2024-09-18T04:25:56.117 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-18T04:25:56.133 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-18T04:25:56.224 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-17_22:16:03-rbd-reef-distro-default-smithi/7909789 branch: reef description: rbd/device/{base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{ubuntu_20.04} thrashers/default thrashosds-health workloads/rbd_nbd} email: ceph-qa@ceph.com first_in_suite: false job_id: '7909789' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-17_22:16:03-rbd-reef-distro-default-smithi no_nested_subset: false openstack: - machine: cpus: 1 disk: 40 ram: 8000 volumes: count: 4 size: 30 os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: reef ceph: conf: global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from sha1: 19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: extra_packages: - rbd-nbd flavor: default sha1: 19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: reef sha1: a477b1894d5c5a613f23260ca0e6b96d79652ba7 owner: scheduled_teuthology@teuthology priority: 930 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mgr.x - osd.3 - osd.4 - osd.5 - - client.0 seed: 4623 sha1: 19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 sleep_before_teardown: 0 subset: 80/128 suite: rbd suite_branch: reef suite_path: /home/teuthworker/src/git.ceph.com_ceph_a477b1894d5c5a613f23260ca0e6b96d79652ba7/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: a477b1894d5c5a613f23260ca0e6b96d79652ba7 targets: smithi028.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= smithi082.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= smithi115.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= tasks: - install: null - ceph: null - thrashosds: timeout: 1200 - workunit: clients: client.0: - rbd/rbd-nbd.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-17_22:16:03 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-18T04:25:56.225 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_a477b1894d5c5a613f23260ca0e6b96d79652ba7/qa; will attempt to use it 2024-09-18T04:25:56.225 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_a477b1894d5c5a613f23260ca0e6b96d79652ba7/qa/tasks 2024-09-18T04:25:56.226 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-18T04:25:56.227 INFO:teuthology.task.internal:Checking packages... 2024-09-18T04:25:56.249 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '19e2a2beb6f41a3ab62650e38614eb29b59dcbb9' 2024-09-18T04:25:56.249 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-18T04:25:56.250 INFO:teuthology.packaging:ref: None 2024-09-18T04:25:56.250 INFO:teuthology.packaging:tag: None 2024-09-18T04:25:56.250 INFO:teuthology.packaging:branch: reef 2024-09-18T04:25:56.250 INFO:teuthology.packaging:sha1: 19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-18T04:25:56.250 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=reef 2024-09-18T04:25:56.524 INFO:teuthology.task.internal:Found packages for ceph version 18.2.4-855-g19e2a2be-1focal 2024-09-18T04:25:56.526 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-18T04:25:56.532 INFO:teuthology.task.internal:no buildpackages task found 2024-09-18T04:25:56.532 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-18T04:25:56.555 INFO:teuthology.task.internal:Saving configuration 2024-09-18T04:25:56.567 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-18T04:25:56.573 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-18T04:25:56.597 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi028.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-17_22:16:03-rbd-reef-distro-default-smithi/7909789', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-18 04:19:59.565505', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2024-09-18T04:25:56.619 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi082.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-17_22:16:03-rbd-reef-distro-default-smithi/7909789', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-18 04:19:59.563606', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2024-09-18T04:25:56.639 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi115.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-17_22:16:03-rbd-reef-distro-default-smithi/7909789', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-18 04:19:59.567215', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2024-09-18T04:25:56.639 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-18T04:25:56.645 INFO:teuthology.task.internal:roles: ubuntu@smithi028.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'] 2024-09-18T04:25:56.645 INFO:teuthology.task.internal:roles: ubuntu@smithi082.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5'] 2024-09-18T04:25:56.645 INFO:teuthology.task.internal:roles: ubuntu@smithi115.front.sepia.ceph.com - ['client.0'] 2024-09-18T04:25:56.645 INFO:teuthology.run_tasks:Running task console_log... 2024-09-18T04:25:56.726 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f95f0f5b130>, signals=[15]) 2024-09-18T04:25:56.726 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-18T04:25:56.732 INFO:teuthology.task.internal:Opening connections... 2024-09-18T04:25:56.732 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi028.front.sepia.ceph.com 2024-09-18T04:25:56.734 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T04:25:56.823 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi082.front.sepia.ceph.com 2024-09-18T04:25:56.824 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T04:25:56.911 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi115.front.sepia.ceph.com 2024-09-18T04:25:56.912 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T04:25:57.000 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-18T04:25:57.007 DEBUG:teuthology.orchestra.run.smithi028:> uname -m 2024-09-18T04:25:57.013 INFO:teuthology.orchestra.run.smithi028.stdout:x86_64 2024-09-18T04:25:57.014 DEBUG:teuthology.orchestra.run.smithi028:> cat /etc/os-release 2024-09-18T04:25:57.061 INFO:teuthology.orchestra.run.smithi028.stdout:NAME="Ubuntu" 2024-09-18T04:25:57.061 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2024-09-18T04:25:57.061 INFO:teuthology.orchestra.run.smithi028.stdout:ID=ubuntu 2024-09-18T04:25:57.061 INFO:teuthology.orchestra.run.smithi028.stdout:ID_LIKE=debian 2024-09-18T04:25:57.061 INFO:teuthology.orchestra.run.smithi028.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2024-09-18T04:25:57.062 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION_ID="20.04" 2024-09-18T04:25:57.062 INFO:teuthology.orchestra.run.smithi028.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-18T04:25:57.062 INFO:teuthology.orchestra.run.smithi028.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-18T04:25:57.062 INFO:teuthology.orchestra.run.smithi028.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-18T04:25:57.062 INFO:teuthology.orchestra.run.smithi028.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-18T04:25:57.062 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION_CODENAME=focal 2024-09-18T04:25:57.062 INFO:teuthology.orchestra.run.smithi028.stdout:UBUNTU_CODENAME=focal 2024-09-18T04:25:57.063 INFO:teuthology.lock.ops:Updating smithi028.front.sepia.ceph.com on lock server 2024-09-18T04:25:57.085 DEBUG:teuthology.orchestra.run.smithi082:> uname -m 2024-09-18T04:25:57.092 INFO:teuthology.orchestra.run.smithi082.stdout:x86_64 2024-09-18T04:25:57.092 DEBUG:teuthology.orchestra.run.smithi082:> cat /etc/os-release 2024-09-18T04:25:57.139 INFO:teuthology.orchestra.run.smithi082.stdout:NAME="Ubuntu" 2024-09-18T04:25:57.139 INFO:teuthology.orchestra.run.smithi082.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2024-09-18T04:25:57.139 INFO:teuthology.orchestra.run.smithi082.stdout:ID=ubuntu 2024-09-18T04:25:57.139 INFO:teuthology.orchestra.run.smithi082.stdout:ID_LIKE=debian 2024-09-18T04:25:57.139 INFO:teuthology.orchestra.run.smithi082.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2024-09-18T04:25:57.139 INFO:teuthology.orchestra.run.smithi082.stdout:VERSION_ID="20.04" 2024-09-18T04:25:57.139 INFO:teuthology.orchestra.run.smithi082.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-18T04:25:57.140 INFO:teuthology.orchestra.run.smithi082.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-18T04:25:57.140 INFO:teuthology.orchestra.run.smithi082.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-18T04:25:57.140 INFO:teuthology.orchestra.run.smithi082.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-18T04:25:57.140 INFO:teuthology.orchestra.run.smithi082.stdout:VERSION_CODENAME=focal 2024-09-18T04:25:57.140 INFO:teuthology.orchestra.run.smithi082.stdout:UBUNTU_CODENAME=focal 2024-09-18T04:25:57.140 INFO:teuthology.lock.ops:Updating smithi082.front.sepia.ceph.com on lock server 2024-09-18T04:25:57.159 DEBUG:teuthology.orchestra.run.smithi115:> uname -m 2024-09-18T04:25:57.166 INFO:teuthology.orchestra.run.smithi115.stdout:x86_64 2024-09-18T04:25:57.166 DEBUG:teuthology.orchestra.run.smithi115:> cat /etc/os-release 2024-09-18T04:25:57.215 INFO:teuthology.orchestra.run.smithi115.stdout:NAME="Ubuntu" 2024-09-18T04:25:57.215 INFO:teuthology.orchestra.run.smithi115.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2024-09-18T04:25:57.216 INFO:teuthology.orchestra.run.smithi115.stdout:ID=ubuntu 2024-09-18T04:25:57.216 INFO:teuthology.orchestra.run.smithi115.stdout:ID_LIKE=debian 2024-09-18T04:25:57.216 INFO:teuthology.orchestra.run.smithi115.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2024-09-18T04:25:57.216 INFO:teuthology.orchestra.run.smithi115.stdout:VERSION_ID="20.04" 2024-09-18T04:25:57.216 INFO:teuthology.orchestra.run.smithi115.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-18T04:25:57.216 INFO:teuthology.orchestra.run.smithi115.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-18T04:25:57.216 INFO:teuthology.orchestra.run.smithi115.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-18T04:25:57.216 INFO:teuthology.orchestra.run.smithi115.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-18T04:25:57.216 INFO:teuthology.orchestra.run.smithi115.stdout:VERSION_CODENAME=focal 2024-09-18T04:25:57.216 INFO:teuthology.orchestra.run.smithi115.stdout:UBUNTU_CODENAME=focal 2024-09-18T04:25:57.217 INFO:teuthology.lock.ops:Updating smithi115.front.sepia.ceph.com on lock server 2024-09-18T04:25:57.235 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-18T04:25:57.246 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-18T04:25:57.251 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-18T04:25:57.251 DEBUG:teuthology.orchestra.run.smithi028:> test '!' -e /home/ubuntu/cephtest 2024-09-18T04:25:57.253 DEBUG:teuthology.orchestra.run.smithi082:> test '!' -e /home/ubuntu/cephtest 2024-09-18T04:25:57.256 DEBUG:teuthology.orchestra.run.smithi115:> test '!' -e /home/ubuntu/cephtest 2024-09-18T04:25:57.261 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-18T04:25:57.267 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-18T04:25:57.267 DEBUG:teuthology.orchestra.run.smithi028:> test -z $(ls -A /var/lib/ceph) 2024-09-18T04:25:57.269 DEBUG:teuthology.orchestra.run.smithi082:> test -z $(ls -A /var/lib/ceph) 2024-09-18T04:25:57.271 DEBUG:teuthology.orchestra.run.smithi115:> test -z $(ls -A /var/lib/ceph) 2024-09-18T04:25:57.297 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-18T04:25:57.374 INFO:teuthology.run_tasks:Running task kernel... 2024-09-18T04:25:57.388 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-18T04:25:57.389 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}, 'client.0': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-18T04:25:57.389 DEBUG:teuthology.orchestra.run.smithi028:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-18T04:25:57.389 DEBUG:teuthology.orchestra.run.smithi082:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-18T04:25:57.391 DEBUG:teuthology.orchestra.run.smithi115:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-18T04:25:57.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T04:25:57.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T04:25:57.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T04:25:57.399 DEBUG:teuthology.orchestra.run.smithi082:> uname -r 2024-09-18T04:25:57.399 DEBUG:teuthology.orchestra.run.smithi115:> uname -r 2024-09-18T04:25:57.400 DEBUG:teuthology.orchestra.run.smithi028:> uname -r 2024-09-18T04:25:57.406 INFO:teuthology.orchestra.run.smithi115.stdout:5.4.0-124-generic 2024-09-18T04:25:57.406 INFO:teuthology.task.kernel:Running kernel on smithi115: 5.4.0-124-generic 2024-09-18T04:25:57.406 DEBUG:teuthology.orchestra.run.smithi115:> sudo apt-get clean 2024-09-18T04:25:57.408 INFO:teuthology.orchestra.run.smithi028.stdout:5.4.0-124-generic 2024-09-18T04:25:57.408 INFO:teuthology.orchestra.run.smithi082.stdout:5.4.0-124-generic 2024-09-18T04:25:57.408 INFO:teuthology.task.kernel:Running kernel on smithi028: 5.4.0-124-generic 2024-09-18T04:25:57.408 DEBUG:teuthology.orchestra.run.smithi028:> sudo apt-get clean 2024-09-18T04:25:57.409 INFO:teuthology.task.kernel:Running kernel on smithi082: 5.4.0-124-generic 2024-09-18T04:25:57.409 DEBUG:teuthology.orchestra.run.smithi082:> sudo apt-get clean 2024-09-18T04:25:57.505 DEBUG:teuthology.orchestra.run.smithi115:> sudo apt-get update 2024-09-18T04:25:57.550 DEBUG:teuthology.orchestra.run.smithi082:> sudo apt-get update 2024-09-18T04:25:57.554 DEBUG:teuthology.orchestra.run.smithi028:> sudo apt-get update 2024-09-18T04:25:57.703 INFO:teuthology.orchestra.run.smithi082.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [128 kB] 2024-09-18T04:25:57.719 INFO:teuthology.orchestra.run.smithi028.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-18T04:25:57.732 INFO:teuthology.orchestra.run.smithi028.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [128 kB] 2024-09-18T04:25:57.733 INFO:teuthology.orchestra.run.smithi115.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-18T04:25:57.733 INFO:teuthology.orchestra.run.smithi115.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [128 kB] 2024-09-18T04:25:57.733 INFO:teuthology.orchestra.run.smithi115.stdout:Get:3 http://security.ubuntu.com/ubuntu focal-security InRelease [128 kB] 2024-09-18T04:25:57.783 INFO:teuthology.orchestra.run.smithi115.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [128 kB] 2024-09-18T04:25:57.788 INFO:teuthology.orchestra.run.smithi028.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease [128 kB] 2024-09-18T04:25:57.819 INFO:teuthology.orchestra.run.smithi082.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-18T04:25:57.826 INFO:teuthology.orchestra.run.smithi028.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security InRelease [128 kB] 2024-09-18T04:25:57.901 INFO:teuthology.orchestra.run.smithi082.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [128 kB] 2024-09-18T04:25:57.986 INFO:teuthology.orchestra.run.smithi082.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [817 kB] 2024-09-18T04:25:58.072 INFO:teuthology.orchestra.run.smithi082.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [3,223 kB] 2024-09-18T04:25:58.099 INFO:teuthology.orchestra.run.smithi028.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3,598 kB] 2024-09-18T04:25:58.113 INFO:teuthology.orchestra.run.smithi082.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [475 kB] 2024-09-18T04:25:58.117 INFO:teuthology.orchestra.run.smithi082.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [14.3 kB] 2024-09-18T04:25:58.117 INFO:teuthology.orchestra.run.smithi082.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [3,152 kB] 2024-09-18T04:25:58.118 INFO:teuthology.orchestra.run.smithi115.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [817 kB] 2024-09-18T04:25:58.137 INFO:teuthology.orchestra.run.smithi082.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [38.3 kB] 2024-09-18T04:25:58.137 INFO:teuthology.orchestra.run.smithi082.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [441 kB] 2024-09-18T04:25:58.141 INFO:teuthology.orchestra.run.smithi082.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [548 B] 2024-09-18T04:25:58.141 INFO:teuthology.orchestra.run.smithi082.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [681 kB] 2024-09-18T04:25:58.145 INFO:teuthology.orchestra.run.smithi082.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [1,012 kB] 2024-09-18T04:25:58.152 INFO:teuthology.orchestra.run.smithi082.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [213 kB] 2024-09-18T04:25:58.153 INFO:teuthology.orchestra.run.smithi082.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [21.4 kB] 2024-09-18T04:25:58.153 INFO:teuthology.orchestra.run.smithi082.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,204 B] 2024-09-18T04:25:58.153 INFO:teuthology.orchestra.run.smithi082.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [24.8 kB] 2024-09-18T04:25:58.165 INFO:teuthology.orchestra.run.smithi082.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,968 B] 2024-09-18T04:25:58.165 INFO:teuthology.orchestra.run.smithi082.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [540 B] 2024-09-18T04:25:58.209 INFO:teuthology.orchestra.run.smithi115.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [3,223 kB] 2024-09-18T04:25:58.251 INFO:teuthology.orchestra.run.smithi028.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [1,036 kB] 2024-09-18T04:25:58.251 INFO:teuthology.orchestra.run.smithi115.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [475 kB] 2024-09-18T04:25:58.252 INFO:teuthology.orchestra.run.smithi115.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3,598 kB] 2024-09-18T04:25:58.253 INFO:teuthology.orchestra.run.smithi082.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-backports InRelease [128 kB] 2024-09-18T04:25:58.255 INFO:teuthology.orchestra.run.smithi115.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [14.3 kB] 2024-09-18T04:25:58.255 INFO:teuthology.orchestra.run.smithi115.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [38.3 kB] 2024-09-18T04:25:58.256 INFO:teuthology.orchestra.run.smithi115.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [3,152 kB] 2024-09-18T04:25:58.266 INFO:teuthology.orchestra.run.smithi028.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [553 kB] 2024-09-18T04:25:58.270 INFO:teuthology.orchestra.run.smithi028.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.7 kB] 2024-09-18T04:25:58.271 INFO:teuthology.orchestra.run.smithi028.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [3,279 kB] 2024-09-18T04:25:58.280 INFO:teuthology.orchestra.run.smithi115.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [441 kB] 2024-09-18T04:25:58.283 INFO:teuthology.orchestra.run.smithi115.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [548 B] 2024-09-18T04:25:58.283 INFO:teuthology.orchestra.run.smithi115.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [1,012 kB] 2024-09-18T04:25:58.290 INFO:teuthology.orchestra.run.smithi115.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [681 kB] 2024-09-18T04:25:58.294 INFO:teuthology.orchestra.run.smithi028.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [39.6 kB] 2024-09-18T04:25:58.295 INFO:teuthology.orchestra.run.smithi028.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [459 kB] 2024-09-18T04:25:58.295 INFO:teuthology.orchestra.run.smithi115.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [213 kB] 2024-09-18T04:25:58.297 INFO:teuthology.orchestra.run.smithi115.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [21.4 kB] 2024-09-18T04:25:58.297 INFO:teuthology.orchestra.run.smithi115.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,204 B] 2024-09-18T04:25:58.297 INFO:teuthology.orchestra.run.smithi115.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [24.8 kB] 2024-09-18T04:25:58.298 INFO:teuthology.orchestra.run.smithi115.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,968 B] 2024-09-18T04:25:58.298 INFO:teuthology.orchestra.run.smithi028.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [548 B] 2024-09-18T04:25:58.298 INFO:teuthology.orchestra.run.smithi028.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [808 kB] 2024-09-18T04:25:58.303 INFO:teuthology.orchestra.run.smithi115.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [540 B] 2024-09-18T04:25:58.304 INFO:teuthology.orchestra.run.smithi028.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,231 kB] 2024-09-18T04:25:58.312 INFO:teuthology.orchestra.run.smithi028.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [295 kB] 2024-09-18T04:25:58.314 INFO:teuthology.orchestra.run.smithi028.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [28.3 kB] 2024-09-18T04:25:58.314 INFO:teuthology.orchestra.run.smithi028.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2024-09-18T04:25:58.315 INFO:teuthology.orchestra.run.smithi028.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [27.1 kB] 2024-09-18T04:25:58.317 INFO:teuthology.orchestra.run.smithi028.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,936 B] 2024-09-18T04:25:58.317 INFO:teuthology.orchestra.run.smithi028.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [616 B] 2024-09-18T04:25:58.317 INFO:teuthology.orchestra.run.smithi028.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2024-09-18T04:25:58.317 INFO:teuthology.orchestra.run.smithi028.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2024-09-18T04:25:58.334 INFO:teuthology.orchestra.run.smithi028.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2024-09-18T04:25:58.334 INFO:teuthology.orchestra.run.smithi028.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2024-09-18T04:25:58.335 INFO:teuthology.orchestra.run.smithi028.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2024-09-18T04:25:58.335 INFO:teuthology.orchestra.run.smithi028.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2024-09-18T04:25:58.367 INFO:teuthology.orchestra.run.smithi115.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [1,036 kB] 2024-09-18T04:25:58.377 INFO:teuthology.orchestra.run.smithi115.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [553 kB] 2024-09-18T04:25:58.382 INFO:teuthology.orchestra.run.smithi115.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.7 kB] 2024-09-18T04:25:58.382 INFO:teuthology.orchestra.run.smithi115.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [39.6 kB] 2024-09-18T04:25:58.382 INFO:teuthology.orchestra.run.smithi115.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [3,279 kB] 2024-09-18T04:25:58.393 INFO:teuthology.orchestra.run.smithi082.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3,598 kB] 2024-09-18T04:25:58.403 INFO:teuthology.orchestra.run.smithi115.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [459 kB] 2024-09-18T04:25:58.407 INFO:teuthology.orchestra.run.smithi115.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [548 B] 2024-09-18T04:25:58.407 INFO:teuthology.orchestra.run.smithi115.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,231 kB] 2024-09-18T04:25:58.415 INFO:teuthology.orchestra.run.smithi115.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [808 kB] 2024-09-18T04:25:58.420 INFO:teuthology.orchestra.run.smithi115.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [295 kB] 2024-09-18T04:25:58.422 INFO:teuthology.orchestra.run.smithi115.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [28.3 kB] 2024-09-18T04:25:58.423 INFO:teuthology.orchestra.run.smithi115.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [27.1 kB] 2024-09-18T04:25:58.423 INFO:teuthology.orchestra.run.smithi115.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2024-09-18T04:25:58.423 INFO:teuthology.orchestra.run.smithi115.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,936 B] 2024-09-18T04:25:58.436 INFO:teuthology.orchestra.run.smithi115.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [616 B] 2024-09-18T04:25:58.436 INFO:teuthology.orchestra.run.smithi115.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2024-09-18T04:25:58.436 INFO:teuthology.orchestra.run.smithi115.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2024-09-18T04:25:58.437 INFO:teuthology.orchestra.run.smithi115.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2024-09-18T04:25:58.438 INFO:teuthology.orchestra.run.smithi028.stdout:Get:27 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [3,223 kB] 2024-09-18T04:25:58.453 INFO:teuthology.orchestra.run.smithi115.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2024-09-18T04:25:58.453 INFO:teuthology.orchestra.run.smithi115.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2024-09-18T04:25:58.453 INFO:teuthology.orchestra.run.smithi115.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2024-09-18T04:25:58.778 INFO:teuthology.orchestra.run.smithi082.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [1,036 kB] 2024-09-18T04:25:58.805 INFO:teuthology.orchestra.run.smithi082.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [553 kB] 2024-09-18T04:25:58.818 INFO:teuthology.orchestra.run.smithi082.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.7 kB] 2024-09-18T04:25:58.819 INFO:teuthology.orchestra.run.smithi082.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [39.6 kB] 2024-09-18T04:25:58.820 INFO:teuthology.orchestra.run.smithi082.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [3,279 kB] 2024-09-18T04:25:58.917 INFO:teuthology.orchestra.run.smithi082.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [459 kB] 2024-09-18T04:25:58.926 INFO:teuthology.orchestra.run.smithi028.stdout:Get:28 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [817 kB] 2024-09-18T04:25:58.935 INFO:teuthology.orchestra.run.smithi082.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [548 B] 2024-09-18T04:25:58.936 INFO:teuthology.orchestra.run.smithi082.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,231 kB] 2024-09-18T04:25:58.940 INFO:teuthology.orchestra.run.smithi028.stdout:Get:29 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [475 kB] 2024-09-18T04:25:58.949 INFO:teuthology.orchestra.run.smithi028.stdout:Get:30 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [14.3 kB] 2024-09-18T04:25:58.949 INFO:teuthology.orchestra.run.smithi028.stdout:Get:31 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [38.3 kB] 2024-09-18T04:25:58.950 INFO:teuthology.orchestra.run.smithi028.stdout:Get:32 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [3,152 kB] 2024-09-18T04:25:58.969 INFO:teuthology.orchestra.run.smithi082.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [808 kB] 2024-09-18T04:25:58.985 INFO:teuthology.orchestra.run.smithi082.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [295 kB] 2024-09-18T04:25:58.990 INFO:teuthology.orchestra.run.smithi082.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [28.3 kB] 2024-09-18T04:25:58.991 INFO:teuthology.orchestra.run.smithi082.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2024-09-18T04:25:58.991 INFO:teuthology.orchestra.run.smithi082.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [27.1 kB] 2024-09-18T04:25:58.992 INFO:teuthology.orchestra.run.smithi082.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,936 B] 2024-09-18T04:25:59.039 INFO:teuthology.orchestra.run.smithi082.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [616 B] 2024-09-18T04:25:59.039 INFO:teuthology.orchestra.run.smithi082.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2024-09-18T04:25:59.040 INFO:teuthology.orchestra.run.smithi082.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2024-09-18T04:25:59.048 INFO:teuthology.orchestra.run.smithi028.stdout:Get:33 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [441 kB] 2024-09-18T04:25:59.073 INFO:teuthology.orchestra.run.smithi028.stdout:Get:34 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [548 B] 2024-09-18T04:25:59.073 INFO:teuthology.orchestra.run.smithi028.stdout:Get:35 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [1,012 kB] 2024-09-18T04:25:59.104 INFO:teuthology.orchestra.run.smithi028.stdout:Get:36 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [681 kB] 2024-09-18T04:25:59.109 INFO:teuthology.orchestra.run.smithi082.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2024-09-18T04:25:59.109 INFO:teuthology.orchestra.run.smithi082.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2024-09-18T04:25:59.110 INFO:teuthology.orchestra.run.smithi082.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2024-09-18T04:25:59.110 INFO:teuthology.orchestra.run.smithi082.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2024-09-18T04:25:59.112 INFO:teuthology.orchestra.run.smithi028.stdout:Get:37 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [213 kB] 2024-09-18T04:25:59.115 INFO:teuthology.orchestra.run.smithi028.stdout:Get:38 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [21.4 kB] 2024-09-18T04:25:59.115 INFO:teuthology.orchestra.run.smithi028.stdout:Get:39 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [24.8 kB] 2024-09-18T04:25:59.115 INFO:teuthology.orchestra.run.smithi028.stdout:Get:40 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,204 B] 2024-09-18T04:25:59.115 INFO:teuthology.orchestra.run.smithi028.stdout:Get:41 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,968 B] 2024-09-18T04:25:59.191 INFO:teuthology.orchestra.run.smithi028.stdout:Get:42 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [540 B] 2024-09-18T04:26:02.588 INFO:teuthology.orchestra.run.smithi115.stdout:Fetched 22.0 MB in 3s (7,189 kB/s) 2024-09-18T04:26:02.589 INFO:teuthology.orchestra.run.smithi082.stdout:Fetched 22.0 MB in 3s (7,559 kB/s) 2024-09-18T04:26:02.921 INFO:teuthology.orchestra.run.smithi028.stdout:Fetched 22.0 MB in 3s (6,777 kB/s) 2024-09-18T04:26:04.169 INFO:teuthology.orchestra.run.smithi115.stdout:Reading package lists... 2024-09-18T04:26:04.195 DEBUG:teuthology.orchestra.run.smithi115:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-18T04:26:04.244 INFO:teuthology.orchestra.run.smithi082.stdout:Reading package lists... 2024-09-18T04:26:04.266 INFO:teuthology.orchestra.run.smithi115.stdout:Reading package lists... 2024-09-18T04:26:04.271 DEBUG:teuthology.orchestra.run.smithi082:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-18T04:26:04.360 INFO:teuthology.orchestra.run.smithi082.stdout:Reading package lists... 2024-09-18T04:26:04.487 INFO:teuthology.orchestra.run.smithi115.stdout:Building dependency tree... 2024-09-18T04:26:04.488 INFO:teuthology.orchestra.run.smithi115.stdout:Reading state information... 2024-09-18T04:26:04.548 INFO:teuthology.orchestra.run.smithi028.stdout:Reading package lists... 2024-09-18T04:26:04.577 DEBUG:teuthology.orchestra.run.smithi028:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-18T04:26:04.581 INFO:teuthology.orchestra.run.smithi082.stdout:Building dependency tree... 2024-09-18T04:26:04.581 INFO:teuthology.orchestra.run.smithi082.stdout:Reading state information... 2024-09-18T04:26:04.667 INFO:teuthology.orchestra.run.smithi028.stdout:Reading package lists... 2024-09-18T04:26:04.692 INFO:teuthology.orchestra.run.smithi115.stdout:The following additional packages will be installed: 2024-09-18T04:26:04.692 INFO:teuthology.orchestra.run.smithi115.stdout: linux-generic linux-headers-5.4.0-196 linux-headers-5.4.0-196-generic 2024-09-18T04:26:04.693 INFO:teuthology.orchestra.run.smithi115.stdout: linux-headers-generic linux-image-5.4.0-196-generic 2024-09-18T04:26:04.693 INFO:teuthology.orchestra.run.smithi115.stdout: linux-modules-5.4.0-196-generic linux-modules-extra-5.4.0-196-generic 2024-09-18T04:26:04.694 INFO:teuthology.orchestra.run.smithi115.stdout:Suggested packages: 2024-09-18T04:26:04.694 INFO:teuthology.orchestra.run.smithi115.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2024-09-18T04:26:04.694 INFO:teuthology.orchestra.run.smithi115.stdout:Recommended packages: 2024-09-18T04:26:04.694 INFO:teuthology.orchestra.run.smithi115.stdout: thermald 2024-09-18T04:26:04.742 INFO:teuthology.orchestra.run.smithi115.stdout:The following NEW packages will be installed: 2024-09-18T04:26:04.742 INFO:teuthology.orchestra.run.smithi115.stdout: linux-headers-5.4.0-196 linux-headers-5.4.0-196-generic 2024-09-18T04:26:04.742 INFO:teuthology.orchestra.run.smithi115.stdout: linux-image-5.4.0-196-generic linux-modules-5.4.0-196-generic 2024-09-18T04:26:04.743 INFO:teuthology.orchestra.run.smithi115.stdout: linux-modules-extra-5.4.0-196-generic 2024-09-18T04:26:04.744 INFO:teuthology.orchestra.run.smithi115.stdout:The following packages will be upgraded: 2024-09-18T04:26:04.744 INFO:teuthology.orchestra.run.smithi115.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-18T04:26:04.792 INFO:teuthology.orchestra.run.smithi082.stdout:The following additional packages will be installed: 2024-09-18T04:26:04.792 INFO:teuthology.orchestra.run.smithi082.stdout: linux-generic linux-headers-5.4.0-196 linux-headers-5.4.0-196-generic 2024-09-18T04:26:04.793 INFO:teuthology.orchestra.run.smithi082.stdout: linux-headers-generic linux-image-5.4.0-196-generic 2024-09-18T04:26:04.793 INFO:teuthology.orchestra.run.smithi082.stdout: linux-modules-5.4.0-196-generic linux-modules-extra-5.4.0-196-generic 2024-09-18T04:26:04.794 INFO:teuthology.orchestra.run.smithi082.stdout:Suggested packages: 2024-09-18T04:26:04.794 INFO:teuthology.orchestra.run.smithi082.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2024-09-18T04:26:04.794 INFO:teuthology.orchestra.run.smithi082.stdout:Recommended packages: 2024-09-18T04:26:04.794 INFO:teuthology.orchestra.run.smithi082.stdout: thermald 2024-09-18T04:26:04.798 INFO:teuthology.orchestra.run.smithi115.stdout:3 upgraded, 5 newly installed, 0 to remove and 316 not upgraded. 2024-09-18T04:26:04.798 INFO:teuthology.orchestra.run.smithi115.stdout:Need to get 77.2 MB of archives. 2024-09-18T04:26:04.799 INFO:teuthology.orchestra.run.smithi115.stdout:After this operation, 381 MB of additional disk space will be used. 2024-09-18T04:26:04.799 INFO:teuthology.orchestra.run.smithi115.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-196-generic amd64 5.4.0-196.216 [15.0 MB] 2024-09-18T04:26:04.841 INFO:teuthology.orchestra.run.smithi082.stdout:The following NEW packages will be installed: 2024-09-18T04:26:04.841 INFO:teuthology.orchestra.run.smithi082.stdout: linux-headers-5.4.0-196 linux-headers-5.4.0-196-generic 2024-09-18T04:26:04.841 INFO:teuthology.orchestra.run.smithi082.stdout: linux-image-5.4.0-196-generic linux-modules-5.4.0-196-generic 2024-09-18T04:26:04.841 INFO:teuthology.orchestra.run.smithi082.stdout: linux-modules-extra-5.4.0-196-generic 2024-09-18T04:26:04.842 INFO:teuthology.orchestra.run.smithi082.stdout:The following packages will be upgraded: 2024-09-18T04:26:04.843 INFO:teuthology.orchestra.run.smithi082.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-18T04:26:04.894 INFO:teuthology.orchestra.run.smithi028.stdout:Building dependency tree... 2024-09-18T04:26:04.894 INFO:teuthology.orchestra.run.smithi028.stdout:Reading state information... 2024-09-18T04:26:04.900 INFO:teuthology.orchestra.run.smithi082.stdout:3 upgraded, 5 newly installed, 0 to remove and 316 not upgraded. 2024-09-18T04:26:04.900 INFO:teuthology.orchestra.run.smithi082.stdout:Need to get 77.2 MB of archives. 2024-09-18T04:26:04.900 INFO:teuthology.orchestra.run.smithi082.stdout:After this operation, 381 MB of additional disk space will be used. 2024-09-18T04:26:04.900 INFO:teuthology.orchestra.run.smithi082.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-196-generic amd64 5.4.0-196.216 [15.0 MB] 2024-09-18T04:26:05.085 INFO:teuthology.orchestra.run.smithi115.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-196-generic amd64 5.4.0-196.216 [10.5 MB] 2024-09-18T04:26:05.107 INFO:teuthology.orchestra.run.smithi028.stdout:The following additional packages will be installed: 2024-09-18T04:26:05.107 INFO:teuthology.orchestra.run.smithi028.stdout: linux-generic linux-headers-5.4.0-196 linux-headers-5.4.0-196-generic 2024-09-18T04:26:05.107 INFO:teuthology.orchestra.run.smithi028.stdout: linux-headers-generic linux-image-5.4.0-196-generic 2024-09-18T04:26:05.107 INFO:teuthology.orchestra.run.smithi028.stdout: linux-modules-5.4.0-196-generic linux-modules-extra-5.4.0-196-generic 2024-09-18T04:26:05.108 INFO:teuthology.orchestra.run.smithi028.stdout:Suggested packages: 2024-09-18T04:26:05.109 INFO:teuthology.orchestra.run.smithi028.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2024-09-18T04:26:05.109 INFO:teuthology.orchestra.run.smithi028.stdout:Recommended packages: 2024-09-18T04:26:05.109 INFO:teuthology.orchestra.run.smithi028.stdout: thermald 2024-09-18T04:26:05.168 INFO:teuthology.orchestra.run.smithi028.stdout:The following NEW packages will be installed: 2024-09-18T04:26:05.168 INFO:teuthology.orchestra.run.smithi028.stdout: linux-headers-5.4.0-196 linux-headers-5.4.0-196-generic 2024-09-18T04:26:05.168 INFO:teuthology.orchestra.run.smithi028.stdout: linux-image-5.4.0-196-generic linux-modules-5.4.0-196-generic 2024-09-18T04:26:05.168 INFO:teuthology.orchestra.run.smithi028.stdout: linux-modules-extra-5.4.0-196-generic 2024-09-18T04:26:05.169 INFO:teuthology.orchestra.run.smithi028.stdout:The following packages will be upgraded: 2024-09-18T04:26:05.170 INFO:teuthology.orchestra.run.smithi028.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-18T04:26:05.175 INFO:teuthology.orchestra.run.smithi115.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-196-generic amd64 5.4.0-196.216 [39.2 MB] 2024-09-18T04:26:05.223 INFO:teuthology.orchestra.run.smithi082.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-196-generic amd64 5.4.0-196.216 [10.5 MB] 2024-09-18T04:26:05.317 INFO:teuthology.orchestra.run.smithi082.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-196-generic amd64 5.4.0-196.216 [39.2 MB] 2024-09-18T04:26:05.354 INFO:teuthology.orchestra.run.smithi028.stdout:3 upgraded, 5 newly installed, 0 to remove and 316 not upgraded. 2024-09-18T04:26:05.355 INFO:teuthology.orchestra.run.smithi028.stdout:Need to get 77.2 MB of archives. 2024-09-18T04:26:05.355 INFO:teuthology.orchestra.run.smithi028.stdout:After this operation, 381 MB of additional disk space will be used. 2024-09-18T04:26:05.355 INFO:teuthology.orchestra.run.smithi028.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-196-generic amd64 5.4.0-196.216 [15.0 MB] 2024-09-18T04:26:05.661 INFO:teuthology.orchestra.run.smithi115.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.196.194 [1,900 B] 2024-09-18T04:26:05.661 INFO:teuthology.orchestra.run.smithi115.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.196.194 [2,592 B] 2024-09-18T04:26:05.661 INFO:teuthology.orchestra.run.smithi115.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-196 all 5.4.0-196.216 [11.0 MB] 2024-09-18T04:26:05.757 INFO:teuthology.orchestra.run.smithi115.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-196-generic amd64 5.4.0-196.216 [1,386 kB] 2024-09-18T04:26:05.767 INFO:teuthology.orchestra.run.smithi115.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.196.194 [2,460 B] 2024-09-18T04:26:05.774 INFO:teuthology.orchestra.run.smithi082.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.196.194 [1,900 B] 2024-09-18T04:26:05.774 INFO:teuthology.orchestra.run.smithi082.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.196.194 [2,592 B] 2024-09-18T04:26:05.775 INFO:teuthology.orchestra.run.smithi082.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-196 all 5.4.0-196.216 [11.0 MB] 2024-09-18T04:26:05.867 INFO:teuthology.orchestra.run.smithi082.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-196-generic amd64 5.4.0-196.216 [1,386 kB] 2024-09-18T04:26:05.878 INFO:teuthology.orchestra.run.smithi082.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.196.194 [2,460 B] 2024-09-18T04:26:06.474 INFO:teuthology.orchestra.run.smithi115.stdout:Fetched 77.2 MB in 1s (76.0 MB/s) 2024-09-18T04:26:06.516 INFO:teuthology.orchestra.run.smithi028.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-196-generic amd64 5.4.0-196.216 [10.5 MB] 2024-09-18T04:26:06.603 INFO:teuthology.orchestra.run.smithi082.stdout:Fetched 77.2 MB in 1s (75.1 MB/s) 2024-09-18T04:26:06.661 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package linux-modules-5.4.0-196-generic. 2024-09-18T04:26:06.811 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package linux-modules-5.4.0-196-generic. 2024-09-18T04:26:06.831 INFO:teuthology.orchestra.run.smithi028.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-196-generic amd64 5.4.0-196.216 [39.2 MB] 2024-09-18T04:26:08.124 INFO:teuthology.orchestra.run.smithi028.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.196.194 [1,900 B] 2024-09-18T04:26:08.124 INFO:teuthology.orchestra.run.smithi028.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.196.194 [2,592 B] 2024-09-18T04:26:08.125 INFO:teuthology.orchestra.run.smithi028.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-196 all 5.4.0-196.216 [11.0 MB] 2024-09-18T04:26:08.389 INFO:teuthology.orchestra.run.smithi115.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2024-09-18T04:26:08.394 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../0-linux-modules-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-18T04:26:08.406 INFO:teuthology.orchestra.run.smithi028.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-196-generic amd64 5.4.0-196.216 [1,386 kB] 2024-09-18T04:26:08.450 INFO:teuthology.orchestra.run.smithi028.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.196.194 [2,460 B] 2024-09-18T04:26:08.531 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking linux-modules-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T04:26:08.552 INFO:teuthology.orchestra.run.smithi082.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2024-09-18T04:26:08.556 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../0-linux-modules-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-18T04:26:08.691 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking linux-modules-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T04:26:09.192 INFO:teuthology.orchestra.run.smithi028.stdout:Fetched 77.2 MB in 3s (23.6 MB/s) 2024-09-18T04:26:09.411 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package linux-modules-5.4.0-196-generic. 2024-09-18T04:26:10.507 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package linux-image-5.4.0-196-generic. 2024-09-18T04:26:10.520 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../1-linux-image-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-18T04:26:10.619 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking linux-image-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T04:26:10.624 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package linux-image-5.4.0-196-generic. 2024-09-18T04:26:10.640 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../1-linux-image-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-18T04:26:10.745 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking linux-image-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T04:26:11.122 INFO:teuthology.orchestra.run.smithi028.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2024-09-18T04:26:11.127 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../0-linux-modules-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-18T04:26:11.315 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking linux-modules-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T04:26:11.718 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-196-generic. 2024-09-18T04:26:11.734 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-18T04:26:11.776 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking linux-modules-extra-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T04:26:11.818 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-196-generic. 2024-09-18T04:26:11.832 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-18T04:26:11.877 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking linux-modules-extra-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T04:26:13.173 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package linux-image-5.4.0-196-generic. 2024-09-18T04:26:13.187 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../1-linux-image-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-18T04:26:13.285 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking linux-image-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T04:26:14.435 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-196-generic. 2024-09-18T04:26:14.448 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-18T04:26:14.493 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking linux-modules-extra-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T04:26:16.728 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../3-linux-generic_5.4.0.196.194_amd64.deb ... 2024-09-18T04:26:16.732 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../3-linux-generic_5.4.0.196.194_amd64.deb ... 2024-09-18T04:26:16.884 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking linux-generic (5.4.0.196.194) over (5.4.0.124.125) ... 2024-09-18T04:26:16.887 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking linux-generic (5.4.0.196.194) over (5.4.0.124.125) ... 2024-09-18T04:26:17.206 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.196.194_amd64.deb ... 2024-09-18T04:26:17.241 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.196.194_amd64.deb ... 2024-09-18T04:26:17.357 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking linux-image-generic (5.4.0.196.194) over (5.4.0.124.125) ... 2024-09-18T04:26:17.370 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking linux-image-generic (5.4.0.196.194) over (5.4.0.124.125) ... 2024-09-18T04:26:17.681 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package linux-headers-5.4.0-196. 2024-09-18T04:26:17.696 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../5-linux-headers-5.4.0-196_5.4.0-196.216_all.deb ... 2024-09-18T04:26:17.727 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package linux-headers-5.4.0-196. 2024-09-18T04:26:17.742 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../5-linux-headers-5.4.0-196_5.4.0-196.216_all.deb ... 2024-09-18T04:26:17.785 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking linux-headers-5.4.0-196 (5.4.0-196.216) ... 2024-09-18T04:26:17.903 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking linux-headers-5.4.0-196 (5.4.0-196.216) ... 2024-09-18T04:26:19.572 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../3-linux-generic_5.4.0.196.194_amd64.deb ... 2024-09-18T04:26:19.670 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking linux-generic (5.4.0.196.194) over (5.4.0.124.125) ... 2024-09-18T04:26:20.297 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.196.194_amd64.deb ... 2024-09-18T04:26:20.423 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking linux-image-generic (5.4.0.196.194) over (5.4.0.124.125) ... 2024-09-18T04:26:20.801 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package linux-headers-5.4.0-196. 2024-09-18T04:26:20.816 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../5-linux-headers-5.4.0-196_5.4.0-196.216_all.deb ... 2024-09-18T04:26:20.859 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking linux-headers-5.4.0-196 (5.4.0-196.216) ... 2024-09-18T04:26:23.442 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package linux-headers-5.4.0-196-generic. 2024-09-18T04:26:23.463 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../6-linux-headers-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-18T04:26:23.500 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking linux-headers-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T04:26:23.592 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package linux-headers-5.4.0-196-generic. 2024-09-18T04:26:23.614 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../6-linux-headers-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-18T04:26:23.658 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking linux-headers-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T04:26:26.279 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.196.194_amd64.deb ... 2024-09-18T04:26:26.417 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking linux-headers-generic (5.4.0.196.194) over (5.4.0.124.125) ... 2024-09-18T04:26:26.466 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.196.194_amd64.deb ... 2024-09-18T04:26:26.595 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking linux-headers-generic (5.4.0.196.194) over (5.4.0.124.125) ... 2024-09-18T04:26:26.747 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up linux-headers-5.4.0-196 (5.4.0-196.216) ... 2024-09-18T04:26:26.850 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package linux-headers-5.4.0-196-generic. 2024-09-18T04:26:26.873 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../6-linux-headers-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-18T04:26:26.896 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up linux-headers-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T04:26:26.917 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking linux-headers-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T04:26:26.939 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up linux-headers-5.4.0-196 (5.4.0-196.216) ... 2024-09-18T04:26:27.022 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up linux-headers-generic (5.4.0.196.194) ... 2024-09-18T04:26:27.083 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up linux-headers-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T04:26:27.174 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up linux-modules-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T04:26:27.217 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up linux-headers-generic (5.4.0.196.194) ... 2024-09-18T04:26:27.368 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up linux-modules-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T04:26:28.426 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up linux-image-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T04:26:28.586 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up linux-image-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T04:26:29.713 INFO:teuthology.orchestra.run.smithi082.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-196-generic 2024-09-18T04:26:29.713 INFO:teuthology.orchestra.run.smithi082.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-196-generic 2024-09-18T04:26:29.820 INFO:teuthology.orchestra.run.smithi115.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-196-generic 2024-09-18T04:26:29.820 INFO:teuthology.orchestra.run.smithi115.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-196-generic 2024-09-18T04:26:29.847 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up linux-modules-extra-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T04:26:29.956 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.196.194_amd64.deb ... 2024-09-18T04:26:30.845 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up linux-modules-extra-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T04:26:31.065 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up linux-image-generic (5.4.0.196.194) ... 2024-09-18T04:26:31.166 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up linux-generic (5.4.0.196.194) ... 2024-09-18T04:26:31.218 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking linux-headers-generic (5.4.0.196.194) over (5.4.0.124.125) ... 2024-09-18T04:26:31.267 INFO:teuthology.orchestra.run.smithi082.stdout:Processing triggers for linux-image-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T04:26:31.353 INFO:teuthology.orchestra.run.smithi082.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-18T04:26:31.353 INFO:teuthology.orchestra.run.smithi082.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-196-generic 2024-09-18T04:26:31.605 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up linux-headers-5.4.0-196 (5.4.0-196.216) ... 2024-09-18T04:26:31.714 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up linux-headers-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T04:26:31.865 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up linux-headers-generic (5.4.0.196.194) ... 2024-09-18T04:26:31.991 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up linux-modules-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T04:26:32.062 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up linux-image-generic (5.4.0.196.194) ... 2024-09-18T04:26:32.205 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up linux-generic (5.4.0.196.194) ... 2024-09-18T04:26:32.340 INFO:teuthology.orchestra.run.smithi115.stdout:Processing triggers for linux-image-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T04:26:32.420 INFO:teuthology.orchestra.run.smithi115.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-18T04:26:32.420 INFO:teuthology.orchestra.run.smithi115.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-196-generic 2024-09-18T04:26:33.318 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up linux-image-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T04:26:34.495 INFO:teuthology.orchestra.run.smithi028.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-196-generic 2024-09-18T04:26:34.495 INFO:teuthology.orchestra.run.smithi028.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-196-generic 2024-09-18T04:26:34.637 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up linux-modules-extra-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T04:26:35.897 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up linux-image-generic (5.4.0.196.194) ... 2024-09-18T04:26:36.056 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up linux-generic (5.4.0.196.194) ... 2024-09-18T04:26:36.216 INFO:teuthology.orchestra.run.smithi028.stdout:Processing triggers for linux-image-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T04:26:36.281 INFO:teuthology.orchestra.run.smithi028.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-18T04:26:36.281 INFO:teuthology.orchestra.run.smithi028.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-196-generic 2024-09-18T04:26:52.965 INFO:teuthology.orchestra.run.smithi082.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-18T04:26:52.965 INFO:teuthology.orchestra.run.smithi082.stdout:Sourcing file `/etc/default/grub' 2024-09-18T04:26:52.983 INFO:teuthology.orchestra.run.smithi082.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-18T04:26:53.011 INFO:teuthology.orchestra.run.smithi082.stdout:Generating grub configuration file ... 2024-09-18T04:26:53.384 INFO:teuthology.orchestra.run.smithi082.stdout:Found linux image: /boot/vmlinuz-5.4.0-196-generic 2024-09-18T04:26:53.405 INFO:teuthology.orchestra.run.smithi082.stdout:Found initrd image: /boot/initrd.img-5.4.0-196-generic 2024-09-18T04:26:53.822 INFO:teuthology.orchestra.run.smithi082.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-18T04:26:53.833 INFO:teuthology.orchestra.run.smithi082.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-18T04:26:54.438 INFO:teuthology.orchestra.run.smithi115.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-18T04:26:54.438 INFO:teuthology.orchestra.run.smithi115.stdout:Sourcing file `/etc/default/grub' 2024-09-18T04:26:54.453 INFO:teuthology.orchestra.run.smithi115.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-18T04:26:54.472 INFO:teuthology.orchestra.run.smithi115.stdout:Generating grub configuration file ... 2024-09-18T04:26:54.811 INFO:teuthology.orchestra.run.smithi115.stdout:Found linux image: /boot/vmlinuz-5.4.0-196-generic 2024-09-18T04:26:54.836 INFO:teuthology.orchestra.run.smithi115.stdout:Found initrd image: /boot/initrd.img-5.4.0-196-generic 2024-09-18T04:26:55.237 INFO:teuthology.orchestra.run.smithi115.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-18T04:26:55.248 INFO:teuthology.orchestra.run.smithi115.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-18T04:26:55.961 INFO:teuthology.orchestra.run.smithi082.stdout:done 2024-09-18T04:26:56.409 DEBUG:teuthology.orchestra.run.smithi082:> dpkg -s linux-image-generic 2024-09-18T04:26:56.464 INFO:teuthology.orchestra.run.smithi082.stdout:Package: linux-image-generic 2024-09-18T04:26:56.464 INFO:teuthology.orchestra.run.smithi082.stdout:Status: install ok installed 2024-09-18T04:26:56.464 INFO:teuthology.orchestra.run.smithi082.stdout:Priority: optional 2024-09-18T04:26:56.464 INFO:teuthology.orchestra.run.smithi082.stdout:Section: kernel 2024-09-18T04:26:56.465 INFO:teuthology.orchestra.run.smithi082.stdout:Installed-Size: 20 2024-09-18T04:26:56.465 INFO:teuthology.orchestra.run.smithi082.stdout:Maintainer: Ubuntu Kernel Team 2024-09-18T04:26:56.465 INFO:teuthology.orchestra.run.smithi082.stdout:Architecture: amd64 2024-09-18T04:26:56.465 INFO:teuthology.orchestra.run.smithi082.stdout:Source: linux-meta 2024-09-18T04:26:56.465 INFO:teuthology.orchestra.run.smithi082.stdout:Version: 5.4.0.196.194 2024-09-18T04:26:56.465 INFO:teuthology.orchestra.run.smithi082.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-18T04:26:56.465 INFO:teuthology.orchestra.run.smithi082.stdout:Depends: linux-image-5.4.0-196-generic, linux-modules-extra-5.4.0-196-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-18T04:26:56.465 INFO:teuthology.orchestra.run.smithi082.stdout:Recommends: thermald 2024-09-18T04:26:56.465 INFO:teuthology.orchestra.run.smithi082.stdout:Description: Generic Linux kernel image 2024-09-18T04:26:56.465 INFO:teuthology.orchestra.run.smithi082.stdout: This package will always depend on the latest generic kernel image 2024-09-18T04:26:56.465 INFO:teuthology.orchestra.run.smithi082.stdout: available. 2024-09-18T04:26:56.466 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-196-generic 2024-09-18T04:26:56.466 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-18T04:26:56.466 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-18T04:26:56.466 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-18T04:26:56.466 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi082.front.sepia.ceph.com, path=None, version=distro) 2024-09-18T04:26:56.466 DEBUG:teuthology.orchestra.run.smithi082:> sudo apt-get clean 2024-09-18T04:26:56.545 DEBUG:teuthology.orchestra.run.smithi082:> sudo apt-get update 2024-09-18T04:26:56.674 INFO:teuthology.orchestra.run.smithi082.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-18T04:26:56.681 INFO:teuthology.orchestra.run.smithi082.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-18T04:26:56.698 INFO:teuthology.orchestra.run.smithi082.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-18T04:26:56.792 INFO:teuthology.orchestra.run.smithi082.stdout:Hit:4 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-18T04:26:57.395 INFO:teuthology.orchestra.run.smithi115.stdout:done 2024-09-18T04:26:57.806 DEBUG:teuthology.orchestra.run.smithi115:> dpkg -s linux-image-generic 2024-09-18T04:26:57.858 INFO:teuthology.orchestra.run.smithi115.stdout:Package: linux-image-generic 2024-09-18T04:26:57.858 INFO:teuthology.orchestra.run.smithi115.stdout:Status: install ok installed 2024-09-18T04:26:57.858 INFO:teuthology.orchestra.run.smithi115.stdout:Priority: optional 2024-09-18T04:26:57.858 INFO:teuthology.orchestra.run.smithi115.stdout:Section: kernel 2024-09-18T04:26:57.858 INFO:teuthology.orchestra.run.smithi115.stdout:Installed-Size: 20 2024-09-18T04:26:57.858 INFO:teuthology.orchestra.run.smithi115.stdout:Maintainer: Ubuntu Kernel Team 2024-09-18T04:26:57.858 INFO:teuthology.orchestra.run.smithi115.stdout:Architecture: amd64 2024-09-18T04:26:57.858 INFO:teuthology.orchestra.run.smithi115.stdout:Source: linux-meta 2024-09-18T04:26:57.858 INFO:teuthology.orchestra.run.smithi115.stdout:Version: 5.4.0.196.194 2024-09-18T04:26:57.859 INFO:teuthology.orchestra.run.smithi115.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-18T04:26:57.859 INFO:teuthology.orchestra.run.smithi115.stdout:Depends: linux-image-5.4.0-196-generic, linux-modules-extra-5.4.0-196-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-18T04:26:57.859 INFO:teuthology.orchestra.run.smithi115.stdout:Recommends: thermald 2024-09-18T04:26:57.859 INFO:teuthology.orchestra.run.smithi115.stdout:Description: Generic Linux kernel image 2024-09-18T04:26:57.859 INFO:teuthology.orchestra.run.smithi115.stdout: This package will always depend on the latest generic kernel image 2024-09-18T04:26:57.859 INFO:teuthology.orchestra.run.smithi115.stdout: available. 2024-09-18T04:26:57.859 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-196-generic 2024-09-18T04:26:57.859 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-18T04:26:57.859 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-18T04:26:57.859 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-09-18T04:26:57.859 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi115.front.sepia.ceph.com, path=None, version=distro) 2024-09-18T04:26:57.860 DEBUG:teuthology.orchestra.run.smithi115:> sudo apt-get clean 2024-09-18T04:26:57.932 DEBUG:teuthology.orchestra.run.smithi115:> sudo apt-get update 2024-09-18T04:26:58.198 INFO:teuthology.orchestra.run.smithi115.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-18T04:26:58.198 INFO:teuthology.orchestra.run.smithi115.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-18T04:26:58.268 INFO:teuthology.orchestra.run.smithi115.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-18T04:26:58.349 INFO:teuthology.orchestra.run.smithi115.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-18T04:26:58.520 INFO:teuthology.orchestra.run.smithi082.stdout:Reading package lists... 2024-09-18T04:26:58.525 INFO:teuthology.orchestra.run.smithi028.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-18T04:26:58.525 INFO:teuthology.orchestra.run.smithi028.stdout:Sourcing file `/etc/default/grub' 2024-09-18T04:26:58.540 INFO:teuthology.orchestra.run.smithi028.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-18T04:26:58.546 DEBUG:teuthology.orchestra.run.smithi082:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-18T04:26:58.562 INFO:teuthology.orchestra.run.smithi028.stdout:Generating grub configuration file ... 2024-09-18T04:26:58.617 INFO:teuthology.orchestra.run.smithi082.stdout:Reading package lists... 2024-09-18T04:26:58.821 INFO:teuthology.orchestra.run.smithi082.stdout:Building dependency tree... 2024-09-18T04:26:58.822 INFO:teuthology.orchestra.run.smithi082.stdout:Reading state information... 2024-09-18T04:26:58.881 INFO:teuthology.orchestra.run.smithi028.stdout:Found linux image: /boot/vmlinuz-5.4.0-196-generic 2024-09-18T04:26:58.900 INFO:teuthology.orchestra.run.smithi028.stdout:Found initrd image: /boot/initrd.img-5.4.0-196-generic 2024-09-18T04:26:59.053 INFO:teuthology.orchestra.run.smithi082.stdout:linux-image-generic is already the newest version (5.4.0.196.194). 2024-09-18T04:26:59.053 INFO:teuthology.orchestra.run.smithi082.stdout:0 upgraded, 0 newly installed, 0 to remove and 316 not upgraded. 2024-09-18T04:26:59.055 DEBUG:teuthology.orchestra.run.smithi082:> dpkg -s linux-image-generic 2024-09-18T04:26:59.072 INFO:teuthology.orchestra.run.smithi082.stdout:Package: linux-image-generic 2024-09-18T04:26:59.072 INFO:teuthology.orchestra.run.smithi082.stdout:Status: install ok installed 2024-09-18T04:26:59.072 INFO:teuthology.orchestra.run.smithi082.stdout:Priority: optional 2024-09-18T04:26:59.072 INFO:teuthology.orchestra.run.smithi082.stdout:Section: kernel 2024-09-18T04:26:59.072 INFO:teuthology.orchestra.run.smithi082.stdout:Installed-Size: 20 2024-09-18T04:26:59.073 INFO:teuthology.orchestra.run.smithi082.stdout:Maintainer: Ubuntu Kernel Team 2024-09-18T04:26:59.073 INFO:teuthology.orchestra.run.smithi082.stdout:Architecture: amd64 2024-09-18T04:26:59.073 INFO:teuthology.orchestra.run.smithi082.stdout:Source: linux-meta 2024-09-18T04:26:59.073 INFO:teuthology.orchestra.run.smithi082.stdout:Version: 5.4.0.196.194 2024-09-18T04:26:59.073 INFO:teuthology.orchestra.run.smithi082.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-18T04:26:59.073 INFO:teuthology.orchestra.run.smithi082.stdout:Depends: linux-image-5.4.0-196-generic, linux-modules-extra-5.4.0-196-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-18T04:26:59.073 INFO:teuthology.orchestra.run.smithi082.stdout:Recommends: thermald 2024-09-18T04:26:59.073 INFO:teuthology.orchestra.run.smithi082.stdout:Description: Generic Linux kernel image 2024-09-18T04:26:59.073 INFO:teuthology.orchestra.run.smithi082.stdout: This package will always depend on the latest generic kernel image 2024-09-18T04:26:59.073 INFO:teuthology.orchestra.run.smithi082.stdout: available. 2024-09-18T04:26:59.074 DEBUG:teuthology.orchestra.run.smithi082:> mktemp 2024-09-18T04:26:59.119 INFO:teuthology.orchestra.run.smithi082.stdout:/tmp/tmp.EljSCWnAjA 2024-09-18T04:26:59.120 DEBUG:teuthology.orchestra.run.smithi082:> sudo cp /boot/grub/grub.cfg /tmp/tmp.EljSCWnAjA 2024-09-18T04:26:59.133 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 0666 /tmp/tmp.EljSCWnAjA 2024-09-18T04:26:59.176 DEBUG:teuthology.orchestra.remote:smithi082:/tmp/tmp.EljSCWnAjA is 9KB 2024-09-18T04:26:59.190 DEBUG:teuthology.orchestra.run.smithi082:> rm -fr /tmp/tmp.EljSCWnAjA 2024-09-18T04:26:59.197 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-18T04:26:59.250 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-09-18T04:26:59.251 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-18T04:26:59.251 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-18T04:26:59.280 INFO:teuthology.orchestra.run.smithi028.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-18T04:26:59.289 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-196-generic 2024-09-18T04:26:59.289 DEBUG:teuthology.orchestra.run.smithi082:> sudo update-grub 2024-09-18T04:26:59.292 INFO:teuthology.orchestra.run.smithi028.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-18T04:27:00.045 INFO:teuthology.orchestra.run.smithi115.stdout:Reading package lists... 2024-09-18T04:27:00.070 DEBUG:teuthology.orchestra.run.smithi115:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-18T04:27:00.136 INFO:teuthology.orchestra.run.smithi115.stdout:Reading package lists... 2024-09-18T04:27:00.329 INFO:teuthology.orchestra.run.smithi115.stdout:Building dependency tree... 2024-09-18T04:27:00.329 INFO:teuthology.orchestra.run.smithi115.stdout:Reading state information... 2024-09-18T04:27:00.567 INFO:teuthology.orchestra.run.smithi115.stdout:linux-image-generic is already the newest version (5.4.0.196.194). 2024-09-18T04:27:00.567 INFO:teuthology.orchestra.run.smithi115.stdout:0 upgraded, 0 newly installed, 0 to remove and 316 not upgraded. 2024-09-18T04:27:00.569 DEBUG:teuthology.orchestra.run.smithi115:> dpkg -s linux-image-generic 2024-09-18T04:27:00.590 INFO:teuthology.orchestra.run.smithi115.stdout:Package: linux-image-generic 2024-09-18T04:27:00.590 INFO:teuthology.orchestra.run.smithi115.stdout:Status: install ok installed 2024-09-18T04:27:00.590 INFO:teuthology.orchestra.run.smithi115.stdout:Priority: optional 2024-09-18T04:27:00.590 INFO:teuthology.orchestra.run.smithi115.stdout:Section: kernel 2024-09-18T04:27:00.590 INFO:teuthology.orchestra.run.smithi115.stdout:Installed-Size: 20 2024-09-18T04:27:00.590 INFO:teuthology.orchestra.run.smithi115.stdout:Maintainer: Ubuntu Kernel Team 2024-09-18T04:27:00.590 INFO:teuthology.orchestra.run.smithi115.stdout:Architecture: amd64 2024-09-18T04:27:00.590 INFO:teuthology.orchestra.run.smithi115.stdout:Source: linux-meta 2024-09-18T04:27:00.590 INFO:teuthology.orchestra.run.smithi115.stdout:Version: 5.4.0.196.194 2024-09-18T04:27:00.591 INFO:teuthology.orchestra.run.smithi115.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-18T04:27:00.591 INFO:teuthology.orchestra.run.smithi115.stdout:Depends: linux-image-5.4.0-196-generic, linux-modules-extra-5.4.0-196-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-18T04:27:00.591 INFO:teuthology.orchestra.run.smithi115.stdout:Recommends: thermald 2024-09-18T04:27:00.591 INFO:teuthology.orchestra.run.smithi115.stdout:Description: Generic Linux kernel image 2024-09-18T04:27:00.591 INFO:teuthology.orchestra.run.smithi115.stdout: This package will always depend on the latest generic kernel image 2024-09-18T04:27:00.591 INFO:teuthology.orchestra.run.smithi115.stdout: available. 2024-09-18T04:27:00.591 DEBUG:teuthology.orchestra.run.smithi115:> mktemp 2024-09-18T04:27:00.596 INFO:teuthology.orchestra.run.smithi115.stdout:/tmp/tmp.gxs8a0YluE 2024-09-18T04:27:00.596 DEBUG:teuthology.orchestra.run.smithi115:> sudo cp /boot/grub/grub.cfg /tmp/tmp.gxs8a0YluE 2024-09-18T04:27:00.650 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod 0666 /tmp/tmp.gxs8a0YluE 2024-09-18T04:27:00.691 DEBUG:teuthology.orchestra.remote:smithi115:/tmp/tmp.gxs8a0YluE is 9KB 2024-09-18T04:27:00.704 DEBUG:teuthology.orchestra.run.smithi115:> rm -fr /tmp/tmp.gxs8a0YluE 2024-09-18T04:27:00.711 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-18T04:27:00.746 INFO:teuthology.orchestra.run.smithi082.stderr:Sourcing file `/etc/default/grub' 2024-09-18T04:27:00.756 INFO:teuthology.orchestra.run.smithi082.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-18T04:27:00.767 INFO:teuthology.orchestra.run.smithi082.stderr:Generating grub configuration file ... 2024-09-18T04:27:00.768 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-18T04:27:00.768 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-18T04:27:00.768 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-18T04:27:00.814 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-196-generic 2024-09-18T04:27:00.814 DEBUG:teuthology.orchestra.run.smithi115:> sudo update-grub 2024-09-18T04:27:01.037 INFO:teuthology.orchestra.run.smithi082.stderr:Found linux image: /boot/vmlinuz-5.4.0-196-generic 2024-09-18T04:27:01.049 INFO:teuthology.orchestra.run.smithi082.stderr:Found initrd image: /boot/initrd.img-5.4.0-196-generic 2024-09-18T04:27:01.362 INFO:teuthology.orchestra.run.smithi028.stdout:done 2024-09-18T04:27:01.438 INFO:teuthology.orchestra.run.smithi082.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-18T04:27:01.449 INFO:teuthology.orchestra.run.smithi082.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-18T04:27:01.809 DEBUG:teuthology.orchestra.run.smithi028:> dpkg -s linux-image-generic 2024-09-18T04:27:01.852 INFO:teuthology.orchestra.run.smithi028.stdout:Package: linux-image-generic 2024-09-18T04:27:01.853 INFO:teuthology.orchestra.run.smithi028.stdout:Status: install ok installed 2024-09-18T04:27:01.853 INFO:teuthology.orchestra.run.smithi028.stdout:Priority: optional 2024-09-18T04:27:01.853 INFO:teuthology.orchestra.run.smithi028.stdout:Section: kernel 2024-09-18T04:27:01.853 INFO:teuthology.orchestra.run.smithi028.stdout:Installed-Size: 20 2024-09-18T04:27:01.853 INFO:teuthology.orchestra.run.smithi028.stdout:Maintainer: Ubuntu Kernel Team 2024-09-18T04:27:01.853 INFO:teuthology.orchestra.run.smithi028.stdout:Architecture: amd64 2024-09-18T04:27:01.853 INFO:teuthology.orchestra.run.smithi028.stdout:Source: linux-meta 2024-09-18T04:27:01.853 INFO:teuthology.orchestra.run.smithi028.stdout:Version: 5.4.0.196.194 2024-09-18T04:27:01.853 INFO:teuthology.orchestra.run.smithi028.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-18T04:27:01.853 INFO:teuthology.orchestra.run.smithi028.stdout:Depends: linux-image-5.4.0-196-generic, linux-modules-extra-5.4.0-196-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-18T04:27:01.853 INFO:teuthology.orchestra.run.smithi028.stdout:Recommends: thermald 2024-09-18T04:27:01.853 INFO:teuthology.orchestra.run.smithi028.stdout:Description: Generic Linux kernel image 2024-09-18T04:27:01.854 INFO:teuthology.orchestra.run.smithi028.stdout: This package will always depend on the latest generic kernel image 2024-09-18T04:27:01.854 INFO:teuthology.orchestra.run.smithi028.stdout: available. 2024-09-18T04:27:01.855 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-196-generic 2024-09-18T04:27:01.855 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-18T04:27:01.855 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-18T04:27:01.855 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-18T04:27:01.855 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi028.front.sepia.ceph.com, path=None, version=distro) 2024-09-18T04:27:01.855 DEBUG:teuthology.orchestra.run.smithi028:> sudo apt-get clean 2024-09-18T04:27:01.935 DEBUG:teuthology.orchestra.run.smithi028:> sudo apt-get update 2024-09-18T04:27:02.197 INFO:teuthology.orchestra.run.smithi028.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-18T04:27:02.197 INFO:teuthology.orchestra.run.smithi028.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-18T04:27:02.226 INFO:teuthology.orchestra.run.smithi115.stderr:Sourcing file `/etc/default/grub' 2024-09-18T04:27:02.234 INFO:teuthology.orchestra.run.smithi115.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-18T04:27:02.249 INFO:teuthology.orchestra.run.smithi115.stderr:Generating grub configuration file ... 2024-09-18T04:27:02.267 INFO:teuthology.orchestra.run.smithi028.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-18T04:27:02.349 INFO:teuthology.orchestra.run.smithi028.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-18T04:27:02.483 INFO:teuthology.orchestra.run.smithi082.stderr:done 2024-09-18T04:27:02.485 DEBUG:teuthology.orchestra.run.smithi082:> sudo shutdown -r now 2024-09-18T04:27:02.522 INFO:teuthology.orchestra.run.smithi115.stderr:Found linux image: /boot/vmlinuz-5.4.0-196-generic 2024-09-18T04:27:02.534 INFO:teuthology.orchestra.run.smithi115.stderr:Found initrd image: /boot/initrd.img-5.4.0-196-generic 2024-09-18T04:27:02.896 INFO:teuthology.orchestra.run.smithi115.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-18T04:27:02.908 INFO:teuthology.orchestra.run.smithi115.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-18T04:27:03.948 INFO:teuthology.orchestra.run.smithi115.stderr:done 2024-09-18T04:27:03.950 DEBUG:teuthology.orchestra.run.smithi115:> sudo shutdown -r now 2024-09-18T04:27:04.118 INFO:teuthology.orchestra.run.smithi028.stdout:Reading package lists... 2024-09-18T04:27:04.147 DEBUG:teuthology.orchestra.run.smithi028:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-18T04:27:04.221 INFO:teuthology.orchestra.run.smithi028.stdout:Reading package lists... 2024-09-18T04:27:04.422 INFO:teuthology.orchestra.run.smithi028.stdout:Building dependency tree... 2024-09-18T04:27:04.422 INFO:teuthology.orchestra.run.smithi028.stdout:Reading state information... 2024-09-18T04:27:04.663 INFO:teuthology.orchestra.run.smithi028.stdout:linux-image-generic is already the newest version (5.4.0.196.194). 2024-09-18T04:27:04.663 INFO:teuthology.orchestra.run.smithi028.stdout:0 upgraded, 0 newly installed, 0 to remove and 316 not upgraded. 2024-09-18T04:27:04.665 DEBUG:teuthology.orchestra.run.smithi028:> dpkg -s linux-image-generic 2024-09-18T04:27:04.686 INFO:teuthology.orchestra.run.smithi028.stdout:Package: linux-image-generic 2024-09-18T04:27:04.686 INFO:teuthology.orchestra.run.smithi028.stdout:Status: install ok installed 2024-09-18T04:27:04.686 INFO:teuthology.orchestra.run.smithi028.stdout:Priority: optional 2024-09-18T04:27:04.686 INFO:teuthology.orchestra.run.smithi028.stdout:Section: kernel 2024-09-18T04:27:04.686 INFO:teuthology.orchestra.run.smithi028.stdout:Installed-Size: 20 2024-09-18T04:27:04.687 INFO:teuthology.orchestra.run.smithi028.stdout:Maintainer: Ubuntu Kernel Team 2024-09-18T04:27:04.687 INFO:teuthology.orchestra.run.smithi028.stdout:Architecture: amd64 2024-09-18T04:27:04.687 INFO:teuthology.orchestra.run.smithi028.stdout:Source: linux-meta 2024-09-18T04:27:04.687 INFO:teuthology.orchestra.run.smithi028.stdout:Version: 5.4.0.196.194 2024-09-18T04:27:04.687 INFO:teuthology.orchestra.run.smithi028.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-18T04:27:04.687 INFO:teuthology.orchestra.run.smithi028.stdout:Depends: linux-image-5.4.0-196-generic, linux-modules-extra-5.4.0-196-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-18T04:27:04.687 INFO:teuthology.orchestra.run.smithi028.stdout:Recommends: thermald 2024-09-18T04:27:04.687 INFO:teuthology.orchestra.run.smithi028.stdout:Description: Generic Linux kernel image 2024-09-18T04:27:04.687 INFO:teuthology.orchestra.run.smithi028.stdout: This package will always depend on the latest generic kernel image 2024-09-18T04:27:04.687 INFO:teuthology.orchestra.run.smithi028.stdout: available. 2024-09-18T04:27:04.688 DEBUG:teuthology.orchestra.run.smithi028:> mktemp 2024-09-18T04:27:04.693 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/tmp.jMXsBEE1Zh 2024-09-18T04:27:04.693 DEBUG:teuthology.orchestra.run.smithi028:> sudo cp /boot/grub/grub.cfg /tmp/tmp.jMXsBEE1Zh 2024-09-18T04:27:04.715 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0666 /tmp/tmp.jMXsBEE1Zh 2024-09-18T04:27:04.770 DEBUG:teuthology.orchestra.remote:smithi028:/tmp/tmp.jMXsBEE1Zh is 9KB 2024-09-18T04:27:04.783 DEBUG:teuthology.orchestra.run.smithi028:> rm -fr /tmp/tmp.jMXsBEE1Zh 2024-09-18T04:27:04.790 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-18T04:27:04.851 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-18T04:27:04.851 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-18T04:27:04.851 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-18T04:27:04.884 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-196-generic 2024-09-18T04:27:04.884 DEBUG:teuthology.orchestra.run.smithi028:> sudo update-grub 2024-09-18T04:27:06.352 INFO:teuthology.orchestra.run.smithi028.stderr:Sourcing file `/etc/default/grub' 2024-09-18T04:27:06.364 INFO:teuthology.orchestra.run.smithi028.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-18T04:27:06.378 INFO:teuthology.orchestra.run.smithi028.stderr:Generating grub configuration file ... 2024-09-18T04:27:06.666 INFO:teuthology.orchestra.run.smithi028.stderr:Found linux image: /boot/vmlinuz-5.4.0-196-generic 2024-09-18T04:27:06.678 INFO:teuthology.orchestra.run.smithi028.stderr:Found initrd image: /boot/initrd.img-5.4.0-196-generic 2024-09-18T04:27:07.096 INFO:teuthology.orchestra.run.smithi028.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-18T04:27:07.108 INFO:teuthology.orchestra.run.smithi028.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-18T04:27:08.145 INFO:teuthology.orchestra.run.smithi028.stderr:done 2024-09-18T04:27:08.147 DEBUG:teuthology.orchestra.run.smithi028:> sudo shutdown -r now 2024-09-18T04:27:32.512 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-18T04:27:32.513 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-09-18T04:27:32.514 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T04:27:33.998 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-09-18T04:27:33.998 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-09-18T04:27:33.999 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T04:27:38.154 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-18T04:27:38.154 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-09-18T04:27:38.155 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T04:27:50.973 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.82 2024-09-18T04:27:52.509 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.115 2024-09-18T04:27:56.605 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.28 2024-09-18T04:27:59.978 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-09-18T04:27:59.978 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T04:28:01.510 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-09-18T04:28:01.510 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T04:28:03.041 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.82 2024-09-18T04:28:04.573 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.115 2024-09-18T04:28:05.606 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-09-18T04:28:05.607 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T04:28:15.042 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-09-18T04:28:15.043 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T04:28:16.576 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-09-18T04:28:16.577 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T04:28:23.997 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.28 2024-09-18T04:28:33.469 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.82 2024-09-18T04:28:36.000 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-09-18T04:28:36.001 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T04:28:39.069 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.28 2024-09-18T04:28:48.471 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-09-18T04:28:48.471 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T04:28:48.713 DEBUG:teuthology.orchestra.run.smithi082:> true 2024-09-18T04:28:48.755 DEBUG:teuthology.orchestra.run.smithi115:> true 2024-09-18T04:28:49.302 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-09-18T04:28:49.302 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.4.0-196-generic"... 2024-09-18T04:28:49.302 DEBUG:teuthology.orchestra.run.smithi082:> uname -r 2024-09-18T04:28:49.308 INFO:teuthology.orchestra.run.smithi082.stdout:5.4.0-196-generic 2024-09-18T04:28:49.309 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-196-generic vs 5.4.0-196-generic 2024-09-18T04:28:49.309 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-18T04:28:49.309 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-18T04:28:49.671 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-09-18T04:28:49.671 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.4.0-196-generic"... 2024-09-18T04:28:49.671 DEBUG:teuthology.orchestra.run.smithi115:> uname -r 2024-09-18T04:28:49.677 INFO:teuthology.orchestra.run.smithi115.stdout:5.4.0-196-generic 2024-09-18T04:28:49.677 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-196-generic vs 5.4.0-196-generic 2024-09-18T04:28:49.678 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-18T04:28:49.678 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-18T04:28:50.309 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-18T04:28:50.310 DEBUG:teuthology.orchestra.run.smithi082:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-18T04:28:50.406 INFO:teuthology.orchestra.run.smithi082.stdout:ttyS1 2024-09-18T04:28:50.424 DEBUG:teuthology.parallel:result is None 2024-09-18T04:28:50.678 INFO:teuthology.task.kernel:Enabling kdb on client.0... 2024-09-18T04:28:50.679 DEBUG:teuthology.orchestra.run.smithi115:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-18T04:28:50.760 INFO:teuthology.orchestra.run.smithi115.stdout:ttyS1 2024-09-18T04:28:50.791 DEBUG:teuthology.parallel:result is None 2024-09-18T04:28:54.070 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-09-18T04:28:54.070 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T04:28:54.329 DEBUG:teuthology.orchestra.run.smithi028:> true 2024-09-18T04:28:54.886 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-09-18T04:28:54.886 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.4.0-196-generic"... 2024-09-18T04:28:54.886 DEBUG:teuthology.orchestra.run.smithi028:> uname -r 2024-09-18T04:28:54.893 INFO:teuthology.orchestra.run.smithi028.stdout:5.4.0-196-generic 2024-09-18T04:28:54.893 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-196-generic vs 5.4.0-196-generic 2024-09-18T04:28:54.893 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-18T04:28:54.893 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-18T04:28:55.896 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-18T04:28:55.896 DEBUG:teuthology.orchestra.run.smithi028:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-18T04:28:55.973 INFO:teuthology.orchestra.run.smithi028.stdout:ttyS1 2024-09-18T04:28:56.005 DEBUG:teuthology.parallel:result is None 2024-09-18T04:28:56.005 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-18T04:28:56.012 INFO:teuthology.task.internal:Creating test directory... 2024-09-18T04:28:56.012 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-18T04:28:56.014 DEBUG:teuthology.orchestra.run.smithi082:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-18T04:28:56.016 DEBUG:teuthology.orchestra.run.smithi115:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-18T04:28:56.023 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-18T04:28:56.030 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-18T04:28:56.039 INFO:teuthology.task.internal:Creating archive directory... 2024-09-18T04:28:56.040 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-18T04:28:56.042 DEBUG:teuthology.orchestra.run.smithi082:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-18T04:28:56.044 DEBUG:teuthology.orchestra.run.smithi115:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-18T04:28:56.066 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-18T04:28:56.074 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-18T04:28:56.074 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-18T04:28:56.077 DEBUG:teuthology.orchestra.run.smithi082:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-18T04:28:56.093 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T04:28:56.103 DEBUG:teuthology.orchestra.run.smithi115:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-18T04:28:56.105 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T04:28:56.119 INFO:teuthology.orchestra.run.smithi082.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T04:28:56.129 INFO:teuthology.orchestra.run.smithi082.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T04:28:56.131 INFO:teuthology.orchestra.run.smithi115.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T04:28:56.142 INFO:teuthology.orchestra.run.smithi115.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T04:28:56.144 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-18T04:28:56.152 INFO:teuthology.task.internal:Configuring sudo... 2024-09-18T04:28:56.152 DEBUG:teuthology.orchestra.run.smithi028:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-18T04:28:56.154 DEBUG:teuthology.orchestra.run.smithi082:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-18T04:28:56.157 DEBUG:teuthology.orchestra.run.smithi115:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-18T04:28:56.205 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-18T04:28:56.214 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-18T04:28:56.214 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-18T04:28:56.216 DEBUG:teuthology.orchestra.run.smithi082:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-18T04:28:56.218 DEBUG:teuthology.orchestra.run.smithi115:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-18T04:28:56.224 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-18T04:28:56.231 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-18T04:28:56.281 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-18T04:28:56.281 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-18T04:28:56.310 DEBUG:teuthology.orchestra.run.smithi082:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-18T04:28:56.318 DEBUG:teuthology.orchestra.run.smithi082:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-18T04:28:56.326 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-09-18T04:28:56.326 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-18T04:28:56.352 DEBUG:teuthology.orchestra.run.smithi115:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-18T04:28:56.360 DEBUG:teuthology.orchestra.run.smithi115:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-18T04:28:56.367 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-18T04:28:56.367 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-18T04:28:56.395 DEBUG:teuthology.orchestra.run.smithi028:> sudo service rsyslog restart 2024-09-18T04:28:56.398 DEBUG:teuthology.orchestra.run.smithi082:> sudo service rsyslog restart 2024-09-18T04:28:56.400 DEBUG:teuthology.orchestra.run.smithi115:> sudo service rsyslog restart 2024-09-18T04:28:56.453 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-18T04:28:56.460 INFO:teuthology.task.internal:Starting timer... 2024-09-18T04:28:56.460 INFO:teuthology.run_tasks:Running task pcp... 2024-09-18T04:28:56.488 INFO:teuthology.run_tasks:Running task selinux... 2024-09-18T04:28:56.497 DEBUG:teuthology.task.selinux:Excluding smithi028: OS 'ubuntu' does not support SELinux 2024-09-18T04:28:56.497 DEBUG:teuthology.task.selinux:Excluding smithi082: OS 'ubuntu' does not support SELinux 2024-09-18T04:28:56.497 DEBUG:teuthology.task.selinux:Excluding smithi115: OS 'ubuntu' does not support SELinux 2024-09-18T04:28:56.498 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-18T04:28:56.498 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-18T04:28:56.498 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-18T04:28:56.498 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-18T04:28:56.510 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-18T04:28:56.521 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-18T04:28:56.522 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi028.front.sepia.ceph.com,smithi082.front.sepia.ceph.com,smithi115.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-18T04:34:33.575 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi028.front.sepia.ceph.com'), Remote(name='ubuntu@smithi082.front.sepia.ceph.com'), Remote(name='ubuntu@smithi115.front.sepia.ceph.com')] 2024-09-18T04:34:33.593 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-09-18T04:34:33.594 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T04:34:33.687 DEBUG:teuthology.orchestra.run.smithi028:> true 2024-09-18T04:34:33.814 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-09-18T04:34:33.815 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-09-18T04:34:33.815 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T04:34:33.912 DEBUG:teuthology.orchestra.run.smithi082:> true 2024-09-18T04:34:34.034 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-09-18T04:34:34.034 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-09-18T04:34:34.035 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T04:34:34.124 DEBUG:teuthology.orchestra.run.smithi115:> true 2024-09-18T04:34:34.246 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-09-18T04:34:34.247 INFO:teuthology.run_tasks:Running task clock... 2024-09-18T04:34:34.257 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-18T04:34:34.257 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-18T04:34:34.257 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-18T04:34:34.260 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-18T04:34:34.260 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-18T04:34:34.262 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-18T04:34:34.262 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-18T04:34:34.291 INFO:teuthology.orchestra.run.smithi028.stdout:18 Sep 04:34:34 ntpd[13656]: ntpd 4.2.8p12@1.3728-o (1): Starting 2024-09-18T04:34:34.291 INFO:teuthology.orchestra.run.smithi028.stdout:18 Sep 04:34:34 ntpd[13656]: Command line: ntpd -gq 2024-09-18T04:34:34.292 INFO:teuthology.orchestra.run.smithi028.stdout:18 Sep 04:34:34 ntpd[13656]: proto: precision = 0.066 usec (-24) 2024-09-18T04:34:34.292 INFO:teuthology.orchestra.run.smithi028.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-18T04:34:34.293 INFO:teuthology.orchestra.run.smithi028.stdout:18 Sep 04:34:34 ntpd[13656]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-18T04:34:34.293 INFO:teuthology.orchestra.run.smithi028.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-18T04:34:34.294 INFO:teuthology.orchestra.run.smithi028.stdout:18 Sep 04:34:34 ntpd[13656]: restrict ::: KOD does nothing without LIMITED. 2024-09-18T04:34:34.294 INFO:teuthology.orchestra.run.smithi028.stdout:18 Sep 04:34:34 ntpd[13656]: Listen and drop on 0 v6wildcard [::]:123 2024-09-18T04:34:34.294 INFO:teuthology.orchestra.run.smithi028.stdout:18 Sep 04:34:34 ntpd[13656]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-18T04:34:34.294 INFO:teuthology.orchestra.run.smithi028.stdout:18 Sep 04:34:34 ntpd[13656]: Listen normally on 2 lo 127.0.0.1:123 2024-09-18T04:34:34.294 INFO:teuthology.orchestra.run.smithi028.stdout:18 Sep 04:34:34 ntpd[13656]: Listen normally on 3 ens1f0 172.21.15.28:123 2024-09-18T04:34:34.294 INFO:teuthology.orchestra.run.smithi028.stdout:18 Sep 04:34:34 ntpd[13656]: Listen normally on 4 lo [::1]:123 2024-09-18T04:34:34.294 INFO:teuthology.orchestra.run.smithi028.stdout:18 Sep 04:34:34 ntpd[13656]: Listen normally on 5 ens1f0 [fe80::ae1f:6bff:fea5:ca3c%3]:123 2024-09-18T04:34:34.294 INFO:teuthology.orchestra.run.smithi028.stdout:18 Sep 04:34:34 ntpd[13656]: Listening on routing socket on fd #22 for interface updates 2024-09-18T04:34:34.296 INFO:teuthology.orchestra.run.smithi082.stdout:18 Sep 04:34:34 ntpd[13740]: ntpd 4.2.8p12@1.3728-o (1): Starting 2024-09-18T04:34:34.296 INFO:teuthology.orchestra.run.smithi082.stdout:18 Sep 04:34:34 ntpd[13740]: Command line: ntpd -gq 2024-09-18T04:34:34.297 INFO:teuthology.orchestra.run.smithi115.stdout:18 Sep 04:34:34 ntpd[13708]: ntpd 4.2.8p12@1.3728-o (1): Starting 2024-09-18T04:34:34.297 INFO:teuthology.orchestra.run.smithi115.stdout:18 Sep 04:34:34 ntpd[13708]: Command line: ntpd -gq 2024-09-18T04:34:34.298 INFO:teuthology.orchestra.run.smithi082.stdout:18 Sep 04:34:34 ntpd[13740]: proto: precision = 0.168 usec (-22) 2024-09-18T04:34:34.298 INFO:teuthology.orchestra.run.smithi082.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-18T04:34:34.298 INFO:teuthology.orchestra.run.smithi082.stdout:18 Sep 04:34:34 ntpd[13740]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-18T04:34:34.298 INFO:teuthology.orchestra.run.smithi082.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-18T04:34:34.299 INFO:teuthology.orchestra.run.smithi082.stdout:18 Sep 04:34:34 ntpd[13740]: restrict ::: KOD does nothing without LIMITED. 2024-09-18T04:34:34.299 INFO:teuthology.orchestra.run.smithi082.stdout:18 Sep 04:34:34 ntpd[13740]: Listen and drop on 0 v6wildcard [::]:123 2024-09-18T04:34:34.300 INFO:teuthology.orchestra.run.smithi082.stdout:18 Sep 04:34:34 ntpd[13740]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-18T04:34:34.300 INFO:teuthology.orchestra.run.smithi082.stdout:18 Sep 04:34:34 ntpd[13740]: Listen normally on 2 lo 127.0.0.1:123 2024-09-18T04:34:34.300 INFO:teuthology.orchestra.run.smithi082.stdout:18 Sep 04:34:34 ntpd[13740]: Listen normally on 3 enp3s0f1 172.21.15.82:123 2024-09-18T04:34:34.300 INFO:teuthology.orchestra.run.smithi082.stdout:18 Sep 04:34:34 ntpd[13740]: Listen normally on 4 lo [::1]:123 2024-09-18T04:34:34.300 INFO:teuthology.orchestra.run.smithi082.stdout:18 Sep 04:34:34 ntpd[13740]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d3ff%5]:123 2024-09-18T04:34:34.300 INFO:teuthology.orchestra.run.smithi082.stdout:18 Sep 04:34:34 ntpd[13740]: Listening on routing socket on fd #22 for interface updates 2024-09-18T04:34:34.300 INFO:teuthology.orchestra.run.smithi115.stdout:18 Sep 04:34:34 ntpd[13708]: proto: precision = 0.283 usec (-22) 2024-09-18T04:34:34.301 INFO:teuthology.orchestra.run.smithi115.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-18T04:34:34.301 INFO:teuthology.orchestra.run.smithi115.stdout:18 Sep 04:34:34 ntpd[13708]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-18T04:34:34.301 INFO:teuthology.orchestra.run.smithi115.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-18T04:34:34.302 INFO:teuthology.orchestra.run.smithi115.stdout:18 Sep 04:34:34 ntpd[13708]: restrict ::: KOD does nothing without LIMITED. 2024-09-18T04:34:34.302 INFO:teuthology.orchestra.run.smithi115.stdout:18 Sep 04:34:34 ntpd[13708]: Listen and drop on 0 v6wildcard [::]:123 2024-09-18T04:34:34.302 INFO:teuthology.orchestra.run.smithi115.stdout:18 Sep 04:34:34 ntpd[13708]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-18T04:34:34.302 INFO:teuthology.orchestra.run.smithi115.stdout:18 Sep 04:34:34 ntpd[13708]: Listen normally on 2 lo 127.0.0.1:123 2024-09-18T04:34:34.303 INFO:teuthology.orchestra.run.smithi115.stdout:18 Sep 04:34:34 ntpd[13708]: Listen normally on 3 enp3s0f1 172.21.15.115:123 2024-09-18T04:34:34.303 INFO:teuthology.orchestra.run.smithi115.stdout:18 Sep 04:34:34 ntpd[13708]: Listen normally on 4 lo [::1]:123 2024-09-18T04:34:34.303 INFO:teuthology.orchestra.run.smithi115.stdout:18 Sep 04:34:34 ntpd[13708]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:ce8f%5]:123 2024-09-18T04:34:34.303 INFO:teuthology.orchestra.run.smithi115.stdout:18 Sep 04:34:34 ntpd[13708]: Listening on routing socket on fd #22 for interface updates 2024-09-18T04:34:35.292 INFO:teuthology.orchestra.run.smithi028.stderr:18 Sep 04:34:35 ntpd[13656]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:35.292 INFO:teuthology.orchestra.run.smithi028.stderr:18 Sep 04:34:35 ntpd[13656]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:35.293 INFO:teuthology.orchestra.run.smithi028.stderr:18 Sep 04:34:35 ntpd[13656]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:35.293 INFO:teuthology.orchestra.run.smithi028.stderr:18 Sep 04:34:35 ntpd[13656]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:35.293 INFO:teuthology.orchestra.run.smithi028.stderr:18 Sep 04:34:35 ntpd[13656]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T04:34:35.293 INFO:teuthology.orchestra.run.smithi028.stderr:18 Sep 04:34:35 ntpd[13656]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T04:34:35.297 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:35 ntpd[13740]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:35.297 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:35 ntpd[13740]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:35.300 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:35 ntpd[13708]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:35.300 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:35 ntpd[13708]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:35.300 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:35 ntpd[13708]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:35.300 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:35 ntpd[13708]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:35.300 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:35 ntpd[13708]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:35.300 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:35 ntpd[13708]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:35.300 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:35 ntpd[13708]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:35.300 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:35 ntpd[13708]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:35.300 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:35 ntpd[13708]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:35.300 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:35 ntpd[13708]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:35.300 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:35 ntpd[13708]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T04:34:35.300 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:35 ntpd[13708]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T04:34:35.300 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:35 ntpd[13708]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:35.301 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:35 ntpd[13708]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:35.301 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:35 ntpd[13708]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T04:34:35.301 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:35 ntpd[13708]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T04:34:35.301 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:35 ntpd[13708]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:35.301 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:35 ntpd[13708]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:35.301 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:35 ntpd[13708]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T04:34:35.301 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:35 ntpd[13708]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T04:34:37.292 INFO:teuthology.orchestra.run.smithi028.stderr:18 Sep 04:34:37 ntpd[13656]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:37.292 INFO:teuthology.orchestra.run.smithi028.stderr:18 Sep 04:34:37 ntpd[13656]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:37.292 INFO:teuthology.orchestra.run.smithi028.stderr:18 Sep 04:34:37 ntpd[13656]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:37.292 INFO:teuthology.orchestra.run.smithi028.stderr:18 Sep 04:34:37 ntpd[13656]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:37.293 INFO:teuthology.orchestra.run.smithi028.stderr:18 Sep 04:34:37 ntpd[13656]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T04:34:37.293 INFO:teuthology.orchestra.run.smithi028.stderr:18 Sep 04:34:37 ntpd[13656]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T04:34:37.297 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:37 ntpd[13740]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:37.297 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:37 ntpd[13740]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:37.297 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:37 ntpd[13740]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:37.297 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:37 ntpd[13740]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:37.297 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:37 ntpd[13740]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T04:34:37.298 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:37 ntpd[13740]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T04:34:37.299 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:37 ntpd[13708]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:37.299 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:37 ntpd[13708]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:37.299 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:37 ntpd[13708]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:37.300 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:37 ntpd[13708]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:37.300 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:37 ntpd[13708]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:37.300 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:37 ntpd[13708]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:37.300 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:37 ntpd[13708]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:37.300 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:37 ntpd[13708]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:37.300 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:37 ntpd[13708]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T04:34:37.300 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:37 ntpd[13708]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T04:34:37.300 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:37 ntpd[13708]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:37.300 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:37 ntpd[13708]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:37.300 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:37 ntpd[13708]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T04:34:37.300 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:37 ntpd[13708]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T04:34:37.300 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:37 ntpd[13708]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:37.301 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:37 ntpd[13708]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:37.301 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:37 ntpd[13708]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T04:34:37.301 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:37 ntpd[13708]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T04:34:38.293 INFO:teuthology.orchestra.run.smithi028.stderr:18 Sep 04:34:38 ntpd[13656]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:38.293 INFO:teuthology.orchestra.run.smithi028.stderr:18 Sep 04:34:38 ntpd[13656]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:38.293 INFO:teuthology.orchestra.run.smithi028.stderr:18 Sep 04:34:38 ntpd[13656]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:38.293 INFO:teuthology.orchestra.run.smithi028.stderr:18 Sep 04:34:38 ntpd[13656]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:38.293 INFO:teuthology.orchestra.run.smithi028.stderr:18 Sep 04:34:38 ntpd[13656]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:38.293 INFO:teuthology.orchestra.run.smithi028.stderr:18 Sep 04:34:38 ntpd[13656]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:38.293 INFO:teuthology.orchestra.run.smithi028.stderr:18 Sep 04:34:38 ntpd[13656]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T04:34:38.293 INFO:teuthology.orchestra.run.smithi028.stderr:18 Sep 04:34:38 ntpd[13656]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T04:34:38.293 INFO:teuthology.orchestra.run.smithi028.stderr:18 Sep 04:34:38 ntpd[13656]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:38.293 INFO:teuthology.orchestra.run.smithi028.stderr:18 Sep 04:34:38 ntpd[13656]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:38.294 INFO:teuthology.orchestra.run.smithi028.stderr:18 Sep 04:34:38 ntpd[13656]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T04:34:38.294 INFO:teuthology.orchestra.run.smithi028.stderr:18 Sep 04:34:38 ntpd[13656]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T04:34:38.298 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:38 ntpd[13740]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:38.298 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:38 ntpd[13740]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:38.298 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:38 ntpd[13740]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:38.298 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:38 ntpd[13740]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:38.298 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:38 ntpd[13740]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:38.298 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:38 ntpd[13740]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:38.298 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:38 ntpd[13740]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T04:34:38.298 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:38 ntpd[13740]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T04:34:38.298 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:38 ntpd[13740]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:38.298 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:38 ntpd[13740]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:38.298 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:38 ntpd[13740]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T04:34:38.299 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:38 ntpd[13740]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T04:34:39.292 INFO:teuthology.orchestra.run.smithi028.stderr:18 Sep 04:34:39 ntpd[13656]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:39.292 INFO:teuthology.orchestra.run.smithi028.stderr:18 Sep 04:34:39 ntpd[13656]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:39.293 INFO:teuthology.orchestra.run.smithi028.stderr:18 Sep 04:34:39 ntpd[13656]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:39.293 INFO:teuthology.orchestra.run.smithi028.stderr:18 Sep 04:34:39 ntpd[13656]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:39.293 INFO:teuthology.orchestra.run.smithi028.stderr:18 Sep 04:34:39 ntpd[13656]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:39.293 INFO:teuthology.orchestra.run.smithi028.stderr:18 Sep 04:34:39 ntpd[13656]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:39.293 INFO:teuthology.orchestra.run.smithi028.stderr:18 Sep 04:34:39 ntpd[13656]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T04:34:39.293 INFO:teuthology.orchestra.run.smithi028.stderr:18 Sep 04:34:39 ntpd[13656]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T04:34:39.297 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:39 ntpd[13740]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:39.297 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:39 ntpd[13740]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:39.298 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:39 ntpd[13740]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:39.298 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:39 ntpd[13740]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:39.298 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:39 ntpd[13740]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T04:34:39.298 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:39 ntpd[13740]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T04:34:39.299 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:39 ntpd[13708]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:39.300 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:39 ntpd[13708]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:39.300 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:39 ntpd[13708]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:39.300 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:39 ntpd[13708]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:39.300 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:39 ntpd[13708]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:39.300 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:39 ntpd[13708]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:39.300 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:39 ntpd[13708]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:39.300 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:39 ntpd[13708]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:39.300 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:39 ntpd[13708]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T04:34:39.300 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:39 ntpd[13708]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T04:34:39.300 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:39 ntpd[13708]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:39.300 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:39 ntpd[13708]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:39.300 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:39 ntpd[13708]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T04:34:39.300 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:39 ntpd[13708]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T04:34:39.301 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:39 ntpd[13708]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:39.301 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:39 ntpd[13708]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:39.301 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:39 ntpd[13708]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T04:34:39.301 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:39 ntpd[13708]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T04:34:40.293 INFO:teuthology.orchestra.run.smithi028.stderr:18 Sep 04:34:40 ntpd[13656]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:40.293 INFO:teuthology.orchestra.run.smithi028.stderr:18 Sep 04:34:40 ntpd[13656]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:40.293 INFO:teuthology.orchestra.run.smithi028.stderr:18 Sep 04:34:40 ntpd[13656]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:40.293 INFO:teuthology.orchestra.run.smithi028.stderr:18 Sep 04:34:40 ntpd[13656]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:40.293 INFO:teuthology.orchestra.run.smithi028.stderr:18 Sep 04:34:40 ntpd[13656]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:40.293 INFO:teuthology.orchestra.run.smithi028.stderr:18 Sep 04:34:40 ntpd[13656]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:40.293 INFO:teuthology.orchestra.run.smithi028.stderr:18 Sep 04:34:40 ntpd[13656]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T04:34:40.293 INFO:teuthology.orchestra.run.smithi028.stderr:18 Sep 04:34:40 ntpd[13656]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T04:34:40.293 INFO:teuthology.orchestra.run.smithi028.stderr:18 Sep 04:34:40 ntpd[13656]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:40.293 INFO:teuthology.orchestra.run.smithi028.stderr:18 Sep 04:34:40 ntpd[13656]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:40.294 INFO:teuthology.orchestra.run.smithi028.stderr:18 Sep 04:34:40 ntpd[13656]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T04:34:40.294 INFO:teuthology.orchestra.run.smithi028.stderr:18 Sep 04:34:40 ntpd[13656]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T04:34:40.298 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:40 ntpd[13740]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:40.298 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:40 ntpd[13740]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:40.298 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:40 ntpd[13740]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:40.298 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:40 ntpd[13740]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:40.298 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:40 ntpd[13740]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:40.298 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:40 ntpd[13740]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:40.298 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:40 ntpd[13740]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T04:34:40.298 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:40 ntpd[13740]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T04:34:40.298 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:40 ntpd[13740]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:40.299 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:40 ntpd[13740]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:40.299 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:40 ntpd[13740]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T04:34:40.299 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:40 ntpd[13740]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T04:34:41.292 INFO:teuthology.orchestra.run.smithi028.stderr:18 Sep 04:34:41 ntpd[13656]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:41.292 INFO:teuthology.orchestra.run.smithi028.stderr:18 Sep 04:34:41 ntpd[13656]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:41.292 INFO:teuthology.orchestra.run.smithi028.stderr:18 Sep 04:34:41 ntpd[13656]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:41.292 INFO:teuthology.orchestra.run.smithi028.stderr:18 Sep 04:34:41 ntpd[13656]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:41.292 INFO:teuthology.orchestra.run.smithi028.stderr:18 Sep 04:34:41 ntpd[13656]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T04:34:41.293 INFO:teuthology.orchestra.run.smithi028.stderr:18 Sep 04:34:41 ntpd[13656]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T04:34:41.293 INFO:teuthology.orchestra.run.smithi028.stdout:18 Sep 04:34:41 ntpd[13656]: ntpd: time slew -0.002012 s 2024-09-18T04:34:41.293 INFO:teuthology.orchestra.run.smithi028.stdout:ntpd: time slew -0.002012s 2024-09-18T04:34:41.294 INFO:teuthology.orchestra.run.smithi028.stderr:18 Sep 04:34:41 ntpd[13656]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-18T04:34:41.294 INFO:teuthology.orchestra.run.smithi028.stderr:18 Sep 04:34:41 ntpd[13656]: can't open /var/log/ntpstats/loopstats.20240918: Permission denied 2024-09-18T04:34:41.297 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:41 ntpd[13740]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:41.297 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:41 ntpd[13740]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:41.297 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:41 ntpd[13740]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:41.298 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:41 ntpd[13740]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:41.298 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:41 ntpd[13740]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T04:34:41.298 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:41 ntpd[13740]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T04:34:41.299 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:41 ntpd[13708]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:41.299 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:41 ntpd[13708]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:41.299 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:41 ntpd[13708]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:41.299 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:41 ntpd[13708]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:41.299 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:41 ntpd[13708]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:41.299 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:41 ntpd[13708]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:41.300 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:41 ntpd[13708]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:41.300 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:41 ntpd[13708]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:41.300 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:41 ntpd[13708]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T04:34:41.300 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:41 ntpd[13708]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T04:34:41.300 INFO:teuthology.orchestra.run.smithi115.stdout:18 Sep 04:34:41 ntpd[13708]: ntpd: time slew -0.001242 s 2024-09-18T04:34:41.300 INFO:teuthology.orchestra.run.smithi115.stdout:ntpd: time slew -0.001242s 2024-09-18T04:34:41.301 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:41 ntpd[13708]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-18T04:34:41.301 INFO:teuthology.orchestra.run.smithi115.stderr:18 Sep 04:34:41 ntpd[13708]: can't open /var/log/ntpstats/loopstats.20240918: Permission denied 2024-09-18T04:34:41.358 INFO:teuthology.orchestra.run.smithi028.stdout: remote refid st t when poll reach delay offset jitter 2024-09-18T04:34:41.358 INFO:teuthology.orchestra.run.smithi028.stdout:============================================================================== 2024-09-18T04:34:41.358 INFO:teuthology.orchestra.run.smithi028.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T04:34:41.358 INFO:teuthology.orchestra.run.smithi028.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T04:34:41.358 INFO:teuthology.orchestra.run.smithi028.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T04:34:41.358 INFO:teuthology.orchestra.run.smithi028.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T04:34:41.384 INFO:teuthology.orchestra.run.smithi115.stdout: remote refid st t when poll reach delay offset jitter 2024-09-18T04:34:41.384 INFO:teuthology.orchestra.run.smithi115.stdout:============================================================================== 2024-09-18T04:34:41.384 INFO:teuthology.orchestra.run.smithi115.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T04:34:41.384 INFO:teuthology.orchestra.run.smithi115.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T04:34:41.384 INFO:teuthology.orchestra.run.smithi115.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T04:34:41.384 INFO:teuthology.orchestra.run.smithi115.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T04:34:42.298 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:42 ntpd[13740]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:42.298 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:42 ntpd[13740]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:42.298 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:42 ntpd[13740]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:42.298 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:42 ntpd[13740]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:42.299 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:42 ntpd[13740]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:42.299 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:42 ntpd[13740]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:42.299 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:42 ntpd[13740]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T04:34:42.299 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:42 ntpd[13740]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T04:34:42.299 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:42 ntpd[13740]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:42.299 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:42 ntpd[13740]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:42.300 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:42 ntpd[13740]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T04:34:42.300 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:42 ntpd[13740]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T04:34:43.300 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:43 ntpd[13740]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:43.301 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:43 ntpd[13740]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:43.301 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:43 ntpd[13740]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T04:34:43.301 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:43 ntpd[13740]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T04:34:43.301 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:43 ntpd[13740]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T04:34:43.301 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:43 ntpd[13740]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T04:34:43.301 INFO:teuthology.orchestra.run.smithi082.stdout:18 Sep 04:34:43 ntpd[13740]: ntpd: time slew +0.001147 s 2024-09-18T04:34:43.302 INFO:teuthology.orchestra.run.smithi082.stdout:ntpd: time slew +0.001147s 2024-09-18T04:34:43.302 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:43 ntpd[13740]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-18T04:34:43.302 INFO:teuthology.orchestra.run.smithi082.stderr:18 Sep 04:34:43 ntpd[13740]: can't open /var/log/ntpstats/loopstats.20240918: Permission denied 2024-09-18T04:34:43.366 INFO:teuthology.orchestra.run.smithi082.stdout: remote refid st t when poll reach delay offset jitter 2024-09-18T04:34:43.366 INFO:teuthology.orchestra.run.smithi082.stdout:============================================================================== 2024-09-18T04:34:43.366 INFO:teuthology.orchestra.run.smithi082.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T04:34:43.366 INFO:teuthology.orchestra.run.smithi082.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T04:34:43.366 INFO:teuthology.orchestra.run.smithi082.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T04:34:43.366 INFO:teuthology.orchestra.run.smithi082.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T04:34:43.367 INFO:teuthology.run_tasks:Running task install... 2024-09-18T04:34:43.375 DEBUG:teuthology.task.install:project ceph 2024-09-18T04:34:43.375 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'extra_packages': ['rbd-nbd'], 'flavor': 'default', 'sha1': '19e2a2beb6f41a3ab62650e38614eb29b59dcbb9'}} 2024-09-18T04:34:43.375 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-nbd'], 'flavor': 'default', 'sha1': '19e2a2beb6f41a3ab62650e38614eb29b59dcbb9'} 2024-09-18T04:34:43.375 INFO:teuthology.task.install:Using flavor: default 2024-09-18T04:34:43.381 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-18T04:34:43.381 INFO:teuthology.task.install:extra packages: ['rbd-nbd'] 2024-09-18T04:34:43.381 DEBUG:teuthology.orchestra.run.smithi028:> sudo apt-key list | grep Ceph 2024-09-18T04:34:43.382 DEBUG:teuthology.orchestra.run.smithi082:> sudo apt-key list | grep Ceph 2024-09-18T04:34:43.382 DEBUG:teuthology.orchestra.run.smithi115:> sudo apt-key list | grep Ceph 2024-09-18T04:34:43.481 INFO:teuthology.orchestra.run.smithi115.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2024-09-18T04:34:43.493 INFO:teuthology.orchestra.run.smithi028.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2024-09-18T04:34:43.508 INFO:teuthology.orchestra.run.smithi082.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2024-09-18T04:34:43.539 INFO:teuthology.orchestra.run.smithi115.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-18T04:34:43.539 INFO:teuthology.orchestra.run.smithi115.stdout:uid [ unknown] Ceph.com (release key) 2024-09-18T04:34:43.540 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-nbd on remote deb x86_64 2024-09-18T04:34:43.540 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-18T04:34:43.554 INFO:teuthology.orchestra.run.smithi028.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-18T04:34:43.555 INFO:teuthology.orchestra.run.smithi028.stdout:uid [ unknown] Ceph.com (release key) 2024-09-18T04:34:43.555 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-nbd on remote deb x86_64 2024-09-18T04:34:43.556 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-18T04:34:43.564 INFO:teuthology.orchestra.run.smithi082.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-18T04:34:43.564 INFO:teuthology.orchestra.run.smithi082.stdout:uid [ unknown] Ceph.com (release key) 2024-09-18T04:34:43.565 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-nbd on remote deb x86_64 2024-09-18T04:34:43.565 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-18T04:34:43.696 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default/ 2024-09-18T04:34:43.696 INFO:teuthology.task.install.deb:Package version is 18.2.4-855-g19e2a2be-1focal 2024-09-18T04:34:43.711 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default/ 2024-09-18T04:34:43.712 INFO:teuthology.task.install.deb:Package version is 18.2.4-855-g19e2a2be-1focal 2024-09-18T04:34:43.718 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default/ 2024-09-18T04:34:43.718 INFO:teuthology.task.install.deb:Package version is 18.2.4-855-g19e2a2be-1focal 2024-09-18T04:34:43.813 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-18T04:34:43.813 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-18T04:34:43.825 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-18T04:34:43.826 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-18T04:34:43.830 DEBUG:teuthology.orchestra.run.smithi115:> sudo apt-get update 2024-09-18T04:34:43.832 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-09-18T04:34:43.832 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-18T04:34:43.842 DEBUG:teuthology.orchestra.run.smithi028:> sudo apt-get update 2024-09-18T04:34:43.848 DEBUG:teuthology.orchestra.run.smithi082:> sudo apt-get update 2024-09-18T04:34:43.990 INFO:teuthology.orchestra.run.smithi028.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-18T04:34:44.013 INFO:teuthology.orchestra.run.smithi115.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-18T04:34:44.013 INFO:teuthology.orchestra.run.smithi115.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-18T04:34:44.019 INFO:teuthology.orchestra.run.smithi115.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-18T04:34:44.034 INFO:teuthology.orchestra.run.smithi082.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-18T04:34:44.034 INFO:teuthology.orchestra.run.smithi082.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-18T04:34:44.036 INFO:teuthology.orchestra.run.smithi115.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-18T04:34:44.039 INFO:teuthology.orchestra.run.smithi082.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-18T04:34:44.051 INFO:teuthology.orchestra.run.smithi028.stdout:Ign:2 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal InRelease 2024-09-18T04:34:44.057 INFO:teuthology.orchestra.run.smithi082.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-18T04:34:44.069 INFO:teuthology.orchestra.run.smithi115.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal InRelease 2024-09-18T04:34:44.074 INFO:teuthology.orchestra.run.smithi028.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal Release [7,662 B] 2024-09-18T04:34:44.088 INFO:teuthology.orchestra.run.smithi082.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal InRelease 2024-09-18T04:34:44.091 INFO:teuthology.orchestra.run.smithi115.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal Release [7,662 B] 2024-09-18T04:34:44.096 INFO:teuthology.orchestra.run.smithi028.stdout:Ign:4 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal Release.gpg 2024-09-18T04:34:44.106 INFO:teuthology.orchestra.run.smithi028.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-18T04:34:44.109 INFO:teuthology.orchestra.run.smithi082.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal Release [7,662 B] 2024-09-18T04:34:44.117 INFO:teuthology.orchestra.run.smithi115.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal Release.gpg 2024-09-18T04:34:44.121 INFO:teuthology.orchestra.run.smithi028.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 Packages [17.6 kB] 2024-09-18T04:34:44.132 INFO:teuthology.orchestra.run.smithi082.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal Release.gpg 2024-09-18T04:34:44.136 INFO:teuthology.orchestra.run.smithi115.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 Packages [17.6 kB] 2024-09-18T04:34:44.154 INFO:teuthology.orchestra.run.smithi082.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main i386 Packages [3,426 B] 2024-09-18T04:34:44.162 INFO:teuthology.orchestra.run.smithi028.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main i386 Packages [3,426 B] 2024-09-18T04:34:44.169 INFO:teuthology.orchestra.run.smithi115.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main i386 Packages [3,426 B] 2024-09-18T04:34:44.187 INFO:teuthology.orchestra.run.smithi028.stdout:Hit:8 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-18T04:34:44.188 INFO:teuthology.orchestra.run.smithi082.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 Packages [17.6 kB] 2024-09-18T04:34:44.269 INFO:teuthology.orchestra.run.smithi028.stdout:Hit:9 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-18T04:34:44.680 INFO:teuthology.orchestra.run.smithi115.stdout:Fetched 28.7 kB in 1s (44.7 kB/s) 2024-09-18T04:34:44.681 INFO:teuthology.orchestra.run.smithi028.stdout:Fetched 28.7 kB in 1s (43.2 kB/s) 2024-09-18T04:34:44.691 INFO:teuthology.orchestra.run.smithi082.stdout:Fetched 28.7 kB in 1s (45.3 kB/s) 2024-09-18T04:34:45.837 INFO:teuthology.orchestra.run.smithi115.stdout:Reading package lists... 2024-09-18T04:34:45.848 INFO:teuthology.orchestra.run.smithi082.stdout:Reading package lists... 2024-09-18T04:34:45.863 DEBUG:teuthology.orchestra.run.smithi115:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.2.4-855-g19e2a2be-1focal cephadm=18.2.4-855-g19e2a2be-1focal ceph-mds=18.2.4-855-g19e2a2be-1focal ceph-mgr=18.2.4-855-g19e2a2be-1focal ceph-common=18.2.4-855-g19e2a2be-1focal ceph-fuse=18.2.4-855-g19e2a2be-1focal ceph-test=18.2.4-855-g19e2a2be-1focal ceph-volume=18.2.4-855-g19e2a2be-1focal radosgw=18.2.4-855-g19e2a2be-1focal python3-rados=18.2.4-855-g19e2a2be-1focal python3-rgw=18.2.4-855-g19e2a2be-1focal python3-cephfs=18.2.4-855-g19e2a2be-1focal python3-rbd=18.2.4-855-g19e2a2be-1focal libcephfs2=18.2.4-855-g19e2a2be-1focal libcephfs-dev=18.2.4-855-g19e2a2be-1focal librados2=18.2.4-855-g19e2a2be-1focal librbd1=18.2.4-855-g19e2a2be-1focal rbd-fuse=18.2.4-855-g19e2a2be-1focal rbd-nbd=18.2.4-855-g19e2a2be-1focal 2024-09-18T04:34:45.873 DEBUG:teuthology.orchestra.run.smithi082:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.2.4-855-g19e2a2be-1focal cephadm=18.2.4-855-g19e2a2be-1focal ceph-mds=18.2.4-855-g19e2a2be-1focal ceph-mgr=18.2.4-855-g19e2a2be-1focal ceph-common=18.2.4-855-g19e2a2be-1focal ceph-fuse=18.2.4-855-g19e2a2be-1focal ceph-test=18.2.4-855-g19e2a2be-1focal ceph-volume=18.2.4-855-g19e2a2be-1focal radosgw=18.2.4-855-g19e2a2be-1focal python3-rados=18.2.4-855-g19e2a2be-1focal python3-rgw=18.2.4-855-g19e2a2be-1focal python3-cephfs=18.2.4-855-g19e2a2be-1focal python3-rbd=18.2.4-855-g19e2a2be-1focal libcephfs2=18.2.4-855-g19e2a2be-1focal libcephfs-dev=18.2.4-855-g19e2a2be-1focal librados2=18.2.4-855-g19e2a2be-1focal librbd1=18.2.4-855-g19e2a2be-1focal rbd-fuse=18.2.4-855-g19e2a2be-1focal rbd-nbd=18.2.4-855-g19e2a2be-1focal 2024-09-18T04:34:45.888 INFO:teuthology.orchestra.run.smithi028.stdout:Reading package lists... 2024-09-18T04:34:45.916 DEBUG:teuthology.orchestra.run.smithi028:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.2.4-855-g19e2a2be-1focal cephadm=18.2.4-855-g19e2a2be-1focal ceph-mds=18.2.4-855-g19e2a2be-1focal ceph-mgr=18.2.4-855-g19e2a2be-1focal ceph-common=18.2.4-855-g19e2a2be-1focal ceph-fuse=18.2.4-855-g19e2a2be-1focal ceph-test=18.2.4-855-g19e2a2be-1focal ceph-volume=18.2.4-855-g19e2a2be-1focal radosgw=18.2.4-855-g19e2a2be-1focal python3-rados=18.2.4-855-g19e2a2be-1focal python3-rgw=18.2.4-855-g19e2a2be-1focal python3-cephfs=18.2.4-855-g19e2a2be-1focal python3-rbd=18.2.4-855-g19e2a2be-1focal libcephfs2=18.2.4-855-g19e2a2be-1focal libcephfs-dev=18.2.4-855-g19e2a2be-1focal librados2=18.2.4-855-g19e2a2be-1focal librbd1=18.2.4-855-g19e2a2be-1focal rbd-fuse=18.2.4-855-g19e2a2be-1focal rbd-nbd=18.2.4-855-g19e2a2be-1focal 2024-09-18T04:34:45.932 INFO:teuthology.orchestra.run.smithi115.stdout:Reading package lists... 2024-09-18T04:34:45.943 INFO:teuthology.orchestra.run.smithi082.stdout:Reading package lists... 2024-09-18T04:34:45.989 INFO:teuthology.orchestra.run.smithi028.stdout:Reading package lists... 2024-09-18T04:34:46.125 INFO:teuthology.orchestra.run.smithi115.stdout:Building dependency tree... 2024-09-18T04:34:46.126 INFO:teuthology.orchestra.run.smithi115.stdout:Reading state information... 2024-09-18T04:34:46.165 INFO:teuthology.orchestra.run.smithi082.stdout:Building dependency tree... 2024-09-18T04:34:46.165 INFO:teuthology.orchestra.run.smithi082.stdout:Reading state information... 2024-09-18T04:34:46.212 INFO:teuthology.orchestra.run.smithi028.stdout:Building dependency tree... 2024-09-18T04:34:46.213 INFO:teuthology.orchestra.run.smithi028.stdout:Reading state information... 2024-09-18T04:34:46.316 INFO:teuthology.orchestra.run.smithi115.stdout:The following packages were automatically installed and are no longer required: 2024-09-18T04:34:46.317 INFO:teuthology.orchestra.run.smithi115.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2024-09-18T04:34:46.317 INFO:teuthology.orchestra.run.smithi115.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-18T04:34:46.317 INFO:teuthology.orchestra.run.smithi115.stdout:The following additional packages will be installed: 2024-09-18T04:34:46.318 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-18T04:34:46.318 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2024-09-18T04:34:46.318 INFO:teuthology.orchestra.run.smithi115.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2024-09-18T04:34:46.318 INFO:teuthology.orchestra.run.smithi115.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2024-09-18T04:34:46.318 INFO:teuthology.orchestra.run.smithi115.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2024-09-18T04:34:46.320 INFO:teuthology.orchestra.run.smithi115.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2024-09-18T04:34:46.320 INFO:teuthology.orchestra.run.smithi115.stdout: python-asyncssh-doc python-natsort-doc python-pastedeploy-tpl 2024-09-18T04:34:46.320 INFO:teuthology.orchestra.run.smithi115.stdout: python3-asyncssh python3-atomicwrites python3-attr python3-bcrypt 2024-09-18T04:34:46.320 INFO:teuthology.orchestra.run.smithi115.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2024-09-18T04:34:46.320 INFO:teuthology.orchestra.run.smithi115.stdout: python3-ceph-common python3-cherrypy3 python3-cryptography python3-decorator 2024-09-18T04:34:46.320 INFO:teuthology.orchestra.run.smithi115.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2024-09-18T04:34:46.320 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2024-09-18T04:34:46.320 INFO:teuthology.orchestra.run.smithi115.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2024-09-18T04:34:46.320 INFO:teuthology.orchestra.run.smithi115.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2024-09-18T04:34:46.320 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2024-09-18T04:34:46.321 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2024-09-18T04:34:46.321 INFO:teuthology.orchestra.run.smithi115.stdout: python3-repoze.lru python3-requests-oauthlib python3-routes python3-rsa 2024-09-18T04:34:46.321 INFO:teuthology.orchestra.run.smithi115.stdout: python3-scipy python3-simplegeneric python3-simplejson 2024-09-18T04:34:46.321 INFO:teuthology.orchestra.run.smithi115.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2024-09-18T04:34:46.321 INFO:teuthology.orchestra.run.smithi115.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2024-09-18T04:34:46.321 INFO:teuthology.orchestra.run.smithi115.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2024-09-18T04:34:46.321 INFO:teuthology.orchestra.run.smithi115.stdout: qttranslations5-l10n socat xmlstarlet 2024-09-18T04:34:46.321 INFO:teuthology.orchestra.run.smithi115.stdout:Suggested packages: 2024-09-18T04:34:46.321 INFO:teuthology.orchestra.run.smithi115.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2024-09-18T04:34:46.321 INFO:teuthology.orchestra.run.smithi115.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2024-09-18T04:34:46.321 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2024-09-18T04:34:46.321 INFO:teuthology.orchestra.run.smithi115.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2024-09-18T04:34:46.321 INFO:teuthology.orchestra.run.smithi115.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2024-09-18T04:34:46.321 INFO:teuthology.orchestra.run.smithi115.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2024-09-18T04:34:46.322 INFO:teuthology.orchestra.run.smithi115.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2024-09-18T04:34:46.322 INFO:teuthology.orchestra.run.smithi115.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-18T04:34:46.322 INFO:teuthology.orchestra.run.smithi115.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2024-09-18T04:34:46.322 INFO:teuthology.orchestra.run.smithi115.stdout:Recommended packages: 2024-09-18T04:34:46.322 INFO:teuthology.orchestra.run.smithi115.stdout: btrfs-tools 2024-09-18T04:34:46.355 INFO:teuthology.orchestra.run.smithi082.stdout:The following packages were automatically installed and are no longer required: 2024-09-18T04:34:46.356 INFO:teuthology.orchestra.run.smithi082.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2024-09-18T04:34:46.356 INFO:teuthology.orchestra.run.smithi082.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-18T04:34:46.356 INFO:teuthology.orchestra.run.smithi082.stdout:The following additional packages will be installed: 2024-09-18T04:34:46.356 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-18T04:34:46.356 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2024-09-18T04:34:46.357 INFO:teuthology.orchestra.run.smithi082.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2024-09-18T04:34:46.357 INFO:teuthology.orchestra.run.smithi082.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2024-09-18T04:34:46.357 INFO:teuthology.orchestra.run.smithi082.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2024-09-18T04:34:46.357 INFO:teuthology.orchestra.run.smithi082.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2024-09-18T04:34:46.357 INFO:teuthology.orchestra.run.smithi082.stdout: python-asyncssh-doc python-natsort-doc python-pastedeploy-tpl 2024-09-18T04:34:46.358 INFO:teuthology.orchestra.run.smithi082.stdout: python3-asyncssh python3-atomicwrites python3-attr python3-bcrypt 2024-09-18T04:34:46.358 INFO:teuthology.orchestra.run.smithi082.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2024-09-18T04:34:46.358 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ceph-common python3-cherrypy3 python3-cryptography python3-decorator 2024-09-18T04:34:46.358 INFO:teuthology.orchestra.run.smithi082.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2024-09-18T04:34:46.358 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2024-09-18T04:34:46.358 INFO:teuthology.orchestra.run.smithi082.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2024-09-18T04:34:46.358 INFO:teuthology.orchestra.run.smithi082.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2024-09-18T04:34:46.359 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2024-09-18T04:34:46.359 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2024-09-18T04:34:46.359 INFO:teuthology.orchestra.run.smithi082.stdout: python3-repoze.lru python3-requests-oauthlib python3-routes python3-rsa 2024-09-18T04:34:46.359 INFO:teuthology.orchestra.run.smithi082.stdout: python3-scipy python3-simplegeneric python3-simplejson 2024-09-18T04:34:46.359 INFO:teuthology.orchestra.run.smithi082.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2024-09-18T04:34:46.359 INFO:teuthology.orchestra.run.smithi082.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2024-09-18T04:34:46.359 INFO:teuthology.orchestra.run.smithi082.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2024-09-18T04:34:46.359 INFO:teuthology.orchestra.run.smithi082.stdout: qttranslations5-l10n socat xmlstarlet 2024-09-18T04:34:46.360 INFO:teuthology.orchestra.run.smithi082.stdout:Suggested packages: 2024-09-18T04:34:46.360 INFO:teuthology.orchestra.run.smithi082.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2024-09-18T04:34:46.360 INFO:teuthology.orchestra.run.smithi082.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2024-09-18T04:34:46.360 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2024-09-18T04:34:46.360 INFO:teuthology.orchestra.run.smithi082.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2024-09-18T04:34:46.360 INFO:teuthology.orchestra.run.smithi082.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2024-09-18T04:34:46.360 INFO:teuthology.orchestra.run.smithi082.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2024-09-18T04:34:46.360 INFO:teuthology.orchestra.run.smithi082.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2024-09-18T04:34:46.360 INFO:teuthology.orchestra.run.smithi082.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-18T04:34:46.360 INFO:teuthology.orchestra.run.smithi082.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2024-09-18T04:34:46.360 INFO:teuthology.orchestra.run.smithi082.stdout:Recommended packages: 2024-09-18T04:34:46.360 INFO:teuthology.orchestra.run.smithi082.stdout: btrfs-tools 2024-09-18T04:34:46.422 INFO:teuthology.orchestra.run.smithi028.stdout:The following packages were automatically installed and are no longer required: 2024-09-18T04:34:46.423 INFO:teuthology.orchestra.run.smithi028.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2024-09-18T04:34:46.423 INFO:teuthology.orchestra.run.smithi028.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-18T04:34:46.423 INFO:teuthology.orchestra.run.smithi028.stdout:The following additional packages will be installed: 2024-09-18T04:34:46.423 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-18T04:34:46.423 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2024-09-18T04:34:46.423 INFO:teuthology.orchestra.run.smithi028.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2024-09-18T04:34:46.424 INFO:teuthology.orchestra.run.smithi028.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2024-09-18T04:34:46.424 INFO:teuthology.orchestra.run.smithi028.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2024-09-18T04:34:46.424 INFO:teuthology.orchestra.run.smithi028.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2024-09-18T04:34:46.424 INFO:teuthology.orchestra.run.smithi028.stdout: python-asyncssh-doc python-natsort-doc python-pastedeploy-tpl 2024-09-18T04:34:46.425 INFO:teuthology.orchestra.run.smithi028.stdout: python3-asyncssh python3-atomicwrites python3-attr python3-bcrypt 2024-09-18T04:34:46.425 INFO:teuthology.orchestra.run.smithi028.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2024-09-18T04:34:46.425 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-common python3-cherrypy3 python3-cryptography python3-decorator 2024-09-18T04:34:46.425 INFO:teuthology.orchestra.run.smithi028.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2024-09-18T04:34:46.425 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2024-09-18T04:34:46.425 INFO:teuthology.orchestra.run.smithi028.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2024-09-18T04:34:46.425 INFO:teuthology.orchestra.run.smithi028.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2024-09-18T04:34:46.425 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2024-09-18T04:34:46.425 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2024-09-18T04:34:46.425 INFO:teuthology.orchestra.run.smithi028.stdout: python3-repoze.lru python3-requests-oauthlib python3-routes python3-rsa 2024-09-18T04:34:46.425 INFO:teuthology.orchestra.run.smithi028.stdout: python3-scipy python3-simplegeneric python3-simplejson 2024-09-18T04:34:46.425 INFO:teuthology.orchestra.run.smithi028.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2024-09-18T04:34:46.425 INFO:teuthology.orchestra.run.smithi028.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2024-09-18T04:34:46.425 INFO:teuthology.orchestra.run.smithi028.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2024-09-18T04:34:46.426 INFO:teuthology.orchestra.run.smithi028.stdout: qttranslations5-l10n socat xmlstarlet 2024-09-18T04:34:46.426 INFO:teuthology.orchestra.run.smithi028.stdout:Suggested packages: 2024-09-18T04:34:46.426 INFO:teuthology.orchestra.run.smithi028.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2024-09-18T04:34:46.426 INFO:teuthology.orchestra.run.smithi028.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2024-09-18T04:34:46.426 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2024-09-18T04:34:46.426 INFO:teuthology.orchestra.run.smithi028.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2024-09-18T04:34:46.426 INFO:teuthology.orchestra.run.smithi028.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2024-09-18T04:34:46.426 INFO:teuthology.orchestra.run.smithi028.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2024-09-18T04:34:46.426 INFO:teuthology.orchestra.run.smithi028.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2024-09-18T04:34:46.427 INFO:teuthology.orchestra.run.smithi028.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-18T04:34:46.427 INFO:teuthology.orchestra.run.smithi028.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2024-09-18T04:34:46.427 INFO:teuthology.orchestra.run.smithi028.stdout:Recommended packages: 2024-09-18T04:34:46.427 INFO:teuthology.orchestra.run.smithi028.stdout: btrfs-tools 2024-09-18T04:34:46.436 INFO:teuthology.orchestra.run.smithi115.stdout:The following NEW packages will be installed: 2024-09-18T04:34:46.436 INFO:teuthology.orchestra.run.smithi115.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-18T04:34:46.436 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-18T04:34:46.436 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-09-18T04:34:46.439 INFO:teuthology.orchestra.run.smithi115.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2024-09-18T04:34:46.439 INFO:teuthology.orchestra.run.smithi115.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2024-09-18T04:34:46.439 INFO:teuthology.orchestra.run.smithi115.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-18T04:34:46.439 INFO:teuthology.orchestra.run.smithi115.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2024-09-18T04:34:46.439 INFO:teuthology.orchestra.run.smithi115.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-asyncssh-doc 2024-09-18T04:34:46.439 INFO:teuthology.orchestra.run.smithi115.stdout: python-natsort-doc python-pastedeploy-tpl python3-asyncssh 2024-09-18T04:34:46.439 INFO:teuthology.orchestra.run.smithi115.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2024-09-18T04:34:46.439 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-09-18T04:34:46.439 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cherrypy3 python3-cryptography python3-decorator python3-google-auth 2024-09-18T04:34:46.439 INFO:teuthology.orchestra.run.smithi115.stdout: python3-html5lib python3-jinja2 python3-joblib python3-jwt 2024-09-18T04:34:46.439 INFO:teuthology.orchestra.run.smithi115.stdout: python3-kubernetes python3-logutils python3-lxml python3-mako 2024-09-18T04:34:46.439 INFO:teuthology.orchestra.run.smithi115.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2024-09-18T04:34:46.440 INFO:teuthology.orchestra.run.smithi115.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2024-09-18T04:34:46.440 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2024-09-18T04:34:46.440 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2024-09-18T04:34:46.440 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2024-09-18T04:34:46.440 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2024-09-18T04:34:46.440 INFO:teuthology.orchestra.run.smithi115.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-09-18T04:34:46.440 INFO:teuthology.orchestra.run.smithi115.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2024-09-18T04:34:46.440 INFO:teuthology.orchestra.run.smithi115.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2024-09-18T04:34:46.440 INFO:teuthology.orchestra.run.smithi115.stdout: python3-webtest python3-werkzeug qttranslations5-l10n radosgw rbd-fuse 2024-09-18T04:34:46.440 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-nbd socat xmlstarlet 2024-09-18T04:34:46.440 INFO:teuthology.orchestra.run.smithi115.stdout:The following packages will be upgraded: 2024-09-18T04:34:46.441 INFO:teuthology.orchestra.run.smithi115.stdout: librados2 librbd1 2024-09-18T04:34:46.474 INFO:teuthology.orchestra.run.smithi082.stdout:The following NEW packages will be installed: 2024-09-18T04:34:46.475 INFO:teuthology.orchestra.run.smithi082.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-18T04:34:46.475 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-18T04:34:46.475 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-09-18T04:34:46.476 INFO:teuthology.orchestra.run.smithi082.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2024-09-18T04:34:46.476 INFO:teuthology.orchestra.run.smithi082.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2024-09-18T04:34:46.476 INFO:teuthology.orchestra.run.smithi082.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-18T04:34:46.476 INFO:teuthology.orchestra.run.smithi082.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2024-09-18T04:34:46.476 INFO:teuthology.orchestra.run.smithi082.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-asyncssh-doc 2024-09-18T04:34:46.476 INFO:teuthology.orchestra.run.smithi082.stdout: python-natsort-doc python-pastedeploy-tpl python3-asyncssh 2024-09-18T04:34:46.476 INFO:teuthology.orchestra.run.smithi082.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2024-09-18T04:34:46.477 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-09-18T04:34:46.477 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cherrypy3 python3-cryptography python3-decorator python3-google-auth 2024-09-18T04:34:46.477 INFO:teuthology.orchestra.run.smithi082.stdout: python3-html5lib python3-jinja2 python3-joblib python3-jwt 2024-09-18T04:34:46.477 INFO:teuthology.orchestra.run.smithi082.stdout: python3-kubernetes python3-logutils python3-lxml python3-mako 2024-09-18T04:34:46.477 INFO:teuthology.orchestra.run.smithi082.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2024-09-18T04:34:46.477 INFO:teuthology.orchestra.run.smithi082.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2024-09-18T04:34:46.477 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2024-09-18T04:34:46.477 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2024-09-18T04:34:46.477 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2024-09-18T04:34:46.477 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2024-09-18T04:34:46.477 INFO:teuthology.orchestra.run.smithi082.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-09-18T04:34:46.477 INFO:teuthology.orchestra.run.smithi082.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2024-09-18T04:34:46.477 INFO:teuthology.orchestra.run.smithi082.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2024-09-18T04:34:46.477 INFO:teuthology.orchestra.run.smithi082.stdout: python3-webtest python3-werkzeug qttranslations5-l10n radosgw rbd-fuse 2024-09-18T04:34:46.478 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-nbd socat xmlstarlet 2024-09-18T04:34:46.478 INFO:teuthology.orchestra.run.smithi082.stdout:The following packages will be upgraded: 2024-09-18T04:34:46.478 INFO:teuthology.orchestra.run.smithi082.stdout: librados2 librbd1 2024-09-18T04:34:46.504 INFO:teuthology.orchestra.run.smithi115.stdout:2 upgraded, 111 newly installed, 0 to remove and 295 not upgraded. 2024-09-18T04:34:46.504 INFO:teuthology.orchestra.run.smithi115.stdout:Need to get 199 MB of archives. 2024-09-18T04:34:46.504 INFO:teuthology.orchestra.run.smithi115.stdout:After this operation, 978 MB of additional disk space will be used. 2024-09-18T04:34:46.504 INFO:teuthology.orchestra.run.smithi115.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2024-09-18T04:34:46.543 INFO:teuthology.orchestra.run.smithi082.stdout:2 upgraded, 111 newly installed, 0 to remove and 295 not upgraded. 2024-09-18T04:34:46.543 INFO:teuthology.orchestra.run.smithi082.stdout:Need to get 199 MB of archives. 2024-09-18T04:34:46.543 INFO:teuthology.orchestra.run.smithi082.stdout:After this operation, 978 MB of additional disk space will be used. 2024-09-18T04:34:46.543 INFO:teuthology.orchestra.run.smithi082.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2024-09-18T04:34:46.547 INFO:teuthology.orchestra.run.smithi028.stdout:The following NEW packages will be installed: 2024-09-18T04:34:46.547 INFO:teuthology.orchestra.run.smithi028.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-18T04:34:46.547 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-18T04:34:46.548 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-09-18T04:34:46.549 INFO:teuthology.orchestra.run.smithi028.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2024-09-18T04:34:46.549 INFO:teuthology.orchestra.run.smithi028.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2024-09-18T04:34:46.549 INFO:teuthology.orchestra.run.smithi028.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-18T04:34:46.549 INFO:teuthology.orchestra.run.smithi028.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2024-09-18T04:34:46.549 INFO:teuthology.orchestra.run.smithi028.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-asyncssh-doc 2024-09-18T04:34:46.549 INFO:teuthology.orchestra.run.smithi028.stdout: python-natsort-doc python-pastedeploy-tpl python3-asyncssh 2024-09-18T04:34:46.549 INFO:teuthology.orchestra.run.smithi028.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2024-09-18T04:34:46.549 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-09-18T04:34:46.549 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cherrypy3 python3-cryptography python3-decorator python3-google-auth 2024-09-18T04:34:46.549 INFO:teuthology.orchestra.run.smithi028.stdout: python3-html5lib python3-jinja2 python3-joblib python3-jwt 2024-09-18T04:34:46.549 INFO:teuthology.orchestra.run.smithi028.stdout: python3-kubernetes python3-logutils python3-lxml python3-mako 2024-09-18T04:34:46.550 INFO:teuthology.orchestra.run.smithi028.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2024-09-18T04:34:46.550 INFO:teuthology.orchestra.run.smithi028.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2024-09-18T04:34:46.550 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2024-09-18T04:34:46.550 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2024-09-18T04:34:46.550 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2024-09-18T04:34:46.550 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2024-09-18T04:34:46.550 INFO:teuthology.orchestra.run.smithi028.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-09-18T04:34:46.550 INFO:teuthology.orchestra.run.smithi028.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2024-09-18T04:34:46.550 INFO:teuthology.orchestra.run.smithi028.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2024-09-18T04:34:46.550 INFO:teuthology.orchestra.run.smithi028.stdout: python3-webtest python3-werkzeug qttranslations5-l10n radosgw rbd-fuse 2024-09-18T04:34:46.550 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-nbd socat xmlstarlet 2024-09-18T04:34:46.551 INFO:teuthology.orchestra.run.smithi028.stdout:The following packages will be upgraded: 2024-09-18T04:34:46.551 INFO:teuthology.orchestra.run.smithi028.stdout: librados2 librbd1 2024-09-18T04:34:46.587 INFO:teuthology.orchestra.run.smithi115.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2024-09-18T04:34:46.590 INFO:teuthology.orchestra.run.smithi115.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2024-09-18T04:34:46.590 INFO:teuthology.orchestra.run.smithi115.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 18.2.4-855-g19e2a2be-1focal [3,453 kB] 2024-09-18T04:34:46.607 INFO:teuthology.orchestra.run.smithi115.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2024-09-18T04:34:46.626 INFO:teuthology.orchestra.run.smithi082.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 18.2.4-855-g19e2a2be-1focal [3,453 kB] 2024-09-18T04:34:46.628 INFO:teuthology.orchestra.run.smithi082.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2024-09-18T04:34:46.631 INFO:teuthology.orchestra.run.smithi082.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2024-09-18T04:34:46.649 INFO:teuthology.orchestra.run.smithi082.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2024-09-18T04:34:46.660 INFO:teuthology.orchestra.run.smithi115.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2024-09-18T04:34:46.662 INFO:teuthology.orchestra.run.smithi115.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2024-09-18T04:34:46.668 INFO:teuthology.orchestra.run.smithi115.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2024-09-18T04:34:46.670 INFO:teuthology.orchestra.run.smithi115.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2024-09-18T04:34:46.670 INFO:teuthology.orchestra.run.smithi115.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2024-09-18T04:34:46.671 INFO:teuthology.orchestra.run.smithi115.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2024-09-18T04:34:46.672 INFO:teuthology.orchestra.run.smithi115.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2024-09-18T04:34:46.675 INFO:teuthology.orchestra.run.smithi115.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2024-09-18T04:34:46.675 INFO:teuthology.orchestra.run.smithi115.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2024-09-18T04:34:46.677 INFO:teuthology.orchestra.run.smithi115.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2024-09-18T04:34:46.694 INFO:teuthology.orchestra.run.smithi115.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2024-09-18T04:34:46.694 INFO:teuthology.orchestra.run.smithi115.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2024-09-18T04:34:46.695 INFO:teuthology.orchestra.run.smithi115.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2024-09-18T04:34:46.695 INFO:teuthology.orchestra.run.smithi115.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-18T04:34:46.696 INFO:teuthology.orchestra.run.smithi115.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2024-09-18T04:34:46.697 INFO:teuthology.orchestra.run.smithi115.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-webob all 1:1.8.5-2ubuntu0.1 [84.5 kB] 2024-09-18T04:34:46.698 INFO:teuthology.orchestra.run.smithi115.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2024-09-18T04:34:46.698 INFO:teuthology.orchestra.run.smithi115.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2024-09-18T04:34:46.699 INFO:teuthology.orchestra.run.smithi028.stdout:2 upgraded, 111 newly installed, 0 to remove and 295 not upgraded. 2024-09-18T04:34:46.699 INFO:teuthology.orchestra.run.smithi028.stdout:Need to get 199 MB of archives. 2024-09-18T04:34:46.699 INFO:teuthology.orchestra.run.smithi028.stdout:After this operation, 978 MB of additional disk space will be used. 2024-09-18T04:34:46.699 INFO:teuthology.orchestra.run.smithi028.stdout:Get:1 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 18.2.4-855-g19e2a2be-1focal [3,453 kB] 2024-09-18T04:34:46.700 INFO:teuthology.orchestra.run.smithi115.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2024-09-18T04:34:46.703 INFO:teuthology.orchestra.run.smithi082.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2024-09-18T04:34:46.705 INFO:teuthology.orchestra.run.smithi082.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2024-09-18T04:34:46.711 INFO:teuthology.orchestra.run.smithi082.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2024-09-18T04:34:46.712 INFO:teuthology.orchestra.run.smithi115.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2024-09-18T04:34:46.712 INFO:teuthology.orchestra.run.smithi115.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2024-09-18T04:34:46.713 INFO:teuthology.orchestra.run.smithi082.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2024-09-18T04:34:46.713 INFO:teuthology.orchestra.run.smithi082.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2024-09-18T04:34:46.715 INFO:teuthology.orchestra.run.smithi082.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2024-09-18T04:34:46.715 INFO:teuthology.orchestra.run.smithi082.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2024-09-18T04:34:46.718 INFO:teuthology.orchestra.run.smithi082.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2024-09-18T04:34:46.718 INFO:teuthology.orchestra.run.smithi082.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2024-09-18T04:34:46.720 INFO:teuthology.orchestra.run.smithi082.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2024-09-18T04:34:46.729 INFO:teuthology.orchestra.run.smithi115.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2024-09-18T04:34:46.730 INFO:teuthology.orchestra.run.smithi115.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2024-09-18T04:34:46.730 INFO:teuthology.orchestra.run.smithi115.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2024-09-18T04:34:46.730 INFO:teuthology.orchestra.run.smithi115.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2024-09-18T04:34:46.731 INFO:teuthology.orchestra.run.smithi115.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.2 [183 kB] 2024-09-18T04:34:46.733 INFO:teuthology.orchestra.run.smithi115.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2024-09-18T04:34:46.734 INFO:teuthology.orchestra.run.smithi115.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.3 [211 kB] 2024-09-18T04:34:46.736 INFO:teuthology.orchestra.run.smithi115.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2024-09-18T04:34:46.738 INFO:teuthology.orchestra.run.smithi082.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2024-09-18T04:34:46.738 INFO:teuthology.orchestra.run.smithi082.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2024-09-18T04:34:46.739 INFO:teuthology.orchestra.run.smithi082.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2024-09-18T04:34:46.740 INFO:teuthology.orchestra.run.smithi082.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-18T04:34:46.740 INFO:teuthology.orchestra.run.smithi082.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2024-09-18T04:34:46.741 INFO:teuthology.orchestra.run.smithi082.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-webob all 1:1.8.5-2ubuntu0.1 [84.5 kB] 2024-09-18T04:34:46.742 INFO:teuthology.orchestra.run.smithi082.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2024-09-18T04:34:46.742 INFO:teuthology.orchestra.run.smithi082.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2024-09-18T04:34:46.744 INFO:teuthology.orchestra.run.smithi082.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2024-09-18T04:34:46.744 INFO:teuthology.orchestra.run.smithi028.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2024-09-18T04:34:46.745 INFO:teuthology.orchestra.run.smithi115.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-asyncssh all 1.12.2-1 [152 kB] 2024-09-18T04:34:46.747 INFO:teuthology.orchestra.run.smithi115.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jinja2 all 2.10.1-2ubuntu0.3 [96.0 kB] 2024-09-18T04:34:46.755 INFO:teuthology.orchestra.run.smithi082.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2024-09-18T04:34:46.762 INFO:teuthology.orchestra.run.smithi115.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2024-09-18T04:34:46.763 INFO:teuthology.orchestra.run.smithi115.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2024-09-18T04:34:46.764 INFO:teuthology.orchestra.run.smithi082.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2024-09-18T04:34:46.764 INFO:teuthology.orchestra.run.smithi115.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2024-09-18T04:34:46.765 INFO:teuthology.orchestra.run.smithi115.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2024-09-18T04:34:46.767 INFO:teuthology.orchestra.run.smithi115.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2024-09-18T04:34:46.767 INFO:teuthology.orchestra.run.smithi115.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2024-09-18T04:34:46.768 INFO:teuthology.orchestra.run.smithi115.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2024-09-18T04:34:46.769 INFO:teuthology.orchestra.run.smithi115.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2024-09-18T04:34:46.771 INFO:teuthology.orchestra.run.smithi082.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2024-09-18T04:34:46.772 INFO:teuthology.orchestra.run.smithi082.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2024-09-18T04:34:46.772 INFO:teuthology.orchestra.run.smithi082.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2024-09-18T04:34:46.773 INFO:teuthology.orchestra.run.smithi082.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2024-09-18T04:34:46.775 INFO:teuthology.orchestra.run.smithi082.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.2 [183 kB] 2024-09-18T04:34:46.778 INFO:teuthology.orchestra.run.smithi115.stdout:Get:45 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 18.2.4-855-g19e2a2be-1focal [4,044 kB] 2024-09-18T04:34:46.779 INFO:teuthology.orchestra.run.smithi082.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2024-09-18T04:34:46.779 INFO:teuthology.orchestra.run.smithi082.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.3 [211 kB] 2024-09-18T04:34:46.780 INFO:teuthology.orchestra.run.smithi115.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-18T04:34:46.780 INFO:teuthology.orchestra.run.smithi115.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2024-09-18T04:34:46.781 INFO:teuthology.orchestra.run.smithi082.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2024-09-18T04:34:46.790 INFO:teuthology.orchestra.run.smithi082.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-asyncssh all 1.12.2-1 [152 kB] 2024-09-18T04:34:46.792 INFO:teuthology.orchestra.run.smithi082.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jinja2 all 2.10.1-2ubuntu0.3 [96.0 kB] 2024-09-18T04:34:46.793 INFO:teuthology.orchestra.run.smithi082.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2024-09-18T04:34:46.796 INFO:teuthology.orchestra.run.smithi115.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2024-09-18T04:34:46.798 INFO:teuthology.orchestra.run.smithi115.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2024-09-18T04:34:46.800 INFO:teuthology.orchestra.run.smithi115.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2024-09-18T04:34:46.801 INFO:teuthology.orchestra.run.smithi115.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2024-09-18T04:34:46.802 INFO:teuthology.orchestra.run.smithi115.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2024-09-18T04:34:46.807 INFO:teuthology.orchestra.run.smithi082.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2024-09-18T04:34:46.808 INFO:teuthology.orchestra.run.smithi082.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2024-09-18T04:34:46.808 INFO:teuthology.orchestra.run.smithi082.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2024-09-18T04:34:46.810 INFO:teuthology.orchestra.run.smithi082.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2024-09-18T04:34:46.810 INFO:teuthology.orchestra.run.smithi082.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2024-09-18T04:34:46.811 INFO:teuthology.orchestra.run.smithi082.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2024-09-18T04:34:46.812 INFO:teuthology.orchestra.run.smithi082.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2024-09-18T04:34:46.824 INFO:teuthology.orchestra.run.smithi082.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-18T04:34:46.825 INFO:teuthology.orchestra.run.smithi082.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2024-09-18T04:34:46.826 INFO:teuthology.orchestra.run.smithi082.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2024-09-18T04:34:46.842 INFO:teuthology.orchestra.run.smithi082.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2024-09-18T04:34:46.844 INFO:teuthology.orchestra.run.smithi082.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2024-09-18T04:34:46.844 INFO:teuthology.orchestra.run.smithi082.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2024-09-18T04:34:46.845 INFO:teuthology.orchestra.run.smithi082.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2024-09-18T04:34:46.846 INFO:teuthology.orchestra.run.smithi082.stdout:Get:52 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 18.2.4-855-g19e2a2be-1focal [4,044 kB] 2024-09-18T04:34:46.880 INFO:teuthology.orchestra.run.smithi115.stdout:Get:53 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 18.2.4-855-g19e2a2be-1focal [746 kB] 2024-09-18T04:34:46.927 INFO:teuthology.orchestra.run.smithi082.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2024-09-18T04:34:46.943 INFO:teuthology.orchestra.run.smithi082.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2024-09-18T04:34:46.944 INFO:teuthology.orchestra.run.smithi082.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2024-09-18T04:34:46.946 INFO:teuthology.orchestra.run.smithi115.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2024-09-18T04:34:46.946 INFO:teuthology.orchestra.run.smithi115.stdout:Get:55 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 18.2.4-855-g19e2a2be-1focal [388 kB] 2024-09-18T04:34:46.952 INFO:teuthology.orchestra.run.smithi115.stdout:Get:56 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 18.2.4-855-g19e2a2be-1focal [31.9 kB] 2024-09-18T04:34:46.953 INFO:teuthology.orchestra.run.smithi115.stdout:Get:57 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 18.2.4-855-g19e2a2be-1focal [201 kB] 2024-09-18T04:34:46.956 INFO:teuthology.orchestra.run.smithi082.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2024-09-18T04:34:46.957 INFO:teuthology.orchestra.run.smithi115.stdout:Get:58 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 18.2.4-855-g19e2a2be-1focal [62.5 kB] 2024-09-18T04:34:46.958 INFO:teuthology.orchestra.run.smithi115.stdout:Get:59 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 18.2.4-855-g19e2a2be-1focal [367 kB] 2024-09-18T04:34:46.962 INFO:teuthology.orchestra.run.smithi115.stdout:Get:60 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 18.2.4-855-g19e2a2be-1focal [6,627 kB] 2024-09-18T04:34:46.967 INFO:teuthology.orchestra.run.smithi115.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2024-09-18T04:34:46.969 INFO:teuthology.orchestra.run.smithi115.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2024-09-18T04:34:46.981 INFO:teuthology.orchestra.run.smithi115.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2024-09-18T04:34:46.981 INFO:teuthology.orchestra.run.smithi115.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2024-09-18T04:34:46.981 INFO:teuthology.orchestra.run.smithi115.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2024-09-18T04:34:46.982 INFO:teuthology.orchestra.run.smithi115.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2024-09-18T04:34:46.982 INFO:teuthology.orchestra.run.smithi115.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2024-09-18T04:34:46.982 INFO:teuthology.orchestra.run.smithi115.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2024-09-18T04:34:46.983 INFO:teuthology.orchestra.run.smithi115.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2024-09-18T04:34:46.985 INFO:teuthology.orchestra.run.smithi115.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2024-09-18T04:34:47.001 INFO:teuthology.orchestra.run.smithi115.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2024-09-18T04:34:47.002 INFO:teuthology.orchestra.run.smithi115.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2024-09-18T04:34:47.005 INFO:teuthology.orchestra.run.smithi115.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2024-09-18T04:34:47.007 INFO:teuthology.orchestra.run.smithi115.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2024-09-18T04:34:47.008 INFO:teuthology.orchestra.run.smithi115.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2024-09-18T04:34:47.009 INFO:teuthology.orchestra.run.smithi115.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2024-09-18T04:34:47.012 INFO:teuthology.orchestra.run.smithi115.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2024-09-18T04:34:47.015 INFO:teuthology.orchestra.run.smithi115.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2024-09-18T04:34:47.021 INFO:teuthology.orchestra.run.smithi082.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2024-09-18T04:34:47.021 INFO:teuthology.orchestra.run.smithi082.stdout:Get:58 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 18.2.4-855-g19e2a2be-1focal [746 kB] 2024-09-18T04:34:47.022 INFO:teuthology.orchestra.run.smithi082.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2024-09-18T04:34:47.023 INFO:teuthology.orchestra.run.smithi082.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2024-09-18T04:34:47.023 INFO:teuthology.orchestra.run.smithi082.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2024-09-18T04:34:47.024 INFO:teuthology.orchestra.run.smithi082.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2024-09-18T04:34:47.025 INFO:teuthology.orchestra.run.smithi082.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2024-09-18T04:34:47.026 INFO:teuthology.orchestra.run.smithi082.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2024-09-18T04:34:47.026 INFO:teuthology.orchestra.run.smithi082.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2024-09-18T04:34:47.031 INFO:teuthology.orchestra.run.smithi115.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.2 [283 kB] 2024-09-18T04:34:47.033 INFO:teuthology.orchestra.run.smithi115.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-asyncssh-doc all 1.12.2-1 [258 kB] 2024-09-18T04:34:47.034 INFO:teuthology.orchestra.run.smithi082.stdout:Get:66 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 18.2.4-855-g19e2a2be-1focal [388 kB] 2024-09-18T04:34:47.035 INFO:teuthology.orchestra.run.smithi115.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2024-09-18T04:34:47.037 INFO:teuthology.orchestra.run.smithi082.stdout:Get:67 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 18.2.4-855-g19e2a2be-1focal [31.9 kB] 2024-09-18T04:34:47.037 INFO:teuthology.orchestra.run.smithi082.stdout:Get:68 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 18.2.4-855-g19e2a2be-1focal [201 kB] 2024-09-18T04:34:47.039 INFO:teuthology.orchestra.run.smithi082.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2024-09-18T04:34:47.039 INFO:teuthology.orchestra.run.smithi082.stdout:Get:70 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 18.2.4-855-g19e2a2be-1focal [62.5 kB] 2024-09-18T04:34:47.039 INFO:teuthology.orchestra.run.smithi082.stdout:Get:71 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 18.2.4-855-g19e2a2be-1focal [367 kB] 2024-09-18T04:34:47.042 INFO:teuthology.orchestra.run.smithi082.stdout:Get:72 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 18.2.4-855-g19e2a2be-1focal [6,627 kB] 2024-09-18T04:34:47.045 INFO:teuthology.orchestra.run.smithi082.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2024-09-18T04:34:47.047 INFO:teuthology.orchestra.run.smithi082.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2024-09-18T04:34:47.049 INFO:teuthology.orchestra.run.smithi082.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2024-09-18T04:34:47.050 INFO:teuthology.orchestra.run.smithi082.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2024-09-18T04:34:47.054 INFO:teuthology.orchestra.run.smithi082.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2024-09-18T04:34:47.058 INFO:teuthology.orchestra.run.smithi082.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2024-09-18T04:34:47.073 INFO:teuthology.orchestra.run.smithi082.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.2 [283 kB] 2024-09-18T04:34:47.074 INFO:teuthology.orchestra.run.smithi115.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2024-09-18T04:34:47.075 INFO:teuthology.orchestra.run.smithi115.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2024-09-18T04:34:47.075 INFO:teuthology.orchestra.run.smithi115.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2024-09-18T04:34:47.076 INFO:teuthology.orchestra.run.smithi082.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-asyncssh-doc all 1.12.2-1 [258 kB] 2024-09-18T04:34:47.076 INFO:teuthology.orchestra.run.smithi115.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2024-09-18T04:34:47.077 INFO:teuthology.orchestra.run.smithi082.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2024-09-18T04:34:47.087 INFO:teuthology.orchestra.run.smithi115.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-18T04:34:47.088 INFO:teuthology.orchestra.run.smithi115.stdout:Get:87 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2024-09-18T04:34:47.089 INFO:teuthology.orchestra.run.smithi115.stdout:Get:88 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2024-09-18T04:34:47.089 INFO:teuthology.orchestra.run.smithi115.stdout:Get:89 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2024-09-18T04:34:47.089 INFO:teuthology.orchestra.run.smithi115.stdout:Get:90 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2024-09-18T04:34:47.114 INFO:teuthology.orchestra.run.smithi082.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2024-09-18T04:34:47.115 INFO:teuthology.orchestra.run.smithi082.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2024-09-18T04:34:47.115 INFO:teuthology.orchestra.run.smithi082.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2024-09-18T04:34:47.116 INFO:teuthology.orchestra.run.smithi082.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2024-09-18T04:34:47.128 INFO:teuthology.orchestra.run.smithi082.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-18T04:34:47.128 INFO:teuthology.orchestra.run.smithi082.stdout:Get:87 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2024-09-18T04:34:47.130 INFO:teuthology.orchestra.run.smithi082.stdout:Get:88 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2024-09-18T04:34:47.130 INFO:teuthology.orchestra.run.smithi082.stdout:Get:89 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2024-09-18T04:34:47.130 INFO:teuthology.orchestra.run.smithi082.stdout:Get:90 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2024-09-18T04:34:47.144 INFO:teuthology.orchestra.run.smithi028.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2024-09-18T04:34:47.144 INFO:teuthology.orchestra.run.smithi115.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 18.2.4-855-g19e2a2be-1focal [113 kB] 2024-09-18T04:34:47.145 INFO:teuthology.orchestra.run.smithi115.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 18.2.4-855-g19e2a2be-1focal [478 kB] 2024-09-18T04:34:47.157 INFO:teuthology.orchestra.run.smithi028.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2024-09-18T04:34:47.169 INFO:teuthology.orchestra.run.smithi115.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 18.2.4-855-g19e2a2be-1focal [25.2 MB] 2024-09-18T04:34:47.241 INFO:teuthology.orchestra.run.smithi028.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2024-09-18T04:34:47.265 INFO:teuthology.orchestra.run.smithi082.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 18.2.4-855-g19e2a2be-1focal [113 kB] 2024-09-18T04:34:47.266 INFO:teuthology.orchestra.run.smithi082.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 18.2.4-855-g19e2a2be-1focal [478 kB] 2024-09-18T04:34:47.273 INFO:teuthology.orchestra.run.smithi082.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 18.2.4-855-g19e2a2be-1focal [25.2 MB] 2024-09-18T04:34:47.484 INFO:teuthology.orchestra.run.smithi028.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2024-09-18T04:34:47.493 INFO:teuthology.orchestra.run.smithi028.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2024-09-18T04:34:47.521 INFO:teuthology.orchestra.run.smithi028.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2024-09-18T04:34:47.529 INFO:teuthology.orchestra.run.smithi028.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2024-09-18T04:34:47.530 INFO:teuthology.orchestra.run.smithi028.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2024-09-18T04:34:47.534 INFO:teuthology.orchestra.run.smithi028.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2024-09-18T04:34:47.535 INFO:teuthology.orchestra.run.smithi028.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2024-09-18T04:34:47.547 INFO:teuthology.orchestra.run.smithi028.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2024-09-18T04:34:47.549 INFO:teuthology.orchestra.run.smithi028.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2024-09-18T04:34:47.566 INFO:teuthology.orchestra.run.smithi028.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2024-09-18T04:34:47.636 INFO:teuthology.orchestra.run.smithi028.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2024-09-18T04:34:47.637 INFO:teuthology.orchestra.run.smithi028.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2024-09-18T04:34:47.638 INFO:teuthology.orchestra.run.smithi028.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2024-09-18T04:34:47.640 INFO:teuthology.orchestra.run.smithi028.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-18T04:34:47.641 INFO:teuthology.orchestra.run.smithi028.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2024-09-18T04:34:47.642 INFO:teuthology.orchestra.run.smithi028.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-webob all 1:1.8.5-2ubuntu0.1 [84.5 kB] 2024-09-18T04:34:47.644 INFO:teuthology.orchestra.run.smithi028.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2024-09-18T04:34:47.646 INFO:teuthology.orchestra.run.smithi028.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2024-09-18T04:34:47.649 INFO:teuthology.orchestra.run.smithi028.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2024-09-18T04:34:47.729 INFO:teuthology.orchestra.run.smithi028.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2024-09-18T04:34:47.730 INFO:teuthology.orchestra.run.smithi028.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2024-09-18T04:34:47.740 INFO:teuthology.orchestra.run.smithi028.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2024-09-18T04:34:47.740 INFO:teuthology.orchestra.run.smithi028.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2024-09-18T04:34:47.741 INFO:teuthology.orchestra.run.smithi028.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2024-09-18T04:34:47.742 INFO:teuthology.orchestra.run.smithi028.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2024-09-18T04:34:47.743 INFO:teuthology.orchestra.run.smithi028.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.2 [183 kB] 2024-09-18T04:34:47.745 INFO:teuthology.orchestra.run.smithi028.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2024-09-18T04:34:47.811 INFO:teuthology.orchestra.run.smithi028.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.3 [211 kB] 2024-09-18T04:34:47.815 INFO:teuthology.orchestra.run.smithi028.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2024-09-18T04:34:47.892 INFO:teuthology.orchestra.run.smithi028.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-asyncssh all 1.12.2-1 [152 kB] 2024-09-18T04:34:47.895 INFO:teuthology.orchestra.run.smithi028.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jinja2 all 2.10.1-2ubuntu0.3 [96.0 kB] 2024-09-18T04:34:47.898 INFO:teuthology.orchestra.run.smithi028.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2024-09-18T04:34:47.900 INFO:teuthology.orchestra.run.smithi028.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2024-09-18T04:34:47.901 INFO:teuthology.orchestra.run.smithi028.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2024-09-18T04:34:47.902 INFO:teuthology.orchestra.run.smithi028.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2024-09-18T04:34:47.903 INFO:teuthology.orchestra.run.smithi028.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2024-09-18T04:34:47.903 INFO:teuthology.orchestra.run.smithi028.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2024-09-18T04:34:47.974 INFO:teuthology.orchestra.run.smithi028.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2024-09-18T04:34:47.974 INFO:teuthology.orchestra.run.smithi028.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2024-09-18T04:34:47.986 INFO:teuthology.orchestra.run.smithi028.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-18T04:34:47.987 INFO:teuthology.orchestra.run.smithi028.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2024-09-18T04:34:47.989 INFO:teuthology.orchestra.run.smithi028.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2024-09-18T04:34:47.990 INFO:teuthology.orchestra.run.smithi028.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2024-09-18T04:34:48.056 INFO:teuthology.orchestra.run.smithi028.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2024-09-18T04:34:48.057 INFO:teuthology.orchestra.run.smithi028.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2024-09-18T04:34:48.058 INFO:teuthology.orchestra.run.smithi028.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2024-09-18T04:34:48.139 INFO:teuthology.orchestra.run.smithi028.stdout:Get:52 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 18.2.4-855-g19e2a2be-1focal [4,044 kB] 2024-09-18T04:34:48.157 INFO:teuthology.orchestra.run.smithi115.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 18.2.4-855-g19e2a2be-1focal [9,605 kB] 2024-09-18T04:34:48.276 INFO:teuthology.orchestra.run.smithi082.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 18.2.4-855-g19e2a2be-1focal [9,605 kB] 2024-09-18T04:34:48.325 INFO:teuthology.orchestra.run.smithi115.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 18.2.4-855-g19e2a2be-1focal [238 kB] 2024-09-18T04:34:48.329 INFO:teuthology.orchestra.run.smithi115.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 18.2.4-855-g19e2a2be-1focal [148 kB] 2024-09-18T04:34:48.332 INFO:teuthology.orchestra.run.smithi115.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 18.2.4-855-g19e2a2be-1focal [1,574 kB] 2024-09-18T04:34:48.339 INFO:teuthology.orchestra.run.smithi028.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2024-09-18T04:34:48.401 INFO:teuthology.orchestra.run.smithi115.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 18.2.4-855-g19e2a2be-1focal [6,640 kB] 2024-09-18T04:34:48.413 INFO:teuthology.orchestra.run.smithi028.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2024-09-18T04:34:48.417 INFO:teuthology.orchestra.run.smithi028.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2024-09-18T04:34:48.484 INFO:teuthology.orchestra.run.smithi028.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2024-09-18T04:34:48.485 INFO:teuthology.orchestra.run.smithi028.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2024-09-18T04:34:48.485 INFO:teuthology.orchestra.run.smithi028.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2024-09-18T04:34:48.486 INFO:teuthology.orchestra.run.smithi028.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2024-09-18T04:34:48.486 INFO:teuthology.orchestra.run.smithi028.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2024-09-18T04:34:48.487 INFO:teuthology.orchestra.run.smithi028.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2024-09-18T04:34:48.487 INFO:teuthology.orchestra.run.smithi028.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2024-09-18T04:34:48.533 INFO:teuthology.orchestra.run.smithi082.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 18.2.4-855-g19e2a2be-1focal [238 kB] 2024-09-18T04:34:48.535 INFO:teuthology.orchestra.run.smithi082.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 18.2.4-855-g19e2a2be-1focal [148 kB] 2024-09-18T04:34:48.537 INFO:teuthology.orchestra.run.smithi082.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 18.2.4-855-g19e2a2be-1focal [1,574 kB] 2024-09-18T04:34:48.546 INFO:teuthology.orchestra.run.smithi028.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2024-09-18T04:34:48.547 INFO:teuthology.orchestra.run.smithi028.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2024-09-18T04:34:48.604 INFO:teuthology.orchestra.run.smithi115.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 18.2.4-855-g19e2a2be-1focal [24.5 MB] 2024-09-18T04:34:48.616 INFO:teuthology.orchestra.run.smithi082.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 18.2.4-855-g19e2a2be-1focal [6,640 kB] 2024-09-18T04:34:48.628 INFO:teuthology.orchestra.run.smithi028.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2024-09-18T04:34:48.633 INFO:teuthology.orchestra.run.smithi028.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2024-09-18T04:34:48.637 INFO:teuthology.orchestra.run.smithi028.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2024-09-18T04:34:48.640 INFO:teuthology.orchestra.run.smithi028.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2024-09-18T04:34:48.642 INFO:teuthology.orchestra.run.smithi028.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2024-09-18T04:34:48.645 INFO:teuthology.orchestra.run.smithi028.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2024-09-18T04:34:48.647 INFO:teuthology.orchestra.run.smithi028.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2024-09-18T04:34:48.720 INFO:teuthology.orchestra.run.smithi028.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.2 [283 kB] 2024-09-18T04:34:48.724 INFO:teuthology.orchestra.run.smithi028.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-asyncssh-doc all 1.12.2-1 [258 kB] 2024-09-18T04:34:48.726 INFO:teuthology.orchestra.run.smithi028.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2024-09-18T04:34:48.862 INFO:teuthology.orchestra.run.smithi082.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 18.2.4-855-g19e2a2be-1focal [24.5 MB] 2024-09-18T04:34:48.889 INFO:teuthology.orchestra.run.smithi028.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2024-09-18T04:34:48.890 INFO:teuthology.orchestra.run.smithi028.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2024-09-18T04:34:48.890 INFO:teuthology.orchestra.run.smithi028.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2024-09-18T04:34:48.891 INFO:teuthology.orchestra.run.smithi028.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2024-09-18T04:34:48.903 INFO:teuthology.orchestra.run.smithi028.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-18T04:34:48.904 INFO:teuthology.orchestra.run.smithi028.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2024-09-18T04:34:48.905 INFO:teuthology.orchestra.run.smithi028.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2024-09-18T04:34:48.906 INFO:teuthology.orchestra.run.smithi028.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2024-09-18T04:34:48.957 INFO:teuthology.orchestra.run.smithi028.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2024-09-18T04:34:49.509 INFO:teuthology.orchestra.run.smithi115.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 18.2.4-855-g19e2a2be-1focal [13.7 kB] 2024-09-18T04:34:49.510 INFO:teuthology.orchestra.run.smithi115.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 18.2.4-855-g19e2a2be-1focal [856 kB] 2024-09-18T04:34:49.516 INFO:teuthology.orchestra.run.smithi028.stdout:Get:84 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 18.2.4-855-g19e2a2be-1focal [746 kB] 2024-09-18T04:34:49.525 INFO:teuthology.orchestra.run.smithi115.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 18.2.4-855-g19e2a2be-1focal [2,295 kB] 2024-09-18T04:34:49.541 INFO:teuthology.orchestra.run.smithi115.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 18.2.4-855-g19e2a2be-1focal [239 kB] 2024-09-18T04:34:49.543 INFO:teuthology.orchestra.run.smithi115.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 18.2.4-855-g19e2a2be-1focal [143 kB] 2024-09-18T04:34:49.544 INFO:teuthology.orchestra.run.smithi115.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 18.2.4-855-g19e2a2be-1focal [7,015 kB] 2024-09-18T04:34:49.769 INFO:teuthology.orchestra.run.smithi115.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 18.2.4-855-g19e2a2be-1focal [7,893 kB] 2024-09-18T04:34:49.779 INFO:teuthology.orchestra.run.smithi028.stdout:Get:85 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 18.2.4-855-g19e2a2be-1focal [388 kB] 2024-09-18T04:34:49.819 INFO:teuthology.orchestra.run.smithi082.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 18.2.4-855-g19e2a2be-1focal [13.7 kB] 2024-09-18T04:34:49.822 INFO:teuthology.orchestra.run.smithi082.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 18.2.4-855-g19e2a2be-1focal [856 kB] 2024-09-18T04:34:49.836 INFO:teuthology.orchestra.run.smithi082.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 18.2.4-855-g19e2a2be-1focal [2,295 kB] 2024-09-18T04:34:49.852 INFO:teuthology.orchestra.run.smithi082.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 18.2.4-855-g19e2a2be-1focal [239 kB] 2024-09-18T04:34:49.871 INFO:teuthology.orchestra.run.smithi082.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 18.2.4-855-g19e2a2be-1focal [143 kB] 2024-09-18T04:34:49.873 INFO:teuthology.orchestra.run.smithi082.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 18.2.4-855-g19e2a2be-1focal [7,015 kB] 2024-09-18T04:34:49.925 INFO:teuthology.orchestra.run.smithi028.stdout:Get:86 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 18.2.4-855-g19e2a2be-1focal [31.9 kB] 2024-09-18T04:34:49.927 INFO:teuthology.orchestra.run.smithi028.stdout:Get:87 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 18.2.4-855-g19e2a2be-1focal [201 kB] 2024-09-18T04:34:49.993 INFO:teuthology.orchestra.run.smithi028.stdout:Get:88 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 18.2.4-855-g19e2a2be-1focal [62.5 kB] 2024-09-18T04:34:50.001 INFO:teuthology.orchestra.run.smithi115.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 18.2.4-855-g19e2a2be-1focal [13.9 kB] 2024-09-18T04:34:50.002 INFO:teuthology.orchestra.run.smithi115.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 18.2.4-855-g19e2a2be-1focal [46.9 MB] 2024-09-18T04:34:50.041 INFO:teuthology.orchestra.run.smithi028.stdout:Get:89 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 18.2.4-855-g19e2a2be-1focal [367 kB] 2024-09-18T04:34:50.129 INFO:teuthology.orchestra.run.smithi082.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 18.2.4-855-g19e2a2be-1focal [7,893 kB] 2024-09-18T04:34:50.170 INFO:teuthology.orchestra.run.smithi028.stdout:Get:90 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 18.2.4-855-g19e2a2be-1focal [6,627 kB] 2024-09-18T04:34:50.412 INFO:teuthology.orchestra.run.smithi082.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 18.2.4-855-g19e2a2be-1focal [13.9 kB] 2024-09-18T04:34:50.412 INFO:teuthology.orchestra.run.smithi082.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 18.2.4-855-g19e2a2be-1focal [46.9 MB] 2024-09-18T04:34:51.757 INFO:teuthology.orchestra.run.smithi115.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 18.2.4-855-g19e2a2be-1focal [120 kB] 2024-09-18T04:34:51.759 INFO:teuthology.orchestra.run.smithi115.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 18.2.4-855-g19e2a2be-1focal [37.4 kB] 2024-09-18T04:34:51.760 INFO:teuthology.orchestra.run.smithi115.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 18.2.4-855-g19e2a2be-1focal [13.5 MB] 2024-09-18T04:34:52.145 INFO:teuthology.orchestra.run.smithi115.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 18.2.4-855-g19e2a2be-1focal [108 kB] 2024-09-18T04:34:52.147 INFO:teuthology.orchestra.run.smithi115.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 rbd-nbd amd64 18.2.4-855-g19e2a2be-1focal [189 kB] 2024-09-18T04:34:52.178 INFO:teuthology.orchestra.run.smithi082.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 18.2.4-855-g19e2a2be-1focal [120 kB] 2024-09-18T04:34:52.181 INFO:teuthology.orchestra.run.smithi082.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 18.2.4-855-g19e2a2be-1focal [37.4 kB] 2024-09-18T04:34:52.182 INFO:teuthology.orchestra.run.smithi082.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 18.2.4-855-g19e2a2be-1focal [13.5 MB] 2024-09-18T04:34:52.345 INFO:teuthology.orchestra.run.smithi028.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 18.2.4-855-g19e2a2be-1focal [113 kB] 2024-09-18T04:34:52.366 INFO:teuthology.orchestra.run.smithi082.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 18.2.4-855-g19e2a2be-1focal [108 kB] 2024-09-18T04:34:52.367 INFO:teuthology.orchestra.run.smithi082.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 rbd-nbd amd64 18.2.4-855-g19e2a2be-1focal [189 kB] 2024-09-18T04:34:52.370 INFO:teuthology.orchestra.run.smithi028.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 18.2.4-855-g19e2a2be-1focal [478 kB] 2024-09-18T04:34:52.414 INFO:teuthology.orchestra.run.smithi028.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 18.2.4-855-g19e2a2be-1focal [25.2 MB] 2024-09-18T04:34:52.801 INFO:teuthology.orchestra.run.smithi115.stdout:Fetched 199 MB in 6s (34.8 MB/s) 2024-09-18T04:34:52.970 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2024-09-18T04:34:53.029 INFO:teuthology.orchestra.run.smithi115.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141720 files and directories currently installed.) 2024-09-18T04:34:53.033 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2024-09-18T04:34:53.074 INFO:teuthology.orchestra.run.smithi082.stdout:Fetched 199 MB in 6s (33.5 MB/s) 2024-09-18T04:34:53.118 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-18T04:34:53.254 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2024-09-18T04:34:53.311 INFO:teuthology.orchestra.run.smithi082.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141720 files and directories currently installed.) 2024-09-18T04:34:53.316 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2024-09-18T04:34:53.420 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-18T04:34:53.497 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-18T04:34:53.514 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2024-09-18T04:34:53.547 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-18T04:34:53.756 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-18T04:34:53.775 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2024-09-18T04:34:53.814 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-18T04:34:53.816 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-18T04:34:53.835 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../002-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2024-09-18T04:34:53.866 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-18T04:34:54.167 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-18T04:34:54.184 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../002-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2024-09-18T04:34:54.251 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-18T04:34:54.328 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-18T04:34:54.346 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../003-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-18T04:34:54.415 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T04:34:54.713 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-18T04:34:54.729 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../003-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-18T04:34:54.781 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T04:34:54.862 INFO:teuthology.orchestra.run.smithi028.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 18.2.4-855-g19e2a2be-1focal [9,605 kB] 2024-09-18T04:34:54.922 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-18T04:34:54.939 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../004-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-18T04:34:54.989 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T04:34:55.299 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-18T04:34:55.308 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-18T04:34:55.315 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../004-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-18T04:34:55.324 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../005-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-18T04:34:55.357 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T04:34:55.358 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T04:34:55.551 INFO:teuthology.orchestra.run.smithi028.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 18.2.4-855-g19e2a2be-1focal [238 kB] 2024-09-18T04:34:55.569 INFO:teuthology.orchestra.run.smithi028.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 18.2.4-855-g19e2a2be-1focal [148 kB] 2024-09-18T04:34:55.573 INFO:teuthology.orchestra.run.smithi028.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 18.2.4-855-g19e2a2be-1focal [1,574 kB] 2024-09-18T04:34:55.693 INFO:teuthology.orchestra.run.smithi028.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 18.2.4-855-g19e2a2be-1focal [6,640 kB] 2024-09-18T04:34:55.711 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-18T04:34:55.726 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../005-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-18T04:34:55.753 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package libthrift-0.13.0. 2024-09-18T04:34:55.769 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../006-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2024-09-18T04:34:55.769 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T04:34:55.811 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-18T04:34:56.150 INFO:teuthology.orchestra.run.smithi028.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 18.2.4-855-g19e2a2be-1focal [24.5 MB] 2024-09-18T04:34:56.164 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libthrift-0.13.0. 2024-09-18T04:34:56.173 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../007-librbd1_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:34:56.183 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../006-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2024-09-18T04:34:56.213 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-18T04:34:56.306 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking librbd1 (18.2.4-855-g19e2a2be-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-18T04:34:56.549 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../007-librbd1_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:34:56.675 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking librbd1 (18.2.4-855-g19e2a2be-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-18T04:34:56.925 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../008-librados2_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:34:57.043 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking librados2 (18.2.4-855-g19e2a2be-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-18T04:34:57.353 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../008-librados2_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:34:57.480 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking librados2 (18.2.4-855-g19e2a2be-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-18T04:34:57.751 INFO:teuthology.orchestra.run.smithi028.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 18.2.4-855-g19e2a2be-1focal [13.7 kB] 2024-09-18T04:34:57.752 INFO:teuthology.orchestra.run.smithi028.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 18.2.4-855-g19e2a2be-1focal [856 kB] 2024-09-18T04:34:57.763 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package libcephfs2. 2024-09-18T04:34:57.773 INFO:teuthology.orchestra.run.smithi028.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 18.2.4-855-g19e2a2be-1focal [2,295 kB] 2024-09-18T04:34:57.779 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../009-libcephfs2_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:34:57.822 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking libcephfs2 (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:34:57.901 INFO:teuthology.orchestra.run.smithi028.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 18.2.4-855-g19e2a2be-1focal [239 kB] 2024-09-18T04:34:57.918 INFO:teuthology.orchestra.run.smithi028.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 18.2.4-855-g19e2a2be-1focal [143 kB] 2024-09-18T04:34:57.920 INFO:teuthology.orchestra.run.smithi028.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 18.2.4-855-g19e2a2be-1focal [7,015 kB] 2024-09-18T04:34:58.158 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libcephfs2. 2024-09-18T04:34:58.166 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-rados. 2024-09-18T04:34:58.177 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../009-libcephfs2_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:34:58.182 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../010-python3-rados_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:34:58.216 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-rados (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:34:58.217 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libcephfs2 (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:34:58.342 INFO:teuthology.orchestra.run.smithi028.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 18.2.4-855-g19e2a2be-1focal [7,893 kB] 2024-09-18T04:34:58.552 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-18T04:34:58.570 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../011-python3-ceph-argparse_18.2.4-855-g19e2a2be-1focal_all.deb ... 2024-09-18T04:34:58.578 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-rados. 2024-09-18T04:34:58.593 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../010-python3-rados_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:34:58.635 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-ceph-argparse (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:34:58.636 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-rados (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:34:58.755 INFO:teuthology.orchestra.run.smithi028.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 18.2.4-855-g19e2a2be-1focal [13.9 kB] 2024-09-18T04:34:58.756 INFO:teuthology.orchestra.run.smithi028.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 18.2.4-855-g19e2a2be-1focal [46.9 MB] 2024-09-18T04:34:58.887 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-cephfs. 2024-09-18T04:34:58.906 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../012-python3-cephfs_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:34:58.946 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-cephfs (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:34:58.963 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-18T04:34:58.983 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../011-python3-ceph-argparse_18.2.4-855-g19e2a2be-1focal_all.deb ... 2024-09-18T04:34:59.022 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-ceph-argparse (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:34:59.241 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-cephfs. 2024-09-18T04:34:59.248 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-18T04:34:59.257 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../012-python3-cephfs_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:34:59.264 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../013-python3-ceph-common_18.2.4-855-g19e2a2be-1focal_all.deb ... 2024-09-18T04:34:59.298 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-ceph-common (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:34:59.299 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-cephfs (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:34:59.635 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-prettytable. 2024-09-18T04:34:59.652 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-18T04:34:59.654 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../014-python3-prettytable_0.7.2-5_all.deb ... 2024-09-18T04:34:59.668 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../013-python3-ceph-common_18.2.4-855-g19e2a2be-1focal_all.deb ... 2024-09-18T04:34:59.709 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2024-09-18T04:34:59.710 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-ceph-common (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:34:59.945 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-rbd. 2024-09-18T04:34:59.961 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../015-python3-rbd_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:34:59.995 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-rbd (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:00.030 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-prettytable. 2024-09-18T04:35:00.046 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../014-python3-prettytable_0.7.2-5_all.deb ... 2024-09-18T04:35:00.088 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2024-09-18T04:35:00.298 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2024-09-18T04:35:00.317 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../016-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2024-09-18T04:35:00.358 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-18T04:35:00.374 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-rbd. 2024-09-18T04:35:00.392 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../015-python3-rbd_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:00.432 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-rbd (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:00.703 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package librabbitmq4:amd64. 2024-09-18T04:35:00.721 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../017-librabbitmq4_0.10.0-1_amd64.deb ... 2024-09-18T04:35:00.760 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2024-09-18T04:35:00.777 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../016-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2024-09-18T04:35:00.777 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2024-09-18T04:35:00.818 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-18T04:35:01.047 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-18T04:35:01.066 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../018-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2024-09-18T04:35:01.105 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-18T04:35:01.149 INFO:teuthology.orchestra.run.smithi028.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 18.2.4-855-g19e2a2be-1focal [120 kB] 2024-09-18T04:35:01.152 INFO:teuthology.orchestra.run.smithi028.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 18.2.4-855-g19e2a2be-1focal [37.4 kB] 2024-09-18T04:35:01.153 INFO:teuthology.orchestra.run.smithi028.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 18.2.4-855-g19e2a2be-1focal [13.5 MB] 2024-09-18T04:35:01.205 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package librabbitmq4:amd64. 2024-09-18T04:35:01.221 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../017-librabbitmq4_0.10.0-1_amd64.deb ... 2024-09-18T04:35:01.271 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2024-09-18T04:35:01.433 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package librgw2. 2024-09-18T04:35:01.453 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../019-librgw2_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:01.491 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking librgw2 (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:01.583 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-18T04:35:01.602 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../018-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2024-09-18T04:35:01.625 INFO:teuthology.orchestra.run.smithi028.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 18.2.4-855-g19e2a2be-1focal [108 kB] 2024-09-18T04:35:01.634 INFO:teuthology.orchestra.run.smithi028.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 rbd-nbd amd64 18.2.4-855-g19e2a2be-1focal [189 kB] 2024-09-18T04:35:01.641 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-18T04:35:02.015 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package librgw2. 2024-09-18T04:35:02.035 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../019-librgw2_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:02.166 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking librgw2 (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:02.320 INFO:teuthology.orchestra.run.smithi028.stdout:Fetched 199 MB in 15s (13.2 MB/s) 2024-09-18T04:35:02.464 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2024-09-18T04:35:02.520 INFO:teuthology.orchestra.run.smithi028.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141720 files and directories currently installed.) 2024-09-18T04:35:02.525 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2024-09-18T04:35:02.569 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-rgw. 2024-09-18T04:35:02.589 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../020-python3-rgw_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:02.604 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-18T04:35:02.627 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-rgw (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:02.941 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-18T04:35:02.947 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package liboath0. 2024-09-18T04:35:02.960 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2024-09-18T04:35:02.964 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2024-09-18T04:35:03.005 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2024-09-18T04:35:03.023 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-18T04:35:03.046 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-rgw. 2024-09-18T04:35:03.065 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../020-python3-rgw_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:03.105 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-rgw (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:03.308 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package libradosstriper1. 2024-09-18T04:35:03.326 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../022-libradosstriper1_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:03.360 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-18T04:35:03.366 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking libradosstriper1 (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:03.377 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../002-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2024-09-18T04:35:03.418 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-18T04:35:03.449 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package liboath0. 2024-09-18T04:35:03.468 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2024-09-18T04:35:03.507 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2024-09-18T04:35:03.770 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package ceph-common. 2024-09-18T04:35:03.786 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../023-ceph-common_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:03.810 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libradosstriper1. 2024-09-18T04:35:03.828 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking ceph-common (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:03.828 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../022-libradosstriper1_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:03.855 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-18T04:35:03.873 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../003-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-18T04:35:03.876 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libradosstriper1 (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:03.918 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T04:35:04.324 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-common. 2024-09-18T04:35:04.341 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../023-ceph-common_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:04.382 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-common (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:04.441 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-18T04:35:04.457 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../004-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-18T04:35:04.499 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T04:35:04.794 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-18T04:35:04.812 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../005-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-18T04:35:04.844 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T04:35:05.230 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libthrift-0.13.0. 2024-09-18T04:35:05.246 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../006-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2024-09-18T04:35:05.288 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-18T04:35:05.658 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../007-librbd1_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:05.807 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking librbd1 (18.2.4-855-g19e2a2be-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-18T04:35:06.358 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package ceph-base. 2024-09-18T04:35:06.375 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../024-ceph-base_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:06.427 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking ceph-base (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:06.437 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../008-librados2_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:06.562 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking librados2 (18.2.4-855-g19e2a2be-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-18T04:35:06.837 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-base. 2024-09-18T04:35:06.853 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../024-ceph-base_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:06.906 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-base (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:07.266 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libcephfs2. 2024-09-18T04:35:07.282 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../009-libcephfs2_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:07.324 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libcephfs2 (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:07.671 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-18T04:35:07.685 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-rados. 2024-09-18T04:35:07.687 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../025-python3-cherrypy3_8.9.1-6_all.deb ... 2024-09-18T04:35:07.702 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../010-python3-rados_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:07.741 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2024-09-18T04:35:08.447 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-18T04:35:08.447 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../025-python3-cherrypy3_8.9.1-6_all.deb ... 2024-09-18T04:35:08.447 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2024-09-18T04:35:08.447 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-natsort. 2024-09-18T04:35:08.448 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../026-python3-natsort_7.0.1-1_all.deb ... 2024-09-18T04:35:08.448 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-natsort (7.0.1-1) ... 2024-09-18T04:35:08.448 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-rados (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:08.448 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-18T04:35:08.448 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../011-python3-ceph-argparse_18.2.4-855-g19e2a2be-1focal_all.deb ... 2024-09-18T04:35:08.448 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-ceph-argparse (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:08.448 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-cephfs. 2024-09-18T04:35:08.448 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../012-python3-cephfs_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:08.452 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-logutils. 2024-09-18T04:35:08.458 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-cephfs (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:08.473 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../027-python3-logutils_0.3.3-7_all.deb ... 2024-09-18T04:35:08.511 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-logutils (0.3.3-7) ... 2024-09-18T04:35:08.578 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-natsort. 2024-09-18T04:35:08.596 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../026-python3-natsort_7.0.1-1_all.deb ... 2024-09-18T04:35:08.636 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-natsort (7.0.1-1) ... 2024-09-18T04:35:08.769 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-18T04:35:08.788 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../013-python3-ceph-common_18.2.4-855-g19e2a2be-1focal_all.deb ... 2024-09-18T04:35:08.813 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-18T04:35:08.819 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-ceph-common (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:08.830 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../028-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2024-09-18T04:35:08.872 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2024-09-18T04:35:08.948 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-logutils. 2024-09-18T04:35:08.965 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../027-python3-logutils_0.3.3-7_all.deb ... 2024-09-18T04:35:09.006 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-logutils (0.3.3-7) ... 2024-09-18T04:35:09.089 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-prettytable. 2024-09-18T04:35:09.105 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../014-python3-prettytable_0.7.2-5_all.deb ... 2024-09-18T04:35:09.147 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2024-09-18T04:35:09.150 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-mako. 2024-09-18T04:35:09.169 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../029-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2024-09-18T04:35:09.208 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-18T04:35:09.309 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-18T04:35:09.325 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../028-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2024-09-18T04:35:09.358 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2024-09-18T04:35:09.399 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-rbd. 2024-09-18T04:35:09.419 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../015-python3-rbd_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:09.457 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-rbd (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:09.544 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-18T04:35:09.562 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../030-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-18T04:35:09.602 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-18T04:35:09.620 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-mako. 2024-09-18T04:35:09.639 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../029-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2024-09-18T04:35:09.678 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-18T04:35:09.819 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2024-09-18T04:35:09.835 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../016-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2024-09-18T04:35:09.877 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-18T04:35:09.905 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-18T04:35:09.925 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../031-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2024-09-18T04:35:09.964 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2024-09-18T04:35:10.039 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-18T04:35:10.057 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../030-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-18T04:35:10.097 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-18T04:35:10.179 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package librabbitmq4:amd64. 2024-09-18T04:35:10.199 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../017-librabbitmq4_0.10.0-1_amd64.deb ... 2024-09-18T04:35:10.229 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2024-09-18T04:35:10.250 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-webob. 2024-09-18T04:35:10.266 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../032-python3-webob_1%3a1.8.5-2ubuntu0.1_all.deb ... 2024-09-18T04:35:10.308 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-18T04:35:10.384 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-18T04:35:10.403 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../031-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2024-09-18T04:35:10.442 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2024-09-18T04:35:10.582 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-18T04:35:10.598 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../018-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2024-09-18T04:35:10.640 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-18T04:35:10.653 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-soupsieve. 2024-09-18T04:35:10.669 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../033-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2024-09-18T04:35:10.694 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-webob. 2024-09-18T04:35:10.710 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-18T04:35:10.713 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../032-python3-webob_1%3a1.8.5-2ubuntu0.1_all.deb ... 2024-09-18T04:35:10.744 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-18T04:35:11.005 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-bs4. 2024-09-18T04:35:11.023 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../034-python3-bs4_4.8.2-1_all.deb ... 2024-09-18T04:35:11.030 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-soupsieve. 2024-09-18T04:35:11.035 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package librgw2. 2024-09-18T04:35:11.051 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../033-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2024-09-18T04:35:11.053 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../019-librgw2_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:11.063 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2024-09-18T04:35:11.080 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-18T04:35:11.093 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking librgw2 (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:11.375 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-bs4. 2024-09-18T04:35:11.391 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../034-python3-bs4_4.8.2-1_all.deb ... 2024-09-18T04:35:11.417 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-waitress. 2024-09-18T04:35:11.433 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2024-09-18T04:35:11.434 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../035-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2024-09-18T04:35:11.503 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-18T04:35:11.786 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-waitress. 2024-09-18T04:35:11.804 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../035-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2024-09-18T04:35:11.828 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-tempita. 2024-09-18T04:35:11.846 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../036-python3-tempita_0.5.2-6_all.deb ... 2024-09-18T04:35:11.847 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-18T04:35:11.886 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-tempita (0.5.2-6) ... 2024-09-18T04:35:11.962 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-rgw. 2024-09-18T04:35:11.979 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../020-python3-rgw_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:12.020 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-rgw (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:12.156 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-tempita. 2024-09-18T04:35:12.175 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../036-python3-tempita_0.5.2-6_all.deb ... 2024-09-18T04:35:12.189 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-paste. 2024-09-18T04:35:12.209 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../037-python3-paste_3.4.0+dfsg1-1_all.deb ... 2024-09-18T04:35:12.214 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-tempita (0.5.2-6) ... 2024-09-18T04:35:12.247 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2024-09-18T04:35:12.423 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package liboath0. 2024-09-18T04:35:12.443 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2024-09-18T04:35:12.481 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2024-09-18T04:35:12.534 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-paste. 2024-09-18T04:35:12.550 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../037-python3-paste_3.4.0+dfsg1-1_all.deb ... 2024-09-18T04:35:12.617 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2024-09-18T04:35:12.618 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-18T04:35:12.637 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../038-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2024-09-18T04:35:12.676 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2024-09-18T04:35:12.784 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libradosstriper1. 2024-09-18T04:35:12.804 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../022-libradosstriper1_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:12.842 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libradosstriper1 (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:12.921 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-18T04:35:12.940 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../038-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2024-09-18T04:35:12.979 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2024-09-18T04:35:12.979 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-18T04:35:12.995 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../039-python3-pastedeploy_2.1.0-1_all.deb ... 2024-09-18T04:35:13.037 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2024-09-18T04:35:13.281 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-18T04:35:13.297 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../039-python3-pastedeploy_2.1.0-1_all.deb ... 2024-09-18T04:35:13.340 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2024-09-18T04:35:13.340 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-webtest. 2024-09-18T04:35:13.356 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../040-python3-webtest_2.0.32-2_all.deb ... 2024-09-18T04:35:13.398 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-webtest (2.0.32-2) ... 2024-09-18T04:35:13.453 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package ceph-common. 2024-09-18T04:35:13.470 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../023-ceph-common_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:13.511 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking ceph-common (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:13.668 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-webtest. 2024-09-18T04:35:13.685 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../040-python3-webtest_2.0.32-2_all.deb ... 2024-09-18T04:35:13.726 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-webtest (2.0.32-2) ... 2024-09-18T04:35:13.732 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-pecan. 2024-09-18T04:35:13.753 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../041-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2024-09-18T04:35:13.791 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2024-09-18T04:35:14.029 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-pecan. 2024-09-18T04:35:14.048 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../041-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2024-09-18T04:35:14.087 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2024-09-18T04:35:14.105 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-18T04:35:14.121 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../042-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.2_all.deb ... 2024-09-18T04:35:14.154 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-18T04:35:14.449 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-18T04:35:14.457 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-18T04:35:14.466 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../043-ceph-mgr-modules-core_18.2.4-855-g19e2a2be-1focal_all.deb ... 2024-09-18T04:35:14.476 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../042-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.2_all.deb ... 2024-09-18T04:35:14.507 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking ceph-mgr-modules-core (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:14.515 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-18T04:35:14.810 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-18T04:35:14.825 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../043-ceph-mgr-modules-core_18.2.4-855-g19e2a2be-1focal_all.deb ... 2024-09-18T04:35:14.868 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-mgr-modules-core (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:14.904 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-18T04:35:14.920 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../044-libsqlite3-mod-ceph_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:14.962 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking libsqlite3-mod-ceph (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:15.248 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-18T04:35:15.267 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../044-libsqlite3-mod-ceph_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:15.273 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-18T04:35:15.289 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../045-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2024-09-18T04:35:15.306 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libsqlite3-mod-ceph (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:15.331 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-18T04:35:15.602 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-cryptography. 2024-09-18T04:35:15.618 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../046-python3-cryptography_2.8-3ubuntu0.3_amd64.deb ... 2024-09-18T04:35:15.634 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-18T04:35:15.650 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../045-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2024-09-18T04:35:15.660 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-18T04:35:15.683 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-18T04:35:15.995 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-cryptography. 2024-09-18T04:35:16.011 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../046-python3-cryptography_2.8-3ubuntu0.3_amd64.deb ... 2024-09-18T04:35:16.012 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package ceph-base. 2024-09-18T04:35:16.030 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-openssl. 2024-09-18T04:35:16.032 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../024-ceph-base_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:16.049 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../047-python3-openssl_19.0.0-1build1_all.deb ... 2024-09-18T04:35:16.070 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-18T04:35:16.085 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking ceph-base (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:16.088 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2024-09-18T04:35:16.416 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-openssl. 2024-09-18T04:35:16.429 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package ceph-mgr. 2024-09-18T04:35:16.435 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../047-python3-openssl_19.0.0-1build1_all.deb ... 2024-09-18T04:35:16.445 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../048-ceph-mgr_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:16.474 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2024-09-18T04:35:16.487 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking ceph-mgr (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:16.818 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-mgr. 2024-09-18T04:35:16.837 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../048-ceph-mgr_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:16.901 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-mgr (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:16.974 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package ceph-mon. 2024-09-18T04:35:16.993 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../049-ceph-mon_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:17.031 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking ceph-mon (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:17.392 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-mon. 2024-09-18T04:35:17.409 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../049-ceph-mon_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:17.450 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-mon (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:18.063 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-18T04:35:18.083 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../025-python3-cherrypy3_8.9.1-6_all.deb ... 2024-09-18T04:35:18.121 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2024-09-18T04:35:18.458 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-natsort. 2024-09-18T04:35:18.474 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../026-python3-natsort_7.0.1-1_all.deb ... 2024-09-18T04:35:18.516 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-natsort (7.0.1-1) ... 2024-09-18T04:35:18.777 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package ceph-osd. 2024-09-18T04:35:18.796 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../050-ceph-osd_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:18.827 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-logutils. 2024-09-18T04:35:18.834 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking ceph-osd (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:18.845 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../027-python3-logutils_0.3.3-7_all.deb ... 2024-09-18T04:35:18.885 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-logutils (0.3.3-7) ... 2024-09-18T04:35:18.962 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-osd. 2024-09-18T04:35:18.978 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../050-ceph-osd_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:19.020 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-osd (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:19.180 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-18T04:35:19.197 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../028-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2024-09-18T04:35:19.229 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2024-09-18T04:35:19.524 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-mako. 2024-09-18T04:35:19.544 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../029-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2024-09-18T04:35:19.582 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-18T04:35:19.952 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-18T04:35:19.969 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../030-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-18T04:35:20.010 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-18T04:35:20.296 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-18T04:35:20.316 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../031-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2024-09-18T04:35:20.354 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2024-09-18T04:35:20.657 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-webob. 2024-09-18T04:35:20.673 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../032-python3-webob_1%3a1.8.5-2ubuntu0.1_all.deb ... 2024-09-18T04:35:20.715 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-18T04:35:20.979 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package ceph. 2024-09-18T04:35:20.998 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../051-ceph_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:21.037 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking ceph (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:21.060 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-soupsieve. 2024-09-18T04:35:21.076 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../033-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2024-09-18T04:35:21.117 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-18T04:35:21.206 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph. 2024-09-18T04:35:21.224 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../051-ceph_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:21.264 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:21.365 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package ceph-fuse. 2024-09-18T04:35:21.381 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../052-ceph-fuse_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:21.412 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-bs4. 2024-09-18T04:35:21.431 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../034-python3-bs4_4.8.2-1_all.deb ... 2024-09-18T04:35:21.448 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking ceph-fuse (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:21.461 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2024-09-18T04:35:21.625 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-fuse. 2024-09-18T04:35:21.642 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../052-ceph-fuse_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:21.683 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-fuse (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:21.831 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-waitress. 2024-09-18T04:35:21.843 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package ceph-mds. 2024-09-18T04:35:21.851 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../035-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2024-09-18T04:35:21.860 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../053-ceph-mds_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:21.892 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-18T04:35:21.902 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking ceph-mds (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:22.111 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-mds. 2024-09-18T04:35:22.129 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../053-ceph-mds_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:22.169 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-mds (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:22.184 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-tempita. 2024-09-18T04:35:22.203 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../036-python3-tempita_0.5.2-6_all.deb ... 2024-09-18T04:35:22.242 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-tempita (0.5.2-6) ... 2024-09-18T04:35:22.422 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package cephadm. 2024-09-18T04:35:22.439 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../054-cephadm_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:22.505 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking cephadm (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:22.528 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-paste. 2024-09-18T04:35:22.548 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../037-python3-paste_3.4.0+dfsg1-1_all.deb ... 2024-09-18T04:35:22.586 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2024-09-18T04:35:22.698 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package cephadm. 2024-09-18T04:35:22.716 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../054-cephadm_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:22.756 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking cephadm (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:22.824 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-18T04:35:22.843 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../055-python3-asyncssh_1.12.2-1_all.deb ... 2024-09-18T04:35:22.882 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-asyncssh (1.12.2-1) ... 2024-09-18T04:35:22.948 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-18T04:35:22.968 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../038-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2024-09-18T04:35:23.006 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2024-09-18T04:35:23.117 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-18T04:35:23.136 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../055-python3-asyncssh_1.12.2-1_all.deb ... 2024-09-18T04:35:23.175 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-asyncssh (1.12.2-1) ... 2024-09-18T04:35:23.236 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-jinja2. 2024-09-18T04:35:23.251 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../056-python3-jinja2_2.10.1-2ubuntu0.3_all.deb ... 2024-09-18T04:35:23.292 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-18T04:35:23.294 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-18T04:35:23.311 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../039-python3-pastedeploy_2.1.0-1_all.deb ... 2024-09-18T04:35:23.350 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2024-09-18T04:35:23.554 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-jinja2. 2024-09-18T04:35:23.570 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../056-python3-jinja2_2.10.1-2ubuntu0.3_all.deb ... 2024-09-18T04:35:23.611 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-18T04:35:23.680 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-18T04:35:23.698 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../057-ceph-mgr-cephadm_18.2.4-855-g19e2a2be-1focal_all.deb ... 2024-09-18T04:35:23.738 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking ceph-mgr-cephadm (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:23.795 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-webtest. 2024-09-18T04:35:23.811 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../040-python3-webtest_2.0.32-2_all.deb ... 2024-09-18T04:35:23.853 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-webtest (2.0.32-2) ... 2024-09-18T04:35:23.990 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-18T04:35:24.006 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../057-ceph-mgr-cephadm_18.2.4-855-g19e2a2be-1focal_all.deb ... 2024-09-18T04:35:24.048 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-mgr-cephadm (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:24.101 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-jwt. 2024-09-18T04:35:24.116 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../058-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2024-09-18T04:35:24.143 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-pecan. 2024-09-18T04:35:24.159 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-18T04:35:24.163 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../041-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2024-09-18T04:35:24.193 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2024-09-18T04:35:24.402 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-jwt. 2024-09-18T04:35:24.418 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../058-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2024-09-18T04:35:24.453 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-attr. 2024-09-18T04:35:24.459 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-18T04:35:24.469 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../059-python3-attr_19.3.0-2_all.deb ... 2024-09-18T04:35:24.506 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-18T04:35:24.511 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-attr (19.3.0-2) ... 2024-09-18T04:35:24.524 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../042-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.2_all.deb ... 2024-09-18T04:35:24.586 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-18T04:35:24.771 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-attr. 2024-09-18T04:35:24.787 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../059-python3-attr_19.3.0-2_all.deb ... 2024-09-18T04:35:24.858 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-attr (19.3.0-2) ... 2024-09-18T04:35:24.872 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-pluggy. 2024-09-18T04:35:24.890 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../060-python3-pluggy_0.13.0-2_all.deb ... 2024-09-18T04:35:24.909 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-18T04:35:24.925 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../043-ceph-mgr-modules-core_18.2.4-855-g19e2a2be-1focal_all.deb ... 2024-09-18T04:35:24.930 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2024-09-18T04:35:24.967 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking ceph-mgr-modules-core (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:25.182 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-pluggy. 2024-09-18T04:35:25.201 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../060-python3-pluggy_0.13.0-2_all.deb ... 2024-09-18T04:35:25.248 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2024-09-18T04:35:25.250 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-py. 2024-09-18T04:35:25.267 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../061-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2024-09-18T04:35:25.309 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2024-09-18T04:35:25.355 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-18T04:35:25.374 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../044-libsqlite3-mod-ceph_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:25.413 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libsqlite3-mod-ceph (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:25.551 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-py. 2024-09-18T04:35:25.569 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../061-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2024-09-18T04:35:25.609 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2024-09-18T04:35:25.628 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-atomicwrites. 2024-09-18T04:35:25.646 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../062-python3-atomicwrites_1.1.5-2build1_all.deb ... 2024-09-18T04:35:25.712 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2024-09-18T04:35:25.757 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-18T04:35:25.774 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../045-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2024-09-18T04:35:25.815 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-18T04:35:25.963 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-atomicwrites. 2024-09-18T04:35:25.979 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../062-python3-atomicwrites_1.1.5-2build1_all.deb ... 2024-09-18T04:35:25.989 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-packaging. 2024-09-18T04:35:26.007 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../063-python3-packaging_20.3-1_all.deb ... 2024-09-18T04:35:26.021 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2024-09-18T04:35:26.039 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-packaging (20.3-1) ... 2024-09-18T04:35:26.085 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-cryptography. 2024-09-18T04:35:26.104 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../046-python3-cryptography_2.8-3ubuntu0.3_amd64.deb ... 2024-09-18T04:35:26.143 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-18T04:35:26.324 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-packaging. 2024-09-18T04:35:26.340 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../063-python3-packaging_20.3-1_all.deb ... 2024-09-18T04:35:26.342 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-18T04:35:26.359 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../064-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2024-09-18T04:35:26.382 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-packaging (20.3-1) ... 2024-09-18T04:35:26.408 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-18T04:35:26.522 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-openssl. 2024-09-18T04:35:26.540 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../047-python3-openssl_19.0.0-1build1_all.deb ... 2024-09-18T04:35:26.582 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2024-09-18T04:35:26.693 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-18T04:35:26.711 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-pytest. 2024-09-18T04:35:26.712 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../064-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2024-09-18T04:35:26.727 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../065-python3-pytest_4.6.9-1_all.deb ... 2024-09-18T04:35:26.751 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-18T04:35:26.794 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-pytest (4.6.9-1) ... 2024-09-18T04:35:26.920 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package ceph-mgr. 2024-09-18T04:35:26.937 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../048-ceph-mgr_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:26.978 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking ceph-mgr (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:27.054 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-pytest. 2024-09-18T04:35:27.070 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../065-python3-pytest_4.6.9-1_all.deb ... 2024-09-18T04:35:27.112 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-pytest (4.6.9-1) ... 2024-09-18T04:35:27.132 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-18T04:35:27.150 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../066-python3-repoze.lru_0.7-2_all.deb ... 2024-09-18T04:35:27.192 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-18T04:35:27.444 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package libjs-underscore. 2024-09-18T04:35:27.459 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../067-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2024-09-18T04:35:27.474 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-18T04:35:27.491 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../066-python3-repoze.lru_0.7-2_all.deb ... 2024-09-18T04:35:27.498 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package ceph-mon. 2024-09-18T04:35:27.501 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-18T04:35:27.514 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../049-ceph-mon_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:27.532 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-18T04:35:27.547 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking ceph-mon (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:27.771 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package libjs-sphinxdoc. 2024-09-18T04:35:27.786 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libjs-underscore. 2024-09-18T04:35:27.787 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../068-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2024-09-18T04:35:27.804 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../067-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2024-09-18T04:35:27.829 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-18T04:35:27.844 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-18T04:35:28.113 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libjs-sphinxdoc. 2024-09-18T04:35:28.129 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../068-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2024-09-18T04:35:28.140 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-routes. 2024-09-18T04:35:28.156 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../069-python3-routes_2.4.1-2_all.deb ... 2024-09-18T04:35:28.171 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-18T04:35:28.198 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-routes (2.4.1-2) ... 2024-09-18T04:35:28.510 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-18T04:35:28.524 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-routes. 2024-09-18T04:35:28.529 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../070-ceph-mgr-dashboard_18.2.4-855-g19e2a2be-1focal_all.deb ... 2024-09-18T04:35:28.541 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../069-python3-routes_2.4.1-2_all.deb ... 2024-09-18T04:35:28.568 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking ceph-mgr-dashboard (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:28.582 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-routes (2.4.1-2) ... 2024-09-18T04:35:28.616 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package ceph-osd. 2024-09-18T04:35:28.636 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../050-ceph-osd_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:28.674 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking ceph-osd (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:28.902 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-18T04:35:28.920 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../070-ceph-mgr-dashboard_18.2.4-855-g19e2a2be-1focal_all.deb ... 2024-09-18T04:35:28.960 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-mgr-dashboard (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:29.856 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-decorator. 2024-09-18T04:35:29.872 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../071-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2024-09-18T04:35:29.906 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2024-09-18T04:35:30.226 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2024-09-18T04:35:30.245 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../072-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2024-09-18T04:35:30.284 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-18T04:35:30.465 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-decorator. 2024-09-18T04:35:30.482 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../071-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2024-09-18T04:35:30.523 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2024-09-18T04:35:30.586 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-scipy. 2024-09-18T04:35:30.605 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../073-python3-scipy_1.3.3-3build1_amd64.deb ... 2024-09-18T04:35:30.644 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2024-09-18T04:35:30.843 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2024-09-18T04:35:30.859 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../072-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2024-09-18T04:35:30.901 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-18T04:35:30.952 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package ceph. 2024-09-18T04:35:30.969 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../051-ceph_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:31.010 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking ceph (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:31.245 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-scipy. 2024-09-18T04:35:31.262 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../073-python3-scipy_1.3.3-3build1_amd64.deb ... 2024-09-18T04:35:31.303 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2024-09-18T04:35:31.371 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package ceph-fuse. 2024-09-18T04:35:31.388 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../052-ceph-fuse_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:31.429 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking ceph-fuse (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:31.800 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package ceph-mds. 2024-09-18T04:35:31.816 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../053-ceph-mds_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:31.850 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking ceph-mds (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:31.987 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-sklearn-lib. 2024-09-18T04:35:32.006 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../074-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2024-09-18T04:35:32.045 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-18T04:35:32.336 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package cephadm. 2024-09-18T04:35:32.353 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../054-cephadm_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:32.394 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking cephadm (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:32.596 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-sklearn-lib. 2024-09-18T04:35:32.612 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../074-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2024-09-18T04:35:32.624 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-joblib. 2024-09-18T04:35:32.640 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../075-python3-joblib_0.14.0-3_all.deb ... 2024-09-18T04:35:32.654 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-18T04:35:32.682 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-joblib (0.14.0-3) ... 2024-09-18T04:35:32.713 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-18T04:35:32.729 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../055-python3-asyncssh_1.12.2-1_all.deb ... 2024-09-18T04:35:32.771 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-asyncssh (1.12.2-1) ... 2024-09-18T04:35:33.036 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-sklearn. 2024-09-18T04:35:33.053 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../076-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2024-09-18T04:35:33.093 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-18T04:35:33.158 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-jinja2. 2024-09-18T04:35:33.177 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../056-python3-jinja2_2.10.1-2ubuntu0.3_all.deb ... 2024-09-18T04:35:33.182 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-joblib. 2024-09-18T04:35:33.199 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../075-python3-joblib_0.14.0-3_all.deb ... 2024-09-18T04:35:33.216 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-18T04:35:33.241 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-joblib (0.14.0-3) ... 2024-09-18T04:35:33.628 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-sklearn. 2024-09-18T04:35:33.643 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../076-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2024-09-18T04:35:33.685 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-18T04:35:33.965 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-18T04:35:33.978 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-18T04:35:33.983 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../057-ceph-mgr-cephadm_18.2.4-855-g19e2a2be-1focal_all.deb ... 2024-09-18T04:35:33.992 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../077-ceph-mgr-diskprediction-local_18.2.4-855-g19e2a2be-1focal_all.deb ... 2024-09-18T04:35:34.026 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking ceph-mgr-cephadm (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:34.212 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking ceph-mgr-diskprediction-local (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:34.347 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-jwt. 2024-09-18T04:35:34.367 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../058-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2024-09-18T04:35:34.405 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-18T04:35:34.725 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-attr. 2024-09-18T04:35:34.741 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../059-python3-attr_19.3.0-2_all.deb ... 2024-09-18T04:35:34.770 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-18T04:35:34.783 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-attr (19.3.0-2) ... 2024-09-18T04:35:34.786 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../077-ceph-mgr-diskprediction-local_18.2.4-855-g19e2a2be-1focal_all.deb ... 2024-09-18T04:35:34.828 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-mgr-diskprediction-local (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:35.111 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-pluggy. 2024-09-18T04:35:35.129 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../060-python3-pluggy_0.13.0-2_all.deb ... 2024-09-18T04:35:35.168 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2024-09-18T04:35:35.290 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-cachetools. 2024-09-18T04:35:35.309 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../078-python3-cachetools_4.0.0-1_all.deb ... 2024-09-18T04:35:35.347 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2024-09-18T04:35:35.479 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-py. 2024-09-18T04:35:35.497 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../061-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2024-09-18T04:35:35.537 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2024-09-18T04:35:35.676 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-18T04:35:35.693 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../079-python3-pyasn1_0.4.2-3build1_all.deb ... 2024-09-18T04:35:35.726 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2024-09-18T04:35:35.891 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-atomicwrites. 2024-09-18T04:35:35.909 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../062-python3-atomicwrites_1.1.5-2build1_all.deb ... 2024-09-18T04:35:35.923 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-cachetools. 2024-09-18T04:35:35.939 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../078-python3-cachetools_4.0.0-1_all.deb ... 2024-09-18T04:35:35.949 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2024-09-18T04:35:35.981 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2024-09-18T04:35:36.038 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-18T04:35:36.054 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../080-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2024-09-18T04:35:36.096 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-18T04:35:36.235 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-packaging. 2024-09-18T04:35:36.251 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../063-python3-packaging_20.3-1_all.deb ... 2024-09-18T04:35:36.293 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-packaging (20.3-1) ... 2024-09-18T04:35:36.308 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-18T04:35:36.325 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../079-python3-pyasn1_0.4.2-3build1_all.deb ... 2024-09-18T04:35:36.366 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2024-09-18T04:35:36.501 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-rsa. 2024-09-18T04:35:36.519 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../081-python3-rsa_4.0-3_all.deb ... 2024-09-18T04:35:36.561 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-rsa (4.0-3) ... 2024-09-18T04:35:36.612 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-18T04:35:36.629 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../064-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2024-09-18T04:35:36.670 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-18T04:35:36.695 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-18T04:35:36.711 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../080-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2024-09-18T04:35:36.753 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-18T04:35:36.869 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-google-auth. 2024-09-18T04:35:36.885 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../082-python3-google-auth_1.5.1-2_all.deb ... 2024-09-18T04:35:36.927 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2024-09-18T04:35:36.999 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-pytest. 2024-09-18T04:35:37.016 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../065-python3-pytest_4.6.9-1_all.deb ... 2024-09-18T04:35:37.057 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-pytest (4.6.9-1) ... 2024-09-18T04:35:37.168 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-rsa. 2024-09-18T04:35:37.185 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../081-python3-rsa_4.0-3_all.deb ... 2024-09-18T04:35:37.225 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-rsa (4.0-3) ... 2024-09-18T04:35:37.248 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-blinker. 2024-09-18T04:35:37.264 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../083-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2024-09-18T04:35:37.306 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-18T04:35:37.402 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-18T04:35:37.421 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../066-python3-repoze.lru_0.7-2_all.deb ... 2024-09-18T04:35:37.461 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-18T04:35:37.538 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-google-auth. 2024-09-18T04:35:37.554 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../082-python3-google-auth_1.5.1-2_all.deb ... 2024-09-18T04:35:37.596 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2024-09-18T04:35:37.609 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-oauthlib. 2024-09-18T04:35:37.627 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../084-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2024-09-18T04:35:37.658 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-18T04:35:37.738 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libjs-underscore. 2024-09-18T04:35:37.754 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../067-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2024-09-18T04:35:37.796 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-18T04:35:37.912 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-blinker. 2024-09-18T04:35:37.929 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../083-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2024-09-18T04:35:37.946 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-18T04:35:37.965 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../085-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2024-09-18T04:35:37.970 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-18T04:35:37.996 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-18T04:35:38.040 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libjs-sphinxdoc. 2024-09-18T04:35:38.060 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../068-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2024-09-18T04:35:38.098 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-18T04:35:38.284 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-oauthlib. 2024-09-18T04:35:38.301 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../084-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2024-09-18T04:35:38.308 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-websocket. 2024-09-18T04:35:38.324 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../086-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2024-09-18T04:35:38.350 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-18T04:35:38.366 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2024-09-18T04:35:38.426 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-routes. 2024-09-18T04:35:38.442 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../069-python3-routes_2.4.1-2_all.deb ... 2024-09-18T04:35:38.484 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-routes (2.4.1-2) ... 2024-09-18T04:35:38.678 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-18T04:35:38.696 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../085-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2024-09-18T04:35:38.703 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-18T04:35:38.719 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../087-python3-kubernetes_7.0.0~a1-2_all.deb ... 2024-09-18T04:35:38.736 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-18T04:35:38.761 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2024-09-18T04:35:38.845 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-18T04:35:38.862 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../070-ceph-mgr-dashboard_18.2.4-855-g19e2a2be-1focal_all.deb ... 2024-09-18T04:35:38.903 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking ceph-mgr-dashboard (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:39.048 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-websocket. 2024-09-18T04:35:39.068 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../086-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2024-09-18T04:35:39.106 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2024-09-18T04:35:39.296 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-18T04:35:39.313 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../088-ceph-mgr-k8sevents_18.2.4-855-g19e2a2be-1focal_all.deb ... 2024-09-18T04:35:39.353 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking ceph-mgr-k8sevents (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:39.418 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-18T04:35:39.434 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../087-python3-kubernetes_7.0.0~a1-2_all.deb ... 2024-09-18T04:35:39.476 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2024-09-18T04:35:39.666 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-18T04:35:39.683 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../089-libonig5_6.9.4-1_amd64.deb ... 2024-09-18T04:35:39.750 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2024-09-18T04:35:40.059 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-18T04:35:40.080 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../088-ceph-mgr-k8sevents_18.2.4-855-g19e2a2be-1focal_all.deb ... 2024-09-18T04:35:40.112 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-18T04:35:40.117 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-mgr-k8sevents (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:40.131 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../090-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-18T04:35:40.170 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-18T04:35:40.242 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-decorator. 2024-09-18T04:35:40.258 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../071-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2024-09-18T04:35:40.300 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2024-09-18T04:35:40.440 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package jq. 2024-09-18T04:35:40.445 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-18T04:35:40.460 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../091-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-18T04:35:40.462 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../089-libonig5_6.9.4-1_amd64.deb ... 2024-09-18T04:35:40.498 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2024-09-18T04:35:40.503 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2024-09-18T04:35:40.636 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2024-09-18T04:35:40.653 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../072-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2024-09-18T04:35:40.694 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-18T04:35:40.776 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package socat. 2024-09-18T04:35:40.794 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../092-socat_1.7.3.3-2_amd64.deb ... 2024-09-18T04:35:40.833 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking socat (1.7.3.3-2) ... 2024-09-18T04:35:40.873 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-18T04:35:40.893 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../090-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-18T04:35:40.931 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-18T04:35:40.997 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-scipy. 2024-09-18T04:35:41.013 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../073-python3-scipy_1.3.3-3build1_amd64.deb ... 2024-09-18T04:35:41.054 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2024-09-18T04:35:41.154 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package xmlstarlet. 2024-09-18T04:35:41.171 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../093-xmlstarlet_1.6.1-2build1_amd64.deb ... 2024-09-18T04:35:41.212 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2024-09-18T04:35:41.234 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package jq. 2024-09-18T04:35:41.250 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../091-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-18T04:35:41.292 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2024-09-18T04:35:41.550 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package ceph-test. 2024-09-18T04:35:41.555 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package socat. 2024-09-18T04:35:41.565 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../094-ceph-test_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:41.573 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../092-socat_1.7.3.3-2_amd64.deb ... 2024-09-18T04:35:41.604 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking socat (1.7.3.3-2) ... 2024-09-18T04:35:41.607 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking ceph-test (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:41.925 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package xmlstarlet. 2024-09-18T04:35:41.944 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../093-xmlstarlet_1.6.1-2build1_amd64.deb ... 2024-09-18T04:35:41.982 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2024-09-18T04:35:42.320 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-test. 2024-09-18T04:35:42.340 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../094-ceph-test_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:42.363 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-sklearn-lib. 2024-09-18T04:35:42.379 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../074-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2024-09-18T04:35:42.401 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-test (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:42.413 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-18T04:35:42.941 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-joblib. 2024-09-18T04:35:42.960 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../075-python3-joblib_0.14.0-3_all.deb ... 2024-09-18T04:35:42.999 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-joblib (0.14.0-3) ... 2024-09-18T04:35:43.378 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-sklearn. 2024-09-18T04:35:43.396 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../076-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2024-09-18T04:35:43.435 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-18T04:35:44.078 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-18T04:35:44.094 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../077-ceph-mgr-diskprediction-local_18.2.4-855-g19e2a2be-1focal_all.deb ... 2024-09-18T04:35:44.161 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking ceph-mgr-diskprediction-local (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:45.230 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-cachetools. 2024-09-18T04:35:45.249 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../078-python3-cachetools_4.0.0-1_all.deb ... 2024-09-18T04:35:45.280 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2024-09-18T04:35:45.583 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-18T04:35:45.601 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../079-python3-pyasn1_0.4.2-3build1_all.deb ... 2024-09-18T04:35:45.650 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2024-09-18T04:35:45.795 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package ceph-volume. 2024-09-18T04:35:45.811 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../095-ceph-volume_18.2.4-855-g19e2a2be-1focal_all.deb ... 2024-09-18T04:35:45.845 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking ceph-volume (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:46.002 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-18T04:35:46.021 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../080-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2024-09-18T04:35:46.061 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-18T04:35:46.149 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package fonts-mathjax. 2024-09-18T04:35:46.168 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../096-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-18T04:35:46.207 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-18T04:35:46.463 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-rsa. 2024-09-18T04:35:46.481 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../081-python3-rsa_4.0-3_all.deb ... 2024-09-18T04:35:46.521 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-rsa (4.0-3) ... 2024-09-18T04:35:46.532 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-volume. 2024-09-18T04:35:46.552 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../095-ceph-volume_18.2.4-855-g19e2a2be-1focal_all.deb ... 2024-09-18T04:35:46.590 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-volume (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:46.694 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-18T04:35:46.711 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../097-libcephfs-dev_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:46.760 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking libcephfs-dev (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:46.833 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-google-auth. 2024-09-18T04:35:46.850 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../082-python3-google-auth_1.5.1-2_all.deb ... 2024-09-18T04:35:46.891 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2024-09-18T04:35:46.907 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package fonts-mathjax. 2024-09-18T04:35:46.928 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../096-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-18T04:35:46.968 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-18T04:35:47.197 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package nvme-cli. 2024-09-18T04:35:47.214 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../098-nvme-cli_1.9-1ubuntu0.2_amd64.deb ... 2024-09-18T04:35:47.254 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-blinker. 2024-09-18T04:35:47.271 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../083-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2024-09-18T04:35:47.273 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking nvme-cli (1.9-1ubuntu0.2) ... 2024-09-18T04:35:47.311 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-18T04:35:47.463 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-18T04:35:47.481 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../097-libcephfs-dev_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:47.521 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libcephfs-dev (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:47.580 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-18T04:35:47.589 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-oauthlib. 2024-09-18T04:35:47.599 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../099-python-asyncssh-doc_1.12.2-1_all.deb ... 2024-09-18T04:35:47.606 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../084-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2024-09-18T04:35:47.638 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python-asyncssh-doc (1.12.2-1) ... 2024-09-18T04:35:47.639 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-18T04:35:47.958 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package nvme-cli. 2024-09-18T04:35:47.966 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package libjs-mathjax. 2024-09-18T04:35:47.974 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../098-nvme-cli_1.9-1ubuntu0.2_amd64.deb ... 2024-09-18T04:35:47.983 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../100-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-18T04:35:47.985 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-18T04:35:48.002 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../085-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2024-09-18T04:35:48.016 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-18T04:35:48.034 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking nvme-cli (1.9-1ubuntu0.2) ... 2024-09-18T04:35:48.043 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-18T04:35:48.354 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-18T04:35:48.373 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../099-python-asyncssh-doc_1.12.2-1_all.deb ... 2024-09-18T04:35:48.379 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-websocket. 2024-09-18T04:35:48.398 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../086-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2024-09-18T04:35:48.412 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python-asyncssh-doc (1.12.2-1) ... 2024-09-18T04:35:48.437 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2024-09-18T04:35:48.752 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-18T04:35:48.770 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../087-python3-kubernetes_7.0.0~a1-2_all.deb ... 2024-09-18T04:35:48.785 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libjs-mathjax. 2024-09-18T04:35:48.802 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../100-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-18T04:35:48.826 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2024-09-18T04:35:48.843 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-18T04:35:49.357 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-18T04:35:49.375 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../088-ceph-mgr-k8sevents_18.2.4-855-g19e2a2be-1focal_all.deb ... 2024-09-18T04:35:49.415 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking ceph-mgr-k8sevents (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:49.693 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python-natsort-doc. 2024-09-18T04:35:49.711 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../101-python-natsort-doc_7.0.1-1_all.deb ... 2024-09-18T04:35:49.751 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2024-09-18T04:35:49.776 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-18T04:35:49.793 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../089-libonig5_6.9.4-1_amd64.deb ... 2024-09-18T04:35:49.834 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2024-09-18T04:35:50.087 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-webencodings. 2024-09-18T04:35:50.107 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../102-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2024-09-18T04:35:50.145 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-18T04:35:50.180 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-18T04:35:50.197 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../090-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-18T04:35:50.238 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-18T04:35:50.464 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python-natsort-doc. 2024-09-18T04:35:50.481 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-html5lib. 2024-09-18T04:35:50.482 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../101-python-natsort-doc_7.0.1-1_all.deb ... 2024-09-18T04:35:50.500 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../103-python3-html5lib_1.0.1-2_all.deb ... 2024-09-18T04:35:50.507 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package jq. 2024-09-18T04:35:50.522 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2024-09-18T04:35:50.525 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../091-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-18T04:35:50.539 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2024-09-18T04:35:50.565 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2024-09-18T04:35:50.843 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package socat. 2024-09-18T04:35:50.860 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../092-socat_1.7.3.3-2_amd64.deb ... 2024-09-18T04:35:50.867 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-webencodings. 2024-09-18T04:35:50.876 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-lxml:amd64. 2024-09-18T04:35:50.885 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../102-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2024-09-18T04:35:50.893 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../104-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2024-09-18T04:35:50.901 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking socat (1.7.3.3-2) ... 2024-09-18T04:35:50.925 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-18T04:35:50.925 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-18T04:35:51.219 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-html5lib. 2024-09-18T04:35:51.221 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package xmlstarlet. 2024-09-18T04:35:51.238 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../093-xmlstarlet_1.6.1-2build1_amd64.deb ... 2024-09-18T04:35:51.239 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../103-python3-html5lib_1.0.1-2_all.deb ... 2024-09-18T04:35:51.279 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2024-09-18T04:35:51.302 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2024-09-18T04:35:51.404 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-pastescript. 2024-09-18T04:35:51.421 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../105-python3-pastescript_2.0.2-4_all.deb ... 2024-09-18T04:35:51.462 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-18T04:35:51.642 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package ceph-test. 2024-09-18T04:35:51.658 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../094-ceph-test_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:51.664 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-lxml:amd64. 2024-09-18T04:35:51.681 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../104-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2024-09-18T04:35:51.699 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking ceph-test (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:51.722 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-18T04:35:51.757 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-psutil. 2024-09-18T04:35:51.778 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../106-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2024-09-18T04:35:51.815 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2024-09-18T04:35:52.127 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-18T04:35:52.144 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../107-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2024-09-18T04:35:52.188 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-18T04:35:52.209 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-pastescript. 2024-09-18T04:35:52.226 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../105-python3-pastescript_2.0.2-4_all.deb ... 2024-09-18T04:35:52.266 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-18T04:35:52.516 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package python3-simplejson. 2024-09-18T04:35:52.533 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../108-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2024-09-18T04:35:52.573 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-18T04:35:52.587 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-psutil. 2024-09-18T04:35:52.605 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../106-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2024-09-18T04:35:52.644 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2024-09-18T04:35:52.851 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-18T04:35:52.871 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../109-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2024-09-18T04:35:52.909 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-18T04:35:52.997 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-18T04:35:53.014 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../107-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2024-09-18T04:35:53.072 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-18T04:35:53.375 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-simplejson. 2024-09-18T04:35:53.396 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../108-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2024-09-18T04:35:53.433 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-18T04:35:53.572 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package radosgw. 2024-09-18T04:35:53.591 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../110-radosgw_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:53.630 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking radosgw (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:53.705 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-18T04:35:53.723 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../109-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2024-09-18T04:35:53.763 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-18T04:35:54.434 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package radosgw. 2024-09-18T04:35:54.450 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../110-radosgw_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:54.491 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking radosgw (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:55.033 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package rbd-fuse. 2024-09-18T04:35:55.050 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../111-rbd-fuse_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:55.091 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking rbd-fuse (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:55.335 INFO:teuthology.orchestra.run.smithi115.stdout:Selecting previously unselected package rbd-nbd. 2024-09-18T04:35:55.357 INFO:teuthology.orchestra.run.smithi115.stdout:Preparing to unpack .../112-rbd-nbd_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:55.393 INFO:teuthology.orchestra.run.smithi115.stdout:Unpacking rbd-nbd (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:55.725 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-attr (19.3.0-2) ... 2024-09-18T04:35:55.795 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package ceph-volume. 2024-09-18T04:35:55.816 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../095-ceph-volume_18.2.4-855-g19e2a2be-1focal_all.deb ... 2024-09-18T04:35:55.862 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking ceph-volume (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:55.953 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package rbd-fuse. 2024-09-18T04:35:55.973 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../111-rbd-fuse_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:56.011 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking rbd-fuse (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:56.026 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2024-09-18T04:35:56.152 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-18T04:35:56.165 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package fonts-mathjax. 2024-09-18T04:35:56.182 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../096-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-18T04:35:56.223 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-18T04:35:56.278 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up nvme-cli (1.9-1ubuntu0.2) ... 2024-09-18T04:35:56.280 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package rbd-nbd. 2024-09-18T04:35:56.300 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../112-rbd-nbd_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:56.338 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking rbd-nbd (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:56.471 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up cephadm (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:56.677 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-attr (19.3.0-2) ... 2024-09-18T04:35:56.752 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-18T04:35:56.770 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../097-libcephfs-dev_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:35:56.810 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libcephfs-dev (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:56.978 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2024-09-18T04:35:57.024 INFO:teuthology.orchestra.run.smithi115.stdout:Adding system user cephadm....done 2024-09-18T04:35:57.104 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-18T04:35:57.187 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-18T04:35:57.230 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up nvme-cli (1.9-1ubuntu0.2) ... 2024-09-18T04:35:57.231 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package nvme-cli. 2024-09-18T04:35:57.249 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../098-nvme-cli_1.9-1ubuntu0.2_amd64.deb ... 2024-09-18T04:35:57.309 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking nvme-cli (1.9-1ubuntu0.2) ... 2024-09-18T04:35:57.506 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up cephadm (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:35:57.506 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-18T04:35:57.621 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-18T04:35:57.639 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../099-python-asyncssh-doc_1.12.2-1_all.deb ... 2024-09-18T04:35:57.665 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-18T04:35:57.679 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python-asyncssh-doc (1.12.2-1) ... 2024-09-18T04:35:57.958 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-18T04:35:57.974 INFO:teuthology.orchestra.run.smithi082.stdout:Adding system user cephadm....done 2024-09-18T04:35:58.048 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libjs-mathjax. 2024-09-18T04:35:58.065 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../100-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-18T04:35:58.075 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up liboath0 (2.6.1-1.3) ... 2024-09-18T04:35:58.130 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-18T04:35:58.138 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-18T04:35:58.193 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2024-09-18T04:35:58.263 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-18T04:35:58.528 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-joblib (0.14.0-3) ... 2024-09-18T04:35:58.650 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-18T04:35:58.926 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-18T04:35:58.947 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-18T04:35:59.052 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up liboath0 (2.6.1-1.3) ... 2024-09-18T04:35:59.178 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2024-09-18T04:35:59.240 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-cachetools (4.0.0-1) ... 2024-09-18T04:35:59.525 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-18T04:35:59.714 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-joblib (0.14.0-3) ... 2024-09-18T04:35:59.714 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python-natsort-doc. 2024-09-18T04:35:59.714 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../101-python-natsort-doc_7.0.1-1_all.deb ... 2024-09-18T04:35:59.750 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2024-09-18T04:35:59.809 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-18T04:35:59.956 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-18T04:36:00.102 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-ceph-argparse (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:00.137 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-webencodings. 2024-09-18T04:36:00.154 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../102-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2024-09-18T04:36:00.194 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-18T04:36:00.249 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-cachetools (4.0.0-1) ... 2024-09-18T04:36:00.402 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-18T04:36:00.514 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-html5lib. 2024-09-18T04:36:00.529 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2024-09-18T04:36:00.531 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../103-python3-html5lib_1.0.1-2_all.deb ... 2024-09-18T04:36:00.534 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-18T04:36:00.672 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2024-09-18T04:36:00.810 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-18T04:36:00.821 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-18T04:36:01.042 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-lxml:amd64. 2024-09-18T04:36:01.059 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../104-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2024-09-18T04:36:01.091 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-18T04:36:01.094 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-ceph-argparse (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:01.106 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-18T04:36:01.232 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2024-09-18T04:36:01.387 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-18T04:36:01.527 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2024-09-18T04:36:01.536 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-pastescript. 2024-09-18T04:36:01.555 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../105-python3-pastescript_2.0.2-4_all.deb ... 2024-09-18T04:36:01.595 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-18T04:36:01.616 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-natsort (7.0.1-1) ... 2024-09-18T04:36:01.839 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-18T04:36:01.918 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2024-09-18T04:36:01.940 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-psutil. 2024-09-18T04:36:01.957 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../106-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2024-09-18T04:36:01.998 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2024-09-18T04:36:02.123 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-18T04:36:02.193 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-18T04:36:02.249 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2024-09-18T04:36:02.320 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-18T04:36:02.338 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../107-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2024-09-18T04:36:02.378 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-18T04:36:02.553 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2024-09-18T04:36:02.617 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-natsort (7.0.1-1) ... 2024-09-18T04:36:02.655 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-simplejson. 2024-09-18T04:36:02.676 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../108-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2024-09-18T04:36:02.713 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-18T04:36:02.837 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-18T04:36:02.934 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2024-09-18T04:36:03.016 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-18T04:36:03.037 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../109-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2024-09-18T04:36:03.074 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-18T04:36:03.189 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-tempita (0.5.2-6) ... 2024-09-18T04:36:03.210 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-18T04:36:03.490 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-packaging (20.3-1) ... 2024-09-18T04:36:03.586 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2024-09-18T04:36:03.753 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package radosgw. 2024-09-18T04:36:03.770 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../110-radosgw_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:36:03.799 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2024-09-18T04:36:03.810 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking radosgw (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:03.862 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-18T04:36:03.925 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-18T04:36:04.043 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-18T04:36:04.180 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-tempita (0.5.2-6) ... 2024-09-18T04:36:04.345 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2024-09-18T04:36:04.481 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-packaging (20.3-1) ... 2024-09-18T04:36:04.757 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-18T04:36:04.782 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2024-09-18T04:36:04.916 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-18T04:36:05.272 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package rbd-fuse. 2024-09-18T04:36:05.290 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../111-rbd-fuse_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:36:05.330 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking rbd-fuse (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:05.406 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-18T04:36:05.467 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-18T04:36:05.618 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package rbd-nbd. 2024-09-18T04:36:05.636 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../112-rbd-nbd_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-18T04:36:05.675 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking rbd-nbd (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:05.766 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up socat (1.7.3.3-2) ... 2024-09-18T04:36:05.891 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-ceph-common (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:05.927 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2024-09-18T04:36:05.998 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-attr (19.3.0-2) ... 2024-09-18T04:36:06.226 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T04:36:06.298 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2024-09-18T04:36:06.311 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-18T04:36:06.361 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-html5lib (1.0.1-2) ... 2024-09-18T04:36:06.423 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-18T04:36:06.574 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up nvme-cli (1.9-1ubuntu0.2) ... 2024-09-18T04:36:06.671 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-18T04:36:06.746 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-18T04:36:06.776 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up cephadm (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:06.863 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2024-09-18T04:36:06.981 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-pluggy (0.13.0-2) ... 2024-09-18T04:36:07.023 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up socat (1.7.3.3-2) ... 2024-09-18T04:36:07.182 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-ceph-common (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:07.266 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-18T04:36:07.478 INFO:teuthology.orchestra.run.smithi028.stdout:Adding system user cephadm....done 2024-09-18T04:36:07.492 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T04:36:07.583 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T04:36:07.626 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-html5lib (1.0.1-2) ... 2024-09-18T04:36:07.641 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-18T04:36:07.709 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-18T04:36:07.767 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-18T04:36:07.836 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2024-09-18T04:36:07.969 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-18T04:36:08.095 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2024-09-18T04:36:08.127 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-18T04:36:08.139 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2024-09-18T04:36:08.221 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-pluggy (0.13.0-2) ... 2024-09-18T04:36:08.420 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-18T04:36:08.423 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-logutils (0.3.3-7) ... 2024-09-18T04:36:08.505 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-18T04:36:08.520 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up liboath0 (2.6.1-1.3) ... 2024-09-18T04:36:08.646 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2024-09-18T04:36:08.708 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-18T04:36:08.831 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T04:36:08.834 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-18T04:36:08.957 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-18T04:36:08.981 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-joblib (0.14.0-3) ... 2024-09-18T04:36:09.075 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2024-09-18T04:36:09.127 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-prettytable (0.7.2-5) ... 2024-09-18T04:36:09.383 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-18T04:36:09.401 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2024-09-18T04:36:09.428 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-18T04:36:09.676 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-cachetools (4.0.0-1) ... 2024-09-18T04:36:09.694 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-logutils (0.3.3-7) ... 2024-09-18T04:36:09.729 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2024-09-18T04:36:09.961 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-18T04:36:09.978 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-18T04:36:10.104 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-18T04:36:10.121 INFO:teuthology.orchestra.run.smithi115.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2024-09-18T04:36:10.253 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-18T04:36:10.373 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2024-09-18T04:36:10.380 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-prettytable (0.7.2-5) ... 2024-09-18T04:36:10.499 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-18T04:36:10.538 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-ceph-argparse (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:10.656 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-18T04:36:10.825 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2024-09-18T04:36:10.847 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-18T04:36:10.949 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2024-09-18T04:36:10.973 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2024-09-18T04:36:11.168 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-18T04:36:11.249 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-18T04:36:11.294 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-18T04:36:11.341 INFO:teuthology.orchestra.run.smithi082.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2024-09-18T04:36:11.420 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-18T04:36:11.533 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-18T04:36:11.626 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2024-09-18T04:36:11.659 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2024-09-18T04:36:11.753 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-18T04:36:11.755 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-18T04:36:12.064 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2024-09-18T04:36:12.068 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-natsort (7.0.1-1) ... 2024-09-18T04:36:12.070 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2024-09-18T04:36:12.370 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2024-09-18T04:36:12.438 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-18T04:36:12.564 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-18T04:36:12.654 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-18T04:36:12.692 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-18T04:36:13.022 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2024-09-18T04:36:13.035 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-18T04:36:13.323 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-18T04:36:13.344 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2024-09-18T04:36:13.528 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-18T04:36:13.654 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-pytest (4.6.9-1) ... 2024-09-18T04:36:13.665 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-tempita (0.5.2-6) ... 2024-09-18T04:36:13.975 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-packaging (20.3-1) ... 2024-09-18T04:36:14.022 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2024-09-18T04:36:14.251 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2024-09-18T04:36:14.323 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2024-09-18T04:36:14.385 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-18T04:36:14.511 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-18T04:36:14.624 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T04:36:14.750 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-bs4 (4.8.2-1) ... 2024-09-18T04:36:14.808 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-18T04:36:14.827 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2024-09-18T04:36:14.959 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-pytest (4.6.9-1) ... 2024-09-18T04:36:15.101 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-18T04:36:15.230 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-18T04:36:15.319 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2024-09-18T04:36:15.436 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-asyncssh (1.12.2-1) ... 2024-09-18T04:36:15.598 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-18T04:36:15.611 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2024-09-18T04:36:15.821 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-18T04:36:15.912 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T04:36:15.950 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up socat (1.7.3.3-2) ... 2024-09-18T04:36:16.038 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-bs4 (4.8.2-1) ... 2024-09-18T04:36:16.076 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-ceph-common (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:16.139 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-18T04:36:16.390 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-18T04:36:16.402 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T04:36:16.537 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-html5lib (1.0.1-2) ... 2024-09-18T04:36:16.725 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-asyncssh (1.12.2-1) ... 2024-09-18T04:36:16.888 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-18T04:36:16.997 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2024-09-18T04:36:17.098 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-pluggy (0.13.0-2) ... 2024-09-18T04:36:17.118 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-18T04:36:17.202 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-18T04:36:17.328 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2024-09-18T04:36:17.383 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-18T04:36:17.452 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-18T04:36:17.463 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2024-09-18T04:36:17.572 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-rsa (4.0-3) ... 2024-09-18T04:36:17.709 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T04:36:17.838 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-18T04:36:17.873 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-18T04:36:17.961 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2024-09-18T04:36:17.999 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-webtest (2.0.32-2) ... 2024-09-18T04:36:18.287 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2024-09-18T04:36:18.309 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-18T04:36:18.524 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-18T04:36:18.580 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-logutils (0.3.3-7) ... 2024-09-18T04:36:18.610 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-18T04:36:18.655 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2024-09-18T04:36:18.776 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2024-09-18T04:36:18.875 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-18T04:36:18.902 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-rsa (4.0-3) ... 2024-09-18T04:36:18.944 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2024-09-18T04:36:19.001 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-18T04:36:19.227 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-18T04:36:19.294 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-prettytable (0.7.2-5) ... 2024-09-18T04:36:19.313 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up librados2 (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:19.354 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-webtest (2.0.32-2) ... 2024-09-18T04:36:19.431 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python-asyncssh-doc (1.12.2-1) ... 2024-09-18T04:36:19.557 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up librgw2 (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:19.586 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-18T04:36:19.655 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-18T04:36:19.683 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up libsqlite3-mod-ceph (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:19.894 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up libcephfs2 (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:19.895 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2024-09-18T04:36:19.948 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-18T04:36:20.020 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-routes (2.4.1-2) ... 2024-09-18T04:36:20.266 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2024-09-18T04:36:20.305 INFO:teuthology.orchestra.run.smithi028.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2024-09-18T04:36:20.347 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up libradosstriper1 (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:20.473 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-google-auth (1.5.1-2) ... 2024-09-18T04:36:20.728 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up librados2 (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:20.728 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2024-09-18T04:36:20.728 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-18T04:36:20.752 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python-asyncssh-doc (1.12.2-1) ... 2024-09-18T04:36:20.774 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up librbd1 (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:20.902 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up ceph-mgr-modules-core (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:20.912 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up librgw2 (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:20.959 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2024-09-18T04:36:21.026 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up ceph-fuse (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:21.079 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libsqlite3-mod-ceph (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:21.256 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-18T04:36:21.257 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-18T04:36:21.299 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libcephfs2 (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:21.344 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-18T04:36:21.425 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-routes (2.4.1-2) ... 2024-09-18T04:36:21.469 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-18T04:36:21.595 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-18T04:36:21.694 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up libcephfs-dev (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:21.718 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libradosstriper1 (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:21.829 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-rados (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:21.869 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-google-auth (1.5.1-2) ... 2024-09-18T04:36:21.922 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-18T04:36:21.955 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2024-09-18T04:36:22.180 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up librbd1 (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:22.401 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2024-09-18T04:36:22.401 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-mgr-modules-core (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:22.423 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-fuse (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:22.648 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-18T04:36:22.648 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-18T04:36:22.867 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-rbd (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:22.993 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up rbd-fuse (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:23.100 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libcephfs-dev (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:23.135 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up rbd-nbd (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:23.201 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-rados (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:23.261 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-rgw (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:23.301 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2024-09-18T04:36:23.387 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up python3-cephfs (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:23.520 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up ceph-common (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:23.745 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-18T04:36:23.871 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-pytest (4.6.9-1) ... 2024-09-18T04:36:24.069 INFO:teuthology.orchestra.run.smithi115.stdout:Adding group ceph....done 2024-09-18T04:36:24.214 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-rbd (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:24.231 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2024-09-18T04:36:24.340 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up rbd-fuse (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:24.434 INFO:teuthology.orchestra.run.smithi115.stdout:Adding system user ceph....done 2024-09-18T04:36:24.499 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up rbd-nbd (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:24.531 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2024-09-18T04:36:24.609 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-rgw (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:24.610 INFO:teuthology.orchestra.run.smithi115.stdout:Setting system user ceph properties....done 2024-09-18T04:36:24.635 INFO:teuthology.orchestra.run.smithi115.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-18T04:36:24.717 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-cephfs (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:24.789 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-18T04:36:24.827 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-common (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:24.833 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T04:36:24.958 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-bs4 (4.8.2-1) ... 2024-09-18T04:36:25.097 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-18T04:36:25.318 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-18T04:36:25.413 INFO:teuthology.orchestra.run.smithi082.stdout:Adding group ceph....done 2024-09-18T04:36:25.589 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up ceph-test (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:25.645 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-asyncssh (1.12.2-1) ... 2024-09-18T04:36:25.740 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up radosgw (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:25.782 INFO:teuthology.orchestra.run.smithi082.stdout:Adding system user ceph....done 2024-09-18T04:36:25.949 INFO:teuthology.orchestra.run.smithi082.stdout:Setting system user ceph properties....done 2024-09-18T04:36:25.969 INFO:teuthology.orchestra.run.smithi082.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-18T04:36:26.029 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-18T04:36:26.123 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-18T04:36:26.229 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-18T04:36:26.230 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-18T04:36:26.347 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-18T04:36:26.446 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-18T04:36:26.667 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up ceph-base (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:26.920 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-test (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:27.069 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up radosgw (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:27.070 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-18T04:36:27.386 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-18T04:36:27.510 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up ceph-mds (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:27.511 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2024-09-18T04:36:27.540 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-18T04:36:27.540 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-18T04:36:27.637 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2024-09-18T04:36:27.740 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-18T04:36:27.740 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-18T04:36:27.763 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-rsa (4.0-3) ... 2024-09-18T04:36:27.998 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-base (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:28.098 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-18T04:36:28.187 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up ceph-mgr (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:28.223 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-webtest (2.0.32-2) ... 2024-09-18T04:36:28.381 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-18T04:36:28.425 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-18T04:36:28.425 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-18T04:36:28.508 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-18T04:36:28.801 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-18T04:36:28.833 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-mds (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:28.846 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up ceph-osd (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:29.068 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-18T04:36:29.068 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-18T04:36:29.127 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2024-09-18T04:36:29.352 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-18T04:36:29.352 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-18T04:36:29.487 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up librados2 (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:29.510 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-mgr (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:29.613 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python-asyncssh-doc (1.12.2-1) ... 2024-09-18T04:36:29.740 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-18T04:36:29.740 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-18T04:36:29.764 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up librgw2 (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:29.798 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up ceph-mgr-k8sevents (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:29.890 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libsqlite3-mod-ceph (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:29.923 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up ceph-mgr-diskprediction-local (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:30.093 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libcephfs2 (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:30.100 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up ceph-mon (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:30.186 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-osd (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:30.219 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-routes (2.4.1-2) ... 2024-09-18T04:36:30.322 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-18T04:36:30.322 INFO:teuthology.orchestra.run.smithi115.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-18T04:36:30.512 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libradosstriper1 (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:30.666 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-18T04:36:30.666 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-18T04:36:30.688 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-google-auth (1.5.1-2) ... 2024-09-18T04:36:30.898 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up ceph-mgr-cephadm (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:30.981 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up librbd1 (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:31.053 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up ceph (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:31.116 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up ceph-mgr-modules-core (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:31.170 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up ceph-mgr-dashboard (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:31.242 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up ceph-fuse (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:31.246 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-mgr-k8sevents (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:31.305 INFO:teuthology.orchestra.run.smithi115.stdout:Setting up ceph-volume (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:31.474 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-18T04:36:31.475 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-18T04:36:31.557 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-mgr-diskprediction-local (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:31.658 INFO:teuthology.orchestra.run.smithi115.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2024-09-18T04:36:31.700 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-mon (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:31.801 INFO:teuthology.orchestra.run.smithi115.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2024-09-18T04:36:31.918 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libcephfs-dev (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:31.919 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-18T04:36:31.919 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-18T04:36:32.044 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-rados (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:32.126 INFO:teuthology.orchestra.run.smithi115.stdout:Processing triggers for man-db (2.9.1-1) ... 2024-09-18T04:36:32.153 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2024-09-18T04:36:32.375 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-mgr-cephadm (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:32.518 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:32.644 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-mgr-dashboard (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:32.787 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-volume (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:33.082 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-rbd (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:33.165 INFO:teuthology.orchestra.run.smithi082.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2024-09-18T04:36:33.216 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up rbd-fuse (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:33.333 INFO:teuthology.orchestra.run.smithi082.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2024-09-18T04:36:33.342 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up rbd-nbd (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:33.485 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-rgw (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:33.611 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-cephfs (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:33.617 INFO:teuthology.orchestra.run.smithi082.stdout:Processing triggers for man-db (2.9.1-1) ... 2024-09-18T04:36:33.729 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up ceph-common (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:34.273 INFO:teuthology.orchestra.run.smithi028.stdout:Adding group ceph....done 2024-09-18T04:36:34.616 INFO:teuthology.orchestra.run.smithi028.stdout:Adding system user ceph....done 2024-09-18T04:36:34.750 INFO:teuthology.orchestra.run.smithi028.stdout:Setting system user ceph properties....done 2024-09-18T04:36:34.771 INFO:teuthology.orchestra.run.smithi028.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-18T04:36:34.936 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-18T04:36:35.021 INFO:teuthology.orchestra.run.smithi115.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2024-09-18T04:36:35.260 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-18T04:36:35.645 INFO:teuthology.orchestra.run.smithi115.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-18T04:36:35.651 DEBUG:teuthology.parallel:result is None 2024-09-18T04:36:35.721 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up ceph-test (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:35.847 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up radosgw (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:36.328 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-18T04:36:36.328 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-18T04:36:36.528 INFO:teuthology.orchestra.run.smithi082.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2024-09-18T04:36:36.791 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up ceph-base (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:37.180 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-18T04:36:37.625 INFO:teuthology.orchestra.run.smithi082.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-18T04:36:37.630 DEBUG:teuthology.parallel:result is None 2024-09-18T04:36:37.642 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up ceph-mds (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:37.889 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-18T04:36:37.889 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-18T04:36:38.344 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up ceph-mgr (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:38.575 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-18T04:36:38.575 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-18T04:36:39.037 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up ceph-osd (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:39.507 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-18T04:36:39.507 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-18T04:36:39.938 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up ceph-mgr-k8sevents (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:40.048 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up ceph-mgr-diskprediction-local (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:40.190 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up ceph-mon (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:40.423 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-18T04:36:40.423 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-18T04:36:40.868 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up ceph-mgr-cephadm (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:41.020 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up ceph (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:41.120 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up ceph-mgr-dashboard (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:41.237 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up ceph-volume (18.2.4-855-g19e2a2be-1focal) ... 2024-09-18T04:36:41.599 INFO:teuthology.orchestra.run.smithi028.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2024-09-18T04:36:41.784 INFO:teuthology.orchestra.run.smithi028.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2024-09-18T04:36:42.085 INFO:teuthology.orchestra.run.smithi028.stdout:Processing triggers for man-db (2.9.1-1) ... 2024-09-18T04:36:45.029 INFO:teuthology.orchestra.run.smithi028.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2024-09-18T04:36:45.569 INFO:teuthology.orchestra.run.smithi028.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-18T04:36:45.574 DEBUG:teuthology.parallel:result is None 2024-09-18T04:36:45.575 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-18T04:36:45.726 DEBUG:teuthology.orchestra.run.smithi028:> dpkg-query -W -f '${Version}' ceph 2024-09-18T04:36:45.752 INFO:teuthology.orchestra.run.smithi028.stdout:18.2.4-855-g19e2a2be-1focal 2024-09-18T04:36:45.753 INFO:teuthology.packaging:The installed version of ceph is 18.2.4-855-g19e2a2be-1focal 2024-09-18T04:36:45.753 INFO:teuthology.task.install:The correct ceph version 18.2.4-855-g19e2a2be-1focal is installed. 2024-09-18T04:36:45.755 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-18T04:36:45.899 DEBUG:teuthology.orchestra.run.smithi082:> dpkg-query -W -f '${Version}' ceph 2024-09-18T04:36:45.923 INFO:teuthology.orchestra.run.smithi082.stdout:18.2.4-855-g19e2a2be-1focal 2024-09-18T04:36:45.923 INFO:teuthology.packaging:The installed version of ceph is 18.2.4-855-g19e2a2be-1focal 2024-09-18T04:36:45.923 INFO:teuthology.task.install:The correct ceph version 18.2.4-855-g19e2a2be-1focal is installed. 2024-09-18T04:36:45.925 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-18T04:36:46.067 DEBUG:teuthology.orchestra.run.smithi115:> dpkg-query -W -f '${Version}' ceph 2024-09-18T04:36:46.090 INFO:teuthology.orchestra.run.smithi115.stdout:18.2.4-855-g19e2a2be-1focal 2024-09-18T04:36:46.090 INFO:teuthology.packaging:The installed version of ceph is 18.2.4-855-g19e2a2be-1focal 2024-09-18T04:36:46.090 INFO:teuthology.task.install:The correct ceph version 18.2.4-855-g19e2a2be-1focal is installed. 2024-09-18T04:36:46.092 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-18T04:36:46.093 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-18T04:36:46.093 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-18T04:36:46.109 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-09-18T04:36:46.109 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-18T04:36:46.125 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-18T04:36:46.126 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-18T04:36:46.147 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-18T04:36:46.147 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-18T04:36:46.147 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/daemon-helper 2024-09-18T04:36:46.163 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-18T04:36:46.174 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-09-18T04:36:46.174 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/usr/bin/daemon-helper 2024-09-18T04:36:46.188 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-18T04:36:46.199 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-18T04:36:46.199 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/usr/bin/daemon-helper 2024-09-18T04:36:46.216 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-18T04:36:46.227 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-18T04:36:46.227 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-18T04:36:46.227 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-18T04:36:46.242 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-18T04:36:46.252 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-09-18T04:36:46.253 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-18T04:36:46.269 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-18T04:36:46.280 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-18T04:36:46.280 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-18T04:36:46.296 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-18T04:36:46.305 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-18T04:36:46.305 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-18T04:36:46.305 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/stdin-killer 2024-09-18T04:36:46.321 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-18T04:36:46.330 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-09-18T04:36:46.330 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/usr/bin/stdin-killer 2024-09-18T04:36:46.345 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-18T04:36:46.358 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-18T04:36:46.358 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/usr/bin/stdin-killer 2024-09-18T04:36:46.374 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-18T04:36:46.429 INFO:teuthology.run_tasks:Running task ceph... 2024-09-18T04:36:46.509 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-18T04:36:46.510 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 777 /var/log/ceph 2024-09-18T04:36:46.513 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 777 /var/log/ceph 2024-09-18T04:36:46.515 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod 777 /var/log/ceph 2024-09-18T04:36:46.531 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-18T04:36:46.531 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-18T04:36:46.572 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-18T04:36:46.576 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-18T04:36:46.594 INFO:tasks.ceph:Creating extra log directories... 2024-09-18T04:36:46.594 DEBUG:teuthology.orchestra.run.smithi028:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-18T04:36:46.598 DEBUG:teuthology.orchestra.run.smithi082:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-18T04:36:46.601 DEBUG:teuthology.orchestra.run.smithi115:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-18T04:36:46.675 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-18T04:36:46.675 INFO:tasks.ceph:config {'conf': {'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-18T04:36:46.675 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-17_22:16:03-rbd-reef-distro-default-smithi/7909789', 'branch': 'reef', 'description': 'rbd/device/{base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{ubuntu_20.04} thrashers/default thrashosds-health workloads/rbd_nbd}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7909789', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-09-17_22:16:03-rbd-reef-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'machine': {'cpus': 1, 'disk': 40, 'ram': 8000}, 'volumes': {'count': 4, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '20.04', 'overrides': {'admin_socket': {'branch': 'reef'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from'], 'sha1': '19e2a2beb6f41a3ab62650e38614eb29b59dcbb9'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'extra_packages': ['rbd-nbd'], 'flavor': 'default', 'sha1': '19e2a2beb6f41a3ab62650e38614eb29b59dcbb9'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'reef', 'sha1': 'a477b1894d5c5a613f23260ca0e6b96d79652ba7'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 930, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'], ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5'], ['client.0']], 'seed': 4623, 'sha1': '19e2a2beb6f41a3ab62650e38614eb29b59dcbb9', 'sleep_before_teardown': 0, 'subset': '80/128', 'suite': 'rbd', 'suite_branch': 'reef', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_a477b1894d5c5a613f23260ca0e6b96d79652ba7/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'a477b1894d5c5a613f23260ca0e6b96d79652ba7', 'targets': {'smithi028.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM=', 'smithi082.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM=', 'smithi115.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rbd/rbd-nbd.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-17_22:16:03', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-18T04:36:46.676 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-18T04:36:46.678 DEBUG:teuthology.orchestra.run.smithi082:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-18T04:36:46.681 DEBUG:teuthology.orchestra.run.smithi115:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-18T04:36:46.687 DEBUG:teuthology.orchestra.run.smithi028:> sudo install -d -m0777 -- /var/run/ceph 2024-09-18T04:36:46.690 DEBUG:teuthology.orchestra.run.smithi082:> sudo install -d -m0777 -- /var/run/ceph 2024-09-18T04:36:46.692 DEBUG:teuthology.orchestra.run.smithi115:> sudo install -d -m0777 -- /var/run/ceph 2024-09-18T04:36:46.761 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-18T04:36:46.761 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/scratch_devs of=/dev/stdout 2024-09-18T04:36:46.768 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-18T04:36:46.768 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_1 2024-09-18T04:36:46.776 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-18T04:36:46.776 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T04:36:46.776 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 6h/6d Inode: 360 Links: 1 2024-09-18T04:36:46.776 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T04:36:46.776 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-09-18 04:33:09.820820983 +0000 2024-09-18T04:36:46.777 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-09-18 04:33:09.592826242 +0000 2024-09-18T04:36:46.777 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-09-18 04:33:09.592826242 +0000 2024-09-18T04:36:46.777 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: - 2024-09-18T04:36:46.777 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-18T04:36:46.791 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-18T04:36:46.791 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-18T04:36:46.791 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000301744 s, 1.7 MB/s 2024-09-18T04:36:46.793 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-18T04:36:46.802 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_2 2024-09-18T04:36:46.860 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-18T04:36:46.860 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T04:36:46.860 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 6h/6d Inode: 365 Links: 1 2024-09-18T04:36:46.860 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T04:36:46.860 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-09-18 04:33:10.220811757 +0000 2024-09-18T04:36:46.860 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-09-18 04:33:10.040815908 +0000 2024-09-18T04:36:46.860 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-09-18 04:33:10.040815908 +0000 2024-09-18T04:36:46.860 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: - 2024-09-18T04:36:46.861 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-18T04:36:46.873 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-18T04:36:46.873 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-18T04:36:46.873 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.00030367 s, 1.7 MB/s 2024-09-18T04:36:46.874 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-18T04:36:46.936 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_3 2024-09-18T04:36:46.997 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-18T04:36:46.997 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T04:36:46.997 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 6h/6d Inode: 370 Links: 1 2024-09-18T04:36:46.997 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T04:36:46.997 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-09-18 04:33:10.644801976 +0000 2024-09-18T04:36:46.997 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-09-18 04:33:10.436806774 +0000 2024-09-18T04:36:46.997 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-09-18 04:33:10.436806774 +0000 2024-09-18T04:36:46.997 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: - 2024-09-18T04:36:46.997 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-18T04:36:47.072 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-18T04:36:47.072 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-18T04:36:47.072 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000309527 s, 1.7 MB/s 2024-09-18T04:36:47.073 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-18T04:36:47.132 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_4 2024-09-18T04:36:47.187 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-18T04:36:47.187 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T04:36:47.187 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 6h/6d Inode: 378 Links: 1 2024-09-18T04:36:47.187 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T04:36:47.187 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-09-18 04:33:11.048792657 +0000 2024-09-18T04:36:47.187 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-09-18 04:33:10.832797640 +0000 2024-09-18T04:36:47.187 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-09-18 04:33:10.832797640 +0000 2024-09-18T04:36:47.187 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: - 2024-09-18T04:36:47.187 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-18T04:36:47.202 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-18T04:36:47.202 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-18T04:36:47.202 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000277391 s, 1.8 MB/s 2024-09-18T04:36:47.203 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-18T04:36:47.212 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-18T04:36:47.212 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-09-18T04:36:47.212 DEBUG:teuthology.orchestra.run.smithi082:> dd if=/scratch_devs of=/dev/stdout 2024-09-18T04:36:47.219 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-18T04:36:47.220 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_1 2024-09-18T04:36:47.269 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-18T04:36:47.269 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T04:36:47.269 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 6h/6d Inode: 359 Links: 1 2024-09-18T04:36:47.269 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T04:36:47.269 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2024-09-18 04:33:08.964349996 +0000 2024-09-18T04:36:47.270 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2024-09-18 04:33:08.764354599 +0000 2024-09-18T04:36:47.270 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2024-09-18 04:33:08.764354599 +0000 2024-09-18T04:36:47.270 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: - 2024-09-18T04:36:47.270 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-18T04:36:47.281 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2024-09-18T04:36:47.281 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2024-09-18T04:36:47.281 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.00043439 s, 1.2 MB/s 2024-09-18T04:36:47.283 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-18T04:36:47.292 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_2 2024-09-18T04:36:47.341 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-18T04:36:47.342 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T04:36:47.342 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 6h/6d Inode: 364 Links: 1 2024-09-18T04:36:47.342 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T04:36:47.342 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2024-09-18 04:33:09.384340331 +0000 2024-09-18T04:36:47.342 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2024-09-18 04:33:09.180345025 +0000 2024-09-18T04:36:47.342 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2024-09-18 04:33:09.180345025 +0000 2024-09-18T04:36:47.342 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: - 2024-09-18T04:36:47.342 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-18T04:36:47.354 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2024-09-18T04:36:47.354 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2024-09-18T04:36:47.354 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.000439687 s, 1.2 MB/s 2024-09-18T04:36:47.355 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-18T04:36:47.365 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_3 2024-09-18T04:36:47.418 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-18T04:36:47.418 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T04:36:47.418 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 6h/6d Inode: 370 Links: 1 2024-09-18T04:36:47.418 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T04:36:47.418 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2024-09-18 04:33:09.812330481 +0000 2024-09-18T04:36:47.418 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2024-09-18 04:33:09.620334900 +0000 2024-09-18T04:36:47.418 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2024-09-18 04:33:09.620334900 +0000 2024-09-18T04:36:47.419 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: - 2024-09-18T04:36:47.419 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-18T04:36:47.433 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2024-09-18T04:36:47.433 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2024-09-18T04:36:47.433 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.000433168 s, 1.2 MB/s 2024-09-18T04:36:47.434 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-18T04:36:47.487 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_4 2024-09-18T04:36:47.494 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-18T04:36:47.494 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T04:36:47.494 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 6h/6d Inode: 377 Links: 1 2024-09-18T04:36:47.494 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T04:36:47.494 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2024-09-18 04:33:10.216321184 +0000 2024-09-18T04:36:47.494 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2024-09-18 04:33:10.020325695 +0000 2024-09-18T04:36:47.494 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2024-09-18 04:33:10.020325695 +0000 2024-09-18T04:36:47.494 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: - 2024-09-18T04:36:47.495 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-18T04:36:47.508 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2024-09-18T04:36:47.508 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2024-09-18T04:36:47.508 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.000376772 s, 1.4 MB/s 2024-09-18T04:36:47.510 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-18T04:36:47.563 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-18T04:36:47.563 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi028.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi082.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-18T04:36:47.563 INFO:tasks.ceph:Generating config... 2024-09-18T04:36:47.564 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-18T04:36:47.564 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-09-18T04:36:47.565 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-18T04:36:47.565 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-18T04:36:47.565 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-18T04:36:47.565 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-18T04:36:47.565 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-18T04:36:47.565 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-18T04:36:47.565 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-18T04:36:47.565 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-09-18T04:36:47.565 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-18T04:36:47.565 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-18T04:36:47.565 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-18T04:36:47.565 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-18T04:36:47.565 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-18T04:36:47.566 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-18T04:36:47.566 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-18T04:36:47.566 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-18T04:36:47.566 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-18T04:36:47.566 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-18T04:36:47.566 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-09-18T04:36:47.566 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-18T04:36:47.566 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-09-18T04:36:47.566 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-18T04:36:47.566 INFO:tasks.ceph:Setting up mon.a... 2024-09-18T04:36:47.566 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-18T04:36:47.608 INFO:teuthology.orchestra.run.smithi028.stdout:creating /etc/ceph/ceph.keyring 2024-09-18T04:36:47.621 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-18T04:36:47.706 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-18T04:36:47.722 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.28'), ('mon.c', '[v2:172.21.15.28:3301,v1:172.21.15.28:6790]'), ('mon.b', '172.21.15.82')] 2024-09-18T04:36:47.722 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.28,[v2:172.21.15.28:3301,v1:172.21.15.28:6790],172.21.15.82', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-09-18T04:36:47.723 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-18T04:36:47.723 DEBUG:teuthology.orchestra.run.smithi028:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-18T04:36:47.730 DEBUG:teuthology.orchestra.run.smithi028:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.28 --addv c '[v2:172.21.15.28:3301,v1:172.21.15.28:6790]' --add b 172.21.15.82 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-18T04:36:47.817 INFO:teuthology.orchestra.run.smithi028.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-18T04:36:47.817 INFO:teuthology.orchestra.run.smithi028.stdout:monmaptool: generated fsid be6146ba-a991-4f6e-b359-99cbedc9f9b1 2024-09-18T04:36:47.817 INFO:teuthology.orchestra.run.smithi028.stdout:setting min_mon_release = pacific 2024-09-18T04:36:47.817 INFO:teuthology.orchestra.run.smithi028.stdout:epoch 0 2024-09-18T04:36:47.817 INFO:teuthology.orchestra.run.smithi028.stdout:fsid be6146ba-a991-4f6e-b359-99cbedc9f9b1 2024-09-18T04:36:47.818 INFO:teuthology.orchestra.run.smithi028.stdout:last_changed 2024-09-18T04:36:47.816468+0000 2024-09-18T04:36:47.818 INFO:teuthology.orchestra.run.smithi028.stdout:created 2024-09-18T04:36:47.816468+0000 2024-09-18T04:36:47.818 INFO:teuthology.orchestra.run.smithi028.stdout:min_mon_release 16 (pacific) 2024-09-18T04:36:47.818 INFO:teuthology.orchestra.run.smithi028.stdout:election_strategy: 1 2024-09-18T04:36:47.818 INFO:teuthology.orchestra.run.smithi028.stdout:0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-09-18T04:36:47.818 INFO:teuthology.orchestra.run.smithi028.stdout:1: [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] mon.b 2024-09-18T04:36:47.818 INFO:teuthology.orchestra.run.smithi028.stdout:2: [v2:172.21.15.28:3301/0,v1:172.21.15.28:6790/0] mon.c 2024-09-18T04:36:47.818 INFO:teuthology.orchestra.run.smithi028.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-18T04:36:47.819 DEBUG:teuthology.orchestra.run.smithi028:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-18T04:36:47.824 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID be6146ba-a991-4f6e-b359-99cbedc9f9b1... 2024-09-18T04:36:47.824 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-18T04:36:47.827 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-18T04:36:47.829 DEBUG:teuthology.orchestra.run.smithi115:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-18T04:36:47.857 INFO:teuthology.orchestra.run.smithi028.stdout:[global] 2024-09-18T04:36:47.857 INFO:teuthology.orchestra.run.smithi028.stdout: chdir = "" 2024-09-18T04:36:47.857 INFO:teuthology.orchestra.run.smithi028.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-18T04:36:47.857 INFO:teuthology.orchestra.run.smithi028.stdout: auth supported = cephx 2024-09-18T04:36:47.857 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:36:47.857 INFO:teuthology.orchestra.run.smithi028.stdout: filestore xattr use omap = true 2024-09-18T04:36:47.858 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:36:47.858 INFO:teuthology.orchestra.run.smithi028.stdout: mon clock drift allowed = 1.000 2024-09-18T04:36:47.858 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:36:47.858 INFO:teuthology.orchestra.run.smithi028.stdout: osd crush chooseleaf type = 0 2024-09-18T04:36:47.858 INFO:teuthology.orchestra.run.smithi028.stdout: auth debug = true 2024-09-18T04:36:47.858 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:36:47.858 INFO:teuthology.orchestra.run.smithi028.stdout: ms die on old message = true 2024-09-18T04:36:47.858 INFO:teuthology.orchestra.run.smithi028.stdout: ms die on bug = true 2024-09-18T04:36:47.858 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:36:47.858 INFO:teuthology.orchestra.run.smithi028.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-18T04:36:47.858 INFO:teuthology.orchestra.run.smithi028.stdout: mon pg warn max object skew = 0 2024-09-18T04:36:47.858 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:36:47.858 INFO:teuthology.orchestra.run.smithi028.stdout: # disable pg_autoscaler by default for new pools 2024-09-18T04:36:47.858 INFO:teuthology.orchestra.run.smithi028.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-18T04:36:47.859 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:36:47.859 INFO:teuthology.orchestra.run.smithi028.stdout: osd pool default size = 2 2024-09-18T04:36:47.859 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:36:47.859 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd allow primary affinity = true 2024-09-18T04:36:47.859 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd allow pg remap = true 2024-09-18T04:36:47.859 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on legacy crush tunables = false 2024-09-18T04:36:47.859 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on crush straw calc version zero = false 2024-09-18T04:36:47.859 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on no sortbitwise = false 2024-09-18T04:36:47.859 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on osd down out interval zero = false 2024-09-18T04:36:47.859 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on too few osds = false 2024-09-18T04:36:47.859 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-18T04:36:47.859 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-18T04:36:47.859 INFO:teuthology.orchestra.run.smithi028.stdout: mon_allow_pool_size_one = true 2024-09-18T04:36:47.860 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:36:47.860 INFO:teuthology.orchestra.run.smithi028.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-18T04:36:47.860 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:36:47.860 INFO:teuthology.orchestra.run.smithi028.stdout: osd default data pool replay window = 5 2024-09-18T04:36:47.860 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:36:47.860 INFO:teuthology.orchestra.run.smithi028.stdout: mon allow pool delete = true 2024-09-18T04:36:47.860 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:36:47.860 INFO:teuthology.orchestra.run.smithi028.stdout: mon cluster log file level = debug 2024-09-18T04:36:47.860 INFO:teuthology.orchestra.run.smithi028.stdout: debug asserts on shutdown = true 2024-09-18T04:36:47.860 INFO:teuthology.orchestra.run.smithi028.stdout: mon health detail to clog = false 2024-09-18T04:36:47.860 INFO:teuthology.orchestra.run.smithi028.stdout: mon host = "172.21.15.28,[v2:172.21.15.28:3301,v1:172.21.15.28:6790],172.21.15.82" 2024-09-18T04:36:47.860 INFO:teuthology.orchestra.run.smithi028.stdout: mon client directed command retry = 5 2024-09-18T04:36:47.860 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on pool no app = False 2024-09-18T04:36:47.860 INFO:teuthology.orchestra.run.smithi028.stdout: ms inject socket failures = 5000 2024-09-18T04:36:47.861 INFO:teuthology.orchestra.run.smithi028.stdout: fsid = be6146ba-a991-4f6e-b359-99cbedc9f9b1 2024-09-18T04:36:47.861 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:36:47.861 INFO:teuthology.orchestra.run.smithi028.stdout:[osd] 2024-09-18T04:36:47.861 INFO:teuthology.orchestra.run.smithi028.stdout: osd journal size = 100 2024-09-18T04:36:47.861 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:36:47.861 INFO:teuthology.orchestra.run.smithi028.stdout: osd scrub load threshold = 5.0 2024-09-18T04:36:47.861 INFO:teuthology.orchestra.run.smithi028.stdout: osd scrub max interval = 600 2024-09-18T04:36:47.861 INFO:teuthology.orchestra.run.smithi028.stdout: osd mclock profile = high_recovery_ops 2024-09-18T04:36:47.861 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:36:47.861 INFO:teuthology.orchestra.run.smithi028.stdout: osd recover clone overlap = true 2024-09-18T04:36:47.861 INFO:teuthology.orchestra.run.smithi028.stdout: osd recovery max chunk = 1048576 2024-09-18T04:36:47.861 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:36:47.861 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug shutdown = true 2024-09-18T04:36:47.861 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug op order = true 2024-09-18T04:36:47.862 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug verify stray on activate = true 2024-09-18T04:36:47.862 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:36:47.862 INFO:teuthology.orchestra.run.smithi028.stdout: osd open classes on start = true 2024-09-18T04:36:47.862 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug pg log writeout = true 2024-09-18T04:36:47.862 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:36:47.862 INFO:teuthology.orchestra.run.smithi028.stdout: osd deep scrub update digest min age = 30 2024-09-18T04:36:47.862 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:36:47.862 INFO:teuthology.orchestra.run.smithi028.stdout: osd map max advance = 10 2024-09-18T04:36:47.862 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:36:47.862 INFO:teuthology.orchestra.run.smithi028.stdout: journal zero on create = true 2024-09-18T04:36:47.862 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:36:47.862 INFO:teuthology.orchestra.run.smithi028.stdout: filestore ondisk finisher threads = 3 2024-09-18T04:36:47.862 INFO:teuthology.orchestra.run.smithi028.stdout: filestore apply finisher threads = 3 2024-09-18T04:36:47.862 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:36:47.863 INFO:teuthology.orchestra.run.smithi028.stdout: bdev debug aio = true 2024-09-18T04:36:47.863 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug misdirected ops = true 2024-09-18T04:36:47.863 INFO:teuthology.orchestra.run.smithi028.stdout: bluestore block size = 96636764160 2024-09-18T04:36:47.863 INFO:teuthology.orchestra.run.smithi028.stdout: bluestore compression algorithm = snappy 2024-09-18T04:36:47.863 INFO:teuthology.orchestra.run.smithi028.stdout: bluestore compression mode = aggressive 2024-09-18T04:36:47.863 INFO:teuthology.orchestra.run.smithi028.stdout: bluestore fsck on mount = True 2024-09-18T04:36:47.863 INFO:teuthology.orchestra.run.smithi028.stdout: debug bluefs = 1/20 2024-09-18T04:36:47.863 INFO:teuthology.orchestra.run.smithi028.stdout: debug bluestore = 1/20 2024-09-18T04:36:47.863 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-09-18T04:36:47.863 INFO:teuthology.orchestra.run.smithi028.stdout: debug osd = 20 2024-09-18T04:36:47.863 INFO:teuthology.orchestra.run.smithi028.stdout: debug rocksdb = 4/10 2024-09-18T04:36:47.863 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd backfillfull_ratio = 0.85 2024-09-18T04:36:47.863 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd full ratio = 0.9 2024-09-18T04:36:47.863 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd nearfull ratio = 0.8 2024-09-18T04:36:47.864 INFO:teuthology.orchestra.run.smithi028.stdout: osd blocked scrub grace period = 3600 2024-09-18T04:36:47.864 INFO:teuthology.orchestra.run.smithi028.stdout: osd failsafe full ratio = 0.95 2024-09-18T04:36:47.864 INFO:teuthology.orchestra.run.smithi028.stdout: osd max markdown count = 1000 2024-09-18T04:36:47.864 INFO:teuthology.orchestra.run.smithi028.stdout: osd objectstore = bluestore 2024-09-18T04:36:47.864 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:36:47.864 INFO:teuthology.orchestra.run.smithi028.stdout:[mgr] 2024-09-18T04:36:47.864 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-09-18T04:36:47.864 INFO:teuthology.orchestra.run.smithi028.stdout: debug mgr = 20 2024-09-18T04:36:47.864 INFO:teuthology.orchestra.run.smithi028.stdout: debug mon = 20 2024-09-18T04:36:47.864 INFO:teuthology.orchestra.run.smithi028.stdout: debug auth = 20 2024-09-18T04:36:47.864 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min pgs per osd = 4 2024-09-18T04:36:47.864 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min bytes per osd = 10 2024-09-18T04:36:47.864 INFO:teuthology.orchestra.run.smithi028.stdout: mgr/telemetry/nag = false 2024-09-18T04:36:47.864 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:36:47.865 INFO:teuthology.orchestra.run.smithi028.stdout:[mon] 2024-09-18T04:36:47.865 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-09-18T04:36:47.865 INFO:teuthology.orchestra.run.smithi028.stdout: debug mon = 20 2024-09-18T04:36:47.865 INFO:teuthology.orchestra.run.smithi028.stdout: debug paxos = 20 2024-09-18T04:36:47.865 INFO:teuthology.orchestra.run.smithi028.stdout: debug auth = 20 2024-09-18T04:36:47.865 INFO:teuthology.orchestra.run.smithi028.stdout: mon data avail warn = 5 2024-09-18T04:36:47.865 INFO:teuthology.orchestra.run.smithi028.stdout: mon mgr mkfs grace = 240 2024-09-18T04:36:47.865 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min pgs per osd = 4 2024-09-18T04:36:47.865 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd reporter subtree level = osd 2024-09-18T04:36:47.865 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd prime pg temp = true 2024-09-18T04:36:47.865 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min bytes per osd = 10 2024-09-18T04:36:47.865 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:36:47.865 INFO:teuthology.orchestra.run.smithi028.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-18T04:36:47.865 INFO:teuthology.orchestra.run.smithi028.stdout: auth mon ticket ttl = 660 # 11m 2024-09-18T04:36:47.866 INFO:teuthology.orchestra.run.smithi028.stdout: auth service ticket ttl = 240 # 4m 2024-09-18T04:36:47.866 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:36:47.866 INFO:teuthology.orchestra.run.smithi028.stdout: # don't complain about insecure global_id in the test suite 2024-09-18T04:36:47.866 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-18T04:36:47.866 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-18T04:36:47.866 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:36:47.866 INFO:teuthology.orchestra.run.smithi028.stdout: # 1m isn't quite enough 2024-09-18T04:36:47.866 INFO:teuthology.orchestra.run.smithi028.stdout: mon_down_mkfs_grace = 2m 2024-09-18T04:36:47.866 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:36:47.867 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_filestore_osds = false 2024-09-18T04:36:47.867 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:36:47.867 INFO:teuthology.orchestra.run.smithi028.stdout:[client] 2024-09-18T04:36:47.867 INFO:teuthology.orchestra.run.smithi028.stdout: rgw cache enabled = true 2024-09-18T04:36:47.867 INFO:teuthology.orchestra.run.smithi028.stdout: rgw enable ops log = true 2024-09-18T04:36:47.867 INFO:teuthology.orchestra.run.smithi028.stdout: rgw enable usage log = true 2024-09-18T04:36:47.867 INFO:teuthology.orchestra.run.smithi028.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-18T04:36:47.867 INFO:teuthology.orchestra.run.smithi028.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-18T04:36:47.867 INFO:teuthology.orchestra.run.smithi028.stdout:[mon.a] 2024-09-18T04:36:47.867 INFO:teuthology.orchestra.run.smithi028.stdout:[mon.c] 2024-09-18T04:36:47.867 INFO:teuthology.orchestra.run.smithi028.stdout:[mon.b] 2024-09-18T04:36:47.868 INFO:teuthology.orchestra.run.smithi082.stdout:[global] 2024-09-18T04:36:47.868 INFO:teuthology.orchestra.run.smithi082.stdout: chdir = "" 2024-09-18T04:36:47.868 INFO:teuthology.orchestra.run.smithi082.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-18T04:36:47.868 INFO:teuthology.orchestra.run.smithi082.stdout: auth supported = cephx 2024-09-18T04:36:47.868 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-18T04:36:47.869 INFO:teuthology.orchestra.run.smithi082.stdout: filestore xattr use omap = true 2024-09-18T04:36:47.869 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-18T04:36:47.869 INFO:teuthology.orchestra.run.smithi082.stdout: mon clock drift allowed = 1.000 2024-09-18T04:36:47.869 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-18T04:36:47.869 INFO:teuthology.orchestra.run.smithi082.stdout: osd crush chooseleaf type = 0 2024-09-18T04:36:47.869 INFO:teuthology.orchestra.run.smithi082.stdout: auth debug = true 2024-09-18T04:36:47.869 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-18T04:36:47.869 INFO:teuthology.orchestra.run.smithi082.stdout: ms die on old message = true 2024-09-18T04:36:47.869 INFO:teuthology.orchestra.run.smithi082.stdout: ms die on bug = true 2024-09-18T04:36:47.869 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-18T04:36:47.869 INFO:teuthology.orchestra.run.smithi082.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-18T04:36:47.869 INFO:teuthology.orchestra.run.smithi082.stdout: mon pg warn max object skew = 0 2024-09-18T04:36:47.869 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-18T04:36:47.869 INFO:teuthology.orchestra.run.smithi082.stdout: # disable pg_autoscaler by default for new pools 2024-09-18T04:36:47.870 INFO:teuthology.orchestra.run.smithi082.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-18T04:36:47.870 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-18T04:36:47.870 INFO:teuthology.orchestra.run.smithi082.stdout: osd pool default size = 2 2024-09-18T04:36:47.870 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-18T04:36:47.870 INFO:teuthology.orchestra.run.smithi082.stdout: mon osd allow primary affinity = true 2024-09-18T04:36:47.870 INFO:teuthology.orchestra.run.smithi082.stdout: mon osd allow pg remap = true 2024-09-18T04:36:47.870 INFO:teuthology.orchestra.run.smithi082.stdout: mon warn on legacy crush tunables = false 2024-09-18T04:36:47.870 INFO:teuthology.orchestra.run.smithi082.stdout: mon warn on crush straw calc version zero = false 2024-09-18T04:36:47.870 INFO:teuthology.orchestra.run.smithi082.stdout: mon warn on no sortbitwise = false 2024-09-18T04:36:47.870 INFO:teuthology.orchestra.run.smithi082.stdout: mon warn on osd down out interval zero = false 2024-09-18T04:36:47.870 INFO:teuthology.orchestra.run.smithi082.stdout: mon warn on too few osds = false 2024-09-18T04:36:47.870 INFO:teuthology.orchestra.run.smithi082.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-18T04:36:47.870 INFO:teuthology.orchestra.run.smithi082.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-18T04:36:47.870 INFO:teuthology.orchestra.run.smithi082.stdout: mon_allow_pool_size_one = true 2024-09-18T04:36:47.871 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-18T04:36:47.871 INFO:teuthology.orchestra.run.smithi082.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-18T04:36:47.871 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-18T04:36:47.871 INFO:teuthology.orchestra.run.smithi082.stdout: osd default data pool replay window = 5 2024-09-18T04:36:47.871 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-18T04:36:47.871 INFO:teuthology.orchestra.run.smithi082.stdout: mon allow pool delete = true 2024-09-18T04:36:47.871 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-18T04:36:47.871 INFO:teuthology.orchestra.run.smithi082.stdout: mon cluster log file level = debug 2024-09-18T04:36:47.871 INFO:teuthology.orchestra.run.smithi082.stdout: debug asserts on shutdown = true 2024-09-18T04:36:47.871 INFO:teuthology.orchestra.run.smithi082.stdout: mon health detail to clog = false 2024-09-18T04:36:47.871 INFO:teuthology.orchestra.run.smithi082.stdout: mon host = "172.21.15.28,[v2:172.21.15.28:3301,v1:172.21.15.28:6790],172.21.15.82" 2024-09-18T04:36:47.871 INFO:teuthology.orchestra.run.smithi082.stdout: mon client directed command retry = 5 2024-09-18T04:36:47.871 INFO:teuthology.orchestra.run.smithi082.stdout: mon warn on pool no app = False 2024-09-18T04:36:47.871 INFO:teuthology.orchestra.run.smithi082.stdout: ms inject socket failures = 5000 2024-09-18T04:36:47.871 INFO:teuthology.orchestra.run.smithi082.stdout: fsid = be6146ba-a991-4f6e-b359-99cbedc9f9b1 2024-09-18T04:36:47.872 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-18T04:36:47.872 INFO:teuthology.orchestra.run.smithi082.stdout:[osd] 2024-09-18T04:36:47.872 INFO:teuthology.orchestra.run.smithi082.stdout: osd journal size = 100 2024-09-18T04:36:47.872 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-18T04:36:47.872 INFO:teuthology.orchestra.run.smithi082.stdout: osd scrub load threshold = 5.0 2024-09-18T04:36:47.872 INFO:teuthology.orchestra.run.smithi082.stdout: osd scrub max interval = 600 2024-09-18T04:36:47.872 INFO:teuthology.orchestra.run.smithi082.stdout: osd mclock profile = high_recovery_ops 2024-09-18T04:36:47.872 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-18T04:36:47.872 INFO:teuthology.orchestra.run.smithi082.stdout: osd recover clone overlap = true 2024-09-18T04:36:47.872 INFO:teuthology.orchestra.run.smithi082.stdout: osd recovery max chunk = 1048576 2024-09-18T04:36:47.872 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-18T04:36:47.872 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug shutdown = true 2024-09-18T04:36:47.872 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug op order = true 2024-09-18T04:36:47.873 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug verify stray on activate = true 2024-09-18T04:36:47.873 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-18T04:36:47.873 INFO:teuthology.orchestra.run.smithi082.stdout: osd open classes on start = true 2024-09-18T04:36:47.873 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug pg log writeout = true 2024-09-18T04:36:47.873 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-18T04:36:47.873 INFO:teuthology.orchestra.run.smithi082.stdout: osd deep scrub update digest min age = 30 2024-09-18T04:36:47.873 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-18T04:36:47.873 INFO:teuthology.orchestra.run.smithi082.stdout: osd map max advance = 10 2024-09-18T04:36:47.873 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-18T04:36:47.873 INFO:teuthology.orchestra.run.smithi082.stdout: journal zero on create = true 2024-09-18T04:36:47.873 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-18T04:36:47.873 INFO:teuthology.orchestra.run.smithi082.stdout: filestore ondisk finisher threads = 3 2024-09-18T04:36:47.873 INFO:teuthology.orchestra.run.smithi082.stdout: filestore apply finisher threads = 3 2024-09-18T04:36:47.873 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-18T04:36:47.874 INFO:teuthology.orchestra.run.smithi082.stdout: bdev debug aio = true 2024-09-18T04:36:47.874 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug misdirected ops = true 2024-09-18T04:36:47.874 INFO:teuthology.orchestra.run.smithi082.stdout: bluestore block size = 96636764160 2024-09-18T04:36:47.874 INFO:teuthology.orchestra.run.smithi082.stdout: bluestore compression algorithm = snappy 2024-09-18T04:36:47.874 INFO:teuthology.orchestra.run.smithi082.stdout: bluestore compression mode = aggressive 2024-09-18T04:36:47.874 INFO:teuthology.orchestra.run.smithi082.stdout: bluestore fsck on mount = True 2024-09-18T04:36:47.874 INFO:teuthology.orchestra.run.smithi082.stdout: debug bluefs = 1/20 2024-09-18T04:36:47.874 INFO:teuthology.orchestra.run.smithi082.stdout: debug bluestore = 1/20 2024-09-18T04:36:47.874 INFO:teuthology.orchestra.run.smithi082.stdout: debug ms = 1 2024-09-18T04:36:47.874 INFO:teuthology.orchestra.run.smithi082.stdout: debug osd = 20 2024-09-18T04:36:47.874 INFO:teuthology.orchestra.run.smithi082.stdout: debug rocksdb = 4/10 2024-09-18T04:36:47.874 INFO:teuthology.orchestra.run.smithi082.stdout: mon osd backfillfull_ratio = 0.85 2024-09-18T04:36:47.874 INFO:teuthology.orchestra.run.smithi082.stdout: mon osd full ratio = 0.9 2024-09-18T04:36:47.874 INFO:teuthology.orchestra.run.smithi082.stdout: mon osd nearfull ratio = 0.8 2024-09-18T04:36:47.875 INFO:teuthology.orchestra.run.smithi082.stdout: osd blocked scrub grace period = 3600 2024-09-18T04:36:47.875 INFO:teuthology.orchestra.run.smithi082.stdout: osd failsafe full ratio = 0.95 2024-09-18T04:36:47.875 INFO:teuthology.orchestra.run.smithi082.stdout: osd max markdown count = 1000 2024-09-18T04:36:47.875 INFO:teuthology.orchestra.run.smithi082.stdout: osd objectstore = bluestore 2024-09-18T04:36:47.875 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-18T04:36:47.875 INFO:teuthology.orchestra.run.smithi082.stdout:[mgr] 2024-09-18T04:36:47.875 INFO:teuthology.orchestra.run.smithi082.stdout: debug ms = 1 2024-09-18T04:36:47.875 INFO:teuthology.orchestra.run.smithi082.stdout: debug mgr = 20 2024-09-18T04:36:47.875 INFO:teuthology.orchestra.run.smithi082.stdout: debug mon = 20 2024-09-18T04:36:47.875 INFO:teuthology.orchestra.run.smithi082.stdout: debug auth = 20 2024-09-18T04:36:47.875 INFO:teuthology.orchestra.run.smithi082.stdout: mon reweight min pgs per osd = 4 2024-09-18T04:36:47.875 INFO:teuthology.orchestra.run.smithi082.stdout: mon reweight min bytes per osd = 10 2024-09-18T04:36:47.875 INFO:teuthology.orchestra.run.smithi082.stdout: mgr/telemetry/nag = false 2024-09-18T04:36:47.875 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-18T04:36:47.876 INFO:teuthology.orchestra.run.smithi082.stdout:[mon] 2024-09-18T04:36:47.876 INFO:teuthology.orchestra.run.smithi082.stdout: debug ms = 1 2024-09-18T04:36:47.876 INFO:teuthology.orchestra.run.smithi082.stdout: debug mon = 20 2024-09-18T04:36:47.876 INFO:teuthology.orchestra.run.smithi082.stdout: debug paxos = 20 2024-09-18T04:36:47.876 INFO:teuthology.orchestra.run.smithi082.stdout: debug auth = 20 2024-09-18T04:36:47.876 INFO:teuthology.orchestra.run.smithi082.stdout: mon data avail warn = 5 2024-09-18T04:36:47.876 INFO:teuthology.orchestra.run.smithi082.stdout: mon mgr mkfs grace = 240 2024-09-18T04:36:47.876 INFO:teuthology.orchestra.run.smithi082.stdout: mon reweight min pgs per osd = 4 2024-09-18T04:36:47.876 INFO:teuthology.orchestra.run.smithi082.stdout: mon osd reporter subtree level = osd 2024-09-18T04:36:47.876 INFO:teuthology.orchestra.run.smithi082.stdout: mon osd prime pg temp = true 2024-09-18T04:36:47.877 INFO:teuthology.orchestra.run.smithi082.stdout: mon reweight min bytes per osd = 10 2024-09-18T04:36:47.877 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-18T04:36:47.877 INFO:teuthology.orchestra.run.smithi082.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-18T04:36:47.877 INFO:teuthology.orchestra.run.smithi082.stdout: auth mon ticket ttl = 660 # 11m 2024-09-18T04:36:47.877 INFO:teuthology.orchestra.run.smithi082.stdout: auth service ticket ttl = 240 # 4m 2024-09-18T04:36:47.877 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-18T04:36:47.877 INFO:teuthology.orchestra.run.smithi082.stdout: # don't complain about insecure global_id in the test suite 2024-09-18T04:36:47.877 INFO:teuthology.orchestra.run.smithi082.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-18T04:36:47.877 INFO:teuthology.orchestra.run.smithi082.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-18T04:36:47.877 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-18T04:36:47.877 INFO:teuthology.orchestra.run.smithi082.stdout: # 1m isn't quite enough 2024-09-18T04:36:47.877 INFO:teuthology.orchestra.run.smithi082.stdout: mon_down_mkfs_grace = 2m 2024-09-18T04:36:47.878 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-18T04:36:47.878 INFO:teuthology.orchestra.run.smithi082.stdout: mon_warn_on_filestore_osds = false 2024-09-18T04:36:47.878 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-09-18T04:36:47.878 INFO:teuthology.orchestra.run.smithi082.stdout:[client] 2024-09-18T04:36:47.878 INFO:teuthology.orchestra.run.smithi082.stdout: rgw cache enabled = true 2024-09-18T04:36:47.878 INFO:teuthology.orchestra.run.smithi082.stdout: rgw enable ops log = true 2024-09-18T04:36:47.878 INFO:teuthology.orchestra.run.smithi082.stdout: rgw enable usage log = true 2024-09-18T04:36:47.878 INFO:teuthology.orchestra.run.smithi082.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-18T04:36:47.878 INFO:teuthology.orchestra.run.smithi082.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-18T04:36:47.879 INFO:teuthology.orchestra.run.smithi082.stdout:[mon.a] 2024-09-18T04:36:47.879 INFO:teuthology.orchestra.run.smithi082.stdout:[mon.c] 2024-09-18T04:36:47.879 INFO:teuthology.orchestra.run.smithi082.stdout:[mon.b] 2024-09-18T04:36:47.879 INFO:teuthology.orchestra.run.smithi115.stdout:[global] 2024-09-18T04:36:47.879 INFO:teuthology.orchestra.run.smithi115.stdout: chdir = "" 2024-09-18T04:36:47.879 INFO:teuthology.orchestra.run.smithi115.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-18T04:36:47.879 INFO:teuthology.orchestra.run.smithi115.stdout: auth supported = cephx 2024-09-18T04:36:47.879 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T04:36:47.880 INFO:teuthology.orchestra.run.smithi115.stdout: filestore xattr use omap = true 2024-09-18T04:36:47.880 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T04:36:47.880 INFO:teuthology.orchestra.run.smithi115.stdout: mon clock drift allowed = 1.000 2024-09-18T04:36:47.880 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T04:36:47.880 INFO:teuthology.orchestra.run.smithi115.stdout: osd crush chooseleaf type = 0 2024-09-18T04:36:47.880 INFO:teuthology.orchestra.run.smithi115.stdout: auth debug = true 2024-09-18T04:36:47.880 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T04:36:47.880 INFO:teuthology.orchestra.run.smithi115.stdout: ms die on old message = true 2024-09-18T04:36:47.880 INFO:teuthology.orchestra.run.smithi115.stdout: ms die on bug = true 2024-09-18T04:36:47.880 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T04:36:47.880 INFO:teuthology.orchestra.run.smithi115.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-18T04:36:47.881 INFO:teuthology.orchestra.run.smithi115.stdout: mon pg warn max object skew = 0 2024-09-18T04:36:47.881 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T04:36:47.881 INFO:teuthology.orchestra.run.smithi115.stdout: # disable pg_autoscaler by default for new pools 2024-09-18T04:36:47.881 INFO:teuthology.orchestra.run.smithi115.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-18T04:36:47.881 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T04:36:47.881 INFO:teuthology.orchestra.run.smithi115.stdout: osd pool default size = 2 2024-09-18T04:36:47.881 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T04:36:47.881 INFO:teuthology.orchestra.run.smithi115.stdout: mon osd allow primary affinity = true 2024-09-18T04:36:47.881 INFO:teuthology.orchestra.run.smithi115.stdout: mon osd allow pg remap = true 2024-09-18T04:36:47.881 INFO:teuthology.orchestra.run.smithi115.stdout: mon warn on legacy crush tunables = false 2024-09-18T04:36:47.882 INFO:teuthology.orchestra.run.smithi115.stdout: mon warn on crush straw calc version zero = false 2024-09-18T04:36:47.882 INFO:teuthology.orchestra.run.smithi115.stdout: mon warn on no sortbitwise = false 2024-09-18T04:36:47.882 INFO:teuthology.orchestra.run.smithi115.stdout: mon warn on osd down out interval zero = false 2024-09-18T04:36:47.882 INFO:teuthology.orchestra.run.smithi115.stdout: mon warn on too few osds = false 2024-09-18T04:36:47.882 INFO:teuthology.orchestra.run.smithi115.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-18T04:36:47.882 INFO:teuthology.orchestra.run.smithi115.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-18T04:36:47.882 INFO:teuthology.orchestra.run.smithi115.stdout: mon_allow_pool_size_one = true 2024-09-18T04:36:47.882 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T04:36:47.882 INFO:teuthology.orchestra.run.smithi115.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-18T04:36:47.882 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T04:36:47.882 INFO:teuthology.orchestra.run.smithi115.stdout: osd default data pool replay window = 5 2024-09-18T04:36:47.883 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T04:36:47.883 INFO:teuthology.orchestra.run.smithi115.stdout: mon allow pool delete = true 2024-09-18T04:36:47.883 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T04:36:47.883 INFO:teuthology.orchestra.run.smithi115.stdout: mon cluster log file level = debug 2024-09-18T04:36:47.883 INFO:teuthology.orchestra.run.smithi115.stdout: debug asserts on shutdown = true 2024-09-18T04:36:47.883 INFO:teuthology.orchestra.run.smithi115.stdout: mon health detail to clog = false 2024-09-18T04:36:47.883 INFO:teuthology.orchestra.run.smithi115.stdout: mon host = "172.21.15.28,[v2:172.21.15.28:3301,v1:172.21.15.28:6790],172.21.15.82" 2024-09-18T04:36:47.883 INFO:teuthology.orchestra.run.smithi115.stdout: mon client directed command retry = 5 2024-09-18T04:36:47.884 INFO:teuthology.orchestra.run.smithi115.stdout: mon warn on pool no app = False 2024-09-18T04:36:47.884 INFO:teuthology.orchestra.run.smithi115.stdout: ms inject socket failures = 5000 2024-09-18T04:36:47.884 INFO:teuthology.orchestra.run.smithi115.stdout: fsid = be6146ba-a991-4f6e-b359-99cbedc9f9b1 2024-09-18T04:36:47.884 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T04:36:47.884 INFO:teuthology.orchestra.run.smithi115.stdout:[osd] 2024-09-18T04:36:47.884 INFO:teuthology.orchestra.run.smithi115.stdout: osd journal size = 100 2024-09-18T04:36:47.884 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T04:36:47.884 INFO:teuthology.orchestra.run.smithi115.stdout: osd scrub load threshold = 5.0 2024-09-18T04:36:47.884 INFO:teuthology.orchestra.run.smithi115.stdout: osd scrub max interval = 600 2024-09-18T04:36:47.884 INFO:teuthology.orchestra.run.smithi115.stdout: osd mclock profile = high_recovery_ops 2024-09-18T04:36:47.884 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T04:36:47.884 INFO:teuthology.orchestra.run.smithi115.stdout: osd recover clone overlap = true 2024-09-18T04:36:47.885 INFO:teuthology.orchestra.run.smithi115.stdout: osd recovery max chunk = 1048576 2024-09-18T04:36:47.885 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T04:36:47.885 INFO:teuthology.orchestra.run.smithi115.stdout: osd debug shutdown = true 2024-09-18T04:36:47.885 INFO:teuthology.orchestra.run.smithi115.stdout: osd debug op order = true 2024-09-18T04:36:47.885 INFO:teuthology.orchestra.run.smithi115.stdout: osd debug verify stray on activate = true 2024-09-18T04:36:47.885 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T04:36:47.885 INFO:teuthology.orchestra.run.smithi115.stdout: osd open classes on start = true 2024-09-18T04:36:47.885 INFO:teuthology.orchestra.run.smithi115.stdout: osd debug pg log writeout = true 2024-09-18T04:36:47.885 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T04:36:47.885 INFO:teuthology.orchestra.run.smithi115.stdout: osd deep scrub update digest min age = 30 2024-09-18T04:36:47.885 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T04:36:47.885 INFO:teuthology.orchestra.run.smithi115.stdout: osd map max advance = 10 2024-09-18T04:36:47.885 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T04:36:47.886 INFO:teuthology.orchestra.run.smithi115.stdout: journal zero on create = true 2024-09-18T04:36:47.886 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T04:36:47.886 INFO:teuthology.orchestra.run.smithi115.stdout: filestore ondisk finisher threads = 3 2024-09-18T04:36:47.886 INFO:teuthology.orchestra.run.smithi115.stdout: filestore apply finisher threads = 3 2024-09-18T04:36:47.886 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T04:36:47.886 INFO:teuthology.orchestra.run.smithi115.stdout: bdev debug aio = true 2024-09-18T04:36:47.886 INFO:teuthology.orchestra.run.smithi115.stdout: osd debug misdirected ops = true 2024-09-18T04:36:47.886 INFO:teuthology.orchestra.run.smithi115.stdout: bluestore block size = 96636764160 2024-09-18T04:36:47.886 INFO:teuthology.orchestra.run.smithi115.stdout: bluestore compression algorithm = snappy 2024-09-18T04:36:47.886 INFO:teuthology.orchestra.run.smithi115.stdout: bluestore compression mode = aggressive 2024-09-18T04:36:47.886 INFO:teuthology.orchestra.run.smithi115.stdout: bluestore fsck on mount = True 2024-09-18T04:36:47.886 INFO:teuthology.orchestra.run.smithi115.stdout: debug bluefs = 1/20 2024-09-18T04:36:47.886 INFO:teuthology.orchestra.run.smithi115.stdout: debug bluestore = 1/20 2024-09-18T04:36:47.887 INFO:teuthology.orchestra.run.smithi115.stdout: debug ms = 1 2024-09-18T04:36:47.887 INFO:teuthology.orchestra.run.smithi115.stdout: debug osd = 20 2024-09-18T04:36:47.887 INFO:teuthology.orchestra.run.smithi115.stdout: debug rocksdb = 4/10 2024-09-18T04:36:47.887 INFO:teuthology.orchestra.run.smithi115.stdout: mon osd backfillfull_ratio = 0.85 2024-09-18T04:36:47.887 INFO:teuthology.orchestra.run.smithi115.stdout: mon osd full ratio = 0.9 2024-09-18T04:36:47.887 INFO:teuthology.orchestra.run.smithi115.stdout: mon osd nearfull ratio = 0.8 2024-09-18T04:36:47.887 INFO:teuthology.orchestra.run.smithi115.stdout: osd blocked scrub grace period = 3600 2024-09-18T04:36:47.887 INFO:teuthology.orchestra.run.smithi115.stdout: osd failsafe full ratio = 0.95 2024-09-18T04:36:47.887 INFO:teuthology.orchestra.run.smithi115.stdout: osd max markdown count = 1000 2024-09-18T04:36:47.887 INFO:teuthology.orchestra.run.smithi115.stdout: osd objectstore = bluestore 2024-09-18T04:36:47.887 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T04:36:47.887 INFO:teuthology.orchestra.run.smithi115.stdout:[mgr] 2024-09-18T04:36:47.887 INFO:teuthology.orchestra.run.smithi115.stdout: debug ms = 1 2024-09-18T04:36:47.887 INFO:teuthology.orchestra.run.smithi115.stdout: debug mgr = 20 2024-09-18T04:36:47.888 INFO:teuthology.orchestra.run.smithi115.stdout: debug mon = 20 2024-09-18T04:36:47.888 INFO:teuthology.orchestra.run.smithi115.stdout: debug auth = 20 2024-09-18T04:36:47.888 INFO:teuthology.orchestra.run.smithi115.stdout: mon reweight min pgs per osd = 4 2024-09-18T04:36:47.888 INFO:teuthology.orchestra.run.smithi115.stdout: mon reweight min bytes per osd = 10 2024-09-18T04:36:47.888 INFO:teuthology.orchestra.run.smithi115.stdout: mgr/telemetry/nag = false 2024-09-18T04:36:47.888 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T04:36:47.888 INFO:teuthology.orchestra.run.smithi115.stdout:[mon] 2024-09-18T04:36:47.888 INFO:teuthology.orchestra.run.smithi115.stdout: debug ms = 1 2024-09-18T04:36:47.888 INFO:teuthology.orchestra.run.smithi115.stdout: debug mon = 20 2024-09-18T04:36:47.888 INFO:teuthology.orchestra.run.smithi115.stdout: debug paxos = 20 2024-09-18T04:36:47.888 INFO:teuthology.orchestra.run.smithi115.stdout: debug auth = 20 2024-09-18T04:36:47.888 INFO:teuthology.orchestra.run.smithi115.stdout: mon data avail warn = 5 2024-09-18T04:36:47.888 INFO:teuthology.orchestra.run.smithi115.stdout: mon mgr mkfs grace = 240 2024-09-18T04:36:47.888 INFO:teuthology.orchestra.run.smithi115.stdout: mon reweight min pgs per osd = 4 2024-09-18T04:36:47.889 INFO:teuthology.orchestra.run.smithi115.stdout: mon osd reporter subtree level = osd 2024-09-18T04:36:47.889 INFO:teuthology.orchestra.run.smithi115.stdout: mon osd prime pg temp = true 2024-09-18T04:36:47.889 INFO:teuthology.orchestra.run.smithi115.stdout: mon reweight min bytes per osd = 10 2024-09-18T04:36:47.889 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T04:36:47.889 INFO:teuthology.orchestra.run.smithi115.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-18T04:36:47.889 INFO:teuthology.orchestra.run.smithi115.stdout: auth mon ticket ttl = 660 # 11m 2024-09-18T04:36:47.889 INFO:teuthology.orchestra.run.smithi115.stdout: auth service ticket ttl = 240 # 4m 2024-09-18T04:36:47.889 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T04:36:47.889 INFO:teuthology.orchestra.run.smithi115.stdout: # don't complain about insecure global_id in the test suite 2024-09-18T04:36:47.889 INFO:teuthology.orchestra.run.smithi115.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-18T04:36:47.889 INFO:teuthology.orchestra.run.smithi115.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-18T04:36:47.889 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T04:36:47.889 INFO:teuthology.orchestra.run.smithi115.stdout: # 1m isn't quite enough 2024-09-18T04:36:47.889 INFO:teuthology.orchestra.run.smithi115.stdout: mon_down_mkfs_grace = 2m 2024-09-18T04:36:47.889 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T04:36:47.890 INFO:teuthology.orchestra.run.smithi115.stdout: mon_warn_on_filestore_osds = false 2024-09-18T04:36:47.890 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-18T04:36:47.890 INFO:teuthology.orchestra.run.smithi115.stdout:[client] 2024-09-18T04:36:47.890 INFO:teuthology.orchestra.run.smithi115.stdout: rgw cache enabled = true 2024-09-18T04:36:47.890 INFO:teuthology.orchestra.run.smithi115.stdout: rgw enable ops log = true 2024-09-18T04:36:47.890 INFO:teuthology.orchestra.run.smithi115.stdout: rgw enable usage log = true 2024-09-18T04:36:47.890 INFO:teuthology.orchestra.run.smithi115.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-18T04:36:47.890 INFO:teuthology.orchestra.run.smithi115.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-18T04:36:47.890 INFO:teuthology.orchestra.run.smithi115.stdout:[mon.a] 2024-09-18T04:36:47.890 INFO:teuthology.orchestra.run.smithi115.stdout:[mon.c] 2024-09-18T04:36:47.890 INFO:teuthology.orchestra.run.smithi115.stdout:[mon.b] 2024-09-18T04:36:47.892 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-18T04:36:47.893 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-18T04:36:47.990 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-18T04:36:47.990 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-18T04:36:47.990 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-18T04:36:47.998 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-18T04:36:47.998 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-18T04:36:48.004 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi028.front.sepia.ceph.com 2024-09-18T04:36:48.005 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-18T04:36:48.005 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-18T04:36:48.005 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-18T04:36:48.072 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-18T04:36:48.072 DEBUG:teuthology.orchestra.run.smithi028:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-18T04:36:48.082 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi082.front.sepia.ceph.com 2024-09-18T04:36:48.082 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-09-18T04:36:48.082 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-18T04:36:48.082 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-18T04:36:48.111 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-09-18T04:36:48.111 DEBUG:teuthology.orchestra.run.smithi082:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-18T04:36:48.117 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi115.front.sepia.ceph.com 2024-09-18T04:36:48.117 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-18T04:36:48.117 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-18T04:36:48.117 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-18T04:36:48.144 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-18T04:36:48.144 DEBUG:teuthology.orchestra.run.smithi115:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-18T04:36:48.198 INFO:tasks.ceph:Setting up mon nodes... 2024-09-18T04:36:48.198 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-18T04:36:48.199 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-18T04:36:48.251 INFO:teuthology.orchestra.run.smithi082.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-18T04:36:48.260 INFO:tasks.ceph:Setting up mds nodes... 2024-09-18T04:36:48.260 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-18T04:36:48.260 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-18T04:36:48.302 INFO:teuthology.orchestra.run.smithi115.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-18T04:36:48.322 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-18T04:36:48.323 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi028.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi082.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-18T04:36:48.323 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-18T04:36:48.341 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-18T04:36:48.341 INFO:tasks.ceph:role: osd.0 2024-09-18T04:36:48.341 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi028.front.sepia.ceph.com 2024-09-18T04:36:48.342 DEBUG:teuthology.orchestra.run.smithi028:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-18T04:36:48.432 INFO:teuthology.orchestra.run.smithi028.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-18T04:36:48.432 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-18T04:36:48.432 INFO:teuthology.orchestra.run.smithi028.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-18T04:36:48.432 INFO:teuthology.orchestra.run.smithi028.stdout: = reflink=1 2024-09-18T04:36:48.432 INFO:teuthology.orchestra.run.smithi028.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-18T04:36:48.432 INFO:teuthology.orchestra.run.smithi028.stdout: = sunit=0 swidth=0 blks 2024-09-18T04:36:48.432 INFO:teuthology.orchestra.run.smithi028.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-18T04:36:48.432 INFO:teuthology.orchestra.run.smithi028.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-18T04:36:48.432 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-18T04:36:48.433 INFO:teuthology.orchestra.run.smithi028.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-18T04:36:48.433 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi028.front.sepia.ceph.com -o noatime 2024-09-18T04:36:48.434 DEBUG:teuthology.orchestra.run.smithi028:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-18T04:36:48.455 DEBUG:teuthology.orchestra.run.smithi028:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-18T04:36:48.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T04:36:48.463 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: /sbin/restorecon: command not found 2024-09-18T04:36:48.463 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-18T04:36:48.523 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-18T04:36:48.523 INFO:tasks.ceph:role: osd.1 2024-09-18T04:36:48.523 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi028.front.sepia.ceph.com 2024-09-18T04:36:48.523 DEBUG:teuthology.orchestra.run.smithi028:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-18T04:36:48.564 INFO:teuthology.orchestra.run.smithi028.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-18T04:36:48.564 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-18T04:36:48.564 INFO:teuthology.orchestra.run.smithi028.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-18T04:36:48.564 INFO:teuthology.orchestra.run.smithi028.stdout: = reflink=1 2024-09-18T04:36:48.564 INFO:teuthology.orchestra.run.smithi028.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-18T04:36:48.565 INFO:teuthology.orchestra.run.smithi028.stdout: = sunit=0 swidth=0 blks 2024-09-18T04:36:48.565 INFO:teuthology.orchestra.run.smithi028.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-18T04:36:48.565 INFO:teuthology.orchestra.run.smithi028.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-18T04:36:48.565 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-18T04:36:48.565 INFO:teuthology.orchestra.run.smithi028.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-18T04:36:48.566 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi028.front.sepia.ceph.com -o noatime 2024-09-18T04:36:48.566 DEBUG:teuthology.orchestra.run.smithi028:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-18T04:36:48.586 DEBUG:teuthology.orchestra.run.smithi028:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-18T04:36:48.596 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: /sbin/restorecon: command not found 2024-09-18T04:36:48.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T04:36:48.597 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-18T04:36:48.606 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-18T04:36:48.607 INFO:tasks.ceph:role: osd.2 2024-09-18T04:36:48.607 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi028.front.sepia.ceph.com 2024-09-18T04:36:48.607 DEBUG:teuthology.orchestra.run.smithi028:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-18T04:36:48.693 INFO:teuthology.orchestra.run.smithi028.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-18T04:36:48.693 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-18T04:36:48.693 INFO:teuthology.orchestra.run.smithi028.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-18T04:36:48.693 INFO:teuthology.orchestra.run.smithi028.stdout: = reflink=1 2024-09-18T04:36:48.693 INFO:teuthology.orchestra.run.smithi028.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-18T04:36:48.693 INFO:teuthology.orchestra.run.smithi028.stdout: = sunit=0 swidth=0 blks 2024-09-18T04:36:48.693 INFO:teuthology.orchestra.run.smithi028.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-18T04:36:48.694 INFO:teuthology.orchestra.run.smithi028.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-18T04:36:48.694 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-18T04:36:48.694 INFO:teuthology.orchestra.run.smithi028.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-18T04:36:48.695 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi028.front.sepia.ceph.com -o noatime 2024-09-18T04:36:48.695 DEBUG:teuthology.orchestra.run.smithi028:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-18T04:36:48.713 DEBUG:teuthology.orchestra.run.smithi028:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-18T04:36:48.725 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: /sbin/restorecon: command not found 2024-09-18T04:36:48.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T04:36:48.726 DEBUG:teuthology.orchestra.run.smithi028:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-18T04:36:48.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-18T04:36:48.841+0000 7f094340f880 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-18T04:36:48.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-18T04:36:48.841+0000 7f094340f880 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-18T04:36:48.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-18T04:36:48.841+0000 7f094340f880 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-18T04:36:48.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-18T04:36:48.841+0000 7f094340f880 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-18T04:36:48.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-18T04:36:48.841+0000 7f094340f880 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-18T04:36:48.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-18T04:36:48.841+0000 7f094340f880 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-18T04:36:51.853 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-18T04:36:51.869 DEBUG:teuthology.orchestra.run.smithi028:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-18T04:36:51.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-18T04:36:51.905+0000 7efd96444880 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-18T04:36:51.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-18T04:36:51.905+0000 7efd96444880 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-18T04:36:51.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-18T04:36:51.905+0000 7efd96444880 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-18T04:36:51.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-18T04:36:51.905+0000 7efd96444880 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-18T04:36:51.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-18T04:36:51.905+0000 7efd96444880 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-18T04:36:51.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-18T04:36:51.905+0000 7efd96444880 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-18T04:36:54.929 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-18T04:36:54.987 DEBUG:teuthology.orchestra.run.smithi028:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-18T04:36:55.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-18T04:36:55.025+0000 7f0722e59880 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-18T04:36:55.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-18T04:36:55.025+0000 7f0722e59880 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-18T04:36:55.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-18T04:36:55.025+0000 7f0722e59880 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-18T04:36:55.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-18T04:36:55.025+0000 7f0722e59880 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-18T04:36:55.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-18T04:36:55.025+0000 7f0722e59880 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-18T04:36:55.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-18T04:36:55.025+0000 7f0722e59880 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-18T04:36:58.033 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-18T04:36:58.048 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-18T04:36:58.065 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-18T04:36:58.065 INFO:tasks.ceph:role: osd.3 2024-09-18T04:36:58.065 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi082.front.sepia.ceph.com 2024-09-18T04:36:58.065 DEBUG:teuthology.orchestra.run.smithi082:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-18T04:36:58.605 INFO:teuthology.orchestra.run.smithi082.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-18T04:36:58.606 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-18T04:36:58.606 INFO:teuthology.orchestra.run.smithi082.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-18T04:36:58.606 INFO:teuthology.orchestra.run.smithi082.stdout: = reflink=1 2024-09-18T04:36:58.606 INFO:teuthology.orchestra.run.smithi082.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-18T04:36:58.606 INFO:teuthology.orchestra.run.smithi082.stdout: = sunit=0 swidth=0 blks 2024-09-18T04:36:58.606 INFO:teuthology.orchestra.run.smithi082.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-18T04:36:58.606 INFO:teuthology.orchestra.run.smithi082.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-18T04:36:58.606 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-18T04:36:58.606 INFO:teuthology.orchestra.run.smithi082.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-18T04:36:58.608 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi082.front.sepia.ceph.com -o noatime 2024-09-18T04:36:58.608 DEBUG:teuthology.orchestra.run.smithi082:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-09-18T04:36:58.688 DEBUG:teuthology.orchestra.run.smithi082:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-18T04:36:58.699 INFO:teuthology.orchestra.run.smithi082.stderr:sudo: /sbin/restorecon: command not found 2024-09-18T04:36:58.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T04:36:58.700 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-18T04:36:58.714 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-18T04:36:58.714 INFO:tasks.ceph:role: osd.4 2024-09-18T04:36:58.714 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi082.front.sepia.ceph.com 2024-09-18T04:36:58.715 DEBUG:teuthology.orchestra.run.smithi082:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-18T04:36:59.246 INFO:teuthology.orchestra.run.smithi082.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-18T04:36:59.246 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-18T04:36:59.246 INFO:teuthology.orchestra.run.smithi082.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-18T04:36:59.246 INFO:teuthology.orchestra.run.smithi082.stdout: = reflink=1 2024-09-18T04:36:59.246 INFO:teuthology.orchestra.run.smithi082.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-18T04:36:59.246 INFO:teuthology.orchestra.run.smithi082.stdout: = sunit=0 swidth=0 blks 2024-09-18T04:36:59.246 INFO:teuthology.orchestra.run.smithi082.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-18T04:36:59.246 INFO:teuthology.orchestra.run.smithi082.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-18T04:36:59.246 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-18T04:36:59.247 INFO:teuthology.orchestra.run.smithi082.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-18T04:36:59.248 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi082.front.sepia.ceph.com -o noatime 2024-09-18T04:36:59.248 DEBUG:teuthology.orchestra.run.smithi082:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-09-18T04:36:59.297 DEBUG:teuthology.orchestra.run.smithi082:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-18T04:36:59.309 INFO:teuthology.orchestra.run.smithi082.stderr:sudo: /sbin/restorecon: command not found 2024-09-18T04:36:59.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T04:36:59.310 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-18T04:36:59.320 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-18T04:36:59.320 INFO:tasks.ceph:role: osd.5 2024-09-18T04:36:59.320 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi082.front.sepia.ceph.com 2024-09-18T04:36:59.320 DEBUG:teuthology.orchestra.run.smithi082:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-18T04:36:59.825 INFO:teuthology.orchestra.run.smithi082.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-18T04:36:59.825 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-18T04:36:59.825 INFO:teuthology.orchestra.run.smithi082.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-18T04:36:59.825 INFO:teuthology.orchestra.run.smithi082.stdout: = reflink=1 2024-09-18T04:36:59.825 INFO:teuthology.orchestra.run.smithi082.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-18T04:36:59.826 INFO:teuthology.orchestra.run.smithi082.stdout: = sunit=0 swidth=0 blks 2024-09-18T04:36:59.826 INFO:teuthology.orchestra.run.smithi082.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-18T04:36:59.826 INFO:teuthology.orchestra.run.smithi082.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-18T04:36:59.826 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-18T04:36:59.826 INFO:teuthology.orchestra.run.smithi082.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-18T04:36:59.827 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi082.front.sepia.ceph.com -o noatime 2024-09-18T04:36:59.827 DEBUG:teuthology.orchestra.run.smithi082:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2024-09-18T04:36:59.881 DEBUG:teuthology.orchestra.run.smithi082:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-18T04:36:59.892 INFO:teuthology.orchestra.run.smithi082.stderr:sudo: /sbin/restorecon: command not found 2024-09-18T04:36:59.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T04:36:59.893 DEBUG:teuthology.orchestra.run.smithi082:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-18T04:36:59.986 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-18T04:36:59.980+0000 7f8bdd094880 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-18T04:36:59.986 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-18T04:36:59.980+0000 7f8bdd094880 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-18T04:36:59.986 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-18T04:36:59.980+0000 7f8bdd094880 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-18T04:36:59.986 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-18T04:36:59.980+0000 7f8bdd094880 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-18T04:36:59.986 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-18T04:36:59.980+0000 7f8bdd094880 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-18T04:36:59.986 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-18T04:36:59.984+0000 7f8bdd094880 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-18T04:37:02.998 DEBUG:teuthology.orchestra.run.smithi082:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-18T04:37:03.017 DEBUG:teuthology.orchestra.run.smithi082:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-18T04:37:03.059 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-18T04:37:03.056+0000 7fd7c0c4c880 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-18T04:37:03.060 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-18T04:37:03.056+0000 7fd7c0c4c880 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-18T04:37:03.060 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-18T04:37:03.056+0000 7fd7c0c4c880 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-18T04:37:03.060 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-18T04:37:03.056+0000 7fd7c0c4c880 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-18T04:37:03.060 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-18T04:37:03.056+0000 7fd7c0c4c880 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-18T04:37:03.060 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-18T04:37:03.056+0000 7fd7c0c4c880 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-18T04:37:06.069 DEBUG:teuthology.orchestra.run.smithi082:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-18T04:37:06.086 DEBUG:teuthology.orchestra.run.smithi082:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-18T04:37:06.177 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-18T04:37:06.172+0000 7f5688e6f880 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-18T04:37:06.178 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-18T04:37:06.172+0000 7f5688e6f880 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-18T04:37:06.178 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-18T04:37:06.172+0000 7f5688e6f880 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-18T04:37:06.178 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-18T04:37:06.172+0000 7f5688e6f880 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-18T04:37:06.178 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-18T04:37:06.172+0000 7f5688e6f880 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-18T04:37:06.178 INFO:teuthology.orchestra.run.smithi082.stderr:2024-09-18T04:37:06.176+0000 7f5688e6f880 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-18T04:37:09.188 DEBUG:teuthology.orchestra.run.smithi082:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-18T04:37:09.205 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-18T04:37:09.206 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-18T04:37:09.206 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-18T04:37:09.223 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-18T04:37:09.223 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-18T04:37:09.287 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-18T04:37:09.288 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-18T04:37:09.299 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-09-18T04:37:09.299 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-18T04:37:09.315 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-09-18T04:37:09.315 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-18T04:37:09.375 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-09-18T04:37:09.376 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-18T04:37:09.389 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-09-18T04:37:09.389 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-18T04:37:09.401 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-18T04:37:09.401 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-18T04:37:09.409 INFO:tasks.ceph:Adding keys to all mons... 2024-09-18T04:37:09.409 DEBUG:teuthology.orchestra.run.smithi028:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-18T04:37:09.412 DEBUG:teuthology.orchestra.run.smithi082:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-18T04:37:09.424 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.0] 2024-09-18T04:37:09.424 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDgWOpmZvllMhAAeDnxXvbwFL+QTIZ3H0xxqA== 2024-09-18T04:37:09.424 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.1] 2024-09-18T04:37:09.425 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDjWOpmi+NBNhAAkDXy3p4sruBTNk/xn95c5Q== 2024-09-18T04:37:09.425 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.2] 2024-09-18T04:37:09.425 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDnWOpmsXDeARAAcy82n0R5e1rM5JflB+Z8jQ== 2024-09-18T04:37:09.425 INFO:teuthology.orchestra.run.smithi028.stdout:[mgr.x] 2024-09-18T04:37:09.425 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDgWOpmLl7kDhAAEUYevmRJghpVJMnwKKSq2Q== 2024-09-18T04:37:09.425 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.3] 2024-09-18T04:37:09.425 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDrWOpmn7mhOhAAtRdI1FrJCa+1hnlGwK3hog== 2024-09-18T04:37:09.425 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.4] 2024-09-18T04:37:09.425 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDvWOpmC1htAxAA5Yvsohq1TNkoUvXF1BjqoA== 2024-09-18T04:37:09.425 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.5] 2024-09-18T04:37:09.425 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDyWOpmseF0ChAA1tOU0G8f0f7HCuwrnapH7w== 2024-09-18T04:37:09.425 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-09-18T04:37:09.425 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDgWOpmUx3gERAAxhmaJTOveFb+EZwJvyUGCA== 2024-09-18T04:37:09.426 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.0] 2024-09-18T04:37:09.426 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQDgWOpmZvllMhAAeDnxXvbwFL+QTIZ3H0xxqA== 2024-09-18T04:37:09.426 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.1] 2024-09-18T04:37:09.426 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQDjWOpmi+NBNhAAkDXy3p4sruBTNk/xn95c5Q== 2024-09-18T04:37:09.426 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.2] 2024-09-18T04:37:09.426 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQDnWOpmsXDeARAAcy82n0R5e1rM5JflB+Z8jQ== 2024-09-18T04:37:09.427 INFO:teuthology.orchestra.run.smithi082.stdout:[mgr.x] 2024-09-18T04:37:09.427 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQDgWOpmLl7kDhAAEUYevmRJghpVJMnwKKSq2Q== 2024-09-18T04:37:09.427 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.3] 2024-09-18T04:37:09.427 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQDrWOpmn7mhOhAAtRdI1FrJCa+1hnlGwK3hog== 2024-09-18T04:37:09.427 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.4] 2024-09-18T04:37:09.427 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQDvWOpmC1htAxAA5Yvsohq1TNkoUvXF1BjqoA== 2024-09-18T04:37:09.427 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.5] 2024-09-18T04:37:09.427 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQDyWOpmseF0ChAA1tOU0G8f0f7HCuwrnapH7w== 2024-09-18T04:37:09.427 INFO:teuthology.orchestra.run.smithi082.stdout:[client.0] 2024-09-18T04:37:09.427 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQDgWOpmUx3gERAAxhmaJTOveFb+EZwJvyUGCA== 2024-09-18T04:37:09.428 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T04:37:09.471 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T04:37:09.524 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T04:37:09.526 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T04:37:09.577 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T04:37:09.580 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T04:37:09.627 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-18T04:37:09.630 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-18T04:37:09.718 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T04:37:09.720 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T04:37:09.768 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T04:37:09.770 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T04:37:09.817 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T04:37:09.819 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T04:37:09.865 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-18T04:37:09.868 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-18T04:37:09.917 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-18T04:37:09.918 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-18T04:37:09.933 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-18T04:37:10.015 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-18T04:37:10.033 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-18T04:37:10.044 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-18T04:37:10.139 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-18T04:37:10.155 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-18T04:37:10.172 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-18T04:37:10.252 DEBUG:teuthology.orchestra.run.smithi082:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-18T04:37:10.268 DEBUG:teuthology.orchestra.run.smithi028:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-18T04:37:10.270 DEBUG:teuthology.orchestra.run.smithi082:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-18T04:37:10.317 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-18T04:37:10.317 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-18T04:37:10.317 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-18T04:37:10.320 INFO:tasks.ceph.mon.a:Started 2024-09-18T04:37:10.320 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-18T04:37:10.320 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-18T04:37:10.322 INFO:tasks.ceph.mon.c:Started 2024-09-18T04:37:10.322 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-18T04:37:10.322 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-18T04:37:10.324 INFO:tasks.ceph.mon.b:Started 2024-09-18T04:37:10.325 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-18T04:37:10.325 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-18T04:37:10.325 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-18T04:37:10.328 INFO:tasks.ceph.mgr.x:Started 2024-09-18T04:37:10.328 INFO:tasks.ceph:Setting crush tunables to default 2024-09-18T04:37:10.328 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --cluster ceph osd crush tunables default 2024-09-18T04:37:15.800 INFO:teuthology.orchestra.run.smithi028.stderr:adjusted tunables profile to default 2024-09-18T04:37:15.813 INFO:tasks.ceph:check_enable_crimson: False 2024-09-18T04:37:15.813 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-18T04:37:15.814 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-18T04:37:15.814 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-18T04:37:15.828 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-18T04:37:15.829 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-18T04:37:15.838 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-18T04:37:15.838 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-18T04:37:15.901 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-09-18T04:37:15.901 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-18T04:37:15.918 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-09-18T04:37:15.918 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-18T04:37:15.978 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-09-18T04:37:15.978 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-18T04:37:15.993 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd new 6177002d-72e6-4888-bcde-29b6a92d9dc4 0 2024-09-18T04:37:16.204 INFO:teuthology.orchestra.run.smithi082.stdout:0 2024-09-18T04:37:16.216 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd new 4f62cb09-3457-418c-bb69-491d9b70914c 1 2024-09-18T04:37:16.420 INFO:teuthology.orchestra.run.smithi082.stdout:1 2024-09-18T04:37:16.433 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd new 854fe031-3b87-4290-9c2f-3fb196fb806c 2 2024-09-18T04:37:16.645 INFO:teuthology.orchestra.run.smithi082.stdout:2 2024-09-18T04:37:16.652 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-09-18T04:37:16.647+0000 7f2e3f7e82c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-18T04:37:16.659 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd new c2625f98-780b-4f86-bf36-a48d2a227756 3 2024-09-18T04:37:16.856 INFO:teuthology.orchestra.run.smithi082.stdout:3 2024-09-18T04:37:16.870 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd new deb22d86-4081-484d-96eb-9cb9d96b60bc 4 2024-09-18T04:37:17.074 INFO:teuthology.orchestra.run.smithi082.stdout:4 2024-09-18T04:37:17.088 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd new 061d332c-f51b-403d-bb18-eba415c97e7f 5 2024-09-18T04:37:17.289 INFO:teuthology.orchestra.run.smithi082.stdout:5 2024-09-18T04:37:17.307 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-18T04:37:17.308 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-18T04:37:17.310 INFO:tasks.ceph.osd.0:Started 2024-09-18T04:37:17.310 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-18T04:37:17.310 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-18T04:37:17.313 INFO:tasks.ceph.osd.1:Started 2024-09-18T04:37:17.313 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-18T04:37:17.313 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-18T04:37:17.315 INFO:tasks.ceph.osd.2:Started 2024-09-18T04:37:17.315 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-18T04:37:17.315 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-18T04:37:17.318 INFO:tasks.ceph.osd.3:Started 2024-09-18T04:37:17.318 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-18T04:37:17.318 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-18T04:37:17.320 INFO:tasks.ceph.osd.4:Started 2024-09-18T04:37:17.320 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-18T04:37:17.320 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-18T04:37:17.322 INFO:tasks.ceph.osd.5:Started 2024-09-18T04:37:17.323 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T04:37:17.406 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-09-18T04:37:17.399+0000 7f2e3f7e82c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-18T04:37:17.464 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-09-18T04:37:17.459+0000 7f2e3f7e82c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-18T04:37:17.529 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:37:17.529 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":8,"fsid":"be6146ba-a991-4f6e-b359-99cbedc9f9b1","created":"2024-09-18T04:37:15.676634+0000","modified":"2024-09-18T04:37:17.283604+0000","last_up_change":"0.000000","last_in_change":"2024-09-18T04:37:17.283604+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6177002d-72e6-4888-bcde-29b6a92d9dc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4f62cb09-3457-418c-bb69-491d9b70914c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"854fe031-3b87-4290-9c2f-3fb196fb806c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c2625f98-780b-4f86-bf36-a48d2a227756","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"deb22d86-4081-484d-96eb-9cb9d96b60bc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"061d332c-f51b-403d-bb18-eba415c97e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T04:37:17.542 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-18T04:37:17.542 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-18T04:37:17.637 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-09-18T04:37:17.631+0000 7f2e3f7e82c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-18T04:37:17.681 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:37:17.675+0000 7fa93c123880 -1 Falling back to public interface 2024-09-18T04:37:17.688 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:37:17.681+0000 7ff1c4ccd880 -1 Falling back to public interface 2024-09-18T04:37:17.688 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:37:17.681+0000 7f6ab195f880 -1 Falling back to public interface 2024-09-18T04:37:17.711 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:37:17.707+0000 7f6b6add9880 -1 Falling back to public interface 2024-09-18T04:37:17.855 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-09-18T04:37:17.851+0000 7f2e3f7e82c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-18T04:37:17.906 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-09-18T04:37:17.899+0000 7f2e3f7e82c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-18T04:37:17.943 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:37:17.937+0000 7fad76945880 -1 Falling back to public interface 2024-09-18T04:37:17.955 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:37:17.951+0000 7fba167d8880 -1 Falling back to public interface 2024-09-18T04:37:18.082 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-09-18T04:37:18.075+0000 7f2e3f7e82c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-18T04:37:18.145 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-09-18T04:37:18.139+0000 7f2e3f7e82c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-18T04:37:18.147 DEBUG:teuthology.orchestra.run.smithi028:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-18T04:37:18.209 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-09-18T04:37:18.203+0000 7f2e3f7e82c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-18T04:37:18.337 INFO:teuthology.misc.health.smithi028.stdout: 2024-09-18T04:37:18.337 INFO:teuthology.misc.health.smithi028.stdout:{"epoch":8,"fsid":"be6146ba-a991-4f6e-b359-99cbedc9f9b1","created":"2024-09-18T04:37:15.676634+0000","modified":"2024-09-18T04:37:17.283604+0000","last_up_change":"0.000000","last_in_change":"2024-09-18T04:37:17.283604+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6177002d-72e6-4888-bcde-29b6a92d9dc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4f62cb09-3457-418c-bb69-491d9b70914c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"854fe031-3b87-4290-9c2f-3fb196fb806c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c2625f98-780b-4f86-bf36-a48d2a227756","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"deb22d86-4081-484d-96eb-9cb9d96b60bc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"061d332c-f51b-403d-bb18-eba415c97e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T04:37:18.349 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-09-18T04:37:18.597 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-09-18T04:37:18.591+0000 7f2e3f7e82c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-18T04:37:18.713 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-09-18T04:37:18.707+0000 7f2e3f7e82c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-18T04:37:19.044 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-09-18T04:37:19.039+0000 7f2e3f7e82c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-18T04:37:19.228 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-09-18T04:37:19.223+0000 7f2e3f7e82c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-18T04:37:19.296 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-09-18T04:37:19.291+0000 7f2e3f7e82c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-18T04:37:19.455 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-09-18T04:37:19.451+0000 7f2e3f7e82c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-18T04:37:20.132 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-09-18T04:37:20.127+0000 7f2e3f7e82c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-18T04:37:20.197 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-09-18T04:37:20.191+0000 7f2e3f7e82c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-18T04:37:20.368 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-09-18T04:37:20.363+0000 7f2e3f7e82c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-18T04:37:20.436 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-09-18T04:37:20.431+0000 7f2e3f7e82c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-18T04:37:20.505 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-09-18T04:37:20.499+0000 7f2e3f7e82c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-18T04:37:20.813 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-09-18T04:37:20.807+0000 7f2e3f7e82c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-18T04:37:20.886 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-09-18T04:37:20.883+0000 7f2e3f7e82c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-18T04:37:21.011 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:37:21.009+0000 7f6ab195f880 -1 osd.1 0 log_to_monitors true 2024-09-18T04:37:21.225 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:37:21.221+0000 7ff1c4ccd880 -1 osd.0 0 log_to_monitors true 2024-09-18T04:37:21.386 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:37:21.383+0000 7fba167d8880 -1 osd.5 0 log_to_monitors true 2024-09-18T04:37:21.517 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:37:21.511+0000 7fa93c123880 -1 osd.4 0 log_to_monitors true 2024-09-18T04:37:21.561 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-09-18T04:37:21.555+0000 7f2e3f7e82c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-18T04:37:21.690 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:37:21.683+0000 7f6b6add9880 -1 osd.3 0 log_to_monitors true 2024-09-18T04:37:21.728 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:37:21.724+0000 7fad76945880 -1 osd.2 0 log_to_monitors true 2024-09-18T04:37:22.603 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-09-18T04:37:22.599+0000 7f2e2635f700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-18T04:37:22.603 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-09-18T04:37:22.599+0000 7f2e2635f700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-18T04:37:23.387 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:37:23.384+0000 7f6aad79e700 -1 osd.1 0 waiting for initial osdmap 2024-09-18T04:37:23.393 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:37:23.388+0000 7f6aa524d700 -1 osd.1 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T04:37:24.324 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:37:24.320+0000 7fad72784700 -1 osd.2 0 waiting for initial osdmap 2024-09-18T04:37:24.329 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:37:24.328+0000 7fad6a233700 -1 osd.2 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T04:37:24.351 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:37:24.347+0000 7f6b66c18700 -1 osd.3 0 waiting for initial osdmap 2024-09-18T04:37:24.358 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:37:24.355+0000 7f6b5e6c7700 -1 osd.3 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T04:37:24.393 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:37:24.387+0000 7fba12617700 -1 osd.5 0 waiting for initial osdmap 2024-09-18T04:37:24.403 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:37:24.399+0000 7fba0a0c6700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T04:37:24.410 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:37:24.403+0000 7fa937f62700 -1 osd.4 0 waiting for initial osdmap 2024-09-18T04:37:24.415 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:37:24.411+0000 7fa92fa11700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T04:37:24.437 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:37:24.436+0000 7ff1c1493700 -1 osd.0 0 waiting for initial osdmap 2024-09-18T04:37:24.448 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:37:24.444+0000 7ff1b85bb700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T04:37:24.956 DEBUG:teuthology.orchestra.run.smithi028:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-18T04:37:25.258 INFO:teuthology.misc.health.smithi028.stdout: 2024-09-18T04:37:25.258 INFO:teuthology.misc.health.smithi028.stdout:{"epoch":12,"fsid":"be6146ba-a991-4f6e-b359-99cbedc9f9b1","created":"2024-09-18T04:37:15.676634+0000","modified":"2024-09-18T04:37:24.388730+0000","last_up_change":"2024-09-18T04:37:24.388730+0000","last_in_change":"2024-09-18T04:37:17.283604+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6177002d-72e6-4888-bcde-29b6a92d9dc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4f62cb09-3457-418c-bb69-491d9b70914c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":511141401},{"type":"v1","addr":"172.21.15.28:6802","nonce":511141401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":511141401},{"type":"v1","addr":"172.21.15.28:6806","nonce":511141401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":511141401},{"type":"v1","addr":"172.21.15.28:6814","nonce":511141401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":511141401},{"type":"v1","addr":"172.21.15.28:6810","nonce":511141401}]},"public_addr":"172.21.15.28:6802/511141401","cluster_addr":"172.21.15.28:6806/511141401","heartbeat_back_addr":"172.21.15.28:6814/511141401","heartbeat_front_addr":"172.21.15.28:6810/511141401","state":["exists","up"]},{"osd":2,"uuid":"854fe031-3b87-4290-9c2f-3fb196fb806c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2623426735},{"type":"v1","addr":"172.21.15.28:6817","nonce":2623426735}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":2623426735},{"type":"v1","addr":"172.21.15.28:6819","nonce":2623426735}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":2623426735},{"type":"v1","addr":"172.21.15.28:6823","nonce":2623426735}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2623426735},{"type":"v1","addr":"172.21.15.28:6821","nonce":2623426735}]},"public_addr":"172.21.15.28:6817/2623426735","cluster_addr":"172.21.15.28:6819/2623426735","heartbeat_back_addr":"172.21.15.28:6823/2623426735","heartbeat_front_addr":"172.21.15.28:6821/2623426735","state":["exists","up"]},{"osd":3,"uuid":"c2625f98-780b-4f86-bf36-a48d2a227756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":2746051132},{"type":"v1","addr":"172.21.15.82:6809","nonce":2746051132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":2746051132},{"type":"v1","addr":"172.21.15.82:6811","nonce":2746051132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":2746051132},{"type":"v1","addr":"172.21.15.82:6815","nonce":2746051132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":2746051132},{"type":"v1","addr":"172.21.15.82:6813","nonce":2746051132}]},"public_addr":"172.21.15.82:6809/2746051132","cluster_addr":"172.21.15.82:6811/2746051132","heartbeat_back_addr":"172.21.15.82:6815/2746051132","heartbeat_front_addr":"172.21.15.82:6813/2746051132","state":["exists","up"]},{"osd":4,"uuid":"deb22d86-4081-484d-96eb-9cb9d96b60bc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"061d332c-f51b-403d-bb18-eba415c97e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T04:37:25.271 DEBUG:teuthology.misc:3 of 6 OSDs are up 2024-09-18T04:37:31.877 DEBUG:teuthology.orchestra.run.smithi028:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-18T04:37:32.185 INFO:teuthology.misc.health.smithi028.stdout: 2024-09-18T04:37:32.314 INFO:teuthology.misc.health.smithi028.stdout:{"epoch":16,"fsid":"be6146ba-a991-4f6e-b359-99cbedc9f9b1","created":"2024-09-18T04:37:15.676634+0000","modified":"2024-09-18T04:37:28.422932+0000","last_up_change":"2024-09-18T04:37:25.403175+0000","last_in_change":"2024-09-18T04:37:17.283604+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-18T04:37:25.654950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6177002d-72e6-4888-bcde-29b6a92d9dc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6801","nonce":1554407851},{"type":"v1","addr":"172.21.15.28:6803","nonce":1554407851}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6805","nonce":1554407851},{"type":"v1","addr":"172.21.15.28:6807","nonce":1554407851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6813","nonce":1554407851},{"type":"v1","addr":"172.21.15.28:6815","nonce":1554407851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":1554407851},{"type":"v1","addr":"172.21.15.28:6811","nonce":1554407851}]},"public_addr":"172.21.15.28:6803/1554407851","cluster_addr":"172.21.15.28:6807/1554407851","heartbeat_back_addr":"172.21.15.28:6815/1554407851","heartbeat_front_addr":"172.21.15.28:6811/1554407851","state":["exists","up"]},{"osd":1,"uuid":"4f62cb09-3457-418c-bb69-491d9b70914c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":511141401},{"type":"v1","addr":"172.21.15.28:6802","nonce":511141401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":511141401},{"type":"v1","addr":"172.21.15.28:6806","nonce":511141401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":511141401},{"type":"v1","addr":"172.21.15.28:6814","nonce":511141401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":511141401},{"type":"v1","addr":"172.21.15.28:6810","nonce":511141401}]},"public_addr":"172.21.15.28:6802/511141401","cluster_addr":"172.21.15.28:6806/511141401","heartbeat_back_addr":"172.21.15.28:6814/511141401","heartbeat_front_addr":"172.21.15.28:6810/511141401","state":["exists","up"]},{"osd":2,"uuid":"854fe031-3b87-4290-9c2f-3fb196fb806c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2623426735},{"type":"v1","addr":"172.21.15.28:6817","nonce":2623426735}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":2623426735},{"type":"v1","addr":"172.21.15.28:6819","nonce":2623426735}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":2623426735},{"type":"v1","addr":"172.21.15.28:6823","nonce":2623426735}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2623426735},{"type":"v1","addr":"172.21.15.28:6821","nonce":2623426735}]},"public_addr":"172.21.15.28:6817/2623426735","cluster_addr":"172.21.15.28:6819/2623426735","heartbeat_back_addr":"172.21.15.28:6823/2623426735","heartbeat_front_addr":"172.21.15.28:6821/2623426735","state":["exists","up"]},{"osd":3,"uuid":"c2625f98-780b-4f86-bf36-a48d2a227756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":2746051132},{"type":"v1","addr":"172.21.15.82:6809","nonce":2746051132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":2746051132},{"type":"v1","addr":"172.21.15.82:6811","nonce":2746051132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":2746051132},{"type":"v1","addr":"172.21.15.82:6815","nonce":2746051132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":2746051132},{"type":"v1","addr":"172.21.15.82:6813","nonce":2746051132}]},"public_addr":"172.21.15.82:6809/2746051132","cluster_addr":"172.21.15.82:6811/2746051132","heartbeat_back_addr":"172.21.15.82:6815/2746051132","heartbeat_front_addr":"172.21.15.82:6813/2746051132","state":["exists","up"]},{"osd":4,"uuid":"deb22d86-4081-484d-96eb-9cb9d96b60bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":2005712460},{"type":"v1","addr":"172.21.15.82:6801","nonce":2005712460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":2005712460},{"type":"v1","addr":"172.21.15.82:6803","nonce":2005712460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":2005712460},{"type":"v1","addr":"172.21.15.82:6807","nonce":2005712460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":2005712460},{"type":"v1","addr":"172.21.15.82:6805","nonce":2005712460}]},"public_addr":"172.21.15.82:6801/2005712460","cluster_addr":"172.21.15.82:6803/2005712460","heartbeat_back_addr":"172.21.15.82:6807/2005712460","heartbeat_front_addr":"172.21.15.82:6805/2005712460","state":["exists","up"]},{"osd":5,"uuid":"061d332c-f51b-403d-bb18-eba415c97e7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":1267644232},{"type":"v1","addr":"172.21.15.82:6817","nonce":1267644232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":1267644232},{"type":"v1","addr":"172.21.15.82:6819","nonce":1267644232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":1267644232},{"type":"v1","addr":"172.21.15.82:6823","nonce":1267644232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":1267644232},{"type":"v1","addr":"172.21.15.82:6821","nonce":1267644232}]},"public_addr":"172.21.15.82:6817/1267644232","cluster_addr":"172.21.15.82:6819/1267644232","heartbeat_back_addr":"172.21.15.82:6823/1267644232","heartbeat_front_addr":"172.21.15.82:6821/1267644232","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T04:37:22.192087+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T04:37:22.017833+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T04:37:22.755774+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T04:37:22.663785+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T04:37:22.531758+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T04:37:22.400774+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T04:37:32.316 DEBUG:teuthology.misc:6 of 6 OSDs are up 2024-09-18T04:37:32.317 INFO:tasks.ceph:Creating RBD pool 2024-09-18T04:37:32.317 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-18T04:37:33.466 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'rbd' created 2024-09-18T04:37:33.480 DEBUG:teuthology.orchestra.run.smithi028:> rbd --cluster ceph pool init rbd 2024-09-18T04:37:36.501 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-18T04:37:36.501 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-18T04:37:36.501 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-18T04:37:36.502 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-18T04:37:36.502 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-18T04:37:36.865 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:37:36.879 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":5,"flags":0,"active_gid":4120,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":406299689},{"type":"v1","addr":"172.21.15.82:6825","nonce":406299689}]},"active_addr":"172.21.15.82:6825/406299689","active_change":"2024-09-18T04:37:21.580351+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.82:0","nonce":696981890}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.82:0","nonce":1176872315}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.82:0","nonce":3260555546}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.82:0","nonce":3388791205}]}]} 2024-09-18T04:37:36.881 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-18T04:37:36.881 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-18T04:37:36.881 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T04:37:37.186 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:37:37.186 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"fsid":"be6146ba-a991-4f6e-b359-99cbedc9f9b1","created":"2024-09-18T04:37:15.676634+0000","modified":"2024-09-18T04:37:36.470579+0000","last_up_change":"2024-09-18T04:37:25.403175+0000","last_in_change":"2024-09-18T04:37:17.283604+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-18T04:37:25.654950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-18T04:37:32.672462+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":20,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6177002d-72e6-4888-bcde-29b6a92d9dc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6801","nonce":1554407851},{"type":"v1","addr":"172.21.15.28:6803","nonce":1554407851}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6805","nonce":1554407851},{"type":"v1","addr":"172.21.15.28:6807","nonce":1554407851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6813","nonce":1554407851},{"type":"v1","addr":"172.21.15.28:6815","nonce":1554407851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":1554407851},{"type":"v1","addr":"172.21.15.28:6811","nonce":1554407851}]},"public_addr":"172.21.15.28:6803/1554407851","cluster_addr":"172.21.15.28:6807/1554407851","heartbeat_back_addr":"172.21.15.28:6815/1554407851","heartbeat_front_addr":"172.21.15.28:6811/1554407851","state":["exists","up"]},{"osd":1,"uuid":"4f62cb09-3457-418c-bb69-491d9b70914c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":511141401},{"type":"v1","addr":"172.21.15.28:6802","nonce":511141401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":511141401},{"type":"v1","addr":"172.21.15.28:6806","nonce":511141401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":511141401},{"type":"v1","addr":"172.21.15.28:6814","nonce":511141401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":511141401},{"type":"v1","addr":"172.21.15.28:6810","nonce":511141401}]},"public_addr":"172.21.15.28:6802/511141401","cluster_addr":"172.21.15.28:6806/511141401","heartbeat_back_addr":"172.21.15.28:6814/511141401","heartbeat_front_addr":"172.21.15.28:6810/511141401","state":["exists","up"]},{"osd":2,"uuid":"854fe031-3b87-4290-9c2f-3fb196fb806c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2623426735},{"type":"v1","addr":"172.21.15.28:6817","nonce":2623426735}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":2623426735},{"type":"v1","addr":"172.21.15.28:6819","nonce":2623426735}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":2623426735},{"type":"v1","addr":"172.21.15.28:6823","nonce":2623426735}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2623426735},{"type":"v1","addr":"172.21.15.28:6821","nonce":2623426735}]},"public_addr":"172.21.15.28:6817/2623426735","cluster_addr":"172.21.15.28:6819/2623426735","heartbeat_back_addr":"172.21.15.28:6823/2623426735","heartbeat_front_addr":"172.21.15.28:6821/2623426735","state":["exists","up"]},{"osd":3,"uuid":"c2625f98-780b-4f86-bf36-a48d2a227756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":2746051132},{"type":"v1","addr":"172.21.15.82:6809","nonce":2746051132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":2746051132},{"type":"v1","addr":"172.21.15.82:6811","nonce":2746051132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":2746051132},{"type":"v1","addr":"172.21.15.82:6815","nonce":2746051132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":2746051132},{"type":"v1","addr":"172.21.15.82:6813","nonce":2746051132}]},"public_addr":"172.21.15.82:6809/2746051132","cluster_addr":"172.21.15.82:6811/2746051132","heartbeat_back_addr":"172.21.15.82:6815/2746051132","heartbeat_front_addr":"172.21.15.82:6813/2746051132","state":["exists","up"]},{"osd":4,"uuid":"deb22d86-4081-484d-96eb-9cb9d96b60bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":2005712460},{"type":"v1","addr":"172.21.15.82:6801","nonce":2005712460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":2005712460},{"type":"v1","addr":"172.21.15.82:6803","nonce":2005712460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":2005712460},{"type":"v1","addr":"172.21.15.82:6807","nonce":2005712460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":2005712460},{"type":"v1","addr":"172.21.15.82:6805","nonce":2005712460}]},"public_addr":"172.21.15.82:6801/2005712460","cluster_addr":"172.21.15.82:6803/2005712460","heartbeat_back_addr":"172.21.15.82:6807/2005712460","heartbeat_front_addr":"172.21.15.82:6805/2005712460","state":["exists","up"]},{"osd":5,"uuid":"061d332c-f51b-403d-bb18-eba415c97e7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":1267644232},{"type":"v1","addr":"172.21.15.82:6817","nonce":1267644232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":1267644232},{"type":"v1","addr":"172.21.15.82:6819","nonce":1267644232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":1267644232},{"type":"v1","addr":"172.21.15.82:6823","nonce":1267644232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":1267644232},{"type":"v1","addr":"172.21.15.82:6821","nonce":1267644232}]},"public_addr":"172.21.15.82:6817/1267644232","cluster_addr":"172.21.15.82:6819/1267644232","heartbeat_back_addr":"172.21.15.82:6823/1267644232","heartbeat_front_addr":"172.21.15.82:6821/1267644232","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T04:37:22.192087+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T04:37:22.017833+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T04:37:22.755774+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T04:37:22.663785+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T04:37:22.531758+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T04:37:22.400774+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T04:37:37.201 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-18T04:37:37.201 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T04:37:37.500 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:37:37.500 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"fsid":"be6146ba-a991-4f6e-b359-99cbedc9f9b1","created":"2024-09-18T04:37:15.676634+0000","modified":"2024-09-18T04:37:36.470579+0000","last_up_change":"2024-09-18T04:37:25.403175+0000","last_in_change":"2024-09-18T04:37:17.283604+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-18T04:37:25.654950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-18T04:37:32.672462+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":20,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6177002d-72e6-4888-bcde-29b6a92d9dc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6801","nonce":1554407851},{"type":"v1","addr":"172.21.15.28:6803","nonce":1554407851}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6805","nonce":1554407851},{"type":"v1","addr":"172.21.15.28:6807","nonce":1554407851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6813","nonce":1554407851},{"type":"v1","addr":"172.21.15.28:6815","nonce":1554407851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":1554407851},{"type":"v1","addr":"172.21.15.28:6811","nonce":1554407851}]},"public_addr":"172.21.15.28:6803/1554407851","cluster_addr":"172.21.15.28:6807/1554407851","heartbeat_back_addr":"172.21.15.28:6815/1554407851","heartbeat_front_addr":"172.21.15.28:6811/1554407851","state":["exists","up"]},{"osd":1,"uuid":"4f62cb09-3457-418c-bb69-491d9b70914c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":511141401},{"type":"v1","addr":"172.21.15.28:6802","nonce":511141401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":511141401},{"type":"v1","addr":"172.21.15.28:6806","nonce":511141401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":511141401},{"type":"v1","addr":"172.21.15.28:6814","nonce":511141401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":511141401},{"type":"v1","addr":"172.21.15.28:6810","nonce":511141401}]},"public_addr":"172.21.15.28:6802/511141401","cluster_addr":"172.21.15.28:6806/511141401","heartbeat_back_addr":"172.21.15.28:6814/511141401","heartbeat_front_addr":"172.21.15.28:6810/511141401","state":["exists","up"]},{"osd":2,"uuid":"854fe031-3b87-4290-9c2f-3fb196fb806c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2623426735},{"type":"v1","addr":"172.21.15.28:6817","nonce":2623426735}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":2623426735},{"type":"v1","addr":"172.21.15.28:6819","nonce":2623426735}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":2623426735},{"type":"v1","addr":"172.21.15.28:6823","nonce":2623426735}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2623426735},{"type":"v1","addr":"172.21.15.28:6821","nonce":2623426735}]},"public_addr":"172.21.15.28:6817/2623426735","cluster_addr":"172.21.15.28:6819/2623426735","heartbeat_back_addr":"172.21.15.28:6823/2623426735","heartbeat_front_addr":"172.21.15.28:6821/2623426735","state":["exists","up"]},{"osd":3,"uuid":"c2625f98-780b-4f86-bf36-a48d2a227756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":2746051132},{"type":"v1","addr":"172.21.15.82:6809","nonce":2746051132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":2746051132},{"type":"v1","addr":"172.21.15.82:6811","nonce":2746051132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":2746051132},{"type":"v1","addr":"172.21.15.82:6815","nonce":2746051132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":2746051132},{"type":"v1","addr":"172.21.15.82:6813","nonce":2746051132}]},"public_addr":"172.21.15.82:6809/2746051132","cluster_addr":"172.21.15.82:6811/2746051132","heartbeat_back_addr":"172.21.15.82:6815/2746051132","heartbeat_front_addr":"172.21.15.82:6813/2746051132","state":["exists","up"]},{"osd":4,"uuid":"deb22d86-4081-484d-96eb-9cb9d96b60bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":2005712460},{"type":"v1","addr":"172.21.15.82:6801","nonce":2005712460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":2005712460},{"type":"v1","addr":"172.21.15.82:6803","nonce":2005712460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":2005712460},{"type":"v1","addr":"172.21.15.82:6807","nonce":2005712460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":2005712460},{"type":"v1","addr":"172.21.15.82:6805","nonce":2005712460}]},"public_addr":"172.21.15.82:6801/2005712460","cluster_addr":"172.21.15.82:6803/2005712460","heartbeat_back_addr":"172.21.15.82:6807/2005712460","heartbeat_front_addr":"172.21.15.82:6805/2005712460","state":["exists","up"]},{"osd":5,"uuid":"061d332c-f51b-403d-bb18-eba415c97e7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":1267644232},{"type":"v1","addr":"172.21.15.82:6817","nonce":1267644232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":1267644232},{"type":"v1","addr":"172.21.15.82:6819","nonce":1267644232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":1267644232},{"type":"v1","addr":"172.21.15.82:6823","nonce":1267644232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":1267644232},{"type":"v1","addr":"172.21.15.82:6821","nonce":1267644232}]},"public_addr":"172.21.15.82:6817/1267644232","cluster_addr":"172.21.15.82:6819/1267644232","heartbeat_back_addr":"172.21.15.82:6823/1267644232","heartbeat_front_addr":"172.21.15.82:6821/1267644232","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T04:37:22.192087+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T04:37:22.017833+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T04:37:22.755774+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T04:37:22.663785+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T04:37:22.531758+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T04:37:22.400774+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T04:37:37.516 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-18T04:37:37.516 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-18T04:37:37.517 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-18T04:37:37.517 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-18T04:37:37.517 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-18T04:37:37.518 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-18T04:37:37.668 INFO:teuthology.orchestra.run.smithi028.stdout:51539607558 2024-09-18T04:37:37.669 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-18T04:37:37.685 INFO:teuthology.orchestra.run.smithi028.stdout:51539607556 2024-09-18T04:37:37.685 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-18T04:37:37.712 INFO:teuthology.orchestra.run.smithi028.stdout:55834574855 2024-09-18T04:37:37.712 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-18T04:37:37.713 INFO:teuthology.orchestra.run.smithi028.stdout:55834574854 2024-09-18T04:37:37.714 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-18T04:37:37.726 INFO:teuthology.orchestra.run.smithi028.stdout:51539607558 2024-09-18T04:37:37.726 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-18T04:37:37.741 INFO:teuthology.orchestra.run.smithi028.stdout:55834574854 2024-09-18T04:37:37.741 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-18T04:37:38.024 INFO:teuthology.orchestra.run.smithi028.stdout:55834574852 2024-09-18T04:37:38.043 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574852 for osd.4 2024-09-18T04:37:38.069 INFO:teuthology.orchestra.run.smithi028.stdout:55834574853 2024-09-18T04:37:38.082 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574853 for osd.0 2024-09-18T04:37:38.087 INFO:teuthology.orchestra.run.smithi028.stdout:51539607557 2024-09-18T04:37:38.102 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607558 got 51539607557 for osd.1 2024-09-18T04:37:38.124 INFO:teuthology.orchestra.run.smithi028.stdout:51539607555 2024-09-18T04:37:38.138 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607555 for osd.2 2024-09-18T04:37:38.170 INFO:teuthology.orchestra.run.smithi028.stdout:55834574852 2024-09-18T04:37:38.185 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574852 for osd.5 2024-09-18T04:37:38.187 INFO:teuthology.orchestra.run.smithi028.stdout:51539607557 2024-09-18T04:37:38.201 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607558 got 51539607557 for osd.3 2024-09-18T04:37:39.043 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-18T04:37:39.083 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-18T04:37:39.103 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-18T04:37:39.139 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-18T04:37:39.186 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-18T04:37:39.202 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-18T04:37:39.458 INFO:teuthology.orchestra.run.smithi028.stdout:51539607555 2024-09-18T04:37:39.475 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607555 for osd.2 2024-09-18T04:37:39.495 INFO:teuthology.orchestra.run.smithi028.stdout:55834574853 2024-09-18T04:37:39.513 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.4 2024-09-18T04:37:39.531 INFO:teuthology.orchestra.run.smithi028.stdout:51539607557 2024-09-18T04:37:39.546 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607558 got 51539607557 for osd.3 2024-09-18T04:37:39.549 INFO:teuthology.orchestra.run.smithi028.stdout:51539607557 2024-09-18T04:37:39.560 INFO:teuthology.orchestra.run.smithi028.stdout:55834574854 2024-09-18T04:37:39.562 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607558 got 51539607557 for osd.1 2024-09-18T04:37:39.577 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.0 2024-09-18T04:37:39.621 INFO:teuthology.orchestra.run.smithi028.stdout:55834574853 2024-09-18T04:37:39.634 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.5 2024-09-18T04:37:40.477 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-18T04:37:40.513 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-18T04:37:40.547 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-18T04:37:40.563 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-18T04:37:40.577 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-18T04:37:40.635 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-18T04:37:40.912 INFO:teuthology.orchestra.run.smithi028.stdout:55834574855 2024-09-18T04:37:40.924 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.0 2024-09-18T04:37:40.925 DEBUG:teuthology.parallel:result is None 2024-09-18T04:37:40.939 INFO:teuthology.orchestra.run.smithi028.stdout:55834574854 2024-09-18T04:37:40.950 INFO:teuthology.orchestra.run.smithi028.stdout:51539607556 2024-09-18T04:37:40.952 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.5 2024-09-18T04:37:40.952 DEBUG:teuthology.parallel:result is None 2024-09-18T04:37:40.967 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607556 for osd.2 2024-09-18T04:37:40.967 DEBUG:teuthology.parallel:result is None 2024-09-18T04:37:40.989 INFO:teuthology.orchestra.run.smithi028.stdout:55834574854 2024-09-18T04:37:41.005 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.4 2024-09-18T04:37:41.006 DEBUG:teuthology.parallel:result is None 2024-09-18T04:37:41.008 INFO:teuthology.orchestra.run.smithi028.stdout:51539607558 2024-09-18T04:37:41.009 INFO:teuthology.orchestra.run.smithi028.stdout:51539607558 2024-09-18T04:37:41.022 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607558 got 51539607558 for osd.1 2024-09-18T04:37:41.022 DEBUG:teuthology.parallel:result is None 2024-09-18T04:37:41.023 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607558 got 51539607558 for osd.3 2024-09-18T04:37:41.023 DEBUG:teuthology.parallel:result is None 2024-09-18T04:37:41.023 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-18T04:37:41.024 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T04:37:41.355 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:37:41.358 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-18T04:37:41.369 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-18T04:37:39.593885+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":179756,"kb_used_data":1240,"kb_used_omap":9,"kb_used_meta":178422,"kb_avail":566051284,"statfs":{"total":579820584960,"available":579636514816,"internally_reserved":0,"allocated":1269760,"data_stored":1553500,"data_compressed":66040,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":9561,"internal_metadata":182704807},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001643"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-18T04:37:36.484136+0000","last_change":"2024-09-18T04:37:36.484517+0000","last_active":"2024-09-18T04:37:36.484136+0000","last_peered":"2024-09-18T04:37:36.484136+0000","last_clean":"2024-09-18T04:37:36.484136+0000","last_became_active":"2024-09-18T04:37:34.893674+0000","last_became_peered":"2024-09-18T04:37:34.893674+0000","last_unstale":"2024-09-18T04:37:36.484136+0000","last_undegraded":"2024-09-18T04:37:36.484136+0000","last_fullsized":"2024-09-18T04:37:36.484136+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T09:53:33.883648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047343899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-18T04:37:36.493875+0000","last_change":"2024-09-18T04:37:36.494193+0000","last_active":"2024-09-18T04:37:36.493875+0000","last_peered":"2024-09-18T04:37:36.493875+0000","last_clean":"2024-09-18T04:37:36.493875+0000","last_became_active":"2024-09-18T04:37:34.472940+0000","last_became_peered":"2024-09-18T04:37:34.472940+0000","last_unstale":"2024-09-18T04:37:36.493875+0000","last_undegraded":"2024-09-18T04:37:36.493875+0000","last_fullsized":"2024-09-18T04:37:36.493875+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:18:29.266815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070553900000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-18T04:37:36.484076+0000","last_change":"2024-09-18T04:37:36.484508+0000","last_active":"2024-09-18T04:37:36.484076+0000","last_peered":"2024-09-18T04:37:36.484076+0000","last_clean":"2024-09-18T04:37:36.484076+0000","last_became_active":"2024-09-18T04:37:34.468968+0000","last_became_peered":"2024-09-18T04:37:34.468968+0000","last_unstale":"2024-09-18T04:37:36.484076+0000","last_undegraded":"2024-09-18T04:37:36.484076+0000","last_fullsized":"2024-09-18T04:37:36.484076+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T08:46:16.183136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059422499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-18T04:37:36.493770+0000","last_change":"2024-09-18T04:37:36.494228+0000","last_active":"2024-09-18T04:37:36.493770+0000","last_peered":"2024-09-18T04:37:36.493770+0000","last_clean":"2024-09-18T04:37:36.493770+0000","last_became_active":"2024-09-18T04:37:34.471265+0000","last_became_peered":"2024-09-18T04:37:34.471265+0000","last_unstale":"2024-09-18T04:37:36.493770+0000","last_undegraded":"2024-09-18T04:37:36.493770+0000","last_fullsized":"2024-09-18T04:37:36.493770+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:23:05.748076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081514499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"20'2","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-18T04:37:36.490461+0000","last_change":"2024-09-18T04:37:36.490461+0000","last_active":"2024-09-18T04:37:36.490461+0000","last_peered":"2024-09-18T04:37:36.490461+0000","last_clean":"2024-09-18T04:37:36.490461+0000","last_became_active":"2024-09-18T04:37:34.473088+0000","last_became_peered":"2024-09-18T04:37:34.473088+0000","last_unstale":"2024-09-18T04:37:36.490461+0000","last_undegraded":"2024-09-18T04:37:36.490461+0000","last_fullsized":"2024-09-18T04:37:36.490461+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:17:57.352397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001879932,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-18T04:37:36.482703+0000","last_change":"2024-09-18T04:37:36.483033+0000","last_active":"2024-09-18T04:37:36.482703+0000","last_peered":"2024-09-18T04:37:36.482703+0000","last_clean":"2024-09-18T04:37:36.482703+0000","last_became_active":"2024-09-18T04:37:34.471649+0000","last_became_peered":"2024-09-18T04:37:34.471649+0000","last_unstale":"2024-09-18T04:37:36.482703+0000","last_undegraded":"2024-09-18T04:37:36.482703+0000","last_fullsized":"2024-09-18T04:37:36.482703+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T07:21:30.888347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048363399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-18T04:37:36.484618+0000","last_change":"2024-09-18T04:37:36.484950+0000","last_active":"2024-09-18T04:37:36.484618+0000","last_peered":"2024-09-18T04:37:36.484618+0000","last_clean":"2024-09-18T04:37:36.484618+0000","last_became_active":"2024-09-18T04:37:34.470841+0000","last_became_peered":"2024-09-18T04:37:34.470841+0000","last_unstale":"2024-09-18T04:37:36.484618+0000","last_undegraded":"2024-09-18T04:37:36.484618+0000","last_fullsized":"2024-09-18T04:37:36.484618+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:16:34.528783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090513900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"18'1","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-18T04:37:36.489928+0000","last_change":"2024-09-18T04:37:36.490456+0000","last_active":"2024-09-18T04:37:36.489928+0000","last_peered":"2024-09-18T04:37:36.489928+0000","last_clean":"2024-09-18T04:37:36.489928+0000","last_became_active":"2024-09-18T04:37:34.473044+0000","last_became_peered":"2024-09-18T04:37:34.473044+0000","last_unstale":"2024-09-18T04:37:36.489928+0000","last_undegraded":"2024-09-18T04:37:36.489928+0000","last_fullsized":"2024-09-18T04:37:36.489928+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:41.239574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018293420000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"16'117","reported_seq":164,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-18T04:37:35.532099+0000","last_change":"2024-09-18T04:37:27.423069+0000","last_active":"2024-09-18T04:37:35.532099+0000","last_peered":"2024-09-18T04:37:35.532099+0000","last_clean":"2024-09-18T04:37:35.532099+0000","last_became_active":"2024-09-18T04:37:27.422350+0000","last_became_peered":"2024-09-18T04:37:27.422350+0000","last_unstale":"2024-09-18T04:37:35.532099+0000","last_undegraded":"2024-09-18T04:37:35.532099+0000","last_fullsized":"2024-09-18T04:37:35.532099+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:26.406292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:26.406292+0000","last_clean_scrub_stamp":"2024-09-18T04:37:26.406292+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:31:45.852149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":66040,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":192512,"data_stored":62140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59999999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94499999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63800000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":62121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71399999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.877}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.745}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80200000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0720000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31368,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340472,"statfs":{"total":96636764160,"available":96604643328,"internally_reserved":0,"allocated":253952,"data_stored":652489,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1220000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.073}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91100000000000003}]}]},{"osd":3,"up_from":12,"seq":51539607558,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31364,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340476,"statfs":{"total":96636764160,"available":96604647424,"internally_reserved":0,"allocated":253952,"data_stored":652489,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.175}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83099999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.101}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78100000000000003}]}]},{"osd":2,"up_from":12,"seq":51539607556,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31300,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340540,"statfs":{"total":96636764160,"available":96604712960,"internally_reserved":0,"allocated":188416,"data_stored":62121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93899999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95799999999999996}]}]},{"osd":1,"up_from":12,"seq":51539607558,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31308,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340532,"statfs":{"total":96636764160,"available":96604704768,"internally_reserved":0,"allocated":192512,"data_stored":62140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93999999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71199999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T04:37:41.370 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T04:37:41.664 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:37:41.666 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-18T04:37:41.677 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-09-18T04:37:41.594477+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":179756,"kb_used_data":1240,"kb_used_omap":9,"kb_used_meta":178422,"kb_avail":566051284,"statfs":{"total":579820584960,"available":579636514816,"internally_reserved":0,"allocated":1269760,"data_stored":1553500,"data_compressed":66040,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":9561,"internal_metadata":182704807},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.132986"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-18T04:37:36.484136+0000","last_change":"2024-09-18T04:37:36.484517+0000","last_active":"2024-09-18T04:37:36.484136+0000","last_peered":"2024-09-18T04:37:36.484136+0000","last_clean":"2024-09-18T04:37:36.484136+0000","last_became_active":"2024-09-18T04:37:34.893674+0000","last_became_peered":"2024-09-18T04:37:34.893674+0000","last_unstale":"2024-09-18T04:37:36.484136+0000","last_undegraded":"2024-09-18T04:37:36.484136+0000","last_fullsized":"2024-09-18T04:37:36.484136+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T09:53:33.883648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047343899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-18T04:37:36.493875+0000","last_change":"2024-09-18T04:37:36.494193+0000","last_active":"2024-09-18T04:37:36.493875+0000","last_peered":"2024-09-18T04:37:36.493875+0000","last_clean":"2024-09-18T04:37:36.493875+0000","last_became_active":"2024-09-18T04:37:34.472940+0000","last_became_peered":"2024-09-18T04:37:34.472940+0000","last_unstale":"2024-09-18T04:37:36.493875+0000","last_undegraded":"2024-09-18T04:37:36.493875+0000","last_fullsized":"2024-09-18T04:37:36.493875+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:18:29.266815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070553900000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-18T04:37:36.484076+0000","last_change":"2024-09-18T04:37:36.484508+0000","last_active":"2024-09-18T04:37:36.484076+0000","last_peered":"2024-09-18T04:37:36.484076+0000","last_clean":"2024-09-18T04:37:36.484076+0000","last_became_active":"2024-09-18T04:37:34.468968+0000","last_became_peered":"2024-09-18T04:37:34.468968+0000","last_unstale":"2024-09-18T04:37:36.484076+0000","last_undegraded":"2024-09-18T04:37:36.484076+0000","last_fullsized":"2024-09-18T04:37:36.484076+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T08:46:16.183136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059422499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-18T04:37:36.493770+0000","last_change":"2024-09-18T04:37:36.494228+0000","last_active":"2024-09-18T04:37:36.493770+0000","last_peered":"2024-09-18T04:37:36.493770+0000","last_clean":"2024-09-18T04:37:36.493770+0000","last_became_active":"2024-09-18T04:37:34.471265+0000","last_became_peered":"2024-09-18T04:37:34.471265+0000","last_unstale":"2024-09-18T04:37:36.493770+0000","last_undegraded":"2024-09-18T04:37:36.493770+0000","last_fullsized":"2024-09-18T04:37:36.493770+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:23:05.748076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081514499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"20'2","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-18T04:37:36.490461+0000","last_change":"2024-09-18T04:37:36.490461+0000","last_active":"2024-09-18T04:37:36.490461+0000","last_peered":"2024-09-18T04:37:36.490461+0000","last_clean":"2024-09-18T04:37:36.490461+0000","last_became_active":"2024-09-18T04:37:34.473088+0000","last_became_peered":"2024-09-18T04:37:34.473088+0000","last_unstale":"2024-09-18T04:37:36.490461+0000","last_undegraded":"2024-09-18T04:37:36.490461+0000","last_fullsized":"2024-09-18T04:37:36.490461+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:17:57.352397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001879932,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-18T04:37:36.482703+0000","last_change":"2024-09-18T04:37:36.483033+0000","last_active":"2024-09-18T04:37:36.482703+0000","last_peered":"2024-09-18T04:37:36.482703+0000","last_clean":"2024-09-18T04:37:36.482703+0000","last_became_active":"2024-09-18T04:37:34.471649+0000","last_became_peered":"2024-09-18T04:37:34.471649+0000","last_unstale":"2024-09-18T04:37:36.482703+0000","last_undegraded":"2024-09-18T04:37:36.482703+0000","last_fullsized":"2024-09-18T04:37:36.482703+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T07:21:30.888347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048363399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-18T04:37:36.484618+0000","last_change":"2024-09-18T04:37:36.484950+0000","last_active":"2024-09-18T04:37:36.484618+0000","last_peered":"2024-09-18T04:37:36.484618+0000","last_clean":"2024-09-18T04:37:36.484618+0000","last_became_active":"2024-09-18T04:37:34.470841+0000","last_became_peered":"2024-09-18T04:37:34.470841+0000","last_unstale":"2024-09-18T04:37:36.484618+0000","last_undegraded":"2024-09-18T04:37:36.484618+0000","last_fullsized":"2024-09-18T04:37:36.484618+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:16:34.528783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090513900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"18'1","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-18T04:37:36.489928+0000","last_change":"2024-09-18T04:37:36.490456+0000","last_active":"2024-09-18T04:37:36.489928+0000","last_peered":"2024-09-18T04:37:36.489928+0000","last_clean":"2024-09-18T04:37:36.489928+0000","last_became_active":"2024-09-18T04:37:34.473044+0000","last_became_peered":"2024-09-18T04:37:34.473044+0000","last_unstale":"2024-09-18T04:37:36.489928+0000","last_undegraded":"2024-09-18T04:37:36.489928+0000","last_fullsized":"2024-09-18T04:37:36.489928+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:41.239574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018293420000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"16'117","reported_seq":164,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-18T04:37:35.532099+0000","last_change":"2024-09-18T04:37:27.423069+0000","last_active":"2024-09-18T04:37:35.532099+0000","last_peered":"2024-09-18T04:37:35.532099+0000","last_clean":"2024-09-18T04:37:35.532099+0000","last_became_active":"2024-09-18T04:37:27.422350+0000","last_became_peered":"2024-09-18T04:37:27.422350+0000","last_unstale":"2024-09-18T04:37:35.532099+0000","last_undegraded":"2024-09-18T04:37:35.532099+0000","last_fullsized":"2024-09-18T04:37:35.532099+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:26.406292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:26.406292+0000","last_clean_scrub_stamp":"2024-09-18T04:37:26.406292+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:31:45.852149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":66040,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":192512,"data_stored":62140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79800000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77100000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96399999999999997}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":62121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71399999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.877}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.745}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80200000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0720000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31368,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340472,"statfs":{"total":96636764160,"available":96604643328,"internally_reserved":0,"allocated":253952,"data_stored":652489,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1220000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.073}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91100000000000003}]}]},{"osd":3,"up_from":12,"seq":51539607559,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31364,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340476,"statfs":{"total":96636764160,"available":96604647424,"internally_reserved":0,"allocated":253952,"data_stored":652489,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79900000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91700000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]}]},{"osd":2,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31300,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340540,"statfs":{"total":96636764160,"available":96604712960,"internally_reserved":0,"allocated":188416,"data_stored":62121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81399999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77700000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92200000000000004}]}]},{"osd":1,"up_from":12,"seq":51539607559,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31308,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340532,"statfs":{"total":96636764160,"available":96604704768,"internally_reserved":0,"allocated":192512,"data_stored":62140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93999999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71199999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T04:37:41.678 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-18T04:37:41.678 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-18T04:37:41.678 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-18T04:37:41.678 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-18T04:37:42.007 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:37:42.007 INFO:teuthology.orchestra.run.smithi028.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-18T04:37:42.027 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-18T04:37:42.027 INFO:teuthology.run_tasks:Running task thrashosds... 2024-09-18T04:37:42.037 INFO:tasks.thrashosds:config is {'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-18T04:37:42.037 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'reef'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from'], 'sha1': '19e2a2beb6f41a3ab62650e38614eb29b59dcbb9'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'extra_packages': ['rbd-nbd'], 'flavor': 'default', 'sha1': '19e2a2beb6f41a3ab62650e38614eb29b59dcbb9'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'reef', 'sha1': 'a477b1894d5c5a613f23260ca0e6b96d79652ba7'}} 2024-09-18T04:37:42.037 INFO:tasks.thrashosds:config is {'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5} 2024-09-18T04:37:42.037 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-09-18T04:37:42.359 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-18T04:37:42.360 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-18T04:37:42.360 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T04:37:42.667 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:37:42.670 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-18T04:37:42.683 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-09-18T04:37:41.594477+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":179756,"kb_used_data":1240,"kb_used_omap":9,"kb_used_meta":178422,"kb_avail":566051284,"statfs":{"total":579820584960,"available":579636514816,"internally_reserved":0,"allocated":1269760,"data_stored":1553500,"data_compressed":66040,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":9561,"internal_metadata":182704807},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.132986"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-18T04:37:36.484136+0000","last_change":"2024-09-18T04:37:36.484517+0000","last_active":"2024-09-18T04:37:36.484136+0000","last_peered":"2024-09-18T04:37:36.484136+0000","last_clean":"2024-09-18T04:37:36.484136+0000","last_became_active":"2024-09-18T04:37:34.893674+0000","last_became_peered":"2024-09-18T04:37:34.893674+0000","last_unstale":"2024-09-18T04:37:36.484136+0000","last_undegraded":"2024-09-18T04:37:36.484136+0000","last_fullsized":"2024-09-18T04:37:36.484136+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T09:53:33.883648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047343899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-18T04:37:36.493875+0000","last_change":"2024-09-18T04:37:36.494193+0000","last_active":"2024-09-18T04:37:36.493875+0000","last_peered":"2024-09-18T04:37:36.493875+0000","last_clean":"2024-09-18T04:37:36.493875+0000","last_became_active":"2024-09-18T04:37:34.472940+0000","last_became_peered":"2024-09-18T04:37:34.472940+0000","last_unstale":"2024-09-18T04:37:36.493875+0000","last_undegraded":"2024-09-18T04:37:36.493875+0000","last_fullsized":"2024-09-18T04:37:36.493875+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:18:29.266815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070553900000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-18T04:37:36.484076+0000","last_change":"2024-09-18T04:37:36.484508+0000","last_active":"2024-09-18T04:37:36.484076+0000","last_peered":"2024-09-18T04:37:36.484076+0000","last_clean":"2024-09-18T04:37:36.484076+0000","last_became_active":"2024-09-18T04:37:34.468968+0000","last_became_peered":"2024-09-18T04:37:34.468968+0000","last_unstale":"2024-09-18T04:37:36.484076+0000","last_undegraded":"2024-09-18T04:37:36.484076+0000","last_fullsized":"2024-09-18T04:37:36.484076+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T08:46:16.183136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059422499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-18T04:37:36.493770+0000","last_change":"2024-09-18T04:37:36.494228+0000","last_active":"2024-09-18T04:37:36.493770+0000","last_peered":"2024-09-18T04:37:36.493770+0000","last_clean":"2024-09-18T04:37:36.493770+0000","last_became_active":"2024-09-18T04:37:34.471265+0000","last_became_peered":"2024-09-18T04:37:34.471265+0000","last_unstale":"2024-09-18T04:37:36.493770+0000","last_undegraded":"2024-09-18T04:37:36.493770+0000","last_fullsized":"2024-09-18T04:37:36.493770+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:23:05.748076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081514499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"20'2","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-18T04:37:36.490461+0000","last_change":"2024-09-18T04:37:36.490461+0000","last_active":"2024-09-18T04:37:36.490461+0000","last_peered":"2024-09-18T04:37:36.490461+0000","last_clean":"2024-09-18T04:37:36.490461+0000","last_became_active":"2024-09-18T04:37:34.473088+0000","last_became_peered":"2024-09-18T04:37:34.473088+0000","last_unstale":"2024-09-18T04:37:36.490461+0000","last_undegraded":"2024-09-18T04:37:36.490461+0000","last_fullsized":"2024-09-18T04:37:36.490461+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:17:57.352397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001879932,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-18T04:37:36.482703+0000","last_change":"2024-09-18T04:37:36.483033+0000","last_active":"2024-09-18T04:37:36.482703+0000","last_peered":"2024-09-18T04:37:36.482703+0000","last_clean":"2024-09-18T04:37:36.482703+0000","last_became_active":"2024-09-18T04:37:34.471649+0000","last_became_peered":"2024-09-18T04:37:34.471649+0000","last_unstale":"2024-09-18T04:37:36.482703+0000","last_undegraded":"2024-09-18T04:37:36.482703+0000","last_fullsized":"2024-09-18T04:37:36.482703+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T07:21:30.888347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048363399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-18T04:37:36.484618+0000","last_change":"2024-09-18T04:37:36.484950+0000","last_active":"2024-09-18T04:37:36.484618+0000","last_peered":"2024-09-18T04:37:36.484618+0000","last_clean":"2024-09-18T04:37:36.484618+0000","last_became_active":"2024-09-18T04:37:34.470841+0000","last_became_peered":"2024-09-18T04:37:34.470841+0000","last_unstale":"2024-09-18T04:37:36.484618+0000","last_undegraded":"2024-09-18T04:37:36.484618+0000","last_fullsized":"2024-09-18T04:37:36.484618+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:16:34.528783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090513900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"18'1","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-18T04:37:36.489928+0000","last_change":"2024-09-18T04:37:36.490456+0000","last_active":"2024-09-18T04:37:36.489928+0000","last_peered":"2024-09-18T04:37:36.489928+0000","last_clean":"2024-09-18T04:37:36.489928+0000","last_became_active":"2024-09-18T04:37:34.473044+0000","last_became_peered":"2024-09-18T04:37:34.473044+0000","last_unstale":"2024-09-18T04:37:36.489928+0000","last_undegraded":"2024-09-18T04:37:36.489928+0000","last_fullsized":"2024-09-18T04:37:36.489928+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:41.239574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018293420000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"16'117","reported_seq":164,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-18T04:37:35.532099+0000","last_change":"2024-09-18T04:37:27.423069+0000","last_active":"2024-09-18T04:37:35.532099+0000","last_peered":"2024-09-18T04:37:35.532099+0000","last_clean":"2024-09-18T04:37:35.532099+0000","last_became_active":"2024-09-18T04:37:27.422350+0000","last_became_peered":"2024-09-18T04:37:27.422350+0000","last_unstale":"2024-09-18T04:37:35.532099+0000","last_undegraded":"2024-09-18T04:37:35.532099+0000","last_fullsized":"2024-09-18T04:37:35.532099+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:26.406292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:26.406292+0000","last_clean_scrub_stamp":"2024-09-18T04:37:26.406292+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:31:45.852149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":66040,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":192512,"data_stored":62140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79800000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77100000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96399999999999997}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":62121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71399999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.877}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.745}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80200000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0720000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31368,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340472,"statfs":{"total":96636764160,"available":96604643328,"internally_reserved":0,"allocated":253952,"data_stored":652489,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1220000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.073}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91100000000000003}]}]},{"osd":3,"up_from":12,"seq":51539607559,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31364,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340476,"statfs":{"total":96636764160,"available":96604647424,"internally_reserved":0,"allocated":253952,"data_stored":652489,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79900000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91700000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]}]},{"osd":2,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31300,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340540,"statfs":{"total":96636764160,"available":96604712960,"internally_reserved":0,"allocated":188416,"data_stored":62121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81399999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77700000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92200000000000004}]}]},{"osd":1,"up_from":12,"seq":51539607559,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31308,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340532,"statfs":{"total":96636764160,"available":96604704768,"internally_reserved":0,"allocated":192512,"data_stored":62140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93999999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71199999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T04:37:42.684 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T04:37:42.981 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:37:42.984 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-18T04:37:42.996 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-09-18T04:37:41.594477+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":179756,"kb_used_data":1240,"kb_used_omap":9,"kb_used_meta":178422,"kb_avail":566051284,"statfs":{"total":579820584960,"available":579636514816,"internally_reserved":0,"allocated":1269760,"data_stored":1553500,"data_compressed":66040,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":9561,"internal_metadata":182704807},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.132986"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-18T04:37:36.484136+0000","last_change":"2024-09-18T04:37:36.484517+0000","last_active":"2024-09-18T04:37:36.484136+0000","last_peered":"2024-09-18T04:37:36.484136+0000","last_clean":"2024-09-18T04:37:36.484136+0000","last_became_active":"2024-09-18T04:37:34.893674+0000","last_became_peered":"2024-09-18T04:37:34.893674+0000","last_unstale":"2024-09-18T04:37:36.484136+0000","last_undegraded":"2024-09-18T04:37:36.484136+0000","last_fullsized":"2024-09-18T04:37:36.484136+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T09:53:33.883648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047343899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-18T04:37:36.493875+0000","last_change":"2024-09-18T04:37:36.494193+0000","last_active":"2024-09-18T04:37:36.493875+0000","last_peered":"2024-09-18T04:37:36.493875+0000","last_clean":"2024-09-18T04:37:36.493875+0000","last_became_active":"2024-09-18T04:37:34.472940+0000","last_became_peered":"2024-09-18T04:37:34.472940+0000","last_unstale":"2024-09-18T04:37:36.493875+0000","last_undegraded":"2024-09-18T04:37:36.493875+0000","last_fullsized":"2024-09-18T04:37:36.493875+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:18:29.266815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070553900000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-18T04:37:36.484076+0000","last_change":"2024-09-18T04:37:36.484508+0000","last_active":"2024-09-18T04:37:36.484076+0000","last_peered":"2024-09-18T04:37:36.484076+0000","last_clean":"2024-09-18T04:37:36.484076+0000","last_became_active":"2024-09-18T04:37:34.468968+0000","last_became_peered":"2024-09-18T04:37:34.468968+0000","last_unstale":"2024-09-18T04:37:36.484076+0000","last_undegraded":"2024-09-18T04:37:36.484076+0000","last_fullsized":"2024-09-18T04:37:36.484076+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T08:46:16.183136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059422499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-18T04:37:36.493770+0000","last_change":"2024-09-18T04:37:36.494228+0000","last_active":"2024-09-18T04:37:36.493770+0000","last_peered":"2024-09-18T04:37:36.493770+0000","last_clean":"2024-09-18T04:37:36.493770+0000","last_became_active":"2024-09-18T04:37:34.471265+0000","last_became_peered":"2024-09-18T04:37:34.471265+0000","last_unstale":"2024-09-18T04:37:36.493770+0000","last_undegraded":"2024-09-18T04:37:36.493770+0000","last_fullsized":"2024-09-18T04:37:36.493770+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:23:05.748076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081514499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"20'2","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-18T04:37:36.490461+0000","last_change":"2024-09-18T04:37:36.490461+0000","last_active":"2024-09-18T04:37:36.490461+0000","last_peered":"2024-09-18T04:37:36.490461+0000","last_clean":"2024-09-18T04:37:36.490461+0000","last_became_active":"2024-09-18T04:37:34.473088+0000","last_became_peered":"2024-09-18T04:37:34.473088+0000","last_unstale":"2024-09-18T04:37:36.490461+0000","last_undegraded":"2024-09-18T04:37:36.490461+0000","last_fullsized":"2024-09-18T04:37:36.490461+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:17:57.352397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001879932,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-18T04:37:36.482703+0000","last_change":"2024-09-18T04:37:36.483033+0000","last_active":"2024-09-18T04:37:36.482703+0000","last_peered":"2024-09-18T04:37:36.482703+0000","last_clean":"2024-09-18T04:37:36.482703+0000","last_became_active":"2024-09-18T04:37:34.471649+0000","last_became_peered":"2024-09-18T04:37:34.471649+0000","last_unstale":"2024-09-18T04:37:36.482703+0000","last_undegraded":"2024-09-18T04:37:36.482703+0000","last_fullsized":"2024-09-18T04:37:36.482703+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T07:21:30.888347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048363399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-18T04:37:36.484618+0000","last_change":"2024-09-18T04:37:36.484950+0000","last_active":"2024-09-18T04:37:36.484618+0000","last_peered":"2024-09-18T04:37:36.484618+0000","last_clean":"2024-09-18T04:37:36.484618+0000","last_became_active":"2024-09-18T04:37:34.470841+0000","last_became_peered":"2024-09-18T04:37:34.470841+0000","last_unstale":"2024-09-18T04:37:36.484618+0000","last_undegraded":"2024-09-18T04:37:36.484618+0000","last_fullsized":"2024-09-18T04:37:36.484618+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:16:34.528783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090513900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"18'1","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-18T04:37:36.489928+0000","last_change":"2024-09-18T04:37:36.490456+0000","last_active":"2024-09-18T04:37:36.489928+0000","last_peered":"2024-09-18T04:37:36.489928+0000","last_clean":"2024-09-18T04:37:36.489928+0000","last_became_active":"2024-09-18T04:37:34.473044+0000","last_became_peered":"2024-09-18T04:37:34.473044+0000","last_unstale":"2024-09-18T04:37:36.489928+0000","last_undegraded":"2024-09-18T04:37:36.489928+0000","last_fullsized":"2024-09-18T04:37:36.489928+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:41.239574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018293420000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"16'117","reported_seq":164,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-18T04:37:35.532099+0000","last_change":"2024-09-18T04:37:27.423069+0000","last_active":"2024-09-18T04:37:35.532099+0000","last_peered":"2024-09-18T04:37:35.532099+0000","last_clean":"2024-09-18T04:37:35.532099+0000","last_became_active":"2024-09-18T04:37:27.422350+0000","last_became_peered":"2024-09-18T04:37:27.422350+0000","last_unstale":"2024-09-18T04:37:35.532099+0000","last_undegraded":"2024-09-18T04:37:35.532099+0000","last_fullsized":"2024-09-18T04:37:35.532099+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:26.406292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:26.406292+0000","last_clean_scrub_stamp":"2024-09-18T04:37:26.406292+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:31:45.852149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":66040,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":192512,"data_stored":62140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79800000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77100000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96399999999999997}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":62121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71399999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.877}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.745}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80200000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0720000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31368,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340472,"statfs":{"total":96636764160,"available":96604643328,"internally_reserved":0,"allocated":253952,"data_stored":652489,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1220000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.073}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91100000000000003}]}]},{"osd":3,"up_from":12,"seq":51539607559,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31364,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340476,"statfs":{"total":96636764160,"available":96604647424,"internally_reserved":0,"allocated":253952,"data_stored":652489,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79900000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91700000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]}]},{"osd":2,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31300,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340540,"statfs":{"total":96636764160,"available":96604712960,"internally_reserved":0,"allocated":188416,"data_stored":62121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81399999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77700000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92200000000000004}]}]},{"osd":1,"up_from":12,"seq":51539607559,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31308,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340532,"statfs":{"total":96636764160,"available":96604704768,"internally_reserved":0,"allocated":192512,"data_stored":62140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93999999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71199999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T04:37:42.997 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-18T04:37:42.997 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-09-18T04:37:43.311 INFO:teuthology.orchestra.run.smithi028.stdout:epoch 20 2024-09-18T04:37:43.311 INFO:teuthology.orchestra.run.smithi028.stdout:fsid be6146ba-a991-4f6e-b359-99cbedc9f9b1 2024-09-18T04:37:43.311 INFO:teuthology.orchestra.run.smithi028.stdout:created 2024-09-18T04:37:15.676634+0000 2024-09-18T04:37:43.311 INFO:teuthology.orchestra.run.smithi028.stdout:modified 2024-09-18T04:37:36.470579+0000 2024-09-18T04:37:43.311 INFO:teuthology.orchestra.run.smithi028.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-18T04:37:43.311 INFO:teuthology.orchestra.run.smithi028.stdout:crush_version 5 2024-09-18T04:37:43.311 INFO:teuthology.orchestra.run.smithi028.stdout:full_ratio 0.95 2024-09-18T04:37:43.311 INFO:teuthology.orchestra.run.smithi028.stdout:backfillfull_ratio 0.9 2024-09-18T04:37:43.311 INFO:teuthology.orchestra.run.smithi028.stdout:nearfull_ratio 0.85 2024-09-18T04:37:43.311 INFO:teuthology.orchestra.run.smithi028.stdout:require_min_compat_client luminous 2024-09-18T04:37:43.312 INFO:teuthology.orchestra.run.smithi028.stdout:min_compat_client jewel 2024-09-18T04:37:43.312 INFO:teuthology.orchestra.run.smithi028.stdout:require_osd_release reef 2024-09-18T04:37:43.312 INFO:teuthology.orchestra.run.smithi028.stdout:stretch_mode_enabled false 2024-09-18T04:37:43.312 INFO:teuthology.orchestra.run.smithi028.stdout:pool 1 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 16 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 6.06 2024-09-18T04:37:43.312 INFO:teuthology.orchestra.run.smithi028.stdout:pool 2 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 20 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.25 2024-09-18T04:37:43.312 INFO:teuthology.orchestra.run.smithi028.stdout: removed_snaps_queue [2~1] 2024-09-18T04:37:43.312 INFO:teuthology.orchestra.run.smithi028.stdout:max_osd 6 2024-09-18T04:37:43.312 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.28:6801/1554407851,v1:172.21.15.28:6803/1554407851] [v2:172.21.15.28:6805/1554407851,v1:172.21.15.28:6807/1554407851] exists,up 6177002d-72e6-4888-bcde-29b6a92d9dc4 2024-09-18T04:37:43.312 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 up in weight 1 up_from 12 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:172.21.15.28:6800/511141401,v1:172.21.15.28:6802/511141401] [v2:172.21.15.28:6804/511141401,v1:172.21.15.28:6806/511141401] exists,up 4f62cb09-3457-418c-bb69-491d9b70914c 2024-09-18T04:37:43.312 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 up in weight 1 up_from 12 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:172.21.15.28:6816/2623426735,v1:172.21.15.28:6817/2623426735] [v2:172.21.15.28:6818/2623426735,v1:172.21.15.28:6819/2623426735] exists,up 854fe031-3b87-4290-9c2f-3fb196fb806c 2024-09-18T04:37:43.312 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 up in weight 1 up_from 12 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:172.21.15.82:6808/2746051132,v1:172.21.15.82:6809/2746051132] [v2:172.21.15.82:6810/2746051132,v1:172.21.15.82:6811/2746051132] exists,up c2625f98-780b-4f86-bf36-a48d2a227756 2024-09-18T04:37:43.312 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.82:6800/2005712460,v1:172.21.15.82:6801/2005712460] [v2:172.21.15.82:6802/2005712460,v1:172.21.15.82:6803/2005712460] exists,up deb22d86-4081-484d-96eb-9cb9d96b60bc 2024-09-18T04:37:43.313 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 up in weight 1 up_from 13 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:172.21.15.82:6816/1267644232,v1:172.21.15.82:6817/1267644232] [v2:172.21.15.82:6818/1267644232,v1:172.21.15.82:6819/1267644232] exists,up 061d332c-f51b-403d-bb18-eba415c97e7f 2024-09-18T04:37:43.331 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.28:6801/1554407851,v1:172.21.15.28:6803/1554407851] [v2:172.21.15.28:6805/1554407851,v1:172.21.15.28:6807/1554407851] exists,up 6177002d-72e6-4888-bcde-29b6a92d9dc4', 'osd.1 up in weight 1 up_from 12 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:172.21.15.28:6800/511141401,v1:172.21.15.28:6802/511141401] [v2:172.21.15.28:6804/511141401,v1:172.21.15.28:6806/511141401] exists,up 4f62cb09-3457-418c-bb69-491d9b70914c', 'osd.2 up in weight 1 up_from 12 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:172.21.15.28:6816/2623426735,v1:172.21.15.28:6817/2623426735] [v2:172.21.15.28:6818/2623426735,v1:172.21.15.28:6819/2623426735] exists,up 854fe031-3b87-4290-9c2f-3fb196fb806c', 'osd.3 up in weight 1 up_from 12 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:172.21.15.82:6808/2746051132,v1:172.21.15.82:6809/2746051132] [v2:172.21.15.82:6810/2746051132,v1:172.21.15.82:6811/2746051132] exists,up c2625f98-780b-4f86-bf36-a48d2a227756', 'osd.4 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.82:6800/2005712460,v1:172.21.15.82:6801/2005712460] [v2:172.21.15.82:6802/2005712460,v1:172.21.15.82:6803/2005712460] exists,up deb22d86-4081-484d-96eb-9cb9d96b60bc', 'osd.5 up in weight 1 up_from 13 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:172.21.15.82:6816/1267644232,v1:172.21.15.82:6817/1267644232] [v2:172.21.15.82:6818/1267644232,v1:172.21.15.82:6819/1267644232] exists,up 061d332c-f51b-403d-bb18-eba415c97e7f'] 2024-09-18T04:37:43.331 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-09-18T04:37:43.432 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:37:43.432 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mon.a", 2024-09-18T04:37:43.432 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster": "ceph", 2024-09-18T04:37:43.432 INFO:teuthology.orchestra.run.smithi028.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-18T04:37:43.432 INFO:teuthology.orchestra.run.smithi028.stdout: "admin_socket_mode": "", 2024-09-18T04:37:43.432 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-09-18T04:37:43.432 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_client_required": "cephx, none", 2024-09-18T04:37:43.432 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_cluster_required": "cephx", 2024-09-18T04:37:43.432 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_debug": "true", 2024-09-18T04:37:43.432 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-18T04:37:43.433 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-18T04:37:43.433 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_service_required": "cephx", 2024-09-18T04:37:43.433 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-18T04:37:43.433 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_supported": "cephx", 2024-09-18T04:37:43.433 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_aio": "true", 2024-09-18T04:37:43.433 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-18T04:37:43.433 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_aio_poll_ms": "250", 2024-09-18T04:37:43.433 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_aio_reap_max": "16", 2024-09-18T04:37:43.433 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_async_discard": "false", 2024-09-18T04:37:43.433 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_block_size": "4096", 2024-09-18T04:37:43.433 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_debug_aio": "false", 2024-09-18T04:37:43.433 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-18T04:37:43.433 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-18T04:37:43.434 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_debug_inflight_ios": "false", 2024-09-18T04:37:43.434 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_enable_discard": "false", 2024-09-18T04:37:43.434 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_flock_retry": "3", 2024-09-18T04:37:43.434 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-18T04:37:43.434 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_inject_crash": "0", 2024-09-18T04:37:43.434 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-18T04:37:43.434 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_ioring": "false", 2024-09-18T04:37:43.434 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_ioring_hipri": "false", 2024-09-18T04:37:43.434 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-18T04:37:43.434 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-18T04:37:43.434 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-18T04:37:43.434 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-18T04:37:43.434 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_type": "", 2024-09-18T04:37:43.435 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_alloc_size": "1048576", 2024-09-18T04:37:43.435 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_allocator": "hybrid", 2024-09-18T04:37:43.435 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_buffered_io": "true", 2024-09-18T04:37:43.435 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_check_for_zeros": "false", 2024-09-18T04:37:43.435 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-18T04:37:43.435 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-18T04:37:43.435 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_compact_log_sync": "false", 2024-09-18T04:37:43.435 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-18T04:37:43.435 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-18T04:37:43.435 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-18T04:37:43.435 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-18T04:37:43.435 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_max_log_runway": "4194304", 2024-09-18T04:37:43.435 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_max_prefetch": "1048576", 2024-09-18T04:37:43.436 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_min_flush_size": "524288", 2024-09-18T04:37:43.436 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_min_log_runway": "1048576", 2024-09-18T04:37:43.436 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_replay_recovery": "false", 2024-09-18T04:37:43.436 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-18T04:37:43.436 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-18T04:37:43.436 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_sync_write": "false", 2024-09-18T04:37:43.436 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-18T04:37:43.436 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-18T04:37:43.436 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-18T04:37:43.436 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_allocation_from_file": "true", 2024-09-18T04:37:43.436 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_allocator": "hybrid", 2024-09-18T04:37:43.436 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-18T04:37:43.436 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-18T04:37:43.436 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-18T04:37:43.437 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-18T04:37:43.437 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-18T04:37:43.437 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-18T04:37:43.437 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-18T04:37:43.437 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_block_create": "true", 2024-09-18T04:37:43.437 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_block_db_create": "false", 2024-09-18T04:37:43.437 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_block_db_path": "", 2024-09-18T04:37:43.437 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_block_db_size": "0", 2024-09-18T04:37:43.437 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_block_path": "", 2024-09-18T04:37:43.437 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_block_preallocate_file": "false", 2024-09-18T04:37:43.437 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_block_size": "107374182400", 2024-09-18T04:37:43.437 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_block_wal_create": "false", 2024-09-18T04:37:43.437 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_block_wal_path": "", 2024-09-18T04:37:43.438 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_block_wal_size": "100663296", 2024-09-18T04:37:43.438 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_bluefs": "true", 2024-09-18T04:37:43.438 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-18T04:37:43.438 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-18T04:37:43.438 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-18T04:37:43.438 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-18T04:37:43.438 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-18T04:37:43.438 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-18T04:37:43.438 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-18T04:37:43.438 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-18T04:37:43.438 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_autotune": "true", 2024-09-18T04:37:43.438 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-18T04:37:43.438 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-18T04:37:43.439 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-18T04:37:43.439 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-18T04:37:43.439 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_size": "0", 2024-09-18T04:37:43.439 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-18T04:37:43.439 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-18T04:37:43.439 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-18T04:37:43.439 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-18T04:37:43.439 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_type": "2q", 2024-09-18T04:37:43.439 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-18T04:37:43.439 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_clone_cow": "true", 2024-09-18T04:37:43.439 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-18T04:37:43.439 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-18T04:37:43.439 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-18T04:37:43.440 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-18T04:37:43.440 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-18T04:37:43.440 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-18T04:37:43.440 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-18T04:37:43.440 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_compression_mode": "none", 2024-09-18T04:37:43.440 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-18T04:37:43.440 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_csum_type": "crc32c", 2024-09-18T04:37:43.440 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-18T04:37:43.440 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_freelist": "false", 2024-09-18T04:37:43.440 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-18T04:37:43.440 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-18T04:37:43.440 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_inject_bug21040": "false", 2024-09-18T04:37:43.440 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-18T04:37:43.441 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-18T04:37:43.441 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-18T04:37:43.441 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-18T04:37:43.441 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-18T04:37:43.441 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-18T04:37:43.441 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-18T04:37:43.441 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_prefill": "0.000000", 2024-09-18T04:37:43.441 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-18T04:37:43.441 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-18T04:37:43.441 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-18T04:37:43.441 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_small_allocations": "0", 2024-09-18T04:37:43.441 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-18T04:37:43.441 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_default_buffered_read": "true", 2024-09-18T04:37:43.442 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_default_buffered_write": "false", 2024-09-18T04:37:43.442 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-18T04:37:43.442 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-18T04:37:43.442 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-18T04:37:43.442 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-18T04:37:43.442 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-18T04:37:43.442 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-18T04:37:43.442 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-18T04:37:43.442 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-18T04:37:43.442 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fail_eio": "false", 2024-09-18T04:37:43.442 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-18T04:37:43.442 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-18T04:37:43.442 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-18T04:37:43.442 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-18T04:37:43.443 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-18T04:37:43.443 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-18T04:37:43.443 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_on_mount": "false", 2024-09-18T04:37:43.443 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-18T04:37:43.443 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_on_umount": "false", 2024-09-18T04:37:43.443 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-18T04:37:43.443 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-18T04:37:43.443 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-18T04:37:43.443 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-18T04:37:43.443 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-18T04:37:43.443 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-18T04:37:43.443 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-18T04:37:43.443 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-18T04:37:43.443 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_ignore_data_csum": "false", 2024-09-18T04:37:43.444 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-18T04:37:43.444 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-18T04:37:43.444 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-18T04:37:43.444 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-18T04:37:43.444 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_log_op_age": "5.000000", 2024-09-18T04:37:43.444 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_max_alloc_size": "0", 2024-09-18T04:37:43.444 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_max_blob_size": "0", 2024-09-18T04:37:43.444 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-18T04:37:43.444 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-18T04:37:43.444 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-18T04:37:43.444 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_max_deferred_txc": "32", 2024-09-18T04:37:43.444 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_min_alloc_size": "0", 2024-09-18T04:37:43.444 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-18T04:37:43.445 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-18T04:37:43.445 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_nid_prealloc": "1024", 2024-09-18T04:37:43.445 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-18T04:37:43.445 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-18T04:37:43.445 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-18T04:37:43.445 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-18T04:37:43.445 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_retry_disk_reads": "3", 2024-09-18T04:37:43.445 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_rocksdb_cf": "true", 2024-09-18T04:37:43.445 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-09-18T04:37:43.445 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-09-18T04:37:43.445 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-18T04:37:43.445 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-18T04:37:43.445 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-18T04:37:43.446 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-18T04:37:43.446 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_spdk_mem": "512", 2024-09-18T04:37:43.446 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-18T04:37:43.446 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-18T04:37:43.446 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-18T04:37:43.446 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-18T04:37:43.446 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-18T04:37:43.446 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-18T04:37:43.446 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-18T04:37:43.446 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_tracing": "false", 2024-09-18T04:37:43.446 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-18T04:37:43.446 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-18T04:37:43.446 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-18T04:37:43.446 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-18T04:37:43.447 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-18T04:37:43.447 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-18T04:37:43.447 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-18T04:37:43.447 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-18T04:37:43.447 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-18T04:37:43.447 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_zero_block_detection": "false", 2024-09-18T04:37:43.447 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-18T04:37:43.447 INFO:teuthology.orchestra.run.smithi028.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-18T04:37:43.447 INFO:teuthology.orchestra.run.smithi028.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-18T04:37:43.447 INFO:teuthology.orchestra.run.smithi028.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-18T04:37:43.447 INFO:teuthology.orchestra.run.smithi028.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-18T04:37:43.447 INFO:teuthology.orchestra.run.smithi028.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-18T04:37:43.447 INFO:teuthology.orchestra.run.smithi028.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-18T04:37:43.448 INFO:teuthology.orchestra.run.smithi028.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-09-18T04:37:43.448 INFO:teuthology.orchestra.run.smithi028.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-18T04:37:43.448 INFO:teuthology.orchestra.run.smithi028.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-18T04:37:43.448 INFO:teuthology.orchestra.run.smithi028.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-18T04:37:43.448 INFO:teuthology.orchestra.run.smithi028.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-18T04:37:43.448 INFO:teuthology.orchestra.run.smithi028.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-18T04:37:43.448 INFO:teuthology.orchestra.run.smithi028.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-18T04:37:43.448 INFO:teuthology.orchestra.run.smithi028.stdout: "cephx_cluster_require_signatures": "false", 2024-09-18T04:37:43.448 INFO:teuthology.orchestra.run.smithi028.stdout: "cephx_cluster_require_version": "2", 2024-09-18T04:37:43.448 INFO:teuthology.orchestra.run.smithi028.stdout: "cephx_require_signatures": "false", 2024-09-18T04:37:43.448 INFO:teuthology.orchestra.run.smithi028.stdout: "cephx_require_version": "2", 2024-09-18T04:37:43.448 INFO:teuthology.orchestra.run.smithi028.stdout: "cephx_service_require_signatures": "false", 2024-09-18T04:37:43.448 INFO:teuthology.orchestra.run.smithi028.stdout: "cephx_service_require_version": "2", 2024-09-18T04:37:43.449 INFO:teuthology.orchestra.run.smithi028.stdout: "cephx_sign_messages": "true", 2024-09-18T04:37:43.449 INFO:teuthology.orchestra.run.smithi028.stdout: "chdir": "", 2024-09-18T04:37:43.449 INFO:teuthology.orchestra.run.smithi028.stdout: "client_acl_type": "", 2024-09-18T04:37:43.449 INFO:teuthology.orchestra.run.smithi028.stdout: "client_asio_thread_count": "2", 2024-09-18T04:37:43.449 INFO:teuthology.orchestra.run.smithi028.stdout: "client_cache_mid": "0.750000", 2024-09-18T04:37:43.449 INFO:teuthology.orchestra.run.smithi028.stdout: "client_cache_size": "16384", 2024-09-18T04:37:43.449 INFO:teuthology.orchestra.run.smithi028.stdout: "client_caps_release_delay": "5", 2024-09-18T04:37:43.449 INFO:teuthology.orchestra.run.smithi028.stdout: "client_check_pool_perm": "true", 2024-09-18T04:37:43.449 INFO:teuthology.orchestra.run.smithi028.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-18T04:37:43.449 INFO:teuthology.orchestra.run.smithi028.stdout: "client_debug_force_sync_read": "false", 2024-09-18T04:37:43.449 INFO:teuthology.orchestra.run.smithi028.stdout: "client_debug_getattr_caps": "false", 2024-09-18T04:37:43.449 INFO:teuthology.orchestra.run.smithi028.stdout: "client_debug_inject_features": "", 2024-09-18T04:37:43.449 INFO:teuthology.orchestra.run.smithi028.stdout: "client_debug_inject_tick_delay": "0", 2024-09-18T04:37:43.450 INFO:teuthology.orchestra.run.smithi028.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-18T04:37:43.450 INFO:teuthology.orchestra.run.smithi028.stdout: "client_die_on_failed_remount": "false", 2024-09-18T04:37:43.450 INFO:teuthology.orchestra.run.smithi028.stdout: "client_dirsize_rbytes": "true", 2024-09-18T04:37:43.450 INFO:teuthology.orchestra.run.smithi028.stdout: "client_force_lazyio": "false", 2024-09-18T04:37:43.450 INFO:teuthology.orchestra.run.smithi028.stdout: "client_fs": "", 2024-09-18T04:37:43.450 INFO:teuthology.orchestra.run.smithi028.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-18T04:37:43.450 INFO:teuthology.orchestra.run.smithi028.stdout: "client_inject_release_failure": "false", 2024-09-18T04:37:43.450 INFO:teuthology.orchestra.run.smithi028.stdout: "client_max_inline_size": "4096", 2024-09-18T04:37:43.450 INFO:teuthology.orchestra.run.smithi028.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-18T04:37:43.450 INFO:teuthology.orchestra.run.smithi028.stdout: "client_mds_namespace": "", 2024-09-18T04:37:43.450 INFO:teuthology.orchestra.run.smithi028.stdout: "client_metadata": "", 2024-09-18T04:37:43.450 INFO:teuthology.orchestra.run.smithi028.stdout: "client_mount_gid": "-1", 2024-09-18T04:37:43.450 INFO:teuthology.orchestra.run.smithi028.stdout: "client_mount_timeout": "300", 2024-09-18T04:37:43.451 INFO:teuthology.orchestra.run.smithi028.stdout: "client_mount_uid": "-1", 2024-09-18T04:37:43.451 INFO:teuthology.orchestra.run.smithi028.stdout: "client_mountpoint": "/", 2024-09-18T04:37:43.451 INFO:teuthology.orchestra.run.smithi028.stdout: "client_notify_timeout": "10", 2024-09-18T04:37:43.451 INFO:teuthology.orchestra.run.smithi028.stdout: "client_oc": "true", 2024-09-18T04:37:43.451 INFO:teuthology.orchestra.run.smithi028.stdout: "client_oc_max_dirty": "104857600", 2024-09-18T04:37:43.451 INFO:teuthology.orchestra.run.smithi028.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-18T04:37:43.451 INFO:teuthology.orchestra.run.smithi028.stdout: "client_oc_max_objects": "1000", 2024-09-18T04:37:43.451 INFO:teuthology.orchestra.run.smithi028.stdout: "client_oc_size": "209715200", 2024-09-18T04:37:43.451 INFO:teuthology.orchestra.run.smithi028.stdout: "client_oc_target_dirty": "8388608", 2024-09-18T04:37:43.451 INFO:teuthology.orchestra.run.smithi028.stdout: "client_permissions": "true", 2024-09-18T04:37:43.451 INFO:teuthology.orchestra.run.smithi028.stdout: "client_quota": "true", 2024-09-18T04:37:43.451 INFO:teuthology.orchestra.run.smithi028.stdout: "client_quota_df": "true", 2024-09-18T04:37:43.451 INFO:teuthology.orchestra.run.smithi028.stdout: "client_readahead_max_bytes": "0", 2024-09-18T04:37:43.451 INFO:teuthology.orchestra.run.smithi028.stdout: "client_readahead_max_periods": "4", 2024-09-18T04:37:43.452 INFO:teuthology.orchestra.run.smithi028.stdout: "client_readahead_min": "131072", 2024-09-18T04:37:43.452 INFO:teuthology.orchestra.run.smithi028.stdout: "client_reconnect_stale": "false", 2024-09-18T04:37:43.452 INFO:teuthology.orchestra.run.smithi028.stdout: "client_shutdown_timeout": "30", 2024-09-18T04:37:43.452 INFO:teuthology.orchestra.run.smithi028.stdout: "client_snapdir": ".snap", 2024-09-18T04:37:43.452 INFO:teuthology.orchestra.run.smithi028.stdout: "client_tick_interval": "1", 2024-09-18T04:37:43.452 INFO:teuthology.orchestra.run.smithi028.stdout: "client_trace": "", 2024-09-18T04:37:43.452 INFO:teuthology.orchestra.run.smithi028.stdout: "client_try_dentry_invalidate": "false", 2024-09-18T04:37:43.452 INFO:teuthology.orchestra.run.smithi028.stdout: "client_use_faked_inos": "false", 2024-09-18T04:37:43.452 INFO:teuthology.orchestra.run.smithi028.stdout: "client_use_random_mds": "false", 2024-09-18T04:37:43.452 INFO:teuthology.orchestra.run.smithi028.stdout: "clog_to_graylog": "false", 2024-09-18T04:37:43.452 INFO:teuthology.orchestra.run.smithi028.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-18T04:37:43.452 INFO:teuthology.orchestra.run.smithi028.stdout: "clog_to_graylog_port": "12201", 2024-09-18T04:37:43.452 INFO:teuthology.orchestra.run.smithi028.stdout: "clog_to_monitors": "default=true", 2024-09-18T04:37:43.453 INFO:teuthology.orchestra.run.smithi028.stdout: "clog_to_syslog": "false", 2024-09-18T04:37:43.453 INFO:teuthology.orchestra.run.smithi028.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-18T04:37:43.453 INFO:teuthology.orchestra.run.smithi028.stdout: "clog_to_syslog_level": "info", 2024-09-18T04:37:43.453 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "-", 2024-09-18T04:37:43.453 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_network": "", 2024-09-18T04:37:43.453 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_network_interface": "", 2024-09-18T04:37:43.453 INFO:teuthology.orchestra.run.smithi028.stdout: "compressor_zlib_isal": "false", 2024-09-18T04:37:43.453 INFO:teuthology.orchestra.run.smithi028.stdout: "compressor_zlib_level": "5", 2024-09-18T04:37:43.453 INFO:teuthology.orchestra.run.smithi028.stdout: "compressor_zlib_winsize": "-15", 2024-09-18T04:37:43.453 INFO:teuthology.orchestra.run.smithi028.stdout: "compressor_zstd_level": "1", 2024-09-18T04:37:43.453 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-09-18T04:37:43.453 INFO:teuthology.orchestra.run.smithi028.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-18T04:37:43.453 INFO:teuthology.orchestra.run.smithi028.stdout: "crimson_alien_op_num_threads": "6", 2024-09-18T04:37:43.453 INFO:teuthology.orchestra.run.smithi028.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-18T04:37:43.454 INFO:teuthology.orchestra.run.smithi028.stdout: "crimson_osd_obc_lru_size": "10", 2024-09-18T04:37:43.454 INFO:teuthology.orchestra.run.smithi028.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-18T04:37:43.454 INFO:teuthology.orchestra.run.smithi028.stdout: "crimson_seastar_smp": "1", 2024-09-18T04:37:43.454 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "", 2024-09-18T04:37:43.454 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location_hook": "", 2024-09-18T04:37:43.454 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location_hook_timeout": "10", 2024-09-18T04:37:43.454 INFO:teuthology.orchestra.run.smithi028.stdout: "daemonize": "false", 2024-09-18T04:37:43.454 INFO:teuthology.orchestra.run.smithi028.stdout: "daos_pool": "tank", 2024-09-18T04:37:43.454 INFO:teuthology.orchestra.run.smithi028.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-09-18T04:37:43.454 INFO:teuthology.orchestra.run.smithi028.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-09-18T04:37:43.454 INFO:teuthology.orchestra.run.smithi028.stdout: "dbstore_db_name_prefix": "dbstore", 2024-09-18T04:37:43.454 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_alienstore": "0/5", 2024-09-18T04:37:43.454 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_allow_any_pool_priority": "false", 2024-09-18T04:37:43.455 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_asok": "1/5", 2024-09-18T04:37:43.455 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_asok_assert_abort": "false", 2024-09-18T04:37:43.455 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_asserts_on_shutdown": "true", 2024-09-18T04:37:43.455 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_auth": "20/20", 2024-09-18T04:37:43.455 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_bdev": "1/3", 2024-09-18T04:37:43.455 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_bluefs": "1/5", 2024-09-18T04:37:43.455 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_bluestore": "1/5", 2024-09-18T04:37:43.455 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_buffer": "0/1", 2024-09-18T04:37:43.455 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_ceph_exporter": "1/5", 2024-09-18T04:37:43.455 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_cephfs_mirror": "0/5", 2024-09-18T04:37:43.455 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_cephsqlite": "0/5", 2024-09-18T04:37:43.455 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_client": "0/5", 2024-09-18T04:37:43.455 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_compressor": "1/5", 2024-09-18T04:37:43.455 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_context": "0/1", 2024-09-18T04:37:43.455 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_crush": "1/1", 2024-09-18T04:37:43.456 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_crypto": "1/5", 2024-09-18T04:37:43.456 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_cyanstore": "0/5", 2024-09-18T04:37:43.456 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_deliberately_leak_memory": "false", 2024-09-18T04:37:43.456 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_disable_randomized_ping": "false", 2024-09-18T04:37:43.456 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_dpdk": "1/5", 2024-09-18T04:37:43.456 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_eventtrace": "1/5", 2024-09-18T04:37:43.456 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_filer": "0/1", 2024-09-18T04:37:43.456 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_filestore": "1/3", 2024-09-18T04:37:43.456 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_finisher": "1/1", 2024-09-18T04:37:43.456 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_fuse": "1/5", 2024-09-18T04:37:43.456 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_heartbeat_testing_span": "0", 2024-09-18T04:37:43.456 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_heartbeatmap": "1/5", 2024-09-18T04:37:43.456 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-18T04:37:43.456 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_javaclient": "1/5", 2024-09-18T04:37:43.457 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_journal": "1/3", 2024-09-18T04:37:43.457 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_journaler": "0/5", 2024-09-18T04:37:43.457 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_kstore": "1/5", 2024-09-18T04:37:43.457 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_leveldb": "4/5", 2024-09-18T04:37:43.457 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_lockdep": "0/1", 2024-09-18T04:37:43.457 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_mclock": "1/5", 2024-09-18T04:37:43.457 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_mds": "1/5", 2024-09-18T04:37:43.457 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_mds_balancer": "1/5", 2024-09-18T04:37:43.457 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_mds_locker": "1/5", 2024-09-18T04:37:43.457 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_mds_log": "1/5", 2024-09-18T04:37:43.457 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_mds_log_expire": "1/5", 2024-09-18T04:37:43.457 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_mds_migrator": "1/5", 2024-09-18T04:37:43.457 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_memstore": "1/5", 2024-09-18T04:37:43.457 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_mgr": "2/5", 2024-09-18T04:37:43.457 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_mgrc": "1/5", 2024-09-18T04:37:43.458 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_mon": "20/20", 2024-09-18T04:37:43.458 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_monc": "0/10", 2024-09-18T04:37:43.458 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_ms": "1/1", 2024-09-18T04:37:43.458 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_none": "0/5", 2024-09-18T04:37:43.458 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_objclass": "0/5", 2024-09-18T04:37:43.458 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_objectcacher": "0/5", 2024-09-18T04:37:43.458 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_objecter": "0/1", 2024-09-18T04:37:43.458 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_optracker": "0/5", 2024-09-18T04:37:43.458 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_osd": "1/5", 2024-09-18T04:37:43.458 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_paxos": "20/20", 2024-09-18T04:37:43.458 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_perfcounter": "1/5", 2024-09-18T04:37:43.458 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_prioritycache": "1/5", 2024-09-18T04:37:43.458 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rados": "0/5", 2024-09-18T04:37:43.458 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rbd": "0/5", 2024-09-18T04:37:43.458 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rbd_mirror": "0/5", 2024-09-18T04:37:43.459 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rbd_pwl": "0/5", 2024-09-18T04:37:43.459 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rbd_replay": "0/5", 2024-09-18T04:37:43.459 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_refs": "0/0", 2024-09-18T04:37:43.459 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_reserver": "1/1", 2024-09-18T04:37:43.459 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rgw": "1/5", 2024-09-18T04:37:43.459 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rgw_access": "1/5", 2024-09-18T04:37:43.459 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rgw_datacache": "1/5", 2024-09-18T04:37:43.459 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rgw_dbstore": "1/5", 2024-09-18T04:37:43.459 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rgw_flight": "1/5", 2024-09-18T04:37:43.459 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rgw_sync": "1/5", 2024-09-18T04:37:43.459 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rocksdb": "4/5", 2024-09-18T04:37:43.459 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore": "0/5", 2024-09-18T04:37:43.459 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_backref": "0/5", 2024-09-18T04:37:43.459 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_cache": "0/5", 2024-09-18T04:37:43.460 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_cleaner": "0/5", 2024-09-18T04:37:43.460 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_device": "0/5", 2024-09-18T04:37:43.460 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_epm": "0/5", 2024-09-18T04:37:43.460 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-09-18T04:37:43.460 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_journal": "0/5", 2024-09-18T04:37:43.460 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_lba": "0/5", 2024-09-18T04:37:43.460 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_odata": "0/5", 2024-09-18T04:37:43.460 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_omap": "0/5", 2024-09-18T04:37:43.460 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_onode": "0/5", 2024-09-18T04:37:43.460 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_t": "0/5", 2024-09-18T04:37:43.460 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_tm": "0/5", 2024-09-18T04:37:43.460 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_striper": "0/1", 2024-09-18T04:37:43.460 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_test": "0/5", 2024-09-18T04:37:43.460 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_throttle": "1/1", 2024-09-18T04:37:43.461 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_timer": "0/1", 2024-09-18T04:37:43.461 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_tp": "0/5", 2024-09-18T04:37:43.461 INFO:teuthology.orchestra.run.smithi028.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-18T04:37:43.461 INFO:teuthology.orchestra.run.smithi028.stdout: "device_failure_prediction_mode": "none", 2024-09-18T04:37:43.461 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-18T04:37:43.461 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_dir": "/usr/lib/ceph/erasure-code", 2024-09-18T04:37:43.461 INFO:teuthology.orchestra.run.smithi028.stdout: "err_to_graylog": "false", 2024-09-18T04:37:43.461 INFO:teuthology.orchestra.run.smithi028.stdout: "err_to_journald": "false", 2024-09-18T04:37:43.461 INFO:teuthology.orchestra.run.smithi028.stdout: "err_to_stderr": "true", 2024-09-18T04:37:43.461 INFO:teuthology.orchestra.run.smithi028.stdout: "err_to_syslog": "false", 2024-09-18T04:37:43.461 INFO:teuthology.orchestra.run.smithi028.stdout: "event_tracing": "false", 2024-09-18T04:37:43.461 INFO:teuthology.orchestra.run.smithi028.stdout: "exporter_addr": "0.0.0.0", 2024-09-18T04:37:43.461 INFO:teuthology.orchestra.run.smithi028.stdout: "exporter_http_port": "9926", 2024-09-18T04:37:43.462 INFO:teuthology.orchestra.run.smithi028.stdout: "exporter_prio_limit": "5", 2024-09-18T04:37:43.462 INFO:teuthology.orchestra.run.smithi028.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-18T04:37:43.462 INFO:teuthology.orchestra.run.smithi028.stdout: "exporter_sort_metrics": "true", 2024-09-18T04:37:43.462 INFO:teuthology.orchestra.run.smithi028.stdout: "exporter_stats_period": "5", 2024-09-18T04:37:43.462 INFO:teuthology.orchestra.run.smithi028.stdout: "fake_statfs_for_testing": "0", 2024-09-18T04:37:43.462 INFO:teuthology.orchestra.run.smithi028.stdout: "fatal_signal_handlers": "true", 2024-09-18T04:37:43.462 INFO:teuthology.orchestra.run.smithi028.stdout: "filer_max_purge_ops": "10", 2024-09-18T04:37:43.462 INFO:teuthology.orchestra.run.smithi028.stdout: "filer_max_truncate_ops": "128", 2024-09-18T04:37:43.462 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_apply_finisher_threads": "1", 2024-09-18T04:37:43.462 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_blackhole": "false", 2024-09-18T04:37:43.462 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_btrfs_clone_range": "true", 2024-09-18T04:37:43.462 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_btrfs_snap": "true", 2024-09-18T04:37:43.462 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_caller_concurrency": "10", 2024-09-18T04:37:43.462 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_collect_device_partition_information": "true", 2024-09-18T04:37:43.463 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_commit_timeout": "600.000000", 2024-09-18T04:37:43.463 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_debug_inject_read_err": "false", 2024-09-18T04:37:43.463 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_debug_omap_check": "false", 2024-09-18T04:37:43.463 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_debug_random_read_err": "0.000000", 2024-09-18T04:37:43.463 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_debug_verify_split": "false", 2024-09-18T04:37:43.463 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_dump_file": "", 2024-09-18T04:37:43.463 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-18T04:37:43.463 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-18T04:37:43.463 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_fadvise": "true", 2024-09-18T04:37:43.463 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_fail_eio": "true", 2024-09-18T04:37:43.463 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_fd_cache_shards": "16", 2024-09-18T04:37:43.463 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_fd_cache_size": "128", 2024-09-18T04:37:43.463 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_fiemap": "false", 2024-09-18T04:37:43.463 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_fiemap_threshold": "4096", 2024-09-18T04:37:43.464 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-18T04:37:43.464 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-18T04:37:43.464 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_inject_stall": "0", 2024-09-18T04:37:43.464 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_journal_parallel": "false", 2024-09-18T04:37:43.464 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_journal_trailing": "false", 2024-09-18T04:37:43.464 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_journal_writeahead": "false", 2024-09-18T04:37:43.464 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_kill_at": "0", 2024-09-18T04:37:43.464 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-18T04:37:43.464 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-18T04:37:43.464 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-18T04:37:43.464 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-18T04:37:43.464 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-18T04:37:43.464 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_inline_xattrs": "0", 2024-09-18T04:37:43.464 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-18T04:37:43.465 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-18T04:37:43.465 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-18T04:37:43.465 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-18T04:37:43.465 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_xattr_value_size": "0", 2024-09-18T04:37:43.465 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-18T04:37:43.465 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-18T04:37:43.465 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-18T04:37:43.465 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_merge_threshold": "-10", 2024-09-18T04:37:43.465 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-18T04:37:43.465 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_odsync_write": "false", 2024-09-18T04:37:43.465 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_omap_backend": "rocksdb", 2024-09-18T04:37:43.465 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_omap_backend_path": "", 2024-09-18T04:37:43.465 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-18T04:37:43.466 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-18T04:37:43.466 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-18T04:37:43.466 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_op_thread_timeout": "60", 2024-09-18T04:37:43.466 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_op_threads": "2", 2024-09-18T04:37:43.466 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_punch_hole": "false", 2024-09-18T04:37:43.466 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-18T04:37:43.466 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-18T04:37:43.466 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-18T04:37:43.466 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-18T04:37:43.466 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-18T04:37:43.466 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-18T04:37:43.466 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-18T04:37:43.466 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-18T04:37:43.467 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-18T04:37:43.467 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_queue_max_ops": "50", 2024-09-18T04:37:43.467 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-18T04:37:43.467 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_seek_data_hole": "false", 2024-09-18T04:37:43.467 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_sloppy_crc": "false", 2024-09-18T04:37:43.467 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-18T04:37:43.467 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_splice": "false", 2024-09-18T04:37:43.467 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_split_multiple": "2", 2024-09-18T04:37:43.467 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_split_rand_factor": "20", 2024-09-18T04:37:43.467 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_update_to": "1000", 2024-09-18T04:37:43.467 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-18T04:37:43.468 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-18T04:37:43.468 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-18T04:37:43.468 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-18T04:37:43.468 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-18T04:37:43.468 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-18T04:37:43.468 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_enable": "true", 2024-09-18T04:37:43.468 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-18T04:37:43.468 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-18T04:37:43.468 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-18T04:37:43.468 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-18T04:37:43.468 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-18T04:37:43.468 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-18T04:37:43.469 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_xfs_extsize": "false", 2024-09-18T04:37:43.469 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_zfs_snap": "false", 2024-09-18T04:37:43.469 INFO:teuthology.orchestra.run.smithi028.stdout: "fio_dir": "/tmp/fio", 2024-09-18T04:37:43.469 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "be6146ba-a991-4f6e-b359-99cbedc9f9b1", 2024-09-18T04:37:43.469 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_allow_other": "true", 2024-09-18T04:37:43.469 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_atomic_o_trunc": "true", 2024-09-18T04:37:43.469 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_big_writes": "true", 2024-09-18T04:37:43.469 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_debug": "false", 2024-09-18T04:37:43.469 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_default_permissions": "false", 2024-09-18T04:37:43.469 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_disable_pagecache": "false", 2024-09-18T04:37:43.470 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_max_write": "0", 2024-09-18T04:37:43.470 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_multithreaded": "true", 2024-09-18T04:37:43.470 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_require_active_mds": "true", 2024-09-18T04:37:43.470 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_set_user_groups": "true", 2024-09-18T04:37:43.470 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_splice_move": "true", 2024-09-18T04:37:43.470 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_splice_read": "true", 2024-09-18T04:37:43.470 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_splice_write": "true", 2024-09-18T04:37:43.470 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-18T04:37:43.470 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_use_invalidate_cb": "true", 2024-09-18T04:37:43.471 INFO:teuthology.orchestra.run.smithi028.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-18T04:37:43.471 INFO:teuthology.orchestra.run.smithi028.stdout: "gss_target_name": "ceph", 2024-09-18T04:37:43.471 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_file": "", 2024-09-18T04:37:43.471 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_inject_failure": "0", 2024-09-18T04:37:43.471 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_interval": "5", 2024-09-18T04:37:43.471 INFO:teuthology.orchestra.run.smithi028.stdout: "host": "smithi028", 2024-09-18T04:37:43.471 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-18T04:37:43.471 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-18T04:37:43.471 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-18T04:37:43.471 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-18T04:37:43.471 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-18T04:37:43.471 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-18T04:37:43.471 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-18T04:37:43.472 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-18T04:37:43.472 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-18T04:37:43.472 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-18T04:37:43.472 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-18T04:37:43.472 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-18T04:37:43.472 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-18T04:37:43.472 INFO:teuthology.orchestra.run.smithi028.stdout: "inject_early_sigterm": "false", 2024-09-18T04:37:43.472 INFO:teuthology.orchestra.run.smithi028.stdout: "jaeger_agent_port": "6799", 2024-09-18T04:37:43.472 INFO:teuthology.orchestra.run.smithi028.stdout: "jaeger_tracing_enable": "false", 2024-09-18T04:37:43.472 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_aio": "true", 2024-09-18T04:37:43.472 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_align_min_size": "65536", 2024-09-18T04:37:43.472 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_block_align": "true", 2024-09-18T04:37:43.473 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_block_size": "4096", 2024-09-18T04:37:43.473 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_dio": "true", 2024-09-18T04:37:43.473 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_discard": "false", 2024-09-18T04:37:43.473 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_force_aio": "false", 2024-09-18T04:37:43.473 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_ignore_corruption": "false", 2024-09-18T04:37:43.473 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_max_write_bytes": "10485760", 2024-09-18T04:37:43.473 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_max_write_entries": "100", 2024-09-18T04:37:43.473 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_replay_from": "0", 2024-09-18T04:37:43.473 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-18T04:37:43.473 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-18T04:37:43.473 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-18T04:37:43.473 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-18T04:37:43.473 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_write_header_frequency": "0", 2024-09-18T04:37:43.474 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_zero_on_create": "false", 2024-09-18T04:37:43.474 INFO:teuthology.orchestra.run.smithi028.stdout: "journaler_prefetch_periods": "10", 2024-09-18T04:37:43.474 INFO:teuthology.orchestra.run.smithi028.stdout: "journaler_prezero_periods": "5", 2024-09-18T04:37:43.474 INFO:teuthology.orchestra.run.smithi028.stdout: "journaler_write_head_interval": "15", 2024-09-18T04:37:43.474 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "", 2024-09-18T04:37:43.474 INFO:teuthology.orchestra.run.smithi028.stdout: "keyfile": "", 2024-09-18T04:37:43.474 INFO:teuthology.orchestra.run.smithi028.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-18T04:37:43.474 INFO:teuthology.orchestra.run.smithi028.stdout: "kstore_backend": "rocksdb", 2024-09-18T04:37:43.474 INFO:teuthology.orchestra.run.smithi028.stdout: "kstore_default_stripe_size": "65536", 2024-09-18T04:37:43.474 INFO:teuthology.orchestra.run.smithi028.stdout: "kstore_fsck_on_mount": "false", 2024-09-18T04:37:43.474 INFO:teuthology.orchestra.run.smithi028.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-18T04:37:43.474 INFO:teuthology.orchestra.run.smithi028.stdout: "kstore_max_bytes": "67108864", 2024-09-18T04:37:43.474 INFO:teuthology.orchestra.run.smithi028.stdout: "kstore_max_ops": "512", 2024-09-18T04:37:43.475 INFO:teuthology.orchestra.run.smithi028.stdout: "kstore_nid_prealloc": "1024", 2024-09-18T04:37:43.475 INFO:teuthology.orchestra.run.smithi028.stdout: "kstore_onode_map_size": "1024", 2024-09-18T04:37:43.475 INFO:teuthology.orchestra.run.smithi028.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-18T04:37:43.475 INFO:teuthology.orchestra.run.smithi028.stdout: "kstore_sync_submit_transaction": "false", 2024-09-18T04:37:43.475 INFO:teuthology.orchestra.run.smithi028.stdout: "kstore_sync_transaction": "false", 2024-09-18T04:37:43.475 INFO:teuthology.orchestra.run.smithi028.stdout: "leveldb_block_size": "65536", 2024-09-18T04:37:43.475 INFO:teuthology.orchestra.run.smithi028.stdout: "leveldb_bloom_size": "0", 2024-09-18T04:37:43.475 INFO:teuthology.orchestra.run.smithi028.stdout: "leveldb_cache_size": "536870912", 2024-09-18T04:37:43.475 INFO:teuthology.orchestra.run.smithi028.stdout: "leveldb_compact_on_mount": "false", 2024-09-18T04:37:43.475 INFO:teuthology.orchestra.run.smithi028.stdout: "leveldb_compression": "false", 2024-09-18T04:37:43.475 INFO:teuthology.orchestra.run.smithi028.stdout: "leveldb_log": "", 2024-09-18T04:37:43.475 INFO:teuthology.orchestra.run.smithi028.stdout: "leveldb_log_to_ceph_log": "true", 2024-09-18T04:37:43.475 INFO:teuthology.orchestra.run.smithi028.stdout: "leveldb_max_open_files": "0", 2024-09-18T04:37:43.475 INFO:teuthology.orchestra.run.smithi028.stdout: "leveldb_paranoid": "false", 2024-09-18T04:37:43.476 INFO:teuthology.orchestra.run.smithi028.stdout: "leveldb_write_buffer_size": "33554432", 2024-09-18T04:37:43.476 INFO:teuthology.orchestra.run.smithi028.stdout: "librados_thread_count": "2", 2024-09-18T04:37:43.476 INFO:teuthology.orchestra.run.smithi028.stdout: "lockdep": "false", 2024-09-18T04:37:43.476 INFO:teuthology.orchestra.run.smithi028.stdout: "lockdep_force_backtrace": "false", 2024-09-18T04:37:43.476 INFO:teuthology.orchestra.run.smithi028.stdout: "log_coarse_timestamps": "true", 2024-09-18T04:37:43.476 INFO:teuthology.orchestra.run.smithi028.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-18T04:37:43.476 INFO:teuthology.orchestra.run.smithi028.stdout: "log_flush_on_exit": "false", 2024-09-18T04:37:43.476 INFO:teuthology.orchestra.run.smithi028.stdout: "log_graylog_host": "127.0.0.1", 2024-09-18T04:37:43.476 INFO:teuthology.orchestra.run.smithi028.stdout: "log_graylog_port": "12201", 2024-09-18T04:37:43.476 INFO:teuthology.orchestra.run.smithi028.stdout: "log_max_new": "1000", 2024-09-18T04:37:43.476 INFO:teuthology.orchestra.run.smithi028.stdout: "log_max_recent": "10000", 2024-09-18T04:37:43.476 INFO:teuthology.orchestra.run.smithi028.stdout: "log_stderr_prefix": "", 2024-09-18T04:37:43.477 INFO:teuthology.orchestra.run.smithi028.stdout: "log_stop_at_utilization": "0.970000", 2024-09-18T04:37:43.477 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": "true", 2024-09-18T04:37:43.477 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_graylog": "false", 2024-09-18T04:37:43.477 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_journald": "false", 2024-09-18T04:37:43.477 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_stderr": "false", 2024-09-18T04:37:43.477 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_syslog": "false", 2024-09-18T04:37:43.477 INFO:teuthology.orchestra.run.smithi028.stdout: "max_rotating_auth_attempts": "10", 2024-09-18T04:37:43.477 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-18T04:37:43.477 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_action_on_write_error": "1", 2024-09-18T04:37:43.477 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_alternate_name_max": "8192", 2024-09-18T04:37:43.477 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_asio_thread_count": "2", 2024-09-18T04:37:43.477 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_export_pin": "true", 2024-09-18T04:37:43.477 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_fragment_dirs": "true", 2024-09-18T04:37:43.477 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-18T04:37:43.478 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_fragment_interval": "5", 2024-09-18T04:37:43.478 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-18T04:37:43.478 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-18T04:37:43.478 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_interval": "10", 2024-09-18T04:37:43.478 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_max": "-1", 2024-09-18T04:37:43.478 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_max_until": "-1", 2024-09-18T04:37:43.478 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_merge_size": "50", 2024-09-18T04:37:43.478 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_midchunk": "0.300000", 2024-09-18T04:37:43.478 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-18T04:37:43.478 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_min_start": "0.200000", 2024-09-18T04:37:43.478 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_minchunk": "0.001000", 2024-09-18T04:37:43.478 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_mode": "0", 2024-09-18T04:37:43.478 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_need_max": "1.200000", 2024-09-18T04:37:43.478 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_need_min": "0.800000", 2024-09-18T04:37:43.479 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-18T04:37:43.479 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-18T04:37:43.479 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_split_bits": "3", 2024-09-18T04:37:43.479 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-18T04:37:43.479 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_split_size": "10000", 2024-09-18T04:37:43.479 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-18T04:37:43.479 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_target_decay": "10.000000", 2024-09-18T04:37:43.479 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-18T04:37:43.479 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_beacon_grace": "15.000000", 2024-09-18T04:37:43.479 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_beacon_interval": "4.000000", 2024-09-18T04:37:43.479 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-18T04:37:43.479 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-18T04:37:43.479 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache_mid": "0.700000", 2024-09-18T04:37:43.479 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache_release_free_interval": "10", 2024-09-18T04:37:43.480 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache_reservation": "0.050000", 2024-09-18T04:37:43.480 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-18T04:37:43.480 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache_trim_interval": "1", 2024-09-18T04:37:43.480 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache_trim_threshold": "262144", 2024-09-18T04:37:43.480 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-18T04:37:43.480 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-18T04:37:43.480 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-18T04:37:43.480 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_client_prealloc_inos": "1000", 2024-09-18T04:37:43.480 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-18T04:37:43.480 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_connect_bootstrapping": "false", 2024-09-18T04:37:43.480 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_damage_table_max_entries": "10000", 2024-09-18T04:37:43.480 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-18T04:37:43.480 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_debug_auth_pins": "false", 2024-09-18T04:37:43.481 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_debug_frag": "false", 2024-09-18T04:37:43.481 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_debug_scatterstat": "false", 2024-09-18T04:37:43.481 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_debug_subtrees": "false", 2024-09-18T04:37:43.481 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_decay_halflife": "5.000000", 2024-09-18T04:37:43.481 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_default_dir_hash": "2", 2024-09-18T04:37:43.481 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_defer_session_stale": "true", 2024-09-18T04:37:43.481 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_deny_all_reconnect": "false", 2024-09-18T04:37:43.481 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_dir_keys_per_op": "16384", 2024-09-18T04:37:43.481 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_dir_max_commit_size": "10", 2024-09-18T04:37:43.481 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_dir_max_entries": "0", 2024-09-18T04:37:43.481 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_dir_prefetch": "true", 2024-09-18T04:37:43.481 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-18T04:37:43.481 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-18T04:37:43.481 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_dump_cache_on_map": "false", 2024-09-18T04:37:43.481 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-18T04:37:43.482 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-18T04:37:43.482 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_early_reply": "true", 2024-09-18T04:37:43.482 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_enable_op_tracker": "true", 2024-09-18T04:37:43.482 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_enforce_unique_name": "true", 2024-09-18T04:37:43.482 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-18T04:37:43.482 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-18T04:37:43.482 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_export_ephemeral_random": "true", 2024-09-18T04:37:43.482 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-18T04:37:43.482 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-09-18T04:37:43.482 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-18T04:37:43.482 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-18T04:37:43.482 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-09-18T04:37:43.482 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-18T04:37:43.482 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-18T04:37:43.483 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-18T04:37:43.483 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_health_summarize_threshold": "10", 2024-09-18T04:37:43.483 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-18T04:37:43.483 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-18T04:37:43.483 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_inject_health_dummy": "false", 2024-09-18T04:37:43.483 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-18T04:37:43.483 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_inject_migrator_session_race": "false", 2024-09-18T04:37:43.483 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-18T04:37:43.483 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-18T04:37:43.483 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-18T04:37:43.483 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_join_fs": "", 2024-09-18T04:37:43.483 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_journal_format": "1", 2024-09-18T04:37:43.483 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-09-18T04:37:43.483 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_create_at": "0", 2024-09-18T04:37:43.484 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_export_at": "0", 2024-09-18T04:37:43.484 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_import_at": "0", 2024-09-18T04:37:43.484 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_journal_at": "0", 2024-09-18T04:37:43.484 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_journal_expire_at": "0", 2024-09-18T04:37:43.484 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_journal_replay_at": "0", 2024-09-18T04:37:43.484 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_link_at": "0", 2024-09-18T04:37:43.484 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_mdstable_at": "0", 2024-09-18T04:37:43.484 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_openc_at": "0", 2024-09-18T04:37:43.484 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_rename_at": "0", 2024-09-18T04:37:43.484 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log_events_per_segment": "1024", 2024-09-18T04:37:43.484 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log_max_events": "-1", 2024-09-18T04:37:43.484 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log_max_segments": "128", 2024-09-18T04:37:43.484 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log_pause": "false", 2024-09-18T04:37:43.484 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log_segment_size": "0", 2024-09-18T04:37:43.485 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-18T04:37:43.485 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log_warn_factor": "2.000000", 2024-09-18T04:37:43.485 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_max_caps_per_client": "1048576", 2024-09-18T04:37:43.485 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_max_completed_flushes": "100000", 2024-09-18T04:37:43.485 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_max_completed_requests": "100000", 2024-09-18T04:37:43.485 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_max_export_size": "20971520", 2024-09-18T04:37:43.485 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_max_file_recover": "32", 2024-09-18T04:37:43.485 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_max_purge_files": "64", 2024-09-18T04:37:43.485 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_max_purge_ops": "8192", 2024-09-18T04:37:43.485 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-18T04:37:43.485 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-18T04:37:43.485 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_max_snaps_per_dir": "100", 2024-09-18T04:37:43.485 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_metrics_update_interval": "2", 2024-09-18T04:37:43.486 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_min_caps_per_client": "100", 2024-09-18T04:37:43.486 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_min_caps_working_set": "10000", 2024-09-18T04:37:43.486 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-18T04:37:43.486 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_numa_node": "-1", 2024-09-18T04:37:43.486 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-18T04:37:43.486 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_op_complaint_time": "30.000000", 2024-09-18T04:37:43.486 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_op_history_duration": "600", 2024-09-18T04:37:43.486 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_op_history_size": "20", 2024-09-18T04:37:43.486 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_op_history_slow_op_size": "20", 2024-09-18T04:37:43.486 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-18T04:37:43.486 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_op_log_threshold": "5", 2024-09-18T04:37:43.486 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_ping_grace": "15", 2024-09-18T04:37:43.486 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_ping_interval": "5", 2024-09-18T04:37:43.486 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-18T04:37:43.487 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-18T04:37:43.487 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_recall_max_caps": "30000", 2024-09-18T04:37:43.487 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-18T04:37:43.487 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-18T04:37:43.487 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-18T04:37:43.487 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_recall_warning_threshold": "262144", 2024-09-18T04:37:43.487 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-18T04:37:43.487 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_replay_interval": "1.000000", 2024-09-18T04:37:43.487 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-18T04:37:43.487 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-18T04:37:43.487 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_root_ino_gid": "0", 2024-09-18T04:37:43.487 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_root_ino_uid": "0", 2024-09-18T04:37:43.487 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-18T04:37:43.487 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-18T04:37:43.488 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-18T04:37:43.488 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-18T04:37:43.488 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-18T04:37:43.488 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-18T04:37:43.488 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-18T04:37:43.488 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-18T04:37:43.488 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-18T04:37:43.488 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-18T04:37:43.488 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_shutdown_check": "0", 2024-09-18T04:37:43.488 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_skip_ino": "0", 2024-09-18T04:37:43.488 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-18T04:37:43.488 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_snap_max_uid": "4294967294", 2024-09-18T04:37:43.488 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_snap_min_uid": "0", 2024-09-18T04:37:43.489 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_snap_rstat": "false", 2024-09-18T04:37:43.489 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_standby_replay_damaged": "false", 2024-09-18T04:37:43.489 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_symlink_recovery": "true", 2024-09-18T04:37:43.489 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-18T04:37:43.489 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_thrash_exports": "0", 2024-09-18T04:37:43.489 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_thrash_fragments": "0", 2024-09-18T04:37:43.489 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_tick_interval": "5.000000", 2024-09-18T04:37:43.489 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_valgrind_exit": "false", 2024-09-18T04:37:43.489 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_verify_backtrace": "1", 2024-09-18T04:37:43.489 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_verify_scatter": "false", 2024-09-18T04:37:43.489 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-18T04:37:43.489 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_wipe_sessions": "false", 2024-09-18T04:37:43.489 INFO:teuthology.orchestra.run.smithi028.stdout: "mempool_debug": "false", 2024-09-18T04:37:43.489 INFO:teuthology.orchestra.run.smithi028.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-18T04:37:43.489 INFO:teuthology.orchestra.run.smithi028.stdout: "memstore_device_bytes": "1073741824", 2024-09-18T04:37:43.490 INFO:teuthology.orchestra.run.smithi028.stdout: "memstore_page_set": "false", 2024-09-18T04:37:43.490 INFO:teuthology.orchestra.run.smithi028.stdout: "memstore_page_size": "65536", 2024-09-18T04:37:43.490 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_client_bytes": "134217728", 2024-09-18T04:37:43.490 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_client_messages": "512", 2024-09-18T04:37:43.490 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-18T04:37:43.490 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-18T04:37:43.490 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-18T04:37:43.490 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-18T04:37:43.490 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_disabled_modules": "", 2024-09-18T04:37:43.490 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-18T04:37:43.490 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_max_pg_num_change": "128", 2024-09-18T04:37:43.490 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_mds_bytes": "134217728", 2024-09-18T04:37:43.490 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_mds_messages": "128", 2024-09-18T04:37:43.490 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-18T04:37:43.491 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_mon_bytes": "134217728", 2024-09-18T04:37:43.491 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_mon_messages": "128", 2024-09-18T04:37:43.491 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_osd_bytes": "536870912", 2024-09-18T04:37:43.491 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_osd_messages": "8192", 2024-09-18T04:37:43.491 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_pool": "true", 2024-09-18T04:37:43.491 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-18T04:37:43.491 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_standby_modules": "true", 2024-09-18T04:37:43.491 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_stats_period": "5", 2024-09-18T04:37:43.491 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_stats_threshold": "5", 2024-09-18T04:37:43.491 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_tick_period": "2", 2024-09-18T04:37:43.491 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-18T04:37:43.491 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-18T04:37:43.491 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_allow_pool_delete": "true", 2024-09-18T04:37:43.491 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_allow_pool_size_one": "true", 2024-09-18T04:37:43.492 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_auth_validate_all_caps": "true", 2024-09-18T04:37:43.492 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-18T04:37:43.492 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-18T04:37:43.492 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_bytes": "104857600", 2024-09-18T04:37:43.492 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_directed_command_retry": "5", 2024-09-18T04:37:43.492 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-18T04:37:43.492 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-18T04:37:43.492 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-18T04:37:43.492 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-18T04:37:43.492 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_hunt_parallel": "3", 2024-09-18T04:37:43.492 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_log_interval": "1.000000", 2024-09-18T04:37:43.492 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-18T04:37:43.492 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_ping_interval": "10.000000", 2024-09-18T04:37:43.493 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-18T04:37:43.493 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-18T04:37:43.493 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-18T04:37:43.493 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-18T04:37:43.493 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cluster_log_file_level": "debug", 2024-09-18T04:37:43.493 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cluster_log_to_file": "true", 2024-09-18T04:37:43.493 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-18T04:37:43.493 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-18T04:37:43.493 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-18T04:37:43.493 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cluster_log_to_journald": "false", 2024-09-18T04:37:43.493 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-18T04:37:43.493 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-18T04:37:43.493 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-18T04:37:43.493 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-09-18T04:37:43.494 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_compact_on_bootstrap": "false", 2024-09-18T04:37:43.494 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_compact_on_start": "false", 2024-09-18T04:37:43.494 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_compact_on_trim": "true", 2024-09-18T04:37:43.494 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-18T04:37:43.494 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-18T04:37:43.494 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-18T04:37:43.494 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cpu_threads": "4", 2024-09-18T04:37:43.494 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_crush_min_required_version": "hammer", 2024-09-18T04:37:43.494 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_daemon_bytes": "419430400", 2024-09-18T04:37:43.494 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-18T04:37:43.494 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_data_avail_crit": "5", 2024-09-18T04:37:43.494 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_data_avail_warn": "5", 2024-09-18T04:37:43.494 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_data_size_warn": "16106127360", 2024-09-18T04:37:43.495 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-18T04:37:43.495 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-18T04:37:43.495 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_debug_dump_json": "false", 2024-09-18T04:37:43.495 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-18T04:37:43.495 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_debug_dump_transactions": "false", 2024-09-18T04:37:43.495 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_debug_extra_checks": "false", 2024-09-18T04:37:43.495 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-18T04:37:43.495 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-18T04:37:43.495 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_debug_no_require_quincy": "false", 2024-09-18T04:37:43.495 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_debug_no_require_reef": "false", 2024-09-18T04:37:43.495 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-18T04:37:43.495 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-18T04:37:43.495 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-18T04:37:43.495 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_down_mkfs_grace": "120", 2024-09-18T04:37:43.496 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_election_default_strategy": "1", 2024-09-18T04:37:43.496 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_election_timeout": "5.000000", 2024-09-18T04:37:43.496 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-18T04:37:43.496 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_elector_ping_divisor": "2", 2024-09-18T04:37:43.496 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-18T04:37:43.496 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_enable_op_tracker": "true", 2024-09-18T04:37:43.496 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_fake_pool_delete": "false", 2024-09-18T04:37:43.496 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_force_quorum_join": "false", 2024-09-18T04:37:43.496 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_globalid_prealloc": "10000", 2024-09-18T04:37:43.496 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_health_detail_to_clog": "false", 2024-09-18T04:37:43.496 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_health_log_update_period": "5", 2024-09-18T04:37:43.496 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_health_max_detail": "50", 2024-09-18T04:37:43.496 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_health_to_clog": "true", 2024-09-18T04:37:43.497 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_health_to_clog_interval": "600", 2024-09-18T04:37:43.497 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-18T04:37:43.497 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_host": "172.21.15.28,[v2:172.21.15.28:3301,v1:172.21.15.28:6790],172.21.15.82", 2024-09-18T04:37:43.497 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_host_override": "", 2024-09-18T04:37:43.497 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_initial_members": "", 2024-09-18T04:37:43.497 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-18T04:37:43.497 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-18T04:37:43.497 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-18T04:37:43.497 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-18T04:37:43.497 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-18T04:37:43.497 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_lease": "5.000000", 2024-09-18T04:37:43.497 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-18T04:37:43.497 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-18T04:37:43.498 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_log_full_interval": "50", 2024-09-18T04:37:43.498 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_log_max": "10000", 2024-09-18T04:37:43.498 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_log_max_summary": "50", 2024-09-18T04:37:43.498 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-18T04:37:43.498 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_max_log_epochs": "500", 2024-09-18T04:37:43.498 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-18T04:37:43.498 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-18T04:37:43.498 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_max_osd": "10000", 2024-09-18T04:37:43.498 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_max_pg_per_osd": "10000", 2024-09-18T04:37:43.498 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_max_pool_pg_num": "65536", 2024-09-18T04:37:43.498 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-18T04:37:43.498 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-18T04:37:43.498 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_mds_force_trim_to": "0", 2024-09-18T04:37:43.498 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_mds_skip_sanity": "false", 2024-09-18T04:37:43.499 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_memory_autotune": "true", 2024-09-18T04:37:43.499 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_memory_target": "2147483648", 2024-09-18T04:37:43.499 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_mgr_beacon_grace": "30", 2024-09-18T04:37:43.499 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-18T04:37:43.499 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_mgr_digest_period": "5", 2024-09-18T04:37:43.499 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_mgr_inactive_grace": "60", 2024-09-18T04:37:43.499 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-18T04:37:43.499 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-18T04:37:43.499 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_min_osdmap_epochs": "500", 2024-09-18T04:37:43.499 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_op_complaint_time": "30", 2024-09-18T04:37:43.499 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_op_history_duration": "600", 2024-09-18T04:37:43.499 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_op_history_size": "20", 2024-09-18T04:37:43.499 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_op_history_slow_op_size": "20", 2024-09-18T04:37:43.499 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-18T04:37:43.499 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_op_log_threshold": "5", 2024-09-18T04:37:43.500 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-18T04:37:43.500 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-18T04:37:43.500 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-18T04:37:43.500 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_auto_mark_in": "false", 2024-09-18T04:37:43.500 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-18T04:37:43.500 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-18T04:37:43.500 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-18T04:37:43.500 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_cache_size": "500", 2024-09-18T04:37:43.500 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-18T04:37:43.500 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-18T04:37:43.500 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-18T04:37:43.500 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_down_out_interval": "600", 2024-09-18T04:37:43.500 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-18T04:37:43.500 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-18T04:37:43.501 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_force_trim_to": "0", 2024-09-18T04:37:43.501 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-18T04:37:43.501 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-18T04:37:43.501 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-18T04:37:43.501 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-18T04:37:43.501 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-18T04:37:43.501 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-18T04:37:43.501 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_max_creating_pgs": "1024", 2024-09-18T04:37:43.501 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-18T04:37:43.501 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_min_down_reporters": "2", 2024-09-18T04:37:43.501 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-18T04:37:43.501 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-18T04:37:43.501 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-18T04:37:43.501 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-18T04:37:43.502 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-18T04:37:43.502 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-18T04:37:43.502 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_report_timeout": "900", 2024-09-18T04:37:43.502 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-18T04:37:43.502 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-18T04:37:43.502 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-18T04:37:43.502 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-18T04:37:43.502 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-18T04:37:43.502 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osdmap_full_prune_interval": "10", 2024-09-18T04:37:43.502 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osdmap_full_prune_min": "10000", 2024-09-18T04:37:43.502 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osdmap_full_prune_txsize": "100", 2024-09-18T04:37:43.502 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-18T04:37:43.502 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_pg_stuck_threshold": "60", 2024-09-18T04:37:43.502 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-18T04:37:43.503 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-18T04:37:43.503 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-18T04:37:43.503 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-18T04:37:43.503 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-18T04:37:43.503 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-18T04:37:43.503 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_probe_timeout": "2.000000", 2024-09-18T04:37:43.503 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_reweight_max_change": "0.050000", 2024-09-18T04:37:43.503 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_reweight_max_osds": "4", 2024-09-18T04:37:43.503 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-18T04:37:43.503 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-18T04:37:43.503 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-18T04:37:43.503 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-18T04:37:43.503 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-18T04:37:43.503 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_scrub_interval": "86400", 2024-09-18T04:37:43.504 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_scrub_max_keys": "100", 2024-09-18T04:37:43.504 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_scrub_timeout": "300", 2024-09-18T04:37:43.504 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_session_timeout": "300", 2024-09-18T04:37:43.504 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_smart_report_timeout": "5", 2024-09-18T04:37:43.504 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_stat_smooth_intervals": "6", 2024-09-18T04:37:43.504 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-18T04:37:43.504 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_stretch_pool_min_size": "2", 2024-09-18T04:37:43.504 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_stretch_pool_size": "4", 2024-09-18T04:37:43.504 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-18T04:37:43.504 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-18T04:37:43.504 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_sync_debug": "false", 2024-09-18T04:37:43.504 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-18T04:37:43.504 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-18T04:37:43.505 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_sync_provider_kill_at": "0", 2024-09-18T04:37:43.505 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_sync_requester_kill_at": "0", 2024-09-18T04:37:43.505 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_sync_timeout": "60.000000", 2024-09-18T04:37:43.505 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_target_pg_per_osd": "100", 2024-09-18T04:37:43.505 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_tick_interval": "5", 2024-09-18T04:37:43.505 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_timecheck_interval": "300.000000", 2024-09-18T04:37:43.505 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-18T04:37:43.505 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_use_min_delay_socket": "false", 2024-09-18T04:37:43.505 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_older_version_delay": "604800", 2024-09-18T04:37:43.505 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-18T04:37:43.505 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-18T04:37:43.505 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-18T04:37:43.505 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_filestore_osds": "false", 2024-09-18T04:37:43.505 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-18T04:37:43.506 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-18T04:37:43.506 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-18T04:37:43.506 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_misplaced": "false", 2024-09-18T04:37:43.506 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-18T04:37:43.506 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_older_version": "true", 2024-09-18T04:37:43.506 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-18T04:37:43.506 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_pool_no_app": "false", 2024-09-18T04:37:43.506 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-09-18T04:37:43.506 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-18T04:37:43.506 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-18T04:37:43.506 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-18T04:37:43.506 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-18T04:37:43.506 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-18T04:37:43.507 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-18T04:37:43.507 INFO:teuthology.orchestra.run.smithi028.stdout: "monmap": "", 2024-09-18T04:37:43.507 INFO:teuthology.orchestra.run.smithi028.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-18T04:37:43.507 INFO:teuthology.orchestra.run.smithi028.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-09-18T04:37:43.507 INFO:teuthology.orchestra.run.smithi028.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-09-18T04:37:43.507 INFO:teuthology.orchestra.run.smithi028.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-18T04:37:43.507 INFO:teuthology.orchestra.run.smithi028.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-09-18T04:37:43.507 INFO:teuthology.orchestra.run.smithi028.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-09-18T04:37:43.507 INFO:teuthology.orchestra.run.smithi028.stdout: "motr_tracing_enabled": "false", 2024-09-18T04:37:43.507 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_op_threads": "3", 2024-09-18T04:37:43.507 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-18T04:37:43.507 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_cm": "false", 2024-09-18T04:37:43.507 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_device_name": "", 2024-09-18T04:37:43.507 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_dscp": "96", 2024-09-18T04:37:43.508 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-18T04:37:43.508 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-18T04:37:43.508 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_local_gid": "", 2024-09-18T04:37:43.508 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-18T04:37:43.508 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_port_num": "1", 2024-09-18T04:37:43.508 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-18T04:37:43.508 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-18T04:37:43.508 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-18T04:37:43.508 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-18T04:37:43.508 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_sl": "3", 2024-09-18T04:37:43.508 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_support_srq": "true", 2024-09-18T04:37:43.508 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_type": "ib", 2024-09-18T04:37:43.508 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_reap_threshold": "5", 2024-09-18T04:37:43.508 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_bind_before_connect": "false", 2024-09-18T04:37:43.509 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_bind_ipv4": "true", 2024-09-18T04:37:43.509 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_bind_ipv6": "false", 2024-09-18T04:37:43.509 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_bind_msgr1": "true", 2024-09-18T04:37:43.509 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_bind_msgr2": "true", 2024-09-18T04:37:43.509 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_bind_port_max": "7568", 2024-09-18T04:37:43.509 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_bind_port_min": "6800", 2024-09-18T04:37:43.509 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-18T04:37:43.509 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_bind_retry_count": "3", 2024-09-18T04:37:43.509 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_bind_retry_delay": "5", 2024-09-18T04:37:43.509 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_blackhole_client": "false", 2024-09-18T04:37:43.509 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_blackhole_mds": "false", 2024-09-18T04:37:43.509 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_blackhole_mgr": "false", 2024-09-18T04:37:43.509 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_blackhole_mon": "false", 2024-09-18T04:37:43.509 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_blackhole_osd": "false", 2024-09-18T04:37:43.510 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_client_mode": "crc secure", 2024-09-18T04:37:43.510 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_cluster_mode": "crc secure", 2024-09-18T04:37:43.510 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_cluster_type": "", 2024-09-18T04:37:43.510 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_compress_secure": "false", 2024-09-18T04:37:43.510 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_connection_idle_timeout": "900", 2024-09-18T04:37:43.510 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_connection_ready_timeout": "10", 2024-09-18T04:37:43.510 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_crc_data": "true", 2024-09-18T04:37:43.510 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_crc_header": "true", 2024-09-18T04:37:43.510 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_die_on_bad_msg": "false", 2024-09-18T04:37:43.510 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_die_on_bug": "true", 2024-09-18T04:37:43.510 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_die_on_old_message": "true", 2024-09-18T04:37:43.510 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_die_on_skipped_message": "false", 2024-09-18T04:37:43.510 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-18T04:37:43.510 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-18T04:37:43.511 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_coremask": "0xF", 2024-09-18T04:37:43.511 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-18T04:37:43.511 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-18T04:37:43.511 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_enable_tso": "true", 2024-09-18T04:37:43.511 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-18T04:37:43.511 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-18T04:37:43.511 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_hugepages": "", 2024-09-18T04:37:43.511 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-18T04:37:43.511 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-18T04:37:43.511 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_lro": "true", 2024-09-18T04:37:43.511 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_memory_channel": "4", 2024-09-18T04:37:43.511 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-18T04:37:43.511 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_pmd": "", 2024-09-18T04:37:43.511 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_port_id": "0", 2024-09-18T04:37:43.512 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-18T04:37:43.512 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-18T04:37:43.512 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dump_on_send": "false", 2024-09-18T04:37:43.512 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_initial_backoff": "0.200000", 2024-09-18T04:37:43.512 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_inject_delay_max": "1.000000", 2024-09-18T04:37:43.512 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_inject_delay_probability": "0.000000", 2024-09-18T04:37:43.512 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_inject_delay_type": "", 2024-09-18T04:37:43.512 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_inject_internal_delays": "0.000000", 2024-09-18T04:37:43.512 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_inject_network_congestion": "0", 2024-09-18T04:37:43.512 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_inject_socket_failures": "5000", 2024-09-18T04:37:43.512 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_learn_addr_from_peer": "true", 2024-09-18T04:37:43.512 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_max_accept_failures": "4", 2024-09-18T04:37:43.512 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_max_backoff": "15.000000", 2024-09-18T04:37:43.513 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_mon_client_mode": "secure crc", 2024-09-18T04:37:43.513 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-18T04:37:43.513 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_mon_service_mode": "secure crc", 2024-09-18T04:37:43.513 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_osd_compress_min_size": "1024", 2024-09-18T04:37:43.513 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_osd_compress_mode": "none", 2024-09-18T04:37:43.513 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_osd_compression_algorithm": "snappy", 2024-09-18T04:37:43.513 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-18T04:37:43.513 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_pq_min_cost": "65536", 2024-09-18T04:37:43.513 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_public_type": "", 2024-09-18T04:37:43.513 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_service_mode": "crc secure", 2024-09-18T04:37:43.513 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_tcp_listen_backlog": "512", 2024-09-18T04:37:43.513 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_tcp_nodelay": "true", 2024-09-18T04:37:43.513 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-18T04:37:43.513 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_tcp_rcvbuf": "0", 2024-09-18T04:37:43.514 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_type": "async+posix", 2024-09-18T04:37:43.514 INFO:teuthology.orchestra.run.smithi028.stdout: "no_config_file": "false", 2024-09-18T04:37:43.514 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter_completion_locks_per_session": "32", 2024-09-18T04:37:43.514 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-18T04:37:43.514 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-18T04:37:43.514 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter_inflight_ops": "1024", 2024-09-18T04:37:43.514 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-18T04:37:43.514 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-18T04:37:43.514 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter_tick_interval": "5.000000", 2024-09-18T04:37:43.514 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter_timeout": "10.000000", 2024-09-18T04:37:43.514 INFO:teuthology.orchestra.run.smithi028.stdout: "objectstore_blackhole": "false", 2024-09-18T04:37:43.514 INFO:teuthology.orchestra.run.smithi028.stdout: "openssl_engine_opts": "", 2024-09-18T04:37:43.514 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_agent_delay_time": "5.000000", 2024-09-18T04:37:43.514 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_agent_hist_halflife": "1000", 2024-09-18T04:37:43.515 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_agent_max_low_ops": "2", 2024-09-18T04:37:43.515 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_agent_max_ops": "4", 2024-09-18T04:37:43.515 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-18T04:37:43.515 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-18T04:37:43.515 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_agent_slop": "0.020000", 2024-09-18T04:37:43.515 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-18T04:37:43.515 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-18T04:37:43.515 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_asio_thread_count": "2", 2024-09-18T04:37:43.515 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_async_recovery_min_cost": "100", 2024-09-18T04:37:43.515 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-18T04:37:43.515 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-18T04:37:43.515 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_backfill_scan_max": "512", 2024-09-18T04:37:43.515 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_backfill_scan_min": "64", 2024-09-18T04:37:43.516 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_backoff_on_degraded": "false", 2024-09-18T04:37:43.516 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_backoff_on_peering": "false", 2024-09-18T04:37:43.516 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_backoff_on_unfound": "true", 2024-09-18T04:37:43.516 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_beacon_report_interval": "300", 2024-09-18T04:37:43.516 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bench_duration": "30", 2024-09-18T04:37:43.516 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-18T04:37:43.516 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bench_max_block_size": "67108864", 2024-09-18T04:37:43.516 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-18T04:37:43.516 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_blkin_trace_all": "false", 2024-09-18T04:37:43.516 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_blocked_scrub_grace_period": "120", 2024-09-18T04:37:43.516 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-18T04:37:43.516 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-09-18T04:37:43.516 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-18T04:37:43.517 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_check_for_log_corruption": "false", 2024-09-18T04:37:43.517 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-18T04:37:43.517 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-18T04:37:43.517 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_class_dir": "/usr/lib/rados-classes", 2024-09-18T04:37:43.517 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-18T04:37:43.517 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_class_update_on_start": "true", 2024-09-18T04:37:43.517 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_client_message_cap": "256", 2024-09-18T04:37:43.517 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_client_message_size_cap": "524288000", 2024-09-18T04:37:43.517 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_client_op_priority": "63", 2024-09-18T04:37:43.517 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_client_watch_timeout": "30", 2024-09-18T04:37:43.517 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_command_max_records": "256", 2024-09-18T04:37:43.517 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_compact_on_start": "false", 2024-09-18T04:37:43.517 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-18T04:37:43.517 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_crush_chooseleaf_type": "0", 2024-09-18T04:37:43.518 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-18T04:37:43.518 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_crush_update_on_start": "true", 2024-09-18T04:37:43.518 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_crush_update_weight_set": "true", 2024-09-18T04:37:43.518 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-18T04:37:43.518 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-18T04:37:43.518 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-18T04:37:43.518 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-18T04:37:43.518 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-18T04:37:43.518 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_feed_pullee": "-1", 2024-09-18T04:37:43.518 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-18T04:37:43.518 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-18T04:37:43.518 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-18T04:37:43.518 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_misdirected_ops": "false", 2024-09-18T04:37:43.518 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_no_acting_change": "false", 2024-09-18T04:37:43.519 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_no_purge_strays": "false", 2024-09-18T04:37:43.519 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_op_order": "false", 2024-09-18T04:37:43.519 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-18T04:37:43.519 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-18T04:37:43.519 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-18T04:37:43.519 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-18T04:37:43.519 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_shutdown": "false", 2024-09-18T04:37:43.519 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-18T04:37:43.519 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-18T04:37:43.519 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-18T04:37:43.519 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-18T04:37:43.519 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_verify_snaps": "false", 2024-09-18T04:37:43.519 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-18T04:37:43.519 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-18T04:37:43.520 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_deep_scrub_keys": "1024", 2024-09-18T04:37:43.520 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-18T04:37:43.520 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-18T04:37:43.520 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-18T04:37:43.520 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_deep_scrub_stride": "524288", 2024-09-18T04:37:43.520 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-18T04:37:43.520 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_default_data_pool_replay_window": "5", 2024-09-18T04:37:43.520 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_default_notify_timeout": "30", 2024-09-18T04:37:43.520 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_delete_sleep": "0.000000", 2024-09-18T04:37:43.520 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-18T04:37:43.520 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-18T04:37:43.520 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-18T04:37:43.520 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_discard_disconnected_ops": "true", 2024-09-18T04:37:43.521 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_enable_op_tracker": "true", 2024-09-18T04:37:43.521 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-18T04:37:43.521 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_extblkdev_plugins": "vdo", 2024-09-18T04:37:43.521 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-18T04:37:43.521 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-18T04:37:43.521 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_fast_info": "true", 2024-09-18T04:37:43.521 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_fast_shutdown": "true", 2024-09-18T04:37:43.521 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-18T04:37:43.521 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-18T04:37:43.521 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-18T04:37:43.521 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-18T04:37:43.521 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-18T04:37:43.521 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_function_tracing": "false", 2024-09-18T04:37:43.522 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_heartbeat_grace": "20", 2024-09-18T04:37:43.522 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_heartbeat_interval": "6", 2024-09-18T04:37:43.522 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-18T04:37:43.522 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_heartbeat_min_peers": "10", 2024-09-18T04:37:43.522 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_heartbeat_min_size": "2000", 2024-09-18T04:37:43.522 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_heartbeat_stale": "600", 2024-09-18T04:37:43.522 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-18T04:37:43.522 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_hit_set_max_size": "100000", 2024-09-18T04:37:43.522 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_hit_set_min_size": "1000", 2024-09-18T04:37:43.522 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-18T04:37:43.522 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-18T04:37:43.522 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-18T04:37:43.522 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-18T04:37:43.522 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-18T04:37:43.523 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-18T04:37:43.523 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_journal_size": "5120", 2024-09-18T04:37:43.523 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_kill_backfill_at": "0", 2024-09-18T04:37:43.523 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-18T04:37:43.523 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_map_cache_size": "50", 2024-09-18T04:37:43.523 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_map_dedup": "true", 2024-09-18T04:37:43.523 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_map_message_max": "40", 2024-09-18T04:37:43.523 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-18T04:37:43.523 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_map_share_max_epochs": "40", 2024-09-18T04:37:43.523 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_attr_name_len": "100", 2024-09-18T04:37:43.523 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_attr_size": "0", 2024-09-18T04:37:43.523 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_backfills": "1", 2024-09-18T04:37:43.523 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_markdown_count": "5", 2024-09-18T04:37:43.523 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_markdown_period": "600", 2024-09-18T04:37:43.524 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_object_name_len": "2048", 2024-09-18T04:37:43.524 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_object_namespace_len": "256", 2024-09-18T04:37:43.524 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_object_size": "134217728", 2024-09-18T04:37:43.524 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-18T04:37:43.524 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-18T04:37:43.524 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_pg_blocked_by": "16", 2024-09-18T04:37:43.524 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_pg_log_entries": "10000", 2024-09-18T04:37:43.524 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-18T04:37:43.524 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_pgls": "1024", 2024-09-18T04:37:43.524 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_push_cost": "8388608", 2024-09-18T04:37:43.524 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_push_objects": "10", 2024-09-18T04:37:43.524 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_scrubs": "3", 2024-09-18T04:37:43.524 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-18T04:37:43.525 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_trimming_pgs": "2", 2024-09-18T04:37:43.525 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_write_op_reply_len": "64", 2024-09-18T04:37:43.525 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_write_size": "90", 2024-09-18T04:37:43.525 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-18T04:37:43.525 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-18T04:37:43.525 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-18T04:37:43.525 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-18T04:37:43.525 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-18T04:37:43.525 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-18T04:37:43.525 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-18T04:37:43.525 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-18T04:37:43.525 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_profile": "balanced", 2024-09-18T04:37:43.525 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-18T04:37:43.526 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-18T04:37:43.526 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-18T04:37:43.526 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-18T04:37:43.526 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-18T04:37:43.526 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-18T04:37:43.526 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-18T04:37:43.526 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-18T04:37:43.526 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-18T04:37:43.526 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-18T04:37:43.526 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-18T04:37:43.526 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_memory_base": "805306368", 2024-09-18T04:37:43.526 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_memory_cache_min": "134217728", 2024-09-18T04:37:43.526 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-18T04:37:43.526 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-18T04:37:43.527 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_memory_target": "4294967296", 2024-09-18T04:37:43.527 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_memory_target_autotune": "false", 2024-09-18T04:37:43.527 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-18T04:37:43.527 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_min_pg_log_entries": "250", 2024-09-18T04:37:43.527 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_min_recovery_priority": "0", 2024-09-18T04:37:43.527 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-18T04:37:43.527 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-18T04:37:43.527 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mon_report_interval": "5", 2024-09-18T04:37:43.527 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-18T04:37:43.527 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-18T04:37:43.527 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_num_cache_shards": "32", 2024-09-18T04:37:43.527 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_num_op_tracker_shard": "32", 2024-09-18T04:37:43.527 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_numa_auto_affinity": "true", 2024-09-18T04:37:43.527 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_numa_node": "-1", 2024-09-18T04:37:43.528 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_numa_prefer_iface": "true", 2024-09-18T04:37:43.528 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-09-18T04:37:43.528 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_objecter_finishers": "1", 2024-09-18T04:37:43.528 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_objectstore": "bluestore", 2024-09-18T04:37:43.528 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_objectstore_fuse": "false", 2024-09-18T04:37:43.528 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_objectstore_tracing": "false", 2024-09-18T04:37:43.528 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_complaint_time": "30.000000", 2024-09-18T04:37:43.528 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_history_duration": "600", 2024-09-18T04:37:43.528 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_history_size": "20", 2024-09-18T04:37:43.528 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_history_slow_op_size": "20", 2024-09-18T04:37:43.528 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-18T04:37:43.528 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_log_threshold": "5", 2024-09-18T04:37:43.528 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_num_shards": "0", 2024-09-18T04:37:43.528 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_num_shards_hdd": "5", 2024-09-18T04:37:43.529 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_num_shards_ssd": "8", 2024-09-18T04:37:43.529 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-18T04:37:43.529 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-18T04:37:43.529 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-18T04:37:43.529 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-18T04:37:43.529 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_pq_min_cost": "65536", 2024-09-18T04:37:43.529 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-18T04:37:43.529 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_queue_cut_off": "high", 2024-09-18T04:37:43.529 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-18T04:37:43.529 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_thread_timeout": "15", 2024-09-18T04:37:43.529 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_open_classes_on_start": "true", 2024-09-18T04:37:43.529 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_os_flags": "0", 2024-09-18T04:37:43.529 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_peering_op_priority": "255", 2024-09-18T04:37:43.529 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pg_delete_cost": "1048576", 2024-09-18T04:37:43.530 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pg_delete_priority": "5", 2024-09-18T04:37:43.530 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-18T04:37:43.530 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-18T04:37:43.530 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-18T04:37:43.530 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pg_log_trim_max": "10000", 2024-09-18T04:37:43.530 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pg_log_trim_min": "100", 2024-09-18T04:37:43.530 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-18T04:37:43.530 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-18T04:37:43.530 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pg_stat_report_interval_max": "500", 2024-09-18T04:37:43.530 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-18T04:37:43.530 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-18T04:37:43.530 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-18T04:37:43.530 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-18T04:37:43.530 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-18T04:37:43.531 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-18T04:37:43.531 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_crimson": "false", 2024-09-18T04:37:43.531 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-18T04:37:43.531 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-18T04:37:43.531 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-18T04:37:43.531 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-18T04:37:43.531 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-18T04:37:43.531 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-18T04:37:43.531 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-18T04:37:43.531 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-18T04:37:43.531 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_flags": "0", 2024-09-18T04:37:43.531 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-18T04:37:43.531 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_min_size": "0", 2024-09-18T04:37:43.531 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-18T04:37:43.532 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_pg_num": "32", 2024-09-18T04:37:43.532 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_pgp_num": "0", 2024-09-18T04:37:43.532 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-18T04:37:43.532 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_size": "2", 2024-09-18T04:37:43.532 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_type": "replicated", 2024-09-18T04:37:43.532 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-18T04:37:43.532 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-18T04:37:43.532 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_push_per_object_cost": "1000", 2024-09-18T04:37:43.532 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-18T04:37:43.532 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recover_clone_overlap": "true", 2024-09-18T04:37:43.532 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-18T04:37:43.532 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_cost": "20971520", 2024-09-18T04:37:43.532 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-18T04:37:43.533 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_max_active": "0", 2024-09-18T04:37:43.533 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-18T04:37:43.533 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-18T04:37:43.533 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-18T04:37:43.533 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-18T04:37:43.533 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_max_single_start": "1", 2024-09-18T04:37:43.533 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_op_priority": "3", 2024-09-18T04:37:43.533 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-18T04:37:43.533 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_priority": "5", 2024-09-18T04:37:43.533 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-18T04:37:43.533 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_sleep": "0.000000", 2024-09-18T04:37:43.533 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-18T04:37:43.534 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-18T04:37:43.534 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-18T04:37:43.534 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_repair_during_recovery": "false", 2024-09-18T04:37:43.534 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_requested_scrub_priority": "120", 2024-09-18T04:37:43.534 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-18T04:37:43.534 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-18T04:37:43.534 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_auto_repair": "false", 2024-09-18T04:37:43.534 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-18T04:37:43.534 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-18T04:37:43.534 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_begin_hour": "0", 2024-09-18T04:37:43.534 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_begin_week_day": "0", 2024-09-18T04:37:43.534 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_chunk_max": "25", 2024-09-18T04:37:43.534 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_chunk_min": "5", 2024-09-18T04:37:43.534 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_cost": "52428800", 2024-09-18T04:37:43.535 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_during_recovery": "false", 2024-09-18T04:37:43.535 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_end_hour": "0", 2024-09-18T04:37:43.535 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_end_week_day": "0", 2024-09-18T04:37:43.535 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_event_cost": "4096", 2024-09-18T04:37:43.535 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-18T04:37:43.535 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-18T04:37:43.535 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_invalid_stats": "true", 2024-09-18T04:37:43.535 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-18T04:37:43.535 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-18T04:37:43.535 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_max_preemptions": "5", 2024-09-18T04:37:43.535 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-18T04:37:43.535 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_priority": "5", 2024-09-18T04:37:43.535 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_reservation_timeout": "5000", 2024-09-18T04:37:43.536 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_sleep": "0.000000", 2024-09-18T04:37:43.536 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_slow_reservation_response": "2200", 2024-09-18T04:37:43.536 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-09-18T04:37:43.536 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-09-18T04:37:43.536 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-18T04:37:43.536 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_skip_data_digest": "false", 2024-09-18T04:37:43.536 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_smart_report_timeout": "5", 2024-09-18T04:37:43.536 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_snap_trim_cost": "1048576", 2024-09-18T04:37:43.536 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_snap_trim_priority": "5", 2024-09-18T04:37:43.536 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-18T04:37:43.536 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-18T04:37:43.536 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-18T04:37:43.536 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-18T04:37:43.537 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-09-18T04:37:43.537 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_stats_update_period_scrubbing": "15", 2024-09-18T04:37:43.537 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-18T04:37:43.537 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_target_transaction_size": "30", 2024-09-18T04:37:43.537 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-18T04:37:43.537 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-18T04:37:43.537 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-18T04:37:43.537 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-18T04:37:43.537 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-18T04:37:43.537 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-18T04:37:43.537 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-18T04:37:43.537 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-18T04:37:43.537 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-18T04:37:43.538 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-18T04:37:43.538 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_tracing": "false", 2024-09-18T04:37:43.538 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_use_stale_snap": "false", 2024-09-18T04:37:43.538 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-18T04:37:43.538 INFO:teuthology.orchestra.run.smithi028.stdout: "osdc_blkin_trace_all": "false", 2024-09-18T04:37:43.538 INFO:teuthology.orchestra.run.smithi028.stdout: "paxos_kill_at": "0", 2024-09-18T04:37:43.538 INFO:teuthology.orchestra.run.smithi028.stdout: "paxos_max_join_drift": "10", 2024-09-18T04:37:43.538 INFO:teuthology.orchestra.run.smithi028.stdout: "paxos_min": "500", 2024-09-18T04:37:43.538 INFO:teuthology.orchestra.run.smithi028.stdout: "paxos_min_wait": "0.050000", 2024-09-18T04:37:43.538 INFO:teuthology.orchestra.run.smithi028.stdout: "paxos_propose_interval": "1.000000", 2024-09-18T04:37:43.538 INFO:teuthology.orchestra.run.smithi028.stdout: "paxos_service_trim_max": "500", 2024-09-18T04:37:43.538 INFO:teuthology.orchestra.run.smithi028.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-18T04:37:43.538 INFO:teuthology.orchestra.run.smithi028.stdout: "paxos_service_trim_min": "250", 2024-09-18T04:37:43.539 INFO:teuthology.orchestra.run.smithi028.stdout: "paxos_stash_full_interval": "25", 2024-09-18T04:37:43.539 INFO:teuthology.orchestra.run.smithi028.stdout: "paxos_trim_max": "500", 2024-09-18T04:37:43.539 INFO:teuthology.orchestra.run.smithi028.stdout: "paxos_trim_min": "250", 2024-09-18T04:37:43.539 INFO:teuthology.orchestra.run.smithi028.stdout: "perf": "true", 2024-09-18T04:37:43.539 INFO:teuthology.orchestra.run.smithi028.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-09-18T04:37:43.539 INFO:teuthology.orchestra.run.smithi028.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-18T04:37:43.539 INFO:teuthology.orchestra.run.smithi028.stdout: "plugin_dir": "/usr/lib/ceph", 2024-09-18T04:37:43.539 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "-", 2024-09-18T04:37:43.539 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrv": "", 2024-09-18T04:37:43.539 INFO:teuthology.orchestra.run.smithi028.stdout: "public_bind_addr": "-", 2024-09-18T04:37:43.539 INFO:teuthology.orchestra.run.smithi028.stdout: "public_network": "", 2024-09-18T04:37:43.539 INFO:teuthology.orchestra.run.smithi028.stdout: "public_network_interface": "", 2024-09-18T04:37:43.539 INFO:teuthology.orchestra.run.smithi028.stdout: "qat_compressor_enabled": "false", 2024-09-18T04:37:43.539 INFO:teuthology.orchestra.run.smithi028.stdout: "qat_compressor_session_max_number": "256", 2024-09-18T04:37:43.539 INFO:teuthology.orchestra.run.smithi028.stdout: "rados_mon_op_timeout": "0", 2024-09-18T04:37:43.540 INFO:teuthology.orchestra.run.smithi028.stdout: "rados_osd_op_timeout": "0", 2024-09-18T04:37:43.540 INFO:teuthology.orchestra.run.smithi028.stdout: "rados_tracing": "false", 2024-09-18T04:37:43.540 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_atime_update_interval": "60", 2024-09-18T04:37:43.540 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-18T04:37:43.540 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_balance_parent_reads": "false", 2024-09-18T04:37:43.540 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_balance_snap_reads": "false", 2024-09-18T04:37:43.540 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_blkin_trace_all": "false", 2024-09-18T04:37:43.540 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-18T04:37:43.540 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-18T04:37:43.540 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_cache": "true", 2024-09-18T04:37:43.540 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-18T04:37:43.540 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-18T04:37:43.540 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-18T04:37:43.541 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-18T04:37:43.541 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_cache_policy": "writearound", 2024-09-18T04:37:43.541 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_cache_size": "33554432", 2024-09-18T04:37:43.541 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-18T04:37:43.541 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-18T04:37:43.541 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_clone_copy_on_read": "false", 2024-09-18T04:37:43.541 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_compression_hint": "none", 2024-09-18T04:37:43.541 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_concurrent_management_ops": "10", 2024-09-18T04:37:43.541 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-18T04:37:43.541 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_default_clone_format": "auto", 2024-09-18T04:37:43.541 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_default_data_pool": "", 2024-09-18T04:37:43.541 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_default_features": "61", 2024-09-18T04:37:43.541 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_default_format": "2", 2024-09-18T04:37:43.541 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_default_map_options": "", 2024-09-18T04:37:43.542 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_default_order": "22", 2024-09-18T04:37:43.542 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_default_pool": "rbd", 2024-09-18T04:37:43.542 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-18T04:37:43.542 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_default_stripe_count": "0", 2024-09-18T04:37:43.542 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_default_stripe_unit": "0", 2024-09-18T04:37:43.542 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-18T04:37:43.542 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-18T04:37:43.542 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-18T04:37:43.542 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_enable_alloc_hint": "true", 2024-09-18T04:37:43.542 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-18T04:37:43.542 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_io_scheduler": "simple", 2024-09-18T04:37:43.542 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-18T04:37:43.542 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-18T04:37:43.542 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-18T04:37:43.543 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-18T04:37:43.543 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-18T04:37:43.543 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-18T04:37:43.543 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-18T04:37:43.543 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-18T04:37:43.543 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-18T04:37:43.543 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_journal_order": "24", 2024-09-18T04:37:43.543 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_journal_pool": "", 2024-09-18T04:37:43.543 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_journal_splay_width": "4", 2024-09-18T04:37:43.543 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_localize_parent_reads": "false", 2024-09-18T04:37:43.543 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_localize_snap_reads": "false", 2024-09-18T04:37:43.543 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-18T04:37:43.543 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-18T04:37:43.544 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-18T04:37:43.544 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-18T04:37:43.544 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-18T04:37:43.544 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-18T04:37:43.544 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-18T04:37:43.544 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-18T04:37:43.544 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-18T04:37:43.544 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-18T04:37:43.544 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-18T04:37:43.544 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-18T04:37:43.544 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-18T04:37:43.544 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-18T04:37:43.544 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-18T04:37:43.544 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-18T04:37:43.545 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-18T04:37:43.545 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-18T04:37:43.545 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-18T04:37:43.545 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-18T04:37:43.545 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-18T04:37:43.545 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-18T04:37:43.545 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-18T04:37:43.545 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-18T04:37:43.545 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-18T04:37:43.545 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-18T04:37:43.545 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-18T04:37:43.545 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-18T04:37:43.545 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-18T04:37:43.545 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-18T04:37:43.546 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-18T04:37:43.546 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mtime_update_interval": "60", 2024-09-18T04:37:43.546 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_non_blocking_aio": "true", 2024-09-18T04:37:43.546 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_op_thread_timeout": "60", 2024-09-18T04:37:43.546 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_op_threads": "1", 2024-09-18T04:37:43.546 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_parent_cache_enabled": "false", 2024-09-18T04:37:43.546 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-18T04:37:43.546 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-18T04:37:43.546 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-18T04:37:43.546 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_plugins": "", 2024-09-18T04:37:43.546 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_bps_burst": "0", 2024-09-18T04:37:43.546 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-18T04:37:43.546 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_bps_limit": "0", 2024-09-18T04:37:43.547 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_exclude_ops": "0", 2024-09-18T04:37:43.547 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_iops_burst": "0", 2024-09-18T04:37:43.547 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-18T04:37:43.547 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_iops_limit": "0", 2024-09-18T04:37:43.547 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-18T04:37:43.547 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-18T04:37:43.547 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-18T04:37:43.547 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-18T04:37:43.547 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-18T04:37:43.547 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-18T04:37:43.547 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-18T04:37:43.547 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-18T04:37:43.547 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-18T04:37:43.547 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-18T04:37:43.548 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-18T04:37:43.548 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-18T04:37:43.548 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-18T04:37:43.548 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-18T04:37:43.548 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_read_from_replica_policy": "default", 2024-09-18T04:37:43.548 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-18T04:37:43.548 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-18T04:37:43.548 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-18T04:37:43.548 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-18T04:37:43.548 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_skip_partial_discard": "true", 2024-09-18T04:37:43.548 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-18T04:37:43.548 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_tracing": "false", 2024-09-18T04:37:43.549 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_validate_names": "true", 2024-09-18T04:37:43.549 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_validate_pool": "true", 2024-09-18T04:37:43.549 INFO:teuthology.orchestra.run.smithi028.stdout: "restapi_base_url": "", 2024-09-18T04:37:43.549 INFO:teuthology.orchestra.run.smithi028.stdout: "restapi_log_level": "", 2024-09-18T04:37:43.549 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_acl_grants_max_num": "100", 2024-09-18T04:37:43.549 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_admin_entry": "admin", 2024-09-18T04:37:43.549 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-09-18T04:37:43.549 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_backend_store": "rados", 2024-09-18T04:37:43.549 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_barbican_url": "", 2024-09-18T04:37:43.549 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_beast_enable_async": "true", 2024-09-18T04:37:43.549 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-18T04:37:43.549 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-18T04:37:43.549 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-18T04:37:43.550 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-09-18T04:37:43.550 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-18T04:37:43.550 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-18T04:37:43.550 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-09-18T04:37:43.550 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_cache_enabled": "true", 2024-09-18T04:37:43.550 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_cache_expiry_interval": "900", 2024-09-18T04:37:43.550 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_cache_lru_size": "10000", 2024-09-18T04:37:43.550 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_config_store": "rados", 2024-09-18T04:37:43.550 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_content_length_compat": "false", 2024-09-18T04:37:43.550 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_copy_obj_progress": "true", 2024-09-18T04:37:43.550 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-18T04:37:43.550 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_cors_rules_max_num": "100", 2024-09-18T04:37:43.550 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_cross_domain_policy": "", 2024-09-18T04:37:43.551 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-18T04:37:43.551 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_kmip_addr": "", 2024-09-18T04:37:43.551 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-18T04:37:43.551 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-18T04:37:43.551 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-18T04:37:43.551 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-18T04:37:43.551 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_kmip_password": "", 2024-09-18T04:37:43.551 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-18T04:37:43.551 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_kmip_username": "", 2024-09-18T04:37:43.551 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_require_ssl": "true", 2024-09-18T04:37:43.551 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-18T04:37:43.551 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-18T04:37:43.552 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-18T04:37:43.552 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-18T04:37:43.552 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-18T04:37:43.552 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-18T04:37:43.552 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-18T04:37:43.552 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-18T04:37:43.552 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-18T04:37:43.552 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-18T04:37:43.552 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-18T04:37:43.552 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-18T04:37:43.552 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-18T04:37:43.552 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-18T04:37:43.552 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-18T04:37:43.553 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_vault_addr": "", 2024-09-18T04:37:43.553 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_vault_auth": "token", 2024-09-18T04:37:43.553 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_vault_namespace": "", 2024-09-18T04:37:43.553 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_vault_prefix": "", 2024-09-18T04:37:43.553 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-18T04:37:43.553 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-18T04:37:43.553 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-18T04:37:43.553 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-18T04:37:43.553 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_vault_token_file": "", 2024-09-18T04:37:43.553 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-18T04:37:43.553 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_curl_buffersize": "524288", 2024-09-18T04:37:43.553 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-18T04:37:43.553 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_curl_low_speed_time": "300", 2024-09-18T04:37:43.554 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_curl_tcp_keepalive": "0", 2024-09-18T04:37:43.554 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-18T04:37:43.554 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-18T04:37:43.554 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-18T04:37:43.554 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-18T04:37:43.554 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-18T04:37:43.554 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-18T04:37:43.554 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-18T04:37:43.554 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-18T04:37:43.554 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-18T04:37:43.554 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-18T04:37:43.554 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_data_log_changes_size": "1000", 2024-09-18T04:37:43.554 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_data_log_num_shards": "128", 2024-09-18T04:37:43.554 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-18T04:37:43.555 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_data_log_window": "30", 2024-09-18T04:37:43.555 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_data_notify_interval_msec": "0", 2024-09-18T04:37:43.555 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_data_sync_poll_interval": "20", 2024-09-18T04:37:43.555 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_data_sync_spawn_window": "20", 2024-09-18T04:37:43.555 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-18T04:37:43.555 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-18T04:37:43.555 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-18T04:37:43.555 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-18T04:37:43.555 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-18T04:37:43.555 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-18T04:37:43.555 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-18T04:37:43.555 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-18T04:37:43.555 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-18T04:37:43.555 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-18T04:37:43.555 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-18T04:37:43.556 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-18T04:37:43.556 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-18T04:37:43.556 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-18T04:37:43.556 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-18T04:37:43.556 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-18T04:37:43.556 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-18T04:37:43.556 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-18T04:37:43.556 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-18T04:37:43.556 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-18T04:37:43.556 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-18T04:37:43.556 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dns_name": "", 2024-09-18T04:37:43.556 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dns_s3website_name": "", 2024-09-18T04:37:43.557 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dynamic_resharding": "true", 2024-09-18T04:37:43.557 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-18T04:37:43.557 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_enable_gc_threads": "true", 2024-09-18T04:37:43.557 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_enable_lc_threads": "true", 2024-09-18T04:37:43.557 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_enable_ops_log": "false", 2024-09-18T04:37:43.557 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_enable_quota_threads": "true", 2024-09-18T04:37:43.557 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_enable_static_website": "false", 2024-09-18T04:37:43.557 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_enable_usage_log": "false", 2024-09-18T04:37:43.557 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_enforce_swift_acls": "true", 2024-09-18T04:37:43.557 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_exit_timeout_secs": "120", 2024-09-18T04:37:43.558 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_expose_bucket": "false", 2024-09-18T04:37:43.558 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_extended_http_attrs": "", 2024-09-18T04:37:43.558 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_filter": "none", 2024-09-18T04:37:43.558 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-18T04:37:43.558 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_frontends": "beast port=7480", 2024-09-18T04:37:43.558 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-18T04:37:43.558 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_gc_max_deferred": "50", 2024-09-18T04:37:43.558 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-18T04:37:43.558 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_gc_max_objs": "32", 2024-09-18T04:37:43.558 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-18T04:37:43.558 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-18T04:37:43.558 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-18T04:37:43.558 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-18T04:37:43.558 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_gc_processor_period": "3600", 2024-09-18T04:37:43.559 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-18T04:37:43.559 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-18T04:37:43.559 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-18T04:37:43.559 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-18T04:37:43.559 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_init_timeout": "300", 2024-09-18T04:37:43.559 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-18T04:37:43.559 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-09-18T04:37:43.559 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-18T04:37:43.559 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-18T04:37:43.559 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_admin_domain": "", 2024-09-18T04:37:43.559 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_admin_password": "", 2024-09-18T04:37:43.559 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_admin_password_path": "", 2024-09-18T04:37:43.559 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_admin_project": "", 2024-09-18T04:37:43.559 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_admin_tenant": "", 2024-09-18T04:37:43.559 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_admin_token": "", 2024-09-18T04:37:43.560 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_admin_token_path": "", 2024-09-18T04:37:43.560 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_admin_user": "", 2024-09-18T04:37:43.560 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_api_version": "2", 2024-09-18T04:37:43.560 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_barbican_domain": "", 2024-09-18T04:37:43.560 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_barbican_password": "", 2024-09-18T04:37:43.560 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_barbican_project": "", 2024-09-18T04:37:43.560 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-18T04:37:43.560 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_barbican_user": "", 2024-09-18T04:37:43.560 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-18T04:37:43.560 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-18T04:37:43.560 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-18T04:37:43.560 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-18T04:37:43.560 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-18T04:37:43.560 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_url": "", 2024-09-18T04:37:43.561 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-18T04:37:43.561 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_lc_debug_interval": "-1", 2024-09-18T04:37:43.561 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_lc_lock_max_time": "90", 2024-09-18T04:37:43.561 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_lc_max_objs": "32", 2024-09-18T04:37:43.561 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_lc_max_rules": "1000", 2024-09-18T04:37:43.561 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_lc_max_worker": "3", 2024-09-18T04:37:43.561 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-18T04:37:43.561 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_lc_thread_delay": "0", 2024-09-18T04:37:43.561 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-18T04:37:43.561 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_ldap_dnattr": "uid", 2024-09-18T04:37:43.561 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-18T04:37:43.561 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_ldap_searchfilter": "", 2024-09-18T04:37:43.561 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-18T04:37:43.561 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_ldap_uri": "ldaps://", 2024-09-18T04:37:43.561 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-18T04:37:43.562 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-18T04:37:43.562 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-18T04:37:43.562 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_log_http_headers": "", 2024-09-18T04:37:43.562 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-18T04:37:43.562 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-18T04:37:43.562 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_log_object_name_utc": "false", 2024-09-18T04:37:43.562 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_luarocks_location": "/tmp/luarocks", 2024-09-18T04:37:43.562 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_attr_name_len": "0", 2024-09-18T04:37:43.562 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_attr_size": "0", 2024-09-18T04:37:43.562 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-18T04:37:43.562 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_chunk_size": "4194304", 2024-09-18T04:37:43.562 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-18T04:37:43.562 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-09-18T04:37:43.562 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-18T04:37:43.563 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_listing_results": "1000", 2024-09-18T04:37:43.563 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_notify_retries": "10", 2024-09-18T04:37:43.563 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-18T04:37:43.563 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_put_param_size": "1048576", 2024-09-18T04:37:43.563 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_put_size": "5368709120", 2024-09-18T04:37:43.563 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_slo_entries": "1000", 2024-09-18T04:37:43.563 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_md_log_max_shards": "64", 2024-09-18T04:37:43.563 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-18T04:37:43.563 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_meta_sync_poll_interval": "20", 2024-09-18T04:37:43.563 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_meta_sync_spawn_window": "20", 2024-09-18T04:37:43.563 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-18T04:37:43.563 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_mp_lock_max_time": "600", 2024-09-18T04:37:43.563 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-18T04:37:43.563 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-18T04:37:43.564 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-18T04:37:43.564 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-18T04:37:43.564 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-18T04:37:43.564 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-09-18T04:37:43.564 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-18T04:37:43.564 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-18T04:37:43.564 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_max_gc": "300", 2024-09-18T04:37:43.564 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-18T04:37:43.564 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-18T04:37:43.564 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-18T04:37:43.564 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-18T04:37:43.564 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-18T04:37:43.564 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-18T04:37:43.564 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-18T04:37:43.565 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_num_async_rados_threads": "32", 2024-09-18T04:37:43.565 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_num_control_oids": "8", 2024-09-18T04:37:43.565 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_numa_node": "-1", 2024-09-18T04:37:43.565 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-18T04:37:43.565 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-18T04:37:43.565 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_objexp_chunk_size": "100", 2024-09-18T04:37:43.565 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_objexp_gc_interval": "600", 2024-09-18T04:37:43.565 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-18T04:37:43.565 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-18T04:37:43.565 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-18T04:37:43.565 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_op_thread_timeout": "600", 2024-09-18T04:37:43.565 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_op_tracing": "false", 2024-09-18T04:37:43.565 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_opa_token": "", 2024-09-18T04:37:43.565 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_opa_url": "", 2024-09-18T04:37:43.566 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_opa_verify_ssl": "true", 2024-09-18T04:37:43.566 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-18T04:37:43.566 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-09-18T04:37:43.566 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_ops_log_rados": "false", 2024-09-18T04:37:43.566 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_ops_log_socket_path": "", 2024-09-18T04:37:43.566 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-18T04:37:43.566 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-09-18T04:37:43.566 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-18T04:37:43.566 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_period_push_interval": "2.000000", 2024-09-18T04:37:43.566 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-18T04:37:43.566 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-18T04:37:43.566 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-09-18T04:37:43.566 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_print_continue": "true", 2024-09-18T04:37:43.567 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-18T04:37:43.567 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-18T04:37:43.567 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-18T04:37:43.567 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-18T04:37:43.567 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-18T04:37:43.567 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_rados_tracing": "false", 2024-09-18T04:37:43.567 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_realm": "", 2024-09-18T04:37:43.567 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_realm_id": "", 2024-09-18T04:37:43.567 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-18T04:37:43.567 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_region": "", 2024-09-18T04:37:43.567 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-18T04:37:43.567 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-18T04:37:43.567 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-18T04:37:43.567 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-18T04:37:43.568 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_request_uri": "", 2024-09-18T04:37:43.568 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_reshard_batch_size": "64", 2024-09-18T04:37:43.568 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-18T04:37:43.568 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_reshard_max_aio": "128", 2024-09-18T04:37:43.568 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_reshard_num_logs": "16", 2024-09-18T04:37:43.568 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_reshard_thread_interval": "600", 2024-09-18T04:37:43.568 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_resolve_cname": "false", 2024-09-18T04:37:43.568 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-18T04:37:43.568 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_run_sync_thread": "true", 2024-09-18T04:37:43.568 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-18T04:37:43.568 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-18T04:37:43.568 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-18T04:37:43.568 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-18T04:37:43.569 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-18T04:37:43.569 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-18T04:37:43.569 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-18T04:37:43.569 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-18T04:37:43.569 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_scheduler_type": "throttler", 2024-09-18T04:37:43.569 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_script_uri": "", 2024-09-18T04:37:43.569 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_service_provider_name": "", 2024-09-18T04:37:43.569 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-18T04:37:43.569 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sts_client_id": "", 2024-09-18T04:37:43.569 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sts_client_secret": "", 2024-09-18T04:37:43.569 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sts_entry": "sts", 2024-09-18T04:37:43.569 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sts_key": "sts", 2024-09-18T04:37:43.569 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-18T04:37:43.570 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sts_min_session_duration": "900", 2024-09-18T04:37:43.570 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sts_token_introspection_url": "", 2024-09-18T04:37:43.570 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_swift_account_in_url": "false", 2024-09-18T04:37:43.570 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_swift_auth_entry": "auth", 2024-09-18T04:37:43.570 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_swift_auth_url": "", 2024-09-18T04:37:43.570 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_swift_custom_header": "", 2024-09-18T04:37:43.570 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-18T04:37:43.570 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_swift_need_stats": "true", 2024-09-18T04:37:43.570 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_swift_tenant_name": "", 2024-09-18T04:37:43.570 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_swift_token_expiration": "86400", 2024-09-18T04:37:43.570 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_swift_url": "", 2024-09-18T04:37:43.570 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_swift_url_prefix": "swift", 2024-09-18T04:37:43.570 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-18T04:37:43.571 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-09-18T04:37:43.571 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-18T04:37:43.571 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_lease_period": "120", 2024-09-18T04:37:43.571 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-18T04:37:43.571 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-18T04:37:43.571 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-18T04:37:43.571 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-18T04:37:43.571 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-18T04:37:43.571 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-18T04:37:43.571 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-18T04:37:43.571 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-18T04:37:43.571 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-18T04:37:43.572 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_thread_pool_size": "512", 2024-09-18T04:37:43.572 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_torrent_comment": "", 2024-09-18T04:37:43.572 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_torrent_createby": "", 2024-09-18T04:37:43.572 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_torrent_encoding": "", 2024-09-18T04:37:43.572 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_torrent_flag": "false", 2024-09-18T04:37:43.572 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_torrent_origin": "", 2024-09-18T04:37:43.572 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-18T04:37:43.572 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_torrent_tracker": "", 2024-09-18T04:37:43.572 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_trust_forwarded_https": "false", 2024-09-18T04:37:43.572 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-18T04:37:43.572 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-18T04:37:43.572 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_usage_max_shards": "32", 2024-09-18T04:37:43.572 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_usage_max_user_shards": "1", 2024-09-18T04:37:43.573 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_use_opa_authz": "false", 2024-09-18T04:37:43.573 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-18T04:37:43.573 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-18T04:37:43.573 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_user_max_buckets": "1000", 2024-09-18T04:37:43.573 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_user_policies_max_num": "100", 2024-09-18T04:37:43.573 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-18T04:37:43.573 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-18T04:37:43.573 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-18T04:37:43.573 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-18T04:37:43.573 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_user_unique_email": "true", 2024-09-18T04:37:43.573 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_verify_ssl": "true", 2024-09-18T04:37:43.573 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-18T04:37:43.573 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_zone": "", 2024-09-18T04:37:43.574 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_zone_id": "", 2024-09-18T04:37:43.574 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-18T04:37:43.574 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_zonegroup": "", 2024-09-18T04:37:43.574 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_zonegroup_id": "", 2024-09-18T04:37:43.574 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-18T04:37:43.574 INFO:teuthology.orchestra.run.smithi028.stdout: "rgwlc_auto_session_clear": "true", 2024-09-18T04:37:43.574 INFO:teuthology.orchestra.run.smithi028.stdout: "rgwlc_skip_bucket_step": "false", 2024-09-18T04:37:43.574 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_block_size": "4096", 2024-09-18T04:37:43.574 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-18T04:37:43.574 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-18T04:37:43.574 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-18T04:37:43.574 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-18T04:37:43.574 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-18T04:37:43.575 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_cache_size": "536870912", 2024-09-18T04:37:43.575 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-18T04:37:43.575 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-18T04:37:43.575 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-18T04:37:43.575 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-18T04:37:43.575 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-18T04:37:43.575 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-18T04:37:43.575 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-18T04:37:43.575 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-18T04:37:43.575 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_index_type": "binary_search", 2024-09-18T04:37:43.575 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-18T04:37:43.575 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-18T04:37:43.575 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_partition_filters": "false", 2024-09-18T04:37:43.575 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_perf": "false", 2024-09-18T04:37:43.576 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-18T04:37:43.576 INFO:teuthology.orchestra.run.smithi028.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-18T04:37:43.576 INFO:teuthology.orchestra.run.smithi028.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-18T04:37:43.576 INFO:teuthology.orchestra.run.smithi028.stdout: "run_dir": "/var/run/ceph", 2024-09-18T04:37:43.576 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_block_create": "true", 2024-09-18T04:37:43.576 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_cache_lru_size": "67108864", 2024-09-18T04:37:43.576 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_cbjournal_size": "5368709120", 2024-09-18T04:37:43.576 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_default_max_object_size": "16777216", 2024-09-18T04:37:43.576 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-18T04:37:43.576 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_device_size": "53687091200", 2024-09-18T04:37:43.576 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_journal_batch_capacity": "16", 2024-09-18T04:37:43.576 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-18T04:37:43.576 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-18T04:37:43.577 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-18T04:37:43.577 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_main_device_type": "SSD", 2024-09-18T04:37:43.577 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_max_concurrent_transactions": "8", 2024-09-18T04:37:43.577 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-09-18T04:37:43.577 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-09-18T04:37:43.577 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-09-18T04:37:43.577 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-09-18T04:37:43.577 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_segment_size": "67108864", 2024-09-18T04:37:43.577 INFO:teuthology.orchestra.run.smithi028.stdout: "set_keepcaps": "false", 2024-09-18T04:37:43.577 INFO:teuthology.orchestra.run.smithi028.stdout: "setgroup": "", 2024-09-18T04:37:43.577 INFO:teuthology.orchestra.run.smithi028.stdout: "setuser": "", 2024-09-18T04:37:43.577 INFO:teuthology.orchestra.run.smithi028.stdout: "setuser_match_path": "", 2024-09-18T04:37:43.577 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-18T04:37:43.578 INFO:teuthology.orchestra.run.smithi028.stdout: "thp": "false", 2024-09-18T04:37:43.578 INFO:teuthology.orchestra.run.smithi028.stdout: "threadpool_default_timeout": "60", 2024-09-18T04:37:43.578 INFO:teuthology.orchestra.run.smithi028.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-18T04:37:43.578 INFO:teuthology.orchestra.run.smithi028.stdout: "throttler_perf_counter": "true" 2024-09-18T04:37:43.578 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:37:43.580 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-18T04:37:43.725 INFO:teuthology.orchestra.run.smithi028.stdout:mon.a: {} 2024-09-18T04:37:43.739 INFO:teuthology.orchestra.run.smithi028.stdout:mon.b: {} 2024-09-18T04:37:43.756 INFO:teuthology.orchestra.run.smithi028.stdout:mon.c: {} 2024-09-18T04:37:43.758 INFO:teuthology.orchestra.run.smithi028.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-18T04:37:43.758 INFO:teuthology.orchestra.run.smithi028.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-18T04:37:43.758 INFO:teuthology.orchestra.run.smithi028.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-18T04:37:43.771 DEBUG:teuthology.orchestra.run.smithi028:> type ceph-objectstore-tool 2024-09-18T04:37:43.777 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-18T04:37:43.777 DEBUG:teuthology.orchestra.run.smithi082:> type ceph-objectstore-tool 2024-09-18T04:37:43.783 INFO:teuthology.orchestra.run.smithi082.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-18T04:37:43.784 INFO:teuthology.run_tasks:Running task workunit... 2024-09-18T04:37:43.796 INFO:tasks.workunit:Pulling workunits from ref a477b1894d5c5a613f23260ca0e6b96d79652ba7 2024-09-18T04:37:43.797 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-18T04:37:43.797 DEBUG:teuthology.orchestra.run.smithi115:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-18T04:37:43.797 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-18T04:37:43.797 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-18T04:37:43.797 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T04:37:43.798 INFO:tasks.thrashosds.thrasher:Removing osd 2, in_osds are: [0, 1, 2, 3, 4, 5] 2024-09-18T04:37:43.798 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 2 2024-09-18T04:37:43.798 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-18T04:37:43.798 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-18T04:37:43.799 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T04:37:43.799 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-18T04:37:43.799 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:37:43.799 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-18T04:37:43.800 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-18T04:37:43.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T04:37:43.807 INFO:teuthology.orchestra.run.smithi115.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-18T04:37:43.807 DEBUG:teuthology.orchestra.run.smithi115:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-18T04:37:43.814 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-18T04:37:43.814 DEBUG:teuthology.orchestra.run.smithi115:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-18T04:37:43.826 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:37:43.823+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:37:43.862 INFO:tasks.workunit:timeout=3h 2024-09-18T04:37:43.862 INFO:tasks.workunit:cleanup=True 2024-09-18T04:37:43.863 DEBUG:teuthology.orchestra.run.smithi115:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a477b1894d5c5a613f23260ca0e6b96d79652ba7 2024-09-18T04:37:43.872 INFO:tasks.workunit.client.0.smithi115.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-18T04:37:43.900 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:37:43.895+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:37:43.916 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:37:43.916 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:37:43.916 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:37:43.916 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:37:43.926 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:37:43.952 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:37:43.974 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:37:43.992 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:37:44.001 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:37:44.000+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:37:44.013 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:37:44.037 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:37:44.061 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:37:44.063 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:37:44.063 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:37:44.064 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:37:44.064 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:37:44.064 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:37:44.064 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:37:44.078 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:44.078 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:44.092 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:37:44.114 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-09-18T04:37:44.112+0000 7f8dc00db700 -1 mon.a@0(leader).osd e20 definitely_dead 0 2024-09-18T04:37:44.114 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:37:44.112+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:37:44.203 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:37:44.199+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:37:44.236 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:44.236 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:44.246 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:37:44.304 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:37:44.300+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:37:44.351 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:44.351 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:44.360 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:37:44.405 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:37:44.404+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:37:44.470 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:44.470 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:44.480 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:37:44.522 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:37:44.520+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:37:44.525 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-09-18T04:37:44.527 INFO:teuthology.orchestra.run.smithi028.stderr:marked out osd.2. 2024-09-18T04:37:44.608 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:37:44.604+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:37:44.627 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:44.627 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:44.637 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:37:44.708 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:37:44.704+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:37:44.747 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:44.747 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:44.757 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:37:44.810 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:37:44.807+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:37:44.869 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:44.869 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:44.879 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:37:44.910 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:37:44.908+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:37:44.988 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:44.988 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:44.998 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:37:45.042 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:37:45.040+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:37:45.113 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:37:45.107+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:37:45.118 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:45.118 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:45.128 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:37:45.213 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:37:45.207+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:37:45.232 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:45.232 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:45.242 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:37:45.314 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:37:45.307+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:37:45.350 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:45.350 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:45.359 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:37:45.414 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:37:45.411+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:37:45.462 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:45.463 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:45.472 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:37:45.518 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:37:45.515+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:37:45.575 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:45.575 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:45.584 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:37:45.616 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:37:45.611+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:37:45.689 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:45.689 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:45.698 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:37:45.742 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:37:45.739+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:37:45.807 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:45.807 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:45.816 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:37:45.817 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:37:45.811+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:37:45.918 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:45.918 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:45.919 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:37:45.915+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:37:45.928 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:37:46.020 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:37:46.016+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:37:46.035 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:46.035 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:46.044 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:37:46.074 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T04:37:46.120 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:37:46.115+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:37:46.160 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:46.160 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:46.171 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:37:46.222 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:37:46.215+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:37:46.226 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:37:46.248 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:37:46.267 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:37:46.277 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:37:46.277 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:37:46.277 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:37:46.277 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:37:46.277 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:37:46.287 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:37:46.289 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:37:46.312 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:37:46.323 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:37:46.319+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:37:46.334 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:37:46.337 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:37:46.337 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:37:46.337 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:37:46.337 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:37:46.337 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:37:46.337 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:37:46.396 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:37:46.396 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:37:46.396 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:37:46.396 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:37:46.397 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.397 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:30 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'16, mlcod=15'14)", 2024-09-18T04:37:46.397 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.486330+0000", 2024-09-18T04:37:46.397 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 18.906380498000001, 2024-09-18T04:37:46.397 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026383700000000001, 2024-09-18T04:37:46.397 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:46.397 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:46.397 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:46.397 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.397 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:46.397 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486330+0000", 2024-09-18T04:37:46.397 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:46.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.398 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.398 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:46.398 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486330+0000", 2024-09-18T04:37:46.398 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:46.398 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.398 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.398 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:46.398 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486328+0000", 2024-09-18T04:37:46.398 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:37:46.398 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.398 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.398 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:46.398 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486342+0000", 2024-09-18T04:37:46.398 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3783e-05 2024-09-18T04:37:46.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.399 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.399 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:46.399 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486373+0000", 2024-09-18T04:37:46.399 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1362999999999998e-05 2024-09-18T04:37:46.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.399 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.399 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:46.399 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486387+0000", 2024-09-18T04:37:46.399 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3219e-05 2024-09-18T04:37:46.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.399 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.399 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:46.399 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486489+0000", 2024-09-18T04:37:46.400 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010232499999999999 2024-09-18T04:37:46.400 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.400 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.400 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:46.400 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486524+0000", 2024-09-18T04:37:46.400 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4972000000000003e-05 2024-09-18T04:37:46.400 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.400 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.400 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:46.400 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.488902+0000", 2024-09-18T04:37:46.400 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023776240000000001 2024-09-18T04:37:46.400 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.400 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.401 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:46.401 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.488968+0000", 2024-09-18T04:37:46.401 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6761000000000007e-05 2024-09-18T04:37:46.401 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:46.401 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:46.401 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:46.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.401 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.401 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:35 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'21, mlcod=15'17)", 2024-09-18T04:37:46.401 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.491817+0000", 2024-09-18T04:37:46.401 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 18.900893580000002, 2024-09-18T04:37:46.401 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023524739999999998, 2024-09-18T04:37:46.401 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:46.401 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:46.402 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:46.402 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.402 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:46.402 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491817+0000", 2024-09-18T04:37:46.402 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:46.402 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.402 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.402 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:46.402 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491817+0000", 2024-09-18T04:37:46.402 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:46.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.403 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.403 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:46.403 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491815+0000", 2024-09-18T04:37:46.403 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:37:46.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.403 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.403 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:46.403 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491890+0000", 2024-09-18T04:37:46.403 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4500999999999997e-05 2024-09-18T04:37:46.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.403 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.404 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:46.404 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491916+0000", 2024-09-18T04:37:46.404 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5666000000000001e-05 2024-09-18T04:37:46.404 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.404 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.404 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:46.404 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491928+0000", 2024-09-18T04:37:46.404 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2585999999999999e-05 2024-09-18T04:37:46.404 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.404 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.404 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:46.404 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.492000+0000", 2024-09-18T04:37:46.405 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2309000000000002e-05 2024-09-18T04:37:46.405 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.405 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.405 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:46.405 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.492027+0000", 2024-09-18T04:37:46.405 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6251e-05 2024-09-18T04:37:46.405 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.405 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.405 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:46.405 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.494114+0000", 2024-09-18T04:37:46.405 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020875260000000001 2024-09-18T04:37:46.406 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.406 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.406 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:46.406 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.494169+0000", 2024-09-18T04:37:46.406 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5145000000000001e-05 2024-09-18T04:37:46.406 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:46.406 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:46.406 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:46.406 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.406 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.406 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:57 1.0 e15/14 1:30306672:devicehealth::main.db.0000000000000000:head v 15'35, mlcod=15'33)", 2024-09-18T04:37:46.406 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.529997+0000", 2024-09-18T04:37:46.406 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 18.862713725999999, 2024-09-18T04:37:46.407 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002623869, 2024-09-18T04:37:46.407 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:46.407 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:46.407 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:46.407 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.407 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:46.407 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.529997+0000", 2024-09-18T04:37:46.407 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:46.407 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.407 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.407 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:46.407 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.529997+0000", 2024-09-18T04:37:46.407 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:46.407 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.408 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.408 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:46.408 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.529995+0000", 2024-09-18T04:37:46.408 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:37:46.408 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.408 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.408 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:46.408 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530116+0000", 2024-09-18T04:37:46.408 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000121141 2024-09-18T04:37:46.408 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.408 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.408 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:46.408 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530146+0000", 2024-09-18T04:37:46.408 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0423e-05 2024-09-18T04:37:46.409 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.409 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.409 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:46.409 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530159+0000", 2024-09-18T04:37:46.409 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2802999999999999e-05 2024-09-18T04:37:46.409 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.409 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.409 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:46.409 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530218+0000", 2024-09-18T04:37:46.409 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8461000000000002e-05 2024-09-18T04:37:46.409 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.409 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.409 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:46.410 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530253+0000", 2024-09-18T04:37:46.410 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4796999999999999e-05 2024-09-18T04:37:46.410 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.410 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.410 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:46.410 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.532559+0000", 2024-09-18T04:37:46.410 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002306939 2024-09-18T04:37:46.410 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.410 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.410 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:46.410 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.532621+0000", 2024-09-18T04:37:46.410 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1131999999999997e-05 2024-09-18T04:37:46.410 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:46.411 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:46.411 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:46.411 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.411 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.411 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:68 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'42, mlcod=15'39)", 2024-09-18T04:37:46.411 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.546404+0000", 2024-09-18T04:37:46.411 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 18.846306315, 2024-09-18T04:37:46.411 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024499769999999999, 2024-09-18T04:37:46.411 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:46.411 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:46.411 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:46.411 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.411 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:46.412 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546404+0000", 2024-09-18T04:37:46.412 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:46.412 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.412 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.412 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:46.412 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546404+0000", 2024-09-18T04:37:46.412 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:46.412 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.412 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.412 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:46.412 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546403+0000", 2024-09-18T04:37:46.412 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:37:46.412 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.412 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.413 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:46.413 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546416+0000", 2024-09-18T04:37:46.413 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.306e-05 2024-09-18T04:37:46.413 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.413 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.413 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:46.413 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546465+0000", 2024-09-18T04:37:46.413 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9441999999999999e-05 2024-09-18T04:37:46.413 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.413 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.413 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:46.414 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546480+0000", 2024-09-18T04:37:46.414 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4919999999999999e-05 2024-09-18T04:37:46.414 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.414 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.414 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:46.414 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546541+0000", 2024-09-18T04:37:46.414 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1085000000000002e-05 2024-09-18T04:37:46.414 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.414 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.414 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:46.414 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546569+0000", 2024-09-18T04:37:46.414 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7803000000000001e-05 2024-09-18T04:37:46.414 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.415 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.415 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:46.415 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.548725+0000", 2024-09-18T04:37:46.415 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021556629999999999 2024-09-18T04:37:46.415 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.415 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.415 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:46.415 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.548854+0000", 2024-09-18T04:37:46.415 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000129524 2024-09-18T04:37:46.415 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:46.415 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:46.415 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:46.415 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.416 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.416 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:69 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'43, mlcod=15'39)", 2024-09-18T04:37:46.416 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.547520+0000", 2024-09-18T04:37:46.416 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 18.845190239000001, 2024-09-18T04:37:46.416 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023506820000000002, 2024-09-18T04:37:46.416 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:46.416 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:46.416 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:46.416 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.416 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:46.416 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547520+0000", 2024-09-18T04:37:46.416 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:46.416 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.417 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.417 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:46.417 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547520+0000", 2024-09-18T04:37:46.417 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:46.417 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.417 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.417 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:46.417 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547519+0000", 2024-09-18T04:37:46.417 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:37:46.417 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.417 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.417 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:46.417 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547614+0000", 2024-09-18T04:37:46.418 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5235000000000006e-05 2024-09-18T04:37:46.418 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.418 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.418 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:46.418 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547640+0000", 2024-09-18T04:37:46.418 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5698999999999999e-05 2024-09-18T04:37:46.418 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.418 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.418 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:46.418 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547651+0000", 2024-09-18T04:37:46.418 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.134e-05 2024-09-18T04:37:46.418 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.418 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.418 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:46.419 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547805+0000", 2024-09-18T04:37:46.419 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015426099999999999 2024-09-18T04:37:46.419 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.419 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.419 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:46.419 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547844+0000", 2024-09-18T04:37:46.419 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8229999999999998e-05 2024-09-18T04:37:46.419 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.419 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.419 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:46.419 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.549810+0000", 2024-09-18T04:37:46.419 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0019660519999999998 2024-09-18T04:37:46.420 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.420 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.420 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:46.420 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.549871+0000", 2024-09-18T04:37:46.420 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1152e-05 2024-09-18T04:37:46.420 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:46.420 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:46.420 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:46.420 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.420 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.420 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:83 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'53, mlcod=15'50)", 2024-09-18T04:37:46.420 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.575989+0000", 2024-09-18T04:37:46.420 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 18.816721291, 2024-09-18T04:37:46.420 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023991310000000001, 2024-09-18T04:37:46.421 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:46.421 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:46.421 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:46.421 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.421 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:46.421 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.575989+0000", 2024-09-18T04:37:46.421 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:46.421 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.421 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.421 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:46.421 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.575989+0000", 2024-09-18T04:37:46.421 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:46.421 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.422 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.422 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:46.422 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.575988+0000", 2024-09-18T04:37:46.422 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:37:46.422 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.422 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.422 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:46.422 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576001+0000", 2024-09-18T04:37:46.422 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2751e-05 2024-09-18T04:37:46.422 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.422 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.422 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:46.422 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576023+0000", 2024-09-18T04:37:46.422 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2453999999999999e-05 2024-09-18T04:37:46.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.423 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.423 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:46.423 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576038+0000", 2024-09-18T04:37:46.423 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5125e-05 2024-09-18T04:37:46.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.423 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.423 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:46.423 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576161+0000", 2024-09-18T04:37:46.423 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000122543 2024-09-18T04:37:46.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.423 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.423 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:46.423 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576221+0000", 2024-09-18T04:37:46.424 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9948000000000003e-05 2024-09-18T04:37:46.424 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.424 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.424 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:46.424 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.578277+0000", 2024-09-18T04:37:46.424 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020567559999999999 2024-09-18T04:37:46.424 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.424 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.424 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:46.424 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.578388+0000", 2024-09-18T04:37:46.424 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011089100000000001 2024-09-18T04:37:46.424 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:46.424 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:46.425 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:46.425 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.425 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.425 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:86 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'56, mlcod=15'52)", 2024-09-18T04:37:46.425 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.580261+0000", 2024-09-18T04:37:46.425 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 18.812449655000002, 2024-09-18T04:37:46.425 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002418108, 2024-09-18T04:37:46.425 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:46.425 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:46.425 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:46.425 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.425 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:46.425 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580261+0000", 2024-09-18T04:37:46.425 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:46.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.426 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.426 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:46.426 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580261+0000", 2024-09-18T04:37:46.426 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:46.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.426 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.426 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:46.426 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580260+0000", 2024-09-18T04:37:46.426 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:37:46.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.426 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.426 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:46.427 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580269+0000", 2024-09-18T04:37:46.427 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5109999999999992e-06 2024-09-18T04:37:46.427 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.427 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.427 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:46.427 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580288+0000", 2024-09-18T04:37:46.427 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8783999999999998e-05 2024-09-18T04:37:46.427 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.427 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.427 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:46.427 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580296+0000", 2024-09-18T04:37:46.427 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3250000000000008e-06 2024-09-18T04:37:46.427 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.427 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.428 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:46.428 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580351+0000", 2024-09-18T04:37:46.428 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4132000000000003e-05 2024-09-18T04:37:46.428 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.428 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.428 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:46.428 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580381+0000", 2024-09-18T04:37:46.428 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0263000000000001e-05 2024-09-18T04:37:46.428 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.428 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.428 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:46.428 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.582615+0000", 2024-09-18T04:37:46.428 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0022339109999999999 2024-09-18T04:37:46.429 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.429 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.429 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:46.429 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.582679+0000", 2024-09-18T04:37:46.429 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4182000000000003e-05 2024-09-18T04:37:46.429 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:46.429 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:46.429 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:46.429 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.429 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.429 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:105 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'67, mlcod=15'64)", 2024-09-18T04:37:46.429 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.789089+0000", 2024-09-18T04:37:46.429 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 18.603621485000001, 2024-09-18T04:37:46.430 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024384229999999999, 2024-09-18T04:37:46.430 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:46.430 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:46.430 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:46.430 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.430 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:46.430 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789089+0000", 2024-09-18T04:37:46.430 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:46.430 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.430 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.430 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:46.430 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789089+0000", 2024-09-18T04:37:46.430 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:46.431 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.431 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.431 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:46.431 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789088+0000", 2024-09-18T04:37:46.431 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:37:46.431 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.431 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.431 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:46.431 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789107+0000", 2024-09-18T04:37:46.431 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9017000000000001e-05 2024-09-18T04:37:46.431 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.431 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.431 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:46.432 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789139+0000", 2024-09-18T04:37:46.432 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2261999999999997e-05 2024-09-18T04:37:46.432 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.432 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.432 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:46.432 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789153+0000", 2024-09-18T04:37:46.432 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4025999999999999e-05 2024-09-18T04:37:46.432 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.432 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.432 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:46.432 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789255+0000", 2024-09-18T04:37:46.432 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000101969 2024-09-18T04:37:46.432 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.433 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.433 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:46.433 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789314+0000", 2024-09-18T04:37:46.433 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8867999999999999e-05 2024-09-18T04:37:46.433 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.433 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.433 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:46.433 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.791453+0000", 2024-09-18T04:37:46.433 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002138938 2024-09-18T04:37:46.433 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.433 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.433 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:46.434 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.791527+0000", 2024-09-18T04:37:46.434 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4464999999999999e-05 2024-09-18T04:37:46.434 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:46.434 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:46.434 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:46.434 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.434 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.434 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:106 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'68, mlcod=15'64)", 2024-09-18T04:37:46.434 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.790295+0000", 2024-09-18T04:37:46.434 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 18.602415634, 2024-09-18T04:37:46.434 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024379390000000001, 2024-09-18T04:37:46.434 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:46.434 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:46.434 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:46.435 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.435 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:46.435 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790295+0000", 2024-09-18T04:37:46.435 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:46.435 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.435 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.435 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:46.435 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790295+0000", 2024-09-18T04:37:46.435 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:46.435 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.435 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.435 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:46.436 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790294+0000", 2024-09-18T04:37:46.436 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:37:46.436 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.436 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.436 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:46.436 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790390+0000", 2024-09-18T04:37:46.436 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6537000000000001e-05 2024-09-18T04:37:46.436 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.436 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.436 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:46.436 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790418+0000", 2024-09-18T04:37:46.436 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8218e-05 2024-09-18T04:37:46.436 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.436 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.437 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:46.437 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790431+0000", 2024-09-18T04:37:46.437 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2571999999999999e-05 2024-09-18T04:37:46.437 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.437 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.437 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:46.437 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790534+0000", 2024-09-18T04:37:46.437 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000103433 2024-09-18T04:37:46.437 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.437 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.437 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:46.437 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790565+0000", 2024-09-18T04:37:46.437 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0409999999999999e-05 2024-09-18T04:37:46.437 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.438 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.438 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:46.438 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.792651+0000", 2024-09-18T04:37:46.438 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020865250000000001 2024-09-18T04:37:46.438 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.438 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.438 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:46.438 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.792733+0000", 2024-09-18T04:37:46.438 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1477999999999995e-05 2024-09-18T04:37:46.438 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:46.438 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:46.438 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:46.439 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.439 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.439 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:109 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'71, mlcod=15'68)", 2024-09-18T04:37:46.439 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.795250+0000", 2024-09-18T04:37:46.439 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 18.59746062, 2024-09-18T04:37:46.439 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023990449999999998, 2024-09-18T04:37:46.439 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:46.439 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:46.439 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:46.439 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.439 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:46.439 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795250+0000", 2024-09-18T04:37:46.439 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:46.440 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.440 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.440 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:46.440 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795250+0000", 2024-09-18T04:37:46.440 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:46.440 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.440 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.440 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:46.440 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795248+0000", 2024-09-18T04:37:46.440 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:37:46.440 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.440 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.440 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:46.440 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795405+0000", 2024-09-18T04:37:46.441 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000157007 2024-09-18T04:37:46.441 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.441 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.441 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:46.441 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795422+0000", 2024-09-18T04:37:46.441 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7013000000000001e-05 2024-09-18T04:37:46.441 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.441 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.441 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:46.441 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795432+0000", 2024-09-18T04:37:46.441 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0448999999999999e-05 2024-09-18T04:37:46.441 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.441 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.442 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:46.442 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795534+0000", 2024-09-18T04:37:46.442 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000101195 2024-09-18T04:37:46.442 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.442 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.442 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:46.442 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795587+0000", 2024-09-18T04:37:46.442 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3356000000000002e-05 2024-09-18T04:37:46.442 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.442 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.442 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:46.443 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.797560+0000", 2024-09-18T04:37:46.443 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001973023 2024-09-18T04:37:46.443 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.443 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.443 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:46.443 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.797649+0000", 2024-09-18T04:37:46.443 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8864999999999997e-05 2024-09-18T04:37:46.443 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:46.443 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:46.443 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:46.443 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.443 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.443 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:115 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'77, mlcod=15'72)", 2024-09-18T04:37:46.443 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.803339+0000", 2024-09-18T04:37:46.444 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 18.589371902, 2024-09-18T04:37:46.444 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002385093, 2024-09-18T04:37:46.444 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:46.444 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:46.444 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:46.444 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.444 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:46.444 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803339+0000", 2024-09-18T04:37:46.444 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:46.444 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.444 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.444 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:46.444 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803339+0000", 2024-09-18T04:37:46.445 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:46.445 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.445 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.445 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:46.445 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803337+0000", 2024-09-18T04:37:46.445 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:37:46.445 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.445 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.445 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:46.445 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803475+0000", 2024-09-18T04:37:46.445 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013718199999999999 2024-09-18T04:37:46.445 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.445 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.446 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:46.446 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803507+0000", 2024-09-18T04:37:46.446 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2351000000000002e-05 2024-09-18T04:37:46.446 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.446 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.446 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:46.446 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803533+0000", 2024-09-18T04:37:46.446 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6010999999999999e-05 2024-09-18T04:37:46.446 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.446 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.446 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:46.446 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803609+0000", 2024-09-18T04:37:46.446 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.64e-05 2024-09-18T04:37:46.447 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.447 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.447 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:46.447 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803671+0000", 2024-09-18T04:37:46.447 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1467e-05 2024-09-18T04:37:46.447 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.447 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.447 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:46.447 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.805676+0000", 2024-09-18T04:37:46.447 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020047849999999998 2024-09-18T04:37:46.447 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.447 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.447 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:46.447 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.805724+0000", 2024-09-18T04:37:46.448 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8151e-05 2024-09-18T04:37:46.448 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:46.448 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:46.448 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:46.448 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.448 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.448 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:118 1.0 e15/14 1:30306672:devicehealth::main.db.0000000000000000:head v 15'80, mlcod=15'78)", 2024-09-18T04:37:46.448 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.807930+0000", 2024-09-18T04:37:46.448 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 18.584780399, 2024-09-18T04:37:46.448 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028002259999999998, 2024-09-18T04:37:46.448 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:46.448 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:46.448 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:46.449 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.449 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:46.449 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.807930+0000", 2024-09-18T04:37:46.449 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:46.449 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.449 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.449 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:46.449 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.807930+0000", 2024-09-18T04:37:46.449 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:46.449 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.449 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.449 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:46.449 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.807929+0000", 2024-09-18T04:37:46.450 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:37:46.450 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.450 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.450 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:46.450 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808024+0000", 2024-09-18T04:37:46.450 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5258000000000002e-05 2024-09-18T04:37:46.450 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.450 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.450 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:46.450 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808044+0000", 2024-09-18T04:37:46.450 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0203e-05 2024-09-18T04:37:46.450 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.450 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.451 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:46.451 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808053+0000", 2024-09-18T04:37:46.451 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2689999999999999e-06 2024-09-18T04:37:46.451 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.451 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.451 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:46.451 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808137+0000", 2024-09-18T04:37:46.451 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4441e-05 2024-09-18T04:37:46.451 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.451 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.451 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:46.451 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808172+0000", 2024-09-18T04:37:46.451 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4453999999999999e-05 2024-09-18T04:37:46.452 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.452 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.452 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:46.452 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.810663+0000", 2024-09-18T04:37:46.452 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024910269999999998 2024-09-18T04:37:46.452 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.452 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.452 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:46.452 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.810730+0000", 2024-09-18T04:37:46.452 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7658999999999997e-05 2024-09-18T04:37:46.452 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:46.452 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:46.453 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:46.453 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.453 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.453 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:129 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'87, mlcod=15'85)", 2024-09-18T04:37:46.453 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.231689+0000", 2024-09-18T04:37:46.453 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 18.161021386000002, 2024-09-18T04:37:46.453 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002376502, 2024-09-18T04:37:46.453 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:46.453 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:46.453 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:46.453 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.453 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:46.453 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231689+0000", 2024-09-18T04:37:46.454 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:46.454 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.454 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.454 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:46.454 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231689+0000", 2024-09-18T04:37:46.454 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:46.454 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.454 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.454 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:46.454 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231687+0000", 2024-09-18T04:37:46.454 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:37:46.454 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.454 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.455 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:46.455 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231702+0000", 2024-09-18T04:37:46.455 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4949999999999999e-05 2024-09-18T04:37:46.455 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.455 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.455 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:46.455 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231735+0000", 2024-09-18T04:37:46.455 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2276000000000001e-05 2024-09-18T04:37:46.455 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.455 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.455 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:46.455 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231750+0000", 2024-09-18T04:37:46.455 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5193e-05 2024-09-18T04:37:46.456 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.456 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.456 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:46.456 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231872+0000", 2024-09-18T04:37:46.456 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000122127 2024-09-18T04:37:46.456 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.456 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.456 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:46.456 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231936+0000", 2024-09-18T04:37:46.456 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4089000000000003e-05 2024-09-18T04:37:46.456 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.456 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.456 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:46.457 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234002+0000", 2024-09-18T04:37:46.457 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002065575 2024-09-18T04:37:46.457 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.457 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.457 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:46.457 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234066+0000", 2024-09-18T04:37:46.457 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3944999999999998e-05 2024-09-18T04:37:46.457 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:46.457 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:46.457 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:46.457 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.457 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.457 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:130 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'88, mlcod=15'85)", 2024-09-18T04:37:46.457 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.232853+0000", 2024-09-18T04:37:46.458 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 18.159857770999999, 2024-09-18T04:37:46.458 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027351739999999999, 2024-09-18T04:37:46.458 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:46.458 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:46.458 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:46.458 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.458 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:46.458 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232853+0000", 2024-09-18T04:37:46.458 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:46.458 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.458 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.458 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:46.458 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232853+0000", 2024-09-18T04:37:46.459 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:46.459 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.459 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.459 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:46.459 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232851+0000", 2024-09-18T04:37:46.459 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:37:46.459 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.459 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.459 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:46.459 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232866+0000", 2024-09-18T04:37:46.459 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4946000000000001e-05 2024-09-18T04:37:46.459 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.459 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.459 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:46.460 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232897+0000", 2024-09-18T04:37:46.460 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0982999999999998e-05 2024-09-18T04:37:46.460 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.460 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.460 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:46.460 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232915+0000", 2024-09-18T04:37:46.460 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8476000000000001e-05 2024-09-18T04:37:46.460 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.460 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.460 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:46.460 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.233029+0000", 2024-09-18T04:37:46.460 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011305400000000001 2024-09-18T04:37:46.460 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.461 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.461 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:46.461 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.233079+0000", 2024-09-18T04:37:46.461 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0668999999999999e-05 2024-09-18T04:37:46.461 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.461 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.461 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:46.461 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235534+0000", 2024-09-18T04:37:46.461 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024546030000000001 2024-09-18T04:37:46.461 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.461 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.461 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:46.461 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235588+0000", 2024-09-18T04:37:46.462 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4098999999999998e-05 2024-09-18T04:37:46.462 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:46.462 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:46.462 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:46.462 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.462 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.462 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:131 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'89, mlcod=15'85)", 2024-09-18T04:37:46.462 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.234089+0000", 2024-09-18T04:37:46.462 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 18.158621917000001, 2024-09-18T04:37:46.462 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026975609999999998, 2024-09-18T04:37:46.462 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:46.462 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:46.463 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:46.463 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.463 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:46.463 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234089+0000", 2024-09-18T04:37:46.463 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:46.463 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.463 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.463 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:46.463 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234089+0000", 2024-09-18T04:37:46.463 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:46.463 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.463 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.463 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:46.463 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234087+0000", 2024-09-18T04:37:46.464 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:37:46.464 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.464 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.464 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:46.464 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234145+0000", 2024-09-18T04:37:46.464 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8687999999999998e-05 2024-09-18T04:37:46.464 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.464 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.464 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:46.464 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234177+0000", 2024-09-18T04:37:46.464 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1526999999999998e-05 2024-09-18T04:37:46.464 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.465 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.465 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:46.465 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234190+0000", 2024-09-18T04:37:46.465 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2965999999999999e-05 2024-09-18T04:37:46.465 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.465 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.465 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:46.465 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234309+0000", 2024-09-18T04:37:46.465 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011875399999999999 2024-09-18T04:37:46.465 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.465 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.465 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:46.465 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234365+0000", 2024-09-18T04:37:46.466 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6436e-05 2024-09-18T04:37:46.466 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.466 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.466 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:46.466 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236675+0000", 2024-09-18T04:37:46.466 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023099520000000001 2024-09-18T04:37:46.466 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.466 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.466 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:46.466 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236786+0000", 2024-09-18T04:37:46.466 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000111025 2024-09-18T04:37:46.466 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:46.466 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:46.466 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:46.467 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.467 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.467 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:132 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'90, mlcod=15'85)", 2024-09-18T04:37:46.467 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.235300+0000", 2024-09-18T04:37:46.467 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 18.157410466999998, 2024-09-18T04:37:46.467 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025982599999999998, 2024-09-18T04:37:46.467 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:46.467 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:46.467 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:46.467 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.467 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:46.467 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235300+0000", 2024-09-18T04:37:46.467 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:46.467 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.467 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.468 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:46.468 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235300+0000", 2024-09-18T04:37:46.468 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:46.468 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.468 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.468 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:46.468 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235298+0000", 2024-09-18T04:37:46.468 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:37:46.468 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.468 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.468 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:46.468 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235318+0000", 2024-09-18T04:37:46.468 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9714000000000001e-05 2024-09-18T04:37:46.469 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.469 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.469 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:46.469 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235337+0000", 2024-09-18T04:37:46.469 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8892000000000001e-05 2024-09-18T04:37:46.469 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.469 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.469 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:46.469 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235348+0000", 2024-09-18T04:37:46.469 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.129e-05 2024-09-18T04:37:46.469 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.469 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.469 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:46.469 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235681+0000", 2024-09-18T04:37:46.470 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00033323000000000001 2024-09-18T04:37:46.470 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.470 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.470 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:46.470 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235733+0000", 2024-09-18T04:37:46.470 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1391999999999999e-05 2024-09-18T04:37:46.470 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.470 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.470 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:46.470 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237839+0000", 2024-09-18T04:37:46.470 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002105647 2024-09-18T04:37:46.470 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.470 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.471 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:46.471 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237898+0000", 2024-09-18T04:37:46.471 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9759000000000002e-05 2024-09-18T04:37:46.471 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:46.471 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:46.471 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:46.471 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.471 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.471 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:133 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'91, mlcod=15'86)", 2024-09-18T04:37:46.471 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.236920+0000", 2024-09-18T04:37:46.471 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 18.155790579000001, 2024-09-18T04:37:46.471 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024318410000000001, 2024-09-18T04:37:46.471 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:46.471 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:46.472 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:46.472 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.472 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:46.472 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236920+0000", 2024-09-18T04:37:46.472 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:46.472 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.472 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.472 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:46.472 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236920+0000", 2024-09-18T04:37:46.472 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:46.472 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.472 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.473 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:46.473 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236918+0000", 2024-09-18T04:37:46.473 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:37:46.473 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.473 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.473 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:46.473 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236931+0000", 2024-09-18T04:37:46.473 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2683999999999999e-05 2024-09-18T04:37:46.473 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.473 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.473 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:46.473 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236951+0000", 2024-09-18T04:37:46.473 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0292000000000002e-05 2024-09-18T04:37:46.474 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.474 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.474 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:46.474 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236962+0000", 2024-09-18T04:37:46.474 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0703000000000001e-05 2024-09-18T04:37:46.474 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.474 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.474 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:46.474 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237095+0000", 2024-09-18T04:37:46.474 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013312400000000001 2024-09-18T04:37:46.474 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.474 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.475 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:46.475 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237133+0000", 2024-09-18T04:37:46.475 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7496000000000001e-05 2024-09-18T04:37:46.475 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.475 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.475 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:46.475 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.239206+0000", 2024-09-18T04:37:46.475 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020730290000000001 2024-09-18T04:37:46.475 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.475 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.475 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:46.475 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.239352+0000", 2024-09-18T04:37:46.475 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014593000000000001 2024-09-18T04:37:46.476 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:46.476 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:46.476 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:46.476 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.476 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.476 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:134 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'92, mlcod=15'86)", 2024-09-18T04:37:46.476 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.237894+0000", 2024-09-18T04:37:46.476 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 18.154816557, 2024-09-18T04:37:46.476 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025299519999999998, 2024-09-18T04:37:46.476 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:46.476 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:46.476 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:46.476 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.477 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:46.477 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237894+0000", 2024-09-18T04:37:46.477 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:46.477 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.477 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.477 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:46.477 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237894+0000", 2024-09-18T04:37:46.477 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:46.477 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.477 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.477 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:46.477 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237892+0000", 2024-09-18T04:37:46.477 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:37:46.478 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.478 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.478 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:46.478 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237986+0000", 2024-09-18T04:37:46.478 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3341e-05 2024-09-18T04:37:46.478 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.478 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.478 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:46.478 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.238012+0000", 2024-09-18T04:37:46.478 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5843e-05 2024-09-18T04:37:46.478 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.478 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.478 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:46.479 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.238027+0000", 2024-09-18T04:37:46.479 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5469e-05 2024-09-18T04:37:46.479 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.479 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.479 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:46.479 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.238114+0000", 2024-09-18T04:37:46.479 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7483000000000002e-05 2024-09-18T04:37:46.479 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.479 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.479 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:46.479 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.238152+0000", 2024-09-18T04:37:46.479 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7565999999999999e-05 2024-09-18T04:37:46.479 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.480 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.480 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:46.480 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.240297+0000", 2024-09-18T04:37:46.480 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021447620000000001 2024-09-18T04:37:46.480 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.480 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.480 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:46.480 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.240424+0000", 2024-09-18T04:37:46.480 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012705400000000001 2024-09-18T04:37:46.481 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:46.481 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:46.481 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:46.481 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.481 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.481 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:139 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'97, mlcod=15'93)", 2024-09-18T04:37:46.481 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.245692+0000", 2024-09-18T04:37:46.481 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 18.147018416000002, 2024-09-18T04:37:46.481 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024260480000000001, 2024-09-18T04:37:46.481 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:46.481 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:46.481 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:46.481 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.482 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:46.482 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245692+0000", 2024-09-18T04:37:46.482 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:46.482 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.482 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.482 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:46.482 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245692+0000", 2024-09-18T04:37:46.482 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:46.482 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.482 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.482 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:46.482 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245690+0000", 2024-09-18T04:37:46.483 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:37:46.483 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.483 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.483 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:46.483 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245712+0000", 2024-09-18T04:37:46.483 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1916000000000002e-05 2024-09-18T04:37:46.483 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.483 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.483 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:46.483 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245741+0000", 2024-09-18T04:37:46.483 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8503000000000001e-05 2024-09-18T04:37:46.483 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.483 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.484 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:46.484 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245759+0000", 2024-09-18T04:37:46.484 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.872e-05 2024-09-18T04:37:46.484 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.484 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.484 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:46.484 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245859+0000", 2024-09-18T04:37:46.484 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9641000000000004e-05 2024-09-18T04:37:46.484 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.484 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.484 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:46.485 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245900+0000", 2024-09-18T04:37:46.485 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0589000000000001e-05 2024-09-18T04:37:46.485 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.485 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.485 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:46.485 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.247961+0000", 2024-09-18T04:37:46.485 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020619190000000002 2024-09-18T04:37:46.485 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.485 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.485 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:46.485 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.248118+0000", 2024-09-18T04:37:46.485 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015664300000000001 2024-09-18T04:37:46.486 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:46.486 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:46.486 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:46.486 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.486 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.486 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:144 1.0 e15/14 1:30306672:devicehealth::main.db.0000000000000000:head v 15'102, mlcod=15'98)", 2024-09-18T04:37:46.486 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.252149+0000", 2024-09-18T04:37:46.486 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 18.140561349999999, 2024-09-18T04:37:46.486 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024174959999999999, 2024-09-18T04:37:46.486 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:46.486 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:46.487 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:46.487 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.487 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:46.487 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252149+0000", 2024-09-18T04:37:46.487 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:46.487 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.487 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.487 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:46.487 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252149+0000", 2024-09-18T04:37:46.487 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:46.487 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.487 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.488 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:46.488 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252148+0000", 2024-09-18T04:37:46.488 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:37:46.488 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.488 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.488 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:46.488 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252225+0000", 2024-09-18T04:37:46.488 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6576999999999995e-05 2024-09-18T04:37:46.488 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.488 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.488 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:46.488 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252242+0000", 2024-09-18T04:37:46.488 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7740999999999998e-05 2024-09-18T04:37:46.489 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.489 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.489 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:46.489 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252254+0000", 2024-09-18T04:37:46.489 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.137e-05 2024-09-18T04:37:46.489 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.489 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.489 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:46.489 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252385+0000", 2024-09-18T04:37:46.489 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013091099999999999 2024-09-18T04:37:46.489 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.489 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.490 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:46.490 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252426+0000", 2024-09-18T04:37:46.490 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1535999999999997e-05 2024-09-18T04:37:46.490 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.490 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.490 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:46.490 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.254498+0000", 2024-09-18T04:37:46.490 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002071477 2024-09-18T04:37:46.490 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.490 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.490 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:46.490 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.254567+0000", 2024-09-18T04:37:46.490 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9014e-05 2024-09-18T04:37:46.491 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:46.491 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:46.491 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:46.491 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:46.491 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:46.491 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:37:46.492 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:37:46.493 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:37:46.487+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:37:46.541 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-18T04:37:46.593 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:37:46.588+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:37:46.625 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:37:46.625 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:37:46.625 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:37:46.625 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:37:46.635 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:37:46.693 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:37:46.687+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:37:46.738 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:37:46.738 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:37:46.738 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:37:46.738 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:37:46.738 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:37:46.748 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:37:46.795 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:37:46.792+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:37:46.849 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:37:46.850 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:37:46.850 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:37:46.850 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:37:46.850 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.850 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4214.0:5 2.2 e19/17 2:5cea7035:::rbd_info:head v 19'1, mlcod=0'0)", 2024-09-18T04:37:46.850 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:35.471260+0000", 2024-09-18T04:37:46.850 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 11.377290117999999, 2024-09-18T04:37:46.850 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021025950000000001, 2024-09-18T04:37:46.850 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:46.850 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:46.850 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:46.850 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.850 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:46.850 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471260+0000", 2024-09-18T04:37:46.851 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:46.851 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.851 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.851 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:46.851 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471260+0000", 2024-09-18T04:37:46.851 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:46.851 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.851 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.851 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:46.851 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471259+0000", 2024-09-18T04:37:46.851 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:37:46.851 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.851 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.851 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:46.851 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471275+0000", 2024-09-18T04:37:46.852 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6135999999999999e-05 2024-09-18T04:37:46.852 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.852 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.852 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:46.852 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471298+0000", 2024-09-18T04:37:46.852 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3632999999999999e-05 2024-09-18T04:37:46.852 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.852 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.852 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:46.852 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471320+0000", 2024-09-18T04:37:46.852 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1852999999999999e-05 2024-09-18T04:37:46.852 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.852 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.852 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:46.853 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471445+0000", 2024-09-18T04:37:46.853 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012461699999999999 2024-09-18T04:37:46.853 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.853 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.853 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:46.853 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471529+0000", 2024-09-18T04:37:46.853 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3875000000000006e-05 2024-09-18T04:37:46.853 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.853 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.853 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:46.853 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.473317+0000", 2024-09-18T04:37:46.853 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017886899999999999 2024-09-18T04:37:46.853 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.854 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.854 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:46.854 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.473362+0000", 2024-09-18T04:37:46.854 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4724000000000001e-05 2024-09-18T04:37:46.854 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:46.854 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:46.854 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:46.854 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.854 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.854 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4214.0:7 2.2 e20/17 2:5cea7035:::rbd_info:head v 20'2, mlcod=0'0)", 2024-09-18T04:37:46.854 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:36.490560+0000", 2024-09-18T04:37:46.854 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 10.357989978999999, 2024-09-18T04:37:46.854 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0015326020000000001, 2024-09-18T04:37:46.855 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:46.855 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:46.855 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:46.855 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.855 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:46.855 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490560+0000", 2024-09-18T04:37:46.855 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:46.855 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.855 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.855 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:46.855 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490560+0000", 2024-09-18T04:37:46.855 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:46.855 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.855 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.856 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:46.856 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490559+0000", 2024-09-18T04:37:46.856 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:37:46.856 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.856 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.856 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:46.856 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490568+0000", 2024-09-18T04:37:46.856 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5699999999999993e-06 2024-09-18T04:37:46.856 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.856 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.856 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:46.856 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490599+0000", 2024-09-18T04:37:46.856 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1566000000000002e-05 2024-09-18T04:37:46.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.857 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.857 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:46.857 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490609+0000", 2024-09-18T04:37:46.857 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.8700000000000004e-06 2024-09-18T04:37:46.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.857 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.857 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:46.857 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490708+0000", 2024-09-18T04:37:46.857 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9139000000000001e-05 2024-09-18T04:37:46.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.857 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.857 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:46.857 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490736+0000", 2024-09-18T04:37:46.858 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7617e-05 2024-09-18T04:37:46.858 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.858 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.858 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:46.858 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.492016+0000", 2024-09-18T04:37:46.858 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0012796579999999999 2024-09-18T04:37:46.858 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:46.858 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:46.858 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:46.858 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.492092+0000", 2024-09-18T04:37:46.858 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6571999999999998e-05 2024-09-18T04:37:46.858 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:46.858 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:46.859 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:46.859 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:46.859 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:46.859 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:37:46.860 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:37:46.896 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:37:46.891+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:37:46.996 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:37:46.991+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:37:47.011 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:37:47.011 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:37:47.011 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:37:47.011 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:37:47.021 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:37:47.097 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:37:47.091+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:37:47.126 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:37:47.126 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:37:47.126 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:37:47.126 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:37:47.126 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:37:47.137 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:37:47.198 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:37:47.195+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:37:47.244 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:37:47.244 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:37:47.244 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:37:47.244 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:37:47.244 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:47.244 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4214.0:1 2.3 e18/17 2:c4c92e5a:::rbd_trash:head v 18'1, mlcod=0'0)", 2024-09-18T04:37:47.244 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:34.475123+0000", 2024-09-18T04:37:47.244 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 12.767960778999999, 2024-09-18T04:37:47.244 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00081394599999999996, 2024-09-18T04:37:47.244 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:47.245 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:47.245 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:47.245 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:47.245 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:47.245 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475123+0000", 2024-09-18T04:37:47.245 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:47.245 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:47.245 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:47.245 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:47.245 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475123+0000", 2024-09-18T04:37:47.245 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:47.245 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:47.245 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:47.246 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:47.246 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475121+0000", 2024-09-18T04:37:47.246 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:37:47.246 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:47.246 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:47.246 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:47.246 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475149+0000", 2024-09-18T04:37:47.246 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7155999999999999e-05 2024-09-18T04:37:47.246 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:47.246 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:47.246 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:47.246 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475165+0000", 2024-09-18T04:37:47.246 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6160000000000001e-05 2024-09-18T04:37:47.246 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:47.247 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:47.247 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:47.247 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475183+0000", 2024-09-18T04:37:47.247 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7918e-05 2024-09-18T04:37:47.247 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:47.247 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:47.247 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:47.247 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475246+0000", 2024-09-18T04:37:47.247 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3176000000000006e-05 2024-09-18T04:37:47.247 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:47.247 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:47.247 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:47.247 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475279+0000", 2024-09-18T04:37:47.247 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3192999999999999e-05 2024-09-18T04:37:47.247 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:47.248 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:47.248 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:47.248 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475871+0000", 2024-09-18T04:37:47.248 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00059206099999999995 2024-09-18T04:37:47.248 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:47.248 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:47.248 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:47.248 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475937+0000", 2024-09-18T04:37:47.248 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5439999999999997e-05 2024-09-18T04:37:47.248 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:47.248 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:47.248 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:47.248 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:47.248 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:47.249 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:37:47.254 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:37:47.299 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:37:47.296+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:37:47.372 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:37:47.372 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:37:47.372 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:37:47.372 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:37:47.382 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:37:47.400 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:37:47.395+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:37:47.489 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:37:47.490 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:37:47.490 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:37:47.490 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:37:47.490 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:37:47.499 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:37:47.500 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:37:47.496+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:37:47.544 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-09-18T04:37:47.601 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:37:47.600+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:37:47.610 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:37:47.610 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:37:47.610 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:37:47.611 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:37:47.611 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.611 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:47.611 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.573328+0000", 2024-09-18T04:37:47.611 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 20.030857266000002, 2024-09-18T04:37:47.611 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011397898, 2024-09-18T04:37:47.611 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:47.611 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:47.611 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:47.611 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:47.611 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:47.611 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 86 2024-09-18T04:37:47.611 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.612 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:47.612 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.612 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:47.612 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573328+0000", 2024-09-18T04:37:47.612 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:47.612 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.612 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.612 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:47.612 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573328+0000", 2024-09-18T04:37:47.612 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:47.612 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.612 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.612 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:47.613 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573327+0000", 2024-09-18T04:37:47.613 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:37:47.613 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.613 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.613 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:47.613 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573331+0000", 2024-09-18T04:37:47.613 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9750000000000001e-06 2024-09-18T04:37:47.613 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.613 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.613 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:47.613 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573346+0000", 2024-09-18T04:37:47.613 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5469999999999999e-05 2024-09-18T04:37:47.613 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.614 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.614 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:47.614 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573370+0000", 2024-09-18T04:37:47.614 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3969999999999999e-05 2024-09-18T04:37:47.614 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.614 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.614 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:47.614 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.578536+0000", 2024-09-18T04:37:47.614 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0051659679999999999 2024-09-18T04:37:47.614 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.614 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.614 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:47.614 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.578689+0000", 2024-09-18T04:37:47.615 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015276299999999999 2024-09-18T04:37:47.615 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.615 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.615 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:47.615 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.578901+0000", 2024-09-18T04:37:47.615 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000211783 2024-09-18T04:37:47.615 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.615 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.615 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:47.615 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.583408+0000", 2024-09-18T04:37:47.615 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045071249999999998 2024-09-18T04:37:47.615 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.615 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.615 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:47.616 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.584645+0000", 2024-09-18T04:37:47.616 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001237007 2024-09-18T04:37:47.616 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.616 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.616 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:47.616 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.584703+0000", 2024-09-18T04:37:47.616 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.817e-05 2024-09-18T04:37:47.616 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.616 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.616 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:47.616 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.584725+0000", 2024-09-18T04:37:47.617 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2155e-05 2024-09-18T04:37:47.617 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:47.617 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:47.617 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:47.617 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.617 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.617 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:47.617 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.573382+0000", 2024-09-18T04:37:47.617 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 20.030802518000002, 2024-09-18T04:37:47.617 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012154974000000001, 2024-09-18T04:37:47.617 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:47.617 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:47.617 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:47.618 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:47.618 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:47.618 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 87 2024-09-18T04:37:47.618 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.618 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:47.618 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.618 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:47.618 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573382+0000", 2024-09-18T04:37:47.618 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:47.618 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.618 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.618 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:47.618 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573382+0000", 2024-09-18T04:37:47.619 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:47.619 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.619 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.619 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:47.619 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573382+0000", 2024-09-18T04:37:47.619 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:37:47.619 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.619 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.619 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:47.619 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573482+0000", 2024-09-18T04:37:47.619 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000100333 2024-09-18T04:37:47.619 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.619 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.620 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:47.620 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573492+0000", 2024-09-18T04:37:47.620 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5389999999999996e-06 2024-09-18T04:37:47.620 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.620 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.620 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:47.620 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573500+0000", 2024-09-18T04:37:47.620 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8929999999999994e-06 2024-09-18T04:37:47.620 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.620 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.620 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:47.620 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.579797+0000", 2024-09-18T04:37:47.620 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0062963239999999998 2024-09-18T04:37:47.621 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.621 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.621 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:47.621 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.579957+0000", 2024-09-18T04:37:47.621 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000160283 2024-09-18T04:37:47.621 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.621 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.621 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:47.621 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.580178+0000", 2024-09-18T04:37:47.621 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00022079200000000001 2024-09-18T04:37:47.621 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.621 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.621 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:47.621 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.583479+0000", 2024-09-18T04:37:47.622 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033007689999999998 2024-09-18T04:37:47.622 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.622 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.622 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:47.622 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585477+0000", 2024-09-18T04:37:47.622 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0019985670000000001 2024-09-18T04:37:47.622 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.622 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.622 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:47.622 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585517+0000", 2024-09-18T04:37:47.622 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9709999999999998e-05 2024-09-18T04:37:47.622 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.622 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.622 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:47.623 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585537+0000", 2024-09-18T04:37:47.623 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0326000000000002e-05 2024-09-18T04:37:47.623 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:47.623 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:47.623 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:47.623 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.623 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.623 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:47.623 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.573508+0000", 2024-09-18T04:37:47.623 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 20.030676811999999, 2024-09-18T04:37:47.623 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012068476, 2024-09-18T04:37:47.623 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:47.623 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:47.623 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:47.624 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:47.624 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:47.624 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 88 2024-09-18T04:37:47.624 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.624 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:47.624 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.624 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:47.624 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573508+0000", 2024-09-18T04:37:47.624 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:47.624 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.624 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.624 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:47.624 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573508+0000", 2024-09-18T04:37:47.625 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:47.625 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.625 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.625 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:47.625 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573507+0000", 2024-09-18T04:37:47.625 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:37:47.625 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.625 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.625 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:47.625 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573512+0000", 2024-09-18T04:37:47.625 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.2629999999999997e-06 2024-09-18T04:37:47.625 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.625 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.625 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:47.626 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573530+0000", 2024-09-18T04:37:47.626 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8507000000000001e-05 2024-09-18T04:37:47.626 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.626 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.626 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:47.626 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573537+0000", 2024-09-18T04:37:47.626 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.832e-06 2024-09-18T04:37:47.626 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.626 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.626 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:47.626 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.582486+0000", 2024-09-18T04:37:47.626 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0089494770000000008 2024-09-18T04:37:47.626 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.626 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.627 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:47.627 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.582630+0000", 2024-09-18T04:37:47.627 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000144005 2024-09-18T04:37:47.627 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.627 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.627 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:47.627 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.582842+0000", 2024-09-18T04:37:47.627 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00021189200000000001 2024-09-18T04:37:47.627 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.627 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.627 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:47.627 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.584777+0000", 2024-09-18T04:37:47.627 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001934753 2024-09-18T04:37:47.628 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.628 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.628 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:47.628 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585543+0000", 2024-09-18T04:37:47.628 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00076540099999999997 2024-09-18T04:37:47.628 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.628 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.628 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:47.628 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585561+0000", 2024-09-18T04:37:47.628 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8474999999999998e-05 2024-09-18T04:37:47.628 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.628 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.628 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:47.629 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585576+0000", 2024-09-18T04:37:47.629 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5427e-05 2024-09-18T04:37:47.629 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:47.629 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:47.629 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:47.629 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.629 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.629 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:47.629 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786394+0000", 2024-09-18T04:37:47.629 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 19.817790365, 2024-09-18T04:37:47.629 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011902085999999999, 2024-09-18T04:37:47.629 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:47.629 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:47.630 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:47.630 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:47.630 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:47.630 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 109 2024-09-18T04:37:47.630 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.630 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:47.630 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.630 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:47.630 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786394+0000", 2024-09-18T04:37:47.630 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:47.630 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.630 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.630 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:47.631 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786394+0000", 2024-09-18T04:37:47.631 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:47.631 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.631 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.631 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:47.631 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786394+0000", 2024-09-18T04:37:47.631 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:37:47.631 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.631 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.631 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:47.631 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786548+0000", 2024-09-18T04:37:47.631 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015463400000000001 2024-09-18T04:37:47.631 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.631 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.632 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:47.632 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786561+0000", 2024-09-18T04:37:47.632 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3093e-05 2024-09-18T04:37:47.632 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.632 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.632 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:47.632 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786571+0000", 2024-09-18T04:37:47.632 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4700000000000008e-06 2024-09-18T04:37:47.632 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.632 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.632 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:47.632 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.793367+0000", 2024-09-18T04:37:47.632 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0067966600000000004 2024-09-18T04:37:47.633 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.633 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.633 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:47.633 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.793526+0000", 2024-09-18T04:37:47.633 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015877 2024-09-18T04:37:47.633 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.633 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.633 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:47.633 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.793752+0000", 2024-09-18T04:37:47.633 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00022555200000000001 2024-09-18T04:37:47.633 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.633 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.633 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:47.634 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798125+0000", 2024-09-18T04:37:47.634 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0043733349999999999 2024-09-18T04:37:47.634 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.634 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.634 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:47.634 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798146+0000", 2024-09-18T04:37:47.634 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1063000000000001e-05 2024-09-18T04:37:47.634 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.634 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.634 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:47.634 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798232+0000", 2024-09-18T04:37:47.634 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5771000000000003e-05 2024-09-18T04:37:47.634 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.634 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.635 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:47.635 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798297+0000", 2024-09-18T04:37:47.635 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.4624999999999999e-05 2024-09-18T04:37:47.635 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:47.635 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:47.635 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:47.635 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.635 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.635 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:47.635 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786584+0000", 2024-09-18T04:37:47.635 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 19.817600286000001, 2024-09-18T04:37:47.635 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012013037000000001, 2024-09-18T04:37:47.635 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:47.636 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:47.636 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:47.636 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:47.636 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:47.636 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 110 2024-09-18T04:37:47.636 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.636 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:47.636 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.636 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:47.636 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786584+0000", 2024-09-18T04:37:47.636 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:47.636 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.636 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.637 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:47.637 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786584+0000", 2024-09-18T04:37:47.637 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:47.637 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.637 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.637 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:47.637 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786584+0000", 2024-09-18T04:37:47.637 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:37:47.637 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.637 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.637 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:47.637 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786590+0000", 2024-09-18T04:37:47.637 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6359999999999997e-06 2024-09-18T04:37:47.637 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.638 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.638 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:47.638 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786599+0000", 2024-09-18T04:37:47.638 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7369999999999995e-06 2024-09-18T04:37:47.638 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.638 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.638 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:47.638 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786606+0000", 2024-09-18T04:37:47.638 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1930000000000004e-06 2024-09-18T04:37:47.638 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.638 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.638 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:47.638 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.795018+0000", 2024-09-18T04:37:47.639 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0084115660000000005 2024-09-18T04:37:47.639 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.639 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.639 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:47.639 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.795184+0000", 2024-09-18T04:37:47.639 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000165833 2024-09-18T04:37:47.639 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.639 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.639 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:47.639 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.795414+0000", 2024-09-18T04:37:47.639 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00023088399999999999 2024-09-18T04:37:47.639 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.639 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.640 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:47.640 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798321+0000", 2024-09-18T04:37:47.640 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0029069539999999998 2024-09-18T04:37:47.640 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.640 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.640 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:47.640 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798404+0000", 2024-09-18T04:37:47.640 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2724000000000003e-05 2024-09-18T04:37:47.640 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.640 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.640 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:47.640 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798488+0000", 2024-09-18T04:37:47.640 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.3505999999999996e-05 2024-09-18T04:37:47.640 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.641 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.641 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:47.641 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798598+0000", 2024-09-18T04:37:47.641 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000109917 2024-09-18T04:37:47.641 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:47.641 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:47.641 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:47.641 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.641 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.641 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:47.641 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786613+0000", 2024-09-18T04:37:47.641 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 19.817571586, 2024-09-18T04:37:47.641 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014245760999999999, 2024-09-18T04:37:47.642 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:47.642 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:47.642 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:47.642 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:47.642 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:47.642 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 111 2024-09-18T04:37:47.642 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.642 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:47.642 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.642 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:47.642 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786613+0000", 2024-09-18T04:37:47.642 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:47.642 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.642 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.643 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:47.643 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786613+0000", 2024-09-18T04:37:47.643 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:47.643 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.643 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.643 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:47.643 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786612+0000", 2024-09-18T04:37:47.643 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:37:47.643 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.643 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.643 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:47.643 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786622+0000", 2024-09-18T04:37:47.643 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4099999999999997e-06 2024-09-18T04:37:47.643 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.644 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.644 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:47.644 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786630+0000", 2024-09-18T04:37:47.644 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4169999999999992e-06 2024-09-18T04:37:47.644 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.644 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.644 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:47.644 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786637+0000", 2024-09-18T04:37:47.644 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8290000000000002e-06 2024-09-18T04:37:47.644 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.644 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.644 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:47.644 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.797099+0000", 2024-09-18T04:37:47.644 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010461747 2024-09-18T04:37:47.645 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.645 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.645 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:47.645 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.797243+0000", 2024-09-18T04:37:47.645 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014388899999999999 2024-09-18T04:37:47.645 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.645 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.645 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:47.645 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.797463+0000", 2024-09-18T04:37:47.645 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000219912 2024-09-18T04:37:47.645 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.645 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.645 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:47.646 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800732+0000", 2024-09-18T04:37:47.646 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0032688719999999999 2024-09-18T04:37:47.646 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.646 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.646 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:47.646 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800751+0000", 2024-09-18T04:37:47.646 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9633999999999999e-05 2024-09-18T04:37:47.646 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.646 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.646 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:47.646 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800824+0000", 2024-09-18T04:37:47.646 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.2805999999999994e-05 2024-09-18T04:37:47.646 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.646 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.647 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:47.647 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800859+0000", 2024-09-18T04:37:47.647 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4952e-05 2024-09-18T04:37:47.647 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:47.647 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:47.647 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:47.647 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.647 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.647 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:47.647 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786649+0000", 2024-09-18T04:37:47.647 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 19.817536177000001, 2024-09-18T04:37:47.647 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.01613442, 2024-09-18T04:37:47.647 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:47.647 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:47.648 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:47.648 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:47.648 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:47.648 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 112 2024-09-18T04:37:47.648 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.648 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:47.648 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.648 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:47.648 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786649+0000", 2024-09-18T04:37:47.648 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:47.648 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.648 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.648 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:47.649 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786649+0000", 2024-09-18T04:37:47.649 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:47.649 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.649 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.649 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:47.649 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786648+0000", 2024-09-18T04:37:47.649 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:37:47.649 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.649 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.649 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:47.649 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786768+0000", 2024-09-18T04:37:47.649 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000120491 2024-09-18T04:37:47.649 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.650 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.650 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:47.650 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786795+0000", 2024-09-18T04:37:47.650 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6556000000000001e-05 2024-09-18T04:37:47.650 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.650 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.650 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:47.650 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786807+0000", 2024-09-18T04:37:47.650 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1739999999999999e-05 2024-09-18T04:37:47.650 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.650 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.650 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:47.650 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798683+0000", 2024-09-18T04:37:47.650 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011876070000000001 2024-09-18T04:37:47.651 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.651 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.651 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:47.651 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798867+0000", 2024-09-18T04:37:47.651 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000184103 2024-09-18T04:37:47.651 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.651 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.651 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:47.651 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.799082+0000", 2024-09-18T04:37:47.651 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00021476000000000001 2024-09-18T04:37:47.651 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.651 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.651 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:47.651 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802610+0000", 2024-09-18T04:37:47.652 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0035282209999999998 2024-09-18T04:37:47.652 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.652 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.652 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:47.652 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802706+0000", 2024-09-18T04:37:47.652 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5922999999999996e-05 2024-09-18T04:37:47.652 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.652 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.652 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:47.652 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802750+0000", 2024-09-18T04:37:47.652 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4511000000000001e-05 2024-09-18T04:37:47.652 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.652 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.653 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:47.653 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802783+0000", 2024-09-18T04:37:47.653 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2752000000000001e-05 2024-09-18T04:37:47.653 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:47.653 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:47.653 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:47.653 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.653 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.653 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:47.653 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786816+0000", 2024-09-18T04:37:47.653 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 19.817369110000001, 2024-09-18T04:37:47.653 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016197989999999999, 2024-09-18T04:37:47.653 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:47.654 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:47.654 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:47.654 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:47.654 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:47.654 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 113 2024-09-18T04:37:47.654 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.654 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:47.654 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.654 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:47.654 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786816+0000", 2024-09-18T04:37:47.654 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:47.654 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.654 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.655 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:47.655 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786816+0000", 2024-09-18T04:37:47.655 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:47.655 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.655 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.655 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:47.655 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786815+0000", 2024-09-18T04:37:47.655 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:37:47.655 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.655 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.655 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:47.655 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786821+0000", 2024-09-18T04:37:47.655 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9529999999999996e-06 2024-09-18T04:37:47.655 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.656 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.656 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:47.656 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786841+0000", 2024-09-18T04:37:47.656 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9743000000000001e-05 2024-09-18T04:37:47.656 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.656 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.656 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:47.656 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786848+0000", 2024-09-18T04:37:47.656 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4900000000000003e-06 2024-09-18T04:37:47.656 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.656 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.656 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:47.656 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.799747+0000", 2024-09-18T04:37:47.656 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012898902 2024-09-18T04:37:47.657 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.657 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.657 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:47.657 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.799898+0000", 2024-09-18T04:37:47.657 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000150877 2024-09-18T04:37:47.657 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.657 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.657 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:47.657 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800095+0000", 2024-09-18T04:37:47.657 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000197023 2024-09-18T04:37:47.657 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.657 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.657 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:47.658 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802829+0000", 2024-09-18T04:37:47.658 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0027343480000000002 2024-09-18T04:37:47.658 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.658 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.658 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:47.658 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802847+0000", 2024-09-18T04:37:47.658 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7283e-05 2024-09-18T04:37:47.658 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.658 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.658 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:47.658 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802974+0000", 2024-09-18T04:37:47.658 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012749199999999999 2024-09-18T04:37:47.658 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.659 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.659 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:47.659 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803014+0000", 2024-09-18T04:37:47.659 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9576000000000002e-05 2024-09-18T04:37:47.659 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:47.659 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:47.659 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:47.659 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.659 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.659 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:47.659 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:37:47.659 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 19.817283519, 2024-09-18T04:37:47.659 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017399218000000001, 2024-09-18T04:37:47.660 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:47.660 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:47.660 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:47.660 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:47.660 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:47.660 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 114 2024-09-18T04:37:47.660 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.660 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:47.660 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.660 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:47.660 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:37:47.660 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:47.660 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.660 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.661 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:47.661 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:37:47.661 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:47.661 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.661 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.661 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:47.661 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786900+0000", 2024-09-18T04:37:47.661 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:37:47.661 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.661 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.661 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:47.661 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786908+0000", 2024-09-18T04:37:47.661 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8199999999999997e-06 2024-09-18T04:37:47.662 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.662 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.662 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:47.662 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786925+0000", 2024-09-18T04:37:47.662 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6849000000000001e-05 2024-09-18T04:37:47.662 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.662 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.662 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:47.662 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786934+0000", 2024-09-18T04:37:47.662 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6299999999999993e-06 2024-09-18T04:37:47.662 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.662 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.662 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:47.662 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800926+0000", 2024-09-18T04:37:47.663 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013991632 2024-09-18T04:37:47.663 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.663 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.663 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:47.663 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801060+0000", 2024-09-18T04:37:47.663 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013378199999999999 2024-09-18T04:37:47.663 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.663 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.663 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:47.663 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801166+0000", 2024-09-18T04:37:47.663 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010607099999999999 2024-09-18T04:37:47.663 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.663 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.663 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:47.664 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803026+0000", 2024-09-18T04:37:47.664 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001860469 2024-09-18T04:37:47.664 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.664 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.664 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:47.664 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804245+0000", 2024-09-18T04:37:47.664 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0012185900000000001 2024-09-18T04:37:47.664 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.664 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.664 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:47.664 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804278+0000", 2024-09-18T04:37:47.664 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3272000000000003e-05 2024-09-18T04:37:47.664 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.665 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.665 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:47.665 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804300+0000", 2024-09-18T04:37:47.665 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2266e-05 2024-09-18T04:37:47.665 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:47.665 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:47.665 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:47.665 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.665 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.665 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:47.665 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787566+0000", 2024-09-18T04:37:47.665 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 19.816618559999998, 2024-09-18T04:37:47.665 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017124120999999999, 2024-09-18T04:37:47.665 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:47.666 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:47.666 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:47.666 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:47.666 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:47.666 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 115 2024-09-18T04:37:47.666 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.666 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:47.666 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.666 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:47.666 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787566+0000", 2024-09-18T04:37:47.666 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:47.666 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.666 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.666 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:47.667 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787566+0000", 2024-09-18T04:37:47.667 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:47.667 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.667 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.667 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:47.667 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787564+0000", 2024-09-18T04:37:47.667 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:37:47.667 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.667 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.667 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:47.667 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787751+0000", 2024-09-18T04:37:47.667 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000186873 2024-09-18T04:37:47.667 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.667 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.668 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:47.668 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787772+0000", 2024-09-18T04:37:47.668 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.126e-05 2024-09-18T04:37:47.668 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.668 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.668 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:47.668 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787794+0000", 2024-09-18T04:37:47.668 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1418999999999999e-05 2024-09-18T04:37:47.668 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.668 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.668 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:47.668 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801659+0000", 2024-09-18T04:37:47.668 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013865125000000001 2024-09-18T04:37:47.669 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.669 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.669 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:47.669 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801750+0000", 2024-09-18T04:37:47.669 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.1427000000000004e-05 2024-09-18T04:37:47.669 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.669 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.669 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:47.669 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801894+0000", 2024-09-18T04:37:47.669 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000143514 2024-09-18T04:37:47.669 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.669 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.669 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:47.669 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804313+0000", 2024-09-18T04:37:47.670 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0024190879999999998 2024-09-18T04:37:47.670 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.670 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.670 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:47.670 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804644+0000", 2024-09-18T04:37:47.670 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00033100799999999999 2024-09-18T04:37:47.670 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.670 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.670 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:47.670 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804672+0000", 2024-09-18T04:37:47.670 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7988e-05 2024-09-18T04:37:47.670 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.670 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.670 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:47.671 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804690+0000", 2024-09-18T04:37:47.671 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8298999999999999e-05 2024-09-18T04:37:47.671 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:47.671 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:47.671 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:47.671 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.671 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.671 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:47.671 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:37:47.671 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 19.816370621000001, 2024-09-18T04:37:47.671 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017521662, 2024-09-18T04:37:47.671 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:47.671 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:47.671 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:47.672 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:47.672 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:47.672 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 116 2024-09-18T04:37:47.672 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.672 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:47.672 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.672 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:47.672 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:37:47.672 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:47.672 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.672 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.672 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:47.672 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:37:47.673 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:47.673 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.673 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.673 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:47.673 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787813+0000", 2024-09-18T04:37:47.673 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:37:47.673 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.673 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.673 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:47.673 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787824+0000", 2024-09-18T04:37:47.673 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1157e-05 2024-09-18T04:37:47.673 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.673 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.674 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:47.674 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787840+0000", 2024-09-18T04:37:47.674 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6269e-05 2024-09-18T04:37:47.674 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.674 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.674 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:47.674 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787853+0000", 2024-09-18T04:37:47.674 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3156000000000001e-05 2024-09-18T04:37:47.674 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.674 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.674 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:47.674 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803110+0000", 2024-09-18T04:37:47.674 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015256736 2024-09-18T04:37:47.674 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.675 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.675 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:47.675 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803197+0000", 2024-09-18T04:37:47.675 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7195999999999997e-05 2024-09-18T04:37:47.675 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.675 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.675 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:47.675 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803308+0000", 2024-09-18T04:37:47.675 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111005 2024-09-18T04:37:47.675 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.675 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.675 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:47.675 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804316+0000", 2024-09-18T04:37:47.676 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010073949999999999 2024-09-18T04:37:47.676 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.676 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.676 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:47.676 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805232+0000", 2024-09-18T04:37:47.676 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091636300000000001 2024-09-18T04:37:47.676 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.676 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.676 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:47.676 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805277+0000", 2024-09-18T04:37:47.676 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4864000000000003e-05 2024-09-18T04:37:47.676 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.676 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.676 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:47.677 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805336+0000", 2024-09-18T04:37:47.677 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8968000000000002e-05 2024-09-18T04:37:47.677 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:47.677 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:47.677 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:47.677 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.677 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.677 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:47.677 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:37:47.677 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 19.816318566, 2024-09-18T04:37:47.677 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017983453, 2024-09-18T04:37:47.677 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:47.677 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:47.677 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:47.678 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:47.678 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:47.678 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 117 2024-09-18T04:37:47.678 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.678 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:47.678 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.678 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:47.678 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:37:47.678 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:47.678 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.678 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.678 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:47.678 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:37:47.678 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:47.679 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.679 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.679 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:47.679 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787865+0000", 2024-09-18T04:37:47.679 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:37:47.679 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.679 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.679 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:47.679 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787889+0000", 2024-09-18T04:37:47.679 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4244999999999999e-05 2024-09-18T04:37:47.679 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.679 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.679 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:47.680 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787905+0000", 2024-09-18T04:37:47.680 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5739999999999998e-05 2024-09-18T04:37:47.680 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.680 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.680 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:47.680 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787918+0000", 2024-09-18T04:37:47.680 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.252e-05 2024-09-18T04:37:47.680 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.680 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.680 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:47.680 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803693+0000", 2024-09-18T04:37:47.680 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015774872999999998 2024-09-18T04:37:47.680 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.680 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.681 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:47.681 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803775+0000", 2024-09-18T04:37:47.681 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2761000000000003e-05 2024-09-18T04:37:47.681 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.681 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.681 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:47.681 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803887+0000", 2024-09-18T04:37:47.681 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111708 2024-09-18T04:37:47.681 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.681 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.681 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:47.681 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804823+0000", 2024-09-18T04:37:47.681 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00093625100000000003 2024-09-18T04:37:47.682 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.682 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.682 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:47.682 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805804+0000", 2024-09-18T04:37:47.682 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00098114699999999992 2024-09-18T04:37:47.682 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.682 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.682 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:47.682 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805831+0000", 2024-09-18T04:37:47.682 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6401999999999999e-05 2024-09-18T04:37:47.682 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.682 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.682 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:47.683 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805850+0000", 2024-09-18T04:37:47.683 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8862e-05 2024-09-18T04:37:47.683 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:47.683 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:47.683 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:47.683 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.683 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.683 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:47.683 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.230830+0000", 2024-09-18T04:37:47.683 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 19.373354686999999, 2024-09-18T04:37:47.683 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011537146999999999, 2024-09-18T04:37:47.683 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:47.683 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:47.683 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:47.684 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:47.684 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:47.684 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 135 2024-09-18T04:37:47.684 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.684 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:47.684 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.684 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:47.684 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230830+0000", 2024-09-18T04:37:47.684 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:47.684 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.684 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.684 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:47.684 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230830+0000", 2024-09-18T04:37:47.685 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:47.685 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.685 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.685 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:47.685 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230829+0000", 2024-09-18T04:37:47.685 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:37:47.685 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.685 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.685 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:47.685 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230835+0000", 2024-09-18T04:37:47.685 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8660000000000002e-06 2024-09-18T04:37:47.685 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.685 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.686 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:47.686 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230844+0000", 2024-09-18T04:37:47.686 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9339999999999995e-06 2024-09-18T04:37:47.686 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.686 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.686 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:47.686 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230853+0000", 2024-09-18T04:37:47.686 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4419999999999998e-06 2024-09-18T04:37:47.686 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.686 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.686 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:47.686 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.237658+0000", 2024-09-18T04:37:47.686 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0068055310000000001 2024-09-18T04:37:47.687 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.687 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.687 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:47.687 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.237837+0000", 2024-09-18T04:37:47.687 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017943200000000001 2024-09-18T04:37:47.687 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.687 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.687 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:47.687 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.238112+0000", 2024-09-18T04:37:47.687 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00027488600000000001 2024-09-18T04:37:47.687 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.687 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.687 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:47.687 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.240505+0000", 2024-09-18T04:37:47.688 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0023930539999999999 2024-09-18T04:37:47.688 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.688 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.688 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:47.688 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242157+0000", 2024-09-18T04:37:47.688 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001651135 2024-09-18T04:37:47.688 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.688 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.688 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:47.688 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242281+0000", 2024-09-18T04:37:47.688 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012451300000000001 2024-09-18T04:37:47.688 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.688 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.689 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:47.689 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242367+0000", 2024-09-18T04:37:47.689 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6178000000000001e-05 2024-09-18T04:37:47.689 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:47.689 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:47.689 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:47.689 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.689 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.689 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:47.689 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.230858+0000", 2024-09-18T04:37:47.689 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 19.373326799000001, 2024-09-18T04:37:47.689 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013005821000000001, 2024-09-18T04:37:47.689 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:47.689 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:47.690 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:47.690 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:47.690 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:47.690 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 136 2024-09-18T04:37:47.690 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.690 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:47.690 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.690 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:47.690 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230858+0000", 2024-09-18T04:37:47.690 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:47.690 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.690 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.690 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:47.691 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230858+0000", 2024-09-18T04:37:47.691 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:47.691 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.691 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.691 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:47.691 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230858+0000", 2024-09-18T04:37:47.691 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:37:47.691 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.691 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.691 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:47.691 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230861+0000", 2024-09-18T04:37:47.691 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7459999999999998e-06 2024-09-18T04:37:47.691 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.691 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.692 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:47.692 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230877+0000", 2024-09-18T04:37:47.692 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5979000000000001e-05 2024-09-18T04:37:47.692 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.692 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.692 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:47.692 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230885+0000", 2024-09-18T04:37:47.692 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.5979999999999999e-06 2024-09-18T04:37:47.692 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.692 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.692 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:47.692 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.239023+0000", 2024-09-18T04:37:47.692 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0081380710000000002 2024-09-18T04:37:47.692 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.693 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.693 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:47.693 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.239216+0000", 2024-09-18T04:37:47.693 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019274799999999999 2024-09-18T04:37:47.693 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.693 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.693 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:47.693 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.239417+0000", 2024-09-18T04:37:47.693 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020093300000000001 2024-09-18T04:37:47.693 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.693 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.693 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:47.693 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242494+0000", 2024-09-18T04:37:47.694 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003077177 2024-09-18T04:37:47.694 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.694 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.694 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:47.694 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.243668+0000", 2024-09-18T04:37:47.694 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001174027 2024-09-18T04:37:47.694 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.694 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.694 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:47.694 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.243777+0000", 2024-09-18T04:37:47.694 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010955600000000001 2024-09-18T04:37:47.694 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.694 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.694 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:47.695 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.243864+0000", 2024-09-18T04:37:47.695 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6441999999999997e-05 2024-09-18T04:37:47.695 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:47.695 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:47.695 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:47.695 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.695 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.695 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:47.695 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.230901+0000", 2024-09-18T04:37:47.695 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 19.373284225999999, 2024-09-18T04:37:47.695 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014120882, 2024-09-18T04:37:47.695 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:47.695 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:47.695 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:47.696 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:47.696 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:47.696 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 137 2024-09-18T04:37:47.696 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.696 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:47.696 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.696 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:47.696 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230901+0000", 2024-09-18T04:37:47.696 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:47.696 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.696 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.696 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:47.696 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230901+0000", 2024-09-18T04:37:47.696 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:47.697 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.697 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.697 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:47.697 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230900+0000", 2024-09-18T04:37:47.697 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:37:47.697 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.697 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.697 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:47.697 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231002+0000", 2024-09-18T04:37:47.697 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000102392 2024-09-18T04:37:47.697 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.697 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.697 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:47.697 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231029+0000", 2024-09-18T04:37:47.698 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7166999999999999e-05 2024-09-18T04:37:47.698 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.698 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.698 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:47.698 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231036+0000", 2024-09-18T04:37:47.698 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7259999999999996e-06 2024-09-18T04:37:47.698 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.698 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.698 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:47.698 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.240562+0000", 2024-09-18T04:37:47.698 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0095257970000000008 2024-09-18T04:37:47.698 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.698 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.699 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:47.699 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.240730+0000", 2024-09-18T04:37:47.699 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016767099999999999 2024-09-18T04:37:47.699 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.699 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.699 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:47.699 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.240927+0000", 2024-09-18T04:37:47.699 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019711899999999999 2024-09-18T04:37:47.699 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.699 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.699 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:47.699 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244885+0000", 2024-09-18T04:37:47.699 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0039577340000000001 2024-09-18T04:37:47.699 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.700 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.700 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:47.700 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244899+0000", 2024-09-18T04:37:47.700 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4403000000000001e-05 2024-09-18T04:37:47.700 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.700 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.700 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:47.700 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244973+0000", 2024-09-18T04:37:47.700 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4177999999999994e-05 2024-09-18T04:37:47.700 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.700 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.700 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:47.700 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.245021+0000", 2024-09-18T04:37:47.701 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8344999999999999e-05 2024-09-18T04:37:47.701 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:47.701 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:47.701 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:47.701 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.701 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.701 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:47.701 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:37:47.701 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 19.373141435000001, 2024-09-18T04:37:47.701 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016528505999999998, 2024-09-18T04:37:47.701 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:47.701 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:47.701 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:47.701 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:47.702 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:47.702 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 138 2024-09-18T04:37:47.702 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.702 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:47.702 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.702 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:47.702 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:37:47.702 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:47.702 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.702 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.702 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:47.702 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:37:47.702 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:47.703 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.703 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.703 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:47.703 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:37:47.703 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:37:47.703 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.703 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.703 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:47.703 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231047+0000", 2024-09-18T04:37:47.703 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5399999999999997e-06 2024-09-18T04:37:47.703 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.703 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.703 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:47.703 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231068+0000", 2024-09-18T04:37:47.704 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0769000000000001e-05 2024-09-18T04:37:47.704 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.704 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.704 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:47.704 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231074+0000", 2024-09-18T04:37:47.704 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6180000000000003e-06 2024-09-18T04:37:47.704 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.704 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.704 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:47.704 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242556+0000", 2024-09-18T04:37:47.704 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011482654 2024-09-18T04:37:47.704 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.704 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.704 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:47.705 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242666+0000", 2024-09-18T04:37:47.705 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011003200000000001 2024-09-18T04:37:47.705 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.705 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.705 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:47.705 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242924+0000", 2024-09-18T04:37:47.705 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00025785899999999999 2024-09-18T04:37:47.705 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.705 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.705 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:47.705 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247263+0000", 2024-09-18T04:37:47.705 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0043387649999999996 2024-09-18T04:37:47.705 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.706 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.706 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:47.706 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247412+0000", 2024-09-18T04:37:47.706 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000149276 2024-09-18T04:37:47.706 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.706 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.706 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:47.706 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247522+0000", 2024-09-18T04:37:47.706 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000109788 2024-09-18T04:37:47.706 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.706 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.706 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:47.706 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247572+0000", 2024-09-18T04:37:47.706 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9808999999999997e-05 2024-09-18T04:37:47.707 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:47.707 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:47.707 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:47.707 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.707 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.707 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:47.707 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231083+0000", 2024-09-18T04:37:47.707 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 19.373101972000001, 2024-09-18T04:37:47.707 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0166731, 2024-09-18T04:37:47.707 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:47.707 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:47.707 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:47.707 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:47.707 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:47.708 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 139 2024-09-18T04:37:47.708 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.708 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:47.708 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.708 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:47.708 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231083+0000", 2024-09-18T04:37:47.708 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:47.708 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.708 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.708 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:47.708 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231083+0000", 2024-09-18T04:37:47.708 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:47.709 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.709 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.709 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:47.709 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231082+0000", 2024-09-18T04:37:47.709 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:37:47.709 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.709 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.709 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:47.709 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231090+0000", 2024-09-18T04:37:47.709 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1280000000000008e-06 2024-09-18T04:37:47.709 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.709 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.709 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:47.709 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231097+0000", 2024-09-18T04:37:47.710 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5119999999999997e-06 2024-09-18T04:37:47.710 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.710 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.710 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:47.710 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231106+0000", 2024-09-18T04:37:47.710 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9230000000000008e-06 2024-09-18T04:37:47.710 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.710 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.710 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:47.710 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.243977+0000", 2024-09-18T04:37:47.710 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012870754 2024-09-18T04:37:47.710 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.710 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.711 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:47.711 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244112+0000", 2024-09-18T04:37:47.711 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000135575 2024-09-18T04:37:47.711 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.711 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.711 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:47.711 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244265+0000", 2024-09-18T04:37:47.711 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015279900000000001 2024-09-18T04:37:47.711 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.711 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.711 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:47.711 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247649+0000", 2024-09-18T04:37:47.711 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003383896 2024-09-18T04:37:47.711 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.712 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.712 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:47.712 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247656+0000", 2024-09-18T04:37:47.712 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8920000000000003e-06 2024-09-18T04:37:47.712 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.712 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.712 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:47.712 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247692+0000", 2024-09-18T04:37:47.712 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6470000000000001e-05 2024-09-18T04:37:47.712 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.712 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.712 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:47.712 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247756+0000", 2024-09-18T04:37:47.713 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3676999999999994e-05 2024-09-18T04:37:47.713 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:47.713 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:47.713 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:47.713 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.713 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.713 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:47.713 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231190+0000", 2024-09-18T04:37:47.713 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 19.372995046, 2024-09-18T04:37:47.713 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016793556000000001, 2024-09-18T04:37:47.713 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:47.713 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:47.713 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:47.714 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:47.714 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:47.714 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 140 2024-09-18T04:37:47.714 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.714 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:47.714 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.714 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:47.714 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231190+0000", 2024-09-18T04:37:47.714 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:47.714 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.714 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.714 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:47.714 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231190+0000", 2024-09-18T04:37:47.715 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:47.715 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.715 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.715 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:47.715 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231189+0000", 2024-09-18T04:37:47.715 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:37:47.715 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.715 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.715 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:47.715 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231273+0000", 2024-09-18T04:37:47.715 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4445999999999998e-05 2024-09-18T04:37:47.715 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.715 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.715 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:47.716 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231282+0000", 2024-09-18T04:37:47.716 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8859999999999993e-06 2024-09-18T04:37:47.716 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.716 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.716 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:47.716 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231288+0000", 2024-09-18T04:37:47.716 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9819999999999996e-06 2024-09-18T04:37:47.716 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.716 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.716 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:47.716 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.245119+0000", 2024-09-18T04:37:47.716 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.01383121 2024-09-18T04:37:47.716 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.717 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.717 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:47.717 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.245247+0000", 2024-09-18T04:37:47.717 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012763299999999999 2024-09-18T04:37:47.717 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.717 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.717 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:47.717 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.245401+0000", 2024-09-18T04:37:47.717 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015399200000000001 2024-09-18T04:37:47.717 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.717 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.717 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:47.718 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247766+0000", 2024-09-18T04:37:47.718 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0023649510000000001 2024-09-18T04:37:47.718 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.718 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.718 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:47.718 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247904+0000", 2024-09-18T04:37:47.718 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013836499999999999 2024-09-18T04:37:47.718 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.718 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.718 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:47.718 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247950+0000", 2024-09-18T04:37:47.718 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5201000000000003e-05 2024-09-18T04:37:47.718 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.719 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.719 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:47.719 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247983+0000", 2024-09-18T04:37:47.719 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3686000000000002e-05 2024-09-18T04:37:47.719 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:47.719 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:47.719 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:47.719 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.719 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.719 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:47.719 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:37:47.719 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 19.372884302999999, 2024-09-18T04:37:47.719 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017344024, 2024-09-18T04:37:47.719 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:47.720 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:47.720 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:47.720 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:47.720 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:47.720 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 141 2024-09-18T04:37:47.720 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.720 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:47.720 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.720 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:47.720 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:37:47.720 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:47.720 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.720 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.721 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:47.721 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:37:47.721 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:47.721 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.721 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.721 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:47.721 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:37:47.721 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:37:47.721 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.721 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.721 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:47.721 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231309+0000", 2024-09-18T04:37:47.721 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5419999999999994e-06 2024-09-18T04:37:47.721 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.722 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.722 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:47.722 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231323+0000", 2024-09-18T04:37:47.722 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3108000000000001e-05 2024-09-18T04:37:47.722 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.722 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.722 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:47.722 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231340+0000", 2024-09-18T04:37:47.722 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.696e-05 2024-09-18T04:37:47.722 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.722 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.722 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:47.722 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.246143+0000", 2024-09-18T04:37:47.723 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014803320999999999 2024-09-18T04:37:47.723 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.723 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.723 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:47.723 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.246323+0000", 2024-09-18T04:37:47.723 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00018061600000000001 2024-09-18T04:37:47.723 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.723 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.723 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:47.723 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.246530+0000", 2024-09-18T04:37:47.723 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020625 2024-09-18T04:37:47.723 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.723 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.723 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:47.724 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248002+0000", 2024-09-18T04:37:47.724 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0014719379999999999 2024-09-18T04:37:47.724 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.724 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.724 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:47.724 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248582+0000", 2024-09-18T04:37:47.724 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00058020599999999999 2024-09-18T04:37:47.724 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.724 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.724 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:47.724 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248618+0000", 2024-09-18T04:37:47.724 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6559e-05 2024-09-18T04:37:47.725 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.725 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.725 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:47.725 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248645+0000", 2024-09-18T04:37:47.725 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6080000000000001e-05 2024-09-18T04:37:47.725 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:47.725 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:47.725 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:47.725 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.725 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.725 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-18T04:37:47.725 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.878463+0000", 2024-09-18T04:37:47.725 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 18.725722268999998, 2024-09-18T04:37:47.725 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010882697, 2024-09-18T04:37:47.726 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:47.726 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:47.726 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:47.726 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:47.726 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:47.726 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 159 2024-09-18T04:37:47.726 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.726 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:47.726 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.726 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:47.726 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878463+0000", 2024-09-18T04:37:47.726 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:47.726 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.726 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.727 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:47.727 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878463+0000", 2024-09-18T04:37:47.727 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:47.727 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.727 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.727 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:47.727 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878462+0000", 2024-09-18T04:37:47.727 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:37:47.727 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.727 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.727 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:47.727 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878467+0000", 2024-09-18T04:37:47.727 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4099999999999999e-06 2024-09-18T04:37:47.727 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.727 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.728 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:47.728 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878490+0000", 2024-09-18T04:37:47.728 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3159999999999998e-05 2024-09-18T04:37:47.728 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.728 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.728 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:47.728 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878511+0000", 2024-09-18T04:37:47.728 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1092000000000001e-05 2024-09-18T04:37:47.728 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.728 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.728 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:47.728 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.887163+0000", 2024-09-18T04:37:47.728 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.008651845 2024-09-18T04:37:47.728 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.728 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.729 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:47.729 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.887273+0000", 2024-09-18T04:37:47.729 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000109823 2024-09-18T04:37:47.729 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.729 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.729 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:47.729 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.887441+0000", 2024-09-18T04:37:47.729 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016777599999999999 2024-09-18T04:37:47.729 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.729 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.729 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:47.729 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.888634+0000", 2024-09-18T04:37:47.729 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001192839 2024-09-18T04:37:47.730 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.730 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.730 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:47.730 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.889153+0000", 2024-09-18T04:37:47.730 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00051953700000000004 2024-09-18T04:37:47.730 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.730 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.730 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:47.730 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.889234+0000", 2024-09-18T04:37:47.730 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0802000000000003e-05 2024-09-18T04:37:47.730 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:47.730 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:47.730 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:47.730 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.889345+0000", 2024-09-18T04:37:47.731 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011112000000000001 2024-09-18T04:37:47.731 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:47.731 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:47.731 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:47.731 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:47.731 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:47.731 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:37:47.732 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:37:47.733 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:37:47.728+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:37:47.833 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:37:47.827+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:37:47.847 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:37:47.847 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:37:47.847 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:37:47.847 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:37:47.857 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:37:47.934 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:37:47.931+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:37:47.959 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:37:47.959 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:37:47.959 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:37:47.959 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:37:47.959 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:37:47.969 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:37:48.035 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:37:48.032+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:37:48.116 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:37:48.116 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:37:48.116 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:37:48.116 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [] 2024-09-18T04:37:48.116 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:37:48.125 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:37:48.136 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:37:48.131+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:37:48.229 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:37:48.230 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:37:48.230 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:37:48.230 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:37:48.237 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:37:48.236+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:37:48.239 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:37:48.338 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:37:48.335+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:37:48.349 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T04:37:48.390 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:37:48.391 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:37:48.391 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:37:48.392 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:37:48.392 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:37:48.400 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:37:48.446 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:37:48.443+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:37:48.503 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:37:48.510 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:37:48.510 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:37:48.511 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:37:48.511 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:37:48.511 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.511 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-18T04:37:48.511 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:37:48.511 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 14.037539026999999, 2024-09-18T04:37:48.511 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0095781189999999995, 2024-09-18T04:37:48.511 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:48.511 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:48.511 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:48.511 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:37:48.511 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:37:48.512 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 1 2024-09-18T04:37:48.512 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.512 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:48.512 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.512 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:48.512 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:37:48.512 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:48.512 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.512 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.512 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:48.513 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:37:48.513 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:48.513 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.513 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.513 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:48.513 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469862+0000", 2024-09-18T04:37:48.513 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:37:48.513 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.513 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.513 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:48.513 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469879+0000", 2024-09-18T04:37:48.514 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7431999999999998e-05 2024-09-18T04:37:48.514 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.514 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.514 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:48.514 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469896+0000", 2024-09-18T04:37:48.514 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6742999999999999e-05 2024-09-18T04:37:48.514 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.514 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.514 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:48.514 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469915+0000", 2024-09-18T04:37:48.515 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8991000000000002e-05 2024-09-18T04:37:48.515 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.515 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.515 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:48.515 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470397+0000", 2024-09-18T04:37:48.515 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048167799999999998 2024-09-18T04:37:48.515 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.515 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.515 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-09-18T04:37:48.516 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470412+0000", 2024-09-18T04:37:48.516 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5526e-05 2024-09-18T04:37:48.516 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.516 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.516 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:48.516 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473279+0000", 2024-09-18T04:37:48.516 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0028667469999999998 2024-09-18T04:37:48.516 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.517 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.517 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:48.517 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473441+0000", 2024-09-18T04:37:48.517 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000162082 2024-09-18T04:37:48.517 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.517 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.517 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:37:48.517 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473573+0000", 2024-09-18T04:37:48.517 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000132032 2024-09-18T04:37:48.517 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.518 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.518 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:48.518 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474826+0000", 2024-09-18T04:37:48.518 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001252717 2024-09-18T04:37:48.518 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.518 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.518 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:48.518 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479307+0000", 2024-09-18T04:37:48.518 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044813220000000003 2024-09-18T04:37:48.519 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.519 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.519 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:48.519 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479366+0000", 2024-09-18T04:37:48.519 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9024999999999998e-05 2024-09-18T04:37:48.519 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.519 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.519 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:48.519 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479443+0000", 2024-09-18T04:37:48.519 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6339000000000002e-05 2024-09-18T04:37:48.519 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:48.519 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:48.519 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:48.520 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.520 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.520 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.4214.0:1 2.3 e18/17 ondisk, result = 0)", 2024-09-18T04:37:48.520 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.474724+0000", 2024-09-18T04:37:48.520 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 14.032679663, 2024-09-18T04:37:48.520 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000105358, 2024-09-18T04:37:48.520 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:48.520 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:37:48.520 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:48.520 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.520 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:48.520 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474724+0000", 2024-09-18T04:37:48.520 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:48.521 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.521 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.521 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:48.521 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474724+0000", 2024-09-18T04:37:48.521 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:48.521 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.521 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.521 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:48.521 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474723+0000", 2024-09-18T04:37:48.521 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:37:48.521 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.521 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.521 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:48.521 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474731+0000", 2024-09-18T04:37:48.522 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2339999999999994e-06 2024-09-18T04:37:48.522 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.522 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.522 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:48.522 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474748+0000", 2024-09-18T04:37:48.522 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6603e-05 2024-09-18T04:37:48.522 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.522 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.522 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:48.522 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474760+0000", 2024-09-18T04:37:48.522 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2215000000000001e-05 2024-09-18T04:37:48.522 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.522 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.522 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:48.523 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474814+0000", 2024-09-18T04:37:48.523 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4267000000000001e-05 2024-09-18T04:37:48.523 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.523 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.523 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:48.523 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474821+0000", 2024-09-18T04:37:48.523 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0929999999999997e-06 2024-09-18T04:37:48.523 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.523 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.523 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:48.523 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474829+0000", 2024-09-18T04:37:48.523 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1200000000000002e-06 2024-09-18T04:37:48.523 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:48.523 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:48.524 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:48.524 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.524 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.524 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18)", 2024-09-18T04:37:48.524 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.482346+0000", 2024-09-18T04:37:48.524 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 14.025057231, 2024-09-18T04:37:48.524 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020435599999999999, 2024-09-18T04:37:48.524 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:48.524 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-09-18T04:37:48.524 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:48.524 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:37:48.524 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:37:48.524 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 2 2024-09-18T04:37:48.524 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.525 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:48.525 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.525 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:48.525 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482346+0000", 2024-09-18T04:37:48.525 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:48.525 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.525 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.525 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:48.525 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482346+0000", 2024-09-18T04:37:48.525 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:48.525 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.525 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.525 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:48.525 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482345+0000", 2024-09-18T04:37:48.526 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:37:48.526 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.526 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.526 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:48.526 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482360+0000", 2024-09-18T04:37:48.526 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4854999999999999e-05 2024-09-18T04:37:48.526 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.526 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.526 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:48.526 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482377+0000", 2024-09-18T04:37:48.526 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7785999999999999e-05 2024-09-18T04:37:48.526 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.526 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.527 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:48.527 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482387+0000", 2024-09-18T04:37:48.527 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6800000000000005e-06 2024-09-18T04:37:48.527 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.527 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.527 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:48.527 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482441+0000", 2024-09-18T04:37:48.527 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4046999999999999e-05 2024-09-18T04:37:48.527 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.528 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.528 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:48.528 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482551+0000", 2024-09-18T04:37:48.528 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000109468 2024-09-18T04:37:48.528 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:48.528 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:48.528 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:48.528 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.528 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.528 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18)", 2024-09-18T04:37:48.528 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.482853+0000", 2024-09-18T04:37:48.528 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 14.024550613000001, 2024-09-18T04:37:48.528 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00029815799999999997, 2024-09-18T04:37:48.529 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:48.529 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-09-18T04:37:48.529 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:48.529 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:37:48.529 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:37:48.529 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 3 2024-09-18T04:37:48.529 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.529 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:48.529 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.529 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:48.529 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482853+0000", 2024-09-18T04:37:48.529 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:48.529 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.530 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.530 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:48.530 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482853+0000", 2024-09-18T04:37:48.530 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:48.530 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.530 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.530 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:48.530 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482852+0000", 2024-09-18T04:37:48.530 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:37:48.530 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.530 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.530 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:48.530 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482859+0000", 2024-09-18T04:37:48.530 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0380000000000002e-06 2024-09-18T04:37:48.531 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.531 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.531 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:48.531 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482880+0000", 2024-09-18T04:37:48.531 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1243e-05 2024-09-18T04:37:48.531 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.531 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.531 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:48.531 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482896+0000", 2024-09-18T04:37:48.531 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5977e-05 2024-09-18T04:37:48.531 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.531 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.531 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:48.532 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482970+0000", 2024-09-18T04:37:48.532 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4603000000000004e-05 2024-09-18T04:37:48.532 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.532 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.532 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:48.532 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.483151+0000", 2024-09-18T04:37:48.532 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00018059799999999999 2024-09-18T04:37:48.532 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:48.532 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:48.532 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:48.532 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.532 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.532 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-18T04:37:48.533 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:35.468698+0000", 2024-09-18T04:37:48.533 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 13.038705776, 2024-09-18T04:37:48.533 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0039954129999999997, 2024-09-18T04:37:48.533 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:48.533 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:48.533 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:48.533 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:37:48.533 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:37:48.533 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 5 2024-09-18T04:37:48.533 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.533 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:48.533 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.534 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:48.534 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468698+0000", 2024-09-18T04:37:48.534 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:48.534 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.534 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.534 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:48.534 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468698+0000", 2024-09-18T04:37:48.534 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:48.534 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.534 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.534 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:48.534 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468696+0000", 2024-09-18T04:37:48.534 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:37:48.535 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.535 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.535 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:48.535 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468714+0000", 2024-09-18T04:37:48.535 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8743e-05 2024-09-18T04:37:48.535 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.535 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.535 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:48.535 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468748+0000", 2024-09-18T04:37:48.535 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3339e-05 2024-09-18T04:37:48.535 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.535 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.535 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:48.535 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468771+0000", 2024-09-18T04:37:48.536 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3119999999999999e-05 2024-09-18T04:37:48.536 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.536 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.536 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:48.536 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468922+0000", 2024-09-18T04:37:48.536 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000151383 2024-09-18T04:37:48.536 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.536 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.536 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:48.536 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.469278+0000", 2024-09-18T04:37:48.536 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00035628599999999998 2024-09-18T04:37:48.536 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.536 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.537 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:37:48.537 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.469629+0000", 2024-09-18T04:37:48.537 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00035014300000000003 2024-09-18T04:37:48.537 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.537 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.537 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:48.537 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.471620+0000", 2024-09-18T04:37:48.537 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001991278 2024-09-18T04:37:48.537 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.537 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.537 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:48.537 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472440+0000", 2024-09-18T04:37:48.538 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00081998599999999998 2024-09-18T04:37:48.538 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.538 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.538 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:48.538 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472592+0000", 2024-09-18T04:37:48.538 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015227400000000001 2024-09-18T04:37:48.538 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.538 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.538 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:48.538 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472693+0000", 2024-09-18T04:37:48.538 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000101074 2024-09-18T04:37:48.538 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:48.538 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:48.539 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:48.539 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.539 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.539 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.4214.0:5 2.2 e18/17 ondisk, result = 0)", 2024-09-18T04:37:48.539 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:35.472226+0000", 2024-09-18T04:37:48.539 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 13.035177876000001, 2024-09-18T04:37:48.539 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048505299999999999, 2024-09-18T04:37:48.539 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:48.539 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:37:48.539 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:48.539 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.539 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:48.539 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472226+0000", 2024-09-18T04:37:48.540 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:48.540 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.540 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.540 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:48.540 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472226+0000", 2024-09-18T04:37:48.540 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:48.540 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.540 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.540 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:48.541 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472224+0000", 2024-09-18T04:37:48.541 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:37:48.541 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.541 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.541 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:48.541 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472238+0000", 2024-09-18T04:37:48.541 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.468e-05 2024-09-18T04:37:48.541 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.541 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.542 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:48.542 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472272+0000", 2024-09-18T04:37:48.542 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3586e-05 2024-09-18T04:37:48.542 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.542 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.542 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:48.542 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472297+0000", 2024-09-18T04:37:48.542 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5282999999999998e-05 2024-09-18T04:37:48.542 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.542 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.542 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:48.542 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472396+0000", 2024-09-18T04:37:48.542 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.8394999999999996e-05 2024-09-18T04:37:48.542 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.543 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.543 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:48.543 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472421+0000", 2024-09-18T04:37:48.543 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5479000000000001e-05 2024-09-18T04:37:48.543 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.543 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.543 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:48.543 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472711+0000", 2024-09-18T04:37:48.543 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000289714 2024-09-18T04:37:48.543 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:48.543 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:48.543 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:48.543 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.544 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.544 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-18T04:37:48.544 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:37:48.544 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 12.024530607000001, 2024-09-18T04:37:48.544 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082699560000000002, 2024-09-18T04:37:48.544 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:48.544 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:48.544 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:48.544 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:37:48.544 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:37:48.544 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 7 2024-09-18T04:37:48.544 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.544 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:48.545 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.545 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:48.545 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:37:48.545 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:48.545 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.545 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.545 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:48.545 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:37:48.545 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:48.545 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.546 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.546 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:48.546 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482870+0000", 2024-09-18T04:37:48.546 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999967 2024-09-18T04:37:48.546 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.546 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.546 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:48.546 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482889+0000", 2024-09-18T04:37:48.546 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.965e-05 2024-09-18T04:37:48.546 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.546 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.546 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:48.546 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482938+0000", 2024-09-18T04:37:48.546 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8548999999999999e-05 2024-09-18T04:37:48.547 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.547 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.547 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:48.547 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482963+0000", 2024-09-18T04:37:48.547 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5075999999999998e-05 2024-09-18T04:37:48.547 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.547 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.547 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:48.547 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483121+0000", 2024-09-18T04:37:48.547 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015819400000000001 2024-09-18T04:37:48.547 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.547 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.547 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op must wait for map", 2024-09-18T04:37:48.548 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483138+0000", 2024-09-18T04:37:48.548 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7068999999999999e-05 2024-09-18T04:37:48.548 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.548 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.548 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:48.548 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488666+0000", 2024-09-18T04:37:48.548 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0055282049999999996 2024-09-18T04:37:48.548 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.548 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.548 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:48.548 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488783+0000", 2024-09-18T04:37:48.548 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000116737 2024-09-18T04:37:48.548 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.548 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.549 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:37:48.549 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488991+0000", 2024-09-18T04:37:48.549 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020781400000000001 2024-09-18T04:37:48.549 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.549 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.549 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:48.549 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490554+0000", 2024-09-18T04:37:48.549 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0015628370000000001 2024-09-18T04:37:48.549 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.549 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.549 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:48.549 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491019+0000", 2024-09-18T04:37:48.549 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000465608 2024-09-18T04:37:48.549 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.550 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.550 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:48.550 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491087+0000", 2024-09-18T04:37:48.550 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7988e-05 2024-09-18T04:37:48.550 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.550 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.550 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:48.550 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491143+0000", 2024-09-18T04:37:48.550 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5538999999999998e-05 2024-09-18T04:37:48.550 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:48.550 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:48.550 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:48.550 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.550 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.551 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.4214.0:7 2.2 e19/17 ondisk, result = 0)", 2024-09-18T04:37:48.551 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:36.490861+0000", 2024-09-18T04:37:48.551 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 12.016542984999999, 2024-09-18T04:37:48.551 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00029639400000000001, 2024-09-18T04:37:48.551 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:48.551 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:37:48.551 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:48.551 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.551 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:48.551 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490861+0000", 2024-09-18T04:37:48.551 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:48.551 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.551 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.552 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:48.552 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490861+0000", 2024-09-18T04:37:48.552 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:48.552 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.552 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.552 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:48.552 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490859+0000", 2024-09-18T04:37:48.552 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:37:48.552 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.552 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.552 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:48.552 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490867+0000", 2024-09-18T04:37:48.552 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.3089999999999996e-06 2024-09-18T04:37:48.553 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.553 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.553 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:48.553 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490909+0000", 2024-09-18T04:37:48.553 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1919000000000003e-05 2024-09-18T04:37:48.553 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.553 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.553 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:48.553 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490922+0000", 2024-09-18T04:37:48.553 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2836999999999999e-05 2024-09-18T04:37:48.553 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.553 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.553 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:48.554 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491003+0000", 2024-09-18T04:37:48.554 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0681999999999998e-05 2024-09-18T04:37:48.554 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.554 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.554 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:48.554 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491013+0000", 2024-09-18T04:37:48.554 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0220999999999999e-05 2024-09-18T04:37:48.554 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:48.554 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:48.554 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:48.554 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491157+0000", 2024-09-18T04:37:48.554 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014401799999999999 2024-09-18T04:37:48.555 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:48.555 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:48.555 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:48.555 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:48.555 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:48.555 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:37:48.556 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:37:48.556 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:37:48.557 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:37:48.551+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:37:48.557 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:37:48.573 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:37:48.595 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:37:48.618 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:37:48.620 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:37:48.620 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:37:48.621 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:37:48.621 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:37:48.621 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:37:48.621 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:37:48.657 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:37:48.655+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:37:48.666 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:48.666 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:48.676 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:37:48.759 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:37:48.755+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:37:48.781 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:48.781 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:48.791 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:37:48.862 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:37:48.856+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:37:48.900 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:48.901 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:48.911 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:37:48.961 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:37:48.955+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:37:49.063 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:37:49.060+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:37:49.063 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:49.063 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:49.074 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:37:49.164 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:37:49.159+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:37:49.185 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:49.185 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:49.192 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:37:49.266 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:37:49.259+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:37:49.300 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:49.300 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:49.311 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:37:49.367 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:37:49.363+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:37:49.421 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:49.421 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:49.431 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:37:49.468 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:37:49.464+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:37:49.537 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:49.537 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:49.540 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-18T04:37:49.540 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T04:37:49.540 INFO:tasks.thrashosds.thrasher:Adding osd 2 2024-09-18T04:37:49.540 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 2 2024-09-18T04:37:49.547 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:37:49.557 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-18T04:37:49.569 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:37:49.563+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:37:49.667 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:49.667 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:49.669 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:37:49.668+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:37:49.678 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:37:49.771 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:37:49.768+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:37:49.790 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:49.790 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:49.800 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:37:49.856 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-09-18T04:37:49.852+0000 7f8dc00db700 -1 mon.a@0(leader).osd e23 definitely_dead 0 2024-09-18T04:37:49.871 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:37:49.868+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:37:49.907 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:49.907 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:49.917 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:37:49.972 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:37:49.967+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:37:50.046 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:50.047 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:50.056 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:37:50.073 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:37:50.068+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:37:50.157 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:50.157 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:50.166 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:37:50.173 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:37:50.172+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:37:50.271 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:50.271 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:50.275 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:37:50.272+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:37:50.281 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:37:50.375 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:37:50.372+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:37:50.389 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:50.389 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:50.399 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:37:50.476 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:37:50.472+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:37:50.510 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:50.510 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:50.520 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:37:50.577 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:37:50.572+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:37:50.627 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:50.627 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:50.636 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T04:37:50.637 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:37:50.672 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-09-18T04:37:50.672 INFO:teuthology.orchestra.run.smithi028.stderr:marked in osd.2. 2024-09-18T04:37:50.682 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:37:50.679+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:37:50.685 INFO:tasks.thrashosds.thrasher:Added osd 2 2024-09-18T04:37:50.744 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:50.744 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:50.754 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:37:50.778 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:37:50.776+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:37:50.787 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:37:50.807 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:37:50.827 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:37:50.849 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:37:50.862 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:37:50.862 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:37:50.862 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:37:50.862 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:37:50.871 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:37:50.872 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:37:50.879 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:37:50.875+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:37:50.897 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:37:50.900 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:37:50.900 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:37:50.900 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:37:50.900 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:37:50.900 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:37:50.900 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:37:50.980 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:37:50.976+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:37:50.980 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:37:50.980 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:37:50.981 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:37:50.981 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:37:50.981 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:37:50.990 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:37:51.080 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:37:51.075+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:37:51.150 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:37:51.150 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:37:51.150 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:37:51.150 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:37:51.150 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.150 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:30 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'16, mlcod=15'14)", 2024-09-18T04:37:51.150 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.486330+0000", 2024-09-18T04:37:51.150 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 23.659881612, 2024-09-18T04:37:51.150 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026383700000000001, 2024-09-18T04:37:51.150 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:51.151 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:51.151 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:51.151 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.151 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:51.151 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486330+0000", 2024-09-18T04:37:51.151 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:51.151 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.151 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.151 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:51.151 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486330+0000", 2024-09-18T04:37:51.151 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:51.151 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.151 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.151 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:51.152 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486328+0000", 2024-09-18T04:37:51.152 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:37:51.152 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.152 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.152 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:51.152 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486342+0000", 2024-09-18T04:37:51.152 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3783e-05 2024-09-18T04:37:51.152 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.152 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.152 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:51.152 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486373+0000", 2024-09-18T04:37:51.152 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1362999999999998e-05 2024-09-18T04:37:51.153 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.153 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.153 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:51.153 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486387+0000", 2024-09-18T04:37:51.153 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3219e-05 2024-09-18T04:37:51.153 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.153 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.153 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:51.153 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486489+0000", 2024-09-18T04:37:51.153 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010232499999999999 2024-09-18T04:37:51.153 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.153 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.154 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:51.154 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486524+0000", 2024-09-18T04:37:51.154 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4972000000000003e-05 2024-09-18T04:37:51.154 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.154 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.154 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:51.154 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.488902+0000", 2024-09-18T04:37:51.154 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023776240000000001 2024-09-18T04:37:51.154 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.154 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.154 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:51.154 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.488968+0000", 2024-09-18T04:37:51.154 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6761000000000007e-05 2024-09-18T04:37:51.155 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:51.155 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:51.155 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:51.155 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.155 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.155 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:35 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'21, mlcod=15'17)", 2024-09-18T04:37:51.155 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.491817+0000", 2024-09-18T04:37:51.155 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 23.654394694, 2024-09-18T04:37:51.155 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023524739999999998, 2024-09-18T04:37:51.155 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:51.155 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:51.155 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:51.155 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.156 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:51.156 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491817+0000", 2024-09-18T04:37:51.156 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:51.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.156 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.156 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:51.156 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491817+0000", 2024-09-18T04:37:51.156 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:51.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.156 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.156 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:51.156 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491815+0000", 2024-09-18T04:37:51.157 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:37:51.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.157 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.157 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:51.157 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491890+0000", 2024-09-18T04:37:51.157 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4500999999999997e-05 2024-09-18T04:37:51.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.157 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.157 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:51.157 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491916+0000", 2024-09-18T04:37:51.157 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5666000000000001e-05 2024-09-18T04:37:51.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.157 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.158 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:51.158 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491928+0000", 2024-09-18T04:37:51.158 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2585999999999999e-05 2024-09-18T04:37:51.158 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.158 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.158 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:51.158 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.492000+0000", 2024-09-18T04:37:51.158 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2309000000000002e-05 2024-09-18T04:37:51.158 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.158 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.158 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:51.158 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.492027+0000", 2024-09-18T04:37:51.158 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6251e-05 2024-09-18T04:37:51.158 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.159 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.159 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:51.159 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.494114+0000", 2024-09-18T04:37:51.159 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020875260000000001 2024-09-18T04:37:51.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.159 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.159 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:51.159 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.494169+0000", 2024-09-18T04:37:51.159 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5145000000000001e-05 2024-09-18T04:37:51.159 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:51.159 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:51.159 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:51.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.160 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.160 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:57 1.0 e15/14 1:30306672:devicehealth::main.db.0000000000000000:head v 15'35, mlcod=15'33)", 2024-09-18T04:37:51.160 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.529997+0000", 2024-09-18T04:37:51.160 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 23.616214840000001, 2024-09-18T04:37:51.160 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002623869, 2024-09-18T04:37:51.160 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:51.160 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:51.160 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:51.160 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.160 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:51.160 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.529997+0000", 2024-09-18T04:37:51.160 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:51.160 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.161 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.161 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:51.161 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.529997+0000", 2024-09-18T04:37:51.161 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:51.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.161 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.161 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:51.161 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.529995+0000", 2024-09-18T04:37:51.161 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:37:51.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.161 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.161 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:51.161 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530116+0000", 2024-09-18T04:37:51.162 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000121141 2024-09-18T04:37:51.162 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.162 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.162 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:51.162 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530146+0000", 2024-09-18T04:37:51.162 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0423e-05 2024-09-18T04:37:51.162 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.162 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.162 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:51.162 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530159+0000", 2024-09-18T04:37:51.162 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2802999999999999e-05 2024-09-18T04:37:51.162 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.162 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.163 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:51.163 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530218+0000", 2024-09-18T04:37:51.163 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8461000000000002e-05 2024-09-18T04:37:51.163 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.163 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.163 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:51.163 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530253+0000", 2024-09-18T04:37:51.163 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4796999999999999e-05 2024-09-18T04:37:51.163 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.163 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.163 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:51.163 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.532559+0000", 2024-09-18T04:37:51.163 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002306939 2024-09-18T04:37:51.163 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.164 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.164 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:51.164 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.532621+0000", 2024-09-18T04:37:51.164 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1131999999999997e-05 2024-09-18T04:37:51.164 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:51.164 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:51.164 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:51.164 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.164 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.164 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:68 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'42, mlcod=15'39)", 2024-09-18T04:37:51.164 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.546404+0000", 2024-09-18T04:37:51.164 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 23.599807428999998, 2024-09-18T04:37:51.164 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024499769999999999, 2024-09-18T04:37:51.164 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:51.165 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:51.165 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:51.165 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.165 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:51.165 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546404+0000", 2024-09-18T04:37:51.165 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:51.165 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.165 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.165 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:51.165 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546404+0000", 2024-09-18T04:37:51.165 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:51.165 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.165 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.166 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:51.166 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546403+0000", 2024-09-18T04:37:51.166 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:37:51.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.166 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.166 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:51.166 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546416+0000", 2024-09-18T04:37:51.166 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.306e-05 2024-09-18T04:37:51.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.166 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.166 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:51.166 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546465+0000", 2024-09-18T04:37:51.166 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9441999999999999e-05 2024-09-18T04:37:51.167 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.167 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.167 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:51.167 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546480+0000", 2024-09-18T04:37:51.167 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4919999999999999e-05 2024-09-18T04:37:51.167 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.167 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.167 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:51.167 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546541+0000", 2024-09-18T04:37:51.167 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1085000000000002e-05 2024-09-18T04:37:51.167 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.167 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.167 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:51.168 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546569+0000", 2024-09-18T04:37:51.168 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7803000000000001e-05 2024-09-18T04:37:51.168 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.168 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.168 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:51.168 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.548725+0000", 2024-09-18T04:37:51.168 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021556629999999999 2024-09-18T04:37:51.168 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.168 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.168 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:51.168 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.548854+0000", 2024-09-18T04:37:51.168 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000129524 2024-09-18T04:37:51.168 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:51.168 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:51.169 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:51.169 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.169 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.169 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:69 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'43, mlcod=15'39)", 2024-09-18T04:37:51.169 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.547520+0000", 2024-09-18T04:37:51.169 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 23.598691353, 2024-09-18T04:37:51.169 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023506820000000002, 2024-09-18T04:37:51.169 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:51.169 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:51.169 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:51.169 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.169 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:51.169 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547520+0000", 2024-09-18T04:37:51.170 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:51.170 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.170 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.170 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:51.170 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547520+0000", 2024-09-18T04:37:51.170 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:51.170 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.170 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.170 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:51.170 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547519+0000", 2024-09-18T04:37:51.170 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:37:51.170 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.170 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.170 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:51.171 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547614+0000", 2024-09-18T04:37:51.171 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5235000000000006e-05 2024-09-18T04:37:51.171 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.171 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.171 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:51.171 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547640+0000", 2024-09-18T04:37:51.171 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5698999999999999e-05 2024-09-18T04:37:51.171 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.171 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.171 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:51.171 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547651+0000", 2024-09-18T04:37:51.171 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.134e-05 2024-09-18T04:37:51.171 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.172 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.172 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:51.172 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547805+0000", 2024-09-18T04:37:51.172 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015426099999999999 2024-09-18T04:37:51.172 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.172 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.172 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:51.172 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547844+0000", 2024-09-18T04:37:51.172 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8229999999999998e-05 2024-09-18T04:37:51.172 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.172 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.172 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:51.172 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.549810+0000", 2024-09-18T04:37:51.173 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0019660519999999998 2024-09-18T04:37:51.173 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.173 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.173 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:51.173 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.549871+0000", 2024-09-18T04:37:51.173 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1152e-05 2024-09-18T04:37:51.173 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:51.173 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:51.173 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:51.173 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.173 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.173 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:83 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'53, mlcod=15'50)", 2024-09-18T04:37:51.173 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.575989+0000", 2024-09-18T04:37:51.173 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 23.570222404999999, 2024-09-18T04:37:51.174 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023991310000000001, 2024-09-18T04:37:51.174 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:51.174 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:51.174 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:51.174 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.174 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:51.174 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.575989+0000", 2024-09-18T04:37:51.174 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:51.174 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.174 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.174 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:51.174 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.575989+0000", 2024-09-18T04:37:51.174 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:51.174 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.175 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.175 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:51.175 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.575988+0000", 2024-09-18T04:37:51.175 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:37:51.175 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.175 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.175 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:51.175 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576001+0000", 2024-09-18T04:37:51.175 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2751e-05 2024-09-18T04:37:51.175 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.175 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.175 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:51.175 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576023+0000", 2024-09-18T04:37:51.176 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2453999999999999e-05 2024-09-18T04:37:51.176 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.176 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.176 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:51.176 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576038+0000", 2024-09-18T04:37:51.176 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5125e-05 2024-09-18T04:37:51.176 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.176 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.176 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:51.176 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576161+0000", 2024-09-18T04:37:51.176 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000122543 2024-09-18T04:37:51.176 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.176 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.177 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:51.177 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576221+0000", 2024-09-18T04:37:51.177 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9948000000000003e-05 2024-09-18T04:37:51.177 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.177 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.177 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:51.177 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.578277+0000", 2024-09-18T04:37:51.177 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020567559999999999 2024-09-18T04:37:51.177 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.177 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.177 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:51.177 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.578388+0000", 2024-09-18T04:37:51.177 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011089100000000001 2024-09-18T04:37:51.178 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:51.178 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:51.178 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:51.178 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.178 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.178 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:86 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'56, mlcod=15'52)", 2024-09-18T04:37:51.178 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.580261+0000", 2024-09-18T04:37:51.178 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 23.565950769000001, 2024-09-18T04:37:51.178 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002418108, 2024-09-18T04:37:51.178 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:51.178 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:51.178 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:51.178 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.179 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:51.179 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580261+0000", 2024-09-18T04:37:51.179 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:51.179 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.179 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.179 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:51.179 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580261+0000", 2024-09-18T04:37:51.179 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:51.179 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.179 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.179 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:51.179 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580260+0000", 2024-09-18T04:37:51.179 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:37:51.179 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.180 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.180 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:51.180 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580269+0000", 2024-09-18T04:37:51.180 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5109999999999992e-06 2024-09-18T04:37:51.180 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.180 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.180 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:51.180 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580288+0000", 2024-09-18T04:37:51.180 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8783999999999998e-05 2024-09-18T04:37:51.180 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.180 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.180 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:51.180 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580296+0000", 2024-09-18T04:37:51.181 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3250000000000008e-06 2024-09-18T04:37:51.181 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.181 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.181 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:51.181 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580351+0000", 2024-09-18T04:37:51.181 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4132000000000003e-05 2024-09-18T04:37:51.181 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.181 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.181 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:51.181 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580381+0000", 2024-09-18T04:37:51.181 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0263000000000001e-05 2024-09-18T04:37:51.181 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.181 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.182 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:51.182 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.582615+0000", 2024-09-18T04:37:51.182 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0022339109999999999 2024-09-18T04:37:51.182 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.182 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.182 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:51.182 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.582679+0000", 2024-09-18T04:37:51.182 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4182000000000003e-05 2024-09-18T04:37:51.182 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:51.182 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:51.182 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:51.182 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.182 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.183 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:105 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'67, mlcod=15'64)", 2024-09-18T04:37:51.183 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.789089+0000", 2024-09-18T04:37:51.183 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 23.357122599, 2024-09-18T04:37:51.183 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024384229999999999, 2024-09-18T04:37:51.183 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:51.183 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:51.183 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:51.183 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.183 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:51.183 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789089+0000", 2024-09-18T04:37:51.183 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:51.183 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.183 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.184 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:51.184 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789089+0000", 2024-09-18T04:37:51.184 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:51.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.184 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.184 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:51.184 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789088+0000", 2024-09-18T04:37:51.184 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:37:51.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.184 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.184 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:51.184 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789107+0000", 2024-09-18T04:37:51.184 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9017000000000001e-05 2024-09-18T04:37:51.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.185 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.185 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:51.185 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789139+0000", 2024-09-18T04:37:51.185 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2261999999999997e-05 2024-09-18T04:37:51.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.185 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.185 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:51.185 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789153+0000", 2024-09-18T04:37:51.185 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4025999999999999e-05 2024-09-18T04:37:51.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.185 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.185 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:51.185 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789255+0000", 2024-09-18T04:37:51.186 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000101969 2024-09-18T04:37:51.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.186 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.186 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:51.186 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789314+0000", 2024-09-18T04:37:51.186 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8867999999999999e-05 2024-09-18T04:37:51.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.186 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.186 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:51.186 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.791453+0000", 2024-09-18T04:37:51.186 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002138938 2024-09-18T04:37:51.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.186 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.186 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:51.187 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.791527+0000", 2024-09-18T04:37:51.187 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4464999999999999e-05 2024-09-18T04:37:51.187 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:51.187 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:51.187 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:51.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.187 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.187 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:106 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'68, mlcod=15'64)", 2024-09-18T04:37:51.187 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.790295+0000", 2024-09-18T04:37:51.187 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 23.355916747999999, 2024-09-18T04:37:51.187 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024379390000000001, 2024-09-18T04:37:51.187 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:51.187 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:51.188 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:51.188 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.188 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:51.188 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790295+0000", 2024-09-18T04:37:51.188 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:51.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.188 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.188 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:51.188 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790295+0000", 2024-09-18T04:37:51.188 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:51.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.188 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.188 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:51.189 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790294+0000", 2024-09-18T04:37:51.189 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:37:51.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.189 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.189 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:51.189 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790390+0000", 2024-09-18T04:37:51.189 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6537000000000001e-05 2024-09-18T04:37:51.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.189 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.189 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:51.189 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790418+0000", 2024-09-18T04:37:51.189 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8218e-05 2024-09-18T04:37:51.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.189 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.190 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:51.190 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790431+0000", 2024-09-18T04:37:51.190 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2571999999999999e-05 2024-09-18T04:37:51.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.190 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.190 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:51.190 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790534+0000", 2024-09-18T04:37:51.190 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000103433 2024-09-18T04:37:51.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.190 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.190 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:51.190 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790565+0000", 2024-09-18T04:37:51.190 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0409999999999999e-05 2024-09-18T04:37:51.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.190 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.191 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:51.191 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.792651+0000", 2024-09-18T04:37:51.191 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020865250000000001 2024-09-18T04:37:51.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.191 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.191 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:51.191 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.792733+0000", 2024-09-18T04:37:51.191 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1477999999999995e-05 2024-09-18T04:37:51.191 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:51.191 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:51.191 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:51.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.191 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.191 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:109 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'71, mlcod=15'68)", 2024-09-18T04:37:51.192 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.795250+0000", 2024-09-18T04:37:51.192 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 23.350961733999998, 2024-09-18T04:37:51.192 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023990449999999998, 2024-09-18T04:37:51.192 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:51.192 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:51.192 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:51.192 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.192 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:51.192 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795250+0000", 2024-09-18T04:37:51.192 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:51.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.192 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.192 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:51.193 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795250+0000", 2024-09-18T04:37:51.193 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:51.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.193 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.193 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:51.193 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795248+0000", 2024-09-18T04:37:51.193 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:37:51.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.193 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.193 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:51.193 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795405+0000", 2024-09-18T04:37:51.193 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000157007 2024-09-18T04:37:51.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.193 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.194 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:51.194 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795422+0000", 2024-09-18T04:37:51.194 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7013000000000001e-05 2024-09-18T04:37:51.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.194 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.194 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:51.194 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795432+0000", 2024-09-18T04:37:51.194 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0448999999999999e-05 2024-09-18T04:37:51.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.194 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.194 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:51.194 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795534+0000", 2024-09-18T04:37:51.194 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000101195 2024-09-18T04:37:51.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.195 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.195 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:51.195 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795587+0000", 2024-09-18T04:37:51.195 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3356000000000002e-05 2024-09-18T04:37:51.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.195 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.195 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:51.195 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.797560+0000", 2024-09-18T04:37:51.195 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001973023 2024-09-18T04:37:51.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.195 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.195 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:51.195 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.797649+0000", 2024-09-18T04:37:51.196 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8864999999999997e-05 2024-09-18T04:37:51.196 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:51.196 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:51.196 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:51.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.196 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.196 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:115 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'77, mlcod=15'72)", 2024-09-18T04:37:51.196 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.803339+0000", 2024-09-18T04:37:51.196 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 23.342873015999999, 2024-09-18T04:37:51.196 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002385093, 2024-09-18T04:37:51.196 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:51.196 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:51.196 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:51.197 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.197 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:51.197 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803339+0000", 2024-09-18T04:37:51.197 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:51.197 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.197 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.197 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:51.197 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803339+0000", 2024-09-18T04:37:51.197 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:51.197 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.197 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.197 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:51.197 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803337+0000", 2024-09-18T04:37:51.197 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:37:51.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.198 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.198 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:51.198 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803475+0000", 2024-09-18T04:37:51.198 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013718199999999999 2024-09-18T04:37:51.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.198 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.198 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:51.198 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803507+0000", 2024-09-18T04:37:51.198 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2351000000000002e-05 2024-09-18T04:37:51.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.198 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.198 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:51.198 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803533+0000", 2024-09-18T04:37:51.199 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6010999999999999e-05 2024-09-18T04:37:51.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.199 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.199 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:51.199 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803609+0000", 2024-09-18T04:37:51.199 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.64e-05 2024-09-18T04:37:51.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.199 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.199 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:51.199 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803671+0000", 2024-09-18T04:37:51.199 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1467e-05 2024-09-18T04:37:51.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.199 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.200 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:51.200 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.805676+0000", 2024-09-18T04:37:51.200 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020047849999999998 2024-09-18T04:37:51.200 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.200 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.200 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:51.200 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.805724+0000", 2024-09-18T04:37:51.200 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8151e-05 2024-09-18T04:37:51.200 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:51.200 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:51.200 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:51.200 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.200 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.200 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:118 1.0 e15/14 1:30306672:devicehealth::main.db.0000000000000000:head v 15'80, mlcod=15'78)", 2024-09-18T04:37:51.201 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.807930+0000", 2024-09-18T04:37:51.201 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 23.338281512999998, 2024-09-18T04:37:51.201 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028002259999999998, 2024-09-18T04:37:51.201 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:51.201 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:51.201 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:51.201 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.201 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:51.201 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.807930+0000", 2024-09-18T04:37:51.201 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:51.201 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.201 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.202 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:51.202 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.807930+0000", 2024-09-18T04:37:51.202 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:51.202 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.202 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.202 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:51.202 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.807929+0000", 2024-09-18T04:37:51.202 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:37:51.202 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.202 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.203 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:51.203 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808024+0000", 2024-09-18T04:37:51.203 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5258000000000002e-05 2024-09-18T04:37:51.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.203 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.203 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:51.203 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808044+0000", 2024-09-18T04:37:51.203 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0203e-05 2024-09-18T04:37:51.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.203 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.203 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:51.203 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808053+0000", 2024-09-18T04:37:51.203 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2689999999999999e-06 2024-09-18T04:37:51.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.204 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.204 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:51.204 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808137+0000", 2024-09-18T04:37:51.204 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4441e-05 2024-09-18T04:37:51.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.204 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.204 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:51.204 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808172+0000", 2024-09-18T04:37:51.204 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4453999999999999e-05 2024-09-18T04:37:51.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.204 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.204 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:51.204 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.810663+0000", 2024-09-18T04:37:51.204 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024910269999999998 2024-09-18T04:37:51.205 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.205 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.205 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:51.205 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.810730+0000", 2024-09-18T04:37:51.205 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7658999999999997e-05 2024-09-18T04:37:51.205 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:51.205 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:51.205 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:51.205 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.205 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.205 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:129 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'87, mlcod=15'85)", 2024-09-18T04:37:51.205 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.231689+0000", 2024-09-18T04:37:51.205 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.9145225, 2024-09-18T04:37:51.205 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002376502, 2024-09-18T04:37:51.206 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:51.206 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:51.206 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:51.206 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.206 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:51.206 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231689+0000", 2024-09-18T04:37:51.206 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:51.206 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.206 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.206 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:51.206 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231689+0000", 2024-09-18T04:37:51.206 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:51.206 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.206 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.207 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:51.207 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231687+0000", 2024-09-18T04:37:51.207 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:37:51.207 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.207 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.207 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:51.207 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231702+0000", 2024-09-18T04:37:51.207 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4949999999999999e-05 2024-09-18T04:37:51.207 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.207 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.207 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:51.207 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231735+0000", 2024-09-18T04:37:51.207 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2276000000000001e-05 2024-09-18T04:37:51.208 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.208 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.208 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:51.208 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231750+0000", 2024-09-18T04:37:51.208 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5193e-05 2024-09-18T04:37:51.208 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.208 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.208 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:51.208 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231872+0000", 2024-09-18T04:37:51.208 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000122127 2024-09-18T04:37:51.208 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.208 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.208 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:51.209 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231936+0000", 2024-09-18T04:37:51.209 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4089000000000003e-05 2024-09-18T04:37:51.209 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.209 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.209 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:51.209 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234002+0000", 2024-09-18T04:37:51.209 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002065575 2024-09-18T04:37:51.209 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.209 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.209 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:51.209 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234066+0000", 2024-09-18T04:37:51.209 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3944999999999998e-05 2024-09-18T04:37:51.209 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:51.209 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:51.210 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:51.210 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.210 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.210 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:130 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'88, mlcod=15'85)", 2024-09-18T04:37:51.210 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.232853+0000", 2024-09-18T04:37:51.210 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.913358885000001, 2024-09-18T04:37:51.210 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027351739999999999, 2024-09-18T04:37:51.210 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:51.210 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:51.210 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:51.210 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.210 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:51.210 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232853+0000", 2024-09-18T04:37:51.210 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:51.211 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.211 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.211 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:51.211 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232853+0000", 2024-09-18T04:37:51.211 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:51.211 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.211 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.211 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:51.211 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232851+0000", 2024-09-18T04:37:51.211 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:37:51.211 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.211 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.211 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:51.212 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232866+0000", 2024-09-18T04:37:51.212 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4946000000000001e-05 2024-09-18T04:37:51.212 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.212 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.212 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:51.212 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232897+0000", 2024-09-18T04:37:51.212 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0982999999999998e-05 2024-09-18T04:37:51.212 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.212 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.212 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:51.212 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232915+0000", 2024-09-18T04:37:51.212 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8476000000000001e-05 2024-09-18T04:37:51.212 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.212 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.213 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:51.213 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.233029+0000", 2024-09-18T04:37:51.213 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011305400000000001 2024-09-18T04:37:51.213 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.213 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.213 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:51.213 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.233079+0000", 2024-09-18T04:37:51.213 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0668999999999999e-05 2024-09-18T04:37:51.213 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.213 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.213 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:51.213 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235534+0000", 2024-09-18T04:37:51.213 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024546030000000001 2024-09-18T04:37:51.214 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.214 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.214 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:51.214 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235588+0000", 2024-09-18T04:37:51.214 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4098999999999998e-05 2024-09-18T04:37:51.214 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:51.214 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:51.214 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:51.214 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.214 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.214 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:131 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'89, mlcod=15'85)", 2024-09-18T04:37:51.214 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.234089+0000", 2024-09-18T04:37:51.214 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.912123031, 2024-09-18T04:37:51.214 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026975609999999998, 2024-09-18T04:37:51.215 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:51.215 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:51.215 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:51.215 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.215 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:51.215 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234089+0000", 2024-09-18T04:37:51.215 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:51.215 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.215 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.215 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:51.215 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234089+0000", 2024-09-18T04:37:51.215 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:51.215 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.215 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.216 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:51.216 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234087+0000", 2024-09-18T04:37:51.216 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:37:51.216 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.216 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.216 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:51.216 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234145+0000", 2024-09-18T04:37:51.216 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8687999999999998e-05 2024-09-18T04:37:51.216 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.216 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.216 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:51.216 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234177+0000", 2024-09-18T04:37:51.216 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1526999999999998e-05 2024-09-18T04:37:51.217 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.217 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.217 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:51.217 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234190+0000", 2024-09-18T04:37:51.217 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2965999999999999e-05 2024-09-18T04:37:51.217 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.217 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.217 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:51.217 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234309+0000", 2024-09-18T04:37:51.217 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011875399999999999 2024-09-18T04:37:51.217 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.217 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.217 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:51.218 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234365+0000", 2024-09-18T04:37:51.218 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6436e-05 2024-09-18T04:37:51.218 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.218 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.218 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:51.218 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236675+0000", 2024-09-18T04:37:51.218 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023099520000000001 2024-09-18T04:37:51.218 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.218 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.218 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:51.218 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236786+0000", 2024-09-18T04:37:51.218 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000111025 2024-09-18T04:37:51.218 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:51.218 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:51.219 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:51.219 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.219 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.219 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:132 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'90, mlcod=15'85)", 2024-09-18T04:37:51.219 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.235300+0000", 2024-09-18T04:37:51.219 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.910911581000001, 2024-09-18T04:37:51.219 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025982599999999998, 2024-09-18T04:37:51.219 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:51.219 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:51.219 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:51.219 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.219 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:51.219 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235300+0000", 2024-09-18T04:37:51.219 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:51.220 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.220 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.220 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:51.220 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235300+0000", 2024-09-18T04:37:51.220 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:51.220 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.220 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.220 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:51.220 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235298+0000", 2024-09-18T04:37:51.220 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:37:51.220 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.220 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.220 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:51.221 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235318+0000", 2024-09-18T04:37:51.221 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9714000000000001e-05 2024-09-18T04:37:51.221 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.221 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.221 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:51.221 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235337+0000", 2024-09-18T04:37:51.221 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8892000000000001e-05 2024-09-18T04:37:51.221 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.221 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.221 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:51.221 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235348+0000", 2024-09-18T04:37:51.221 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.129e-05 2024-09-18T04:37:51.221 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.222 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.222 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:51.222 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235681+0000", 2024-09-18T04:37:51.222 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00033323000000000001 2024-09-18T04:37:51.222 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.222 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.222 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:51.222 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235733+0000", 2024-09-18T04:37:51.222 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1391999999999999e-05 2024-09-18T04:37:51.222 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.222 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.222 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:51.222 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237839+0000", 2024-09-18T04:37:51.223 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002105647 2024-09-18T04:37:51.223 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.223 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.223 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:51.223 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237898+0000", 2024-09-18T04:37:51.223 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9759000000000002e-05 2024-09-18T04:37:51.223 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:51.223 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:51.223 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:51.223 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.223 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.223 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:133 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'91, mlcod=15'86)", 2024-09-18T04:37:51.223 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.236920+0000", 2024-09-18T04:37:51.224 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.909291693, 2024-09-18T04:37:51.224 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024318410000000001, 2024-09-18T04:37:51.224 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:51.224 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:51.224 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:51.224 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.224 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:51.224 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236920+0000", 2024-09-18T04:37:51.224 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:51.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.224 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.224 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:51.224 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236920+0000", 2024-09-18T04:37:51.225 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:51.225 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.225 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.225 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:51.225 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236918+0000", 2024-09-18T04:37:51.225 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:37:51.225 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.225 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.225 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:51.225 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236931+0000", 2024-09-18T04:37:51.225 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2683999999999999e-05 2024-09-18T04:37:51.225 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.226 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.226 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:51.226 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236951+0000", 2024-09-18T04:37:51.226 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0292000000000002e-05 2024-09-18T04:37:51.226 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.226 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.226 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:51.226 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236962+0000", 2024-09-18T04:37:51.226 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0703000000000001e-05 2024-09-18T04:37:51.226 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.226 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.226 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:51.226 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237095+0000", 2024-09-18T04:37:51.226 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013312400000000001 2024-09-18T04:37:51.227 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.227 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.227 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:51.227 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237133+0000", 2024-09-18T04:37:51.227 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7496000000000001e-05 2024-09-18T04:37:51.227 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.227 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.227 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:51.227 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.239206+0000", 2024-09-18T04:37:51.227 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020730290000000001 2024-09-18T04:37:51.227 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.227 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.227 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:51.228 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.239352+0000", 2024-09-18T04:37:51.228 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014593000000000001 2024-09-18T04:37:51.228 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:51.228 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:51.228 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:51.228 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.228 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.228 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:134 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'92, mlcod=15'86)", 2024-09-18T04:37:51.228 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.237894+0000", 2024-09-18T04:37:51.228 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.908317670999999, 2024-09-18T04:37:51.228 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025299519999999998, 2024-09-18T04:37:51.228 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:51.228 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:51.229 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:51.229 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.229 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:51.229 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237894+0000", 2024-09-18T04:37:51.229 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:51.229 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.229 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.229 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:51.229 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237894+0000", 2024-09-18T04:37:51.229 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:51.229 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.229 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.229 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:51.230 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237892+0000", 2024-09-18T04:37:51.230 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:37:51.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.230 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.230 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:51.230 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237986+0000", 2024-09-18T04:37:51.230 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3341e-05 2024-09-18T04:37:51.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.230 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.230 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:51.230 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.238012+0000", 2024-09-18T04:37:51.230 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5843e-05 2024-09-18T04:37:51.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.230 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.231 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:51.231 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.238027+0000", 2024-09-18T04:37:51.231 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5469e-05 2024-09-18T04:37:51.231 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.231 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.231 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:51.231 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.238114+0000", 2024-09-18T04:37:51.231 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7483000000000002e-05 2024-09-18T04:37:51.231 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.231 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.231 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:51.231 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.238152+0000", 2024-09-18T04:37:51.231 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7565999999999999e-05 2024-09-18T04:37:51.231 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.232 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.232 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:51.232 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.240297+0000", 2024-09-18T04:37:51.232 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021447620000000001 2024-09-18T04:37:51.232 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.232 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.232 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:51.232 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.240424+0000", 2024-09-18T04:37:51.232 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012705400000000001 2024-09-18T04:37:51.232 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:51.232 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:51.232 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:51.232 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.233 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.233 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:139 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'97, mlcod=15'93)", 2024-09-18T04:37:51.233 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.245692+0000", 2024-09-18T04:37:51.233 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.90051953, 2024-09-18T04:37:51.233 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024260480000000001, 2024-09-18T04:37:51.233 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:51.233 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:51.233 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:51.233 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.233 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:51.233 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245692+0000", 2024-09-18T04:37:51.233 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:51.233 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.234 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.234 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:51.234 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245692+0000", 2024-09-18T04:37:51.234 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:51.234 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.234 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.234 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:51.234 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245690+0000", 2024-09-18T04:37:51.234 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:37:51.234 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.234 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.234 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:51.234 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245712+0000", 2024-09-18T04:37:51.235 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1916000000000002e-05 2024-09-18T04:37:51.235 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.235 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.235 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:51.235 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245741+0000", 2024-09-18T04:37:51.235 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8503000000000001e-05 2024-09-18T04:37:51.235 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.235 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.235 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:51.235 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245759+0000", 2024-09-18T04:37:51.235 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.872e-05 2024-09-18T04:37:51.235 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.235 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.235 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:51.236 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245859+0000", 2024-09-18T04:37:51.236 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9641000000000004e-05 2024-09-18T04:37:51.236 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.236 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.236 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:51.236 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245900+0000", 2024-09-18T04:37:51.236 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0589000000000001e-05 2024-09-18T04:37:51.236 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.236 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.236 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:51.236 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.247961+0000", 2024-09-18T04:37:51.236 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020619190000000002 2024-09-18T04:37:51.236 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.237 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.237 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:51.237 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.248118+0000", 2024-09-18T04:37:51.237 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015664300000000001 2024-09-18T04:37:51.237 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:51.237 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:51.237 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:51.237 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.237 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.237 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:144 1.0 e15/14 1:30306672:devicehealth::main.db.0000000000000000:head v 15'102, mlcod=15'98)", 2024-09-18T04:37:51.237 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.252149+0000", 2024-09-18T04:37:51.237 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.894062464000001, 2024-09-18T04:37:51.237 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024174959999999999, 2024-09-18T04:37:51.238 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:51.238 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:51.238 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:51.238 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.238 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:51.238 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252149+0000", 2024-09-18T04:37:51.238 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:51.238 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.238 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.238 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:51.238 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252149+0000", 2024-09-18T04:37:51.238 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:51.238 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.238 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.238 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:51.239 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252148+0000", 2024-09-18T04:37:51.239 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:37:51.239 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.239 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.239 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:51.239 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252225+0000", 2024-09-18T04:37:51.239 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6576999999999995e-05 2024-09-18T04:37:51.239 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.239 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.239 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:51.239 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252242+0000", 2024-09-18T04:37:51.239 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7740999999999998e-05 2024-09-18T04:37:51.239 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.240 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.240 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:51.240 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252254+0000", 2024-09-18T04:37:51.240 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.137e-05 2024-09-18T04:37:51.240 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.240 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.240 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:51.240 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252385+0000", 2024-09-18T04:37:51.240 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013091099999999999 2024-09-18T04:37:51.240 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.240 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.240 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:51.240 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252426+0000", 2024-09-18T04:37:51.241 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1535999999999997e-05 2024-09-18T04:37:51.241 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.241 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.241 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:51.241 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.254498+0000", 2024-09-18T04:37:51.241 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002071477 2024-09-18T04:37:51.241 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.241 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.241 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:51.241 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.254567+0000", 2024-09-18T04:37:51.241 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9014e-05 2024-09-18T04:37:51.241 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:51.241 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:51.241 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:51.242 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:51.242 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:51.242 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:37:51.243 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:37:51.244 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:37:51.240+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:37:51.344 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:37:51.338+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:37:51.356 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:37:51.356 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:37:51.356 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:37:51.356 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:37:51.367 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:37:51.445 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:37:51.440+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:37:51.468 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:37:51.468 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:37:51.468 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:37:51.468 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:37:51.468 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:37:51.478 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:37:51.545 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:37:51.538+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:37:51.635 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:37:51.635 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:37:51.635 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:37:51.635 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:37:51.635 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.635 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4214.0:5 2.2 e19/17 2:5cea7035:::rbd_info:head v 19'1, mlcod=0'0)", 2024-09-18T04:37:51.635 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:35.471260+0000", 2024-09-18T04:37:51.635 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 16.163016445, 2024-09-18T04:37:51.635 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021025950000000001, 2024-09-18T04:37:51.636 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:51.636 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:51.636 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:51.636 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.636 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:51.636 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471260+0000", 2024-09-18T04:37:51.636 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:51.636 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.636 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.636 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:51.636 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471260+0000", 2024-09-18T04:37:51.636 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:51.636 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.636 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.637 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:51.637 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471259+0000", 2024-09-18T04:37:51.637 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:37:51.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.637 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.637 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:51.637 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471275+0000", 2024-09-18T04:37:51.637 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6135999999999999e-05 2024-09-18T04:37:51.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.637 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.637 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:51.637 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471298+0000", 2024-09-18T04:37:51.637 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3632999999999999e-05 2024-09-18T04:37:51.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.638 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.638 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:51.638 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471320+0000", 2024-09-18T04:37:51.638 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1852999999999999e-05 2024-09-18T04:37:51.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.638 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.638 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:51.638 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471445+0000", 2024-09-18T04:37:51.638 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012461699999999999 2024-09-18T04:37:51.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.638 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.638 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:51.638 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471529+0000", 2024-09-18T04:37:51.638 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3875000000000006e-05 2024-09-18T04:37:51.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.639 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.639 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:51.639 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.473317+0000", 2024-09-18T04:37:51.639 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017886899999999999 2024-09-18T04:37:51.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.639 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.639 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:51.639 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.473362+0000", 2024-09-18T04:37:51.639 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4724000000000001e-05 2024-09-18T04:37:51.639 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:51.639 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:51.639 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:51.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.640 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.640 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4214.0:7 2.2 e20/17 2:5cea7035:::rbd_info:head v 20'2, mlcod=0'0)", 2024-09-18T04:37:51.640 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:36.490560+0000", 2024-09-18T04:37:51.640 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 15.143716306, 2024-09-18T04:37:51.640 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0015326020000000001, 2024-09-18T04:37:51.640 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:51.640 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:51.640 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:51.640 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.640 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:51.640 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490560+0000", 2024-09-18T04:37:51.640 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:51.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.641 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.641 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:51.641 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490560+0000", 2024-09-18T04:37:51.641 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:51.641 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.641 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.641 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:51.641 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490559+0000", 2024-09-18T04:37:51.641 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:37:51.641 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.641 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.641 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:51.641 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490568+0000", 2024-09-18T04:37:51.641 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5699999999999993e-06 2024-09-18T04:37:51.642 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.642 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.642 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:51.642 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490599+0000", 2024-09-18T04:37:51.642 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1566000000000002e-05 2024-09-18T04:37:51.642 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.642 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.642 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:51.642 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490609+0000", 2024-09-18T04:37:51.642 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.8700000000000004e-06 2024-09-18T04:37:51.642 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.642 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.642 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:51.642 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490708+0000", 2024-09-18T04:37:51.643 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9139000000000001e-05 2024-09-18T04:37:51.643 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.643 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.643 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:51.643 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490736+0000", 2024-09-18T04:37:51.643 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7617e-05 2024-09-18T04:37:51.643 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.643 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.643 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:51.643 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.492016+0000", 2024-09-18T04:37:51.643 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0012796579999999999 2024-09-18T04:37:51.643 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:51.643 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:51.643 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:51.644 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.492092+0000", 2024-09-18T04:37:51.644 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6571999999999998e-05 2024-09-18T04:37:51.644 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:51.644 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:51.644 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:51.644 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:51.644 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:51.644 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:37:51.645 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:37:51.646 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:37:51.642+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:37:51.747 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:37:51.744+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:37:51.758 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:37:51.758 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:37:51.758 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:37:51.758 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:37:51.768 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:37:51.848 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:37:51.844+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:37:51.918 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:37:51.918 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:37:51.918 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:37:51.919 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:37:51.919 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:37:51.928 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:37:51.949 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:37:51.944+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:37:52.039 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:37:52.040 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:37:52.040 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:37:52.040 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:37:52.040 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:52.040 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4214.0:1 2.3 e18/17 2:c4c92e5a:::rbd_trash:head v 18'1, mlcod=0'0)", 2024-09-18T04:37:52.040 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:34.475123+0000", 2024-09-18T04:37:52.040 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.563575171, 2024-09-18T04:37:52.040 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00081394599999999996, 2024-09-18T04:37:52.040 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:52.040 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:52.040 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:52.040 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:52.041 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:52.041 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475123+0000", 2024-09-18T04:37:52.041 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:52.041 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:52.041 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:52.041 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:52.041 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475123+0000", 2024-09-18T04:37:52.041 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:52.041 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:52.041 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:52.041 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:52.041 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475121+0000", 2024-09-18T04:37:52.041 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:37:52.041 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:52.042 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:52.042 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:52.042 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475149+0000", 2024-09-18T04:37:52.042 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7155999999999999e-05 2024-09-18T04:37:52.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:52.042 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:52.042 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:52.042 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475165+0000", 2024-09-18T04:37:52.042 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6160000000000001e-05 2024-09-18T04:37:52.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:52.042 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:52.042 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:52.042 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475183+0000", 2024-09-18T04:37:52.042 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7918e-05 2024-09-18T04:37:52.043 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:52.043 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:52.043 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:52.043 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475246+0000", 2024-09-18T04:37:52.043 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3176000000000006e-05 2024-09-18T04:37:52.043 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:52.043 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:52.043 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:52.043 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475279+0000", 2024-09-18T04:37:52.043 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3192999999999999e-05 2024-09-18T04:37:52.043 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:52.044 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:52.044 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:52.044 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475871+0000", 2024-09-18T04:37:52.044 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00059206099999999995 2024-09-18T04:37:52.044 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:52.044 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:52.044 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:52.044 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475937+0000", 2024-09-18T04:37:52.044 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5439999999999997e-05 2024-09-18T04:37:52.044 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:52.044 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:52.044 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:52.044 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:52.045 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:52.045 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 25/24 [PushOp(2:c4c92e5a:::rbd_trash:head, version: 18'1, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:c4c92e5a:::rbd_trash:head@18'1, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-18T04:37:52.045 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:51.680544+0000", 2024-09-18T04:37:52.045 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.35815332900000002, 2024-09-18T04:37:52.045 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00050365200000000005, 2024-09-18T04:37:52.045 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:52.045 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:37:52.045 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:52.045 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:52.045 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:52.045 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680544+0000", 2024-09-18T04:37:52.045 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:52.045 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:52.045 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:52.046 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:52.046 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680544+0000", 2024-09-18T04:37:52.046 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:52.046 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:52.046 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:52.046 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:52.046 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680543+0000", 2024-09-18T04:37:52.046 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:37:52.046 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:52.046 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:52.046 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:52.046 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680571+0000", 2024-09-18T04:37:52.047 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7953000000000002e-05 2024-09-18T04:37:52.047 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:52.047 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:52.047 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:52.047 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680605+0000", 2024-09-18T04:37:52.047 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3933000000000002e-05 2024-09-18T04:37:52.047 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:52.047 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:52.047 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:52.047 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680636+0000", 2024-09-18T04:37:52.047 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1288999999999998e-05 2024-09-18T04:37:52.047 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:52.048 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:52.048 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:52.048 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680748+0000", 2024-09-18T04:37:52.048 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000111431 2024-09-18T04:37:52.048 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:52.048 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:52.048 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:52.048 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680758+0000", 2024-09-18T04:37:52.048 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0244999999999999e-05 2024-09-18T04:37:52.048 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:52.048 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:52.048 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:52.049 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.681048+0000", 2024-09-18T04:37:52.049 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00029003399999999999 2024-09-18T04:37:52.049 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:52.049 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:52.049 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:52.049 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:52.049 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:52.050 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:37:52.053 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:37:52.054 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:37:52.048+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:37:52.152 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:37:52.146+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:37:52.169 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:37:52.169 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:37:52.169 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:37:52.169 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:37:52.179 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:37:52.253 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:37:52.246+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:37:52.283 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:37:52.283 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:37:52.283 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:37:52.283 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:37:52.283 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:37:52.292 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:37:52.354 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:37:52.352+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:37:52.402 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:37:52.402 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:37:52.402 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:37:52.402 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:37:52.402 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.403 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:52.403 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.573328+0000", 2024-09-18T04:37:52.403 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 24.823258275000001, 2024-09-18T04:37:52.403 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011397898, 2024-09-18T04:37:52.403 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:52.403 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:52.403 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:52.403 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:52.403 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:52.403 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 86 2024-09-18T04:37:52.404 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.404 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:52.404 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.404 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:52.404 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573328+0000", 2024-09-18T04:37:52.404 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:52.404 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.404 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.404 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:52.404 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573328+0000", 2024-09-18T04:37:52.404 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:52.405 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.405 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.405 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:52.405 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573327+0000", 2024-09-18T04:37:52.405 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:37:52.405 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.405 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.405 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:52.405 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573331+0000", 2024-09-18T04:37:52.405 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9750000000000001e-06 2024-09-18T04:37:52.405 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.405 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.406 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:52.406 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573346+0000", 2024-09-18T04:37:52.406 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5469999999999999e-05 2024-09-18T04:37:52.406 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.406 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.407 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:52.407 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573370+0000", 2024-09-18T04:37:52.407 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3969999999999999e-05 2024-09-18T04:37:52.407 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.407 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.407 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:52.407 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.578536+0000", 2024-09-18T04:37:52.407 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0051659679999999999 2024-09-18T04:37:52.407 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.407 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.407 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:52.407 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.578689+0000", 2024-09-18T04:37:52.408 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015276299999999999 2024-09-18T04:37:52.408 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.408 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.408 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:52.408 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.578901+0000", 2024-09-18T04:37:52.408 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000211783 2024-09-18T04:37:52.408 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.408 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.408 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:52.408 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.583408+0000", 2024-09-18T04:37:52.408 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045071249999999998 2024-09-18T04:37:52.408 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.409 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.409 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:52.409 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.584645+0000", 2024-09-18T04:37:52.409 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001237007 2024-09-18T04:37:52.409 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.409 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.409 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:52.409 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.584703+0000", 2024-09-18T04:37:52.409 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.817e-05 2024-09-18T04:37:52.409 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.409 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.409 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:52.409 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.584725+0000", 2024-09-18T04:37:52.409 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2155e-05 2024-09-18T04:37:52.410 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:52.410 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:52.410 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:52.410 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.410 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.410 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:52.410 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.573382+0000", 2024-09-18T04:37:52.410 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 24.823203527, 2024-09-18T04:37:52.410 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012154974000000001, 2024-09-18T04:37:52.410 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:52.410 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:52.410 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:52.410 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:52.411 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:52.411 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 87 2024-09-18T04:37:52.411 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.411 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:52.411 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.411 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:52.411 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573382+0000", 2024-09-18T04:37:52.411 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:52.411 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.411 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.411 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:52.411 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573382+0000", 2024-09-18T04:37:52.412 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:52.412 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.412 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.412 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:52.412 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573382+0000", 2024-09-18T04:37:52.412 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:37:52.412 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.412 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.412 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:52.412 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573482+0000", 2024-09-18T04:37:52.412 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000100333 2024-09-18T04:37:52.412 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.412 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.413 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:52.413 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573492+0000", 2024-09-18T04:37:52.413 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5389999999999996e-06 2024-09-18T04:37:52.413 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.413 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.413 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:52.413 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573500+0000", 2024-09-18T04:37:52.413 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8929999999999994e-06 2024-09-18T04:37:52.413 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.413 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.413 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:52.414 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.579797+0000", 2024-09-18T04:37:52.414 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0062963239999999998 2024-09-18T04:37:52.414 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.414 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.414 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:52.414 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.579957+0000", 2024-09-18T04:37:52.414 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000160283 2024-09-18T04:37:52.414 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.414 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.414 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:52.415 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.580178+0000", 2024-09-18T04:37:52.415 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00022079200000000001 2024-09-18T04:37:52.415 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.415 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.415 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:52.415 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.583479+0000", 2024-09-18T04:37:52.416 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033007689999999998 2024-09-18T04:37:52.416 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.416 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.416 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:52.416 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585477+0000", 2024-09-18T04:37:52.416 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0019985670000000001 2024-09-18T04:37:52.416 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.416 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.416 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:52.416 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585517+0000", 2024-09-18T04:37:52.417 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9709999999999998e-05 2024-09-18T04:37:52.417 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.417 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.417 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:52.417 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585537+0000", 2024-09-18T04:37:52.417 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0326000000000002e-05 2024-09-18T04:37:52.417 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:52.417 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:52.418 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:52.418 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.418 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.418 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:52.418 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.573508+0000", 2024-09-18T04:37:52.418 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 24.823077820999998, 2024-09-18T04:37:52.418 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012068476, 2024-09-18T04:37:52.418 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:52.419 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:52.419 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:52.419 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:52.419 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:52.419 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 88 2024-09-18T04:37:52.419 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.419 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:52.419 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.419 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:52.419 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573508+0000", 2024-09-18T04:37:52.419 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:52.419 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.419 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.419 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:52.420 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573508+0000", 2024-09-18T04:37:52.420 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:52.420 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.420 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.420 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:52.420 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573507+0000", 2024-09-18T04:37:52.420 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:37:52.420 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.420 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.420 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:52.420 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573512+0000", 2024-09-18T04:37:52.420 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.2629999999999997e-06 2024-09-18T04:37:52.420 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.421 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.421 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:52.421 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573530+0000", 2024-09-18T04:37:52.421 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8507000000000001e-05 2024-09-18T04:37:52.421 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.421 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.421 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:52.421 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573537+0000", 2024-09-18T04:37:52.421 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.832e-06 2024-09-18T04:37:52.421 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.421 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.421 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:52.421 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.582486+0000", 2024-09-18T04:37:52.422 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0089494770000000008 2024-09-18T04:37:52.422 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.422 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.422 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:52.422 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.582630+0000", 2024-09-18T04:37:52.422 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000144005 2024-09-18T04:37:52.423 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.423 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.423 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:52.423 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.582842+0000", 2024-09-18T04:37:52.423 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00021189200000000001 2024-09-18T04:37:52.423 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.424 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.424 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:52.424 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.584777+0000", 2024-09-18T04:37:52.424 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001934753 2024-09-18T04:37:52.424 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.424 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.425 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:52.425 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585543+0000", 2024-09-18T04:37:52.425 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00076540099999999997 2024-09-18T04:37:52.425 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.425 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.425 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:52.425 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585561+0000", 2024-09-18T04:37:52.426 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8474999999999998e-05 2024-09-18T04:37:52.426 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.426 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.426 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:52.426 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585576+0000", 2024-09-18T04:37:52.426 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5427e-05 2024-09-18T04:37:52.427 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:52.427 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:52.427 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:52.427 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.427 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.427 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:52.427 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786394+0000", 2024-09-18T04:37:52.427 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 24.610191373999999, 2024-09-18T04:37:52.427 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011902085999999999, 2024-09-18T04:37:52.427 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:52.427 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:52.427 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:52.427 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:52.428 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:52.428 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 109 2024-09-18T04:37:52.428 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.428 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:52.428 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.428 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:52.428 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786394+0000", 2024-09-18T04:37:52.428 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:52.428 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.428 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.428 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:52.429 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786394+0000", 2024-09-18T04:37:52.429 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:52.429 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.429 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.429 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:52.429 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786394+0000", 2024-09-18T04:37:52.429 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:37:52.429 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.429 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.429 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:52.429 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786548+0000", 2024-09-18T04:37:52.429 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015463400000000001 2024-09-18T04:37:52.429 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.429 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.430 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:52.430 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786561+0000", 2024-09-18T04:37:52.430 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3093e-05 2024-09-18T04:37:52.430 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.430 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.430 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:52.430 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786571+0000", 2024-09-18T04:37:52.430 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4700000000000008e-06 2024-09-18T04:37:52.430 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.430 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.430 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:52.430 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.793367+0000", 2024-09-18T04:37:52.430 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0067966600000000004 2024-09-18T04:37:52.430 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.431 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.431 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:52.431 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.793526+0000", 2024-09-18T04:37:52.431 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015877 2024-09-18T04:37:52.431 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.431 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.431 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:52.431 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.793752+0000", 2024-09-18T04:37:52.431 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00022555200000000001 2024-09-18T04:37:52.431 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.431 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.431 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:52.431 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798125+0000", 2024-09-18T04:37:52.431 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0043733349999999999 2024-09-18T04:37:52.432 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.432 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.432 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:52.432 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798146+0000", 2024-09-18T04:37:52.432 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1063000000000001e-05 2024-09-18T04:37:52.432 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.432 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.432 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:52.432 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798232+0000", 2024-09-18T04:37:52.432 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5771000000000003e-05 2024-09-18T04:37:52.432 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.432 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.432 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:52.433 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798297+0000", 2024-09-18T04:37:52.433 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.4624999999999999e-05 2024-09-18T04:37:52.433 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:52.433 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:52.433 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:52.433 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.433 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.433 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:52.433 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786584+0000", 2024-09-18T04:37:52.433 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 24.610001295, 2024-09-18T04:37:52.433 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012013037000000001, 2024-09-18T04:37:52.433 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:52.433 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:52.434 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:52.434 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:52.434 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:52.434 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 110 2024-09-18T04:37:52.434 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.434 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:52.434 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.434 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:52.434 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786584+0000", 2024-09-18T04:37:52.434 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:52.434 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.434 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.434 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:52.434 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786584+0000", 2024-09-18T04:37:52.434 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:52.435 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.435 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.435 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:52.435 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786584+0000", 2024-09-18T04:37:52.435 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:37:52.435 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.435 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.435 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:52.435 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786590+0000", 2024-09-18T04:37:52.435 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6359999999999997e-06 2024-09-18T04:37:52.435 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.435 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.435 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:52.436 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786599+0000", 2024-09-18T04:37:52.436 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7369999999999995e-06 2024-09-18T04:37:52.436 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.436 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.436 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:52.436 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786606+0000", 2024-09-18T04:37:52.436 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1930000000000004e-06 2024-09-18T04:37:52.436 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.436 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.437 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:52.437 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.795018+0000", 2024-09-18T04:37:52.437 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0084115660000000005 2024-09-18T04:37:52.437 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.437 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.438 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:52.438 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.795184+0000", 2024-09-18T04:37:52.438 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000165833 2024-09-18T04:37:52.438 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.438 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.438 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:52.438 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.795414+0000", 2024-09-18T04:37:52.439 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00023088399999999999 2024-09-18T04:37:52.439 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.439 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.439 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:52.439 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798321+0000", 2024-09-18T04:37:52.439 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0029069539999999998 2024-09-18T04:37:52.439 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.439 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.439 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:52.439 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798404+0000", 2024-09-18T04:37:52.439 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2724000000000003e-05 2024-09-18T04:37:52.439 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.439 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.440 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:52.440 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798488+0000", 2024-09-18T04:37:52.440 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.3505999999999996e-05 2024-09-18T04:37:52.440 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.440 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.440 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:52.440 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798598+0000", 2024-09-18T04:37:52.440 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000109917 2024-09-18T04:37:52.440 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:52.440 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:52.440 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:52.440 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.440 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.440 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:52.441 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786613+0000", 2024-09-18T04:37:52.441 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 24.609972594999999, 2024-09-18T04:37:52.441 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014245760999999999, 2024-09-18T04:37:52.441 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:52.441 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:52.441 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:52.441 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:52.441 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:52.442 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 111 2024-09-18T04:37:52.442 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.442 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:52.442 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.442 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:52.442 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786613+0000", 2024-09-18T04:37:52.442 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:52.442 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.442 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.442 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:52.443 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786613+0000", 2024-09-18T04:37:52.443 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:52.443 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.443 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.443 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:52.443 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786612+0000", 2024-09-18T04:37:52.444 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:37:52.444 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.444 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.444 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:52.444 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786622+0000", 2024-09-18T04:37:52.444 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4099999999999997e-06 2024-09-18T04:37:52.445 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.445 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.445 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:52.445 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786630+0000", 2024-09-18T04:37:52.445 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4169999999999992e-06 2024-09-18T04:37:52.445 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.445 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.445 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:52.445 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786637+0000", 2024-09-18T04:37:52.445 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8290000000000002e-06 2024-09-18T04:37:52.445 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.446 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.446 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:52.446 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.797099+0000", 2024-09-18T04:37:52.446 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010461747 2024-09-18T04:37:52.446 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.446 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.447 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:52.447 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.797243+0000", 2024-09-18T04:37:52.447 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014388899999999999 2024-09-18T04:37:52.447 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.447 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.447 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:52.447 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.797463+0000", 2024-09-18T04:37:52.447 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000219912 2024-09-18T04:37:52.447 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.447 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.447 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:52.447 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800732+0000", 2024-09-18T04:37:52.447 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0032688719999999999 2024-09-18T04:37:52.448 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.448 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.448 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:52.448 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800751+0000", 2024-09-18T04:37:52.448 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9633999999999999e-05 2024-09-18T04:37:52.448 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.448 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.448 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:52.448 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800824+0000", 2024-09-18T04:37:52.448 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.2805999999999994e-05 2024-09-18T04:37:52.448 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.449 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.449 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:52.449 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800859+0000", 2024-09-18T04:37:52.449 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4952e-05 2024-09-18T04:37:52.449 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:52.449 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:52.449 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:52.449 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.449 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.449 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:52.449 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786649+0000", 2024-09-18T04:37:52.449 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 24.609937186, 2024-09-18T04:37:52.449 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.01613442, 2024-09-18T04:37:52.450 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:52.450 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:52.450 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:52.450 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:52.450 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:52.450 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 112 2024-09-18T04:37:52.450 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.450 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:52.450 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.450 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:52.450 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786649+0000", 2024-09-18T04:37:52.450 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:52.451 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.451 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.451 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:52.451 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786649+0000", 2024-09-18T04:37:52.451 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:52.451 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.451 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.451 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:52.451 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786648+0000", 2024-09-18T04:37:52.451 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:37:52.451 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.451 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.451 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:52.451 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786768+0000", 2024-09-18T04:37:52.452 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000120491 2024-09-18T04:37:52.452 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.452 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.452 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:52.452 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786795+0000", 2024-09-18T04:37:52.452 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6556000000000001e-05 2024-09-18T04:37:52.452 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.452 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.452 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:52.453 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786807+0000", 2024-09-18T04:37:52.453 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1739999999999999e-05 2024-09-18T04:37:52.453 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.453 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.453 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:52.453 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798683+0000", 2024-09-18T04:37:52.453 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011876070000000001 2024-09-18T04:37:52.453 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.453 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.453 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:52.453 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798867+0000", 2024-09-18T04:37:52.453 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000184103 2024-09-18T04:37:52.453 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.454 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.454 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:52.454 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.799082+0000", 2024-09-18T04:37:52.454 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00021476000000000001 2024-09-18T04:37:52.454 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.454 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.454 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:52.454 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802610+0000", 2024-09-18T04:37:52.454 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0035282209999999998 2024-09-18T04:37:52.455 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.455 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.455 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:52.455 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802706+0000", 2024-09-18T04:37:52.455 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5922999999999996e-05 2024-09-18T04:37:52.455 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.455 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.455 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:52.455 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802750+0000", 2024-09-18T04:37:52.456 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4511000000000001e-05 2024-09-18T04:37:52.456 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.456 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.456 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:52.456 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802783+0000", 2024-09-18T04:37:52.456 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2752000000000001e-05 2024-09-18T04:37:52.456 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:52.456 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:52.456 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:52.456 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.456 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.456 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:52.457 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786816+0000", 2024-09-18T04:37:52.457 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 24.609770119, 2024-09-18T04:37:52.457 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016197989999999999, 2024-09-18T04:37:52.457 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:52.457 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:52.457 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:52.457 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:52.457 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:52.457 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 113 2024-09-18T04:37:52.457 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.457 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:52.457 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.457 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:52.458 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786816+0000", 2024-09-18T04:37:52.458 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:52.458 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.458 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.458 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:52.458 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786816+0000", 2024-09-18T04:37:52.458 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:52.458 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.458 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.458 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:52.458 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786815+0000", 2024-09-18T04:37:52.458 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:37:52.458 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.459 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.459 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:52.459 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786821+0000", 2024-09-18T04:37:52.459 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9529999999999996e-06 2024-09-18T04:37:52.459 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.459 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.459 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:52.459 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786841+0000", 2024-09-18T04:37:52.459 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9743000000000001e-05 2024-09-18T04:37:52.459 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.459 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.459 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:52.459 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786848+0000", 2024-09-18T04:37:52.459 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4900000000000003e-06 2024-09-18T04:37:52.460 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.460 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.460 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:52.460 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.799747+0000", 2024-09-18T04:37:52.460 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012898902 2024-09-18T04:37:52.460 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.460 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.460 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:52.460 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.799898+0000", 2024-09-18T04:37:52.460 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000150877 2024-09-18T04:37:52.460 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.460 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.461 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:52.461 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800095+0000", 2024-09-18T04:37:52.461 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000197023 2024-09-18T04:37:52.461 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.461 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.461 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:52.461 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802829+0000", 2024-09-18T04:37:52.461 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0027343480000000002 2024-09-18T04:37:52.462 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.462 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.462 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:52.462 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802847+0000", 2024-09-18T04:37:52.462 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7283e-05 2024-09-18T04:37:52.462 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.462 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.462 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:52.462 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802974+0000", 2024-09-18T04:37:52.462 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012749199999999999 2024-09-18T04:37:52.462 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.463 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.463 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:52.463 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803014+0000", 2024-09-18T04:37:52.463 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9576000000000002e-05 2024-09-18T04:37:52.463 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:52.463 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:52.463 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:52.463 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.463 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.463 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:52.464 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:37:52.464 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 24.609684527999999, 2024-09-18T04:37:52.464 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017399218000000001, 2024-09-18T04:37:52.464 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:52.464 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:52.464 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:52.464 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:52.464 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:52.464 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 114 2024-09-18T04:37:52.464 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.464 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:52.464 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.464 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:52.465 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:37:52.465 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:52.465 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.465 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.465 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:52.465 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:37:52.465 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:52.465 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.465 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.465 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:52.465 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786900+0000", 2024-09-18T04:37:52.465 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:37:52.465 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.466 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.466 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:52.466 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786908+0000", 2024-09-18T04:37:52.466 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8199999999999997e-06 2024-09-18T04:37:52.466 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.466 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.466 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:52.466 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786925+0000", 2024-09-18T04:37:52.466 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6849000000000001e-05 2024-09-18T04:37:52.466 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.466 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.466 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:52.466 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786934+0000", 2024-09-18T04:37:52.467 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6299999999999993e-06 2024-09-18T04:37:52.467 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.467 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.467 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:52.467 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800926+0000", 2024-09-18T04:37:52.467 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013991632 2024-09-18T04:37:52.467 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.467 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.467 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:52.467 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801060+0000", 2024-09-18T04:37:52.467 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013378199999999999 2024-09-18T04:37:52.468 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.468 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.468 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:52.468 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801166+0000", 2024-09-18T04:37:52.468 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010607099999999999 2024-09-18T04:37:52.468 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.468 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.468 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:52.468 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803026+0000", 2024-09-18T04:37:52.468 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001860469 2024-09-18T04:37:52.468 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.468 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.469 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:52.469 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804245+0000", 2024-09-18T04:37:52.469 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0012185900000000001 2024-09-18T04:37:52.469 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.469 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.469 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:52.469 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804278+0000", 2024-09-18T04:37:52.469 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3272000000000003e-05 2024-09-18T04:37:52.469 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.469 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.469 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:52.469 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804300+0000", 2024-09-18T04:37:52.469 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2266e-05 2024-09-18T04:37:52.469 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:52.470 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:52.470 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:52.470 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.470 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.470 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:52.470 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787566+0000", 2024-09-18T04:37:52.470 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 24.609019569000001, 2024-09-18T04:37:52.470 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017124120999999999, 2024-09-18T04:37:52.470 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:52.470 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:52.470 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:52.470 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:52.470 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:52.470 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 115 2024-09-18T04:37:52.471 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.471 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:52.471 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.471 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:52.471 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787566+0000", 2024-09-18T04:37:52.471 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:52.471 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.472 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.472 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:52.472 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787566+0000", 2024-09-18T04:37:52.472 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:52.472 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.472 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.472 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:52.472 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787564+0000", 2024-09-18T04:37:52.472 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:37:52.472 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.472 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.473 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:52.473 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787751+0000", 2024-09-18T04:37:52.473 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000186873 2024-09-18T04:37:52.473 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.473 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.474 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:52.474 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787772+0000", 2024-09-18T04:37:52.474 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.126e-05 2024-09-18T04:37:52.474 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.474 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.474 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:52.474 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787794+0000", 2024-09-18T04:37:52.475 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1418999999999999e-05 2024-09-18T04:37:52.475 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.475 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.475 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:52.475 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801659+0000", 2024-09-18T04:37:52.475 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013865125000000001 2024-09-18T04:37:52.475 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.475 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.475 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:52.475 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801750+0000", 2024-09-18T04:37:52.475 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.1427000000000004e-05 2024-09-18T04:37:52.475 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.475 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.476 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:52.476 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801894+0000", 2024-09-18T04:37:52.476 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000143514 2024-09-18T04:37:52.476 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.476 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.476 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:52.476 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804313+0000", 2024-09-18T04:37:52.476 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0024190879999999998 2024-09-18T04:37:52.476 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.476 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.476 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:52.476 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804644+0000", 2024-09-18T04:37:52.476 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00033100799999999999 2024-09-18T04:37:52.477 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.477 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.477 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:52.477 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804672+0000", 2024-09-18T04:37:52.477 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7988e-05 2024-09-18T04:37:52.477 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.477 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.477 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:52.477 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804690+0000", 2024-09-18T04:37:52.477 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8298999999999999e-05 2024-09-18T04:37:52.477 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:52.477 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:52.477 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:52.478 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.478 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.478 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:52.478 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:37:52.478 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 24.60877163, 2024-09-18T04:37:52.478 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017521662, 2024-09-18T04:37:52.478 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:52.478 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:52.478 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:52.478 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:52.478 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:52.478 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 116 2024-09-18T04:37:52.478 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.478 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:52.479 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.479 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:52.479 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:37:52.479 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:52.479 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.479 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.479 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:52.479 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:37:52.479 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:52.479 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.479 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.480 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:52.480 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787813+0000", 2024-09-18T04:37:52.480 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:37:52.480 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.480 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.480 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:52.480 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787824+0000", 2024-09-18T04:37:52.480 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1157e-05 2024-09-18T04:37:52.480 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.480 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.480 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:52.480 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787840+0000", 2024-09-18T04:37:52.480 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6269e-05 2024-09-18T04:37:52.481 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.481 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.481 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:52.481 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787853+0000", 2024-09-18T04:37:52.481 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3156000000000001e-05 2024-09-18T04:37:52.481 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.481 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.481 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:52.481 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803110+0000", 2024-09-18T04:37:52.481 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015256736 2024-09-18T04:37:52.481 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.481 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.481 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:52.482 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803197+0000", 2024-09-18T04:37:52.482 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7195999999999997e-05 2024-09-18T04:37:52.482 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.482 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.482 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:52.482 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803308+0000", 2024-09-18T04:37:52.482 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111005 2024-09-18T04:37:52.482 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.482 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.482 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:52.482 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804316+0000", 2024-09-18T04:37:52.482 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010073949999999999 2024-09-18T04:37:52.482 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.483 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.483 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:52.483 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805232+0000", 2024-09-18T04:37:52.483 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091636300000000001 2024-09-18T04:37:52.483 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.483 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.483 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:52.483 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805277+0000", 2024-09-18T04:37:52.483 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4864000000000003e-05 2024-09-18T04:37:52.483 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.483 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.483 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:52.483 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805336+0000", 2024-09-18T04:37:52.484 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8968000000000002e-05 2024-09-18T04:37:52.484 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:52.484 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:52.484 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:52.484 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.484 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.484 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:52.484 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:37:52.484 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 24.608719574999999, 2024-09-18T04:37:52.484 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017983453, 2024-09-18T04:37:52.484 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:52.484 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:52.485 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:52.485 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:52.485 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:52.485 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 117 2024-09-18T04:37:52.485 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.485 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:52.485 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.485 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:52.485 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:37:52.485 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:52.485 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.485 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.485 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:52.486 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:37:52.486 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:52.486 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.486 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.486 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:52.486 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787865+0000", 2024-09-18T04:37:52.486 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:37:52.486 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.486 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.486 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:52.486 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787889+0000", 2024-09-18T04:37:52.486 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4244999999999999e-05 2024-09-18T04:37:52.486 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.486 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.487 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:52.487 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787905+0000", 2024-09-18T04:37:52.487 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5739999999999998e-05 2024-09-18T04:37:52.487 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.487 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.487 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:52.487 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787918+0000", 2024-09-18T04:37:52.487 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.252e-05 2024-09-18T04:37:52.487 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.487 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.487 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:52.487 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803693+0000", 2024-09-18T04:37:52.487 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015774872999999998 2024-09-18T04:37:52.488 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.488 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.488 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:52.488 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803775+0000", 2024-09-18T04:37:52.488 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2761000000000003e-05 2024-09-18T04:37:52.488 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.488 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.488 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:52.488 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803887+0000", 2024-09-18T04:37:52.488 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111708 2024-09-18T04:37:52.488 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.488 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.489 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:52.489 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804823+0000", 2024-09-18T04:37:52.489 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00093625100000000003 2024-09-18T04:37:52.489 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.489 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.489 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:52.489 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805804+0000", 2024-09-18T04:37:52.489 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00098114699999999992 2024-09-18T04:37:52.489 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.489 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.489 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:52.489 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805831+0000", 2024-09-18T04:37:52.490 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6401999999999999e-05 2024-09-18T04:37:52.490 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.490 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.490 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:52.490 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805850+0000", 2024-09-18T04:37:52.490 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8862e-05 2024-09-18T04:37:52.490 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:52.490 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:52.490 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:52.490 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.490 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.490 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:52.490 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.230830+0000", 2024-09-18T04:37:52.490 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 24.165755696000002, 2024-09-18T04:37:52.491 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011537146999999999, 2024-09-18T04:37:52.491 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:52.491 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:52.491 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:52.491 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:52.491 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:52.491 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 135 2024-09-18T04:37:52.491 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.491 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:52.491 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.491 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:52.491 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230830+0000", 2024-09-18T04:37:52.491 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:52.491 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.492 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.492 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:52.492 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230830+0000", 2024-09-18T04:37:52.492 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:52.492 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.492 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.492 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:52.492 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230829+0000", 2024-09-18T04:37:52.492 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:37:52.492 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.492 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.492 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:52.493 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230835+0000", 2024-09-18T04:37:52.493 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8660000000000002e-06 2024-09-18T04:37:52.493 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.493 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.493 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:52.493 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230844+0000", 2024-09-18T04:37:52.493 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9339999999999995e-06 2024-09-18T04:37:52.493 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.493 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.493 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:52.493 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230853+0000", 2024-09-18T04:37:52.494 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4419999999999998e-06 2024-09-18T04:37:52.494 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.494 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.494 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:52.494 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.237658+0000", 2024-09-18T04:37:52.494 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0068055310000000001 2024-09-18T04:37:52.494 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.494 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.494 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:52.494 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.237837+0000", 2024-09-18T04:37:52.494 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017943200000000001 2024-09-18T04:37:52.494 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.494 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.494 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:52.495 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.238112+0000", 2024-09-18T04:37:52.495 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00027488600000000001 2024-09-18T04:37:52.495 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.495 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.495 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:52.495 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.240505+0000", 2024-09-18T04:37:52.495 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0023930539999999999 2024-09-18T04:37:52.495 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.495 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.495 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:52.495 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242157+0000", 2024-09-18T04:37:52.495 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001651135 2024-09-18T04:37:52.495 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.495 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.496 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:52.496 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242281+0000", 2024-09-18T04:37:52.496 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012451300000000001 2024-09-18T04:37:52.496 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.496 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.496 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:52.496 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242367+0000", 2024-09-18T04:37:52.496 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6178000000000001e-05 2024-09-18T04:37:52.496 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:52.496 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:52.496 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:52.496 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.496 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.497 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:52.497 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.230858+0000", 2024-09-18T04:37:52.497 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 24.165727808, 2024-09-18T04:37:52.497 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013005821000000001, 2024-09-18T04:37:52.497 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:52.497 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:52.497 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:52.497 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:52.497 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:52.497 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 136 2024-09-18T04:37:52.497 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.497 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:52.497 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.498 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:52.498 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230858+0000", 2024-09-18T04:37:52.498 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:52.498 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.498 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.498 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:52.498 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230858+0000", 2024-09-18T04:37:52.498 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:52.498 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.498 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.498 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:52.498 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230858+0000", 2024-09-18T04:37:52.498 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:37:52.499 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.499 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.499 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:52.499 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230861+0000", 2024-09-18T04:37:52.499 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7459999999999998e-06 2024-09-18T04:37:52.499 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.499 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.499 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:52.499 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230877+0000", 2024-09-18T04:37:52.499 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5979000000000001e-05 2024-09-18T04:37:52.499 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.499 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.499 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:52.499 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230885+0000", 2024-09-18T04:37:52.500 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.5979999999999999e-06 2024-09-18T04:37:52.500 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.500 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.500 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:52.500 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.239023+0000", 2024-09-18T04:37:52.500 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0081380710000000002 2024-09-18T04:37:52.500 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.500 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.500 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:52.500 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.239216+0000", 2024-09-18T04:37:52.501 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019274799999999999 2024-09-18T04:37:52.501 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.501 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.501 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:52.501 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.239417+0000", 2024-09-18T04:37:52.501 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020093300000000001 2024-09-18T04:37:52.501 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.501 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.501 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:52.501 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242494+0000", 2024-09-18T04:37:52.501 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003077177 2024-09-18T04:37:52.501 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.501 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.502 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:52.502 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.243668+0000", 2024-09-18T04:37:52.502 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001174027 2024-09-18T04:37:52.502 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.502 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.502 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:52.502 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.243777+0000", 2024-09-18T04:37:52.502 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010955600000000001 2024-09-18T04:37:52.502 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.502 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.502 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:52.502 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.243864+0000", 2024-09-18T04:37:52.503 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6441999999999997e-05 2024-09-18T04:37:52.503 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:52.503 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:52.503 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:52.503 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.503 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.503 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:52.503 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.230901+0000", 2024-09-18T04:37:52.503 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 24.165685235000002, 2024-09-18T04:37:52.503 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014120882, 2024-09-18T04:37:52.504 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:52.504 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:52.504 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:52.504 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:52.504 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:52.504 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 137 2024-09-18T04:37:52.504 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.504 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:52.504 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.505 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:52.505 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230901+0000", 2024-09-18T04:37:52.505 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:52.505 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.505 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.505 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:52.505 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230901+0000", 2024-09-18T04:37:52.505 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:52.505 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.505 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.505 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:52.505 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230900+0000", 2024-09-18T04:37:52.505 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:37:52.506 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.506 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.506 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:52.506 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231002+0000", 2024-09-18T04:37:52.506 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000102392 2024-09-18T04:37:52.506 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.506 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.506 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:52.506 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231029+0000", 2024-09-18T04:37:52.506 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7166999999999999e-05 2024-09-18T04:37:52.506 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.506 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.506 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:52.507 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231036+0000", 2024-09-18T04:37:52.507 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7259999999999996e-06 2024-09-18T04:37:52.507 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.507 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.507 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:52.507 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.240562+0000", 2024-09-18T04:37:52.507 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0095257970000000008 2024-09-18T04:37:52.507 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.507 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.507 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:52.507 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.240730+0000", 2024-09-18T04:37:52.507 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016767099999999999 2024-09-18T04:37:52.508 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.508 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.508 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:52.508 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.240927+0000", 2024-09-18T04:37:52.508 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019711899999999999 2024-09-18T04:37:52.508 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.508 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.508 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:52.508 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244885+0000", 2024-09-18T04:37:52.508 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0039577340000000001 2024-09-18T04:37:52.508 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.508 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.509 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:52.509 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244899+0000", 2024-09-18T04:37:52.509 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4403000000000001e-05 2024-09-18T04:37:52.509 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.509 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.509 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:52.509 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244973+0000", 2024-09-18T04:37:52.509 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4177999999999994e-05 2024-09-18T04:37:52.509 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.509 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.510 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:52.510 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.245021+0000", 2024-09-18T04:37:52.510 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8344999999999999e-05 2024-09-18T04:37:52.510 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:52.510 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:52.510 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:52.510 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.510 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.510 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:52.510 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:37:52.510 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 24.165542444, 2024-09-18T04:37:52.510 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016528505999999998, 2024-09-18T04:37:52.510 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:52.510 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:52.511 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:52.511 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:52.511 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:52.511 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 138 2024-09-18T04:37:52.511 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.511 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:52.511 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.511 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:52.511 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:37:52.511 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:52.511 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.511 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.511 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:52.511 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:37:52.512 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:52.512 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.512 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.512 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:52.512 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:37:52.512 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:37:52.512 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.512 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.512 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:52.512 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231047+0000", 2024-09-18T04:37:52.512 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5399999999999997e-06 2024-09-18T04:37:52.513 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.513 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.513 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:52.513 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231068+0000", 2024-09-18T04:37:52.513 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0769000000000001e-05 2024-09-18T04:37:52.513 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.513 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.513 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:52.513 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231074+0000", 2024-09-18T04:37:52.513 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6180000000000003e-06 2024-09-18T04:37:52.513 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.513 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.513 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:52.514 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242556+0000", 2024-09-18T04:37:52.514 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011482654 2024-09-18T04:37:52.514 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.514 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.514 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:52.514 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242666+0000", 2024-09-18T04:37:52.514 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011003200000000001 2024-09-18T04:37:52.514 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.514 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.514 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:52.514 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242924+0000", 2024-09-18T04:37:52.514 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00025785899999999999 2024-09-18T04:37:52.514 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.515 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.515 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:52.515 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247263+0000", 2024-09-18T04:37:52.515 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0043387649999999996 2024-09-18T04:37:52.515 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.515 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.515 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:52.515 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247412+0000", 2024-09-18T04:37:52.515 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000149276 2024-09-18T04:37:52.515 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.515 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.515 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:52.515 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247522+0000", 2024-09-18T04:37:52.515 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000109788 2024-09-18T04:37:52.515 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.516 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.516 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:52.516 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247572+0000", 2024-09-18T04:37:52.516 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9808999999999997e-05 2024-09-18T04:37:52.516 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:52.516 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:52.516 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:52.516 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.516 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.516 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:52.516 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231083+0000", 2024-09-18T04:37:52.516 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 24.165502980999999, 2024-09-18T04:37:52.516 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0166731, 2024-09-18T04:37:52.517 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:52.517 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:52.517 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:52.517 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:52.517 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:52.517 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 139 2024-09-18T04:37:52.517 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.517 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:52.517 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.517 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:52.517 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231083+0000", 2024-09-18T04:37:52.517 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:52.517 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.517 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.518 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:52.518 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231083+0000", 2024-09-18T04:37:52.518 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:52.518 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.518 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.518 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:52.518 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231082+0000", 2024-09-18T04:37:52.518 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:37:52.518 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.518 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.518 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:52.518 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231090+0000", 2024-09-18T04:37:52.518 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1280000000000008e-06 2024-09-18T04:37:52.519 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.519 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.519 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:52.519 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231097+0000", 2024-09-18T04:37:52.519 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5119999999999997e-06 2024-09-18T04:37:52.519 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.519 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.519 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:52.519 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231106+0000", 2024-09-18T04:37:52.519 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9230000000000008e-06 2024-09-18T04:37:52.519 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.519 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.519 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:52.520 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.243977+0000", 2024-09-18T04:37:52.520 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012870754 2024-09-18T04:37:52.520 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.520 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.520 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:52.520 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244112+0000", 2024-09-18T04:37:52.520 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000135575 2024-09-18T04:37:52.520 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.520 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.520 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:52.520 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244265+0000", 2024-09-18T04:37:52.520 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015279900000000001 2024-09-18T04:37:52.521 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.521 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.521 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:52.521 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247649+0000", 2024-09-18T04:37:52.521 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003383896 2024-09-18T04:37:52.521 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.521 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.521 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:52.521 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247656+0000", 2024-09-18T04:37:52.521 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8920000000000003e-06 2024-09-18T04:37:52.521 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.522 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.522 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:52.522 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247692+0000", 2024-09-18T04:37:52.522 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6470000000000001e-05 2024-09-18T04:37:52.522 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.522 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.522 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:52.522 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247756+0000", 2024-09-18T04:37:52.522 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3676999999999994e-05 2024-09-18T04:37:52.522 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:52.522 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:52.522 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:52.523 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.523 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.523 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:52.523 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231190+0000", 2024-09-18T04:37:52.523 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 24.165396054999999, 2024-09-18T04:37:52.523 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016793556000000001, 2024-09-18T04:37:52.523 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:52.523 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:52.523 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:52.523 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:52.523 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:52.523 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 140 2024-09-18T04:37:52.523 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.523 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:52.524 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.524 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:52.524 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231190+0000", 2024-09-18T04:37:52.524 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:52.524 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.525 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.525 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:52.525 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231190+0000", 2024-09-18T04:37:52.525 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:52.525 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.525 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.525 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:52.525 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231189+0000", 2024-09-18T04:37:52.525 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:37:52.525 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.525 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.525 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:52.525 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231273+0000", 2024-09-18T04:37:52.526 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4445999999999998e-05 2024-09-18T04:37:52.526 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.526 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.526 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:52.526 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231282+0000", 2024-09-18T04:37:52.526 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8859999999999993e-06 2024-09-18T04:37:52.526 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.526 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.526 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:52.526 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231288+0000", 2024-09-18T04:37:52.526 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9819999999999996e-06 2024-09-18T04:37:52.526 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.526 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.527 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:52.527 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.245119+0000", 2024-09-18T04:37:52.527 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.01383121 2024-09-18T04:37:52.527 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.527 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.527 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:52.527 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.245247+0000", 2024-09-18T04:37:52.527 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012763299999999999 2024-09-18T04:37:52.527 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.527 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.527 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:52.527 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.245401+0000", 2024-09-18T04:37:52.527 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015399200000000001 2024-09-18T04:37:52.527 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.528 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.528 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:52.528 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247766+0000", 2024-09-18T04:37:52.528 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0023649510000000001 2024-09-18T04:37:52.528 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.528 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.528 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:52.528 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247904+0000", 2024-09-18T04:37:52.528 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013836499999999999 2024-09-18T04:37:52.528 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.528 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.528 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:52.529 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247950+0000", 2024-09-18T04:37:52.529 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5201000000000003e-05 2024-09-18T04:37:52.529 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.529 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.529 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:52.529 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247983+0000", 2024-09-18T04:37:52.529 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3686000000000002e-05 2024-09-18T04:37:52.529 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:52.529 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:52.529 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:52.529 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.529 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.529 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:52.530 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:37:52.530 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 24.165285312000002, 2024-09-18T04:37:52.530 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017344024, 2024-09-18T04:37:52.530 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:52.530 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:52.530 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:52.530 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:52.530 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:52.530 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 141 2024-09-18T04:37:52.530 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.530 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:52.530 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.530 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:52.530 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:37:52.531 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:52.531 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.531 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.531 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:52.531 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:37:52.531 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:52.531 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.531 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.531 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:52.531 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:37:52.531 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:37:52.532 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.532 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.532 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:52.532 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231309+0000", 2024-09-18T04:37:52.532 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5419999999999994e-06 2024-09-18T04:37:52.532 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.532 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.532 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:52.532 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231323+0000", 2024-09-18T04:37:52.532 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3108000000000001e-05 2024-09-18T04:37:52.532 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.532 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.532 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:52.533 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231340+0000", 2024-09-18T04:37:52.533 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.696e-05 2024-09-18T04:37:52.533 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.533 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.533 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:52.533 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.246143+0000", 2024-09-18T04:37:52.533 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014803320999999999 2024-09-18T04:37:52.533 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.533 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.533 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:52.533 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.246323+0000", 2024-09-18T04:37:52.533 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00018061600000000001 2024-09-18T04:37:52.533 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.534 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.534 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:52.534 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.246530+0000", 2024-09-18T04:37:52.534 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020625 2024-09-18T04:37:52.534 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.534 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.534 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:52.534 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248002+0000", 2024-09-18T04:37:52.534 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0014719379999999999 2024-09-18T04:37:52.534 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.534 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.534 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:52.534 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248582+0000", 2024-09-18T04:37:52.535 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00058020599999999999 2024-09-18T04:37:52.535 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.535 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.535 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:52.535 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248618+0000", 2024-09-18T04:37:52.535 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6559e-05 2024-09-18T04:37:52.535 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.535 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.535 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:52.535 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248645+0000", 2024-09-18T04:37:52.535 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6080000000000001e-05 2024-09-18T04:37:52.535 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:52.535 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:52.535 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:52.536 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.536 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.536 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-18T04:37:52.536 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.878463+0000", 2024-09-18T04:37:52.536 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 23.518123278000001, 2024-09-18T04:37:52.536 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010882697, 2024-09-18T04:37:52.536 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:52.536 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:52.536 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:52.536 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:52.536 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:52.536 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 159 2024-09-18T04:37:52.536 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.537 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:52.537 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.537 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:52.537 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878463+0000", 2024-09-18T04:37:52.537 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:52.537 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.537 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.537 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:52.537 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878463+0000", 2024-09-18T04:37:52.537 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:52.537 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.537 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.537 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:52.538 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878462+0000", 2024-09-18T04:37:52.538 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:37:52.538 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.538 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.538 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:52.538 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878467+0000", 2024-09-18T04:37:52.538 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4099999999999999e-06 2024-09-18T04:37:52.538 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.538 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.538 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:52.538 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878490+0000", 2024-09-18T04:37:52.538 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3159999999999998e-05 2024-09-18T04:37:52.538 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.538 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.539 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:52.539 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878511+0000", 2024-09-18T04:37:52.539 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1092000000000001e-05 2024-09-18T04:37:52.539 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.539 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.539 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:52.539 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.887163+0000", 2024-09-18T04:37:52.539 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.008651845 2024-09-18T04:37:52.539 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.539 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.539 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:52.539 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.887273+0000", 2024-09-18T04:37:52.539 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000109823 2024-09-18T04:37:52.539 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.540 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.540 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:52.540 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.887441+0000", 2024-09-18T04:37:52.540 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016777599999999999 2024-09-18T04:37:52.540 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.540 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.540 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:52.540 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.888634+0000", 2024-09-18T04:37:52.540 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001192839 2024-09-18T04:37:52.540 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.540 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.540 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:52.540 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.889153+0000", 2024-09-18T04:37:52.541 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00051953700000000004 2024-09-18T04:37:52.541 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.541 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.541 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:52.541 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.889234+0000", 2024-09-18T04:37:52.541 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0802000000000003e-05 2024-09-18T04:37:52.541 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:52.541 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:52.541 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:52.541 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.889345+0000", 2024-09-18T04:37:52.541 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011112000000000001 2024-09-18T04:37:52.541 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:52.541 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:52.542 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:52.542 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:52.542 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:52.542 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:37:52.543 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:37:52.544 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:37:52.540+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:37:52.644 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:37:52.640+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:37:52.661 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:37:52.661 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:37:52.661 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:37:52.661 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:37:52.672 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:37:52.686 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-18T04:37:52.745 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:37:52.744+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:37:52.774 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:37:52.775 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:37:52.775 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:37:52.775 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:37:52.775 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:37:52.784 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:37:52.846 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:37:52.842+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:37:52.886 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:37:52.887 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:37:52.887 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:37:52.887 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [] 2024-09-18T04:37:52.887 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:37:52.896 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:37:52.916 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T04:37:52.947 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:37:52.942+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:37:53.048 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:37:53.042+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:37:53.057 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:37:53.058 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:37:53.059 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:37:53.059 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:37:53.059 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:37:53.069 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:37:53.083 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:37:53.102 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:37:53.125 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:37:53.148 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:37:53.149 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:37:53.142+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:37:53.171 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:37:53.174 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:37:53.174 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:37:53.174 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:37:53.174 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:37:53.174 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:37:53.174 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:37:53.218 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:53.218 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:53.228 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:37:53.250 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:37:53.242+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:37:53.338 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:53.338 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:53.348 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:37:53.350 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:37:53.348+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:37:53.450 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:37:53.446+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:37:53.468 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:53.468 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:53.479 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:37:53.551 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:37:53.546+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:37:53.585 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:53.586 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:53.595 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:37:53.652 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:37:53.648+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:37:53.688 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-09-18T04:37:53.703 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:53.704 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:53.714 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:37:53.757 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:37:53.756+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:37:53.854 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:37:53.850+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:37:53.869 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:53.869 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:53.879 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:37:53.955 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:37:53.950+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:37:53.988 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:53.989 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:53.999 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:37:54.057 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:37:54.052+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:37:54.106 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:54.106 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:54.116 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:37:54.157 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:37:54.150+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:37:54.229 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:54.229 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:54.239 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:37:54.258 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:37:54.254+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:37:54.350 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:54.350 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:54.359 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:37:54.356+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:37:54.360 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:37:54.460 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:37:54.456+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:37:54.468 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:54.468 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:54.479 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:37:54.561 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:37:54.556+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:37:54.595 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:54.595 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:54.604 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:37:54.661 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:37:54.654+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:37:54.714 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:54.714 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:54.724 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:37:54.763 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:37:54.758+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:37:54.826 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:54.826 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:54.837 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:37:54.864 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:37:54.860+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:37:54.964 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:37:54.958+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:37:54.986 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:54.987 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:54.996 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:37:55.065 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:37:55.058+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:37:55.102 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:55.103 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:55.112 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:37:55.166 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:37:55.164+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:37:55.187 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T04:37:55.218 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:55.219 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:55.228 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:37:55.267 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:37:55.262+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:37:55.336 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:55.337 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:55.341 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:37:55.347 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:37:55.365 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:37:55.367 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:37:55.364+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:37:55.391 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:37:55.414 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:37:55.436 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:37:55.457 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:37:55.459 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:37:55.459 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:37:55.459 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:37:55.459 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:37:55.459 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:37:55.459 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:37:55.468 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:37:55.464+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:37:55.503 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:37:55.503 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:37:55.503 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:37:55.503 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:37:55.503 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:37:55.513 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:37:55.569 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:37:55.564+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:37:55.624 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:37:55.624 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:37:55.624 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:37:55.624 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:37:55.624 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.624 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-18T04:37:55.624 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:37:55.624 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 21.150408550000002, 2024-09-18T04:37:55.624 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0095781189999999995, 2024-09-18T04:37:55.624 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:55.624 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:55.625 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:55.625 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:37:55.625 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:37:55.625 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 1 2024-09-18T04:37:55.625 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.625 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:55.625 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.625 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:55.625 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:37:55.625 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:55.625 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.625 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.626 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:55.626 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:37:55.626 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:55.626 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.626 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.626 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:55.626 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469862+0000", 2024-09-18T04:37:55.626 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:37:55.626 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.626 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.626 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:55.626 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469879+0000", 2024-09-18T04:37:55.626 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7431999999999998e-05 2024-09-18T04:37:55.626 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.627 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.627 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:55.627 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469896+0000", 2024-09-18T04:37:55.627 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6742999999999999e-05 2024-09-18T04:37:55.627 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.627 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.627 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:55.627 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469915+0000", 2024-09-18T04:37:55.627 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8991000000000002e-05 2024-09-18T04:37:55.627 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.627 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.627 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:55.627 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470397+0000", 2024-09-18T04:37:55.628 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048167799999999998 2024-09-18T04:37:55.628 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.628 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.628 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-09-18T04:37:55.628 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470412+0000", 2024-09-18T04:37:55.628 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5526e-05 2024-09-18T04:37:55.628 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.628 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.628 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:55.628 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473279+0000", 2024-09-18T04:37:55.628 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0028667469999999998 2024-09-18T04:37:55.628 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.628 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.628 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:55.629 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473441+0000", 2024-09-18T04:37:55.629 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000162082 2024-09-18T04:37:55.629 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.629 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.629 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:37:55.629 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473573+0000", 2024-09-18T04:37:55.629 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000132032 2024-09-18T04:37:55.629 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.629 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.629 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:55.629 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474826+0000", 2024-09-18T04:37:55.629 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001252717 2024-09-18T04:37:55.629 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.630 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.630 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:55.630 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479307+0000", 2024-09-18T04:37:55.630 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044813220000000003 2024-09-18T04:37:55.630 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.630 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.630 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:55.630 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479366+0000", 2024-09-18T04:37:55.630 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9024999999999998e-05 2024-09-18T04:37:55.630 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.630 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.630 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:55.630 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479443+0000", 2024-09-18T04:37:55.631 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6339000000000002e-05 2024-09-18T04:37:55.631 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:55.631 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:55.631 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:55.631 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.631 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.631 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.4214.0:1 2.3 e18/17 ondisk, result = 0)", 2024-09-18T04:37:55.631 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.474724+0000", 2024-09-18T04:37:55.631 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 21.145549186, 2024-09-18T04:37:55.631 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000105358, 2024-09-18T04:37:55.631 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:55.631 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:37:55.631 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:55.632 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.632 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:55.632 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474724+0000", 2024-09-18T04:37:55.632 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:55.632 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.632 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.632 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:55.632 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474724+0000", 2024-09-18T04:37:55.632 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:55.632 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.632 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.632 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:55.632 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474723+0000", 2024-09-18T04:37:55.633 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:37:55.633 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.633 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.633 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:55.633 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474731+0000", 2024-09-18T04:37:55.633 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2339999999999994e-06 2024-09-18T04:37:55.633 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.633 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.633 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:55.633 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474748+0000", 2024-09-18T04:37:55.633 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6603e-05 2024-09-18T04:37:55.633 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.633 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.634 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:55.634 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474760+0000", 2024-09-18T04:37:55.634 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2215000000000001e-05 2024-09-18T04:37:55.634 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.634 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.634 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:55.634 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474814+0000", 2024-09-18T04:37:55.634 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4267000000000001e-05 2024-09-18T04:37:55.634 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.634 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.634 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:55.634 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474821+0000", 2024-09-18T04:37:55.634 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0929999999999997e-06 2024-09-18T04:37:55.635 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.635 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.635 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:55.635 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474829+0000", 2024-09-18T04:37:55.635 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1200000000000002e-06 2024-09-18T04:37:55.635 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:55.635 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:55.635 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:55.635 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.635 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.635 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18)", 2024-09-18T04:37:55.635 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.482346+0000", 2024-09-18T04:37:55.635 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 21.137926753999999, 2024-09-18T04:37:55.635 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020435599999999999, 2024-09-18T04:37:55.636 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:55.636 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-09-18T04:37:55.636 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:55.636 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:37:55.636 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:37:55.636 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 2 2024-09-18T04:37:55.636 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.636 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:55.636 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.636 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:55.636 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482346+0000", 2024-09-18T04:37:55.636 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:55.636 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.637 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.637 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:55.637 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482346+0000", 2024-09-18T04:37:55.637 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:55.637 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.637 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.637 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:55.637 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482345+0000", 2024-09-18T04:37:55.637 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:37:55.637 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.637 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.637 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:55.637 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482360+0000", 2024-09-18T04:37:55.638 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4854999999999999e-05 2024-09-18T04:37:55.638 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.638 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.638 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:55.638 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482377+0000", 2024-09-18T04:37:55.638 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7785999999999999e-05 2024-09-18T04:37:55.638 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.638 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.638 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:55.638 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482387+0000", 2024-09-18T04:37:55.638 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6800000000000005e-06 2024-09-18T04:37:55.638 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.638 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.638 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:55.639 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482441+0000", 2024-09-18T04:37:55.639 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4046999999999999e-05 2024-09-18T04:37:55.639 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.639 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.639 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:55.639 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482551+0000", 2024-09-18T04:37:55.639 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000109468 2024-09-18T04:37:55.639 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:55.639 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:55.639 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:55.639 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.639 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.640 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18)", 2024-09-18T04:37:55.640 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.482853+0000", 2024-09-18T04:37:55.640 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 21.137420135999999, 2024-09-18T04:37:55.640 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00029815799999999997, 2024-09-18T04:37:55.640 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:55.640 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-09-18T04:37:55.640 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:55.640 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:37:55.640 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:37:55.640 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 3 2024-09-18T04:37:55.640 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.640 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:55.640 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.641 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:55.641 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482853+0000", 2024-09-18T04:37:55.641 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:55.641 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.641 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.641 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:55.641 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482853+0000", 2024-09-18T04:37:55.641 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:55.641 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.641 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.641 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:55.641 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482852+0000", 2024-09-18T04:37:55.641 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:37:55.642 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.642 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.642 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:55.642 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482859+0000", 2024-09-18T04:37:55.642 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0380000000000002e-06 2024-09-18T04:37:55.642 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.642 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.642 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:55.642 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482880+0000", 2024-09-18T04:37:55.642 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1243e-05 2024-09-18T04:37:55.642 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.642 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.642 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:55.643 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482896+0000", 2024-09-18T04:37:55.643 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5977e-05 2024-09-18T04:37:55.643 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.643 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.643 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:55.643 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482970+0000", 2024-09-18T04:37:55.643 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4603000000000004e-05 2024-09-18T04:37:55.643 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.643 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.643 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:55.643 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.483151+0000", 2024-09-18T04:37:55.643 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00018059799999999999 2024-09-18T04:37:55.644 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:55.644 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:55.644 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:55.644 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.644 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.644 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-18T04:37:55.644 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:35.468698+0000", 2024-09-18T04:37:55.644 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 20.151575299000001, 2024-09-18T04:37:55.644 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0039954129999999997, 2024-09-18T04:37:55.644 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:55.644 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:55.644 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:55.644 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:37:55.645 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:37:55.645 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 5 2024-09-18T04:37:55.645 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.645 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:55.645 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.645 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:55.645 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468698+0000", 2024-09-18T04:37:55.645 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:55.645 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.645 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.645 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:55.645 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468698+0000", 2024-09-18T04:37:55.645 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:55.646 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.646 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.646 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:55.646 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468696+0000", 2024-09-18T04:37:55.646 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:37:55.646 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.646 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.646 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:55.646 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468714+0000", 2024-09-18T04:37:55.646 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8743e-05 2024-09-18T04:37:55.646 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.646 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.646 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:55.646 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468748+0000", 2024-09-18T04:37:55.647 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3339e-05 2024-09-18T04:37:55.647 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.647 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.647 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:55.647 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468771+0000", 2024-09-18T04:37:55.647 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3119999999999999e-05 2024-09-18T04:37:55.647 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.647 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.647 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:55.647 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468922+0000", 2024-09-18T04:37:55.647 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000151383 2024-09-18T04:37:55.647 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.647 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.647 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:55.648 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.469278+0000", 2024-09-18T04:37:55.648 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00035628599999999998 2024-09-18T04:37:55.648 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.648 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.648 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:37:55.648 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.469629+0000", 2024-09-18T04:37:55.648 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00035014300000000003 2024-09-18T04:37:55.648 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.648 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.648 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:55.648 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.471620+0000", 2024-09-18T04:37:55.648 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001991278 2024-09-18T04:37:55.648 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.648 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.649 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:55.649 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472440+0000", 2024-09-18T04:37:55.649 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00081998599999999998 2024-09-18T04:37:55.649 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.649 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.649 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:55.649 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472592+0000", 2024-09-18T04:37:55.649 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015227400000000001 2024-09-18T04:37:55.649 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.649 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.649 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:55.649 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472693+0000", 2024-09-18T04:37:55.649 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000101074 2024-09-18T04:37:55.650 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:55.650 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:55.650 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:55.650 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.650 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.650 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.4214.0:5 2.2 e18/17 ondisk, result = 0)", 2024-09-18T04:37:55.650 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:35.472226+0000", 2024-09-18T04:37:55.650 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 20.148047398999999, 2024-09-18T04:37:55.650 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048505299999999999, 2024-09-18T04:37:55.650 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:55.650 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:37:55.650 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:55.650 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.650 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:55.651 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472226+0000", 2024-09-18T04:37:55.651 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:55.651 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.651 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.651 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:55.651 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472226+0000", 2024-09-18T04:37:55.651 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:55.651 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.651 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.651 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:55.651 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472224+0000", 2024-09-18T04:37:55.651 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:37:55.651 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.651 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.652 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:55.652 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472238+0000", 2024-09-18T04:37:55.652 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.468e-05 2024-09-18T04:37:55.652 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.652 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.652 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:55.652 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472272+0000", 2024-09-18T04:37:55.652 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3586e-05 2024-09-18T04:37:55.652 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.652 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.652 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:55.652 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472297+0000", 2024-09-18T04:37:55.652 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5282999999999998e-05 2024-09-18T04:37:55.653 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.653 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.653 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:55.653 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472396+0000", 2024-09-18T04:37:55.653 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.8394999999999996e-05 2024-09-18T04:37:55.653 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.653 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.653 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:55.653 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472421+0000", 2024-09-18T04:37:55.653 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5479000000000001e-05 2024-09-18T04:37:55.653 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.653 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.653 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:55.653 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472711+0000", 2024-09-18T04:37:55.654 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000289714 2024-09-18T04:37:55.654 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:55.654 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:55.654 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:55.654 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.654 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.654 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-18T04:37:55.654 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:37:55.654 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 19.13740013, 2024-09-18T04:37:55.654 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082699560000000002, 2024-09-18T04:37:55.654 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:55.654 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:55.654 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:55.655 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:37:55.655 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:37:55.655 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 7 2024-09-18T04:37:55.655 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.655 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:55.655 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.655 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:55.655 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:37:55.655 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:55.655 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.655 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.655 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:55.655 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:37:55.656 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:55.656 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.656 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.656 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:55.656 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482870+0000", 2024-09-18T04:37:55.656 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999967 2024-09-18T04:37:55.656 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.656 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.656 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:55.656 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482889+0000", 2024-09-18T04:37:55.656 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.965e-05 2024-09-18T04:37:55.656 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.656 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.656 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:55.657 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482938+0000", 2024-09-18T04:37:55.657 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8548999999999999e-05 2024-09-18T04:37:55.657 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.657 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.657 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:55.657 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482963+0000", 2024-09-18T04:37:55.657 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5075999999999998e-05 2024-09-18T04:37:55.657 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.657 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.657 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:55.657 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483121+0000", 2024-09-18T04:37:55.657 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015819400000000001 2024-09-18T04:37:55.657 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.658 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.658 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op must wait for map", 2024-09-18T04:37:55.658 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483138+0000", 2024-09-18T04:37:55.658 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7068999999999999e-05 2024-09-18T04:37:55.658 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.658 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.658 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:55.658 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488666+0000", 2024-09-18T04:37:55.658 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0055282049999999996 2024-09-18T04:37:55.658 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.658 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.658 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:55.658 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488783+0000", 2024-09-18T04:37:55.658 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000116737 2024-09-18T04:37:55.659 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.659 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.659 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:37:55.659 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488991+0000", 2024-09-18T04:37:55.659 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020781400000000001 2024-09-18T04:37:55.659 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.659 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.659 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:55.659 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490554+0000", 2024-09-18T04:37:55.659 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0015628370000000001 2024-09-18T04:37:55.659 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.659 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.659 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:55.659 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491019+0000", 2024-09-18T04:37:55.660 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000465608 2024-09-18T04:37:55.660 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.660 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.660 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:55.660 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491087+0000", 2024-09-18T04:37:55.660 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7988e-05 2024-09-18T04:37:55.660 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.660 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.660 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:55.660 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491143+0000", 2024-09-18T04:37:55.660 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5538999999999998e-05 2024-09-18T04:37:55.660 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:55.660 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:55.661 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:55.661 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.661 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.661 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.4214.0:7 2.2 e19/17 ondisk, result = 0)", 2024-09-18T04:37:55.661 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:36.490861+0000", 2024-09-18T04:37:55.661 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 19.129412508000001, 2024-09-18T04:37:55.661 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00029639400000000001, 2024-09-18T04:37:55.661 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:55.661 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:37:55.661 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:55.661 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.661 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:55.661 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490861+0000", 2024-09-18T04:37:55.662 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:55.662 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.662 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.662 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:55.662 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490861+0000", 2024-09-18T04:37:55.662 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:55.662 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.662 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.662 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:55.662 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490859+0000", 2024-09-18T04:37:55.662 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:37:55.662 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.662 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.662 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:55.663 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490867+0000", 2024-09-18T04:37:55.663 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.3089999999999996e-06 2024-09-18T04:37:55.663 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.663 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.663 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:55.663 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490909+0000", 2024-09-18T04:37:55.663 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1919000000000003e-05 2024-09-18T04:37:55.663 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.663 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.663 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:55.663 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490922+0000", 2024-09-18T04:37:55.663 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2836999999999999e-05 2024-09-18T04:37:55.663 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.663 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.664 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:55.664 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491003+0000", 2024-09-18T04:37:55.664 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0681999999999998e-05 2024-09-18T04:37:55.664 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.664 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.664 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:55.664 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491013+0000", 2024-09-18T04:37:55.664 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0220999999999999e-05 2024-09-18T04:37:55.664 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.664 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.664 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:55.664 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491157+0000", 2024-09-18T04:37:55.664 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014401799999999999 2024-09-18T04:37:55.665 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:55.665 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:55.665 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:55.665 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.665 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.665 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "MOSDPGPushReply(2.3 25/24 [PushReplyOp(2:c4c92e5a:::rbd_trash:head)])", 2024-09-18T04:37:55.665 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:51.680323+0000", 2024-09-18T04:37:55.665 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 3.939950133, 2024-09-18T04:37:55.665 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00054411499999999996, 2024-09-18T04:37:55.665 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:55.666 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-09-18T04:37:55.666 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:55.666 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.666 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:55.666 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680323+0000", 2024-09-18T04:37:55.666 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:55.666 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.666 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.666 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:55.666 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680323+0000", 2024-09-18T04:37:55.666 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:55.666 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.666 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.667 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:55.667 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680322+0000", 2024-09-18T04:37:55.667 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:37:55.667 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.667 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.667 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:55.667 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680333+0000", 2024-09-18T04:37:55.667 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1153e-05 2024-09-18T04:37:55.667 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.667 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.667 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:55.667 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680357+0000", 2024-09-18T04:37:55.667 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3252999999999999e-05 2024-09-18T04:37:55.667 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.668 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.668 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:55.668 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680378+0000", 2024-09-18T04:37:55.668 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1503000000000001e-05 2024-09-18T04:37:55.668 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.668 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.668 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:55.668 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680522+0000", 2024-09-18T04:37:55.668 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014410700000000001 2024-09-18T04:37:55.668 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:55.668 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:55.668 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:55.668 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680867+0000", 2024-09-18T04:37:55.668 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00034475899999999998 2024-09-18T04:37:55.669 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:55.669 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:55.669 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:55.669 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:55.669 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:55.669 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:37:55.670 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:37:55.666+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:37:55.671 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:37:55.685 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-18T04:37:55.685 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T04:37:55.685 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-18T04:37:55.686 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-18T04:37:55.700 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-18T04:37:55.771 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:37:55.768+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:37:55.789 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:37:55.790 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:37:55.790 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:37:55.790 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:37:55.800 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:37:55.872 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:37:55.868+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:37:55.908 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:37:55.909 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:37:55.909 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:37:55.909 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:37:55.909 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:37:55.918 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:37:55.973 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:37:55.968+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:37:56.015 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:37:56.015 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:37:56.015 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 26, 2024-09-18T04:37:56.015 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "be6146ba-a991-4f6e-b359-99cbedc9f9b1", 2024-09-18T04:37:56.015 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-09-18T04:37:15.676634+0000", 2024-09-18T04:37:56.015 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-09-18T04:37:53.669818+0000", 2024-09-18T04:37:56.015 INFO:teuthology.orchestra.run.smithi028.stdout: "last_up_change": "2024-09-18T04:37:25.403175+0000", 2024-09-18T04:37:56.015 INFO:teuthology.orchestra.run.smithi028.stdout: "last_in_change": "2024-09-18T04:37:50.652169+0000", 2024-09-18T04:37:56.016 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-18T04:37:56.016 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_num": 5799936, 2024-09-18T04:37:56.016 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_set": [ 2024-09-18T04:37:56.016 INFO:teuthology.orchestra.run.smithi028.stdout: "pglog_hardlimit", 2024-09-18T04:37:56.016 INFO:teuthology.orchestra.run.smithi028.stdout: "purged_snapdirs", 2024-09-18T04:37:56.016 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_deletes", 2024-09-18T04:37:56.016 INFO:teuthology.orchestra.run.smithi028.stdout: "sortbitwise" 2024-09-18T04:37:56.016 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-18T04:37:56.016 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_version": 5, 2024-09-18T04:37:56.016 INFO:teuthology.orchestra.run.smithi028.stdout: "full_ratio": 0.94999998807907104, 2024-09-18T04:37:56.016 INFO:teuthology.orchestra.run.smithi028.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-18T04:37:56.016 INFO:teuthology.orchestra.run.smithi028.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-18T04:37:56.016 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_snapshot": "", 2024-09-18T04:37:56.017 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_max": 2, 2024-09-18T04:37:56.017 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd": 6, 2024-09-18T04:37:56.017 INFO:teuthology.orchestra.run.smithi028.stdout: "require_min_compat_client": "luminous", 2024-09-18T04:37:56.017 INFO:teuthology.orchestra.run.smithi028.stdout: "min_compat_client": "jewel", 2024-09-18T04:37:56.017 INFO:teuthology.orchestra.run.smithi028.stdout: "require_osd_release": "reef", 2024-09-18T04:37:56.017 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_crimson": false, 2024-09-18T04:37:56.017 INFO:teuthology.orchestra.run.smithi028.stdout: "pools": [ 2024-09-18T04:37:56.017 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.017 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 1, 2024-09-18T04:37:56.017 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": ".mgr", 2024-09-18T04:37:56.017 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-18T04:37:25.654950+0000", 2024-09-18T04:37:56.017 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-18T04:37:56.017 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-09-18T04:37:56.018 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-18T04:37:56.018 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 2, 2024-09-18T04:37:56.018 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 1, 2024-09-18T04:37:56.018 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-18T04:37:56.018 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-18T04:37:56.018 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-18T04:37:56.018 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T04:37:56.018 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T04:37:56.018 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-18T04:37:56.018 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-18T04:37:56.018 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 1, 2024-09-18T04:37:56.018 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 1, 2024-09-18T04:37:56.019 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 1, 2024-09-18T04:37:56.019 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 1, 2024-09-18T04:37:56.019 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 1, 2024-09-18T04:37:56.019 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-18T04:37:56.019 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-18T04:37:56.019 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-18T04:37:56.019 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-18T04:37:56.019 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-18T04:37:56.019 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-18T04:37:56.019 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-18T04:37:56.019 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.019 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "16", 2024-09-18T04:37:56.019 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-18T04:37:56.019 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T04:37:56.020 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T04:37:56.020 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-18T04:37:56.020 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-18T04:37:56.020 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-09-18T04:37:56.020 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-09-18T04:37:56.020 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-18T04:37:56.020 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-18T04:37:56.020 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-18T04:37:56.020 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-18T04:37:56.020 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-18T04:37:56.020 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-18T04:37:56.020 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-18T04:37:56.020 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-18T04:37:56.021 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-18T04:37:56.021 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-18T04:37:56.021 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-18T04:37:56.021 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T04:37:56.021 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T04:37:56.021 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T04:37:56.021 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-18T04:37:56.021 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-18T04:37:56.021 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-18T04:37:56.021 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-18T04:37:56.021 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-18T04:37:56.021 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.021 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-18T04:37:56.021 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-18T04:37:56.022 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-18T04:37:56.022 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-18T04:37:56.022 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-18T04:37:56.022 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T04:37:56.022 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-18T04:37:56.022 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-18T04:37:56.022 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-18T04:37:56.022 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-18T04:37:56.022 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-18T04:37:56.022 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-09-18T04:37:56.022 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_max": 32, 2024-09-18T04:37:56.022 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_min": 1 2024-09-18T04:37:56.022 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.023 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-18T04:37:56.023 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": {} 2024-09-18T04:37:56.023 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.023 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-09-18T04:37:56.023 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 6.059999942779541, 2024-09-18T04:37:56.023 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 6.059999942779541, 2024-09-18T04:37:56.023 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.33000001311302185, 2024-09-18T04:37:56.023 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2, 2024-09-18T04:37:56.023 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2, 2024-09-18T04:37:56.023 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 1, 2024-09-18T04:37:56.023 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 1, 2024-09-18T04:37:56.023 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 1 2024-09-18T04:37:56.023 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.023 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.024 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.024 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 2, 2024-09-18T04:37:56.024 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "rbd", 2024-09-18T04:37:56.024 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-18T04:37:32.672462+0000", 2024-09-18T04:37:56.024 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 8193, 2024-09-18T04:37:56.024 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-18T04:37:56.024 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-18T04:37:56.024 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 2, 2024-09-18T04:37:56.024 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 1, 2024-09-18T04:37:56.024 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-18T04:37:56.024 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-18T04:37:56.024 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-18T04:37:56.025 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T04:37:56.025 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T04:37:56.025 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-18T04:37:56.025 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-18T04:37:56.025 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 8, 2024-09-18T04:37:56.025 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 8, 2024-09-18T04:37:56.025 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 8, 2024-09-18T04:37:56.025 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 8, 2024-09-18T04:37:56.025 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 8, 2024-09-18T04:37:56.025 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-18T04:37:56.025 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-18T04:37:56.026 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-18T04:37:56.026 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-18T04:37:56.026 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-18T04:37:56.026 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-18T04:37:56.026 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-18T04:37:56.026 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.026 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "20", 2024-09-18T04:37:56.026 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-18T04:37:56.026 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T04:37:56.026 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T04:37:56.026 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-18T04:37:56.026 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-18T04:37:56.026 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 3, 2024-09-18T04:37:56.027 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 20, 2024-09-18T04:37:56.027 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-18T04:37:56.027 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-18T04:37:56.027 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-18T04:37:56.027 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-18T04:37:56.027 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-18T04:37:56.027 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-18T04:37:56.027 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-18T04:37:56.027 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-18T04:37:56.027 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-18T04:37:56.027 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-18T04:37:56.027 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-18T04:37:56.027 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T04:37:56.028 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T04:37:56.028 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T04:37:56.028 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-18T04:37:56.028 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-18T04:37:56.028 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-18T04:37:56.028 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-18T04:37:56.028 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-18T04:37:56.028 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.028 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-18T04:37:56.028 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-18T04:37:56.028 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-18T04:37:56.028 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-18T04:37:56.028 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-18T04:37:56.028 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T04:37:56.029 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-18T04:37:56.029 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-18T04:37:56.029 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-18T04:37:56.029 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-18T04:37:56.029 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-18T04:37:56.029 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-09-18T04:37:56.029 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-18T04:37:56.029 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd": {} 2024-09-18T04:37:56.029 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.029 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-09-18T04:37:56.029 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2.25, 2024-09-18T04:37:56.029 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2.25, 2024-09-18T04:37:56.029 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-09-18T04:37:56.030 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2.25, 2024-09-18T04:37:56.030 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2.25, 2024-09-18T04:37:56.030 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 1, 2024-09-18T04:37:56.030 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 1, 2024-09-18T04:37:56.030 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 1 2024-09-18T04:37:56.030 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.030 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.030 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-18T04:37:56.030 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-09-18T04:37:56.030 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.030 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-09-18T04:37:56.030 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "6177002d-72e6-4888-bcde-29b6a92d9dc4", 2024-09-18T04:37:56.030 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-18T04:37:56.030 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-18T04:37:56.031 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-18T04:37:56.031 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-18T04:37:56.031 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-18T04:37:56.031 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-18T04:37:56.031 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 13, 2024-09-18T04:37:56.031 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 0, 2024-09-18T04:37:56.031 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-18T04:37:56.031 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-18T04:37:56.031 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-18T04:37:56.031 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:37:56.031 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.031 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:37:56.032 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6801", 2024-09-18T04:37:56.032 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1554407851 2024-09-18T04:37:56.032 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.032 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.032 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:37:56.032 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6803", 2024-09-18T04:37:56.032 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1554407851 2024-09-18T04:37:56.032 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.032 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.032 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.032 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-18T04:37:56.032 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:37:56.032 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.032 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:37:56.033 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6805", 2024-09-18T04:37:56.033 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1554407851 2024-09-18T04:37:56.033 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.033 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.033 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:37:56.033 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6807", 2024-09-18T04:37:56.033 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1554407851 2024-09-18T04:37:56.033 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.033 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.033 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.033 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-18T04:37:56.033 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:37:56.033 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.034 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:37:56.034 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6813", 2024-09-18T04:37:56.034 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1554407851 2024-09-18T04:37:56.034 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.034 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.034 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:37:56.034 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6815", 2024-09-18T04:37:56.034 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1554407851 2024-09-18T04:37:56.034 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.034 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.034 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.034 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-18T04:37:56.034 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:37:56.035 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.035 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:37:56.035 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6809", 2024-09-18T04:37:56.035 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1554407851 2024-09-18T04:37:56.035 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.035 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.035 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:37:56.035 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6811", 2024-09-18T04:37:56.035 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1554407851 2024-09-18T04:37:56.035 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.035 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.035 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.036 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6803/1554407851", 2024-09-18T04:37:56.036 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6807/1554407851", 2024-09-18T04:37:56.036 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6815/1554407851", 2024-09-18T04:37:56.036 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6811/1554407851", 2024-09-18T04:37:56.036 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-18T04:37:56.036 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-18T04:37:56.036 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-18T04:37:56.036 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.036 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.036 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-09-18T04:37:56.036 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "4f62cb09-3457-418c-bb69-491d9b70914c", 2024-09-18T04:37:56.037 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-18T04:37:56.037 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-18T04:37:56.037 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-18T04:37:56.037 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-18T04:37:56.037 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-18T04:37:56.037 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-18T04:37:56.037 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 12, 2024-09-18T04:37:56.037 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 21, 2024-09-18T04:37:56.037 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-18T04:37:56.037 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-18T04:37:56.037 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-18T04:37:56.037 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:37:56.037 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.038 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:37:56.038 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6800", 2024-09-18T04:37:56.038 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 511141401 2024-09-18T04:37:56.038 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.038 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.038 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:37:56.038 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6802", 2024-09-18T04:37:56.038 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 511141401 2024-09-18T04:37:56.038 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.038 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.038 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.038 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-18T04:37:56.038 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:37:56.038 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.039 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:37:56.039 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6804", 2024-09-18T04:37:56.039 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 511141401 2024-09-18T04:37:56.039 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.039 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.039 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:37:56.039 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6806", 2024-09-18T04:37:56.039 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 511141401 2024-09-18T04:37:56.039 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.039 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.039 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.039 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-18T04:37:56.039 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:37:56.040 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.040 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:37:56.040 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6812", 2024-09-18T04:37:56.040 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 511141401 2024-09-18T04:37:56.040 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.040 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.040 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:37:56.040 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6814", 2024-09-18T04:37:56.040 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 511141401 2024-09-18T04:37:56.040 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.040 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.040 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.040 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-18T04:37:56.041 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:37:56.041 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.041 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:37:56.041 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6808", 2024-09-18T04:37:56.041 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 511141401 2024-09-18T04:37:56.041 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.041 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.041 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:37:56.041 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6810", 2024-09-18T04:37:56.041 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 511141401 2024-09-18T04:37:56.041 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.041 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.042 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6802/511141401", 2024-09-18T04:37:56.042 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6806/511141401", 2024-09-18T04:37:56.042 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6814/511141401", 2024-09-18T04:37:56.042 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6810/511141401", 2024-09-18T04:37:56.042 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-18T04:37:56.042 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-18T04:37:56.042 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-18T04:37:56.042 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.042 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.042 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-09-18T04:37:56.043 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "854fe031-3b87-4290-9c2f-3fb196fb806c", 2024-09-18T04:37:56.043 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-18T04:37:56.043 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-18T04:37:56.043 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-18T04:37:56.043 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-18T04:37:56.043 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-18T04:37:56.043 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-18T04:37:56.043 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 12, 2024-09-18T04:37:56.043 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 24, 2024-09-18T04:37:56.043 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-18T04:37:56.043 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-18T04:37:56.043 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-18T04:37:56.043 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:37:56.044 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.044 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:37:56.044 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6816", 2024-09-18T04:37:56.044 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2623426735 2024-09-18T04:37:56.044 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.044 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.044 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:37:56.044 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6817", 2024-09-18T04:37:56.044 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2623426735 2024-09-18T04:37:56.044 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.044 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.044 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.044 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-18T04:37:56.045 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:37:56.045 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.045 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:37:56.045 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6818", 2024-09-18T04:37:56.045 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2623426735 2024-09-18T04:37:56.045 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.045 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.045 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:37:56.045 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6819", 2024-09-18T04:37:56.045 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2623426735 2024-09-18T04:37:56.045 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.045 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.046 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.046 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-18T04:37:56.046 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:37:56.046 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.046 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:37:56.046 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6822", 2024-09-18T04:37:56.046 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2623426735 2024-09-18T04:37:56.046 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.046 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.046 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:37:56.046 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6823", 2024-09-18T04:37:56.046 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2623426735 2024-09-18T04:37:56.046 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.046 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.047 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.047 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-18T04:37:56.047 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:37:56.047 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.047 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:37:56.047 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6820", 2024-09-18T04:37:56.047 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2623426735 2024-09-18T04:37:56.047 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.047 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.047 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:37:56.047 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6821", 2024-09-18T04:37:56.047 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2623426735 2024-09-18T04:37:56.047 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.047 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.048 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.048 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6817/2623426735", 2024-09-18T04:37:56.048 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6819/2623426735", 2024-09-18T04:37:56.048 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6823/2623426735", 2024-09-18T04:37:56.048 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6821/2623426735", 2024-09-18T04:37:56.048 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-18T04:37:56.048 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-18T04:37:56.048 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-18T04:37:56.048 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.048 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.048 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.048 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-09-18T04:37:56.049 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "c2625f98-780b-4f86-bf36-a48d2a227756", 2024-09-18T04:37:56.049 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-18T04:37:56.049 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-18T04:37:56.049 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-18T04:37:56.049 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-18T04:37:56.049 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-18T04:37:56.049 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-18T04:37:56.049 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 12, 2024-09-18T04:37:56.049 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 17, 2024-09-18T04:37:56.049 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-18T04:37:56.049 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-18T04:37:56.049 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-18T04:37:56.050 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:37:56.050 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.050 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:37:56.050 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6808", 2024-09-18T04:37:56.050 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2746051132 2024-09-18T04:37:56.050 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.050 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.050 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:37:56.050 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6809", 2024-09-18T04:37:56.050 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2746051132 2024-09-18T04:37:56.050 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.050 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.050 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.050 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-18T04:37:56.051 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:37:56.051 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.051 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:37:56.051 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6810", 2024-09-18T04:37:56.051 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2746051132 2024-09-18T04:37:56.051 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.051 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.051 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:37:56.051 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6811", 2024-09-18T04:37:56.051 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2746051132 2024-09-18T04:37:56.051 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.051 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.051 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.052 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-18T04:37:56.052 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:37:56.052 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.052 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:37:56.052 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6814", 2024-09-18T04:37:56.052 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2746051132 2024-09-18T04:37:56.052 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.052 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.052 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:37:56.052 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6815", 2024-09-18T04:37:56.052 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2746051132 2024-09-18T04:37:56.052 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.052 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.053 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.053 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-18T04:37:56.053 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:37:56.053 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.053 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:37:56.053 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6812", 2024-09-18T04:37:56.053 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2746051132 2024-09-18T04:37:56.053 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.053 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.053 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:37:56.053 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6813", 2024-09-18T04:37:56.053 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2746051132 2024-09-18T04:37:56.053 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.054 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.054 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.054 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.82:6809/2746051132", 2024-09-18T04:37:56.054 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.82:6811/2746051132", 2024-09-18T04:37:56.054 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.82:6815/2746051132", 2024-09-18T04:37:56.054 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.82:6813/2746051132", 2024-09-18T04:37:56.054 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-18T04:37:56.054 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-18T04:37:56.054 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-18T04:37:56.054 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.054 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.054 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.055 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-09-18T04:37:56.055 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "deb22d86-4081-484d-96eb-9cb9d96b60bc", 2024-09-18T04:37:56.055 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-18T04:37:56.055 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-18T04:37:56.055 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-18T04:37:56.055 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-18T04:37:56.055 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-18T04:37:56.055 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-18T04:37:56.055 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 13, 2024-09-18T04:37:56.055 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 0, 2024-09-18T04:37:56.055 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-18T04:37:56.055 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-18T04:37:56.055 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-18T04:37:56.056 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:37:56.056 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.056 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:37:56.056 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6800", 2024-09-18T04:37:56.056 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2005712460 2024-09-18T04:37:56.056 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.056 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.056 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:37:56.056 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6801", 2024-09-18T04:37:56.056 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2005712460 2024-09-18T04:37:56.056 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.056 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.056 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.057 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-18T04:37:56.057 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:37:56.057 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.057 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:37:56.057 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6802", 2024-09-18T04:37:56.057 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2005712460 2024-09-18T04:37:56.057 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.057 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.057 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:37:56.058 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6803", 2024-09-18T04:37:56.058 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2005712460 2024-09-18T04:37:56.058 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.058 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.058 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.058 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-18T04:37:56.058 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:37:56.058 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.058 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:37:56.058 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6806", 2024-09-18T04:37:56.058 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2005712460 2024-09-18T04:37:56.058 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.058 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.059 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:37:56.059 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6807", 2024-09-18T04:37:56.059 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2005712460 2024-09-18T04:37:56.059 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.059 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.059 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.059 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-18T04:37:56.059 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:37:56.059 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.059 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:37:56.059 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6804", 2024-09-18T04:37:56.059 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2005712460 2024-09-18T04:37:56.059 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.059 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.060 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:37:56.060 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6805", 2024-09-18T04:37:56.060 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2005712460 2024-09-18T04:37:56.060 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.060 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.060 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.060 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.82:6801/2005712460", 2024-09-18T04:37:56.060 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.82:6803/2005712460", 2024-09-18T04:37:56.060 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.82:6807/2005712460", 2024-09-18T04:37:56.060 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.82:6805/2005712460", 2024-09-18T04:37:56.060 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-18T04:37:56.060 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-18T04:37:56.060 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-18T04:37:56.061 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.061 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.061 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.061 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-09-18T04:37:56.061 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "061d332c-f51b-403d-bb18-eba415c97e7f", 2024-09-18T04:37:56.061 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-18T04:37:56.061 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-18T04:37:56.061 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-18T04:37:56.061 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-18T04:37:56.061 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-18T04:37:56.061 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-18T04:37:56.061 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 13, 2024-09-18T04:37:56.061 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 24, 2024-09-18T04:37:56.062 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-18T04:37:56.062 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-18T04:37:56.062 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-18T04:37:56.062 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:37:56.062 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.062 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:37:56.062 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6816", 2024-09-18T04:37:56.062 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1267644232 2024-09-18T04:37:56.062 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.062 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.062 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:37:56.062 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6817", 2024-09-18T04:37:56.062 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1267644232 2024-09-18T04:37:56.063 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.063 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.063 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.063 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-18T04:37:56.063 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:37:56.063 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.063 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:37:56.063 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6818", 2024-09-18T04:37:56.063 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1267644232 2024-09-18T04:37:56.063 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.063 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.063 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:37:56.063 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6819", 2024-09-18T04:37:56.064 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1267644232 2024-09-18T04:37:56.064 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.064 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.064 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.064 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-18T04:37:56.064 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:37:56.064 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.064 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:37:56.064 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6822", 2024-09-18T04:37:56.064 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1267644232 2024-09-18T04:37:56.064 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.064 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.064 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:37:56.065 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6823", 2024-09-18T04:37:56.065 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1267644232 2024-09-18T04:37:56.065 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.065 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.065 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.065 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-18T04:37:56.065 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:37:56.065 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.065 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:37:56.065 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6820", 2024-09-18T04:37:56.065 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1267644232 2024-09-18T04:37:56.065 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.066 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.066 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:37:56.066 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6821", 2024-09-18T04:37:56.066 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1267644232 2024-09-18T04:37:56.066 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.066 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.066 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.82:6817/1267644232", 2024-09-18T04:37:56.066 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.82:6819/1267644232", 2024-09-18T04:37:56.066 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.82:6823/1267644232", 2024-09-18T04:37:56.066 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.82:6821/1267644232", 2024-09-18T04:37:56.066 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-18T04:37:56.067 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-18T04:37:56.067 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-18T04:37:56.067 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.067 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.067 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-18T04:37:56.067 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_xinfo": [ 2024-09-18T04:37:56.067 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.067 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-09-18T04:37:56.067 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-18T04:37:56.067 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-18T04:37:56.067 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-18T04:37:56.067 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138322906710015, 2024-09-18T04:37:56.067 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-18T04:37:56.068 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-18T04:37:22.192087+0000", 2024-09-18T04:37:56.068 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-18T04:37:56.068 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.068 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.068 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-09-18T04:37:56.068 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-18T04:37:56.068 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-18T04:37:56.068 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-18T04:37:56.068 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138322906710015, 2024-09-18T04:37:56.068 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-18T04:37:56.068 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-18T04:37:22.017833+0000", 2024-09-18T04:37:56.068 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-18T04:37:56.068 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.069 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.069 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-09-18T04:37:56.069 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-18T04:37:56.069 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-18T04:37:56.069 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-18T04:37:56.069 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138322906710015, 2024-09-18T04:37:56.069 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-18T04:37:56.069 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-18T04:37:22.755774+0000", 2024-09-18T04:37:56.069 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-18T04:37:56.069 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.069 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.069 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-09-18T04:37:56.069 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-18T04:37:56.070 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-18T04:37:56.070 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-18T04:37:56.070 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138322906710015, 2024-09-18T04:37:56.070 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-18T04:37:56.070 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-18T04:37:22.663785+0000", 2024-09-18T04:37:56.070 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-18T04:37:56.070 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.070 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.070 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-09-18T04:37:56.070 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-18T04:37:56.070 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-18T04:37:56.070 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-18T04:37:56.070 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138322906710015, 2024-09-18T04:37:56.070 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-18T04:37:56.070 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-18T04:37:22.531758+0000", 2024-09-18T04:37:56.071 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-18T04:37:56.071 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.071 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.071 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-09-18T04:37:56.071 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-18T04:37:56.071 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-18T04:37:56.071 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-18T04:37:56.071 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138322906710015, 2024-09-18T04:37:56.071 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-18T04:37:56.071 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-18T04:37:22.400774+0000", 2024-09-18T04:37:56.071 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-18T04:37:56.071 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.071 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-18T04:37:56.071 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap": [], 2024-09-18T04:37:56.072 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap_items": [], 2024-09-18T04:37:56.072 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap_primaries": [], 2024-09-18T04:37:56.072 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_temp": [], 2024-09-18T04:37:56.072 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_temp": [], 2024-09-18T04:37:56.072 INFO:teuthology.orchestra.run.smithi028.stdout: "blocklist": {}, 2024-09-18T04:37:56.072 INFO:teuthology.orchestra.run.smithi028.stdout: "range_blocklist": {}, 2024-09-18T04:37:56.072 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profiles": { 2024-09-18T04:37:56.072 INFO:teuthology.orchestra.run.smithi028.stdout: "default": { 2024-09-18T04:37:56.072 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-failure-domain": "osd", 2024-09-18T04:37:56.072 INFO:teuthology.orchestra.run.smithi028.stdout: "k": "2", 2024-09-18T04:37:56.072 INFO:teuthology.orchestra.run.smithi028.stdout: "m": "1", 2024-09-18T04:37:56.072 INFO:teuthology.orchestra.run.smithi028.stdout: "plugin": "jerasure", 2024-09-18T04:37:56.072 INFO:teuthology.orchestra.run.smithi028.stdout: "technique": "reed_sol_van" 2024-09-18T04:37:56.072 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.073 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.073 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps_queue": [], 2024-09-18T04:37:56.073 INFO:teuthology.orchestra.run.smithi028.stdout: "new_removed_snaps": [], 2024-09-18T04:37:56.073 INFO:teuthology.orchestra.run.smithi028.stdout: "new_purged_snaps": [], 2024-09-18T04:37:56.073 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_node_flags": {}, 2024-09-18T04:37:56.073 INFO:teuthology.orchestra.run.smithi028.stdout: "device_class_flags": {}, 2024-09-18T04:37:56.073 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": { 2024-09-18T04:37:56.073 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_enabled": false, 2024-09-18T04:37:56.073 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_bucket_count": 0, 2024-09-18T04:37:56.073 INFO:teuthology.orchestra.run.smithi028.stdout: "degraded_stretch_mode": 0, 2024-09-18T04:37:56.073 INFO:teuthology.orchestra.run.smithi028.stdout: "recovering_stretch_mode": 0, 2024-09-18T04:37:56.073 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_bucket": 0 2024-09-18T04:37:56.073 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.073 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:37:56.075 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:37:56.075 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:37:56.075 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:37:56.075 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:37:56.076 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.076 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:30 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'16, mlcod=15'14)", 2024-09-18T04:37:56.076 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.486330+0000", 2024-09-18T04:37:56.076 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 28.532941032, 2024-09-18T04:37:56.076 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026383700000000001, 2024-09-18T04:37:56.076 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:56.076 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:56.076 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:56.076 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.076 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:56.076 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486330+0000", 2024-09-18T04:37:56.077 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:56.077 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.077 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.077 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:56.077 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486330+0000", 2024-09-18T04:37:56.077 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:56.077 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.077 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.077 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:56.077 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486328+0000", 2024-09-18T04:37:56.077 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:37:56.078 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.078 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.078 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:56.078 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486342+0000", 2024-09-18T04:37:56.078 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3783e-05 2024-09-18T04:37:56.078 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.078 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.078 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:56.078 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486373+0000", 2024-09-18T04:37:56.078 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1362999999999998e-05 2024-09-18T04:37:56.078 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.078 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.079 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:56.079 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486387+0000", 2024-09-18T04:37:56.079 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3219e-05 2024-09-18T04:37:56.079 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.079 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.079 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:56.079 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486489+0000", 2024-09-18T04:37:56.079 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010232499999999999 2024-09-18T04:37:56.079 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.079 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.079 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:56.079 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486524+0000", 2024-09-18T04:37:56.079 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4972000000000003e-05 2024-09-18T04:37:56.079 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.080 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.080 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:56.080 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.488902+0000", 2024-09-18T04:37:56.080 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023776240000000001 2024-09-18T04:37:56.080 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.080 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.080 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:56.080 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.488968+0000", 2024-09-18T04:37:56.080 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6761000000000007e-05 2024-09-18T04:37:56.080 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.080 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.080 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.080 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.081 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.081 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:35 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'21, mlcod=15'17)", 2024-09-18T04:37:56.081 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.491817+0000", 2024-09-18T04:37:56.081 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 28.527454114000001, 2024-09-18T04:37:56.081 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023524739999999998, 2024-09-18T04:37:56.081 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:56.081 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:56.081 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:56.081 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.081 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:56.081 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491817+0000", 2024-09-18T04:37:56.081 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:56.081 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.082 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.082 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:56.082 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491817+0000", 2024-09-18T04:37:56.082 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:56.082 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.082 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.082 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:56.082 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491815+0000", 2024-09-18T04:37:56.082 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:37:56.082 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.082 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.082 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:56.082 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491890+0000", 2024-09-18T04:37:56.082 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4500999999999997e-05 2024-09-18T04:37:56.083 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.083 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.083 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:56.083 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491916+0000", 2024-09-18T04:37:56.083 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5666000000000001e-05 2024-09-18T04:37:56.083 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.083 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.083 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:56.083 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491928+0000", 2024-09-18T04:37:56.083 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2585999999999999e-05 2024-09-18T04:37:56.083 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.083 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.083 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:56.083 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.492000+0000", 2024-09-18T04:37:56.083 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2309000000000002e-05 2024-09-18T04:37:56.084 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.084 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.084 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:56.084 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.492027+0000", 2024-09-18T04:37:56.084 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6251e-05 2024-09-18T04:37:56.084 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.084 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.084 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:56.084 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.494114+0000", 2024-09-18T04:37:56.084 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020875260000000001 2024-09-18T04:37:56.084 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.084 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.084 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:56.085 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.494169+0000", 2024-09-18T04:37:56.085 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5145000000000001e-05 2024-09-18T04:37:56.085 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.085 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.085 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.085 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.085 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.085 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:57 1.0 e15/14 1:30306672:devicehealth::main.db.0000000000000000:head v 15'35, mlcod=15'33)", 2024-09-18T04:37:56.085 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.529997+0000", 2024-09-18T04:37:56.085 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 28.489274259999998, 2024-09-18T04:37:56.085 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002623869, 2024-09-18T04:37:56.085 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:56.086 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:56.086 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:56.086 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.086 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:56.086 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.529997+0000", 2024-09-18T04:37:56.086 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:56.086 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.086 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.086 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:56.086 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.529997+0000", 2024-09-18T04:37:56.086 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:56.086 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.086 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.086 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:56.087 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.529995+0000", 2024-09-18T04:37:56.087 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:37:56.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.087 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.087 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:56.087 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530116+0000", 2024-09-18T04:37:56.087 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000121141 2024-09-18T04:37:56.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.087 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.087 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:56.087 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530146+0000", 2024-09-18T04:37:56.087 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0423e-05 2024-09-18T04:37:56.088 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.088 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.088 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:56.088 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530159+0000", 2024-09-18T04:37:56.088 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2802999999999999e-05 2024-09-18T04:37:56.088 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.088 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.088 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:56.088 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530218+0000", 2024-09-18T04:37:56.088 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8461000000000002e-05 2024-09-18T04:37:56.088 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.088 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.089 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:56.089 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530253+0000", 2024-09-18T04:37:56.089 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4796999999999999e-05 2024-09-18T04:37:56.089 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.089 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.089 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:56.089 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.532559+0000", 2024-09-18T04:37:56.089 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002306939 2024-09-18T04:37:56.089 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.089 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.089 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:56.089 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.532621+0000", 2024-09-18T04:37:56.089 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1131999999999997e-05 2024-09-18T04:37:56.089 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.090 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.090 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.090 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.090 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:68 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'42, mlcod=15'39)", 2024-09-18T04:37:56.090 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.546404+0000", 2024-09-18T04:37:56.090 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 28.472866848999999, 2024-09-18T04:37:56.090 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024499769999999999, 2024-09-18T04:37:56.090 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:56.090 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:56.090 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:56.090 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.090 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:56.091 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546404+0000", 2024-09-18T04:37:56.091 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:56.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.091 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.091 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:56.091 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546404+0000", 2024-09-18T04:37:56.091 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:56.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.091 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.091 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:56.091 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546403+0000", 2024-09-18T04:37:56.091 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:37:56.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.091 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.092 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:56.092 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546416+0000", 2024-09-18T04:37:56.092 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.306e-05 2024-09-18T04:37:56.092 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.092 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.092 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:56.092 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546465+0000", 2024-09-18T04:37:56.092 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9441999999999999e-05 2024-09-18T04:37:56.092 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.092 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.092 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:56.092 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546480+0000", 2024-09-18T04:37:56.093 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4919999999999999e-05 2024-09-18T04:37:56.093 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.093 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.093 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:56.093 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546541+0000", 2024-09-18T04:37:56.093 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1085000000000002e-05 2024-09-18T04:37:56.093 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.093 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.093 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:56.093 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546569+0000", 2024-09-18T04:37:56.093 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7803000000000001e-05 2024-09-18T04:37:56.093 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.093 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.094 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:56.094 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.548725+0000", 2024-09-18T04:37:56.094 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021556629999999999 2024-09-18T04:37:56.094 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.094 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.094 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:56.094 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.548854+0000", 2024-09-18T04:37:56.094 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000129524 2024-09-18T04:37:56.094 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.094 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.094 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.094 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.094 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.094 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:69 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'43, mlcod=15'39)", 2024-09-18T04:37:56.095 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.547520+0000", 2024-09-18T04:37:56.095 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 28.471750773, 2024-09-18T04:37:56.095 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023506820000000002, 2024-09-18T04:37:56.095 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:56.095 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:56.095 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:56.095 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.095 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:56.095 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547520+0000", 2024-09-18T04:37:56.095 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:56.095 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.095 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.096 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:56.096 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547520+0000", 2024-09-18T04:37:56.096 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:56.096 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.096 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.096 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:56.096 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547519+0000", 2024-09-18T04:37:56.096 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:37:56.096 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.096 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.096 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:56.096 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547614+0000", 2024-09-18T04:37:56.096 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5235000000000006e-05 2024-09-18T04:37:56.096 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.097 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.097 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:56.097 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547640+0000", 2024-09-18T04:37:56.097 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5698999999999999e-05 2024-09-18T04:37:56.097 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.097 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.097 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:56.097 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547651+0000", 2024-09-18T04:37:56.097 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.134e-05 2024-09-18T04:37:56.097 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.097 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.097 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:56.097 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547805+0000", 2024-09-18T04:37:56.097 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015426099999999999 2024-09-18T04:37:56.098 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.098 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.098 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:56.098 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547844+0000", 2024-09-18T04:37:56.098 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8229999999999998e-05 2024-09-18T04:37:56.098 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.098 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.098 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:56.098 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.549810+0000", 2024-09-18T04:37:56.098 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0019660519999999998 2024-09-18T04:37:56.098 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.098 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.098 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:56.098 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.549871+0000", 2024-09-18T04:37:56.099 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1152e-05 2024-09-18T04:37:56.099 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.099 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.099 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.099 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.099 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.099 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:83 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'53, mlcod=15'50)", 2024-09-18T04:37:56.099 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.575989+0000", 2024-09-18T04:37:56.099 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 28.443281825, 2024-09-18T04:37:56.099 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023991310000000001, 2024-09-18T04:37:56.099 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:56.099 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:56.099 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:56.099 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.100 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:56.100 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.575989+0000", 2024-09-18T04:37:56.100 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:56.100 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.100 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.100 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:56.100 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.575989+0000", 2024-09-18T04:37:56.100 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:56.100 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.100 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.100 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:56.100 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.575988+0000", 2024-09-18T04:37:56.100 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:37:56.100 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.101 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.101 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:56.101 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576001+0000", 2024-09-18T04:37:56.101 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2751e-05 2024-09-18T04:37:56.101 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.101 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.101 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:56.101 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576023+0000", 2024-09-18T04:37:56.101 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2453999999999999e-05 2024-09-18T04:37:56.101 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.102 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.102 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:56.102 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576038+0000", 2024-09-18T04:37:56.102 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5125e-05 2024-09-18T04:37:56.102 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.102 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.102 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:56.102 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576161+0000", 2024-09-18T04:37:56.102 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000122543 2024-09-18T04:37:56.102 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.102 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.102 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:56.102 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576221+0000", 2024-09-18T04:37:56.103 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9948000000000003e-05 2024-09-18T04:37:56.103 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.103 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.103 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:56.103 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.578277+0000", 2024-09-18T04:37:56.103 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020567559999999999 2024-09-18T04:37:56.103 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.103 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.103 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:56.103 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.578388+0000", 2024-09-18T04:37:56.103 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011089100000000001 2024-09-18T04:37:56.103 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.103 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.103 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.104 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.104 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.104 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:86 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'56, mlcod=15'52)", 2024-09-18T04:37:56.104 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.580261+0000", 2024-09-18T04:37:56.104 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 28.439010189000001, 2024-09-18T04:37:56.104 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002418108, 2024-09-18T04:37:56.104 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:56.104 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:56.104 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:56.104 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.104 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:56.104 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580261+0000", 2024-09-18T04:37:56.104 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:56.104 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.104 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.105 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:56.105 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580261+0000", 2024-09-18T04:37:56.105 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:56.105 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.105 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.105 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:56.105 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580260+0000", 2024-09-18T04:37:56.105 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:37:56.105 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.105 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.105 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:56.106 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580269+0000", 2024-09-18T04:37:56.106 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5109999999999992e-06 2024-09-18T04:37:56.106 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.106 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.106 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:56.106 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580288+0000", 2024-09-18T04:37:56.106 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8783999999999998e-05 2024-09-18T04:37:56.106 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.106 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.106 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:56.106 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580296+0000", 2024-09-18T04:37:56.106 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3250000000000008e-06 2024-09-18T04:37:56.106 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.106 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.107 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:56.107 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580351+0000", 2024-09-18T04:37:56.107 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4132000000000003e-05 2024-09-18T04:37:56.107 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.107 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.107 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:56.107 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580381+0000", 2024-09-18T04:37:56.107 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0263000000000001e-05 2024-09-18T04:37:56.107 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.107 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.107 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:56.107 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.582615+0000", 2024-09-18T04:37:56.108 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0022339109999999999 2024-09-18T04:37:56.108 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.108 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.108 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:56.108 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.582679+0000", 2024-09-18T04:37:56.108 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4182000000000003e-05 2024-09-18T04:37:56.108 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.108 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.108 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.108 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.108 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.108 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:105 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'67, mlcod=15'64)", 2024-09-18T04:37:56.109 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.789089+0000", 2024-09-18T04:37:56.109 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 28.230182019000001, 2024-09-18T04:37:56.109 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024384229999999999, 2024-09-18T04:37:56.109 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:56.109 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:56.109 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:56.109 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.109 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:56.109 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789089+0000", 2024-09-18T04:37:56.109 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:56.109 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.109 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.109 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:56.109 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789089+0000", 2024-09-18T04:37:56.110 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:56.110 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.110 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.110 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:56.110 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789088+0000", 2024-09-18T04:37:56.110 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:37:56.110 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.110 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.110 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:56.110 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789107+0000", 2024-09-18T04:37:56.110 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9017000000000001e-05 2024-09-18T04:37:56.110 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.110 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.110 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:56.111 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789139+0000", 2024-09-18T04:37:56.111 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2261999999999997e-05 2024-09-18T04:37:56.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.111 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.111 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:56.111 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789153+0000", 2024-09-18T04:37:56.111 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4025999999999999e-05 2024-09-18T04:37:56.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.111 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.111 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:56.111 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789255+0000", 2024-09-18T04:37:56.111 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000101969 2024-09-18T04:37:56.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.111 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.111 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:56.112 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789314+0000", 2024-09-18T04:37:56.112 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8867999999999999e-05 2024-09-18T04:37:56.112 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.112 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.112 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:56.112 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.791453+0000", 2024-09-18T04:37:56.112 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002138938 2024-09-18T04:37:56.112 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.112 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.112 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:56.112 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.791527+0000", 2024-09-18T04:37:56.112 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4464999999999999e-05 2024-09-18T04:37:56.112 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.112 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.113 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.113 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.113 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.113 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:106 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'68, mlcod=15'64)", 2024-09-18T04:37:56.113 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.790295+0000", 2024-09-18T04:37:56.113 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 28.228976167999999, 2024-09-18T04:37:56.113 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024379390000000001, 2024-09-18T04:37:56.113 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:56.113 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:56.113 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:56.113 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.113 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:56.113 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790295+0000", 2024-09-18T04:37:56.113 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:56.114 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.114 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.114 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:56.114 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790295+0000", 2024-09-18T04:37:56.114 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:56.114 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.114 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.114 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:56.114 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790294+0000", 2024-09-18T04:37:56.114 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:37:56.114 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.114 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.114 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:56.114 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790390+0000", 2024-09-18T04:37:56.115 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6537000000000001e-05 2024-09-18T04:37:56.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.115 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.115 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:56.115 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790418+0000", 2024-09-18T04:37:56.115 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8218e-05 2024-09-18T04:37:56.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.115 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.115 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:56.115 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790431+0000", 2024-09-18T04:37:56.115 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2571999999999999e-05 2024-09-18T04:37:56.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.115 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.115 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:56.116 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790534+0000", 2024-09-18T04:37:56.116 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000103433 2024-09-18T04:37:56.116 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.116 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.116 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:56.116 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790565+0000", 2024-09-18T04:37:56.116 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0409999999999999e-05 2024-09-18T04:37:56.116 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.116 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.116 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:56.116 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.792651+0000", 2024-09-18T04:37:56.116 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020865250000000001 2024-09-18T04:37:56.116 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.117 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.117 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:56.117 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.792733+0000", 2024-09-18T04:37:56.117 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1477999999999995e-05 2024-09-18T04:37:56.117 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.117 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.117 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.117 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.117 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.117 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:109 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'71, mlcod=15'68)", 2024-09-18T04:37:56.117 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.795250+0000", 2024-09-18T04:37:56.117 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 28.224021153999999, 2024-09-18T04:37:56.117 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023990449999999998, 2024-09-18T04:37:56.117 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:56.118 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:56.118 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:56.118 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.118 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:56.118 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795250+0000", 2024-09-18T04:37:56.118 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:56.118 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.118 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.118 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:56.118 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795250+0000", 2024-09-18T04:37:56.118 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:56.118 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.118 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.118 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:56.118 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795248+0000", 2024-09-18T04:37:56.119 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:37:56.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.119 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.119 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:56.119 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795405+0000", 2024-09-18T04:37:56.119 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000157007 2024-09-18T04:37:56.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.119 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.119 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:56.119 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795422+0000", 2024-09-18T04:37:56.119 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7013000000000001e-05 2024-09-18T04:37:56.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.119 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.119 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:56.120 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795432+0000", 2024-09-18T04:37:56.120 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0448999999999999e-05 2024-09-18T04:37:56.120 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.120 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.120 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:56.120 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795534+0000", 2024-09-18T04:37:56.120 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000101195 2024-09-18T04:37:56.120 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.120 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.120 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:56.120 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795587+0000", 2024-09-18T04:37:56.120 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3356000000000002e-05 2024-09-18T04:37:56.120 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.121 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.121 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:56.121 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.797560+0000", 2024-09-18T04:37:56.121 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001973023 2024-09-18T04:37:56.121 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.121 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.121 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:56.121 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.797649+0000", 2024-09-18T04:37:56.121 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8864999999999997e-05 2024-09-18T04:37:56.121 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.121 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.121 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.121 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.121 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.122 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:115 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'77, mlcod=15'72)", 2024-09-18T04:37:56.122 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.803339+0000", 2024-09-18T04:37:56.122 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 28.215932435999999, 2024-09-18T04:37:56.122 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002385093, 2024-09-18T04:37:56.122 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:56.122 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:56.122 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:56.122 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.123 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:56.123 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803339+0000", 2024-09-18T04:37:56.123 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:56.123 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.123 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.123 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:56.123 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803339+0000", 2024-09-18T04:37:56.123 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:56.123 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.123 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.123 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:56.124 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803337+0000", 2024-09-18T04:37:56.124 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:37:56.124 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.124 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.124 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:56.124 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803475+0000", 2024-09-18T04:37:56.124 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013718199999999999 2024-09-18T04:37:56.124 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.124 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.124 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:56.124 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803507+0000", 2024-09-18T04:37:56.124 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2351000000000002e-05 2024-09-18T04:37:56.125 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.125 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.125 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:56.125 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803533+0000", 2024-09-18T04:37:56.125 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6010999999999999e-05 2024-09-18T04:37:56.125 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.125 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.125 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:56.125 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803609+0000", 2024-09-18T04:37:56.125 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.64e-05 2024-09-18T04:37:56.125 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.125 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.125 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:56.126 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803671+0000", 2024-09-18T04:37:56.126 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1467e-05 2024-09-18T04:37:56.126 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.126 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.126 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:56.126 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.805676+0000", 2024-09-18T04:37:56.126 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020047849999999998 2024-09-18T04:37:56.126 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.126 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.126 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:56.126 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.805724+0000", 2024-09-18T04:37:56.126 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8151e-05 2024-09-18T04:37:56.126 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.126 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.127 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.127 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.127 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.127 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:118 1.0 e15/14 1:30306672:devicehealth::main.db.0000000000000000:head v 15'80, mlcod=15'78)", 2024-09-18T04:37:56.127 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.807930+0000", 2024-09-18T04:37:56.127 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 28.211340932999999, 2024-09-18T04:37:56.127 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028002259999999998, 2024-09-18T04:37:56.127 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:56.127 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:56.127 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:56.127 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.127 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:56.127 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.807930+0000", 2024-09-18T04:37:56.127 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:56.128 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.128 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.128 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:56.128 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.807930+0000", 2024-09-18T04:37:56.128 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:56.128 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.128 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.128 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:56.128 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.807929+0000", 2024-09-18T04:37:56.128 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:37:56.128 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.129 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.129 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:56.129 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808024+0000", 2024-09-18T04:37:56.129 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5258000000000002e-05 2024-09-18T04:37:56.129 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.129 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.129 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:56.129 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808044+0000", 2024-09-18T04:37:56.129 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0203e-05 2024-09-18T04:37:56.129 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.129 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.129 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:56.129 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808053+0000", 2024-09-18T04:37:56.129 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2689999999999999e-06 2024-09-18T04:37:56.130 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.130 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.130 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:56.130 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808137+0000", 2024-09-18T04:37:56.130 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4441e-05 2024-09-18T04:37:56.130 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.130 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.130 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:56.130 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808172+0000", 2024-09-18T04:37:56.130 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4453999999999999e-05 2024-09-18T04:37:56.130 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.130 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.130 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:56.130 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.810663+0000", 2024-09-18T04:37:56.131 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024910269999999998 2024-09-18T04:37:56.131 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.131 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.131 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:56.131 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.810730+0000", 2024-09-18T04:37:56.131 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7658999999999997e-05 2024-09-18T04:37:56.131 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.131 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.131 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.131 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.131 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.131 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:129 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'87, mlcod=15'85)", 2024-09-18T04:37:56.131 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.231689+0000", 2024-09-18T04:37:56.131 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 27.787581920000001, 2024-09-18T04:37:56.132 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002376502, 2024-09-18T04:37:56.132 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:56.132 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:56.132 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:56.132 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.132 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:56.132 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231689+0000", 2024-09-18T04:37:56.132 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:56.132 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.132 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.132 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:56.132 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231689+0000", 2024-09-18T04:37:56.132 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:56.133 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.133 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.133 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:56.133 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231687+0000", 2024-09-18T04:37:56.133 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:37:56.133 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.133 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.133 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:56.133 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231702+0000", 2024-09-18T04:37:56.133 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4949999999999999e-05 2024-09-18T04:37:56.133 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.133 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.133 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:56.133 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231735+0000", 2024-09-18T04:37:56.133 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2276000000000001e-05 2024-09-18T04:37:56.134 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.134 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.134 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:56.134 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231750+0000", 2024-09-18T04:37:56.134 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5193e-05 2024-09-18T04:37:56.134 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.134 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.134 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:56.134 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231872+0000", 2024-09-18T04:37:56.134 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000122127 2024-09-18T04:37:56.134 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.134 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.134 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:56.135 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231936+0000", 2024-09-18T04:37:56.135 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4089000000000003e-05 2024-09-18T04:37:56.135 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.135 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.135 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:56.135 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234002+0000", 2024-09-18T04:37:56.135 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002065575 2024-09-18T04:37:56.135 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.135 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.135 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:56.135 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234066+0000", 2024-09-18T04:37:56.135 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3944999999999998e-05 2024-09-18T04:37:56.135 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.136 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.136 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.136 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.136 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.136 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:130 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'88, mlcod=15'85)", 2024-09-18T04:37:56.136 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.232853+0000", 2024-09-18T04:37:56.136 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 27.786418305000002, 2024-09-18T04:37:56.136 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027351739999999999, 2024-09-18T04:37:56.136 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:56.136 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:56.136 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:56.136 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.136 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:56.137 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232853+0000", 2024-09-18T04:37:56.137 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:56.137 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.137 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.137 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:56.137 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232853+0000", 2024-09-18T04:37:56.137 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:56.137 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.137 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.137 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:56.137 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232851+0000", 2024-09-18T04:37:56.137 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:37:56.137 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.137 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.138 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:56.138 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232866+0000", 2024-09-18T04:37:56.138 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4946000000000001e-05 2024-09-18T04:37:56.138 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.138 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.138 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:56.138 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232897+0000", 2024-09-18T04:37:56.138 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0982999999999998e-05 2024-09-18T04:37:56.138 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.138 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.138 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:56.138 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232915+0000", 2024-09-18T04:37:56.138 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8476000000000001e-05 2024-09-18T04:37:56.138 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.139 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.139 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:56.139 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.233029+0000", 2024-09-18T04:37:56.139 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011305400000000001 2024-09-18T04:37:56.139 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.139 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.139 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:56.139 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.233079+0000", 2024-09-18T04:37:56.139 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0668999999999999e-05 2024-09-18T04:37:56.139 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.139 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.139 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:56.139 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235534+0000", 2024-09-18T04:37:56.140 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024546030000000001 2024-09-18T04:37:56.140 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.140 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.140 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:56.140 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235588+0000", 2024-09-18T04:37:56.140 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4098999999999998e-05 2024-09-18T04:37:56.140 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.140 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.140 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.140 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.140 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.140 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:131 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'89, mlcod=15'85)", 2024-09-18T04:37:56.140 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.234089+0000", 2024-09-18T04:37:56.140 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 27.785182451000001, 2024-09-18T04:37:56.141 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026975609999999998, 2024-09-18T04:37:56.141 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:56.141 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:56.141 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:56.141 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.141 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:56.141 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234089+0000", 2024-09-18T04:37:56.141 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:56.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.141 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.141 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:56.141 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234089+0000", 2024-09-18T04:37:56.141 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:56.142 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.142 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.142 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:56.142 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234087+0000", 2024-09-18T04:37:56.142 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:37:56.142 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.142 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.142 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:56.142 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234145+0000", 2024-09-18T04:37:56.142 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8687999999999998e-05 2024-09-18T04:37:56.142 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.142 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.142 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:56.142 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234177+0000", 2024-09-18T04:37:56.143 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1526999999999998e-05 2024-09-18T04:37:56.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.143 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.143 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:56.143 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234190+0000", 2024-09-18T04:37:56.143 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2965999999999999e-05 2024-09-18T04:37:56.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.143 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.143 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:56.143 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234309+0000", 2024-09-18T04:37:56.143 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011875399999999999 2024-09-18T04:37:56.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.143 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.143 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:56.144 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234365+0000", 2024-09-18T04:37:56.144 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6436e-05 2024-09-18T04:37:56.144 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.144 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.144 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:56.144 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236675+0000", 2024-09-18T04:37:56.144 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023099520000000001 2024-09-18T04:37:56.144 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.144 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.144 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:56.144 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236786+0000", 2024-09-18T04:37:56.144 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000111025 2024-09-18T04:37:56.145 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.145 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.145 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.145 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.145 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.145 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:132 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'90, mlcod=15'85)", 2024-09-18T04:37:56.145 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.235300+0000", 2024-09-18T04:37:56.145 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 27.783971001000001, 2024-09-18T04:37:56.145 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025982599999999998, 2024-09-18T04:37:56.145 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:56.145 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:56.145 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:56.145 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.146 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:56.146 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235300+0000", 2024-09-18T04:37:56.146 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:56.146 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.146 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.146 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:56.146 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235300+0000", 2024-09-18T04:37:56.146 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:56.146 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.146 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.146 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:56.146 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235298+0000", 2024-09-18T04:37:56.146 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:37:56.147 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.147 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.147 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:56.147 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235318+0000", 2024-09-18T04:37:56.147 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9714000000000001e-05 2024-09-18T04:37:56.147 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.147 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.147 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:56.147 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235337+0000", 2024-09-18T04:37:56.147 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8892000000000001e-05 2024-09-18T04:37:56.147 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.147 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.147 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:56.147 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235348+0000", 2024-09-18T04:37:56.148 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.129e-05 2024-09-18T04:37:56.148 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.148 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.148 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:56.148 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235681+0000", 2024-09-18T04:37:56.148 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00033323000000000001 2024-09-18T04:37:56.148 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.148 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.148 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:56.148 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235733+0000", 2024-09-18T04:37:56.148 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1391999999999999e-05 2024-09-18T04:37:56.148 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.148 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.149 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:56.149 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237839+0000", 2024-09-18T04:37:56.149 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002105647 2024-09-18T04:37:56.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.149 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.149 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:56.149 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237898+0000", 2024-09-18T04:37:56.149 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9759000000000002e-05 2024-09-18T04:37:56.149 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.149 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.149 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.149 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.150 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:133 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'91, mlcod=15'86)", 2024-09-18T04:37:56.150 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.236920+0000", 2024-09-18T04:37:56.150 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 27.782351113000001, 2024-09-18T04:37:56.150 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024318410000000001, 2024-09-18T04:37:56.150 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:56.150 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:56.150 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:56.150 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.150 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:56.150 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236920+0000", 2024-09-18T04:37:56.150 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:56.150 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.150 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.151 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:56.151 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236920+0000", 2024-09-18T04:37:56.151 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:56.151 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.151 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.151 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:56.151 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236918+0000", 2024-09-18T04:37:56.151 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:37:56.151 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.151 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.151 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:56.151 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236931+0000", 2024-09-18T04:37:56.151 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2683999999999999e-05 2024-09-18T04:37:56.152 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.152 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.152 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:56.152 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236951+0000", 2024-09-18T04:37:56.152 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0292000000000002e-05 2024-09-18T04:37:56.152 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.152 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.152 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:56.152 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236962+0000", 2024-09-18T04:37:56.152 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0703000000000001e-05 2024-09-18T04:37:56.152 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.152 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.152 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:56.153 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237095+0000", 2024-09-18T04:37:56.153 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013312400000000001 2024-09-18T04:37:56.153 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.153 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.153 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:56.153 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237133+0000", 2024-09-18T04:37:56.153 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7496000000000001e-05 2024-09-18T04:37:56.153 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.153 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.153 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:56.153 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.239206+0000", 2024-09-18T04:37:56.153 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020730290000000001 2024-09-18T04:37:56.154 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.154 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.154 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:56.154 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.239352+0000", 2024-09-18T04:37:56.154 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014593000000000001 2024-09-18T04:37:56.154 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.154 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.154 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.154 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.154 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.154 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:134 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'92, mlcod=15'86)", 2024-09-18T04:37:56.154 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.237894+0000", 2024-09-18T04:37:56.155 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 27.781377091, 2024-09-18T04:37:56.155 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025299519999999998, 2024-09-18T04:37:56.155 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:56.155 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:56.155 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:56.155 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.155 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:56.155 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237894+0000", 2024-09-18T04:37:56.155 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:56.155 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.155 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.155 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:56.156 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237894+0000", 2024-09-18T04:37:56.156 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:56.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.156 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.156 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:56.156 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237892+0000", 2024-09-18T04:37:56.156 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:37:56.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.156 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.156 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:56.156 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237986+0000", 2024-09-18T04:37:56.156 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3341e-05 2024-09-18T04:37:56.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.156 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.157 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:56.157 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.238012+0000", 2024-09-18T04:37:56.157 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5843e-05 2024-09-18T04:37:56.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.157 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.157 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:56.157 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.238027+0000", 2024-09-18T04:37:56.157 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5469e-05 2024-09-18T04:37:56.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.157 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.157 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:56.157 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.238114+0000", 2024-09-18T04:37:56.157 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7483000000000002e-05 2024-09-18T04:37:56.158 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.158 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.158 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:56.158 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.238152+0000", 2024-09-18T04:37:56.158 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7565999999999999e-05 2024-09-18T04:37:56.158 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.158 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.158 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:56.158 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.240297+0000", 2024-09-18T04:37:56.158 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021447620000000001 2024-09-18T04:37:56.158 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.158 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.158 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:56.159 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.240424+0000", 2024-09-18T04:37:56.159 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012705400000000001 2024-09-18T04:37:56.159 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.159 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.159 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.159 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.159 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:139 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'97, mlcod=15'93)", 2024-09-18T04:37:56.159 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.245692+0000", 2024-09-18T04:37:56.159 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 27.773578950000001, 2024-09-18T04:37:56.159 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024260480000000001, 2024-09-18T04:37:56.159 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:56.159 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:56.160 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:56.160 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.160 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:56.160 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245692+0000", 2024-09-18T04:37:56.160 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:56.160 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.160 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.160 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:56.160 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245692+0000", 2024-09-18T04:37:56.160 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:56.160 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.160 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.161 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:56.161 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245690+0000", 2024-09-18T04:37:56.161 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:37:56.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.161 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.161 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:56.161 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245712+0000", 2024-09-18T04:37:56.161 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1916000000000002e-05 2024-09-18T04:37:56.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.161 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.161 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:56.161 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245741+0000", 2024-09-18T04:37:56.161 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8503000000000001e-05 2024-09-18T04:37:56.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.162 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.162 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:56.162 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245759+0000", 2024-09-18T04:37:56.162 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.872e-05 2024-09-18T04:37:56.162 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.162 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.162 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:56.162 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245859+0000", 2024-09-18T04:37:56.162 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9641000000000004e-05 2024-09-18T04:37:56.162 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.162 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.162 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:56.162 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245900+0000", 2024-09-18T04:37:56.162 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0589000000000001e-05 2024-09-18T04:37:56.163 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.163 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.163 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:56.163 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.247961+0000", 2024-09-18T04:37:56.163 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020619190000000002 2024-09-18T04:37:56.163 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.163 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.163 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:56.163 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.248118+0000", 2024-09-18T04:37:56.163 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015664300000000001 2024-09-18T04:37:56.163 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.163 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.163 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.163 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.163 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.164 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:144 1.0 e15/14 1:30306672:devicehealth::main.db.0000000000000000:head v 15'102, mlcod=15'98)", 2024-09-18T04:37:56.164 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.252149+0000", 2024-09-18T04:37:56.164 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 27.767121884000002, 2024-09-18T04:37:56.164 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024174959999999999, 2024-09-18T04:37:56.164 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:56.164 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:56.164 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:56.164 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.164 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:56.164 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252149+0000", 2024-09-18T04:37:56.164 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:56.164 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.164 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.165 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:56.165 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252149+0000", 2024-09-18T04:37:56.165 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:56.165 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.165 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.165 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:56.165 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252148+0000", 2024-09-18T04:37:56.165 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:37:56.165 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.165 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.165 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:56.165 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252225+0000", 2024-09-18T04:37:56.165 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6576999999999995e-05 2024-09-18T04:37:56.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.166 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.166 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:56.166 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252242+0000", 2024-09-18T04:37:56.166 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7740999999999998e-05 2024-09-18T04:37:56.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.166 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.166 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:56.166 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252254+0000", 2024-09-18T04:37:56.166 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.137e-05 2024-09-18T04:37:56.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.166 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.166 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:56.167 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252385+0000", 2024-09-18T04:37:56.167 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013091099999999999 2024-09-18T04:37:56.167 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.167 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.167 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:56.167 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252426+0000", 2024-09-18T04:37:56.167 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1535999999999997e-05 2024-09-18T04:37:56.167 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.167 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.167 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:56.167 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.254498+0000", 2024-09-18T04:37:56.167 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002071477 2024-09-18T04:37:56.167 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.168 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.168 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:56.168 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.254567+0000", 2024-09-18T04:37:56.168 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9014e-05 2024-09-18T04:37:56.168 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.168 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.168 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.168 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.168 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.168 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:37:56.170 INFO:tasks.thrashosds.thrasher:j is {'epoch': 26, 'fsid': 'be6146ba-a991-4f6e-b359-99cbedc9f9b1', 'created': '2024-09-18T04:37:15.676634+0000', 'modified': '2024-09-18T04:37:53.669818+0000', 'last_up_change': '2024-09-18T04:37:25.403175+0000', 'last_in_change': '2024-09-18T04:37:50.652169+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-18T04:37:25.654950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-18T04:37:32.672462+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 20, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '6177002d-72e6-4888-bcde-29b6a92d9dc4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6801', 'nonce': 1554407851}, {'type': 'v1', 'addr': '172.21.15.28:6803', 'nonce': 1554407851}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6805', 'nonce': 1554407851}, {'type': 'v1', 'addr': '172.21.15.28:6807', 'nonce': 1554407851}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6813', 'nonce': 1554407851}, {'type': 'v1', 'addr': '172.21.15.28:6815', 'nonce': 1554407851}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6809', 'nonce': 1554407851}, {'type': 'v1', 'addr': '172.21.15.28:6811', 'nonce': 1554407851}]}, 'public_addr': '172.21.15.28:6803/1554407851', 'cluster_addr': '172.21.15.28:6807/1554407851', 'heartbeat_back_addr': '172.21.15.28:6815/1554407851', 'heartbeat_front_addr': '172.21.15.28:6811/1554407851', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '4f62cb09-3457-418c-bb69-491d9b70914c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 21, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6800', 'nonce': 511141401}, {'type': 'v1', 'addr': '172.21.15.28:6802', 'nonce': 511141401}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6804', 'nonce': 511141401}, {'type': 'v1', 'addr': '172.21.15.28:6806', 'nonce': 511141401}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6812', 'nonce': 511141401}, {'type': 'v1', 'addr': '172.21.15.28:6814', 'nonce': 511141401}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6808', 'nonce': 511141401}, {'type': 'v1', 'addr': '172.21.15.28:6810', 'nonce': 511141401}]}, 'public_addr': '172.21.15.28:6802/511141401', 'cluster_addr': '172.21.15.28:6806/511141401', 'heartbeat_back_addr': '172.21.15.28:6814/511141401', 'heartbeat_front_addr': '172.21.15.28:6810/511141401', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '854fe031-3b87-4290-9c2f-3fb196fb806c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 24, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6816', 'nonce': 2623426735}, {'type': 'v1', 'addr': '172.21.15.28:6817', 'nonce': 2623426735}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6818', 'nonce': 2623426735}, {'type': 'v1', 'addr': '172.21.15.28:6819', 'nonce': 2623426735}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6822', 'nonce': 2623426735}, {'type': 'v1', 'addr': '172.21.15.28:6823', 'nonce': 2623426735}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6820', 'nonce': 2623426735}, {'type': 'v1', 'addr': '172.21.15.28:6821', 'nonce': 2623426735}]}, 'public_addr': '172.21.15.28:6817/2623426735', 'cluster_addr': '172.21.15.28:6819/2623426735', 'heartbeat_back_addr': '172.21.15.28:6823/2623426735', 'heartbeat_front_addr': '172.21.15.28:6821/2623426735', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'c2625f98-780b-4f86-bf36-a48d2a227756', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 17, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6808', 'nonce': 2746051132}, {'type': 'v1', 'addr': '172.21.15.82:6809', 'nonce': 2746051132}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6810', 'nonce': 2746051132}, {'type': 'v1', 'addr': '172.21.15.82:6811', 'nonce': 2746051132}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6814', 'nonce': 2746051132}, {'type': 'v1', 'addr': '172.21.15.82:6815', 'nonce': 2746051132}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6812', 'nonce': 2746051132}, {'type': 'v1', 'addr': '172.21.15.82:6813', 'nonce': 2746051132}]}, 'public_addr': '172.21.15.82:6809/2746051132', 'cluster_addr': '172.21.15.82:6811/2746051132', 'heartbeat_back_addr': '172.21.15.82:6815/2746051132', 'heartbeat_front_addr': '172.21.15.82:6813/2746051132', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'deb22d86-4081-484d-96eb-9cb9d96b60bc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6800', 'nonce': 2005712460}, {'type': 'v1', 'addr': '172.21.15.82:6801', 'nonce': 2005712460}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6802', 'nonce': 2005712460}, {'type': 'v1', 'addr': '172.21.15.82:6803', 'nonce': 2005712460}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6806', 'nonce': 2005712460}, {'type': 'v1', 'addr': '172.21.15.82:6807', 'nonce': 2005712460}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6804', 'nonce': 2005712460}, {'type': 'v1', 'addr': '172.21.15.82:6805', 'nonce': 2005712460}]}, 'public_addr': '172.21.15.82:6801/2005712460', 'cluster_addr': '172.21.15.82:6803/2005712460', 'heartbeat_back_addr': '172.21.15.82:6807/2005712460', 'heartbeat_front_addr': '172.21.15.82:6805/2005712460', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '061d332c-f51b-403d-bb18-eba415c97e7f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 24, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6816', 'nonce': 1267644232}, {'type': 'v1', 'addr': '172.21.15.82:6817', 'nonce': 1267644232}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6818', 'nonce': 1267644232}, {'type': 'v1', 'addr': '172.21.15.82:6819', 'nonce': 1267644232}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6822', 'nonce': 1267644232}, {'type': 'v1', 'addr': '172.21.15.82:6823', 'nonce': 1267644232}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6820', 'nonce': 1267644232}, {'type': 'v1', 'addr': '172.21.15.82:6821', 'nonce': 1267644232}]}, 'public_addr': '172.21.15.82:6817/1267644232', 'cluster_addr': '172.21.15.82:6819/1267644232', 'heartbeat_back_addr': '172.21.15.82:6823/1267644232', 'heartbeat_front_addr': '172.21.15.82:6821/1267644232', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T04:37:22.192087+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T04:37:22.017833+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T04:37:22.755774+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T04:37:22.663785+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T04:37:22.531758+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T04:37:22.400774+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-18T04:37:56.170 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T04:37:56.171 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:37:56.171 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:37:56.072+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:37:56.214 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:37:56.212+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:37:56.276 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:37:56.272+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:37:56.326 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:37:56.326 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:37:56.326 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:37:56.326 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:37:56.336 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:37:56.382 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:37:56.380+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:37:56.435 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:37:56.435 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:37:56.435 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:37:56.436 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:37:56.436 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:37:56.445 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:37:56.479 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:37:56.476+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:37:56.519 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:37:56.522 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-18T04:37:56.534 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2024-09-18T04:37:55.598185+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":19,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":180076,"kb_used_data":1564,"kb_used_omap":9,"kb_used_meta":178422,"kb_avail":566050964,"statfs":{"total":579820584960,"available":579636187136,"internally_reserved":0,"allocated":1601536,"data_stored":1677790,"data_compressed":66040,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":9561,"internal_metadata":182704807},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001743"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-18T04:37:51.670359+0000","last_change":"2024-09-18T04:37:36.484517+0000","last_active":"2024-09-18T04:37:51.670359+0000","last_peered":"2024-09-18T04:37:51.670359+0000","last_clean":"2024-09-18T04:37:51.670359+0000","last_became_active":"2024-09-18T04:37:34.893674+0000","last_became_peered":"2024-09-18T04:37:34.893674+0000","last_unstale":"2024-09-18T04:37:51.670359+0000","last_undegraded":"2024-09-18T04:37:51.670359+0000","last_fullsized":"2024-09-18T04:37:51.670359+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T09:53:33.883648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047343899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-18T04:37:51.678111+0000","last_change":"2024-09-18T04:37:36.494193+0000","last_active":"2024-09-18T04:37:51.678111+0000","last_peered":"2024-09-18T04:37:51.678111+0000","last_clean":"2024-09-18T04:37:51.678111+0000","last_became_active":"2024-09-18T04:37:34.472940+0000","last_became_peered":"2024-09-18T04:37:34.472940+0000","last_unstale":"2024-09-18T04:37:51.678111+0000","last_undegraded":"2024-09-18T04:37:51.678111+0000","last_fullsized":"2024-09-18T04:37:51.678111+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:18:29.266815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070553900000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-18T04:37:51.670264+0000","last_change":"2024-09-18T04:37:36.484508+0000","last_active":"2024-09-18T04:37:51.670264+0000","last_peered":"2024-09-18T04:37:51.670264+0000","last_clean":"2024-09-18T04:37:51.670264+0000","last_became_active":"2024-09-18T04:37:34.468968+0000","last_became_peered":"2024-09-18T04:37:34.468968+0000","last_unstale":"2024-09-18T04:37:51.670264+0000","last_undegraded":"2024-09-18T04:37:51.670264+0000","last_fullsized":"2024-09-18T04:37:51.670264+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T08:46:16.183136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059422499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-18T04:37:51.677539+0000","last_change":"2024-09-18T04:37:36.494228+0000","last_active":"2024-09-18T04:37:51.677539+0000","last_peered":"2024-09-18T04:37:51.677539+0000","last_clean":"2024-09-18T04:37:51.677539+0000","last_became_active":"2024-09-18T04:37:34.471265+0000","last_became_peered":"2024-09-18T04:37:34.471265+0000","last_unstale":"2024-09-18T04:37:51.677539+0000","last_undegraded":"2024-09-18T04:37:51.677539+0000","last_fullsized":"2024-09-18T04:37:51.677539+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:23:05.748076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081514499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"20'2","reported_seq":22,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-18T04:37:47.836960+0000","last_change":"2024-09-18T04:37:36.490461+0000","last_active":"2024-09-18T04:37:47.836960+0000","last_peered":"2024-09-18T04:37:47.836960+0000","last_clean":"2024-09-18T04:37:47.836960+0000","last_became_active":"2024-09-18T04:37:34.473088+0000","last_became_peered":"2024-09-18T04:37:34.473088+0000","last_unstale":"2024-09-18T04:37:47.836960+0000","last_undegraded":"2024-09-18T04:37:47.836960+0000","last_fullsized":"2024-09-18T04:37:47.836960+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:17:57.352397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001879932,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-18T04:37:53.681898+0000","last_change":"2024-09-18T04:37:51.679120+0000","last_active":"2024-09-18T04:37:53.681898+0000","last_peered":"2024-09-18T04:37:53.681898+0000","last_clean":"2024-09-18T04:37:53.681898+0000","last_became_active":"2024-09-18T04:37:51.678711+0000","last_became_peered":"2024-09-18T04:37:51.678711+0000","last_unstale":"2024-09-18T04:37:53.681898+0000","last_undegraded":"2024-09-18T04:37:53.681898+0000","last_fullsized":"2024-09-18T04:37:53.681898+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T04:46:09.459905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-18T04:37:51.670160+0000","last_change":"2024-09-18T04:37:36.484950+0000","last_active":"2024-09-18T04:37:51.670160+0000","last_peered":"2024-09-18T04:37:51.670160+0000","last_clean":"2024-09-18T04:37:51.670160+0000","last_became_active":"2024-09-18T04:37:34.470841+0000","last_became_peered":"2024-09-18T04:37:34.470841+0000","last_unstale":"2024-09-18T04:37:51.670160+0000","last_undegraded":"2024-09-18T04:37:51.670160+0000","last_fullsized":"2024-09-18T04:37:51.670160+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:16:34.528783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090513900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"18'1","reported_seq":33,"reported_epoch":24,"state":"peering","last_fresh":"2024-09-18T04:37:50.669992+0000","last_change":"2024-09-18T04:37:50.670209+0000","last_active":"2024-09-18T04:37:50.669992+0000","last_peered":"2024-09-18T04:37:47.836960+0000","last_clean":"2024-09-18T04:37:47.836960+0000","last_became_active":"2024-09-18T04:37:45.426803+0000","last_became_peered":"2024-09-18T04:37:45.426803+0000","last_unstale":"2024-09-18T04:37:50.669992+0000","last_undegraded":"2024-09-18T04:37:50.669992+0000","last_fullsized":"2024-09-18T04:37:50.669992+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T07:40:38.786085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018293420000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":169,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-18T04:37:51.670210+0000","last_change":"2024-09-18T04:37:27.423069+0000","last_active":"2024-09-18T04:37:51.670210+0000","last_peered":"2024-09-18T04:37:51.670210+0000","last_clean":"2024-09-18T04:37:51.670210+0000","last_became_active":"2024-09-18T04:37:27.422350+0000","last_became_peered":"2024-09-18T04:37:27.422350+0000","last_unstale":"2024-09-18T04:37:51.670210+0000","last_undegraded":"2024-09-18T04:37:51.670210+0000","last_fullsized":"2024-09-18T04:37:51.670210+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:26.406292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:26.406292+0000","last_clean_scrub_stamp":"2024-09-18T04:37:26.406292+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:31:45.852149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":66040,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":229376,"data_stored":75978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78800000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]}]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":225280,"data_stored":75959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96999999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83699999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574858,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":290816,"data_stored":666327,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70599999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90800000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]}]},{"osd":3,"up_from":12,"seq":51539607562,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31440,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340400,"statfs":{"total":96636764160,"available":96604569600,"internally_reserved":0,"allocated":327680,"data_stored":680081,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93500000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66300000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]}]},{"osd":2,"up_from":12,"seq":51539607560,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31376,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340464,"statfs":{"total":96636764160,"available":96604635136,"internally_reserved":0,"allocated":262144,"data_stored":89713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94499999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.012}]}]},{"osd":1,"up_from":12,"seq":51539607562,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31376,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340464,"statfs":{"total":96636764160,"available":96604635136,"internally_reserved":0,"allocated":266240,"data_stored":89732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81999999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.071}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0509999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T04:37:56.535 INFO:tasks.thrashosds.thrasher:Setting 2.5 to [2, 0] 2024-09-18T04:37:56.536 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '2.5', '2', '0'] 2024-09-18T04:37:56.536 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 2.5 2 0 2024-09-18T04:37:56.550 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:37:56.550 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:37:56.550 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:37:56.550 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:37:56.550 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.550 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4214.0:5 2.2 e19/17 2:5cea7035:::rbd_info:head v 19'1, mlcod=0'0)", 2024-09-18T04:37:56.550 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:35.471260+0000", 2024-09-18T04:37:56.551 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 21.078298958000001, 2024-09-18T04:37:56.551 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021025950000000001, 2024-09-18T04:37:56.551 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:56.551 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:56.551 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:56.551 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.551 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:56.551 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471260+0000", 2024-09-18T04:37:56.551 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:56.551 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.551 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.552 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:56.552 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471260+0000", 2024-09-18T04:37:56.552 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:56.552 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.552 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.552 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:56.552 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471259+0000", 2024-09-18T04:37:56.552 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:37:56.552 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.552 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.553 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:56.553 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471275+0000", 2024-09-18T04:37:56.553 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6135999999999999e-05 2024-09-18T04:37:56.553 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.553 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.553 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:56.553 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471298+0000", 2024-09-18T04:37:56.553 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3632999999999999e-05 2024-09-18T04:37:56.553 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.553 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.553 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:56.554 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471320+0000", 2024-09-18T04:37:56.554 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1852999999999999e-05 2024-09-18T04:37:56.554 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.554 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.554 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:56.554 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471445+0000", 2024-09-18T04:37:56.554 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012461699999999999 2024-09-18T04:37:56.554 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.554 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.554 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:56.554 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471529+0000", 2024-09-18T04:37:56.555 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3875000000000006e-05 2024-09-18T04:37:56.555 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.555 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.555 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:56.555 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.473317+0000", 2024-09-18T04:37:56.555 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017886899999999999 2024-09-18T04:37:56.555 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.555 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.555 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:56.555 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.473362+0000", 2024-09-18T04:37:56.555 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4724000000000001e-05 2024-09-18T04:37:56.555 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.556 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.556 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.556 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.556 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.556 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4214.0:7 2.2 e20/17 2:5cea7035:::rbd_info:head v 20'2, mlcod=0'0)", 2024-09-18T04:37:56.556 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:36.490560+0000", 2024-09-18T04:37:56.556 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 20.058998818999999, 2024-09-18T04:37:56.556 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0015326020000000001, 2024-09-18T04:37:56.556 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:56.556 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:56.557 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:56.557 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.557 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:56.557 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490560+0000", 2024-09-18T04:37:56.557 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:56.557 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.557 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.557 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:56.557 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490560+0000", 2024-09-18T04:37:56.557 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:56.557 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.558 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.558 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:56.558 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490559+0000", 2024-09-18T04:37:56.558 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:37:56.558 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.558 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.558 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:56.558 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490568+0000", 2024-09-18T04:37:56.558 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5699999999999993e-06 2024-09-18T04:37:56.558 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.559 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.559 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:56.559 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490599+0000", 2024-09-18T04:37:56.559 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1566000000000002e-05 2024-09-18T04:37:56.559 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.559 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.559 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:56.559 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490609+0000", 2024-09-18T04:37:56.559 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.8700000000000004e-06 2024-09-18T04:37:56.559 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.559 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.559 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:56.560 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490708+0000", 2024-09-18T04:37:56.560 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9139000000000001e-05 2024-09-18T04:37:56.560 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.560 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.560 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:56.560 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490736+0000", 2024-09-18T04:37:56.560 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7617e-05 2024-09-18T04:37:56.560 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.560 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.560 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:56.560 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.492016+0000", 2024-09-18T04:37:56.560 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0012796579999999999 2024-09-18T04:37:56.560 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.561 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.561 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:56.561 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.492092+0000", 2024-09-18T04:37:56.561 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6571999999999998e-05 2024-09-18T04:37:56.561 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.561 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.561 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.561 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.561 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.561 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:37:56.563 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:37:56.606 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:37:56.604+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:37:56.681 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:37:56.674+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:37:56.689 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-09-18T04:37:56.719 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:37:56.719 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:37:56.719 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:37:56.719 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:37:56.729 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:37:56.783 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:37:56.778+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:37:56.829 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:37:56.829 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:37:56.830 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:37:56.830 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:37:56.830 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:37:56.839 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:37:56.883 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:37:56.878+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:37:56.945 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:37:56.945 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:37:56.945 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:37:56.945 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:37:56.945 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.946 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4214.0:1 2.3 e18/17 2:c4c92e5a:::rbd_trash:head v 18'1, mlcod=0'0)", 2024-09-18T04:37:56.946 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:34.475123+0000", 2024-09-18T04:37:56.946 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.469286308000001, 2024-09-18T04:37:56.946 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00081394599999999996, 2024-09-18T04:37:56.946 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:56.946 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:56.946 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:56.946 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.946 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:56.946 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475123+0000", 2024-09-18T04:37:56.946 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:56.946 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.946 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.946 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:56.947 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475123+0000", 2024-09-18T04:37:56.947 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:56.947 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.947 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.947 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:56.947 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475121+0000", 2024-09-18T04:37:56.947 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:37:56.947 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.947 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.947 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:56.947 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475149+0000", 2024-09-18T04:37:56.947 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7155999999999999e-05 2024-09-18T04:37:56.947 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.947 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.947 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:56.948 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475165+0000", 2024-09-18T04:37:56.948 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6160000000000001e-05 2024-09-18T04:37:56.948 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.948 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.948 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:56.948 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475183+0000", 2024-09-18T04:37:56.948 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7918e-05 2024-09-18T04:37:56.948 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.948 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.948 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:56.948 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475246+0000", 2024-09-18T04:37:56.948 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3176000000000006e-05 2024-09-18T04:37:56.948 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.949 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.949 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:56.949 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475279+0000", 2024-09-18T04:37:56.949 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3192999999999999e-05 2024-09-18T04:37:56.949 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.949 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.949 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:37:56.949 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475871+0000", 2024-09-18T04:37:56.949 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00059206099999999995 2024-09-18T04:37:56.949 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.949 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.949 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:56.949 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475937+0000", 2024-09-18T04:37:56.950 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5439999999999997e-05 2024-09-18T04:37:56.950 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.950 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.950 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.950 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.950 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.950 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 25/24 [PushOp(2:c4c92e5a:::rbd_trash:head, version: 18'1, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:c4c92e5a:::rbd_trash:head@18'1, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-18T04:37:56.950 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:51.680544+0000", 2024-09-18T04:37:56.950 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 5.2638644660000002, 2024-09-18T04:37:56.950 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00050365200000000005, 2024-09-18T04:37:56.950 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:37:56.950 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:37:56.950 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:37:56.950 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.951 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:37:56.951 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680544+0000", 2024-09-18T04:37:56.951 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:56.951 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.951 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.951 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:37:56.951 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680544+0000", 2024-09-18T04:37:56.951 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:37:56.951 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.951 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.951 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:37:56.951 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680543+0000", 2024-09-18T04:37:56.951 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:37:56.951 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.951 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.952 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:37:56.952 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680571+0000", 2024-09-18T04:37:56.952 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7953000000000002e-05 2024-09-18T04:37:56.952 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.952 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.952 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:37:56.952 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680605+0000", 2024-09-18T04:37:56.952 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3933000000000002e-05 2024-09-18T04:37:56.952 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.952 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.952 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:37:56.952 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680636+0000", 2024-09-18T04:37:56.952 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1288999999999998e-05 2024-09-18T04:37:56.952 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.953 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.953 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:37:56.953 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680748+0000", 2024-09-18T04:37:56.953 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000111431 2024-09-18T04:37:56.953 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.953 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.953 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:37:56.953 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680758+0000", 2024-09-18T04:37:56.953 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0244999999999999e-05 2024-09-18T04:37:56.953 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:37:56.953 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:37:56.953 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:37:56.953 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.681048+0000", 2024-09-18T04:37:56.954 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00029003399999999999 2024-09-18T04:37:56.954 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.954 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.954 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.954 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:37:56.954 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:37:56.954 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:37:56.955 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:37:56.984 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:37:56.980+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:37:57.071 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:37:57.071 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:37:57.071 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:37:57.071 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:37:57.081 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:37:57.084 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:37:57.078+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:37:57.185 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:37:57.184+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:37:57.191 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:37:57.191 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:37:57.191 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:37:57.191 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:37:57.191 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:37:57.201 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:37:57.286 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:37:57.284+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:37:57.312 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:37:57.313 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:37:57.313 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:37:57.313 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:37:57.313 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.313 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:57.313 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.573328+0000", 2024-09-18T04:37:57.313 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 29.733057037999998, 2024-09-18T04:37:57.313 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011397898, 2024-09-18T04:37:57.313 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:57.314 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:57.314 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:57.314 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:57.314 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:57.314 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 86 2024-09-18T04:37:57.314 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.314 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:57.314 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.314 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:57.315 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573328+0000", 2024-09-18T04:37:57.315 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:57.315 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.315 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.315 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:57.315 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573328+0000", 2024-09-18T04:37:57.315 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:57.315 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.315 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.315 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:57.315 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573327+0000", 2024-09-18T04:37:57.315 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:37:57.315 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.315 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.315 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:57.316 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573331+0000", 2024-09-18T04:37:57.316 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9750000000000001e-06 2024-09-18T04:37:57.316 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.316 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.316 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:57.316 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573346+0000", 2024-09-18T04:37:57.316 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5469999999999999e-05 2024-09-18T04:37:57.316 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.316 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.316 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:57.316 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573370+0000", 2024-09-18T04:37:57.316 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3969999999999999e-05 2024-09-18T04:37:57.316 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.317 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.317 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:57.317 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.578536+0000", 2024-09-18T04:37:57.317 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0051659679999999999 2024-09-18T04:37:57.317 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.317 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.317 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:57.317 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.578689+0000", 2024-09-18T04:37:57.317 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015276299999999999 2024-09-18T04:37:57.317 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.317 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.317 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:57.317 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.578901+0000", 2024-09-18T04:37:57.318 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000211783 2024-09-18T04:37:57.318 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.318 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.318 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:57.318 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.583408+0000", 2024-09-18T04:37:57.318 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045071249999999998 2024-09-18T04:37:57.318 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.318 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.318 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:57.318 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.584645+0000", 2024-09-18T04:37:57.319 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001237007 2024-09-18T04:37:57.319 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.319 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.319 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:57.319 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.584703+0000", 2024-09-18T04:37:57.319 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.817e-05 2024-09-18T04:37:57.319 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.319 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.319 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:57.319 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.584725+0000", 2024-09-18T04:37:57.319 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2155e-05 2024-09-18T04:37:57.319 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:57.319 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:57.320 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:57.320 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.320 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.320 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:57.320 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.573382+0000", 2024-09-18T04:37:57.320 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 29.733002290000002, 2024-09-18T04:37:57.320 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012154974000000001, 2024-09-18T04:37:57.320 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:57.320 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:57.320 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:57.320 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:57.320 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:57.321 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 87 2024-09-18T04:37:57.321 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.321 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:57.321 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.321 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:57.321 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573382+0000", 2024-09-18T04:37:57.321 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:57.321 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.321 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.321 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:57.321 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573382+0000", 2024-09-18T04:37:57.321 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:57.321 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.321 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.322 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:57.322 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573382+0000", 2024-09-18T04:37:57.322 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:37:57.322 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.322 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.322 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:57.322 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573482+0000", 2024-09-18T04:37:57.322 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000100333 2024-09-18T04:37:57.322 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.322 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.322 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:57.322 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573492+0000", 2024-09-18T04:37:57.322 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5389999999999996e-06 2024-09-18T04:37:57.323 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.323 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.323 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:57.323 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573500+0000", 2024-09-18T04:37:57.323 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8929999999999994e-06 2024-09-18T04:37:57.323 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.323 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.323 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:57.323 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.579797+0000", 2024-09-18T04:37:57.323 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0062963239999999998 2024-09-18T04:37:57.323 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.323 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.323 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:57.324 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.579957+0000", 2024-09-18T04:37:57.324 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000160283 2024-09-18T04:37:57.324 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.324 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.324 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:57.324 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.580178+0000", 2024-09-18T04:37:57.324 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00022079200000000001 2024-09-18T04:37:57.324 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.324 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.324 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:57.324 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.583479+0000", 2024-09-18T04:37:57.324 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033007689999999998 2024-09-18T04:37:57.324 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.325 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.325 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:57.325 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585477+0000", 2024-09-18T04:37:57.325 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0019985670000000001 2024-09-18T04:37:57.325 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.325 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.325 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:57.325 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585517+0000", 2024-09-18T04:37:57.325 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9709999999999998e-05 2024-09-18T04:37:57.325 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.325 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.325 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:57.325 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585537+0000", 2024-09-18T04:37:57.326 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0326000000000002e-05 2024-09-18T04:37:57.326 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:57.326 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:57.326 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:57.326 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.326 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.326 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:57.326 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.573508+0000", 2024-09-18T04:37:57.326 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 29.732876584, 2024-09-18T04:37:57.326 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012068476, 2024-09-18T04:37:57.326 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:57.326 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:57.326 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:57.327 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:57.327 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:57.327 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 88 2024-09-18T04:37:57.327 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.327 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:57.327 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.327 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:57.327 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573508+0000", 2024-09-18T04:37:57.327 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:57.327 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.327 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.327 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:57.327 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573508+0000", 2024-09-18T04:37:57.328 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:57.328 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.328 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.328 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:57.328 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573507+0000", 2024-09-18T04:37:57.328 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:37:57.328 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.328 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.328 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:57.328 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573512+0000", 2024-09-18T04:37:57.328 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.2629999999999997e-06 2024-09-18T04:37:57.328 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.329 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.329 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:57.329 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573530+0000", 2024-09-18T04:37:57.329 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8507000000000001e-05 2024-09-18T04:37:57.329 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.329 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.329 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:57.329 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573537+0000", 2024-09-18T04:37:57.329 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.832e-06 2024-09-18T04:37:57.329 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.329 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.329 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:57.329 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.582486+0000", 2024-09-18T04:37:57.329 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0089494770000000008 2024-09-18T04:37:57.330 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.330 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.330 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:57.330 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.582630+0000", 2024-09-18T04:37:57.330 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000144005 2024-09-18T04:37:57.330 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.330 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.330 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:57.330 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.582842+0000", 2024-09-18T04:37:57.330 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00021189200000000001 2024-09-18T04:37:57.330 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.330 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.330 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:57.330 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.584777+0000", 2024-09-18T04:37:57.330 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001934753 2024-09-18T04:37:57.331 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.331 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.331 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:57.331 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585543+0000", 2024-09-18T04:37:57.331 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00076540099999999997 2024-09-18T04:37:57.331 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.331 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.331 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:57.331 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585561+0000", 2024-09-18T04:37:57.331 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8474999999999998e-05 2024-09-18T04:37:57.331 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.331 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.332 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:57.332 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585576+0000", 2024-09-18T04:37:57.332 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5427e-05 2024-09-18T04:37:57.332 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:57.332 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:57.332 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:57.332 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.332 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.332 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:57.332 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786394+0000", 2024-09-18T04:37:57.332 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 29.519990137000001, 2024-09-18T04:37:57.333 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011902085999999999, 2024-09-18T04:37:57.333 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:57.333 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:57.333 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:57.333 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:57.333 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:57.333 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 109 2024-09-18T04:37:57.333 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.333 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:57.333 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.333 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:57.333 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786394+0000", 2024-09-18T04:37:57.333 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:57.333 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.334 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.334 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:57.334 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786394+0000", 2024-09-18T04:37:57.334 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:57.334 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.334 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.334 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:57.334 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786394+0000", 2024-09-18T04:37:57.334 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:37:57.334 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.334 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.334 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:57.334 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786548+0000", 2024-09-18T04:37:57.334 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015463400000000001 2024-09-18T04:37:57.335 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.335 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.335 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:57.335 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786561+0000", 2024-09-18T04:37:57.335 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3093e-05 2024-09-18T04:37:57.335 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.335 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.335 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:57.335 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786571+0000", 2024-09-18T04:37:57.335 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4700000000000008e-06 2024-09-18T04:37:57.335 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.335 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.335 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:57.335 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.793367+0000", 2024-09-18T04:37:57.335 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0067966600000000004 2024-09-18T04:37:57.336 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.336 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.336 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:57.336 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.793526+0000", 2024-09-18T04:37:57.336 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015877 2024-09-18T04:37:57.336 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.336 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.336 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:57.336 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.793752+0000", 2024-09-18T04:37:57.336 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00022555200000000001 2024-09-18T04:37:57.337 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.337 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.337 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:57.337 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798125+0000", 2024-09-18T04:37:57.337 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0043733349999999999 2024-09-18T04:37:57.337 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.337 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.337 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:57.337 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798146+0000", 2024-09-18T04:37:57.337 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1063000000000001e-05 2024-09-18T04:37:57.337 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.337 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.337 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:57.338 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798232+0000", 2024-09-18T04:37:57.338 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5771000000000003e-05 2024-09-18T04:37:57.338 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.338 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.338 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:57.338 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798297+0000", 2024-09-18T04:37:57.338 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.4624999999999999e-05 2024-09-18T04:37:57.338 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:57.338 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:57.338 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:57.338 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.338 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.338 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:57.339 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786584+0000", 2024-09-18T04:37:57.339 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 29.519800058000001, 2024-09-18T04:37:57.339 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012013037000000001, 2024-09-18T04:37:57.339 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:57.339 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:57.339 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:57.339 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:57.339 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:57.339 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 110 2024-09-18T04:37:57.339 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.340 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:57.340 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.340 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:57.340 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786584+0000", 2024-09-18T04:37:57.340 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:57.340 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.340 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.340 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:57.340 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786584+0000", 2024-09-18T04:37:57.340 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:57.341 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.341 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.341 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:57.341 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786584+0000", 2024-09-18T04:37:57.341 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:37:57.341 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.341 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.341 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:57.341 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786590+0000", 2024-09-18T04:37:57.342 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6359999999999997e-06 2024-09-18T04:37:57.342 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.342 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.342 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:57.342 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786599+0000", 2024-09-18T04:37:57.342 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7369999999999995e-06 2024-09-18T04:37:57.342 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.342 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.342 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:57.342 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786606+0000", 2024-09-18T04:37:57.342 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1930000000000004e-06 2024-09-18T04:37:57.342 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.343 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.343 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:57.343 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.795018+0000", 2024-09-18T04:37:57.343 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0084115660000000005 2024-09-18T04:37:57.343 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.343 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.343 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:57.343 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.795184+0000", 2024-09-18T04:37:57.343 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000165833 2024-09-18T04:37:57.343 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.344 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.344 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:57.344 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.795414+0000", 2024-09-18T04:37:57.344 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00023088399999999999 2024-09-18T04:37:57.344 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.344 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.344 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:57.344 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798321+0000", 2024-09-18T04:37:57.344 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0029069539999999998 2024-09-18T04:37:57.344 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.345 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.345 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:57.345 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798404+0000", 2024-09-18T04:37:57.345 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2724000000000003e-05 2024-09-18T04:37:57.345 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.345 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.345 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:57.345 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798488+0000", 2024-09-18T04:37:57.345 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.3505999999999996e-05 2024-09-18T04:37:57.345 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.345 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.346 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:57.346 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798598+0000", 2024-09-18T04:37:57.346 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000109917 2024-09-18T04:37:57.346 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:57.346 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:57.346 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:57.346 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.346 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.346 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:57.347 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786613+0000", 2024-09-18T04:37:57.347 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 29.519771358, 2024-09-18T04:37:57.347 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014245760999999999, 2024-09-18T04:37:57.347 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:57.347 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:57.347 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:57.347 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:57.347 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:57.347 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 111 2024-09-18T04:37:57.347 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.348 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:57.348 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.348 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:57.348 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786613+0000", 2024-09-18T04:37:57.348 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:57.348 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.348 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.348 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:57.349 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786613+0000", 2024-09-18T04:37:57.349 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:57.349 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.349 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.349 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:57.349 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786612+0000", 2024-09-18T04:37:57.349 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:37:57.349 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.349 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.349 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:57.350 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786622+0000", 2024-09-18T04:37:57.350 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4099999999999997e-06 2024-09-18T04:37:57.350 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.350 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.350 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:57.350 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786630+0000", 2024-09-18T04:37:57.350 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4169999999999992e-06 2024-09-18T04:37:57.350 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.350 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.350 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:57.350 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786637+0000", 2024-09-18T04:37:57.350 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8290000000000002e-06 2024-09-18T04:37:57.350 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.350 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.351 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:57.351 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.797099+0000", 2024-09-18T04:37:57.351 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010461747 2024-09-18T04:37:57.351 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.351 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.351 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:57.351 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.797243+0000", 2024-09-18T04:37:57.351 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014388899999999999 2024-09-18T04:37:57.351 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.351 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.351 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:57.351 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.797463+0000", 2024-09-18T04:37:57.351 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000219912 2024-09-18T04:37:57.351 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.352 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.352 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:57.352 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800732+0000", 2024-09-18T04:37:57.352 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0032688719999999999 2024-09-18T04:37:57.352 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.352 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.352 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:57.352 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800751+0000", 2024-09-18T04:37:57.352 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9633999999999999e-05 2024-09-18T04:37:57.352 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.352 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.352 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:57.353 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800824+0000", 2024-09-18T04:37:57.353 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.2805999999999994e-05 2024-09-18T04:37:57.353 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.353 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.353 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:57.353 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800859+0000", 2024-09-18T04:37:57.353 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4952e-05 2024-09-18T04:37:57.353 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:57.353 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:57.353 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:57.353 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.353 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.353 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:57.353 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786649+0000", 2024-09-18T04:37:57.354 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 29.519735949000001, 2024-09-18T04:37:57.354 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.01613442, 2024-09-18T04:37:57.354 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:57.354 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:57.354 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:57.354 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:57.354 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:57.354 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 112 2024-09-18T04:37:57.354 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.354 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:57.354 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.354 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:57.354 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786649+0000", 2024-09-18T04:37:57.354 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:57.355 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.355 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.355 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:57.355 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786649+0000", 2024-09-18T04:37:57.355 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:57.355 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.355 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.355 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:57.355 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786648+0000", 2024-09-18T04:37:57.355 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:37:57.355 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.355 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.355 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:57.355 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786768+0000", 2024-09-18T04:37:57.355 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000120491 2024-09-18T04:37:57.356 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.356 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.356 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:57.356 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786795+0000", 2024-09-18T04:37:57.356 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6556000000000001e-05 2024-09-18T04:37:57.356 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.356 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.356 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:57.356 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786807+0000", 2024-09-18T04:37:57.356 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1739999999999999e-05 2024-09-18T04:37:57.356 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.356 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.356 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:57.357 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798683+0000", 2024-09-18T04:37:57.357 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011876070000000001 2024-09-18T04:37:57.357 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.357 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.357 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:57.357 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798867+0000", 2024-09-18T04:37:57.357 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000184103 2024-09-18T04:37:57.357 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.357 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.357 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:57.357 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.799082+0000", 2024-09-18T04:37:57.357 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00021476000000000001 2024-09-18T04:37:57.357 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.358 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.358 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:57.358 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802610+0000", 2024-09-18T04:37:57.358 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0035282209999999998 2024-09-18T04:37:57.358 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.358 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.358 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:57.358 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802706+0000", 2024-09-18T04:37:57.358 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5922999999999996e-05 2024-09-18T04:37:57.358 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.358 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.358 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:57.358 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802750+0000", 2024-09-18T04:37:57.358 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4511000000000001e-05 2024-09-18T04:37:57.358 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.359 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.359 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:57.359 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802783+0000", 2024-09-18T04:37:57.359 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2752000000000001e-05 2024-09-18T04:37:57.359 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:57.359 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:57.359 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:57.359 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.359 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.359 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:57.359 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786816+0000", 2024-09-18T04:37:57.359 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 29.519568882000002, 2024-09-18T04:37:57.360 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016197989999999999, 2024-09-18T04:37:57.360 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:57.360 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:57.360 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:57.360 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:57.360 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:57.360 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 113 2024-09-18T04:37:57.360 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.360 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:57.360 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.360 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:57.360 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786816+0000", 2024-09-18T04:37:57.361 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:57.361 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.361 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.361 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:57.361 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786816+0000", 2024-09-18T04:37:57.361 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:57.361 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.361 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.361 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:57.361 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786815+0000", 2024-09-18T04:37:57.361 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:37:57.361 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.361 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.362 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:57.362 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786821+0000", 2024-09-18T04:37:57.362 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9529999999999996e-06 2024-09-18T04:37:57.362 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.362 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.362 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:57.362 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786841+0000", 2024-09-18T04:37:57.362 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9743000000000001e-05 2024-09-18T04:37:57.362 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.362 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.362 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:57.362 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786848+0000", 2024-09-18T04:37:57.362 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4900000000000003e-06 2024-09-18T04:37:57.363 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.363 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.363 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:57.363 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.799747+0000", 2024-09-18T04:37:57.363 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012898902 2024-09-18T04:37:57.363 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.363 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.363 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:57.363 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.799898+0000", 2024-09-18T04:37:57.363 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000150877 2024-09-18T04:37:57.363 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.363 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.363 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:57.364 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800095+0000", 2024-09-18T04:37:57.364 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000197023 2024-09-18T04:37:57.364 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.364 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.364 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:57.364 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802829+0000", 2024-09-18T04:37:57.364 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0027343480000000002 2024-09-18T04:37:57.364 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.364 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.364 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:57.364 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802847+0000", 2024-09-18T04:37:57.364 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7283e-05 2024-09-18T04:37:57.364 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.365 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.365 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:57.365 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802974+0000", 2024-09-18T04:37:57.365 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012749199999999999 2024-09-18T04:37:57.365 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.365 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.365 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:57.365 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803014+0000", 2024-09-18T04:37:57.365 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9576000000000002e-05 2024-09-18T04:37:57.365 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:57.365 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:57.365 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:57.365 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.365 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.366 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:57.366 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:37:57.366 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 29.519483291, 2024-09-18T04:37:57.366 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017399218000000001, 2024-09-18T04:37:57.366 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:57.366 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:57.366 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:57.366 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:57.366 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:57.366 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 114 2024-09-18T04:37:57.366 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.366 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:57.366 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.367 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:57.367 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:37:57.367 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:57.367 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.367 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.367 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:57.367 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:37:57.367 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:57.367 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.367 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.367 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:57.367 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786900+0000", 2024-09-18T04:37:57.367 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:37:57.367 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.368 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.368 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:57.368 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786908+0000", 2024-09-18T04:37:57.368 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8199999999999997e-06 2024-09-18T04:37:57.368 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.368 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.368 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:57.368 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786925+0000", 2024-09-18T04:37:57.368 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6849000000000001e-05 2024-09-18T04:37:57.368 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.368 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.368 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:57.368 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786934+0000", 2024-09-18T04:37:57.369 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6299999999999993e-06 2024-09-18T04:37:57.369 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.369 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.369 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:57.369 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800926+0000", 2024-09-18T04:37:57.369 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013991632 2024-09-18T04:37:57.369 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.369 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.369 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:57.369 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801060+0000", 2024-09-18T04:37:57.369 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013378199999999999 2024-09-18T04:37:57.369 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.369 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.370 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:57.370 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801166+0000", 2024-09-18T04:37:57.370 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010607099999999999 2024-09-18T04:37:57.370 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.370 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.370 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:57.370 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803026+0000", 2024-09-18T04:37:57.370 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001860469 2024-09-18T04:37:57.370 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.370 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.370 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:57.371 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804245+0000", 2024-09-18T04:37:57.371 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0012185900000000001 2024-09-18T04:37:57.371 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.371 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.371 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:57.371 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804278+0000", 2024-09-18T04:37:57.371 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3272000000000003e-05 2024-09-18T04:37:57.371 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.371 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.371 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:57.371 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804300+0000", 2024-09-18T04:37:57.371 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2266e-05 2024-09-18T04:37:57.372 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:57.372 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:57.372 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:57.372 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.372 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.372 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:57.372 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787566+0000", 2024-09-18T04:37:57.372 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 29.518818331999999, 2024-09-18T04:37:57.372 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017124120999999999, 2024-09-18T04:37:57.372 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:57.372 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:57.372 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:57.372 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:57.372 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:57.372 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 115 2024-09-18T04:37:57.373 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.373 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:57.373 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.373 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:57.373 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787566+0000", 2024-09-18T04:37:57.373 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:57.373 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.373 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.374 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:57.374 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787566+0000", 2024-09-18T04:37:57.374 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:57.374 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.374 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.374 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:57.374 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787564+0000", 2024-09-18T04:37:57.374 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:37:57.374 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.374 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.374 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:57.374 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787751+0000", 2024-09-18T04:37:57.374 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000186873 2024-09-18T04:37:57.375 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.375 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.375 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:57.375 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787772+0000", 2024-09-18T04:37:57.375 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.126e-05 2024-09-18T04:37:57.375 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.375 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.375 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:57.375 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787794+0000", 2024-09-18T04:37:57.375 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1418999999999999e-05 2024-09-18T04:37:57.376 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.376 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.376 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:57.376 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801659+0000", 2024-09-18T04:37:57.376 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013865125000000001 2024-09-18T04:37:57.376 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.376 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.377 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:57.377 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801750+0000", 2024-09-18T04:37:57.377 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.1427000000000004e-05 2024-09-18T04:37:57.377 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.377 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.377 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:57.377 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801894+0000", 2024-09-18T04:37:57.377 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000143514 2024-09-18T04:37:57.380 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.381 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.381 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:57.381 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804313+0000", 2024-09-18T04:37:57.381 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0024190879999999998 2024-09-18T04:37:57.381 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.381 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.381 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:57.381 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804644+0000", 2024-09-18T04:37:57.381 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00033100799999999999 2024-09-18T04:37:57.381 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.382 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.382 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:57.382 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804672+0000", 2024-09-18T04:37:57.382 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7988e-05 2024-09-18T04:37:57.382 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.382 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.382 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:57.382 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804690+0000", 2024-09-18T04:37:57.382 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8298999999999999e-05 2024-09-18T04:37:57.382 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:57.383 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:57.383 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:57.383 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.383 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.383 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:57.383 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:37:57.383 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 29.518570393000001, 2024-09-18T04:37:57.383 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017521662, 2024-09-18T04:37:57.383 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:57.383 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:57.383 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:57.384 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:57.384 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:57.384 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 116 2024-09-18T04:37:57.384 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.384 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:57.384 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.384 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:57.384 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:37:57.384 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:57.384 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.384 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.384 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:57.385 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:37:57.385 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:57.385 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.385 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.385 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:57.385 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787813+0000", 2024-09-18T04:37:57.385 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:37:57.385 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.385 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.385 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:57.385 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787824+0000", 2024-09-18T04:37:57.385 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1157e-05 2024-09-18T04:37:57.385 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.386 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.386 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:57.386 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787840+0000", 2024-09-18T04:37:57.386 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6269e-05 2024-09-18T04:37:57.386 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.386 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.386 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:57.386 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787853+0000", 2024-09-18T04:37:57.386 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3156000000000001e-05 2024-09-18T04:37:57.386 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.386 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.386 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:57.386 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803110+0000", 2024-09-18T04:37:57.387 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015256736 2024-09-18T04:37:57.387 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.387 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.387 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:57.387 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803197+0000", 2024-09-18T04:37:57.387 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7195999999999997e-05 2024-09-18T04:37:57.387 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.387 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.387 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:57.387 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803308+0000", 2024-09-18T04:37:57.387 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111005 2024-09-18T04:37:57.387 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.388 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.388 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:57.388 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804316+0000", 2024-09-18T04:37:57.388 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010073949999999999 2024-09-18T04:37:57.388 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.388 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.388 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:57.388 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805232+0000", 2024-09-18T04:37:57.389 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091636300000000001 2024-09-18T04:37:57.389 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.389 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.389 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:57.389 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805277+0000", 2024-09-18T04:37:57.389 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4864000000000003e-05 2024-09-18T04:37:57.389 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.389 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.389 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:57.390 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805336+0000", 2024-09-18T04:37:57.390 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8968000000000002e-05 2024-09-18T04:37:57.390 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:57.390 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:57.390 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:57.390 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.390 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.391 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:57.391 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:37:57.391 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 29.518518338, 2024-09-18T04:37:57.391 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017983453, 2024-09-18T04:37:57.391 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:57.391 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:57.391 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:57.391 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:57.391 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:57.391 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 117 2024-09-18T04:37:57.391 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.392 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:57.392 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.392 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:57.392 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:37:57.392 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:57.392 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.392 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.392 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:57.392 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:37:57.392 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:57.392 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.392 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.392 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:57.393 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787865+0000", 2024-09-18T04:37:57.393 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:37:57.393 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.393 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.393 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:57.393 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787889+0000", 2024-09-18T04:37:57.393 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4244999999999999e-05 2024-09-18T04:37:57.393 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.393 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.393 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:57.393 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787905+0000", 2024-09-18T04:37:57.393 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5739999999999998e-05 2024-09-18T04:37:57.393 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.394 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.394 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:57.394 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787918+0000", 2024-09-18T04:37:57.394 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.252e-05 2024-09-18T04:37:57.394 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.394 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.394 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:57.394 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803693+0000", 2024-09-18T04:37:57.394 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015774872999999998 2024-09-18T04:37:57.394 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.394 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.394 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:57.395 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803775+0000", 2024-09-18T04:37:57.395 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2761000000000003e-05 2024-09-18T04:37:57.395 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.395 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.395 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:57.395 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803887+0000", 2024-09-18T04:37:57.395 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111708 2024-09-18T04:37:57.395 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.395 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.395 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:57.395 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804823+0000", 2024-09-18T04:37:57.395 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00093625100000000003 2024-09-18T04:37:57.396 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.396 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.396 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:57.396 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805804+0000", 2024-09-18T04:37:57.396 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00098114699999999992 2024-09-18T04:37:57.396 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.396 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.396 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:57.396 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805831+0000", 2024-09-18T04:37:57.396 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6401999999999999e-05 2024-09-18T04:37:57.396 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.396 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.397 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:57.397 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805850+0000", 2024-09-18T04:37:57.397 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8862e-05 2024-09-18T04:37:57.397 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:57.397 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:57.397 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:57.397 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.397 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.397 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:57.397 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.230830+0000", 2024-09-18T04:37:57.397 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 29.075554458999999, 2024-09-18T04:37:57.397 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011537146999999999, 2024-09-18T04:37:57.398 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:57.398 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:57.398 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:57.398 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:57.398 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:57.398 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 135 2024-09-18T04:37:57.398 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.398 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:57.398 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.398 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:57.398 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230830+0000", 2024-09-18T04:37:57.398 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:57.399 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.399 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.399 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:57.399 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230830+0000", 2024-09-18T04:37:57.399 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:57.399 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.399 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.399 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:57.399 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230829+0000", 2024-09-18T04:37:57.399 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:37:57.399 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.399 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.400 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:57.400 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230835+0000", 2024-09-18T04:37:57.400 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8660000000000002e-06 2024-09-18T04:37:57.400 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.400 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.400 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:57.400 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230844+0000", 2024-09-18T04:37:57.400 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9339999999999995e-06 2024-09-18T04:37:57.400 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.400 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.400 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:57.400 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230853+0000", 2024-09-18T04:37:57.400 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4419999999999998e-06 2024-09-18T04:37:57.401 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.401 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.401 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:57.401 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.237658+0000", 2024-09-18T04:37:57.401 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0068055310000000001 2024-09-18T04:37:57.401 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.401 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.401 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:57.401 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.237837+0000", 2024-09-18T04:37:57.401 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017943200000000001 2024-09-18T04:37:57.401 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.401 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.401 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:57.402 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.238112+0000", 2024-09-18T04:37:57.402 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00027488600000000001 2024-09-18T04:37:57.402 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.402 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.402 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:57.402 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.240505+0000", 2024-09-18T04:37:57.402 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0023930539999999999 2024-09-18T04:37:57.402 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.402 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.402 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:57.402 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242157+0000", 2024-09-18T04:37:57.402 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001651135 2024-09-18T04:37:57.402 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.403 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.403 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:57.403 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242281+0000", 2024-09-18T04:37:57.403 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012451300000000001 2024-09-18T04:37:57.403 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.403 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.403 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:57.403 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242367+0000", 2024-09-18T04:37:57.403 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6178000000000001e-05 2024-09-18T04:37:57.403 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:57.403 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:57.403 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:57.403 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.404 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.404 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:57.404 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.230858+0000", 2024-09-18T04:37:57.404 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 29.075526571000001, 2024-09-18T04:37:57.404 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013005821000000001, 2024-09-18T04:37:57.404 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:57.404 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:57.404 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:57.404 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:57.404 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:57.404 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 136 2024-09-18T04:37:57.404 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.404 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:57.405 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.405 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:57.405 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230858+0000", 2024-09-18T04:37:57.405 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:57.405 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.405 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.405 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:57.405 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230858+0000", 2024-09-18T04:37:57.405 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:57.405 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.405 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.405 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:57.406 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230858+0000", 2024-09-18T04:37:57.406 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:37:57.406 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.406 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.406 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:57.406 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230861+0000", 2024-09-18T04:37:57.406 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7459999999999998e-06 2024-09-18T04:37:57.406 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.406 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.406 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:57.406 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230877+0000", 2024-09-18T04:37:57.406 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5979000000000001e-05 2024-09-18T04:37:57.406 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.406 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.407 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:57.407 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230885+0000", 2024-09-18T04:37:57.407 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.5979999999999999e-06 2024-09-18T04:37:57.407 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.407 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.407 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:57.407 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.239023+0000", 2024-09-18T04:37:57.407 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0081380710000000002 2024-09-18T04:37:57.407 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.407 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.407 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:57.407 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.239216+0000", 2024-09-18T04:37:57.407 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019274799999999999 2024-09-18T04:37:57.407 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.408 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.408 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:57.408 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.239417+0000", 2024-09-18T04:37:57.408 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020093300000000001 2024-09-18T04:37:57.408 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.408 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.408 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:57.408 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242494+0000", 2024-09-18T04:37:57.408 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003077177 2024-09-18T04:37:57.408 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.408 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.408 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:57.408 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.243668+0000", 2024-09-18T04:37:57.409 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001174027 2024-09-18T04:37:57.409 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.409 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.409 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:57.409 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.243777+0000", 2024-09-18T04:37:57.409 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010955600000000001 2024-09-18T04:37:57.409 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.409 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.409 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:57.409 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.243864+0000", 2024-09-18T04:37:57.409 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6441999999999997e-05 2024-09-18T04:37:57.409 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:57.409 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:57.410 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:57.410 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.410 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.410 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:57.410 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.230901+0000", 2024-09-18T04:37:57.410 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 29.075483997999999, 2024-09-18T04:37:57.410 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014120882, 2024-09-18T04:37:57.410 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:57.410 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:57.410 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:57.410 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:57.410 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:57.410 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 137 2024-09-18T04:37:57.410 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.411 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:57.411 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.411 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:57.411 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230901+0000", 2024-09-18T04:37:57.411 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:57.411 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.411 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.411 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:57.411 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230901+0000", 2024-09-18T04:37:57.411 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:57.411 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.411 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.411 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:57.411 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230900+0000", 2024-09-18T04:37:57.412 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:37:57.412 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.412 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.412 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:57.412 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231002+0000", 2024-09-18T04:37:57.412 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000102392 2024-09-18T04:37:57.412 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.412 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.412 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:57.412 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231029+0000", 2024-09-18T04:37:57.412 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7166999999999999e-05 2024-09-18T04:37:57.412 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.412 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.413 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:57.413 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231036+0000", 2024-09-18T04:37:57.413 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7259999999999996e-06 2024-09-18T04:37:57.413 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.413 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.413 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:57.413 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.240562+0000", 2024-09-18T04:37:57.413 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0095257970000000008 2024-09-18T04:37:57.414 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.414 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.414 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:57.414 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.240730+0000", 2024-09-18T04:37:57.414 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016767099999999999 2024-09-18T04:37:57.414 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.414 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.414 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:57.414 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.240927+0000", 2024-09-18T04:37:57.414 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019711899999999999 2024-09-18T04:37:57.414 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.414 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.414 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:57.415 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244885+0000", 2024-09-18T04:37:57.415 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0039577340000000001 2024-09-18T04:37:57.415 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.415 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.415 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:57.415 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244899+0000", 2024-09-18T04:37:57.415 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4403000000000001e-05 2024-09-18T04:37:57.415 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.415 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.416 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:57.416 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244973+0000", 2024-09-18T04:37:57.416 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4177999999999994e-05 2024-09-18T04:37:57.416 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.416 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.416 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:57.416 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.245021+0000", 2024-09-18T04:37:57.416 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8344999999999999e-05 2024-09-18T04:37:57.416 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:57.416 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:57.416 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:57.416 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.417 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.417 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:57.417 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:37:57.417 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 29.075341207000001, 2024-09-18T04:37:57.417 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016528505999999998, 2024-09-18T04:37:57.417 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:57.417 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:57.417 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:57.417 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:57.417 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:57.418 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 138 2024-09-18T04:37:57.418 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.418 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:57.418 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.418 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:57.418 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:37:57.418 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:57.418 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.418 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.418 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:57.418 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:37:57.418 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:57.419 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.419 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.419 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:57.419 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:37:57.419 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:37:57.419 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.419 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.419 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:57.419 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231047+0000", 2024-09-18T04:37:57.420 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5399999999999997e-06 2024-09-18T04:37:57.420 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.420 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.420 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:57.420 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231068+0000", 2024-09-18T04:37:57.420 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0769000000000001e-05 2024-09-18T04:37:57.420 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.420 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.420 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:57.420 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231074+0000", 2024-09-18T04:37:57.420 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6180000000000003e-06 2024-09-18T04:37:57.421 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.421 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.421 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:57.421 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242556+0000", 2024-09-18T04:37:57.421 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011482654 2024-09-18T04:37:57.421 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.421 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.421 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:57.421 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242666+0000", 2024-09-18T04:37:57.422 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011003200000000001 2024-09-18T04:37:57.422 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.422 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.422 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:57.422 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242924+0000", 2024-09-18T04:37:57.422 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00025785899999999999 2024-09-18T04:37:57.422 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.422 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.422 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:57.423 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247263+0000", 2024-09-18T04:37:57.423 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0043387649999999996 2024-09-18T04:37:57.423 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.423 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.423 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:57.423 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247412+0000", 2024-09-18T04:37:57.423 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000149276 2024-09-18T04:37:57.423 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.423 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.423 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:57.424 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247522+0000", 2024-09-18T04:37:57.424 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000109788 2024-09-18T04:37:57.424 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.424 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.424 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:57.424 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247572+0000", 2024-09-18T04:37:57.424 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9808999999999997e-05 2024-09-18T04:37:57.424 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:57.425 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:57.425 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:57.425 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.425 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.425 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:57.425 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231083+0000", 2024-09-18T04:37:57.425 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 29.075301744000001, 2024-09-18T04:37:57.425 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0166731, 2024-09-18T04:37:57.426 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:57.426 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:57.426 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:57.426 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:57.426 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:57.426 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 139 2024-09-18T04:37:57.426 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.426 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:57.426 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.426 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:57.426 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231083+0000", 2024-09-18T04:37:57.426 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:57.426 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.426 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.426 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:57.427 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231083+0000", 2024-09-18T04:37:57.427 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:57.427 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.427 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.427 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:57.427 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231082+0000", 2024-09-18T04:37:57.427 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:37:57.427 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.427 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.427 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:57.427 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231090+0000", 2024-09-18T04:37:57.427 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1280000000000008e-06 2024-09-18T04:37:57.427 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.428 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.428 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:57.428 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231097+0000", 2024-09-18T04:37:57.428 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5119999999999997e-06 2024-09-18T04:37:57.428 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.428 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.428 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:57.428 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231106+0000", 2024-09-18T04:37:57.428 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9230000000000008e-06 2024-09-18T04:37:57.428 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.428 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.428 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:57.429 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.243977+0000", 2024-09-18T04:37:57.429 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012870754 2024-09-18T04:37:57.429 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.429 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.429 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:57.429 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244112+0000", 2024-09-18T04:37:57.429 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000135575 2024-09-18T04:37:57.429 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.429 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.429 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:57.429 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244265+0000", 2024-09-18T04:37:57.429 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015279900000000001 2024-09-18T04:37:57.429 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.429 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.430 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:57.430 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247649+0000", 2024-09-18T04:37:57.430 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003383896 2024-09-18T04:37:57.430 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.430 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.430 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:57.430 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247656+0000", 2024-09-18T04:37:57.430 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8920000000000003e-06 2024-09-18T04:37:57.430 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.430 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.430 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:57.430 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247692+0000", 2024-09-18T04:37:57.430 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6470000000000001e-05 2024-09-18T04:37:57.431 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.431 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.431 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:57.431 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247756+0000", 2024-09-18T04:37:57.431 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3676999999999994e-05 2024-09-18T04:37:57.431 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:57.431 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:57.431 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:57.431 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.431 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.431 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:57.431 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231190+0000", 2024-09-18T04:37:57.431 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 29.075194818, 2024-09-18T04:37:57.431 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016793556000000001, 2024-09-18T04:37:57.432 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:57.432 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:57.432 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:57.432 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:57.432 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:57.432 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 140 2024-09-18T04:37:57.432 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.432 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:57.432 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.432 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:57.432 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231190+0000", 2024-09-18T04:37:57.432 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:57.432 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.432 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.433 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:57.433 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231190+0000", 2024-09-18T04:37:57.433 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:57.433 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.433 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.433 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:57.433 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231189+0000", 2024-09-18T04:37:57.433 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:37:57.433 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.433 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.433 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:57.433 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231273+0000", 2024-09-18T04:37:57.433 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4445999999999998e-05 2024-09-18T04:37:57.434 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.434 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.434 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:57.434 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231282+0000", 2024-09-18T04:37:57.434 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8859999999999993e-06 2024-09-18T04:37:57.434 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.434 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.434 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:57.434 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231288+0000", 2024-09-18T04:37:57.434 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9819999999999996e-06 2024-09-18T04:37:57.434 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.434 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.434 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:57.434 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.245119+0000", 2024-09-18T04:37:57.435 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.01383121 2024-09-18T04:37:57.435 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.435 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.435 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:57.435 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.245247+0000", 2024-09-18T04:37:57.435 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012763299999999999 2024-09-18T04:37:57.435 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.435 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.435 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:57.435 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.245401+0000", 2024-09-18T04:37:57.435 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015399200000000001 2024-09-18T04:37:57.435 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.435 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.436 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:57.436 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247766+0000", 2024-09-18T04:37:57.436 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0023649510000000001 2024-09-18T04:37:57.436 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.436 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.436 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:57.436 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247904+0000", 2024-09-18T04:37:57.436 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013836499999999999 2024-09-18T04:37:57.436 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.436 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.436 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:57.436 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247950+0000", 2024-09-18T04:37:57.437 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5201000000000003e-05 2024-09-18T04:37:57.437 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.437 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.437 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:57.437 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247983+0000", 2024-09-18T04:37:57.437 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3686000000000002e-05 2024-09-18T04:37:57.437 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:57.437 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:57.437 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:57.437 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.437 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.437 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:37:57.438 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:37:57.438 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 29.075084074999999, 2024-09-18T04:37:57.438 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017344024, 2024-09-18T04:37:57.438 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:57.438 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:57.438 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:57.438 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:57.438 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:57.438 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 141 2024-09-18T04:37:57.438 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.438 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:57.438 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.438 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:57.438 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:37:57.439 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:57.439 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.439 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.439 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:57.439 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:37:57.439 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:57.439 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.439 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.439 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:57.439 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:37:57.439 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:37:57.439 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.439 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.440 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:57.440 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231309+0000", 2024-09-18T04:37:57.440 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5419999999999994e-06 2024-09-18T04:37:57.440 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.440 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.440 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:57.440 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231323+0000", 2024-09-18T04:37:57.440 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3108000000000001e-05 2024-09-18T04:37:57.440 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.440 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.440 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:57.440 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231340+0000", 2024-09-18T04:37:57.440 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.696e-05 2024-09-18T04:37:57.441 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.441 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.441 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:57.441 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.246143+0000", 2024-09-18T04:37:57.441 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014803320999999999 2024-09-18T04:37:57.441 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.441 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.441 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:57.441 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.246323+0000", 2024-09-18T04:37:57.441 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00018061600000000001 2024-09-18T04:37:57.441 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.441 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.441 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:57.442 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.246530+0000", 2024-09-18T04:37:57.442 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020625 2024-09-18T04:37:57.442 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.442 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.442 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:57.442 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248002+0000", 2024-09-18T04:37:57.442 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0014719379999999999 2024-09-18T04:37:57.442 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.442 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.442 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:57.442 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248582+0000", 2024-09-18T04:37:57.442 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00058020599999999999 2024-09-18T04:37:57.442 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.443 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.443 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:57.443 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248618+0000", 2024-09-18T04:37:57.443 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6559e-05 2024-09-18T04:37:57.443 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.443 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.443 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:57.443 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248645+0000", 2024-09-18T04:37:57.443 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6080000000000001e-05 2024-09-18T04:37:57.443 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:57.443 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:57.443 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:57.443 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.444 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.444 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-18T04:37:57.444 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.878463+0000", 2024-09-18T04:37:57.444 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 28.427922040999999, 2024-09-18T04:37:57.444 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010882697, 2024-09-18T04:37:57.444 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:37:57.444 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:37:57.444 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:37:57.444 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:37:57.444 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:37:57.444 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 159 2024-09-18T04:37:57.444 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.444 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:37:57.445 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.445 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:37:57.445 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878463+0000", 2024-09-18T04:37:57.445 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:57.445 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.445 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.445 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:37:57.445 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878463+0000", 2024-09-18T04:37:57.445 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:37:57.445 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.445 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.445 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:37:57.445 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878462+0000", 2024-09-18T04:37:57.446 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:37:57.446 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.446 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.446 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:37:57.446 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878467+0000", 2024-09-18T04:37:57.446 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4099999999999999e-06 2024-09-18T04:37:57.446 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.446 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.446 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:37:57.446 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878490+0000", 2024-09-18T04:37:57.446 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3159999999999998e-05 2024-09-18T04:37:57.446 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.446 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.446 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:37:57.447 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878511+0000", 2024-09-18T04:37:57.447 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1092000000000001e-05 2024-09-18T04:37:57.447 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.447 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.447 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:37:57.447 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.887163+0000", 2024-09-18T04:37:57.447 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.008651845 2024-09-18T04:37:57.447 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.447 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.447 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:37:57.447 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.887273+0000", 2024-09-18T04:37:57.447 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000109823 2024-09-18T04:37:57.447 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.447 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.448 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:37:57.448 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.887441+0000", 2024-09-18T04:37:57.448 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016777599999999999 2024-09-18T04:37:57.448 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.448 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.448 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:37:57.448 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.888634+0000", 2024-09-18T04:37:57.448 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001192839 2024-09-18T04:37:57.448 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.448 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.449 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:37:57.449 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.889153+0000", 2024-09-18T04:37:57.449 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00051953700000000004 2024-09-18T04:37:57.449 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.449 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.449 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:37:57.449 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.889234+0000", 2024-09-18T04:37:57.449 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0802000000000003e-05 2024-09-18T04:37:57.449 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:37:57.449 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:37:57.449 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:37:57.450 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.889345+0000", 2024-09-18T04:37:57.450 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011112000000000001 2024-09-18T04:37:57.450 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:57.450 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:57.450 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:57.450 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:37:57.450 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:37:57.450 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:37:57.452 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:37:57.448+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:37:57.452 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:37:57.471 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T04:37:57.552 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:37:57.548+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:37:57.566 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:37:57.566 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:37:57.566 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:37:57.566 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:37:57.576 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:37:57.625 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:37:57.647 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:37:57.653 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:37:57.652+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:37:57.667 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:37:57.682 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:37:57.683 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:37:57.683 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:37:57.683 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:37:57.683 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:37:57.691 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:37:57.692 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:37:57.706 INFO:teuthology.orchestra.run.smithi028.stderr:set 2.5 pg_upmap mapping to [2,0] 2024-09-18T04:37:57.718 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:37:57.741 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:37:57.743 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:37:57.743 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:37:57.743 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:37:57.743 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:37:57.743 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:37:57.744 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:37:57.754 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:37:57.750+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:37:57.794 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:57.794 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:57.804 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:37:57.855 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:37:57.850+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:37:57.905 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:57.905 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:57.915 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:37:57.955 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:37:57.950+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:37:58.023 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:58.023 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:58.033 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:37:58.056 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:37:58.052+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:37:58.157 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:37:58.156+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:37:58.185 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:58.185 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:58.194 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:37:58.258 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:37:58.254+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:37:58.313 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:58.313 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:58.324 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:37:58.359 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:37:58.356+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:37:58.437 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:58.437 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:58.445 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:37:58.461 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:37:58.454+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:37:58.556 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:58.556 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:58.561 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:37:58.554+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:37:58.566 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:37:58.662 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:37:58.660+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:37:58.675 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:58.675 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:58.685 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:37:58.706 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-18T04:37:58.764 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:37:58.758+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:37:58.839 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:58.839 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:58.849 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:37:58.864 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:37:58.860+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:37:58.954 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:58.954 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:58.964 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:37:58.965 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:37:58.958+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:37:59.067 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:37:59.062+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:37:59.110 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:59.111 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:59.120 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:37:59.167 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:37:59.164+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:37:59.225 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:59.225 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:59.236 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:37:59.278 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:37:59.276+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:37:59.370 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:37:59.366+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:37:59.388 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:59.388 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:59.398 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:37:59.472 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:37:59.468+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:37:59.519 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:59.520 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:59.529 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:37:59.572 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:37:59.566+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:37:59.640 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:59.640 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:59.650 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:37:59.674 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:37:59.672+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:37:59.724 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-09-18T04:37:59.755 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T04:37:59.758 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:59.759 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:59.768 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:37:59.774 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:37:59.766+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:37:59.871 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:59.871 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:37:59.875 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:37:59.872+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:37:59.881 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:37:59.935 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:37:59.954 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:37:59.975 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:37:59.976 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:37:59.972+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:37:59.997 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:37:59.997 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:00.006 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:38:00.007 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:38:00.028 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:38:00.049 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:38:00.051 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:38:00.051 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:38:00.051 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:38:00.051 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:38:00.051 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:38:00.051 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:38:00.077 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:00.072+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:00.108 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:00.108 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:38:00.108 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:38:00.108 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [] 2024-09-18T04:38:00.108 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:00.118 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:38:00.178 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:00.170+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:00.221 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:00.221 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:00.221 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:38:00.221 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:00.231 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:38:00.278 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:00.276+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:00.379 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:00.374+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:00.386 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:00.386 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:00.386 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:38:00.386 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:38:00.386 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:00.396 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:38:00.480 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:00.476+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:00.504 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:00.504 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:38:00.504 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:38:00.504 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:38:00.505 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.505 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-18T04:38:00.505 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:38:00.505 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 26.030904159999999, 2024-09-18T04:38:00.505 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0095781189999999995, 2024-09-18T04:38:00.505 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:00.505 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:00.505 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:00.505 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:38:00.505 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:38:00.505 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 1 2024-09-18T04:38:00.505 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.505 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:00.505 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.506 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:00.506 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:38:00.506 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:00.506 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.506 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.506 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:00.506 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:38:00.506 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:00.506 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.506 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.506 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:00.506 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469862+0000", 2024-09-18T04:38:00.506 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:00.507 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.507 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.507 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:00.507 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469879+0000", 2024-09-18T04:38:00.507 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7431999999999998e-05 2024-09-18T04:38:00.507 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.507 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.507 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:00.507 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469896+0000", 2024-09-18T04:38:00.507 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6742999999999999e-05 2024-09-18T04:38:00.507 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.507 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.507 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:00.508 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469915+0000", 2024-09-18T04:38:00.508 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8991000000000002e-05 2024-09-18T04:38:00.508 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.508 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.508 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:00.508 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470397+0000", 2024-09-18T04:38:00.508 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048167799999999998 2024-09-18T04:38:00.508 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.508 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.508 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-09-18T04:38:00.508 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470412+0000", 2024-09-18T04:38:00.508 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5526e-05 2024-09-18T04:38:00.508 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.509 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.509 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:00.509 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473279+0000", 2024-09-18T04:38:00.509 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0028667469999999998 2024-09-18T04:38:00.509 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.509 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.509 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:00.509 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473441+0000", 2024-09-18T04:38:00.509 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000162082 2024-09-18T04:38:00.509 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.509 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.509 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:38:00.509 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473573+0000", 2024-09-18T04:38:00.510 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000132032 2024-09-18T04:38:00.510 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.510 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.510 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:00.510 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474826+0000", 2024-09-18T04:38:00.510 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001252717 2024-09-18T04:38:00.510 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.510 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.510 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:00.510 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479307+0000", 2024-09-18T04:38:00.510 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044813220000000003 2024-09-18T04:38:00.510 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.510 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.510 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:00.510 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479366+0000", 2024-09-18T04:38:00.511 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9024999999999998e-05 2024-09-18T04:38:00.511 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.511 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.511 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:00.511 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479443+0000", 2024-09-18T04:38:00.511 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6339000000000002e-05 2024-09-18T04:38:00.511 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:00.511 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:00.511 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:00.511 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.511 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.511 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.4214.0:1 2.3 e18/17 ondisk, result = 0)", 2024-09-18T04:38:00.511 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.474724+0000", 2024-09-18T04:38:00.512 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 26.026044796000001, 2024-09-18T04:38:00.512 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000105358, 2024-09-18T04:38:00.512 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:00.512 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:38:00.512 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:00.512 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.512 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:00.512 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474724+0000", 2024-09-18T04:38:00.512 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:00.512 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.512 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.512 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:00.512 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474724+0000", 2024-09-18T04:38:00.513 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:00.513 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.513 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.513 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:00.513 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474723+0000", 2024-09-18T04:38:00.513 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:00.513 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.513 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.513 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:00.513 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474731+0000", 2024-09-18T04:38:00.513 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2339999999999994e-06 2024-09-18T04:38:00.513 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.513 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.513 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:00.513 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474748+0000", 2024-09-18T04:38:00.514 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6603e-05 2024-09-18T04:38:00.514 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.514 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.514 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:00.514 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474760+0000", 2024-09-18T04:38:00.514 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2215000000000001e-05 2024-09-18T04:38:00.514 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.514 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.514 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:00.514 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474814+0000", 2024-09-18T04:38:00.514 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4267000000000001e-05 2024-09-18T04:38:00.514 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.514 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.515 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:00.515 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474821+0000", 2024-09-18T04:38:00.515 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0929999999999997e-06 2024-09-18T04:38:00.515 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.515 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.515 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:00.515 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474829+0000", 2024-09-18T04:38:00.515 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1200000000000002e-06 2024-09-18T04:38:00.515 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:00.515 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:00.515 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:00.515 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.515 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.515 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18)", 2024-09-18T04:38:00.516 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.482346+0000", 2024-09-18T04:38:00.516 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 26.018422363999999, 2024-09-18T04:38:00.516 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020435599999999999, 2024-09-18T04:38:00.516 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:00.516 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-09-18T04:38:00.516 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:00.516 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:38:00.516 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:38:00.516 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 2 2024-09-18T04:38:00.516 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.516 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:00.516 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.516 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:00.517 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482346+0000", 2024-09-18T04:38:00.517 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:00.517 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.517 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.517 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:00.517 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482346+0000", 2024-09-18T04:38:00.517 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:00.517 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.517 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.517 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:00.517 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482345+0000", 2024-09-18T04:38:00.517 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:00.517 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.517 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.518 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:00.518 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482360+0000", 2024-09-18T04:38:00.518 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4854999999999999e-05 2024-09-18T04:38:00.518 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.518 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.518 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:00.518 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482377+0000", 2024-09-18T04:38:00.518 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7785999999999999e-05 2024-09-18T04:38:00.518 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.518 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.518 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:00.518 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482387+0000", 2024-09-18T04:38:00.518 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6800000000000005e-06 2024-09-18T04:38:00.519 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.519 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.519 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:00.519 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482441+0000", 2024-09-18T04:38:00.519 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4046999999999999e-05 2024-09-18T04:38:00.519 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.519 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.519 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:00.519 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482551+0000", 2024-09-18T04:38:00.519 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000109468 2024-09-18T04:38:00.519 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:00.519 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:00.519 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:00.520 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.520 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.520 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18)", 2024-09-18T04:38:00.520 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.482853+0000", 2024-09-18T04:38:00.520 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 26.017915746, 2024-09-18T04:38:00.520 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00029815799999999997, 2024-09-18T04:38:00.520 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:00.520 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-09-18T04:38:00.520 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:00.520 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:38:00.520 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:38:00.520 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 3 2024-09-18T04:38:00.521 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.521 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:00.521 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.521 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:00.521 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482853+0000", 2024-09-18T04:38:00.521 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:00.521 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.521 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.521 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:00.521 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482853+0000", 2024-09-18T04:38:00.521 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:00.521 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.521 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.521 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:00.522 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482852+0000", 2024-09-18T04:38:00.522 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:00.522 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.522 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.522 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:00.522 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482859+0000", 2024-09-18T04:38:00.522 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0380000000000002e-06 2024-09-18T04:38:00.522 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.522 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.522 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:00.522 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482880+0000", 2024-09-18T04:38:00.523 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1243e-05 2024-09-18T04:38:00.523 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.523 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.523 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:00.523 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482896+0000", 2024-09-18T04:38:00.523 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5977e-05 2024-09-18T04:38:00.523 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.523 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.523 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:00.523 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482970+0000", 2024-09-18T04:38:00.523 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4603000000000004e-05 2024-09-18T04:38:00.523 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.523 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.523 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:00.524 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.483151+0000", 2024-09-18T04:38:00.524 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00018059799999999999 2024-09-18T04:38:00.524 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:00.524 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:00.524 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:00.524 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.524 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.524 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-18T04:38:00.524 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:35.468698+0000", 2024-09-18T04:38:00.524 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 25.032070909000002, 2024-09-18T04:38:00.524 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0039954129999999997, 2024-09-18T04:38:00.524 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:00.524 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:00.524 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:00.525 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:38:00.525 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:38:00.525 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 5 2024-09-18T04:38:00.525 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.525 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:00.525 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.525 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:00.525 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468698+0000", 2024-09-18T04:38:00.525 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:00.525 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.525 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.525 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:00.525 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468698+0000", 2024-09-18T04:38:00.525 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:00.526 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.526 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.526 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:00.526 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468696+0000", 2024-09-18T04:38:00.526 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:00.526 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.526 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.526 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:00.526 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468714+0000", 2024-09-18T04:38:00.526 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8743e-05 2024-09-18T04:38:00.526 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.526 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.526 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:00.527 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468748+0000", 2024-09-18T04:38:00.527 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3339e-05 2024-09-18T04:38:00.527 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.527 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.527 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:00.527 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468771+0000", 2024-09-18T04:38:00.527 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3119999999999999e-05 2024-09-18T04:38:00.527 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.527 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.527 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:00.527 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468922+0000", 2024-09-18T04:38:00.527 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000151383 2024-09-18T04:38:00.527 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.527 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.528 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:00.528 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.469278+0000", 2024-09-18T04:38:00.528 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00035628599999999998 2024-09-18T04:38:00.528 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.528 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.528 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:00.528 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.469629+0000", 2024-09-18T04:38:00.528 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00035014300000000003 2024-09-18T04:38:00.528 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.528 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.528 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:00.528 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.471620+0000", 2024-09-18T04:38:00.528 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001991278 2024-09-18T04:38:00.529 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.529 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.529 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:00.529 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472440+0000", 2024-09-18T04:38:00.529 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00081998599999999998 2024-09-18T04:38:00.529 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.529 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.529 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:00.529 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472592+0000", 2024-09-18T04:38:00.529 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015227400000000001 2024-09-18T04:38:00.529 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.529 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.529 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:00.529 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472693+0000", 2024-09-18T04:38:00.530 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000101074 2024-09-18T04:38:00.530 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:00.530 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:00.530 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:00.530 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.530 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.530 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.4214.0:5 2.2 e18/17 ondisk, result = 0)", 2024-09-18T04:38:00.530 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:35.472226+0000", 2024-09-18T04:38:00.530 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 25.028543009, 2024-09-18T04:38:00.530 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048505299999999999, 2024-09-18T04:38:00.530 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:00.530 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:38:00.530 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:00.530 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.530 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:00.531 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472226+0000", 2024-09-18T04:38:00.531 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:00.531 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.531 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.531 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:00.531 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472226+0000", 2024-09-18T04:38:00.531 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:00.531 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.531 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.531 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:00.531 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472224+0000", 2024-09-18T04:38:00.531 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:00.531 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.532 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.532 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:00.532 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472238+0000", 2024-09-18T04:38:00.532 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.468e-05 2024-09-18T04:38:00.532 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.532 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.532 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:00.532 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472272+0000", 2024-09-18T04:38:00.532 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3586e-05 2024-09-18T04:38:00.532 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.532 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.532 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:00.532 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472297+0000", 2024-09-18T04:38:00.532 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5282999999999998e-05 2024-09-18T04:38:00.533 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.533 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.533 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:00.533 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472396+0000", 2024-09-18T04:38:00.533 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.8394999999999996e-05 2024-09-18T04:38:00.533 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.533 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.533 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:00.533 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472421+0000", 2024-09-18T04:38:00.533 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5479000000000001e-05 2024-09-18T04:38:00.533 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.533 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.533 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:00.534 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472711+0000", 2024-09-18T04:38:00.534 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000289714 2024-09-18T04:38:00.534 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:00.534 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:00.534 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:00.534 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.534 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.534 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-18T04:38:00.534 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:38:00.534 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 24.01789574, 2024-09-18T04:38:00.534 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082699560000000002, 2024-09-18T04:38:00.534 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:00.535 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:00.535 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:00.535 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:38:00.535 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:38:00.535 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 7 2024-09-18T04:38:00.535 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.535 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:00.535 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.535 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:00.535 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:38:00.535 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:00.535 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.535 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.535 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:00.536 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:38:00.536 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:00.536 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.536 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.536 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:00.536 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482870+0000", 2024-09-18T04:38:00.536 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999967 2024-09-18T04:38:00.536 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.536 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.536 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:00.536 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482889+0000", 2024-09-18T04:38:00.536 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.965e-05 2024-09-18T04:38:00.536 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.536 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.537 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:00.537 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482938+0000", 2024-09-18T04:38:00.537 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8548999999999999e-05 2024-09-18T04:38:00.537 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.537 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.537 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:00.537 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482963+0000", 2024-09-18T04:38:00.537 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5075999999999998e-05 2024-09-18T04:38:00.537 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.537 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.537 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:00.537 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483121+0000", 2024-09-18T04:38:00.537 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015819400000000001 2024-09-18T04:38:00.537 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.538 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.538 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op must wait for map", 2024-09-18T04:38:00.538 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483138+0000", 2024-09-18T04:38:00.538 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7068999999999999e-05 2024-09-18T04:38:00.538 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.538 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.538 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:00.538 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488666+0000", 2024-09-18T04:38:00.538 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0055282049999999996 2024-09-18T04:38:00.538 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.538 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.538 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:00.538 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488783+0000", 2024-09-18T04:38:00.538 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000116737 2024-09-18T04:38:00.539 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.539 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.539 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:00.539 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488991+0000", 2024-09-18T04:38:00.539 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020781400000000001 2024-09-18T04:38:00.539 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.539 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.539 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:00.539 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490554+0000", 2024-09-18T04:38:00.539 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0015628370000000001 2024-09-18T04:38:00.539 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.539 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.539 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:00.539 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491019+0000", 2024-09-18T04:38:00.540 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000465608 2024-09-18T04:38:00.540 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.540 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.540 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:00.540 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491087+0000", 2024-09-18T04:38:00.540 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7988e-05 2024-09-18T04:38:00.540 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.540 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.540 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:00.540 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491143+0000", 2024-09-18T04:38:00.540 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5538999999999998e-05 2024-09-18T04:38:00.540 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:00.540 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:00.540 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:00.541 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.541 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.541 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.4214.0:7 2.2 e19/17 ondisk, result = 0)", 2024-09-18T04:38:00.541 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:36.490861+0000", 2024-09-18T04:38:00.541 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 24.009908117999998, 2024-09-18T04:38:00.541 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00029639400000000001, 2024-09-18T04:38:00.541 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:00.541 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:38:00.541 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:00.541 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.541 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:00.541 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490861+0000", 2024-09-18T04:38:00.541 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:00.542 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.542 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.542 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:00.542 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490861+0000", 2024-09-18T04:38:00.542 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:00.542 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.542 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.542 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:00.542 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490859+0000", 2024-09-18T04:38:00.542 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:00.542 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.542 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.542 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:00.543 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490867+0000", 2024-09-18T04:38:00.543 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.3089999999999996e-06 2024-09-18T04:38:00.543 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.543 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.543 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:00.543 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490909+0000", 2024-09-18T04:38:00.543 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1919000000000003e-05 2024-09-18T04:38:00.543 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.543 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.543 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:00.543 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490922+0000", 2024-09-18T04:38:00.543 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2836999999999999e-05 2024-09-18T04:38:00.543 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.543 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.543 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:00.544 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491003+0000", 2024-09-18T04:38:00.544 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0681999999999998e-05 2024-09-18T04:38:00.544 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.544 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.544 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:00.544 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491013+0000", 2024-09-18T04:38:00.544 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0220999999999999e-05 2024-09-18T04:38:00.544 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.544 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.544 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:00.544 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491157+0000", 2024-09-18T04:38:00.544 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014401799999999999 2024-09-18T04:38:00.544 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:00.545 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:00.545 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:00.545 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.545 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.545 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "MOSDPGPushReply(2.3 25/24 [PushReplyOp(2:c4c92e5a:::rbd_trash:head)])", 2024-09-18T04:38:00.545 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:51.680323+0000", 2024-09-18T04:38:00.545 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 8.8204457430000005, 2024-09-18T04:38:00.545 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00054411499999999996, 2024-09-18T04:38:00.545 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:00.545 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-09-18T04:38:00.545 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:00.545 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.546 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:00.546 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680323+0000", 2024-09-18T04:38:00.546 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:00.546 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.546 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.546 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:00.546 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680323+0000", 2024-09-18T04:38:00.546 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:00.546 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.546 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.546 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:00.546 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680322+0000", 2024-09-18T04:38:00.546 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:00.547 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.547 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.547 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:00.547 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680333+0000", 2024-09-18T04:38:00.547 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1153e-05 2024-09-18T04:38:00.547 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.547 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.547 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:00.547 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680357+0000", 2024-09-18T04:38:00.547 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3252999999999999e-05 2024-09-18T04:38:00.547 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.547 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.548 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:00.548 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680378+0000", 2024-09-18T04:38:00.548 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1503000000000001e-05 2024-09-18T04:38:00.548 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.548 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.548 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:00.548 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680522+0000", 2024-09-18T04:38:00.548 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014410700000000001 2024-09-18T04:38:00.548 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:00.548 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:00.548 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:00.548 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680867+0000", 2024-09-18T04:38:00.548 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00034475899999999998 2024-09-18T04:38:00.548 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:00.549 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:00.549 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:00.549 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:00.549 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:00.549 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:00.550 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:38:00.598 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:00.596+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:00.667 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:00.667 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:00.667 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:38:00.667 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:00.678 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:38:00.681 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:00.674+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:00.782 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:00.774+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:00.783 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:00.783 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:00.783 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:38:00.783 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:38:00.784 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:00.793 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:38:00.883 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:00.879+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:00.899 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:00.899 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:38:00.900 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:38:00.900 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:38:00.900 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.900 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:30 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'16, mlcod=15'14)", 2024-09-18T04:38:00.900 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.486330+0000", 2024-09-18T04:38:00.900 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 33.409772613000001, 2024-09-18T04:38:00.900 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026383700000000001, 2024-09-18T04:38:00.900 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:00.900 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:00.900 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:00.900 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.900 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:00.900 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486330+0000", 2024-09-18T04:38:00.901 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:00.901 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.901 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.901 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:00.901 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486330+0000", 2024-09-18T04:38:00.901 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:00.901 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.901 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.901 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:00.901 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486328+0000", 2024-09-18T04:38:00.901 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:00.901 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.901 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.902 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:00.902 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486342+0000", 2024-09-18T04:38:00.902 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3783e-05 2024-09-18T04:38:00.902 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.902 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.902 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:00.902 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486373+0000", 2024-09-18T04:38:00.902 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1362999999999998e-05 2024-09-18T04:38:00.902 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.902 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.902 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:00.902 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486387+0000", 2024-09-18T04:38:00.902 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3219e-05 2024-09-18T04:38:00.902 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.903 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.903 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:00.903 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486489+0000", 2024-09-18T04:38:00.903 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010232499999999999 2024-09-18T04:38:00.903 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.903 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.903 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:00.903 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486524+0000", 2024-09-18T04:38:00.903 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4972000000000003e-05 2024-09-18T04:38:00.903 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.903 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.903 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:00.903 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.488902+0000", 2024-09-18T04:38:00.903 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023776240000000001 2024-09-18T04:38:00.904 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.904 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.904 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:00.904 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.488968+0000", 2024-09-18T04:38:00.904 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6761000000000007e-05 2024-09-18T04:38:00.904 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:00.904 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:00.904 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:00.904 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.904 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.904 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:35 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'21, mlcod=15'17)", 2024-09-18T04:38:00.904 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.491817+0000", 2024-09-18T04:38:00.905 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 33.404285694999999, 2024-09-18T04:38:00.905 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023524739999999998, 2024-09-18T04:38:00.905 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:00.905 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:00.905 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:00.905 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.905 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:00.905 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491817+0000", 2024-09-18T04:38:00.905 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:00.905 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.905 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.905 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:00.905 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491817+0000", 2024-09-18T04:38:00.906 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:00.906 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.906 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.906 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:00.906 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491815+0000", 2024-09-18T04:38:00.906 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:00.906 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.906 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.906 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:00.906 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491890+0000", 2024-09-18T04:38:00.906 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4500999999999997e-05 2024-09-18T04:38:00.906 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.906 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.907 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:00.907 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491916+0000", 2024-09-18T04:38:00.907 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5666000000000001e-05 2024-09-18T04:38:00.907 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.907 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.907 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:00.907 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491928+0000", 2024-09-18T04:38:00.907 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2585999999999999e-05 2024-09-18T04:38:00.907 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.907 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.907 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:00.907 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.492000+0000", 2024-09-18T04:38:00.907 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2309000000000002e-05 2024-09-18T04:38:00.907 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.908 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.908 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:00.908 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.492027+0000", 2024-09-18T04:38:00.908 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6251e-05 2024-09-18T04:38:00.908 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.908 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.908 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:00.908 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.494114+0000", 2024-09-18T04:38:00.908 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020875260000000001 2024-09-18T04:38:00.908 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.908 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.908 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:00.908 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.494169+0000", 2024-09-18T04:38:00.908 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5145000000000001e-05 2024-09-18T04:38:00.909 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:00.909 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:00.909 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:00.909 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.909 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.909 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:57 1.0 e15/14 1:30306672:devicehealth::main.db.0000000000000000:head v 15'35, mlcod=15'33)", 2024-09-18T04:38:00.909 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.529997+0000", 2024-09-18T04:38:00.909 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 33.366105841, 2024-09-18T04:38:00.909 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002623869, 2024-09-18T04:38:00.909 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:00.909 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:00.909 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:00.909 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.909 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:00.910 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.529997+0000", 2024-09-18T04:38:00.910 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:00.910 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.910 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.910 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:00.910 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.529997+0000", 2024-09-18T04:38:00.910 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:00.910 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.910 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.910 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:00.910 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.529995+0000", 2024-09-18T04:38:00.910 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:00.910 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.911 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.911 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:00.911 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530116+0000", 2024-09-18T04:38:00.911 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000121141 2024-09-18T04:38:00.911 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.911 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.911 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:00.911 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530146+0000", 2024-09-18T04:38:00.911 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0423e-05 2024-09-18T04:38:00.911 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.911 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.911 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:00.912 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530159+0000", 2024-09-18T04:38:00.912 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2802999999999999e-05 2024-09-18T04:38:00.912 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.912 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.912 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:00.912 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530218+0000", 2024-09-18T04:38:00.912 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8461000000000002e-05 2024-09-18T04:38:00.912 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.912 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.912 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:00.912 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530253+0000", 2024-09-18T04:38:00.912 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4796999999999999e-05 2024-09-18T04:38:00.912 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.913 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.913 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:00.913 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.532559+0000", 2024-09-18T04:38:00.913 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002306939 2024-09-18T04:38:00.913 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.913 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.913 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:00.913 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.532621+0000", 2024-09-18T04:38:00.913 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1131999999999997e-05 2024-09-18T04:38:00.913 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:00.913 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:00.913 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:00.913 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.913 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.914 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:68 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'42, mlcod=15'39)", 2024-09-18T04:38:00.914 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.546404+0000", 2024-09-18T04:38:00.914 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 33.349698429999997, 2024-09-18T04:38:00.914 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024499769999999999, 2024-09-18T04:38:00.914 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:00.914 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:00.914 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:00.914 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.914 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:00.914 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546404+0000", 2024-09-18T04:38:00.914 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:00.914 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.914 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.914 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:00.915 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546404+0000", 2024-09-18T04:38:00.915 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:00.915 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.915 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.915 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:00.915 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546403+0000", 2024-09-18T04:38:00.915 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:00.915 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.915 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.915 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:00.915 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546416+0000", 2024-09-18T04:38:00.915 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.306e-05 2024-09-18T04:38:00.915 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.916 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.916 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:00.916 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546465+0000", 2024-09-18T04:38:00.916 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9441999999999999e-05 2024-09-18T04:38:00.916 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.916 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.916 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:00.916 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546480+0000", 2024-09-18T04:38:00.916 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4919999999999999e-05 2024-09-18T04:38:00.916 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.916 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.916 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:00.916 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546541+0000", 2024-09-18T04:38:00.916 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1085000000000002e-05 2024-09-18T04:38:00.917 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.917 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.917 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:00.917 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546569+0000", 2024-09-18T04:38:00.917 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7803000000000001e-05 2024-09-18T04:38:00.917 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.917 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.917 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:00.917 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.548725+0000", 2024-09-18T04:38:00.917 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021556629999999999 2024-09-18T04:38:00.917 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.917 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.917 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:00.917 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.548854+0000", 2024-09-18T04:38:00.918 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000129524 2024-09-18T04:38:00.918 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:00.918 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:00.918 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:00.918 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.918 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.918 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:69 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'43, mlcod=15'39)", 2024-09-18T04:38:00.918 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.547520+0000", 2024-09-18T04:38:00.918 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 33.348582354000001, 2024-09-18T04:38:00.918 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023506820000000002, 2024-09-18T04:38:00.918 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:00.918 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:00.918 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:00.918 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.919 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:00.919 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547520+0000", 2024-09-18T04:38:00.919 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:00.919 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.919 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.919 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:00.919 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547520+0000", 2024-09-18T04:38:00.919 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:00.919 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.919 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.919 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:00.919 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547519+0000", 2024-09-18T04:38:00.919 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:00.919 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.920 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.920 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:00.920 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547614+0000", 2024-09-18T04:38:00.920 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5235000000000006e-05 2024-09-18T04:38:00.920 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.920 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.920 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:00.920 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547640+0000", 2024-09-18T04:38:00.920 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5698999999999999e-05 2024-09-18T04:38:00.920 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.920 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.920 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:00.921 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547651+0000", 2024-09-18T04:38:00.921 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.134e-05 2024-09-18T04:38:00.921 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.921 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.921 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:00.921 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547805+0000", 2024-09-18T04:38:00.921 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015426099999999999 2024-09-18T04:38:00.921 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.921 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.921 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:00.921 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547844+0000", 2024-09-18T04:38:00.921 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8229999999999998e-05 2024-09-18T04:38:00.921 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.922 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.922 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:00.922 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.549810+0000", 2024-09-18T04:38:00.922 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0019660519999999998 2024-09-18T04:38:00.922 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.922 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.922 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:00.922 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.549871+0000", 2024-09-18T04:38:00.922 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1152e-05 2024-09-18T04:38:00.922 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:00.922 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:00.922 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:00.922 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.923 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.923 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:83 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'53, mlcod=15'50)", 2024-09-18T04:38:00.923 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.575989+0000", 2024-09-18T04:38:00.923 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 33.320113405999997, 2024-09-18T04:38:00.923 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023991310000000001, 2024-09-18T04:38:00.923 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:00.923 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:00.923 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:00.923 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.924 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:00.924 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.575989+0000", 2024-09-18T04:38:00.924 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:00.924 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.924 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.924 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:00.924 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.575989+0000", 2024-09-18T04:38:00.924 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:00.924 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.924 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.924 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:00.924 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.575988+0000", 2024-09-18T04:38:00.924 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:00.924 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.925 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.925 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:00.925 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576001+0000", 2024-09-18T04:38:00.925 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2751e-05 2024-09-18T04:38:00.925 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.925 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.925 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:00.925 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576023+0000", 2024-09-18T04:38:00.925 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2453999999999999e-05 2024-09-18T04:38:00.925 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.925 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.925 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:00.925 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576038+0000", 2024-09-18T04:38:00.926 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5125e-05 2024-09-18T04:38:00.926 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.926 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.926 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:00.926 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576161+0000", 2024-09-18T04:38:00.926 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000122543 2024-09-18T04:38:00.926 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.926 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.926 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:00.926 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576221+0000", 2024-09-18T04:38:00.926 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9948000000000003e-05 2024-09-18T04:38:00.926 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.926 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.926 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:00.927 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.578277+0000", 2024-09-18T04:38:00.927 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020567559999999999 2024-09-18T04:38:00.927 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.927 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.927 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:00.927 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.578388+0000", 2024-09-18T04:38:00.927 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011089100000000001 2024-09-18T04:38:00.927 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:00.927 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:00.927 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:00.927 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.927 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.927 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:86 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'56, mlcod=15'52)", 2024-09-18T04:38:00.927 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.580261+0000", 2024-09-18T04:38:00.927 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 33.315841769999999, 2024-09-18T04:38:00.928 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002418108, 2024-09-18T04:38:00.928 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:00.928 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:00.928 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:00.928 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.928 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:00.928 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580261+0000", 2024-09-18T04:38:00.928 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:00.928 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.928 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.928 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:00.928 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580261+0000", 2024-09-18T04:38:00.928 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:00.929 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.929 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.929 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:00.929 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580260+0000", 2024-09-18T04:38:00.929 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:00.929 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.929 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.929 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:00.929 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580269+0000", 2024-09-18T04:38:00.929 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5109999999999992e-06 2024-09-18T04:38:00.929 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.929 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.929 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:00.930 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580288+0000", 2024-09-18T04:38:00.930 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8783999999999998e-05 2024-09-18T04:38:00.930 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.930 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.930 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:00.930 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580296+0000", 2024-09-18T04:38:00.930 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3250000000000008e-06 2024-09-18T04:38:00.930 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.930 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.930 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:00.930 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580351+0000", 2024-09-18T04:38:00.930 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4132000000000003e-05 2024-09-18T04:38:00.930 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.931 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.931 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:00.931 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580381+0000", 2024-09-18T04:38:00.931 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0263000000000001e-05 2024-09-18T04:38:00.931 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.931 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.931 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:00.931 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.582615+0000", 2024-09-18T04:38:00.931 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0022339109999999999 2024-09-18T04:38:00.931 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.931 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.931 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:00.931 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.582679+0000", 2024-09-18T04:38:00.931 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4182000000000003e-05 2024-09-18T04:38:00.932 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:00.932 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:00.932 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:00.932 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.932 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.932 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:105 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'67, mlcod=15'64)", 2024-09-18T04:38:00.932 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.789089+0000", 2024-09-18T04:38:00.932 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 33.107013600000002, 2024-09-18T04:38:00.932 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024384229999999999, 2024-09-18T04:38:00.932 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:00.932 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:00.932 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:00.932 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.932 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:00.933 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789089+0000", 2024-09-18T04:38:00.933 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:00.933 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.933 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.933 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:00.933 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789089+0000", 2024-09-18T04:38:00.933 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:00.933 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.933 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.933 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:00.933 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789088+0000", 2024-09-18T04:38:00.933 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:00.933 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.934 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.934 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:00.934 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789107+0000", 2024-09-18T04:38:00.934 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9017000000000001e-05 2024-09-18T04:38:00.934 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.934 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.934 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:00.934 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789139+0000", 2024-09-18T04:38:00.934 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2261999999999997e-05 2024-09-18T04:38:00.934 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.934 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.934 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:00.934 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789153+0000", 2024-09-18T04:38:00.934 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4025999999999999e-05 2024-09-18T04:38:00.935 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.935 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.935 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:00.935 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789255+0000", 2024-09-18T04:38:00.935 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000101969 2024-09-18T04:38:00.935 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.935 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.935 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:00.935 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789314+0000", 2024-09-18T04:38:00.935 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8867999999999999e-05 2024-09-18T04:38:00.935 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.935 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.935 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:00.936 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.791453+0000", 2024-09-18T04:38:00.936 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002138938 2024-09-18T04:38:00.936 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.936 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.936 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:00.936 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.791527+0000", 2024-09-18T04:38:00.936 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4464999999999999e-05 2024-09-18T04:38:00.936 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:00.936 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:00.936 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:00.936 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.936 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.936 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:106 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'68, mlcod=15'64)", 2024-09-18T04:38:00.936 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.790295+0000", 2024-09-18T04:38:00.937 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 33.105807749, 2024-09-18T04:38:00.937 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024379390000000001, 2024-09-18T04:38:00.937 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:00.937 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:00.937 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:00.937 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.937 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:00.937 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790295+0000", 2024-09-18T04:38:00.937 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:00.937 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.937 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.937 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:00.938 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790295+0000", 2024-09-18T04:38:00.938 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:00.938 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.938 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.938 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:00.938 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790294+0000", 2024-09-18T04:38:00.938 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:00.938 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.938 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.938 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:00.938 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790390+0000", 2024-09-18T04:38:00.938 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6537000000000001e-05 2024-09-18T04:38:00.938 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.938 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.938 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:00.939 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790418+0000", 2024-09-18T04:38:00.939 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8218e-05 2024-09-18T04:38:00.939 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.939 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.939 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:00.939 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790431+0000", 2024-09-18T04:38:00.939 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2571999999999999e-05 2024-09-18T04:38:00.939 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.939 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.939 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:00.939 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790534+0000", 2024-09-18T04:38:00.939 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000103433 2024-09-18T04:38:00.939 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.939 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.940 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:00.940 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790565+0000", 2024-09-18T04:38:00.940 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0409999999999999e-05 2024-09-18T04:38:00.940 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.940 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.940 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:00.940 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.792651+0000", 2024-09-18T04:38:00.940 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020865250000000001 2024-09-18T04:38:00.940 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.940 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.940 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:00.940 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.792733+0000", 2024-09-18T04:38:00.940 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1477999999999995e-05 2024-09-18T04:38:00.940 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:00.941 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:00.941 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:00.941 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.941 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.941 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:109 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'71, mlcod=15'68)", 2024-09-18T04:38:00.941 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.795250+0000", 2024-09-18T04:38:00.941 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 33.100852734999997, 2024-09-18T04:38:00.941 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023990449999999998, 2024-09-18T04:38:00.941 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:00.941 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:00.941 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:00.941 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.941 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:00.941 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795250+0000", 2024-09-18T04:38:00.942 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:00.942 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.942 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.942 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:00.942 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795250+0000", 2024-09-18T04:38:00.942 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:00.942 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.942 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.942 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:00.942 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795248+0000", 2024-09-18T04:38:00.942 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:00.942 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.942 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.943 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:00.943 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795405+0000", 2024-09-18T04:38:00.943 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000157007 2024-09-18T04:38:00.943 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.943 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.943 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:00.943 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795422+0000", 2024-09-18T04:38:00.943 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7013000000000001e-05 2024-09-18T04:38:00.943 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.943 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.943 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:00.943 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795432+0000", 2024-09-18T04:38:00.943 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0448999999999999e-05 2024-09-18T04:38:00.943 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.944 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.944 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:00.944 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795534+0000", 2024-09-18T04:38:00.944 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000101195 2024-09-18T04:38:00.944 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.944 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.944 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:00.944 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795587+0000", 2024-09-18T04:38:00.944 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3356000000000002e-05 2024-09-18T04:38:00.944 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.944 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.944 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:00.944 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.797560+0000", 2024-09-18T04:38:00.944 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001973023 2024-09-18T04:38:00.945 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.945 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.945 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:00.945 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.797649+0000", 2024-09-18T04:38:00.945 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8864999999999997e-05 2024-09-18T04:38:00.945 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:00.945 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:00.945 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:00.945 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.945 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.945 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:115 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'77, mlcod=15'72)", 2024-09-18T04:38:00.945 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.803339+0000", 2024-09-18T04:38:00.946 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 33.092764017, 2024-09-18T04:38:00.946 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002385093, 2024-09-18T04:38:00.946 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:00.946 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:00.946 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:00.946 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.946 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:00.946 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803339+0000", 2024-09-18T04:38:00.946 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:00.946 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.946 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.946 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:00.946 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803339+0000", 2024-09-18T04:38:00.946 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:00.947 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.947 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.947 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:00.947 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803337+0000", 2024-09-18T04:38:00.947 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:00.947 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.947 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.947 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:00.947 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803475+0000", 2024-09-18T04:38:00.947 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013718199999999999 2024-09-18T04:38:00.947 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.947 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.947 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:00.947 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803507+0000", 2024-09-18T04:38:00.948 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2351000000000002e-05 2024-09-18T04:38:00.948 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.948 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.948 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:00.948 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803533+0000", 2024-09-18T04:38:00.948 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6010999999999999e-05 2024-09-18T04:38:00.948 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.948 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.948 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:00.948 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803609+0000", 2024-09-18T04:38:00.948 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.64e-05 2024-09-18T04:38:00.948 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.948 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.949 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:00.949 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803671+0000", 2024-09-18T04:38:00.949 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1467e-05 2024-09-18T04:38:00.949 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.949 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.949 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:00.949 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.805676+0000", 2024-09-18T04:38:00.949 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020047849999999998 2024-09-18T04:38:00.949 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.949 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.949 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:00.949 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.805724+0000", 2024-09-18T04:38:00.949 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8151e-05 2024-09-18T04:38:00.950 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:00.950 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:00.950 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:00.950 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.950 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.950 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:118 1.0 e15/14 1:30306672:devicehealth::main.db.0000000000000000:head v 15'80, mlcod=15'78)", 2024-09-18T04:38:00.950 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.807930+0000", 2024-09-18T04:38:00.950 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 33.088172514, 2024-09-18T04:38:00.950 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028002259999999998, 2024-09-18T04:38:00.950 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:00.950 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:00.950 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:00.950 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.951 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:00.951 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.807930+0000", 2024-09-18T04:38:00.951 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:00.951 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.951 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.951 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:00.951 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.807930+0000", 2024-09-18T04:38:00.951 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:00.951 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.951 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.951 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:00.951 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.807929+0000", 2024-09-18T04:38:00.951 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:00.951 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.952 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.952 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:00.952 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808024+0000", 2024-09-18T04:38:00.952 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5258000000000002e-05 2024-09-18T04:38:00.952 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.952 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.952 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:00.952 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808044+0000", 2024-09-18T04:38:00.952 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0203e-05 2024-09-18T04:38:00.952 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.952 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.952 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:00.952 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808053+0000", 2024-09-18T04:38:00.953 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2689999999999999e-06 2024-09-18T04:38:00.953 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.953 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.953 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:00.953 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808137+0000", 2024-09-18T04:38:00.953 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4441e-05 2024-09-18T04:38:00.953 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.953 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.953 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:00.953 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808172+0000", 2024-09-18T04:38:00.953 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4453999999999999e-05 2024-09-18T04:38:00.953 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.953 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.954 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:00.954 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.810663+0000", 2024-09-18T04:38:00.954 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024910269999999998 2024-09-18T04:38:00.954 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.954 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.954 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:00.954 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.810730+0000", 2024-09-18T04:38:00.954 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7658999999999997e-05 2024-09-18T04:38:00.954 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:00.954 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:00.954 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:00.954 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.954 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.955 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:129 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'87, mlcod=15'85)", 2024-09-18T04:38:00.955 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.231689+0000", 2024-09-18T04:38:00.955 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 32.664413500999999, 2024-09-18T04:38:00.955 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002376502, 2024-09-18T04:38:00.955 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:00.955 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:00.955 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:00.955 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.955 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:00.955 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231689+0000", 2024-09-18T04:38:00.955 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:00.955 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.955 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.955 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:00.956 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231689+0000", 2024-09-18T04:38:00.956 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:00.956 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.956 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.956 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:00.956 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231687+0000", 2024-09-18T04:38:00.956 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:00.956 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.956 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.956 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:00.956 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231702+0000", 2024-09-18T04:38:00.956 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4949999999999999e-05 2024-09-18T04:38:00.956 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.957 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.957 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:00.957 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231735+0000", 2024-09-18T04:38:00.957 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2276000000000001e-05 2024-09-18T04:38:00.957 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.957 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.957 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:00.957 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231750+0000", 2024-09-18T04:38:00.957 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5193e-05 2024-09-18T04:38:00.957 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.957 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.957 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:00.957 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231872+0000", 2024-09-18T04:38:00.957 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000122127 2024-09-18T04:38:00.957 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.957 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.958 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:00.958 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231936+0000", 2024-09-18T04:38:00.958 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4089000000000003e-05 2024-09-18T04:38:00.958 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.958 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.958 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:00.958 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234002+0000", 2024-09-18T04:38:00.958 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002065575 2024-09-18T04:38:00.958 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.958 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.958 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:00.958 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234066+0000", 2024-09-18T04:38:00.958 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3944999999999998e-05 2024-09-18T04:38:00.958 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:00.959 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:00.959 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:00.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.959 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.959 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:130 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'88, mlcod=15'85)", 2024-09-18T04:38:00.959 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.232853+0000", 2024-09-18T04:38:00.959 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 32.663249886000003, 2024-09-18T04:38:00.959 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027351739999999999, 2024-09-18T04:38:00.959 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:00.959 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:00.959 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:00.959 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.959 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:00.959 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232853+0000", 2024-09-18T04:38:00.959 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:00.960 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.960 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.960 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:00.960 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232853+0000", 2024-09-18T04:38:00.960 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:00.960 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.960 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.960 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:00.960 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232851+0000", 2024-09-18T04:38:00.960 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:00.960 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.960 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.960 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:00.960 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232866+0000", 2024-09-18T04:38:00.960 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4946000000000001e-05 2024-09-18T04:38:00.961 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.961 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.961 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:00.961 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232897+0000", 2024-09-18T04:38:00.961 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0982999999999998e-05 2024-09-18T04:38:00.961 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.961 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.961 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:00.961 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232915+0000", 2024-09-18T04:38:00.961 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8476000000000001e-05 2024-09-18T04:38:00.961 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.961 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.961 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:00.962 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.233029+0000", 2024-09-18T04:38:00.962 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011305400000000001 2024-09-18T04:38:00.962 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.962 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.962 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:00.962 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.233079+0000", 2024-09-18T04:38:00.962 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0668999999999999e-05 2024-09-18T04:38:00.962 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.962 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.962 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:00.962 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235534+0000", 2024-09-18T04:38:00.962 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024546030000000001 2024-09-18T04:38:00.962 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.963 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.963 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:00.963 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235588+0000", 2024-09-18T04:38:00.963 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4098999999999998e-05 2024-09-18T04:38:00.963 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:00.963 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:00.963 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:00.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.963 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.963 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:131 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'89, mlcod=15'85)", 2024-09-18T04:38:00.963 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.234089+0000", 2024-09-18T04:38:00.963 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 32.662014032000002, 2024-09-18T04:38:00.963 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026975609999999998, 2024-09-18T04:38:00.964 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:00.964 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:00.964 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:00.964 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.964 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:00.964 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234089+0000", 2024-09-18T04:38:00.964 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:00.964 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.964 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.964 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:00.964 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234089+0000", 2024-09-18T04:38:00.964 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:00.964 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.964 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.965 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:00.965 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234087+0000", 2024-09-18T04:38:00.965 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:00.965 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.965 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.965 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:00.965 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234145+0000", 2024-09-18T04:38:00.965 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8687999999999998e-05 2024-09-18T04:38:00.965 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.965 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.965 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:00.965 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234177+0000", 2024-09-18T04:38:00.965 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1526999999999998e-05 2024-09-18T04:38:00.966 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.966 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.966 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:00.966 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234190+0000", 2024-09-18T04:38:00.966 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2965999999999999e-05 2024-09-18T04:38:00.966 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.966 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.966 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:00.966 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234309+0000", 2024-09-18T04:38:00.966 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011875399999999999 2024-09-18T04:38:00.966 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.966 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.966 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:00.966 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234365+0000", 2024-09-18T04:38:00.967 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6436e-05 2024-09-18T04:38:00.967 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.967 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.967 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:00.967 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236675+0000", 2024-09-18T04:38:00.967 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023099520000000001 2024-09-18T04:38:00.967 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.967 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.967 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:00.967 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236786+0000", 2024-09-18T04:38:00.967 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000111025 2024-09-18T04:38:00.967 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:00.967 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:00.967 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:00.968 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.968 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.968 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:132 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'90, mlcod=15'85)", 2024-09-18T04:38:00.968 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.235300+0000", 2024-09-18T04:38:00.968 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 32.660802582000002, 2024-09-18T04:38:00.968 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025982599999999998, 2024-09-18T04:38:00.968 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:00.968 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:00.968 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:00.968 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.968 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:00.968 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235300+0000", 2024-09-18T04:38:00.968 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:00.968 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.969 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.969 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:00.969 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235300+0000", 2024-09-18T04:38:00.969 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:00.969 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.969 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.969 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:00.969 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235298+0000", 2024-09-18T04:38:00.969 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:00.969 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.969 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.969 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:00.969 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235318+0000", 2024-09-18T04:38:00.969 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9714000000000001e-05 2024-09-18T04:38:00.970 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.970 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.970 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:00.970 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235337+0000", 2024-09-18T04:38:00.970 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8892000000000001e-05 2024-09-18T04:38:00.970 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.970 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.970 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:00.970 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235348+0000", 2024-09-18T04:38:00.970 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.129e-05 2024-09-18T04:38:00.970 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.970 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.970 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:00.970 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235681+0000", 2024-09-18T04:38:00.970 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00033323000000000001 2024-09-18T04:38:00.971 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.971 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.971 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:00.971 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235733+0000", 2024-09-18T04:38:00.971 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1391999999999999e-05 2024-09-18T04:38:00.971 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.971 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.971 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:00.971 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237839+0000", 2024-09-18T04:38:00.971 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002105647 2024-09-18T04:38:00.971 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.971 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.971 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:00.971 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237898+0000", 2024-09-18T04:38:00.972 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9759000000000002e-05 2024-09-18T04:38:00.972 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:00.972 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:00.972 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:00.972 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.972 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.972 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:133 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'91, mlcod=15'86)", 2024-09-18T04:38:00.972 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.236920+0000", 2024-09-18T04:38:00.972 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 32.659182694000002, 2024-09-18T04:38:00.972 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024318410000000001, 2024-09-18T04:38:00.972 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:00.972 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:00.972 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:00.972 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.973 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:00.973 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236920+0000", 2024-09-18T04:38:00.973 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:00.973 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.973 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.973 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:00.973 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236920+0000", 2024-09-18T04:38:00.973 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:00.973 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.973 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.973 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:00.973 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236918+0000", 2024-09-18T04:38:00.973 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:00.973 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.974 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.974 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:00.974 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236931+0000", 2024-09-18T04:38:00.974 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2683999999999999e-05 2024-09-18T04:38:00.974 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.974 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.974 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:00.974 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236951+0000", 2024-09-18T04:38:00.974 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0292000000000002e-05 2024-09-18T04:38:00.974 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.974 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.975 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:00.975 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236962+0000", 2024-09-18T04:38:00.975 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0703000000000001e-05 2024-09-18T04:38:00.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.975 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.975 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:00.975 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237095+0000", 2024-09-18T04:38:00.975 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013312400000000001 2024-09-18T04:38:00.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.975 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.975 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:00.975 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237133+0000", 2024-09-18T04:38:00.975 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7496000000000001e-05 2024-09-18T04:38:00.976 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.976 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.976 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:00.976 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.239206+0000", 2024-09-18T04:38:00.976 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020730290000000001 2024-09-18T04:38:00.976 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.976 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.976 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:00.976 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.239352+0000", 2024-09-18T04:38:00.976 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014593000000000001 2024-09-18T04:38:00.976 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:00.976 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:00.976 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:00.976 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.977 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.977 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:134 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'92, mlcod=15'86)", 2024-09-18T04:38:00.977 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.237894+0000", 2024-09-18T04:38:00.977 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 32.658208672000001, 2024-09-18T04:38:00.977 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025299519999999998, 2024-09-18T04:38:00.977 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:00.977 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:00.977 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:00.977 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.977 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:00.977 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237894+0000", 2024-09-18T04:38:00.977 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:00.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.977 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.978 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:00.978 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237894+0000", 2024-09-18T04:38:00.978 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:00.978 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.978 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.978 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:00.978 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237892+0000", 2024-09-18T04:38:00.978 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:00.978 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.978 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.978 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:00.978 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237986+0000", 2024-09-18T04:38:00.978 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3341e-05 2024-09-18T04:38:00.978 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.979 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.979 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:00.979 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.238012+0000", 2024-09-18T04:38:00.979 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5843e-05 2024-09-18T04:38:00.979 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.979 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.979 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:00.979 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.238027+0000", 2024-09-18T04:38:00.979 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5469e-05 2024-09-18T04:38:00.979 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.979 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.979 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:00.979 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.238114+0000", 2024-09-18T04:38:00.980 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7483000000000002e-05 2024-09-18T04:38:00.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.980 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.980 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:00.980 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.238152+0000", 2024-09-18T04:38:00.980 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7565999999999999e-05 2024-09-18T04:38:00.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.980 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.980 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:00.980 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.240297+0000", 2024-09-18T04:38:00.980 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021447620000000001 2024-09-18T04:38:00.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.980 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.980 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:00.981 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.240424+0000", 2024-09-18T04:38:00.981 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012705400000000001 2024-09-18T04:38:00.981 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:00.981 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:00.981 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:00.981 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.981 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.981 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:139 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'97, mlcod=15'93)", 2024-09-18T04:38:00.981 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.245692+0000", 2024-09-18T04:38:00.981 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 32.650410530999999, 2024-09-18T04:38:00.981 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024260480000000001, 2024-09-18T04:38:00.981 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:00.981 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:00.981 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:00.981 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.982 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:00.982 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245692+0000", 2024-09-18T04:38:00.982 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:00.982 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.982 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.982 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:00.982 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245692+0000", 2024-09-18T04:38:00.982 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:00.982 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.982 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.982 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:00.982 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245690+0000", 2024-09-18T04:38:00.982 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:00.983 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.983 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.983 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:00.983 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245712+0000", 2024-09-18T04:38:00.983 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1916000000000002e-05 2024-09-18T04:38:00.983 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.983 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.983 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:00.983 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245741+0000", 2024-09-18T04:38:00.983 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8503000000000001e-05 2024-09-18T04:38:00.983 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.983 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.983 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:00.984 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245759+0000", 2024-09-18T04:38:00.984 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.872e-05 2024-09-18T04:38:00.984 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.984 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.984 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:00.984 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245859+0000", 2024-09-18T04:38:00.984 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9641000000000004e-05 2024-09-18T04:38:00.984 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.984 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.984 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:00.984 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245900+0000", 2024-09-18T04:38:00.984 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0589000000000001e-05 2024-09-18T04:38:00.984 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.985 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.985 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:00.985 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.247961+0000", 2024-09-18T04:38:00.985 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020619190000000002 2024-09-18T04:38:00.985 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.985 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.985 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:00.985 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.248118+0000", 2024-09-18T04:38:00.985 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015664300000000001 2024-09-18T04:38:00.985 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:00.985 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:00.985 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:00.986 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.986 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.986 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:144 1.0 e15/14 1:30306672:devicehealth::main.db.0000000000000000:head v 15'102, mlcod=15'98)", 2024-09-18T04:38:00.986 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.252149+0000", 2024-09-18T04:38:00.986 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 32.643953465000003, 2024-09-18T04:38:00.986 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024174959999999999, 2024-09-18T04:38:00.986 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:00.986 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:00.986 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:00.986 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.986 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:00.986 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252149+0000", 2024-09-18T04:38:00.986 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:00.987 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.987 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.987 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:00.987 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252149+0000", 2024-09-18T04:38:00.987 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:00.987 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.987 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.987 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:00.987 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252148+0000", 2024-09-18T04:38:00.987 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:00.987 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.987 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.987 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:00.988 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252225+0000", 2024-09-18T04:38:00.988 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6576999999999995e-05 2024-09-18T04:38:00.988 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.988 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.988 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:00.988 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252242+0000", 2024-09-18T04:38:00.988 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7740999999999998e-05 2024-09-18T04:38:00.988 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.988 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.988 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:00.988 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252254+0000", 2024-09-18T04:38:00.988 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.137e-05 2024-09-18T04:38:00.988 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.988 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.988 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:00.989 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252385+0000", 2024-09-18T04:38:00.989 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013091099999999999 2024-09-18T04:38:00.989 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.989 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.989 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:00.989 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252426+0000", 2024-09-18T04:38:00.989 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1535999999999997e-05 2024-09-18T04:38:00.989 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.989 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.989 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:00.989 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.254498+0000", 2024-09-18T04:38:00.989 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002071477 2024-09-18T04:38:00.989 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:00.989 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:00.989 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:00.990 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.254567+0000", 2024-09-18T04:38:00.990 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9014e-05 2024-09-18T04:38:00.990 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:00.990 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:00.990 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:00.990 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:00.990 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:00.990 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:00.992 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:38:00.992 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:00.986+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:01.093 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:01.087+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:01.106 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:01.106 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:01.106 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:38:01.106 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:01.116 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:38:01.194 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:01.190+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:01.225 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:01.225 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:01.226 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:38:01.226 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:38:01.226 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:01.235 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:38:01.295 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:01.291+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:01.344 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:01.344 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:38:01.344 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:38:01.344 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:38:01.345 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:01.345 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4214.0:5 2.2 e19/17 2:5cea7035:::rbd_info:head v 19'1, mlcod=0'0)", 2024-09-18T04:38:01.345 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:35.471260+0000", 2024-09-18T04:38:01.345 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 25.871983415999999, 2024-09-18T04:38:01.345 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021025950000000001, 2024-09-18T04:38:01.345 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:01.345 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:01.345 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:01.345 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:01.345 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:01.345 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471260+0000", 2024-09-18T04:38:01.345 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:01.345 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:01.345 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:01.346 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:01.346 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471260+0000", 2024-09-18T04:38:01.346 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:01.346 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:01.346 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:01.346 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:01.346 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471259+0000", 2024-09-18T04:38:01.346 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:01.346 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:01.346 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:01.346 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:01.347 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471275+0000", 2024-09-18T04:38:01.347 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6135999999999999e-05 2024-09-18T04:38:01.347 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:01.347 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:01.347 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:01.347 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471298+0000", 2024-09-18T04:38:01.347 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3632999999999999e-05 2024-09-18T04:38:01.347 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:01.347 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:01.347 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:01.347 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471320+0000", 2024-09-18T04:38:01.347 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1852999999999999e-05 2024-09-18T04:38:01.347 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:01.348 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:01.348 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:01.348 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471445+0000", 2024-09-18T04:38:01.348 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012461699999999999 2024-09-18T04:38:01.348 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:01.348 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:01.348 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:01.348 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471529+0000", 2024-09-18T04:38:01.348 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3875000000000006e-05 2024-09-18T04:38:01.348 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:01.349 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:01.349 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:01.349 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.473317+0000", 2024-09-18T04:38:01.349 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017886899999999999 2024-09-18T04:38:01.349 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:01.349 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:01.349 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:01.349 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.473362+0000", 2024-09-18T04:38:01.349 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4724000000000001e-05 2024-09-18T04:38:01.349 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:01.349 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:01.349 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:01.349 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:01.350 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:01.350 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4214.0:7 2.2 e20/17 2:5cea7035:::rbd_info:head v 20'2, mlcod=0'0)", 2024-09-18T04:38:01.350 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:36.490560+0000", 2024-09-18T04:38:01.350 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 24.852683277000001, 2024-09-18T04:38:01.350 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0015326020000000001, 2024-09-18T04:38:01.350 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:01.350 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:01.350 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:01.350 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:01.350 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:01.351 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490560+0000", 2024-09-18T04:38:01.351 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:01.351 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:01.351 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:01.351 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:01.351 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490560+0000", 2024-09-18T04:38:01.351 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:01.351 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:01.351 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:01.351 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:01.352 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490559+0000", 2024-09-18T04:38:01.352 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:01.352 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:01.352 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:01.352 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:01.352 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490568+0000", 2024-09-18T04:38:01.352 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5699999999999993e-06 2024-09-18T04:38:01.352 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:01.352 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:01.352 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:01.352 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490599+0000", 2024-09-18T04:38:01.353 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1566000000000002e-05 2024-09-18T04:38:01.353 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:01.353 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:01.353 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:01.353 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490609+0000", 2024-09-18T04:38:01.353 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.8700000000000004e-06 2024-09-18T04:38:01.353 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:01.353 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:01.353 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:01.353 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490708+0000", 2024-09-18T04:38:01.354 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9139000000000001e-05 2024-09-18T04:38:01.354 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:01.354 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:01.354 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:01.354 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490736+0000", 2024-09-18T04:38:01.354 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7617e-05 2024-09-18T04:38:01.354 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:01.354 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:01.354 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:01.355 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.492016+0000", 2024-09-18T04:38:01.355 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0012796579999999999 2024-09-18T04:38:01.355 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:01.355 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:01.355 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:01.355 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.492092+0000", 2024-09-18T04:38:01.355 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6571999999999998e-05 2024-09-18T04:38:01.355 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:01.355 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:01.355 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:01.355 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:01.355 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:01.355 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:01.356 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:38:01.396 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:01.390+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:01.465 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:01.465 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:01.465 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:38:01.465 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:01.475 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:38:01.497 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:01.491+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:01.586 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:01.586 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:01.586 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:38:01.586 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:38:01.586 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:01.597 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:38:01.598 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:01.594+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:01.699 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:01.694+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:01.739 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-18T04:38:01.751 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:01.751 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:38:01.751 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:38:01.751 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:38:01.752 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:01.752 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4214.0:1 2.3 e18/17 2:c4c92e5a:::rbd_trash:head v 18'1, mlcod=0'0)", 2024-09-18T04:38:01.752 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:34.475123+0000", 2024-09-18T04:38:01.752 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 27.275434689000001, 2024-09-18T04:38:01.752 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00081394599999999996, 2024-09-18T04:38:01.752 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:01.752 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:01.752 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:01.752 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:01.752 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:01.752 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475123+0000", 2024-09-18T04:38:01.752 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:01.752 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:01.753 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:01.753 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:01.753 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475123+0000", 2024-09-18T04:38:01.753 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:01.753 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:01.753 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:01.753 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:01.753 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475121+0000", 2024-09-18T04:38:01.753 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:01.753 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:01.753 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:01.753 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:01.754 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475149+0000", 2024-09-18T04:38:01.754 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7155999999999999e-05 2024-09-18T04:38:01.754 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:01.754 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:01.754 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:01.754 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475165+0000", 2024-09-18T04:38:01.754 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6160000000000001e-05 2024-09-18T04:38:01.754 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:01.754 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:01.754 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:01.754 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475183+0000", 2024-09-18T04:38:01.754 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7918e-05 2024-09-18T04:38:01.755 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:01.755 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:01.755 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:01.755 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475246+0000", 2024-09-18T04:38:01.755 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3176000000000006e-05 2024-09-18T04:38:01.755 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:01.755 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:01.755 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:01.755 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475279+0000", 2024-09-18T04:38:01.755 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3192999999999999e-05 2024-09-18T04:38:01.755 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:01.755 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:01.755 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:01.755 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475871+0000", 2024-09-18T04:38:01.756 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00059206099999999995 2024-09-18T04:38:01.756 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:01.756 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:01.756 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:01.756 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475937+0000", 2024-09-18T04:38:01.756 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5439999999999997e-05 2024-09-18T04:38:01.756 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:01.756 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:01.756 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:01.756 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:01.756 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:01.756 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 25/24 [PushOp(2:c4c92e5a:::rbd_trash:head, version: 18'1, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:c4c92e5a:::rbd_trash:head@18'1, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-18T04:38:01.757 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:51.680544+0000", 2024-09-18T04:38:01.757 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 10.070012846999999, 2024-09-18T04:38:01.757 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00050365200000000005, 2024-09-18T04:38:01.757 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:01.757 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:38:01.757 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:01.757 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:01.757 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:01.757 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680544+0000", 2024-09-18T04:38:01.757 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:01.757 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:01.757 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:01.757 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:01.757 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680544+0000", 2024-09-18T04:38:01.758 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:01.758 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:01.758 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:01.758 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:01.758 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680543+0000", 2024-09-18T04:38:01.758 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:01.758 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:01.758 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:01.758 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:01.758 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680571+0000", 2024-09-18T04:38:01.758 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7953000000000002e-05 2024-09-18T04:38:01.758 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:01.758 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:01.758 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:01.759 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680605+0000", 2024-09-18T04:38:01.759 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3933000000000002e-05 2024-09-18T04:38:01.759 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:01.759 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:01.759 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:01.759 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680636+0000", 2024-09-18T04:38:01.759 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1288999999999998e-05 2024-09-18T04:38:01.759 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:01.759 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:01.759 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:01.759 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680748+0000", 2024-09-18T04:38:01.759 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000111431 2024-09-18T04:38:01.759 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:01.760 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:01.760 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:01.760 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680758+0000", 2024-09-18T04:38:01.760 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0244999999999999e-05 2024-09-18T04:38:01.760 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:01.760 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:01.760 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:01.760 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.681048+0000", 2024-09-18T04:38:01.760 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00029003399999999999 2024-09-18T04:38:01.760 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:01.760 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:01.760 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:01.760 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:01.760 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:01.761 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:01.762 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:38:01.800 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:01.795+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:01.875 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:01.876 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:01.876 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:38:01.876 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:01.886 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:38:01.901 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:01.899+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:01.991 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:01.991 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:01.991 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:38:01.991 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:38:01.991 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:02.001 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:38:02.002 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:01.994+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:02.063 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T04:38:02.110 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:02.107+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:02.111 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:02.111 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:38:02.111 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:38:02.111 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:38:02.111 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.111 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:02.111 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.573328+0000", 2024-09-18T04:38:02.111 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 34.531328694000003, 2024-09-18T04:38:02.111 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011397898, 2024-09-18T04:38:02.111 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:02.112 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:02.112 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:02.112 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:02.112 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:02.112 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 86 2024-09-18T04:38:02.112 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.112 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:02.112 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.112 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:02.112 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573328+0000", 2024-09-18T04:38:02.112 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:02.112 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.112 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.113 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:02.113 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573328+0000", 2024-09-18T04:38:02.113 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:02.113 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.113 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.113 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:02.113 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573327+0000", 2024-09-18T04:38:02.113 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:02.113 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.113 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.113 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:02.113 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573331+0000", 2024-09-18T04:38:02.113 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9750000000000001e-06 2024-09-18T04:38:02.114 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.114 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.114 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:02.114 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573346+0000", 2024-09-18T04:38:02.114 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5469999999999999e-05 2024-09-18T04:38:02.114 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.114 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.114 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:02.114 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573370+0000", 2024-09-18T04:38:02.114 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3969999999999999e-05 2024-09-18T04:38:02.114 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.114 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.114 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:02.114 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.578536+0000", 2024-09-18T04:38:02.115 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0051659679999999999 2024-09-18T04:38:02.115 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.115 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.115 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:02.115 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.578689+0000", 2024-09-18T04:38:02.115 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015276299999999999 2024-09-18T04:38:02.115 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.115 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.115 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:02.115 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.578901+0000", 2024-09-18T04:38:02.115 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000211783 2024-09-18T04:38:02.115 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.115 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.115 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:02.116 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.583408+0000", 2024-09-18T04:38:02.116 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045071249999999998 2024-09-18T04:38:02.116 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.116 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.116 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:02.116 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.584645+0000", 2024-09-18T04:38:02.116 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001237007 2024-09-18T04:38:02.116 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.116 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.116 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:02.116 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.584703+0000", 2024-09-18T04:38:02.116 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.817e-05 2024-09-18T04:38:02.116 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.116 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.117 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:02.117 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.584725+0000", 2024-09-18T04:38:02.117 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2155e-05 2024-09-18T04:38:02.117 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:02.117 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:02.117 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:02.117 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.117 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.117 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:02.117 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.573382+0000", 2024-09-18T04:38:02.117 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 34.531273945999999, 2024-09-18T04:38:02.117 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012154974000000001, 2024-09-18T04:38:02.117 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:02.117 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:02.118 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:02.118 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:02.118 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:02.118 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 87 2024-09-18T04:38:02.118 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.118 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:02.118 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.118 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:02.118 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573382+0000", 2024-09-18T04:38:02.118 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:02.118 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.118 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.118 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:02.118 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573382+0000", 2024-09-18T04:38:02.119 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:02.119 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.119 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.119 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:02.119 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573382+0000", 2024-09-18T04:38:02.119 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:02.119 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.119 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.119 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:02.119 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573482+0000", 2024-09-18T04:38:02.119 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000100333 2024-09-18T04:38:02.119 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.119 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.120 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:02.120 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573492+0000", 2024-09-18T04:38:02.120 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5389999999999996e-06 2024-09-18T04:38:02.120 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.120 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.120 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:02.120 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573500+0000", 2024-09-18T04:38:02.120 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8929999999999994e-06 2024-09-18T04:38:02.120 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.120 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.120 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:02.120 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.579797+0000", 2024-09-18T04:38:02.120 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0062963239999999998 2024-09-18T04:38:02.120 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.121 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.121 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:02.121 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.579957+0000", 2024-09-18T04:38:02.121 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000160283 2024-09-18T04:38:02.121 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.121 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.121 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:02.121 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.580178+0000", 2024-09-18T04:38:02.121 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00022079200000000001 2024-09-18T04:38:02.121 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.121 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.121 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:02.121 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.583479+0000", 2024-09-18T04:38:02.121 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033007689999999998 2024-09-18T04:38:02.122 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.122 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.122 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:02.122 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585477+0000", 2024-09-18T04:38:02.122 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0019985670000000001 2024-09-18T04:38:02.122 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.122 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.122 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:02.122 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585517+0000", 2024-09-18T04:38:02.122 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9709999999999998e-05 2024-09-18T04:38:02.122 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.122 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.122 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:02.123 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585537+0000", 2024-09-18T04:38:02.123 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0326000000000002e-05 2024-09-18T04:38:02.123 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:02.123 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:02.123 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:02.123 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.123 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.123 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:02.123 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.573508+0000", 2024-09-18T04:38:02.123 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 34.53114824, 2024-09-18T04:38:02.123 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012068476, 2024-09-18T04:38:02.123 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:02.123 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:02.123 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:02.123 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:02.124 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:02.124 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 88 2024-09-18T04:38:02.124 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.124 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:02.124 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.124 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:02.124 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573508+0000", 2024-09-18T04:38:02.124 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:02.124 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.124 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.125 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:02.125 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573508+0000", 2024-09-18T04:38:02.125 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:02.125 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.125 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.125 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:02.125 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573507+0000", 2024-09-18T04:38:02.125 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:02.125 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.125 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.125 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:02.126 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573512+0000", 2024-09-18T04:38:02.126 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.2629999999999997e-06 2024-09-18T04:38:02.126 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.126 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.126 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:02.126 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573530+0000", 2024-09-18T04:38:02.126 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8507000000000001e-05 2024-09-18T04:38:02.126 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.126 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.126 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:02.126 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573537+0000", 2024-09-18T04:38:02.126 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.832e-06 2024-09-18T04:38:02.126 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.126 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.127 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:02.127 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.582486+0000", 2024-09-18T04:38:02.127 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0089494770000000008 2024-09-18T04:38:02.127 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.127 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.127 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:02.127 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.582630+0000", 2024-09-18T04:38:02.127 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000144005 2024-09-18T04:38:02.127 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.127 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.128 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:02.128 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.582842+0000", 2024-09-18T04:38:02.128 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00021189200000000001 2024-09-18T04:38:02.128 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.128 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.128 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:02.129 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.584777+0000", 2024-09-18T04:38:02.129 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001934753 2024-09-18T04:38:02.129 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.129 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.129 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:02.129 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585543+0000", 2024-09-18T04:38:02.129 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00076540099999999997 2024-09-18T04:38:02.129 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.129 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.129 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:02.129 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585561+0000", 2024-09-18T04:38:02.130 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8474999999999998e-05 2024-09-18T04:38:02.130 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.130 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.130 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:02.130 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585576+0000", 2024-09-18T04:38:02.131 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5427e-05 2024-09-18T04:38:02.131 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:02.131 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:02.131 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:02.131 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.131 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.131 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:02.131 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786394+0000", 2024-09-18T04:38:02.131 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 34.318261792999998, 2024-09-18T04:38:02.131 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011902085999999999, 2024-09-18T04:38:02.131 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:02.131 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:02.132 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:02.132 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:02.132 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:02.132 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 109 2024-09-18T04:38:02.132 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.132 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:02.132 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.132 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:02.132 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786394+0000", 2024-09-18T04:38:02.132 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:02.132 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.132 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.132 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:02.133 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786394+0000", 2024-09-18T04:38:02.133 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:02.133 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.133 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.133 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:02.133 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786394+0000", 2024-09-18T04:38:02.133 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:02.133 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.133 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.133 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:02.133 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786548+0000", 2024-09-18T04:38:02.133 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015463400000000001 2024-09-18T04:38:02.133 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.133 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.134 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:02.134 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786561+0000", 2024-09-18T04:38:02.134 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3093e-05 2024-09-18T04:38:02.134 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.134 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.134 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:02.134 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786571+0000", 2024-09-18T04:38:02.134 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4700000000000008e-06 2024-09-18T04:38:02.134 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.134 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.134 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:02.134 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.793367+0000", 2024-09-18T04:38:02.134 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0067966600000000004 2024-09-18T04:38:02.134 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.135 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.135 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:02.135 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.793526+0000", 2024-09-18T04:38:02.135 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015877 2024-09-18T04:38:02.135 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.135 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.135 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:02.135 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.793752+0000", 2024-09-18T04:38:02.135 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00022555200000000001 2024-09-18T04:38:02.135 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.135 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.135 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:02.135 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798125+0000", 2024-09-18T04:38:02.135 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0043733349999999999 2024-09-18T04:38:02.136 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.136 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.136 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:02.136 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798146+0000", 2024-09-18T04:38:02.136 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1063000000000001e-05 2024-09-18T04:38:02.136 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.136 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.136 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:02.136 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798232+0000", 2024-09-18T04:38:02.136 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5771000000000003e-05 2024-09-18T04:38:02.137 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.137 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.137 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:02.137 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798297+0000", 2024-09-18T04:38:02.137 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.4624999999999999e-05 2024-09-18T04:38:02.137 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:02.137 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:02.137 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:02.137 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.137 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.137 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:02.137 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786584+0000", 2024-09-18T04:38:02.137 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 34.318071713999998, 2024-09-18T04:38:02.138 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012013037000000001, 2024-09-18T04:38:02.138 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:02.138 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:02.138 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:02.138 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:02.138 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:02.138 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 110 2024-09-18T04:38:02.138 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.138 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:02.138 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.138 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:02.138 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786584+0000", 2024-09-18T04:38:02.138 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:02.138 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.139 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.139 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:02.139 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786584+0000", 2024-09-18T04:38:02.139 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:02.139 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.139 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.139 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:02.139 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786584+0000", 2024-09-18T04:38:02.139 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:02.139 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.139 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.139 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:02.140 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786590+0000", 2024-09-18T04:38:02.140 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6359999999999997e-06 2024-09-18T04:38:02.140 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.140 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.140 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:02.140 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786599+0000", 2024-09-18T04:38:02.140 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7369999999999995e-06 2024-09-18T04:38:02.140 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.140 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.140 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:02.140 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786606+0000", 2024-09-18T04:38:02.140 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1930000000000004e-06 2024-09-18T04:38:02.140 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.141 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.141 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:02.141 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.795018+0000", 2024-09-18T04:38:02.141 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0084115660000000005 2024-09-18T04:38:02.141 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.141 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.141 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:02.141 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.795184+0000", 2024-09-18T04:38:02.141 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000165833 2024-09-18T04:38:02.141 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.141 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.141 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:02.141 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.795414+0000", 2024-09-18T04:38:02.142 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00023088399999999999 2024-09-18T04:38:02.142 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.142 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.142 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:02.142 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798321+0000", 2024-09-18T04:38:02.142 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0029069539999999998 2024-09-18T04:38:02.142 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.142 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.142 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:02.142 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798404+0000", 2024-09-18T04:38:02.142 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2724000000000003e-05 2024-09-18T04:38:02.142 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.142 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.142 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:02.143 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798488+0000", 2024-09-18T04:38:02.143 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.3505999999999996e-05 2024-09-18T04:38:02.143 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.143 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.143 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:02.143 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798598+0000", 2024-09-18T04:38:02.143 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000109917 2024-09-18T04:38:02.143 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:02.143 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:02.143 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:02.143 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.143 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.143 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:02.143 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786613+0000", 2024-09-18T04:38:02.144 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 34.318043013999997, 2024-09-18T04:38:02.144 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014245760999999999, 2024-09-18T04:38:02.144 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:02.144 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:02.144 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:02.144 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:02.144 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:02.144 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 111 2024-09-18T04:38:02.144 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.144 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:02.144 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.144 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:02.145 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786613+0000", 2024-09-18T04:38:02.145 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:02.145 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.145 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.145 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:02.145 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786613+0000", 2024-09-18T04:38:02.145 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:02.145 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.145 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.145 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:02.145 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786612+0000", 2024-09-18T04:38:02.145 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:02.145 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.145 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.146 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:02.146 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786622+0000", 2024-09-18T04:38:02.146 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4099999999999997e-06 2024-09-18T04:38:02.146 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.146 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.146 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:02.146 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786630+0000", 2024-09-18T04:38:02.146 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4169999999999992e-06 2024-09-18T04:38:02.146 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.146 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.146 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:02.146 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786637+0000", 2024-09-18T04:38:02.146 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8290000000000002e-06 2024-09-18T04:38:02.146 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.147 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.147 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:02.147 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.797099+0000", 2024-09-18T04:38:02.147 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010461747 2024-09-18T04:38:02.147 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.147 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.147 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:02.147 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.797243+0000", 2024-09-18T04:38:02.147 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014388899999999999 2024-09-18T04:38:02.147 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.147 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.147 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:02.147 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.797463+0000", 2024-09-18T04:38:02.148 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000219912 2024-09-18T04:38:02.148 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.148 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.148 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:02.148 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800732+0000", 2024-09-18T04:38:02.148 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0032688719999999999 2024-09-18T04:38:02.148 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.148 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.148 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:02.148 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800751+0000", 2024-09-18T04:38:02.148 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9633999999999999e-05 2024-09-18T04:38:02.148 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.148 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.148 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:02.149 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800824+0000", 2024-09-18T04:38:02.149 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.2805999999999994e-05 2024-09-18T04:38:02.149 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.149 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.149 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:02.149 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800859+0000", 2024-09-18T04:38:02.149 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4952e-05 2024-09-18T04:38:02.149 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:02.149 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:02.149 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:02.150 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.150 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.150 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:02.150 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786649+0000", 2024-09-18T04:38:02.150 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 34.318007604999998, 2024-09-18T04:38:02.151 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.01613442, 2024-09-18T04:38:02.151 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:02.151 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:02.151 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:02.151 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:02.152 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:02.152 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 112 2024-09-18T04:38:02.152 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.152 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:02.152 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.152 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:02.152 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786649+0000", 2024-09-18T04:38:02.152 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:02.152 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.152 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.152 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:02.152 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786649+0000", 2024-09-18T04:38:02.152 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:02.153 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.153 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.153 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:02.153 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786648+0000", 2024-09-18T04:38:02.153 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:02.153 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.153 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.153 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:02.153 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786768+0000", 2024-09-18T04:38:02.153 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000120491 2024-09-18T04:38:02.153 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.153 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.153 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:02.154 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786795+0000", 2024-09-18T04:38:02.154 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6556000000000001e-05 2024-09-18T04:38:02.154 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.154 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.154 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:02.154 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786807+0000", 2024-09-18T04:38:02.154 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1739999999999999e-05 2024-09-18T04:38:02.154 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.154 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.154 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:02.154 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798683+0000", 2024-09-18T04:38:02.154 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011876070000000001 2024-09-18T04:38:02.154 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.154 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.155 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:02.155 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798867+0000", 2024-09-18T04:38:02.155 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000184103 2024-09-18T04:38:02.155 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.155 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.155 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:02.155 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.799082+0000", 2024-09-18T04:38:02.155 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00021476000000000001 2024-09-18T04:38:02.155 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.155 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.156 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:02.156 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802610+0000", 2024-09-18T04:38:02.156 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0035282209999999998 2024-09-18T04:38:02.156 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.156 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.156 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:02.156 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802706+0000", 2024-09-18T04:38:02.156 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5922999999999996e-05 2024-09-18T04:38:02.156 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.157 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.157 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:02.157 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802750+0000", 2024-09-18T04:38:02.157 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4511000000000001e-05 2024-09-18T04:38:02.157 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.157 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.157 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:02.157 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802783+0000", 2024-09-18T04:38:02.157 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2752000000000001e-05 2024-09-18T04:38:02.157 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:02.157 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:02.157 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:02.157 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.157 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.158 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:02.158 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786816+0000", 2024-09-18T04:38:02.158 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 34.317840537999999, 2024-09-18T04:38:02.158 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016197989999999999, 2024-09-18T04:38:02.158 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:02.158 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:02.158 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:02.158 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:02.158 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:02.158 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 113 2024-09-18T04:38:02.158 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.158 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:02.158 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.159 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:02.159 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786816+0000", 2024-09-18T04:38:02.159 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:02.159 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.159 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.159 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:02.159 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786816+0000", 2024-09-18T04:38:02.159 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:02.159 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.159 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.159 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:02.159 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786815+0000", 2024-09-18T04:38:02.159 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:02.159 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.160 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.160 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:02.160 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786821+0000", 2024-09-18T04:38:02.160 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9529999999999996e-06 2024-09-18T04:38:02.160 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.160 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.160 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:02.160 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786841+0000", 2024-09-18T04:38:02.160 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9743000000000001e-05 2024-09-18T04:38:02.161 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.161 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.161 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:02.161 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786848+0000", 2024-09-18T04:38:02.161 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4900000000000003e-06 2024-09-18T04:38:02.161 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.161 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.161 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:02.161 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.799747+0000", 2024-09-18T04:38:02.162 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012898902 2024-09-18T04:38:02.162 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.162 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.162 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:02.162 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.799898+0000", 2024-09-18T04:38:02.162 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000150877 2024-09-18T04:38:02.162 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.162 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.162 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:02.162 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800095+0000", 2024-09-18T04:38:02.162 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000197023 2024-09-18T04:38:02.162 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.162 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.163 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:02.163 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802829+0000", 2024-09-18T04:38:02.163 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0027343480000000002 2024-09-18T04:38:02.163 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.163 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.163 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:02.163 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802847+0000", 2024-09-18T04:38:02.163 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7283e-05 2024-09-18T04:38:02.164 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.164 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.164 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:02.164 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802974+0000", 2024-09-18T04:38:02.164 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012749199999999999 2024-09-18T04:38:02.164 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.165 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.165 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:02.165 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803014+0000", 2024-09-18T04:38:02.165 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9576000000000002e-05 2024-09-18T04:38:02.165 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:02.165 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:02.165 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:02.165 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.165 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.165 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:02.165 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:38:02.165 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 34.317754946999997, 2024-09-18T04:38:02.166 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017399218000000001, 2024-09-18T04:38:02.166 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:02.166 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:02.166 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:02.166 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:02.166 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:02.167 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 114 2024-09-18T04:38:02.167 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.167 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:02.167 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.167 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:02.167 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:38:02.168 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:02.168 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.168 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.168 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:02.168 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:38:02.168 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:02.168 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.168 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.168 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:02.168 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786900+0000", 2024-09-18T04:38:02.168 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:02.168 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.168 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.168 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:02.169 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786908+0000", 2024-09-18T04:38:02.169 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8199999999999997e-06 2024-09-18T04:38:02.169 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.169 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.169 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:02.169 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786925+0000", 2024-09-18T04:38:02.169 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6849000000000001e-05 2024-09-18T04:38:02.169 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.169 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.169 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:02.169 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786934+0000", 2024-09-18T04:38:02.170 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6299999999999993e-06 2024-09-18T04:38:02.170 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.170 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.170 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:02.170 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800926+0000", 2024-09-18T04:38:02.170 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013991632 2024-09-18T04:38:02.170 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.170 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.170 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:02.170 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801060+0000", 2024-09-18T04:38:02.170 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013378199999999999 2024-09-18T04:38:02.170 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.170 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.171 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:02.171 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801166+0000", 2024-09-18T04:38:02.171 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010607099999999999 2024-09-18T04:38:02.171 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.171 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.171 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:02.171 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803026+0000", 2024-09-18T04:38:02.171 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001860469 2024-09-18T04:38:02.171 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.171 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.171 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:02.171 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804245+0000", 2024-09-18T04:38:02.171 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0012185900000000001 2024-09-18T04:38:02.171 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.171 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.172 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:02.172 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804278+0000", 2024-09-18T04:38:02.172 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3272000000000003e-05 2024-09-18T04:38:02.172 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.172 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.172 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:02.172 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804300+0000", 2024-09-18T04:38:02.172 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2266e-05 2024-09-18T04:38:02.172 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:02.172 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:02.172 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:02.172 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.173 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.173 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:02.173 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787566+0000", 2024-09-18T04:38:02.173 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 34.317089987999999, 2024-09-18T04:38:02.173 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017124120999999999, 2024-09-18T04:38:02.173 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:02.173 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:02.173 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:02.173 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:02.173 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:02.174 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 115 2024-09-18T04:38:02.174 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.174 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:02.174 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.174 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:02.175 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787566+0000", 2024-09-18T04:38:02.175 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:02.175 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.175 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.175 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:02.175 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787566+0000", 2024-09-18T04:38:02.176 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:02.176 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.176 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.176 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:02.176 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787564+0000", 2024-09-18T04:38:02.176 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:02.177 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.177 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.177 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:02.177 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787751+0000", 2024-09-18T04:38:02.177 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000186873 2024-09-18T04:38:02.177 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.177 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.177 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:02.177 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787772+0000", 2024-09-18T04:38:02.177 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.126e-05 2024-09-18T04:38:02.177 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.177 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.177 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:02.178 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787794+0000", 2024-09-18T04:38:02.178 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1418999999999999e-05 2024-09-18T04:38:02.178 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.178 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.178 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:02.178 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801659+0000", 2024-09-18T04:38:02.178 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013865125000000001 2024-09-18T04:38:02.178 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.178 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.178 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:02.178 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801750+0000", 2024-09-18T04:38:02.178 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.1427000000000004e-05 2024-09-18T04:38:02.178 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.179 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.179 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:02.179 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801894+0000", 2024-09-18T04:38:02.179 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000143514 2024-09-18T04:38:02.179 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.179 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.179 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:02.179 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804313+0000", 2024-09-18T04:38:02.179 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0024190879999999998 2024-09-18T04:38:02.179 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.179 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.180 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:02.180 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804644+0000", 2024-09-18T04:38:02.180 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00033100799999999999 2024-09-18T04:38:02.180 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.180 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.180 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:02.180 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804672+0000", 2024-09-18T04:38:02.180 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7988e-05 2024-09-18T04:38:02.181 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.181 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.181 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:02.181 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804690+0000", 2024-09-18T04:38:02.181 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8298999999999999e-05 2024-09-18T04:38:02.181 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:02.181 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:02.181 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:02.181 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.181 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.181 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:02.181 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:38:02.181 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 34.316842049000002, 2024-09-18T04:38:02.182 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017521662, 2024-09-18T04:38:02.182 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:02.182 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:02.182 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:02.182 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:02.182 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:02.182 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 116 2024-09-18T04:38:02.182 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.182 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:02.182 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.182 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:02.182 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:38:02.182 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:02.182 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.183 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.183 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:02.183 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:38:02.183 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:02.183 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.183 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.183 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:02.183 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787813+0000", 2024-09-18T04:38:02.183 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:02.183 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.183 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.184 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:02.184 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787824+0000", 2024-09-18T04:38:02.184 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1157e-05 2024-09-18T04:38:02.184 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.184 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.184 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:02.184 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787840+0000", 2024-09-18T04:38:02.184 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6269e-05 2024-09-18T04:38:02.184 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.184 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.184 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:02.184 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787853+0000", 2024-09-18T04:38:02.184 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3156000000000001e-05 2024-09-18T04:38:02.184 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.185 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.185 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:02.185 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803110+0000", 2024-09-18T04:38:02.185 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015256736 2024-09-18T04:38:02.185 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.185 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.185 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:02.185 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803197+0000", 2024-09-18T04:38:02.186 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7195999999999997e-05 2024-09-18T04:38:02.186 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.186 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.186 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:02.186 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803308+0000", 2024-09-18T04:38:02.186 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111005 2024-09-18T04:38:02.187 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.187 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.187 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:02.187 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804316+0000", 2024-09-18T04:38:02.187 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010073949999999999 2024-09-18T04:38:02.187 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.187 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.188 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:02.188 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805232+0000", 2024-09-18T04:38:02.188 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091636300000000001 2024-09-18T04:38:02.188 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.188 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.188 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:02.188 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805277+0000", 2024-09-18T04:38:02.188 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4864000000000003e-05 2024-09-18T04:38:02.188 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.188 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.188 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:02.188 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805336+0000", 2024-09-18T04:38:02.189 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8968000000000002e-05 2024-09-18T04:38:02.189 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:02.189 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:02.189 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:02.189 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.189 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.189 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:02.189 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:38:02.189 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 34.316789993999997, 2024-09-18T04:38:02.189 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017983453, 2024-09-18T04:38:02.189 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:02.189 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:02.189 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:02.190 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:02.190 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:02.190 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 117 2024-09-18T04:38:02.190 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.190 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:02.190 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.190 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:02.190 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:38:02.190 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:02.190 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.190 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.190 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:02.190 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:38:02.190 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:02.191 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.191 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.191 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:02.191 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787865+0000", 2024-09-18T04:38:02.191 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:02.191 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.191 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.191 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:02.191 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787889+0000", 2024-09-18T04:38:02.191 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4244999999999999e-05 2024-09-18T04:38:02.191 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.191 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.191 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:02.192 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787905+0000", 2024-09-18T04:38:02.192 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5739999999999998e-05 2024-09-18T04:38:02.192 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.192 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.192 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:02.192 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787918+0000", 2024-09-18T04:38:02.192 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.252e-05 2024-09-18T04:38:02.192 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.192 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.192 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:02.192 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803693+0000", 2024-09-18T04:38:02.192 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015774872999999998 2024-09-18T04:38:02.192 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.192 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.193 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:02.193 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803775+0000", 2024-09-18T04:38:02.193 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2761000000000003e-05 2024-09-18T04:38:02.193 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.193 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.193 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:02.193 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803887+0000", 2024-09-18T04:38:02.193 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111708 2024-09-18T04:38:02.193 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.193 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.193 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:02.193 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804823+0000", 2024-09-18T04:38:02.194 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00093625100000000003 2024-09-18T04:38:02.194 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.194 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.194 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:02.194 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805804+0000", 2024-09-18T04:38:02.194 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00098114699999999992 2024-09-18T04:38:02.194 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.194 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.194 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:02.194 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805831+0000", 2024-09-18T04:38:02.194 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6401999999999999e-05 2024-09-18T04:38:02.194 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.194 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.194 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:02.195 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805850+0000", 2024-09-18T04:38:02.195 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8862e-05 2024-09-18T04:38:02.195 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:02.195 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:02.195 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:02.195 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.195 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.195 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:02.195 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.230830+0000", 2024-09-18T04:38:02.195 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 33.873826115, 2024-09-18T04:38:02.195 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011537146999999999, 2024-09-18T04:38:02.195 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:02.195 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:02.195 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:02.196 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:02.196 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:02.196 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 135 2024-09-18T04:38:02.196 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.196 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:02.196 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.196 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:02.196 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230830+0000", 2024-09-18T04:38:02.196 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:02.196 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.196 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.196 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:02.196 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230830+0000", 2024-09-18T04:38:02.196 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:02.197 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.197 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.197 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:02.197 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230829+0000", 2024-09-18T04:38:02.197 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:02.197 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.197 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.197 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:02.197 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230835+0000", 2024-09-18T04:38:02.197 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8660000000000002e-06 2024-09-18T04:38:02.197 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.197 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.197 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:02.197 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230844+0000", 2024-09-18T04:38:02.198 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9339999999999995e-06 2024-09-18T04:38:02.198 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.198 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.198 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:02.198 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230853+0000", 2024-09-18T04:38:02.198 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4419999999999998e-06 2024-09-18T04:38:02.198 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.198 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.198 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:02.198 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.237658+0000", 2024-09-18T04:38:02.198 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0068055310000000001 2024-09-18T04:38:02.198 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.198 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.199 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:02.199 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.237837+0000", 2024-09-18T04:38:02.199 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017943200000000001 2024-09-18T04:38:02.199 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.199 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.199 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:02.199 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.238112+0000", 2024-09-18T04:38:02.199 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00027488600000000001 2024-09-18T04:38:02.199 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.199 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.199 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:02.199 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.240505+0000", 2024-09-18T04:38:02.199 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0023930539999999999 2024-09-18T04:38:02.199 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.200 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.200 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:02.200 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242157+0000", 2024-09-18T04:38:02.200 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001651135 2024-09-18T04:38:02.200 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.200 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.200 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:02.200 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242281+0000", 2024-09-18T04:38:02.200 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012451300000000001 2024-09-18T04:38:02.200 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.200 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.200 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:02.200 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242367+0000", 2024-09-18T04:38:02.201 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6178000000000001e-05 2024-09-18T04:38:02.201 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:02.201 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:02.201 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:02.201 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.201 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.201 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:02.201 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.230858+0000", 2024-09-18T04:38:02.201 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 33.873798227000002, 2024-09-18T04:38:02.201 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013005821000000001, 2024-09-18T04:38:02.201 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:02.201 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:02.201 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:02.201 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:02.202 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:02.202 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 136 2024-09-18T04:38:02.202 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.202 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:02.202 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.202 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:02.202 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230858+0000", 2024-09-18T04:38:02.202 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:02.202 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.202 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.202 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:02.202 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230858+0000", 2024-09-18T04:38:02.202 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:02.202 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.203 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.203 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:02.203 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230858+0000", 2024-09-18T04:38:02.203 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:02.203 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.203 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.203 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:02.203 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230861+0000", 2024-09-18T04:38:02.203 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7459999999999998e-06 2024-09-18T04:38:02.203 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.203 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.203 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:02.203 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230877+0000", 2024-09-18T04:38:02.204 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5979000000000001e-05 2024-09-18T04:38:02.204 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.204 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.204 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:02.204 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230885+0000", 2024-09-18T04:38:02.204 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.5979999999999999e-06 2024-09-18T04:38:02.204 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.204 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.204 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:02.204 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.239023+0000", 2024-09-18T04:38:02.204 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0081380710000000002 2024-09-18T04:38:02.204 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.204 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.204 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:02.205 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.239216+0000", 2024-09-18T04:38:02.205 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019274799999999999 2024-09-18T04:38:02.205 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.205 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.205 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:02.205 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.239417+0000", 2024-09-18T04:38:02.205 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020093300000000001 2024-09-18T04:38:02.205 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.205 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.205 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:02.205 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242494+0000", 2024-09-18T04:38:02.205 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003077177 2024-09-18T04:38:02.205 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.206 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.206 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:02.206 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.243668+0000", 2024-09-18T04:38:02.206 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001174027 2024-09-18T04:38:02.206 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.206 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.206 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:02.206 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.243777+0000", 2024-09-18T04:38:02.206 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010955600000000001 2024-09-18T04:38:02.206 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.206 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.206 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:02.207 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.243864+0000", 2024-09-18T04:38:02.207 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6441999999999997e-05 2024-09-18T04:38:02.207 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:02.207 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:02.207 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:02.207 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.207 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.207 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:02.207 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.230901+0000", 2024-09-18T04:38:02.207 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 33.873755654, 2024-09-18T04:38:02.207 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014120882, 2024-09-18T04:38:02.207 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:02.208 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:02.208 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:02.208 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:02.208 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:02.208 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 137 2024-09-18T04:38:02.208 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.208 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:02.208 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.208 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:02.208 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230901+0000", 2024-09-18T04:38:02.208 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:02.208 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.208 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.209 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:02.209 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230901+0000", 2024-09-18T04:38:02.209 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:02.209 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.209 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.209 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:02.209 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230900+0000", 2024-09-18T04:38:02.209 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:02.209 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.209 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.209 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:02.209 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231002+0000", 2024-09-18T04:38:02.209 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000102392 2024-09-18T04:38:02.210 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.210 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.210 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:02.210 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231029+0000", 2024-09-18T04:38:02.210 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7166999999999999e-05 2024-09-18T04:38:02.210 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.210 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.210 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:02.210 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231036+0000", 2024-09-18T04:38:02.210 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7259999999999996e-06 2024-09-18T04:38:02.210 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.210 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.211 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:02.211 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.240562+0000", 2024-09-18T04:38:02.211 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0095257970000000008 2024-09-18T04:38:02.211 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.211 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.211 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:02.211 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.240730+0000", 2024-09-18T04:38:02.211 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016767099999999999 2024-09-18T04:38:02.211 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.211 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.211 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:02.211 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.240927+0000", 2024-09-18T04:38:02.211 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019711899999999999 2024-09-18T04:38:02.211 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.212 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.212 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:02.212 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244885+0000", 2024-09-18T04:38:02.212 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0039577340000000001 2024-09-18T04:38:02.212 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.212 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.212 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:02.212 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244899+0000", 2024-09-18T04:38:02.212 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4403000000000001e-05 2024-09-18T04:38:02.212 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.212 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.212 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:02.213 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244973+0000", 2024-09-18T04:38:02.213 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4177999999999994e-05 2024-09-18T04:38:02.213 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.213 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.213 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:02.213 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.245021+0000", 2024-09-18T04:38:02.213 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8344999999999999e-05 2024-09-18T04:38:02.213 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:02.213 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:02.213 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:02.213 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.213 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.213 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:02.213 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:38:02.214 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 33.873612862999998, 2024-09-18T04:38:02.214 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016528505999999998, 2024-09-18T04:38:02.214 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:02.214 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:02.214 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:02.214 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:02.214 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:02.214 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 138 2024-09-18T04:38:02.214 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.214 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:02.214 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.214 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:02.214 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:38:02.214 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:02.215 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.215 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.215 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:02.215 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:38:02.215 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:02.215 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.215 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.215 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:02.215 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:38:02.215 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:02.215 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.215 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.215 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:02.215 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231047+0000", 2024-09-18T04:38:02.216 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5399999999999997e-06 2024-09-18T04:38:02.216 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.216 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.216 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:02.216 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231068+0000", 2024-09-18T04:38:02.216 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0769000000000001e-05 2024-09-18T04:38:02.216 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.216 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.216 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:02.216 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231074+0000", 2024-09-18T04:38:02.216 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6180000000000003e-06 2024-09-18T04:38:02.216 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.216 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.217 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:02.217 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242556+0000", 2024-09-18T04:38:02.217 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011482654 2024-09-18T04:38:02.217 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.217 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.217 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:02.217 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242666+0000", 2024-09-18T04:38:02.217 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011003200000000001 2024-09-18T04:38:02.217 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.217 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.217 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:02.217 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242924+0000", 2024-09-18T04:38:02.217 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00025785899999999999 2024-09-18T04:38:02.218 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.218 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.218 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:02.218 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247263+0000", 2024-09-18T04:38:02.218 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0043387649999999996 2024-09-18T04:38:02.218 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.218 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.218 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:02.218 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247412+0000", 2024-09-18T04:38:02.218 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000149276 2024-09-18T04:38:02.218 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.218 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.218 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:02.218 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247522+0000", 2024-09-18T04:38:02.219 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000109788 2024-09-18T04:38:02.219 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.219 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.219 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:02.219 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247572+0000", 2024-09-18T04:38:02.219 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9808999999999997e-05 2024-09-18T04:38:02.219 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:02.219 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:02.219 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:02.219 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.219 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.219 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:02.219 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231083+0000", 2024-09-18T04:38:02.219 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 33.873573399999998, 2024-09-18T04:38:02.220 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0166731, 2024-09-18T04:38:02.220 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:02.220 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:02.220 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:02.220 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:02.220 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:02.220 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 139 2024-09-18T04:38:02.220 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.220 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:02.220 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.220 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:02.220 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231083+0000", 2024-09-18T04:38:02.220 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:02.220 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.221 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.221 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:02.221 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231083+0000", 2024-09-18T04:38:02.221 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:02.221 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.221 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.221 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:02.221 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231082+0000", 2024-09-18T04:38:02.221 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:02.221 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.221 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.221 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:02.221 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231090+0000", 2024-09-18T04:38:02.222 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1280000000000008e-06 2024-09-18T04:38:02.222 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.222 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.222 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:02.222 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231097+0000", 2024-09-18T04:38:02.222 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5119999999999997e-06 2024-09-18T04:38:02.222 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.222 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.222 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:02.222 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231106+0000", 2024-09-18T04:38:02.222 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9230000000000008e-06 2024-09-18T04:38:02.222 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.222 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.222 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:02.223 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.243977+0000", 2024-09-18T04:38:02.223 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012870754 2024-09-18T04:38:02.223 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.223 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.223 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:02.223 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244112+0000", 2024-09-18T04:38:02.223 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000135575 2024-09-18T04:38:02.223 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.223 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.223 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:02.223 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244265+0000", 2024-09-18T04:38:02.223 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015279900000000001 2024-09-18T04:38:02.223 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.223 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.223 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:02.224 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247649+0000", 2024-09-18T04:38:02.224 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003383896 2024-09-18T04:38:02.224 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.224 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.224 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:02.224 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247656+0000", 2024-09-18T04:38:02.224 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8920000000000003e-06 2024-09-18T04:38:02.224 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.224 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.224 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:02.224 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247692+0000", 2024-09-18T04:38:02.224 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6470000000000001e-05 2024-09-18T04:38:02.224 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.224 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.225 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:02.225 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247756+0000", 2024-09-18T04:38:02.225 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3676999999999994e-05 2024-09-18T04:38:02.225 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:02.225 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:02.225 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:02.225 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.225 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.225 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:02.225 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231190+0000", 2024-09-18T04:38:02.225 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 33.873466473999997, 2024-09-18T04:38:02.225 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016793556000000001, 2024-09-18T04:38:02.225 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:02.226 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:02.226 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:02.226 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:02.226 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:02.226 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 140 2024-09-18T04:38:02.226 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.226 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:02.226 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.226 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:02.226 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231190+0000", 2024-09-18T04:38:02.226 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:02.226 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.226 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.226 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:02.226 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231190+0000", 2024-09-18T04:38:02.227 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:02.227 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.227 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.227 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:02.227 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231189+0000", 2024-09-18T04:38:02.227 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:02.227 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.227 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.227 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:02.227 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231273+0000", 2024-09-18T04:38:02.227 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4445999999999998e-05 2024-09-18T04:38:02.227 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.227 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.227 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:02.228 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231282+0000", 2024-09-18T04:38:02.228 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8859999999999993e-06 2024-09-18T04:38:02.228 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.228 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.228 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:02.228 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231288+0000", 2024-09-18T04:38:02.228 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9819999999999996e-06 2024-09-18T04:38:02.228 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.228 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.228 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:02.228 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.245119+0000", 2024-09-18T04:38:02.228 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.01383121 2024-09-18T04:38:02.228 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.229 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.229 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:02.229 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.245247+0000", 2024-09-18T04:38:02.229 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012763299999999999 2024-09-18T04:38:02.229 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.229 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.229 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:02.229 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.245401+0000", 2024-09-18T04:38:02.229 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015399200000000001 2024-09-18T04:38:02.229 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.229 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.229 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:02.229 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247766+0000", 2024-09-18T04:38:02.229 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0023649510000000001 2024-09-18T04:38:02.230 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.230 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.230 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:02.230 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247904+0000", 2024-09-18T04:38:02.230 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013836499999999999 2024-09-18T04:38:02.230 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.230 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.230 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:02.230 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247950+0000", 2024-09-18T04:38:02.230 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5201000000000003e-05 2024-09-18T04:38:02.230 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.230 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.230 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:02.231 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247983+0000", 2024-09-18T04:38:02.231 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3686000000000002e-05 2024-09-18T04:38:02.231 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:02.231 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:02.231 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:02.231 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.231 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.231 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:02.231 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:38:02.231 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 33.873355730999997, 2024-09-18T04:38:02.231 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017344024, 2024-09-18T04:38:02.231 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:02.231 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:02.231 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:02.232 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:02.232 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:02.232 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 141 2024-09-18T04:38:02.232 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.232 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:02.232 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.232 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:02.232 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:38:02.232 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:02.232 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.232 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.232 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:02.232 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:38:02.232 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:02.233 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.233 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.233 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:02.233 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:38:02.233 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:02.233 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.233 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.233 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:02.233 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231309+0000", 2024-09-18T04:38:02.233 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5419999999999994e-06 2024-09-18T04:38:02.233 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.234 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.234 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:02.234 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231323+0000", 2024-09-18T04:38:02.234 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3108000000000001e-05 2024-09-18T04:38:02.234 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.234 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.234 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:02.234 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231340+0000", 2024-09-18T04:38:02.234 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.696e-05 2024-09-18T04:38:02.234 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.234 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.234 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:02.234 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.246143+0000", 2024-09-18T04:38:02.234 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014803320999999999 2024-09-18T04:38:02.234 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.235 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.235 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:02.235 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.246323+0000", 2024-09-18T04:38:02.235 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00018061600000000001 2024-09-18T04:38:02.235 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.235 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.235 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:02.235 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.246530+0000", 2024-09-18T04:38:02.235 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020625 2024-09-18T04:38:02.235 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.235 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.235 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:02.235 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248002+0000", 2024-09-18T04:38:02.235 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0014719379999999999 2024-09-18T04:38:02.236 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.236 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.236 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:02.236 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248582+0000", 2024-09-18T04:38:02.236 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00058020599999999999 2024-09-18T04:38:02.236 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.236 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.236 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:02.236 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248618+0000", 2024-09-18T04:38:02.236 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6559e-05 2024-09-18T04:38:02.236 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.236 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.236 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:02.237 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248645+0000", 2024-09-18T04:38:02.237 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6080000000000001e-05 2024-09-18T04:38:02.237 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:02.237 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:02.237 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:02.237 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.237 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.237 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-18T04:38:02.237 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.878463+0000", 2024-09-18T04:38:02.237 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 33.226193696999999, 2024-09-18T04:38:02.237 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010882697, 2024-09-18T04:38:02.237 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:02.238 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:02.238 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:02.238 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:02.238 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:02.238 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 159 2024-09-18T04:38:02.238 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.238 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:02.238 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.238 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:02.238 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878463+0000", 2024-09-18T04:38:02.238 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:02.238 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.238 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.238 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:02.238 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878463+0000", 2024-09-18T04:38:02.239 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:02.239 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.239 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.239 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:02.239 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878462+0000", 2024-09-18T04:38:02.239 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:02.239 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.239 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.239 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:02.239 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878467+0000", 2024-09-18T04:38:02.239 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4099999999999999e-06 2024-09-18T04:38:02.239 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.239 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.240 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:02.240 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878490+0000", 2024-09-18T04:38:02.240 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3159999999999998e-05 2024-09-18T04:38:02.240 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.240 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.240 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:02.240 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878511+0000", 2024-09-18T04:38:02.240 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1092000000000001e-05 2024-09-18T04:38:02.240 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.240 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.240 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:02.240 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.887163+0000", 2024-09-18T04:38:02.240 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.008651845 2024-09-18T04:38:02.241 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.241 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.241 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:02.241 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.887273+0000", 2024-09-18T04:38:02.241 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000109823 2024-09-18T04:38:02.241 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.241 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.241 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:02.241 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.887441+0000", 2024-09-18T04:38:02.241 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016777599999999999 2024-09-18T04:38:02.241 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.241 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.241 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:02.241 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.888634+0000", 2024-09-18T04:38:02.242 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001192839 2024-09-18T04:38:02.242 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.242 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.242 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:02.242 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.889153+0000", 2024-09-18T04:38:02.242 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00051953700000000004 2024-09-18T04:38:02.242 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.242 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.242 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:02.242 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.889234+0000", 2024-09-18T04:38:02.242 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0802000000000003e-05 2024-09-18T04:38:02.242 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:02.242 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:02.242 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:02.243 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.889345+0000", 2024-09-18T04:38:02.243 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011112000000000001 2024-09-18T04:38:02.243 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:02.243 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:02.243 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:02.243 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:02.243 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:02.243 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:02.244 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:38:02.245 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:38:02.245 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:02.239+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:02.245 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:38:02.247 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:38:02.272 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:38:02.297 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:38:02.321 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:38:02.323 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:38:02.323 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:38:02.324 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:38:02.324 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:38:02.324 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:38:02.324 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:38:02.345 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:02.339+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:02.363 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:02.363 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:02.372 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:38:02.446 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:02.442+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:02.475 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:02.475 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:02.484 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:38:02.547 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:02.543+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:02.596 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:02.596 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:02.605 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:38:02.648 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:02.643+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:02.708 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:02.708 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:02.717 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:38:02.718 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-18T04:38:02.718 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T04:38:02.718 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-18T04:38:02.718 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-18T04:38:02.736 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-09-18T04:38:02.748 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:02.743+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:02.826 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:02.826 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:02.836 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:38:02.849 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:02.842+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:02.937 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:02.937 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:02.947 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:38:02.950 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:02.947+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:03.017 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:38:03.017 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:03.017 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 31, 2024-09-18T04:38:03.017 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "be6146ba-a991-4f6e-b359-99cbedc9f9b1", 2024-09-18T04:38:03.017 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-09-18T04:37:15.676634+0000", 2024-09-18T04:38:03.017 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-09-18T04:38:02.718552+0000", 2024-09-18T04:38:03.017 INFO:teuthology.orchestra.run.smithi028.stdout: "last_up_change": "2024-09-18T04:37:25.403175+0000", 2024-09-18T04:38:03.017 INFO:teuthology.orchestra.run.smithi028.stdout: "last_in_change": "2024-09-18T04:37:50.652169+0000", 2024-09-18T04:38:03.017 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-18T04:38:03.017 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_num": 5804032, 2024-09-18T04:38:03.018 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_set": [ 2024-09-18T04:38:03.018 INFO:teuthology.orchestra.run.smithi028.stdout: "nodeep-scrub", 2024-09-18T04:38:03.018 INFO:teuthology.orchestra.run.smithi028.stdout: "pglog_hardlimit", 2024-09-18T04:38:03.018 INFO:teuthology.orchestra.run.smithi028.stdout: "purged_snapdirs", 2024-09-18T04:38:03.018 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_deletes", 2024-09-18T04:38:03.018 INFO:teuthology.orchestra.run.smithi028.stdout: "sortbitwise" 2024-09-18T04:38:03.018 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-18T04:38:03.018 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_version": 5, 2024-09-18T04:38:03.018 INFO:teuthology.orchestra.run.smithi028.stdout: "full_ratio": 0.94999998807907104, 2024-09-18T04:38:03.018 INFO:teuthology.orchestra.run.smithi028.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-18T04:38:03.018 INFO:teuthology.orchestra.run.smithi028.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-18T04:38:03.018 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_snapshot": "", 2024-09-18T04:38:03.018 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_max": 2, 2024-09-18T04:38:03.018 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd": 6, 2024-09-18T04:38:03.018 INFO:teuthology.orchestra.run.smithi028.stdout: "require_min_compat_client": "luminous", 2024-09-18T04:38:03.019 INFO:teuthology.orchestra.run.smithi028.stdout: "min_compat_client": "luminous", 2024-09-18T04:38:03.019 INFO:teuthology.orchestra.run.smithi028.stdout: "require_osd_release": "reef", 2024-09-18T04:38:03.019 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_crimson": false, 2024-09-18T04:38:03.019 INFO:teuthology.orchestra.run.smithi028.stdout: "pools": [ 2024-09-18T04:38:03.019 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.019 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 1, 2024-09-18T04:38:03.019 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": ".mgr", 2024-09-18T04:38:03.019 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-18T04:37:25.654950+0000", 2024-09-18T04:38:03.019 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-18T04:38:03.019 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-09-18T04:38:03.019 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-18T04:38:03.019 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 2, 2024-09-18T04:38:03.019 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 1, 2024-09-18T04:38:03.019 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-18T04:38:03.020 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-18T04:38:03.020 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-18T04:38:03.020 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T04:38:03.020 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T04:38:03.020 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-18T04:38:03.020 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-18T04:38:03.020 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 1, 2024-09-18T04:38:03.020 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 1, 2024-09-18T04:38:03.020 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 1, 2024-09-18T04:38:03.020 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 1, 2024-09-18T04:38:03.020 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 1, 2024-09-18T04:38:03.020 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-18T04:38:03.020 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-18T04:38:03.020 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-18T04:38:03.021 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-18T04:38:03.021 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-18T04:38:03.021 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-18T04:38:03.021 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-18T04:38:03.021 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.021 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "16", 2024-09-18T04:38:03.021 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-18T04:38:03.021 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T04:38:03.021 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T04:38:03.021 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-18T04:38:03.021 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-18T04:38:03.021 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-09-18T04:38:03.021 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-09-18T04:38:03.021 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-18T04:38:03.022 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-18T04:38:03.022 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-18T04:38:03.022 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-18T04:38:03.022 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-18T04:38:03.022 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-18T04:38:03.022 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-18T04:38:03.022 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-18T04:38:03.022 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-18T04:38:03.022 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-18T04:38:03.022 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-18T04:38:03.022 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T04:38:03.022 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T04:38:03.022 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T04:38:03.022 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-18T04:38:03.022 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-18T04:38:03.023 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-18T04:38:03.023 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-18T04:38:03.023 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-18T04:38:03.023 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.023 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-18T04:38:03.023 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-18T04:38:03.023 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-18T04:38:03.023 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-18T04:38:03.023 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-18T04:38:03.023 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T04:38:03.023 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-18T04:38:03.023 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-18T04:38:03.023 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-18T04:38:03.023 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-18T04:38:03.024 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-18T04:38:03.024 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-09-18T04:38:03.024 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_max": 32, 2024-09-18T04:38:03.024 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_min": 1 2024-09-18T04:38:03.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.024 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-18T04:38:03.024 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": {} 2024-09-18T04:38:03.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.024 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-09-18T04:38:03.024 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 6.059999942779541, 2024-09-18T04:38:03.024 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 6.059999942779541, 2024-09-18T04:38:03.024 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.33000001311302185, 2024-09-18T04:38:03.024 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2, 2024-09-18T04:38:03.024 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2, 2024-09-18T04:38:03.025 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 1, 2024-09-18T04:38:03.025 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 1, 2024-09-18T04:38:03.025 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 1 2024-09-18T04:38:03.025 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:03.025 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.025 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.025 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 2, 2024-09-18T04:38:03.025 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "rbd", 2024-09-18T04:38:03.025 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-18T04:37:32.672462+0000", 2024-09-18T04:38:03.025 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 8193, 2024-09-18T04:38:03.025 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-18T04:38:03.025 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-18T04:38:03.026 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 2, 2024-09-18T04:38:03.026 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 1, 2024-09-18T04:38:03.026 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-18T04:38:03.026 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-18T04:38:03.026 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-18T04:38:03.026 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T04:38:03.026 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T04:38:03.026 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-18T04:38:03.026 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-18T04:38:03.026 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 8, 2024-09-18T04:38:03.026 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 8, 2024-09-18T04:38:03.026 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 8, 2024-09-18T04:38:03.026 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 8, 2024-09-18T04:38:03.027 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 8, 2024-09-18T04:38:03.027 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-18T04:38:03.027 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-18T04:38:03.027 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-18T04:38:03.027 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-18T04:38:03.027 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-18T04:38:03.027 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-18T04:38:03.027 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-18T04:38:03.027 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.027 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "20", 2024-09-18T04:38:03.027 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-18T04:38:03.027 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T04:38:03.027 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T04:38:03.027 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-18T04:38:03.028 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-18T04:38:03.028 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 3, 2024-09-18T04:38:03.028 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 20, 2024-09-18T04:38:03.028 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-18T04:38:03.028 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-18T04:38:03.028 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-18T04:38:03.028 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-18T04:38:03.028 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-18T04:38:03.028 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-18T04:38:03.028 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-18T04:38:03.028 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-18T04:38:03.028 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-18T04:38:03.028 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-18T04:38:03.029 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-18T04:38:03.029 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T04:38:03.029 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T04:38:03.029 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T04:38:03.029 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-18T04:38:03.029 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-18T04:38:03.029 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-18T04:38:03.029 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-18T04:38:03.029 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-18T04:38:03.029 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.029 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-18T04:38:03.029 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-18T04:38:03.029 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-18T04:38:03.029 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-18T04:38:03.030 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-18T04:38:03.030 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T04:38:03.030 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-18T04:38:03.030 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-18T04:38:03.030 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-18T04:38:03.030 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-18T04:38:03.030 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-18T04:38:03.030 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-09-18T04:38:03.030 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-18T04:38:03.030 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd": {} 2024-09-18T04:38:03.030 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.030 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-09-18T04:38:03.030 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.5, 2024-09-18T04:38:03.030 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.5, 2024-09-18T04:38:03.031 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-09-18T04:38:03.031 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.5, 2024-09-18T04:38:03.031 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.5, 2024-09-18T04:38:03.031 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 1, 2024-09-18T04:38:03.031 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 1, 2024-09-18T04:38:03.031 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 1 2024-09-18T04:38:03.031 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:03.031 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:03.031 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-18T04:38:03.031 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-09-18T04:38:03.031 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.031 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-09-18T04:38:03.031 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "6177002d-72e6-4888-bcde-29b6a92d9dc4", 2024-09-18T04:38:03.031 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-18T04:38:03.031 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-18T04:38:03.032 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-18T04:38:03.032 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-18T04:38:03.032 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-18T04:38:03.032 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-18T04:38:03.032 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 13, 2024-09-18T04:38:03.032 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 0, 2024-09-18T04:38:03.032 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-18T04:38:03.032 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-18T04:38:03.032 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-18T04:38:03.032 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:03.032 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.032 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:03.032 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6801", 2024-09-18T04:38:03.032 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1554407851 2024-09-18T04:38:03.033 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.033 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.033 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:03.033 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6803", 2024-09-18T04:38:03.033 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1554407851 2024-09-18T04:38:03.033 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:03.033 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:03.033 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.033 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-18T04:38:03.033 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:03.033 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.033 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:03.033 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6805", 2024-09-18T04:38:03.033 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1554407851 2024-09-18T04:38:03.033 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.034 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.034 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:03.034 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6807", 2024-09-18T04:38:03.034 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1554407851 2024-09-18T04:38:03.034 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:03.034 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:03.034 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.034 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-18T04:38:03.034 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:03.034 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.034 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:03.034 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6813", 2024-09-18T04:38:03.034 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1554407851 2024-09-18T04:38:03.035 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.035 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.035 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:03.035 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6815", 2024-09-18T04:38:03.035 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1554407851 2024-09-18T04:38:03.035 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:03.035 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:03.035 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.035 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-18T04:38:03.035 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:03.035 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.035 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:03.035 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6809", 2024-09-18T04:38:03.035 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1554407851 2024-09-18T04:38:03.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.036 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.036 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:03.036 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6811", 2024-09-18T04:38:03.036 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1554407851 2024-09-18T04:38:03.036 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:03.036 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:03.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.036 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6803/1554407851", 2024-09-18T04:38:03.036 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6807/1554407851", 2024-09-18T04:38:03.036 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6815/1554407851", 2024-09-18T04:38:03.036 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6811/1554407851", 2024-09-18T04:38:03.036 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-18T04:38:03.036 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-18T04:38:03.037 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-18T04:38:03.037 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:03.037 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.037 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.037 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-09-18T04:38:03.037 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "4f62cb09-3457-418c-bb69-491d9b70914c", 2024-09-18T04:38:03.037 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-18T04:38:03.037 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-18T04:38:03.037 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-18T04:38:03.037 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-18T04:38:03.037 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-18T04:38:03.037 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-18T04:38:03.037 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 12, 2024-09-18T04:38:03.037 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 21, 2024-09-18T04:38:03.038 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-18T04:38:03.038 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-18T04:38:03.038 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-18T04:38:03.038 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:03.038 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.038 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:03.038 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6800", 2024-09-18T04:38:03.038 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 511141401 2024-09-18T04:38:03.038 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.038 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.038 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:03.038 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6802", 2024-09-18T04:38:03.038 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 511141401 2024-09-18T04:38:03.038 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:03.039 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:03.039 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.039 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-18T04:38:03.039 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:03.039 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.039 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:03.039 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6804", 2024-09-18T04:38:03.039 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 511141401 2024-09-18T04:38:03.039 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.039 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.039 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:03.039 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6806", 2024-09-18T04:38:03.039 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 511141401 2024-09-18T04:38:03.039 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:03.040 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:03.040 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.040 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-18T04:38:03.040 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:03.040 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.040 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:03.040 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6812", 2024-09-18T04:38:03.040 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 511141401 2024-09-18T04:38:03.040 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.040 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.040 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:03.040 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6814", 2024-09-18T04:38:03.040 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 511141401 2024-09-18T04:38:03.041 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:03.041 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:03.041 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.041 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-18T04:38:03.041 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:03.041 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.041 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:03.041 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6808", 2024-09-18T04:38:03.041 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 511141401 2024-09-18T04:38:03.041 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.041 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.041 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:03.041 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6810", 2024-09-18T04:38:03.041 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 511141401 2024-09-18T04:38:03.042 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:03.042 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:03.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.042 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6802/511141401", 2024-09-18T04:38:03.042 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6806/511141401", 2024-09-18T04:38:03.042 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6814/511141401", 2024-09-18T04:38:03.042 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6810/511141401", 2024-09-18T04:38:03.042 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-18T04:38:03.042 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-18T04:38:03.042 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-18T04:38:03.042 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:03.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.042 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.043 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-09-18T04:38:03.043 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "854fe031-3b87-4290-9c2f-3fb196fb806c", 2024-09-18T04:38:03.043 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-18T04:38:03.043 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-18T04:38:03.043 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-18T04:38:03.043 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-18T04:38:03.043 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-18T04:38:03.043 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-18T04:38:03.043 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 12, 2024-09-18T04:38:03.043 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 28, 2024-09-18T04:38:03.043 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-18T04:38:03.043 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-18T04:38:03.043 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-18T04:38:03.043 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:03.044 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.044 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:03.044 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6816", 2024-09-18T04:38:03.044 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2623426735 2024-09-18T04:38:03.044 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.044 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.044 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:03.044 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6817", 2024-09-18T04:38:03.044 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2623426735 2024-09-18T04:38:03.044 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:03.044 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:03.044 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.044 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-18T04:38:03.044 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:03.045 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.045 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:03.045 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6818", 2024-09-18T04:38:03.045 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2623426735 2024-09-18T04:38:03.045 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.045 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.045 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:03.045 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6819", 2024-09-18T04:38:03.045 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2623426735 2024-09-18T04:38:03.045 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:03.045 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:03.045 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.045 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-18T04:38:03.045 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:03.046 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.046 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:03.046 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6822", 2024-09-18T04:38:03.046 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2623426735 2024-09-18T04:38:03.046 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.046 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.046 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:03.046 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6823", 2024-09-18T04:38:03.046 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2623426735 2024-09-18T04:38:03.046 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:03.046 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:03.046 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.046 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-18T04:38:03.046 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:03.047 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.047 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:03.047 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6820", 2024-09-18T04:38:03.047 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2623426735 2024-09-18T04:38:03.047 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.047 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.047 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:03.047 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6821", 2024-09-18T04:38:03.047 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2623426735 2024-09-18T04:38:03.047 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:03.047 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:03.048 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.048 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6817/2623426735", 2024-09-18T04:38:03.048 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6819/2623426735", 2024-09-18T04:38:03.048 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6823/2623426735", 2024-09-18T04:38:03.048 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6821/2623426735", 2024-09-18T04:38:03.048 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-18T04:38:03.048 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-18T04:38:03.048 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-18T04:38:03.048 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:03.048 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.048 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.048 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-09-18T04:38:03.048 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "c2625f98-780b-4f86-bf36-a48d2a227756", 2024-09-18T04:38:03.049 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-18T04:38:03.049 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-18T04:38:03.049 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-18T04:38:03.049 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-18T04:38:03.049 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-18T04:38:03.049 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-18T04:38:03.049 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 12, 2024-09-18T04:38:03.049 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 17, 2024-09-18T04:38:03.049 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-18T04:38:03.049 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-18T04:38:03.049 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-18T04:38:03.049 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:03.049 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.049 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:03.050 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6808", 2024-09-18T04:38:03.050 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2746051132 2024-09-18T04:38:03.050 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.050 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.050 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:03.050 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6809", 2024-09-18T04:38:03.050 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2746051132 2024-09-18T04:38:03.050 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:03.050 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:03.050 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.050 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-18T04:38:03.050 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:03.050 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.050 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:03.051 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6810", 2024-09-18T04:38:03.051 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2746051132 2024-09-18T04:38:03.051 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.051 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.051 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:03.051 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6811", 2024-09-18T04:38:03.051 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2746051132 2024-09-18T04:38:03.051 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:03.051 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:03.051 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.051 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-18T04:38:03.051 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:03.051 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.052 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:03.052 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6814", 2024-09-18T04:38:03.052 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2746051132 2024-09-18T04:38:03.052 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.052 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.052 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:03.052 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6815", 2024-09-18T04:38:03.052 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2746051132 2024-09-18T04:38:03.052 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:03.052 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:03.052 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.052 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-18T04:38:03.052 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:03.053 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.053 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:03.053 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6812", 2024-09-18T04:38:03.053 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2746051132 2024-09-18T04:38:03.053 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.053 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.053 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:03.053 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6813", 2024-09-18T04:38:03.053 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2746051132 2024-09-18T04:38:03.053 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:03.053 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:03.053 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.053 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.82:6809/2746051132", 2024-09-18T04:38:03.053 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.82:6811/2746051132", 2024-09-18T04:38:03.054 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.82:6815/2746051132", 2024-09-18T04:38:03.054 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.82:6813/2746051132", 2024-09-18T04:38:03.054 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-18T04:38:03.054 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-18T04:38:03.054 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-18T04:38:03.054 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:03.054 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.054 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.054 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-09-18T04:38:03.054 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "deb22d86-4081-484d-96eb-9cb9d96b60bc", 2024-09-18T04:38:03.054 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-18T04:38:03.054 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-18T04:38:03.054 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-18T04:38:03.054 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-18T04:38:03.055 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-18T04:38:03.055 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-18T04:38:03.055 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 13, 2024-09-18T04:38:03.055 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 0, 2024-09-18T04:38:03.055 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-18T04:38:03.055 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-18T04:38:03.055 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-18T04:38:03.055 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:03.055 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.055 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:03.055 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6800", 2024-09-18T04:38:03.055 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2005712460 2024-09-18T04:38:03.055 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.055 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.056 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:03.056 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6801", 2024-09-18T04:38:03.056 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2005712460 2024-09-18T04:38:03.056 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:03.056 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:03.056 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.056 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-18T04:38:03.056 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:03.056 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.056 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:03.056 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6802", 2024-09-18T04:38:03.056 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2005712460 2024-09-18T04:38:03.056 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.056 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.057 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:03.057 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6803", 2024-09-18T04:38:03.057 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2005712460 2024-09-18T04:38:03.057 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:03.057 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:03.057 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.057 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-18T04:38:03.057 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:03.057 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.057 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:03.057 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6806", 2024-09-18T04:38:03.057 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2005712460 2024-09-18T04:38:03.057 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.058 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.058 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:03.058 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6807", 2024-09-18T04:38:03.058 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2005712460 2024-09-18T04:38:03.058 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:03.058 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:03.058 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.058 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-18T04:38:03.058 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:03.058 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.058 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:03.058 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6804", 2024-09-18T04:38:03.058 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2005712460 2024-09-18T04:38:03.059 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.059 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.059 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:03.059 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6805", 2024-09-18T04:38:03.059 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2005712460 2024-09-18T04:38:03.059 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:03.059 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:03.059 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.059 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.82:6801/2005712460", 2024-09-18T04:38:03.059 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.82:6803/2005712460", 2024-09-18T04:38:03.059 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.82:6807/2005712460", 2024-09-18T04:38:03.059 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.82:6805/2005712460", 2024-09-18T04:38:03.059 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-18T04:38:03.059 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-18T04:38:03.060 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-18T04:38:03.060 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:03.060 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.060 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.060 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-09-18T04:38:03.060 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "061d332c-f51b-403d-bb18-eba415c97e7f", 2024-09-18T04:38:03.060 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-18T04:38:03.060 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-18T04:38:03.060 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-18T04:38:03.060 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-18T04:38:03.060 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-18T04:38:03.060 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-18T04:38:03.060 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 13, 2024-09-18T04:38:03.060 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 24, 2024-09-18T04:38:03.061 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-18T04:38:03.061 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-18T04:38:03.061 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-18T04:38:03.061 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:03.061 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.061 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:03.061 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6816", 2024-09-18T04:38:03.061 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1267644232 2024-09-18T04:38:03.061 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.061 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.061 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:03.061 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6817", 2024-09-18T04:38:03.061 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1267644232 2024-09-18T04:38:03.062 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:03.062 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:03.062 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.062 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-18T04:38:03.062 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:03.062 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.062 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:03.062 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6818", 2024-09-18T04:38:03.062 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1267644232 2024-09-18T04:38:03.062 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.062 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.062 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:03.062 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6819", 2024-09-18T04:38:03.062 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1267644232 2024-09-18T04:38:03.063 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:03.063 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:03.063 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.063 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-18T04:38:03.063 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:03.063 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.063 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:03.063 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6822", 2024-09-18T04:38:03.063 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1267644232 2024-09-18T04:38:03.063 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.063 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.063 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:03.063 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6823", 2024-09-18T04:38:03.063 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1267644232 2024-09-18T04:38:03.064 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:03.064 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:03.064 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.064 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-18T04:38:03.064 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:03.064 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.064 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:03.064 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6820", 2024-09-18T04:38:03.064 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1267644232 2024-09-18T04:38:03.064 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.064 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.064 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:03.064 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6821", 2024-09-18T04:38:03.064 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1267644232 2024-09-18T04:38:03.065 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:03.065 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:03.065 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.065 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.82:6817/1267644232", 2024-09-18T04:38:03.065 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.82:6819/1267644232", 2024-09-18T04:38:03.065 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.82:6823/1267644232", 2024-09-18T04:38:03.065 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.82:6821/1267644232", 2024-09-18T04:38:03.065 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-18T04:38:03.065 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-18T04:38:03.065 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-18T04:38:03.065 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:03.065 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:03.065 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-18T04:38:03.066 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_xinfo": [ 2024-09-18T04:38:03.066 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.066 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-09-18T04:38:03.066 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-18T04:38:03.066 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-18T04:38:03.066 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-18T04:38:03.066 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138322906710015, 2024-09-18T04:38:03.066 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-18T04:38:03.066 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-18T04:37:22.192087+0000", 2024-09-18T04:38:03.066 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-18T04:38:03.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.066 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.066 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-09-18T04:38:03.066 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-18T04:38:03.066 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-18T04:38:03.067 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-18T04:38:03.067 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138322906710015, 2024-09-18T04:38:03.067 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-18T04:38:03.067 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-18T04:37:22.017833+0000", 2024-09-18T04:38:03.067 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-18T04:38:03.067 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.067 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.067 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-09-18T04:38:03.067 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-18T04:38:03.067 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-18T04:38:03.067 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-18T04:38:03.067 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138322906710015, 2024-09-18T04:38:03.067 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-18T04:38:03.067 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-18T04:37:22.755774+0000", 2024-09-18T04:38:03.068 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-18T04:38:03.068 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.068 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.068 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-09-18T04:38:03.068 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-18T04:38:03.068 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-18T04:38:03.068 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-18T04:38:03.068 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138322906710015, 2024-09-18T04:38:03.068 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-18T04:38:03.068 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-18T04:37:22.663785+0000", 2024-09-18T04:38:03.068 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-18T04:38:03.068 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.068 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.068 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-09-18T04:38:03.068 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-18T04:38:03.069 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-18T04:38:03.069 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-18T04:38:03.069 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138322906710015, 2024-09-18T04:38:03.069 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-18T04:38:03.069 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-18T04:37:22.531758+0000", 2024-09-18T04:38:03.069 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-18T04:38:03.069 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.069 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.069 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-09-18T04:38:03.069 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-18T04:38:03.069 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-18T04:38:03.069 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-18T04:38:03.069 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138322906710015, 2024-09-18T04:38:03.070 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-18T04:38:03.070 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-18T04:37:22.400774+0000", 2024-09-18T04:38:03.070 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-18T04:38:03.070 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:03.070 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-18T04:38:03.070 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap": [ 2024-09-18T04:38:03.070 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:03.070 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "2.5", 2024-09-18T04:38:03.070 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-09-18T04:38:03.070 INFO:teuthology.orchestra.run.smithi028.stdout: 2, 2024-09-18T04:38:03.070 INFO:teuthology.orchestra.run.smithi028.stdout: 0 2024-09-18T04:38:03.070 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:03.070 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:03.071 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-18T04:38:03.071 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap_items": [], 2024-09-18T04:38:03.071 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap_primaries": [], 2024-09-18T04:38:03.071 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_temp": [], 2024-09-18T04:38:03.071 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_temp": [], 2024-09-18T04:38:03.071 INFO:teuthology.orchestra.run.smithi028.stdout: "blocklist": {}, 2024-09-18T04:38:03.071 INFO:teuthology.orchestra.run.smithi028.stdout: "range_blocklist": {}, 2024-09-18T04:38:03.071 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profiles": { 2024-09-18T04:38:03.071 INFO:teuthology.orchestra.run.smithi028.stdout: "default": { 2024-09-18T04:38:03.071 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-failure-domain": "osd", 2024-09-18T04:38:03.071 INFO:teuthology.orchestra.run.smithi028.stdout: "k": "2", 2024-09-18T04:38:03.071 INFO:teuthology.orchestra.run.smithi028.stdout: "m": "1", 2024-09-18T04:38:03.071 INFO:teuthology.orchestra.run.smithi028.stdout: "plugin": "jerasure", 2024-09-18T04:38:03.071 INFO:teuthology.orchestra.run.smithi028.stdout: "technique": "reed_sol_van" 2024-09-18T04:38:03.072 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:03.072 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:03.072 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps_queue": [], 2024-09-18T04:38:03.072 INFO:teuthology.orchestra.run.smithi028.stdout: "new_removed_snaps": [], 2024-09-18T04:38:03.072 INFO:teuthology.orchestra.run.smithi028.stdout: "new_purged_snaps": [], 2024-09-18T04:38:03.072 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_node_flags": {}, 2024-09-18T04:38:03.072 INFO:teuthology.orchestra.run.smithi028.stdout: "device_class_flags": {}, 2024-09-18T04:38:03.072 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": { 2024-09-18T04:38:03.072 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_enabled": false, 2024-09-18T04:38:03.072 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_bucket_count": 0, 2024-09-18T04:38:03.072 INFO:teuthology.orchestra.run.smithi028.stdout: "degraded_stretch_mode": 0, 2024-09-18T04:38:03.072 INFO:teuthology.orchestra.run.smithi028.stdout: "recovering_stretch_mode": 0, 2024-09-18T04:38:03.072 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_bucket": 0 2024-09-18T04:38:03.073 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:03.073 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:03.074 INFO:tasks.thrashosds.thrasher:j is {'epoch': 31, 'fsid': 'be6146ba-a991-4f6e-b359-99cbedc9f9b1', 'created': '2024-09-18T04:37:15.676634+0000', 'modified': '2024-09-18T04:38:02.718552+0000', 'last_up_change': '2024-09-18T04:37:25.403175+0000', 'last_in_change': '2024-09-18T04:37:50.652169+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-18T04:37:25.654950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-18T04:37:32.672462+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 20, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '6177002d-72e6-4888-bcde-29b6a92d9dc4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6801', 'nonce': 1554407851}, {'type': 'v1', 'addr': '172.21.15.28:6803', 'nonce': 1554407851}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6805', 'nonce': 1554407851}, {'type': 'v1', 'addr': '172.21.15.28:6807', 'nonce': 1554407851}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6813', 'nonce': 1554407851}, {'type': 'v1', 'addr': '172.21.15.28:6815', 'nonce': 1554407851}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6809', 'nonce': 1554407851}, {'type': 'v1', 'addr': '172.21.15.28:6811', 'nonce': 1554407851}]}, 'public_addr': '172.21.15.28:6803/1554407851', 'cluster_addr': '172.21.15.28:6807/1554407851', 'heartbeat_back_addr': '172.21.15.28:6815/1554407851', 'heartbeat_front_addr': '172.21.15.28:6811/1554407851', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '4f62cb09-3457-418c-bb69-491d9b70914c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 21, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6800', 'nonce': 511141401}, {'type': 'v1', 'addr': '172.21.15.28:6802', 'nonce': 511141401}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6804', 'nonce': 511141401}, {'type': 'v1', 'addr': '172.21.15.28:6806', 'nonce': 511141401}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6812', 'nonce': 511141401}, {'type': 'v1', 'addr': '172.21.15.28:6814', 'nonce': 511141401}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6808', 'nonce': 511141401}, {'type': 'v1', 'addr': '172.21.15.28:6810', 'nonce': 511141401}]}, 'public_addr': '172.21.15.28:6802/511141401', 'cluster_addr': '172.21.15.28:6806/511141401', 'heartbeat_back_addr': '172.21.15.28:6814/511141401', 'heartbeat_front_addr': '172.21.15.28:6810/511141401', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '854fe031-3b87-4290-9c2f-3fb196fb806c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 28, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6816', 'nonce': 2623426735}, {'type': 'v1', 'addr': '172.21.15.28:6817', 'nonce': 2623426735}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6818', 'nonce': 2623426735}, {'type': 'v1', 'addr': '172.21.15.28:6819', 'nonce': 2623426735}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6822', 'nonce': 2623426735}, {'type': 'v1', 'addr': '172.21.15.28:6823', 'nonce': 2623426735}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6820', 'nonce': 2623426735}, {'type': 'v1', 'addr': '172.21.15.28:6821', 'nonce': 2623426735}]}, 'public_addr': '172.21.15.28:6817/2623426735', 'cluster_addr': '172.21.15.28:6819/2623426735', 'heartbeat_back_addr': '172.21.15.28:6823/2623426735', 'heartbeat_front_addr': '172.21.15.28:6821/2623426735', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'c2625f98-780b-4f86-bf36-a48d2a227756', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 17, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6808', 'nonce': 2746051132}, {'type': 'v1', 'addr': '172.21.15.82:6809', 'nonce': 2746051132}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6810', 'nonce': 2746051132}, {'type': 'v1', 'addr': '172.21.15.82:6811', 'nonce': 2746051132}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6814', 'nonce': 2746051132}, {'type': 'v1', 'addr': '172.21.15.82:6815', 'nonce': 2746051132}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6812', 'nonce': 2746051132}, {'type': 'v1', 'addr': '172.21.15.82:6813', 'nonce': 2746051132}]}, 'public_addr': '172.21.15.82:6809/2746051132', 'cluster_addr': '172.21.15.82:6811/2746051132', 'heartbeat_back_addr': '172.21.15.82:6815/2746051132', 'heartbeat_front_addr': '172.21.15.82:6813/2746051132', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'deb22d86-4081-484d-96eb-9cb9d96b60bc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6800', 'nonce': 2005712460}, {'type': 'v1', 'addr': '172.21.15.82:6801', 'nonce': 2005712460}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6802', 'nonce': 2005712460}, {'type': 'v1', 'addr': '172.21.15.82:6803', 'nonce': 2005712460}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6806', 'nonce': 2005712460}, {'type': 'v1', 'addr': '172.21.15.82:6807', 'nonce': 2005712460}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6804', 'nonce': 2005712460}, {'type': 'v1', 'addr': '172.21.15.82:6805', 'nonce': 2005712460}]}, 'public_addr': '172.21.15.82:6801/2005712460', 'cluster_addr': '172.21.15.82:6803/2005712460', 'heartbeat_back_addr': '172.21.15.82:6807/2005712460', 'heartbeat_front_addr': '172.21.15.82:6805/2005712460', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '061d332c-f51b-403d-bb18-eba415c97e7f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 24, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6816', 'nonce': 1267644232}, {'type': 'v1', 'addr': '172.21.15.82:6817', 'nonce': 1267644232}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6818', 'nonce': 1267644232}, {'type': 'v1', 'addr': '172.21.15.82:6819', 'nonce': 1267644232}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6822', 'nonce': 1267644232}, {'type': 'v1', 'addr': '172.21.15.82:6823', 'nonce': 1267644232}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6820', 'nonce': 1267644232}, {'type': 'v1', 'addr': '172.21.15.82:6821', 'nonce': 1267644232}]}, 'public_addr': '172.21.15.82:6817/1267644232', 'cluster_addr': '172.21.15.82:6819/1267644232', 'heartbeat_back_addr': '172.21.15.82:6823/1267644232', 'heartbeat_front_addr': '172.21.15.82:6821/1267644232', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T04:37:22.192087+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T04:37:22.017833+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T04:37:22.755774+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T04:37:22.663785+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T04:37:22.531758+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T04:37:22.400774+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.5', 'osds': [2, 0]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-18T04:38:03.075 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T04:38:03.076 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:03.076 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:03.076 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:38:03.077 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:03.071+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:03.175 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:03.170+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:03.184 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:03.184 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:03.195 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:38:03.276 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:03.271+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:03.297 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:03.297 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:03.306 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:38:03.377 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:03.370+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:03.380 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:38:03.383 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-18T04:38:03.395 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":44,"stamp":"2024-09-18T04:38:02.725920+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":184524,"kb_used_data":1900,"kb_used_omap":9,"kb_used_meta":182518,"kb_avail":566046516,"statfs":{"total":579820584960,"available":579631632384,"internally_reserved":0,"allocated":1945600,"data_stored":1806025,"data_compressed":66040,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":9561,"internal_metadata":186899111},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.127215"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-18T04:37:58.703221+0000","last_change":"2024-09-18T04:37:36.484517+0000","last_active":"2024-09-18T04:37:58.703221+0000","last_peered":"2024-09-18T04:37:58.703221+0000","last_clean":"2024-09-18T04:37:58.703221+0000","last_became_active":"2024-09-18T04:37:34.893674+0000","last_became_peered":"2024-09-18T04:37:34.893674+0000","last_unstale":"2024-09-18T04:37:58.703221+0000","last_undegraded":"2024-09-18T04:37:58.703221+0000","last_fullsized":"2024-09-18T04:37:58.703221+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T09:53:33.883648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047343899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":24,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-18T04:37:57.321532+0000","last_change":"2024-09-18T04:37:36.494193+0000","last_active":"2024-09-18T04:37:57.321532+0000","last_peered":"2024-09-18T04:37:57.321532+0000","last_clean":"2024-09-18T04:37:57.321532+0000","last_became_active":"2024-09-18T04:37:34.472940+0000","last_became_peered":"2024-09-18T04:37:34.472940+0000","last_unstale":"2024-09-18T04:37:57.321532+0000","last_undegraded":"2024-09-18T04:37:57.321532+0000","last_fullsized":"2024-09-18T04:37:57.321532+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:18:29.266815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070553900000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T04:37:59.718273+0000","last_change":"2024-09-18T04:37:59.284861+0000","last_active":"2024-09-18T04:37:59.718273+0000","last_peered":"2024-09-18T04:37:59.718273+0000","last_clean":"2024-09-18T04:37:59.718273+0000","last_became_active":"2024-09-18T04:37:59.283691+0000","last_became_peered":"2024-09-18T04:37:59.283691+0000","last_unstale":"2024-09-18T04:37:59.718273+0000","last_undegraded":"2024-09-18T04:37:59.718273+0000","last_fullsized":"2024-09-18T04:37:59.718273+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:19:36.536686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-18T04:37:57.321480+0000","last_change":"2024-09-18T04:37:36.494228+0000","last_active":"2024-09-18T04:37:57.321480+0000","last_peered":"2024-09-18T04:37:57.321480+0000","last_clean":"2024-09-18T04:37:57.321480+0000","last_became_active":"2024-09-18T04:37:34.471265+0000","last_became_peered":"2024-09-18T04:37:34.471265+0000","last_unstale":"2024-09-18T04:37:57.321480+0000","last_undegraded":"2024-09-18T04:37:57.321480+0000","last_fullsized":"2024-09-18T04:37:57.321480+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:23:05.748076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081514499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"20'2","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-18T04:37:59.296042+0000","last_change":"2024-09-18T04:37:36.490461+0000","last_active":"2024-09-18T04:37:59.296042+0000","last_peered":"2024-09-18T04:37:59.296042+0000","last_clean":"2024-09-18T04:37:59.296042+0000","last_became_active":"2024-09-18T04:37:34.473088+0000","last_became_peered":"2024-09-18T04:37:34.473088+0000","last_unstale":"2024-09-18T04:37:59.296042+0000","last_undegraded":"2024-09-18T04:37:59.296042+0000","last_fullsized":"2024-09-18T04:37:59.296042+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:17:57.352397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001879932,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-18T04:37:58.703627+0000","last_change":"2024-09-18T04:37:51.679120+0000","last_active":"2024-09-18T04:37:58.703627+0000","last_peered":"2024-09-18T04:37:58.703627+0000","last_clean":"2024-09-18T04:37:58.703627+0000","last_became_active":"2024-09-18T04:37:51.678711+0000","last_became_peered":"2024-09-18T04:37:51.678711+0000","last_unstale":"2024-09-18T04:37:58.703627+0000","last_undegraded":"2024-09-18T04:37:58.703627+0000","last_fullsized":"2024-09-18T04:37:58.703627+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T04:46:09.459905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-18T04:37:58.703090+0000","last_change":"2024-09-18T04:37:36.484950+0000","last_active":"2024-09-18T04:37:58.703090+0000","last_peered":"2024-09-18T04:37:58.703090+0000","last_clean":"2024-09-18T04:37:58.703090+0000","last_became_active":"2024-09-18T04:37:34.470841+0000","last_became_peered":"2024-09-18T04:37:34.470841+0000","last_unstale":"2024-09-18T04:37:58.703090+0000","last_undegraded":"2024-09-18T04:37:58.703090+0000","last_fullsized":"2024-09-18T04:37:58.703090+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:16:34.528783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090513900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"18'1","reported_seq":49,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-18T04:37:59.295967+0000","last_change":"2024-09-18T04:37:51.681357+0000","last_active":"2024-09-18T04:37:59.295967+0000","last_peered":"2024-09-18T04:37:59.295967+0000","last_clean":"2024-09-18T04:37:59.295967+0000","last_became_active":"2024-09-18T04:37:51.676131+0000","last_became_peered":"2024-09-18T04:37:51.676131+0000","last_unstale":"2024-09-18T04:37:59.295967+0000","last_undegraded":"2024-09-18T04:37:59.295967+0000","last_fullsized":"2024-09-18T04:37:59.295967+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:45:22.605009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018293420000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":173,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-18T04:37:58.703181+0000","last_change":"2024-09-18T04:37:27.423069+0000","last_active":"2024-09-18T04:37:58.703181+0000","last_peered":"2024-09-18T04:37:58.703181+0000","last_clean":"2024-09-18T04:37:58.703181+0000","last_became_active":"2024-09-18T04:37:27.422350+0000","last_became_peered":"2024-09-18T04:37:27.422350+0000","last_unstale":"2024-09-18T04:37:58.703181+0000","last_undegraded":"2024-09-18T04:37:58.703181+0000","last_fullsized":"2024-09-18T04:37:58.703181+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:26.406292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:26.406292+0000","last_clean_scrub_stamp":"2024-09-18T04:37:26.406292+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:31:45.852149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":66040,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":13,"seq":55834574859,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27320,"kb_used_data":296,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344520,"statfs":{"total":96636764160,"available":96608788480,"internally_reserved":0,"allocated":303104,"data_stored":103465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.071}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.262}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.999}]}]},{"osd":4,"up_from":13,"seq":55834574859,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27324,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344516,"statfs":{"total":96636764160,"available":96608784384,"internally_reserved":0,"allocated":311296,"data_stored":108021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93000000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96599999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.129}]}]},{"osd":0,"up_from":13,"seq":55834574860,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35576,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94336264,"statfs":{"total":96636764160,"available":96600334336,"internally_reserved":0,"allocated":364544,"data_stored":693814,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94899999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.169}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0009999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0720000000000001}]}]},{"osd":3,"up_from":12,"seq":51539607563,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31476,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340364,"statfs":{"total":96636764160,"available":96604532736,"internally_reserved":0,"allocated":364544,"data_stored":693814,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94799999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73399999999999999}]}]},{"osd":2,"up_from":12,"seq":51539607561,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":292,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":299008,"data_stored":103446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94499999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.012}]}]},{"osd":1,"up_from":12,"seq":51539607563,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31416,"kb_used_data":296,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340424,"statfs":{"total":96636764160,"available":96604594176,"internally_reserved":0,"allocated":303104,"data_stored":103465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71599999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84999999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T04:38:03.396 INFO:tasks.thrashosds.thrasher:Setting 2.7 to [3, 5] 2024-09-18T04:38:03.396 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '2.7', '3', '5'] 2024-09-18T04:38:03.396 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 2.7 3 5 2024-09-18T04:38:03.414 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:03.414 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:03.424 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:38:03.477 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:03.475+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:03.539 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:03.540 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:03.549 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:38:03.578 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:03.575+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:03.655 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:03.655 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:03.665 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:38:03.710 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:03.707+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:03.780 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:03.774+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:03.818 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:03.818 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:03.828 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:38:03.881 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:03.874+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:03.931 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:03.931 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:03.941 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:38:03.986 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:03.983+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:04.044 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:04.044 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:04.054 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:38:04.083 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:04.079+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:04.168 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:04.168 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:04.178 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:38:04.184 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:04.178+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:04.285 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:04.285 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:04.286 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:04.279+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:04.294 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:38:04.336 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T04:38:04.386 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:04.383+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:04.402 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:04.402 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:04.412 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:38:04.486 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:04.482+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:04.492 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:38:04.511 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:38:04.515 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:04.515 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:04.525 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:38:04.533 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:38:04.556 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:38:04.577 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:38:04.587 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:04.582+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:04.599 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:38:04.601 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:38:04.601 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:38:04.601 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:38:04.601 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:38:04.602 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:38:04.602 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:38:04.631 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:04.631 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:04.632 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:38:04.632 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:38:04.632 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:04.641 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:38:04.688 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:04.683+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:04.746 INFO:teuthology.orchestra.run.smithi028.stderr:set 2.7 pg_upmap mapping to [3,5] 2024-09-18T04:38:04.749 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-18T04:38:04.791 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:04.791 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:38:04.791 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:38:04.791 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [] 2024-09-18T04:38:04.791 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:04.801 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:38:04.810 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:04.807+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:04.890 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:04.886+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:04.906 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:04.906 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:04.906 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:38:04.906 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:04.916 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:38:04.991 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:04.987+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:05.065 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:05.066 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:05.066 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:38:05.066 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:38:05.066 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:05.075 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:38:05.092 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:05.086+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:05.180 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:05.180 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:38:05.180 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:38:05.180 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:38:05.180 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.180 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-18T04:38:05.180 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:38:05.180 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 30.706431579, 2024-09-18T04:38:05.180 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0095781189999999995, 2024-09-18T04:38:05.180 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:05.181 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:05.181 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:05.181 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:38:05.181 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:38:05.181 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 1 2024-09-18T04:38:05.181 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.181 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:05.181 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.181 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:05.181 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:38:05.181 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:05.181 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.181 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.181 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:05.182 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:38:05.182 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:05.182 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.182 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.182 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:05.182 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469862+0000", 2024-09-18T04:38:05.182 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:05.182 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.182 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.182 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:05.182 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469879+0000", 2024-09-18T04:38:05.182 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7431999999999998e-05 2024-09-18T04:38:05.182 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.182 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.183 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:05.183 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469896+0000", 2024-09-18T04:38:05.183 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6742999999999999e-05 2024-09-18T04:38:05.183 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.183 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.183 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:05.183 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469915+0000", 2024-09-18T04:38:05.183 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8991000000000002e-05 2024-09-18T04:38:05.183 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.183 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.183 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:05.183 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470397+0000", 2024-09-18T04:38:05.183 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048167799999999998 2024-09-18T04:38:05.183 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.184 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.184 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-09-18T04:38:05.184 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470412+0000", 2024-09-18T04:38:05.184 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5526e-05 2024-09-18T04:38:05.184 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.184 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.184 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:05.184 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473279+0000", 2024-09-18T04:38:05.184 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0028667469999999998 2024-09-18T04:38:05.184 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.184 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.184 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:05.184 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473441+0000", 2024-09-18T04:38:05.185 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000162082 2024-09-18T04:38:05.185 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.185 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.185 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:38:05.185 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473573+0000", 2024-09-18T04:38:05.185 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000132032 2024-09-18T04:38:05.185 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.185 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.185 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:05.185 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474826+0000", 2024-09-18T04:38:05.185 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001252717 2024-09-18T04:38:05.185 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.185 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.186 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:05.186 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479307+0000", 2024-09-18T04:38:05.186 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044813220000000003 2024-09-18T04:38:05.186 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.186 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.186 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:05.186 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479366+0000", 2024-09-18T04:38:05.186 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9024999999999998e-05 2024-09-18T04:38:05.186 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.186 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.186 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:05.186 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479443+0000", 2024-09-18T04:38:05.186 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6339000000000002e-05 2024-09-18T04:38:05.187 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:05.187 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:05.187 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:05.187 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.187 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.187 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.4214.0:1 2.3 e18/17 ondisk, result = 0)", 2024-09-18T04:38:05.187 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.474724+0000", 2024-09-18T04:38:05.187 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 30.701572214999999, 2024-09-18T04:38:05.187 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000105358, 2024-09-18T04:38:05.187 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:05.187 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:38:05.187 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:05.187 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.187 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:05.188 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474724+0000", 2024-09-18T04:38:05.188 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:05.188 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.188 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.188 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:05.188 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474724+0000", 2024-09-18T04:38:05.188 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:05.188 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.188 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.188 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:05.188 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474723+0000", 2024-09-18T04:38:05.188 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:05.188 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.189 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.189 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:05.189 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474731+0000", 2024-09-18T04:38:05.189 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2339999999999994e-06 2024-09-18T04:38:05.189 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.189 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.189 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:05.189 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474748+0000", 2024-09-18T04:38:05.189 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6603e-05 2024-09-18T04:38:05.189 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.189 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.189 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:05.189 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474760+0000", 2024-09-18T04:38:05.189 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2215000000000001e-05 2024-09-18T04:38:05.190 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.190 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.190 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:05.190 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474814+0000", 2024-09-18T04:38:05.190 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4267000000000001e-05 2024-09-18T04:38:05.190 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.190 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.190 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:05.190 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474821+0000", 2024-09-18T04:38:05.190 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0929999999999997e-06 2024-09-18T04:38:05.190 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.190 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.190 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:05.190 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474829+0000", 2024-09-18T04:38:05.191 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1200000000000002e-06 2024-09-18T04:38:05.191 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:05.191 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:05.191 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:05.191 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.191 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.191 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18)", 2024-09-18T04:38:05.191 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.482346+0000", 2024-09-18T04:38:05.191 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 30.693949783000001, 2024-09-18T04:38:05.191 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020435599999999999, 2024-09-18T04:38:05.191 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:05.191 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-09-18T04:38:05.191 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:05.191 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:38:05.192 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:38:05.192 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 2 2024-09-18T04:38:05.192 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.192 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:05.192 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.192 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:05.192 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482346+0000", 2024-09-18T04:38:05.192 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:05.192 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.192 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.192 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:05.192 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482346+0000", 2024-09-18T04:38:05.192 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:05.192 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.192 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.193 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:05.193 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482345+0000", 2024-09-18T04:38:05.193 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:05.193 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.193 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.193 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:05.193 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482360+0000", 2024-09-18T04:38:05.193 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4854999999999999e-05 2024-09-18T04:38:05.193 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.193 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.193 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:05.193 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482377+0000", 2024-09-18T04:38:05.193 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7785999999999999e-05 2024-09-18T04:38:05.193 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.194 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.194 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:05.194 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482387+0000", 2024-09-18T04:38:05.194 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6800000000000005e-06 2024-09-18T04:38:05.194 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.194 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.194 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:05.194 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482441+0000", 2024-09-18T04:38:05.194 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4046999999999999e-05 2024-09-18T04:38:05.194 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.194 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.194 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:05.194 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482551+0000", 2024-09-18T04:38:05.195 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000109468 2024-09-18T04:38:05.195 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:05.195 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:05.195 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:05.195 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.195 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.195 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18)", 2024-09-18T04:38:05.195 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.482853+0000", 2024-09-18T04:38:05.195 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 30.693443165000001, 2024-09-18T04:38:05.195 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00029815799999999997, 2024-09-18T04:38:05.195 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:05.195 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-09-18T04:38:05.195 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:05.195 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:38:05.196 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:38:05.196 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 3 2024-09-18T04:38:05.196 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.196 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:05.196 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.196 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:05.196 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482853+0000", 2024-09-18T04:38:05.196 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:05.196 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.196 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.196 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:05.196 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482853+0000", 2024-09-18T04:38:05.196 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:05.196 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.196 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.197 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:05.197 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482852+0000", 2024-09-18T04:38:05.197 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:05.197 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.197 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.197 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:05.197 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482859+0000", 2024-09-18T04:38:05.197 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0380000000000002e-06 2024-09-18T04:38:05.197 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.197 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.197 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:05.197 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482880+0000", 2024-09-18T04:38:05.197 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1243e-05 2024-09-18T04:38:05.197 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.198 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.198 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:05.198 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482896+0000", 2024-09-18T04:38:05.198 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5977e-05 2024-09-18T04:38:05.198 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.198 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.198 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:05.198 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482970+0000", 2024-09-18T04:38:05.198 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4603000000000004e-05 2024-09-18T04:38:05.198 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.198 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.198 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:05.198 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.483151+0000", 2024-09-18T04:38:05.198 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00018059799999999999 2024-09-18T04:38:05.199 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:05.199 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:05.199 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:05.199 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.199 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.199 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-18T04:38:05.199 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:35.468698+0000", 2024-09-18T04:38:05.199 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 29.707598328, 2024-09-18T04:38:05.199 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0039954129999999997, 2024-09-18T04:38:05.199 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:05.199 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:05.199 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:05.199 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:38:05.199 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:38:05.199 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 5 2024-09-18T04:38:05.200 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.200 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:05.200 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.200 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:05.200 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468698+0000", 2024-09-18T04:38:05.200 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:05.200 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.200 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.200 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:05.200 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468698+0000", 2024-09-18T04:38:05.200 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:05.200 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.200 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.200 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:05.201 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468696+0000", 2024-09-18T04:38:05.201 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:05.201 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.201 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.201 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:05.201 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468714+0000", 2024-09-18T04:38:05.201 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8743e-05 2024-09-18T04:38:05.201 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.201 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.201 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:05.201 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468748+0000", 2024-09-18T04:38:05.201 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3339e-05 2024-09-18T04:38:05.201 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.201 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.201 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:05.201 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468771+0000", 2024-09-18T04:38:05.202 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3119999999999999e-05 2024-09-18T04:38:05.202 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.202 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.202 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:05.202 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468922+0000", 2024-09-18T04:38:05.202 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000151383 2024-09-18T04:38:05.202 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.202 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.202 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:05.202 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.469278+0000", 2024-09-18T04:38:05.202 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00035628599999999998 2024-09-18T04:38:05.202 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.202 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.202 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:05.202 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.469629+0000", 2024-09-18T04:38:05.202 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00035014300000000003 2024-09-18T04:38:05.203 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.203 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.203 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:05.203 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.471620+0000", 2024-09-18T04:38:05.203 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001991278 2024-09-18T04:38:05.203 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.203 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.203 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:05.203 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472440+0000", 2024-09-18T04:38:05.203 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00081998599999999998 2024-09-18T04:38:05.203 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.203 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.203 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:05.203 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472592+0000", 2024-09-18T04:38:05.204 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015227400000000001 2024-09-18T04:38:05.204 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.204 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.204 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:05.204 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472693+0000", 2024-09-18T04:38:05.204 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000101074 2024-09-18T04:38:05.204 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:05.204 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:05.204 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:05.204 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.204 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.204 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.4214.0:5 2.2 e18/17 ondisk, result = 0)", 2024-09-18T04:38:05.204 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:35.472226+0000", 2024-09-18T04:38:05.204 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 29.704070428000001, 2024-09-18T04:38:05.205 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048505299999999999, 2024-09-18T04:38:05.205 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:05.205 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:38:05.205 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:05.205 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.205 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:05.205 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472226+0000", 2024-09-18T04:38:05.205 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:05.205 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.205 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.205 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:05.205 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472226+0000", 2024-09-18T04:38:05.205 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:05.205 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.205 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.206 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:05.206 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472224+0000", 2024-09-18T04:38:05.206 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:05.206 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.206 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.206 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:05.206 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472238+0000", 2024-09-18T04:38:05.206 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.468e-05 2024-09-18T04:38:05.206 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.206 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.206 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:05.206 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472272+0000", 2024-09-18T04:38:05.206 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3586e-05 2024-09-18T04:38:05.206 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.207 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.207 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:05.207 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472297+0000", 2024-09-18T04:38:05.207 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5282999999999998e-05 2024-09-18T04:38:05.207 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.207 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.207 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:05.207 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472396+0000", 2024-09-18T04:38:05.207 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.8394999999999996e-05 2024-09-18T04:38:05.207 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.207 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.207 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:05.207 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472421+0000", 2024-09-18T04:38:05.208 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5479000000000001e-05 2024-09-18T04:38:05.208 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.208 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.208 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:05.208 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472711+0000", 2024-09-18T04:38:05.208 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000289714 2024-09-18T04:38:05.208 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:05.208 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:05.208 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:05.208 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.208 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.208 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-18T04:38:05.208 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:38:05.208 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 28.693423158999998, 2024-09-18T04:38:05.208 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082699560000000002, 2024-09-18T04:38:05.208 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:05.209 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:05.209 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:05.209 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:38:05.209 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:38:05.209 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 7 2024-09-18T04:38:05.209 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.209 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:05.209 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.209 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:05.209 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:38:05.209 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:05.209 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.209 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.209 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:05.210 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:38:05.210 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:05.210 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.210 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.210 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:05.210 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482870+0000", 2024-09-18T04:38:05.210 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999967 2024-09-18T04:38:05.210 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.210 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.210 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:05.210 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482889+0000", 2024-09-18T04:38:05.210 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.965e-05 2024-09-18T04:38:05.210 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.210 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.211 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:05.211 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482938+0000", 2024-09-18T04:38:05.211 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8548999999999999e-05 2024-09-18T04:38:05.211 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.211 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.211 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:05.211 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482963+0000", 2024-09-18T04:38:05.211 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5075999999999998e-05 2024-09-18T04:38:05.211 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.211 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.211 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:05.211 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483121+0000", 2024-09-18T04:38:05.211 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015819400000000001 2024-09-18T04:38:05.211 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.212 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.212 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op must wait for map", 2024-09-18T04:38:05.212 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483138+0000", 2024-09-18T04:38:05.212 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7068999999999999e-05 2024-09-18T04:38:05.212 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.212 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.212 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:05.212 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488666+0000", 2024-09-18T04:38:05.212 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0055282049999999996 2024-09-18T04:38:05.212 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.212 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.212 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:05.212 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488783+0000", 2024-09-18T04:38:05.212 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000116737 2024-09-18T04:38:05.213 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.213 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.213 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:05.213 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488991+0000", 2024-09-18T04:38:05.213 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020781400000000001 2024-09-18T04:38:05.213 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.213 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.213 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:05.213 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490554+0000", 2024-09-18T04:38:05.213 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0015628370000000001 2024-09-18T04:38:05.213 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.213 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.213 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:05.213 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491019+0000", 2024-09-18T04:38:05.214 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000465608 2024-09-18T04:38:05.214 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.214 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.214 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:05.214 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491087+0000", 2024-09-18T04:38:05.214 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7988e-05 2024-09-18T04:38:05.214 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.214 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.214 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:05.214 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491143+0000", 2024-09-18T04:38:05.214 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5538999999999998e-05 2024-09-18T04:38:05.214 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:05.214 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:05.214 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:05.214 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.214 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.215 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.4214.0:7 2.2 e19/17 ondisk, result = 0)", 2024-09-18T04:38:05.215 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:36.490861+0000", 2024-09-18T04:38:05.215 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 28.685435537, 2024-09-18T04:38:05.215 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00029639400000000001, 2024-09-18T04:38:05.215 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:05.215 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:38:05.215 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:05.215 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.215 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:05.215 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490861+0000", 2024-09-18T04:38:05.215 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:05.215 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.215 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.215 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:05.216 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490861+0000", 2024-09-18T04:38:05.216 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:05.216 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.216 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.216 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:05.216 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490859+0000", 2024-09-18T04:38:05.216 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:05.216 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.216 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.216 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:05.216 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490867+0000", 2024-09-18T04:38:05.216 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.3089999999999996e-06 2024-09-18T04:38:05.216 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.216 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.216 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:05.217 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490909+0000", 2024-09-18T04:38:05.217 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1919000000000003e-05 2024-09-18T04:38:05.217 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.217 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.217 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:05.217 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490922+0000", 2024-09-18T04:38:05.217 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2836999999999999e-05 2024-09-18T04:38:05.217 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.217 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.217 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:05.217 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491003+0000", 2024-09-18T04:38:05.217 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0681999999999998e-05 2024-09-18T04:38:05.217 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.217 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.218 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:05.218 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491013+0000", 2024-09-18T04:38:05.218 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0220999999999999e-05 2024-09-18T04:38:05.218 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.218 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.218 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:05.218 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491157+0000", 2024-09-18T04:38:05.218 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014401799999999999 2024-09-18T04:38:05.218 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:05.218 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:05.218 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:05.218 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.218 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.218 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "MOSDPGPushReply(2.3 25/24 [PushReplyOp(2:c4c92e5a:::rbd_trash:head)])", 2024-09-18T04:38:05.219 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:51.680323+0000", 2024-09-18T04:38:05.219 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 13.495973162, 2024-09-18T04:38:05.219 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00054411499999999996, 2024-09-18T04:38:05.219 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:05.219 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-09-18T04:38:05.219 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:05.219 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.219 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:05.219 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680323+0000", 2024-09-18T04:38:05.219 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:05.219 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.219 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.219 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:05.219 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680323+0000", 2024-09-18T04:38:05.220 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:05.220 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.220 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.220 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:05.220 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680322+0000", 2024-09-18T04:38:05.220 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:05.220 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.220 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.220 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:05.220 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680333+0000", 2024-09-18T04:38:05.220 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1153e-05 2024-09-18T04:38:05.220 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.220 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.220 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:05.220 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680357+0000", 2024-09-18T04:38:05.221 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3252999999999999e-05 2024-09-18T04:38:05.221 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.221 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.221 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:05.221 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680378+0000", 2024-09-18T04:38:05.221 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1503000000000001e-05 2024-09-18T04:38:05.221 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.221 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.221 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:05.221 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680522+0000", 2024-09-18T04:38:05.221 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014410700000000001 2024-09-18T04:38:05.221 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:05.221 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:05.221 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:05.222 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680867+0000", 2024-09-18T04:38:05.222 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00034475899999999998 2024-09-18T04:38:05.222 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:05.222 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:05.222 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:05.222 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:05.222 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:05.222 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:05.223 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:05.218+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:05.224 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:38:05.325 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:05.318+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:05.339 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:05.339 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:05.340 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:38:05.340 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:05.350 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:38:05.425 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:05.419+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:05.458 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:05.458 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:05.459 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:38:05.459 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:38:05.459 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:05.468 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:38:05.525 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:05.518+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:05.578 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:05.578 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:38:05.578 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:38:05.578 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:38:05.578 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.578 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:30 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'16, mlcod=15'14)", 2024-09-18T04:38:05.578 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.486330+0000", 2024-09-18T04:38:05.578 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 38.088011389999998, 2024-09-18T04:38:05.578 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026383700000000001, 2024-09-18T04:38:05.578 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:05.579 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:05.579 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:05.579 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.579 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:05.579 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486330+0000", 2024-09-18T04:38:05.579 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:05.579 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.579 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.579 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:05.579 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486330+0000", 2024-09-18T04:38:05.579 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:05.579 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.579 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.579 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:05.580 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486328+0000", 2024-09-18T04:38:05.580 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:05.580 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.580 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.580 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:05.580 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486342+0000", 2024-09-18T04:38:05.580 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3783e-05 2024-09-18T04:38:05.580 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.580 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.580 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:05.580 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486373+0000", 2024-09-18T04:38:05.580 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1362999999999998e-05 2024-09-18T04:38:05.581 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.581 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.581 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:05.581 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486387+0000", 2024-09-18T04:38:05.581 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3219e-05 2024-09-18T04:38:05.581 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.581 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.581 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:05.581 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486489+0000", 2024-09-18T04:38:05.581 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010232499999999999 2024-09-18T04:38:05.581 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.581 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.582 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:05.582 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486524+0000", 2024-09-18T04:38:05.582 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4972000000000003e-05 2024-09-18T04:38:05.582 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.582 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.582 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:05.582 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.488902+0000", 2024-09-18T04:38:05.582 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023776240000000001 2024-09-18T04:38:05.582 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.582 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.582 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:05.582 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.488968+0000", 2024-09-18T04:38:05.582 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6761000000000007e-05 2024-09-18T04:38:05.582 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:05.583 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:05.583 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:05.583 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.583 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.583 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:35 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'21, mlcod=15'17)", 2024-09-18T04:38:05.583 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.491817+0000", 2024-09-18T04:38:05.583 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 38.082524472000003, 2024-09-18T04:38:05.583 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023524739999999998, 2024-09-18T04:38:05.583 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:05.583 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:05.583 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:05.583 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.583 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:05.584 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491817+0000", 2024-09-18T04:38:05.584 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:05.584 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.584 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.584 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:05.584 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491817+0000", 2024-09-18T04:38:05.584 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:05.584 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.584 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.584 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:05.584 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491815+0000", 2024-09-18T04:38:05.584 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:05.584 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.584 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.585 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:05.585 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491890+0000", 2024-09-18T04:38:05.585 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4500999999999997e-05 2024-09-18T04:38:05.585 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.585 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.585 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:05.585 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491916+0000", 2024-09-18T04:38:05.585 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5666000000000001e-05 2024-09-18T04:38:05.585 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.585 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.585 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:05.585 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491928+0000", 2024-09-18T04:38:05.585 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2585999999999999e-05 2024-09-18T04:38:05.586 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.586 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.586 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:05.586 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.492000+0000", 2024-09-18T04:38:05.586 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2309000000000002e-05 2024-09-18T04:38:05.586 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.586 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.586 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:05.586 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.492027+0000", 2024-09-18T04:38:05.586 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6251e-05 2024-09-18T04:38:05.586 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.586 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.586 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:05.586 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.494114+0000", 2024-09-18T04:38:05.587 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020875260000000001 2024-09-18T04:38:05.587 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.587 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.587 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:05.587 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.494169+0000", 2024-09-18T04:38:05.587 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5145000000000001e-05 2024-09-18T04:38:05.587 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:05.587 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:05.587 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:05.587 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.587 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.587 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:57 1.0 e15/14 1:30306672:devicehealth::main.db.0000000000000000:head v 15'35, mlcod=15'33)", 2024-09-18T04:38:05.587 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.529997+0000", 2024-09-18T04:38:05.587 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 38.044344617999997, 2024-09-18T04:38:05.588 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002623869, 2024-09-18T04:38:05.588 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:05.588 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:05.588 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:05.588 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.588 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:05.588 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.529997+0000", 2024-09-18T04:38:05.588 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:05.588 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.588 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.588 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:05.588 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.529997+0000", 2024-09-18T04:38:05.588 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:05.588 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.589 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.589 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:05.589 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.529995+0000", 2024-09-18T04:38:05.589 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:05.589 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.589 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.589 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:05.589 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530116+0000", 2024-09-18T04:38:05.589 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000121141 2024-09-18T04:38:05.589 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.590 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.590 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:05.590 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530146+0000", 2024-09-18T04:38:05.590 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0423e-05 2024-09-18T04:38:05.590 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.590 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.590 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:05.590 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530159+0000", 2024-09-18T04:38:05.590 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2802999999999999e-05 2024-09-18T04:38:05.590 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.590 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.591 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:05.591 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530218+0000", 2024-09-18T04:38:05.591 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8461000000000002e-05 2024-09-18T04:38:05.591 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.591 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.591 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:05.591 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530253+0000", 2024-09-18T04:38:05.591 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4796999999999999e-05 2024-09-18T04:38:05.591 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.591 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.591 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:05.591 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.532559+0000", 2024-09-18T04:38:05.591 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002306939 2024-09-18T04:38:05.592 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.592 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.592 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:05.592 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.532621+0000", 2024-09-18T04:38:05.592 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1131999999999997e-05 2024-09-18T04:38:05.592 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:05.592 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:05.592 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:05.592 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.592 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.592 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:68 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'42, mlcod=15'39)", 2024-09-18T04:38:05.592 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.546404+0000", 2024-09-18T04:38:05.593 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 38.027937207000001, 2024-09-18T04:38:05.593 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024499769999999999, 2024-09-18T04:38:05.593 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:05.593 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:05.593 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:05.593 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.593 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:05.593 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546404+0000", 2024-09-18T04:38:05.593 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:05.593 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.593 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.593 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:05.594 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546404+0000", 2024-09-18T04:38:05.594 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:05.594 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.594 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.594 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:05.594 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546403+0000", 2024-09-18T04:38:05.594 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:05.594 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.594 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.594 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:05.594 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546416+0000", 2024-09-18T04:38:05.594 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.306e-05 2024-09-18T04:38:05.594 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.594 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.595 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:05.595 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546465+0000", 2024-09-18T04:38:05.595 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9441999999999999e-05 2024-09-18T04:38:05.595 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.595 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.595 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:05.595 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546480+0000", 2024-09-18T04:38:05.595 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4919999999999999e-05 2024-09-18T04:38:05.595 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.595 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.595 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:05.595 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546541+0000", 2024-09-18T04:38:05.595 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1085000000000002e-05 2024-09-18T04:38:05.595 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.596 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.596 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:05.596 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546569+0000", 2024-09-18T04:38:05.596 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7803000000000001e-05 2024-09-18T04:38:05.596 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.596 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.596 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:05.596 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.548725+0000", 2024-09-18T04:38:05.596 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021556629999999999 2024-09-18T04:38:05.596 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.596 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.596 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:05.597 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.548854+0000", 2024-09-18T04:38:05.597 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000129524 2024-09-18T04:38:05.597 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:05.597 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:05.597 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:05.597 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.597 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.597 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:69 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'43, mlcod=15'39)", 2024-09-18T04:38:05.597 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.547520+0000", 2024-09-18T04:38:05.597 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 38.026821130999998, 2024-09-18T04:38:05.597 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023506820000000002, 2024-09-18T04:38:05.597 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:05.597 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:05.929 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:05.929 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.929 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:05.930 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547520+0000", 2024-09-18T04:38:05.930 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:05.930 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.930 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.930 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:05.930 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547520+0000", 2024-09-18T04:38:05.930 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:05.930 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.930 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.930 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:05.930 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547519+0000", 2024-09-18T04:38:05.931 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:05.931 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.931 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.931 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:05.931 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547614+0000", 2024-09-18T04:38:05.931 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5235000000000006e-05 2024-09-18T04:38:05.931 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.931 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.931 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:05.931 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547640+0000", 2024-09-18T04:38:05.931 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5698999999999999e-05 2024-09-18T04:38:05.931 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.931 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.932 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:05.932 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547651+0000", 2024-09-18T04:38:05.932 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.134e-05 2024-09-18T04:38:05.932 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.932 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.932 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:05.932 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547805+0000", 2024-09-18T04:38:05.932 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015426099999999999 2024-09-18T04:38:05.932 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.932 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.932 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:05.932 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547844+0000", 2024-09-18T04:38:05.932 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8229999999999998e-05 2024-09-18T04:38:05.932 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.933 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.933 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:05.933 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.549810+0000", 2024-09-18T04:38:05.933 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0019660519999999998 2024-09-18T04:38:05.933 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.933 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.933 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:05.933 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.549871+0000", 2024-09-18T04:38:05.933 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1152e-05 2024-09-18T04:38:05.933 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:05.934 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:05.934 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:05.934 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.934 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.934 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:83 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'53, mlcod=15'50)", 2024-09-18T04:38:05.934 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.575989+0000", 2024-09-18T04:38:05.934 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 37.998352183000002, 2024-09-18T04:38:05.935 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023991310000000001, 2024-09-18T04:38:05.935 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:05.935 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:05.935 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:05.935 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.935 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:05.935 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.575989+0000", 2024-09-18T04:38:05.936 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:05.936 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.936 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.936 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:05.936 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.575989+0000", 2024-09-18T04:38:05.936 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:05.936 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.936 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.936 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:05.936 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.575988+0000", 2024-09-18T04:38:05.936 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:05.936 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.936 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.937 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:05.937 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576001+0000", 2024-09-18T04:38:05.937 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2751e-05 2024-09-18T04:38:05.937 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.937 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.938 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:05.938 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576023+0000", 2024-09-18T04:38:05.938 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2453999999999999e-05 2024-09-18T04:38:05.938 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.938 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.938 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:05.938 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576038+0000", 2024-09-18T04:38:05.938 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5125e-05 2024-09-18T04:38:05.938 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.938 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.938 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:05.939 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576161+0000", 2024-09-18T04:38:05.939 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000122543 2024-09-18T04:38:05.939 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.939 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.939 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:05.939 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576221+0000", 2024-09-18T04:38:05.939 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9948000000000003e-05 2024-09-18T04:38:05.939 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.939 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.939 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:05.939 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.578277+0000", 2024-09-18T04:38:05.939 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020567559999999999 2024-09-18T04:38:05.940 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.940 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.940 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:05.940 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.578388+0000", 2024-09-18T04:38:05.940 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011089100000000001 2024-09-18T04:38:05.940 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:05.940 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:05.940 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:05.940 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.940 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.940 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:86 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'56, mlcod=15'52)", 2024-09-18T04:38:05.940 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.580261+0000", 2024-09-18T04:38:05.940 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 37.994080547000003, 2024-09-18T04:38:05.941 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002418108, 2024-09-18T04:38:05.941 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:05.941 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:05.941 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:05.941 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.941 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:05.941 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580261+0000", 2024-09-18T04:38:05.941 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:05.941 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.941 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.941 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:05.941 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580261+0000", 2024-09-18T04:38:05.941 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:05.941 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.942 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.942 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:05.942 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580260+0000", 2024-09-18T04:38:05.942 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:05.942 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.942 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.942 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:05.942 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580269+0000", 2024-09-18T04:38:05.942 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5109999999999992e-06 2024-09-18T04:38:05.942 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.942 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.942 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:05.942 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580288+0000", 2024-09-18T04:38:05.942 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8783999999999998e-05 2024-09-18T04:38:05.943 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.943 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.943 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:05.943 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580296+0000", 2024-09-18T04:38:05.943 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3250000000000008e-06 2024-09-18T04:38:05.943 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.943 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.943 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:05.943 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580351+0000", 2024-09-18T04:38:05.943 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4132000000000003e-05 2024-09-18T04:38:05.943 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.943 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.943 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:05.943 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580381+0000", 2024-09-18T04:38:05.944 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0263000000000001e-05 2024-09-18T04:38:05.944 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.944 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.944 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:05.944 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.582615+0000", 2024-09-18T04:38:05.944 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0022339109999999999 2024-09-18T04:38:05.944 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.944 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.944 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:05.944 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.582679+0000", 2024-09-18T04:38:05.944 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4182000000000003e-05 2024-09-18T04:38:05.944 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:05.945 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:05.945 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:05.945 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.945 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.945 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:105 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'67, mlcod=15'64)", 2024-09-18T04:38:05.945 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.789089+0000", 2024-09-18T04:38:05.945 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 37.785252376999999, 2024-09-18T04:38:05.945 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024384229999999999, 2024-09-18T04:38:05.945 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:05.945 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:05.945 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:05.945 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.946 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:05.946 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789089+0000", 2024-09-18T04:38:05.946 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:05.946 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.946 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.946 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:05.946 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789089+0000", 2024-09-18T04:38:05.946 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:05.946 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.946 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.946 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:05.946 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789088+0000", 2024-09-18T04:38:05.946 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:05.947 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.947 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.947 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:05.947 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789107+0000", 2024-09-18T04:38:05.947 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9017000000000001e-05 2024-09-18T04:38:05.947 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.947 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.947 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:05.947 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789139+0000", 2024-09-18T04:38:05.947 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2261999999999997e-05 2024-09-18T04:38:05.947 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.947 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.947 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:05.947 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789153+0000", 2024-09-18T04:38:05.948 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4025999999999999e-05 2024-09-18T04:38:05.948 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.948 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.948 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:05.948 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789255+0000", 2024-09-18T04:38:05.948 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000101969 2024-09-18T04:38:05.948 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.948 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.948 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:05.948 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789314+0000", 2024-09-18T04:38:05.948 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8867999999999999e-05 2024-09-18T04:38:05.948 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.949 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.949 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:05.949 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.791453+0000", 2024-09-18T04:38:05.949 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002138938 2024-09-18T04:38:05.949 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.949 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.949 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:05.949 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.791527+0000", 2024-09-18T04:38:05.949 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4464999999999999e-05 2024-09-18T04:38:05.949 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:05.950 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:05.950 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:05.950 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.950 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.950 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:106 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'68, mlcod=15'64)", 2024-09-18T04:38:05.950 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.790295+0000", 2024-09-18T04:38:05.950 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 37.784046525999997, 2024-09-18T04:38:05.950 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024379390000000001, 2024-09-18T04:38:05.950 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:05.950 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:05.951 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:05.951 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.951 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:05.951 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790295+0000", 2024-09-18T04:38:05.951 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:05.951 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.951 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.951 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:05.951 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790295+0000", 2024-09-18T04:38:05.951 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:05.952 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.952 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.952 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:05.952 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790294+0000", 2024-09-18T04:38:05.952 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:05.952 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.952 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.952 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:05.952 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790390+0000", 2024-09-18T04:38:05.952 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6537000000000001e-05 2024-09-18T04:38:05.952 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.952 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.952 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:05.953 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790418+0000", 2024-09-18T04:38:05.953 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8218e-05 2024-09-18T04:38:05.953 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.953 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.953 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:05.953 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790431+0000", 2024-09-18T04:38:05.953 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2571999999999999e-05 2024-09-18T04:38:05.953 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.953 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.953 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:05.953 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790534+0000", 2024-09-18T04:38:05.953 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000103433 2024-09-18T04:38:05.953 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.954 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.954 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:05.954 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790565+0000", 2024-09-18T04:38:05.956 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0409999999999999e-05 2024-09-18T04:38:05.957 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.957 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.957 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:05.957 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.792651+0000", 2024-09-18T04:38:05.957 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020865250000000001 2024-09-18T04:38:05.957 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.957 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.957 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:05.957 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.792733+0000", 2024-09-18T04:38:05.957 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1477999999999995e-05 2024-09-18T04:38:05.957 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:05.957 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:05.958 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:05.958 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.958 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.958 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:109 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'71, mlcod=15'68)", 2024-09-18T04:38:05.958 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.795250+0000", 2024-09-18T04:38:05.958 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 37.779091512000001, 2024-09-18T04:38:05.958 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023990449999999998, 2024-09-18T04:38:05.958 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:05.958 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:05.958 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:05.958 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.958 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:05.958 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795250+0000", 2024-09-18T04:38:05.958 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:05.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.959 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.959 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:05.959 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795250+0000", 2024-09-18T04:38:05.959 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:05.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.959 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.959 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:05.959 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795248+0000", 2024-09-18T04:38:05.959 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:05.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.960 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.960 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:05.960 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795405+0000", 2024-09-18T04:38:05.960 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000157007 2024-09-18T04:38:05.960 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.960 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.960 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:05.960 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795422+0000", 2024-09-18T04:38:05.960 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7013000000000001e-05 2024-09-18T04:38:05.960 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.960 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.961 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:05.961 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795432+0000", 2024-09-18T04:38:05.961 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0448999999999999e-05 2024-09-18T04:38:05.961 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.961 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.961 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:05.961 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795534+0000", 2024-09-18T04:38:05.962 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000101195 2024-09-18T04:38:05.962 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.962 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.962 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:05.962 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795587+0000", 2024-09-18T04:38:05.962 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3356000000000002e-05 2024-09-18T04:38:05.962 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.962 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.962 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:05.962 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.797560+0000", 2024-09-18T04:38:05.962 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001973023 2024-09-18T04:38:05.962 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.963 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.963 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:05.963 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.797649+0000", 2024-09-18T04:38:05.963 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8864999999999997e-05 2024-09-18T04:38:05.963 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:05.963 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:05.963 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:05.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.963 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.963 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:115 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'77, mlcod=15'72)", 2024-09-18T04:38:05.963 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.803339+0000", 2024-09-18T04:38:05.964 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 37.771002793999997, 2024-09-18T04:38:05.964 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002385093, 2024-09-18T04:38:05.964 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:05.964 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:05.964 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:05.964 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.964 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:05.964 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803339+0000", 2024-09-18T04:38:05.964 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:05.964 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.964 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.965 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:05.965 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803339+0000", 2024-09-18T04:38:05.965 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:05.965 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.965 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.965 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:05.965 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803337+0000", 2024-09-18T04:38:05.965 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:05.965 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.965 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.965 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:05.966 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803475+0000", 2024-09-18T04:38:05.966 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013718199999999999 2024-09-18T04:38:05.966 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.966 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.966 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:05.966 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803507+0000", 2024-09-18T04:38:05.966 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2351000000000002e-05 2024-09-18T04:38:05.966 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.966 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.966 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:05.967 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803533+0000", 2024-09-18T04:38:05.967 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6010999999999999e-05 2024-09-18T04:38:05.967 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.967 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.967 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:05.968 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803609+0000", 2024-09-18T04:38:05.968 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.64e-05 2024-09-18T04:38:05.968 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.968 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.968 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:05.968 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803671+0000", 2024-09-18T04:38:05.968 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1467e-05 2024-09-18T04:38:05.968 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.969 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.969 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:05.969 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.805676+0000", 2024-09-18T04:38:05.969 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020047849999999998 2024-09-18T04:38:05.969 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.969 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.969 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:05.969 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.805724+0000", 2024-09-18T04:38:05.969 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8151e-05 2024-09-18T04:38:05.969 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:05.969 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:05.969 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:05.969 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.970 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.970 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:118 1.0 e15/14 1:30306672:devicehealth::main.db.0000000000000000:head v 15'80, mlcod=15'78)", 2024-09-18T04:38:05.970 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.807930+0000", 2024-09-18T04:38:05.970 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 37.766411290999997, 2024-09-18T04:38:05.970 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028002259999999998, 2024-09-18T04:38:05.970 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:05.970 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:05.970 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:05.970 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.970 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:05.970 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.807930+0000", 2024-09-18T04:38:05.970 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:05.970 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.970 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.971 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:05.971 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.807930+0000", 2024-09-18T04:38:05.971 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:05.971 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.971 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.971 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:05.971 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.807929+0000", 2024-09-18T04:38:05.971 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:05.971 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.971 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.971 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:05.971 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808024+0000", 2024-09-18T04:38:05.972 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5258000000000002e-05 2024-09-18T04:38:05.972 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.972 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.972 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:05.972 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808044+0000", 2024-09-18T04:38:05.973 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0203e-05 2024-09-18T04:38:05.973 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.973 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.973 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:05.974 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808053+0000", 2024-09-18T04:38:05.974 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2689999999999999e-06 2024-09-18T04:38:05.974 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.974 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.974 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:05.974 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808137+0000", 2024-09-18T04:38:05.974 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4441e-05 2024-09-18T04:38:05.974 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.974 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.974 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:05.975 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808172+0000", 2024-09-18T04:38:05.975 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4453999999999999e-05 2024-09-18T04:38:05.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.975 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.975 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:05.975 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.810663+0000", 2024-09-18T04:38:05.975 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024910269999999998 2024-09-18T04:38:05.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.975 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.975 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:05.975 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.810730+0000", 2024-09-18T04:38:05.975 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7658999999999997e-05 2024-09-18T04:38:05.975 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:05.976 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:05.976 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:05.976 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.976 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.976 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:129 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'87, mlcod=15'85)", 2024-09-18T04:38:05.976 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.231689+0000", 2024-09-18T04:38:05.976 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 37.342652278000003, 2024-09-18T04:38:05.976 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002376502, 2024-09-18T04:38:05.976 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:05.976 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:05.976 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:05.976 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.976 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:05.976 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231689+0000", 2024-09-18T04:38:05.977 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:05.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.977 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.977 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:05.977 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231689+0000", 2024-09-18T04:38:05.977 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:05.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.977 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.977 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:05.977 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231687+0000", 2024-09-18T04:38:05.977 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:05.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.977 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.978 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:05.978 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231702+0000", 2024-09-18T04:38:05.978 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4949999999999999e-05 2024-09-18T04:38:05.978 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.978 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.978 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:05.978 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231735+0000", 2024-09-18T04:38:05.978 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2276000000000001e-05 2024-09-18T04:38:05.978 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.978 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.978 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:05.978 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231750+0000", 2024-09-18T04:38:05.979 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5193e-05 2024-09-18T04:38:05.979 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.979 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.979 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:05.979 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231872+0000", 2024-09-18T04:38:05.979 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000122127 2024-09-18T04:38:05.979 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.979 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.979 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:05.979 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231936+0000", 2024-09-18T04:38:05.979 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4089000000000003e-05 2024-09-18T04:38:05.979 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.979 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.980 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:05.980 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234002+0000", 2024-09-18T04:38:05.980 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002065575 2024-09-18T04:38:05.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.980 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.980 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:05.980 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234066+0000", 2024-09-18T04:38:05.980 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3944999999999998e-05 2024-09-18T04:38:05.980 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:05.980 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:05.980 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:05.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.980 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.981 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:130 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'88, mlcod=15'85)", 2024-09-18T04:38:05.981 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.232853+0000", 2024-09-18T04:38:05.981 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 37.341488663, 2024-09-18T04:38:05.981 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027351739999999999, 2024-09-18T04:38:05.981 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:05.981 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:05.981 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:05.981 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.981 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:05.981 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232853+0000", 2024-09-18T04:38:05.981 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:05.981 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.982 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.982 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:05.982 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232853+0000", 2024-09-18T04:38:05.982 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:05.982 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.982 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.982 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:05.982 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232851+0000", 2024-09-18T04:38:05.982 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:05.982 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.982 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.983 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:05.983 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232866+0000", 2024-09-18T04:38:05.983 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4946000000000001e-05 2024-09-18T04:38:05.983 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.983 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.983 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:05.983 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232897+0000", 2024-09-18T04:38:05.983 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0982999999999998e-05 2024-09-18T04:38:05.983 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.983 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.983 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:05.984 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232915+0000", 2024-09-18T04:38:05.984 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8476000000000001e-05 2024-09-18T04:38:05.984 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.984 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.984 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:05.984 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.233029+0000", 2024-09-18T04:38:05.984 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011305400000000001 2024-09-18T04:38:05.985 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.985 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.985 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:05.985 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.233079+0000", 2024-09-18T04:38:05.985 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0668999999999999e-05 2024-09-18T04:38:05.985 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.985 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.985 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:05.985 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235534+0000", 2024-09-18T04:38:05.985 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024546030000000001 2024-09-18T04:38:05.985 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.985 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.985 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:05.985 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235588+0000", 2024-09-18T04:38:05.985 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4098999999999998e-05 2024-09-18T04:38:05.986 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:05.986 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:05.986 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:05.986 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.986 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.986 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:131 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'89, mlcod=15'85)", 2024-09-18T04:38:05.986 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.234089+0000", 2024-09-18T04:38:05.986 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 37.340252808999999, 2024-09-18T04:38:05.986 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026975609999999998, 2024-09-18T04:38:05.986 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:05.986 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:05.987 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:05.987 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.987 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:05.987 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234089+0000", 2024-09-18T04:38:05.987 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:05.987 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.987 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.987 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:05.987 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234089+0000", 2024-09-18T04:38:05.987 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:05.987 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.987 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.987 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:05.988 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234087+0000", 2024-09-18T04:38:05.988 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:05.988 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.988 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.988 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:05.988 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234145+0000", 2024-09-18T04:38:05.988 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8687999999999998e-05 2024-09-18T04:38:05.988 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.988 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.988 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:05.988 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234177+0000", 2024-09-18T04:38:05.988 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1526999999999998e-05 2024-09-18T04:38:05.988 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.988 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.988 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:05.989 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234190+0000", 2024-09-18T04:38:05.989 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2965999999999999e-05 2024-09-18T04:38:05.989 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.989 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.989 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:05.989 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234309+0000", 2024-09-18T04:38:05.990 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011875399999999999 2024-09-18T04:38:05.990 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.990 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.990 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:05.990 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234365+0000", 2024-09-18T04:38:05.991 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6436e-05 2024-09-18T04:38:05.991 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.991 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.991 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:05.991 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236675+0000", 2024-09-18T04:38:05.992 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023099520000000001 2024-09-18T04:38:05.992 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.992 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.992 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:05.992 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236786+0000", 2024-09-18T04:38:05.992 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000111025 2024-09-18T04:38:05.992 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:05.992 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:05.993 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:05.993 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.993 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.993 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:132 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'90, mlcod=15'85)", 2024-09-18T04:38:05.993 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.235300+0000", 2024-09-18T04:38:05.993 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 37.339041358999999, 2024-09-18T04:38:05.993 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025982599999999998, 2024-09-18T04:38:05.993 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:05.993 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:05.993 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:05.993 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.994 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:05.994 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235300+0000", 2024-09-18T04:38:05.994 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:05.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.994 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.994 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:05.994 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235300+0000", 2024-09-18T04:38:05.994 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:05.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.995 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.995 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:05.995 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235298+0000", 2024-09-18T04:38:05.995 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:05.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.995 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.995 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:05.995 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235318+0000", 2024-09-18T04:38:05.995 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9714000000000001e-05 2024-09-18T04:38:05.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.995 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.995 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:05.995 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235337+0000", 2024-09-18T04:38:05.996 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8892000000000001e-05 2024-09-18T04:38:05.996 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.996 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.996 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:05.996 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235348+0000", 2024-09-18T04:38:05.996 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.129e-05 2024-09-18T04:38:05.996 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.996 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.996 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:05.996 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235681+0000", 2024-09-18T04:38:05.996 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00033323000000000001 2024-09-18T04:38:05.997 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.997 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.997 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:05.997 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235733+0000", 2024-09-18T04:38:05.997 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1391999999999999e-05 2024-09-18T04:38:05.997 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.997 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.997 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:05.997 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237839+0000", 2024-09-18T04:38:05.998 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002105647 2024-09-18T04:38:05.998 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.998 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.998 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:05.998 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237898+0000", 2024-09-18T04:38:05.998 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9759000000000002e-05 2024-09-18T04:38:05.998 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:05.998 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:05.999 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:05.999 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:05.999 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:05.999 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:133 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'91, mlcod=15'86)", 2024-09-18T04:38:05.999 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.236920+0000", 2024-09-18T04:38:05.999 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 37.337421470999999, 2024-09-18T04:38:05.999 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024318410000000001, 2024-09-18T04:38:06.000 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:06.000 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:06.000 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:06.000 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.000 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:06.000 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236920+0000", 2024-09-18T04:38:06.000 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:06.000 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.000 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.000 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:06.001 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236920+0000", 2024-09-18T04:38:06.001 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:06.001 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.001 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.001 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:06.001 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236918+0000", 2024-09-18T04:38:06.001 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:06.001 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.001 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.002 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:06.002 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236931+0000", 2024-09-18T04:38:06.002 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2683999999999999e-05 2024-09-18T04:38:06.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.002 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.002 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:06.002 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236951+0000", 2024-09-18T04:38:06.002 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0292000000000002e-05 2024-09-18T04:38:06.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.002 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.002 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:06.002 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236962+0000", 2024-09-18T04:38:06.003 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0703000000000001e-05 2024-09-18T04:38:06.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.003 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.003 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:06.003 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237095+0000", 2024-09-18T04:38:06.003 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013312400000000001 2024-09-18T04:38:06.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.003 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.003 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:06.003 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237133+0000", 2024-09-18T04:38:06.004 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7496000000000001e-05 2024-09-18T04:38:06.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.004 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.004 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:06.004 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.239206+0000", 2024-09-18T04:38:06.004 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020730290000000001 2024-09-18T04:38:06.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.004 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.004 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:06.004 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.239352+0000", 2024-09-18T04:38:06.005 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014593000000000001 2024-09-18T04:38:06.005 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:06.005 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:06.005 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:06.005 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.005 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.005 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:134 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'92, mlcod=15'86)", 2024-09-18T04:38:06.005 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.237894+0000", 2024-09-18T04:38:06.005 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 37.336447448999998, 2024-09-18T04:38:06.005 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025299519999999998, 2024-09-18T04:38:06.005 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:06.006 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:06.006 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:06.006 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.006 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:06.006 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237894+0000", 2024-09-18T04:38:06.006 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:06.006 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.006 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.006 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:06.006 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237894+0000", 2024-09-18T04:38:06.006 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:06.006 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.006 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.006 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:06.007 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237892+0000", 2024-09-18T04:38:06.007 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:06.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.007 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.007 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:06.007 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237986+0000", 2024-09-18T04:38:06.007 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3341e-05 2024-09-18T04:38:06.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.007 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.007 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:06.007 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.238012+0000", 2024-09-18T04:38:06.008 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5843e-05 2024-09-18T04:38:06.008 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.008 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.008 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:06.008 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.238027+0000", 2024-09-18T04:38:06.008 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5469e-05 2024-09-18T04:38:06.008 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.008 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.008 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:06.008 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.238114+0000", 2024-09-18T04:38:06.008 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7483000000000002e-05 2024-09-18T04:38:06.008 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.008 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.009 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:06.009 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.238152+0000", 2024-09-18T04:38:06.009 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7565999999999999e-05 2024-09-18T04:38:06.009 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.009 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.009 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:06.009 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.240297+0000", 2024-09-18T04:38:06.009 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021447620000000001 2024-09-18T04:38:06.009 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.009 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.009 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:06.009 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.240424+0000", 2024-09-18T04:38:06.010 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012705400000000001 2024-09-18T04:38:06.010 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:06.010 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:06.010 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:06.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.010 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.010 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:139 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'97, mlcod=15'93)", 2024-09-18T04:38:06.010 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.245692+0000", 2024-09-18T04:38:06.010 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 37.328649308000003, 2024-09-18T04:38:06.010 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024260480000000001, 2024-09-18T04:38:06.010 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:06.010 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:06.010 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:06.010 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.010 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:06.011 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245692+0000", 2024-09-18T04:38:06.011 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:06.011 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.011 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.011 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:06.011 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245692+0000", 2024-09-18T04:38:06.011 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:06.011 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.011 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.011 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:06.011 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245690+0000", 2024-09-18T04:38:06.011 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:06.012 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.012 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.012 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:06.012 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245712+0000", 2024-09-18T04:38:06.012 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1916000000000002e-05 2024-09-18T04:38:06.012 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.012 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.012 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:06.012 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245741+0000", 2024-09-18T04:38:06.012 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8503000000000001e-05 2024-09-18T04:38:06.012 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.012 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.013 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:06.013 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245759+0000", 2024-09-18T04:38:06.013 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.872e-05 2024-09-18T04:38:06.013 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.013 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.013 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:06.013 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245859+0000", 2024-09-18T04:38:06.013 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9641000000000004e-05 2024-09-18T04:38:06.013 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.013 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.013 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:06.013 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245900+0000", 2024-09-18T04:38:06.013 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0589000000000001e-05 2024-09-18T04:38:06.013 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.014 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.014 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:06.014 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.247961+0000", 2024-09-18T04:38:06.014 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020619190000000002 2024-09-18T04:38:06.014 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.014 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.014 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:06.014 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.248118+0000", 2024-09-18T04:38:06.014 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015664300000000001 2024-09-18T04:38:06.014 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:06.014 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:06.014 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:06.014 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.014 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.015 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:144 1.0 e15/14 1:30306672:devicehealth::main.db.0000000000000000:head v 15'102, mlcod=15'98)", 2024-09-18T04:38:06.015 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.252149+0000", 2024-09-18T04:38:06.015 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 37.322192242, 2024-09-18T04:38:06.015 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024174959999999999, 2024-09-18T04:38:06.015 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:06.015 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:06.015 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:06.015 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.015 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:06.015 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252149+0000", 2024-09-18T04:38:06.016 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:06.016 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.016 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.016 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:06.016 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252149+0000", 2024-09-18T04:38:06.017 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:06.017 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.017 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.017 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:06.017 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252148+0000", 2024-09-18T04:38:06.017 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:06.018 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.018 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.018 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:06.018 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252225+0000", 2024-09-18T04:38:06.018 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6576999999999995e-05 2024-09-18T04:38:06.019 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.019 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.019 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:06.019 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252242+0000", 2024-09-18T04:38:06.019 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7740999999999998e-05 2024-09-18T04:38:06.020 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.020 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.020 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:06.020 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252254+0000", 2024-09-18T04:38:06.020 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.137e-05 2024-09-18T04:38:06.021 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.021 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.021 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:06.021 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252385+0000", 2024-09-18T04:38:06.021 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013091099999999999 2024-09-18T04:38:06.022 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.022 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.022 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:06.022 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252426+0000", 2024-09-18T04:38:06.022 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1535999999999997e-05 2024-09-18T04:38:06.022 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.023 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.023 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:06.023 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.254498+0000", 2024-09-18T04:38:06.023 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002071477 2024-09-18T04:38:06.023 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.024 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.024 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:06.024 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.254567+0000", 2024-09-18T04:38:06.024 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9014e-05 2024-09-18T04:38:06.024 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:06.025 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:06.025 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:06.025 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:06.025 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:06.025 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:06.028 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:06.022+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:06.029 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:38:06.031 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-09-18T04:38:06.128 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:06.122+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:06.189 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:06.189 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:06.189 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:38:06.189 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:06.199 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:38:06.228 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:06.223+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:06.308 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:06.308 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:06.308 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:38:06.308 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:38:06.308 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:06.318 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:38:06.366 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:06.363+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:06.427 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:06.427 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:38:06.427 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:38:06.427 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:38:06.427 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.427 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4214.0:5 2.2 e19/17 2:5cea7035:::rbd_info:head v 19'1, mlcod=0'0)", 2024-09-18T04:38:06.427 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:35.471260+0000", 2024-09-18T04:38:06.427 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 30.954771402999999, 2024-09-18T04:38:06.428 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021025950000000001, 2024-09-18T04:38:06.428 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:06.428 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:06.428 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:06.428 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.428 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:06.428 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471260+0000", 2024-09-18T04:38:06.428 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:06.428 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.428 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.428 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:06.429 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471260+0000", 2024-09-18T04:38:06.429 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:06.429 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.429 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.429 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:06.429 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471259+0000", 2024-09-18T04:38:06.429 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:06.429 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.429 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.430 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:06.430 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471275+0000", 2024-09-18T04:38:06.430 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6135999999999999e-05 2024-09-18T04:38:06.430 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.430 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.430 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:06.430 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471298+0000", 2024-09-18T04:38:06.430 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3632999999999999e-05 2024-09-18T04:38:06.430 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.430 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.430 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:06.431 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471320+0000", 2024-09-18T04:38:06.431 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1852999999999999e-05 2024-09-18T04:38:06.431 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.431 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.431 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:06.431 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471445+0000", 2024-09-18T04:38:06.431 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012461699999999999 2024-09-18T04:38:06.431 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.431 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.431 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:06.431 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471529+0000", 2024-09-18T04:38:06.431 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3875000000000006e-05 2024-09-18T04:38:06.432 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.432 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.432 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:06.432 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.473317+0000", 2024-09-18T04:38:06.432 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017886899999999999 2024-09-18T04:38:06.432 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.432 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.432 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:06.432 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.473362+0000", 2024-09-18T04:38:06.432 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4724000000000001e-05 2024-09-18T04:38:06.433 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:06.433 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:06.433 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:06.433 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.433 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.433 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4214.0:7 2.2 e20/17 2:5cea7035:::rbd_info:head v 20'2, mlcod=0'0)", 2024-09-18T04:38:06.433 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:36.490560+0000", 2024-09-18T04:38:06.433 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 29.935471264, 2024-09-18T04:38:06.433 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0015326020000000001, 2024-09-18T04:38:06.433 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:06.433 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:06.434 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:06.434 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.434 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:06.434 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490560+0000", 2024-09-18T04:38:06.434 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:06.434 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.434 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.434 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:06.434 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490560+0000", 2024-09-18T04:38:06.434 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:06.435 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.435 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.435 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:06.435 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490559+0000", 2024-09-18T04:38:06.435 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:06.435 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.435 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.435 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:06.435 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490568+0000", 2024-09-18T04:38:06.435 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5699999999999993e-06 2024-09-18T04:38:06.435 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.435 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.436 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:06.436 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490599+0000", 2024-09-18T04:38:06.436 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1566000000000002e-05 2024-09-18T04:38:06.436 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.436 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.436 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:06.436 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490609+0000", 2024-09-18T04:38:06.436 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.8700000000000004e-06 2024-09-18T04:38:06.436 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.436 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.436 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:06.436 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490708+0000", 2024-09-18T04:38:06.437 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9139000000000001e-05 2024-09-18T04:38:06.437 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.437 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.437 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:06.437 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490736+0000", 2024-09-18T04:38:06.437 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7617e-05 2024-09-18T04:38:06.437 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.437 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.437 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:06.437 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.492016+0000", 2024-09-18T04:38:06.437 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0012796579999999999 2024-09-18T04:38:06.437 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.437 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.438 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:06.438 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.492092+0000", 2024-09-18T04:38:06.438 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6571999999999998e-05 2024-09-18T04:38:06.438 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:06.438 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:06.438 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:06.438 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:06.438 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:06.438 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:06.440 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:38:06.440 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:06.434+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:06.540 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:06.535+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:06.549 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:06.550 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:06.550 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:38:06.550 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:06.560 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:38:06.614 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T04:38:06.662 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:06.659+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:06.670 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:06.670 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:06.671 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:38:06.671 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:38:06.671 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:06.681 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:38:06.743 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:06.739+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:06.756 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:38:06.777 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:38:06.788 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:06.788 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:38:06.788 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:38:06.789 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:38:06.789 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.789 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4214.0:1 2.3 e18/17 2:c4c92e5a:::rbd_trash:head v 18'1, mlcod=0'0)", 2024-09-18T04:38:06.789 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:34.475123+0000", 2024-09-18T04:38:06.789 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 32.312547166999998, 2024-09-18T04:38:06.789 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00081394599999999996, 2024-09-18T04:38:06.789 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:06.789 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:06.789 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:06.789 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.789 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:06.789 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475123+0000", 2024-09-18T04:38:06.790 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:06.790 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.790 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.790 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:06.790 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475123+0000", 2024-09-18T04:38:06.790 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:06.790 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.790 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.790 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:06.790 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475121+0000", 2024-09-18T04:38:06.791 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:06.791 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.791 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.791 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:06.791 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475149+0000", 2024-09-18T04:38:06.791 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7155999999999999e-05 2024-09-18T04:38:06.791 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.791 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.791 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:06.791 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475165+0000", 2024-09-18T04:38:06.791 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6160000000000001e-05 2024-09-18T04:38:06.791 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.791 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.791 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:06.792 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475183+0000", 2024-09-18T04:38:06.792 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7918e-05 2024-09-18T04:38:06.792 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.792 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.792 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:06.792 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475246+0000", 2024-09-18T04:38:06.792 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3176000000000006e-05 2024-09-18T04:38:06.792 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.792 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.792 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:06.792 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475279+0000", 2024-09-18T04:38:06.792 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3192999999999999e-05 2024-09-18T04:38:06.792 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.792 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.793 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:06.793 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475871+0000", 2024-09-18T04:38:06.793 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00059206099999999995 2024-09-18T04:38:06.793 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.793 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.793 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:06.793 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475937+0000", 2024-09-18T04:38:06.793 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5439999999999997e-05 2024-09-18T04:38:06.793 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:06.793 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:06.794 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:06.794 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.794 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.794 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 25/24 [PushOp(2:c4c92e5a:::rbd_trash:head, version: 18'1, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:c4c92e5a:::rbd_trash:head@18'1, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-18T04:38:06.794 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:51.680544+0000", 2024-09-18T04:38:06.794 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 15.107125325, 2024-09-18T04:38:06.794 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00050365200000000005, 2024-09-18T04:38:06.794 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:06.794 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:38:06.794 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:06.794 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.794 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:06.794 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680544+0000", 2024-09-18T04:38:06.795 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:06.795 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.795 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.795 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:06.795 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680544+0000", 2024-09-18T04:38:06.795 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:06.795 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.795 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.795 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:06.795 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680543+0000", 2024-09-18T04:38:06.795 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:06.795 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.795 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.795 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:06.795 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680571+0000", 2024-09-18T04:38:06.796 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7953000000000002e-05 2024-09-18T04:38:06.796 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.796 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.796 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:06.796 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680605+0000", 2024-09-18T04:38:06.796 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3933000000000002e-05 2024-09-18T04:38:06.796 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.796 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.796 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:06.796 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680636+0000", 2024-09-18T04:38:06.796 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1288999999999998e-05 2024-09-18T04:38:06.797 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.797 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.797 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:06.797 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680748+0000", 2024-09-18T04:38:06.797 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000111431 2024-09-18T04:38:06.797 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.797 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.797 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:06.797 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680758+0000", 2024-09-18T04:38:06.797 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0244999999999999e-05 2024-09-18T04:38:06.797 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:06.797 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:06.797 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:06.798 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.681048+0000", 2024-09-18T04:38:06.798 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00029003399999999999 2024-09-18T04:38:06.798 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:06.798 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:06.798 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:06.798 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:06.798 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:06.798 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:06.799 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:38:06.800 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:38:06.820 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:38:06.841 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:38:06.847 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:06.842+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:06.863 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:38:06.865 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:38:06.865 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:38:06.865 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:38:06.865 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:38:06.865 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:38:06.865 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:38:06.918 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:06.918 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:06.928 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:38:06.945 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:06.939+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:07.037 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:07.038 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:07.046 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:07.042+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:07.047 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:38:07.148 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:07.143+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:07.157 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:07.157 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:07.166 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:38:07.248 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:07.243+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:07.275 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:07.275 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:07.284 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:38:07.349 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:07.342+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:07.391 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:07.391 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:07.401 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:38:07.450 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:07.446+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:07.550 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:07.550 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:07.551 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:07.546+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:07.559 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:38:07.651 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:07.647+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:07.718 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:07.718 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:07.727 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:38:07.752 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:07.747+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:07.839 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:07.839 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:07.850 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:38:07.853 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:07.851+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:07.953 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:07.951+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:07.954 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:07.954 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:07.963 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:38:08.032 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-18T04:38:08.054 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:08.051+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:08.083 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:08.083 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:08.094 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:38:08.154 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:08.151+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:08.247 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:08.248 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:08.255 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:08.250+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:08.257 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:38:08.356 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:08.351+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:08.366 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:08.366 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:08.376 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:38:08.456 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:08.451+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:08.485 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:08.486 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:08.496 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:38:08.557 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:08.550+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:08.651 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:08.651 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:08.658 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:08.654+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:08.660 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:38:08.759 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:08.755+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:08.763 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-09-18T04:38:08.775 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:08.775 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:08.785 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:38:08.861 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:08.854+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:08.883 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T04:38:08.938 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:08.938 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:08.947 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:38:08.961 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:08.955+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:09.025 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:38:09.043 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:38:09.055 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:09.055 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:09.061 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:09.055+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:09.065 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:38:09.066 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:38:09.088 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:38:09.112 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:38:09.150 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:38:09.152 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:38:09.152 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:38:09.152 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:38:09.152 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:38:09.152 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:38:09.153 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:38:09.162 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:09.159+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:09.212 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:09.212 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:38:09.213 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:38:09.213 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:38:09.213 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:09.213 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4214.0:1 2.3 e18/17 2:c4c92e5a:::rbd_trash:head v 18'1, mlcod=0'0)", 2024-09-18T04:38:09.213 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:34.475123+0000", 2024-09-18T04:38:09.213 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 34.736265764999999, 2024-09-18T04:38:09.213 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00081394599999999996, 2024-09-18T04:38:09.213 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:09.213 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:09.213 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:09.213 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:09.213 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:09.214 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475123+0000", 2024-09-18T04:38:09.214 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:09.214 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:09.214 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:09.214 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:09.214 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475123+0000", 2024-09-18T04:38:09.214 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:09.214 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:09.214 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:09.214 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:09.214 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475121+0000", 2024-09-18T04:38:09.214 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:09.214 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:09.214 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:09.215 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:09.215 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475149+0000", 2024-09-18T04:38:09.215 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7155999999999999e-05 2024-09-18T04:38:09.215 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:09.215 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:09.215 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:09.215 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475165+0000", 2024-09-18T04:38:09.215 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6160000000000001e-05 2024-09-18T04:38:09.215 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:09.215 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:09.215 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:09.215 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475183+0000", 2024-09-18T04:38:09.215 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7918e-05 2024-09-18T04:38:09.215 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:09.216 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:09.216 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:09.216 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475246+0000", 2024-09-18T04:38:09.216 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3176000000000006e-05 2024-09-18T04:38:09.216 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:09.216 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:09.216 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:09.216 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475279+0000", 2024-09-18T04:38:09.216 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3192999999999999e-05 2024-09-18T04:38:09.216 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:09.216 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:09.216 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:09.216 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475871+0000", 2024-09-18T04:38:09.216 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00059206099999999995 2024-09-18T04:38:09.216 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:09.217 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:09.217 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:09.217 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475937+0000", 2024-09-18T04:38:09.217 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5439999999999997e-05 2024-09-18T04:38:09.217 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:09.217 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:09.217 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:09.217 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:09.217 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:09.217 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 25/24 [PushOp(2:c4c92e5a:::rbd_trash:head, version: 18'1, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:c4c92e5a:::rbd_trash:head@18'1, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-18T04:38:09.217 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:51.680544+0000", 2024-09-18T04:38:09.217 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.530843922999999, 2024-09-18T04:38:09.217 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00050365200000000005, 2024-09-18T04:38:09.217 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:09.218 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:38:09.218 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:09.218 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:09.218 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:09.218 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680544+0000", 2024-09-18T04:38:09.218 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:09.218 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:09.218 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:09.218 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:09.218 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680544+0000", 2024-09-18T04:38:09.218 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:09.218 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:09.218 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:09.218 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:09.218 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680543+0000", 2024-09-18T04:38:09.218 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:09.219 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:09.219 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:09.219 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:09.219 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680571+0000", 2024-09-18T04:38:09.219 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7953000000000002e-05 2024-09-18T04:38:09.219 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:09.219 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:09.219 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:09.219 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680605+0000", 2024-09-18T04:38:09.219 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3933000000000002e-05 2024-09-18T04:38:09.219 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:09.219 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:09.219 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:09.220 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680636+0000", 2024-09-18T04:38:09.220 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1288999999999998e-05 2024-09-18T04:38:09.220 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:09.220 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:09.220 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:09.220 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680748+0000", 2024-09-18T04:38:09.220 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000111431 2024-09-18T04:38:09.220 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:09.220 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:09.220 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:09.220 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680758+0000", 2024-09-18T04:38:09.220 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0244999999999999e-05 2024-09-18T04:38:09.220 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:09.221 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:09.221 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:09.221 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.681048+0000", 2024-09-18T04:38:09.221 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00029003399999999999 2024-09-18T04:38:09.221 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:09.221 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:09.221 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:09.221 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:09.221 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:09.221 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:09.223 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:38:09.270 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:09.266+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:09.338 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:09.338 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:09.338 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:38:09.338 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:09.348 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:38:09.364 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:09.359+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:09.453 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:09.453 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:09.453 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:38:09.453 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:38:09.453 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:09.462 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:38:09.465 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:09.459+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:09.566 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:09.563+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:09.574 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:09.574 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:38:09.574 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:38:09.574 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:38:09.574 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.574 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:09.574 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.573328+0000", 2024-09-18T04:38:09.575 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 41.994412840000003, 2024-09-18T04:38:09.575 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011397898, 2024-09-18T04:38:09.575 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:09.575 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:09.575 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:09.575 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:09.575 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:09.575 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 86 2024-09-18T04:38:09.575 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.575 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:09.575 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.575 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:09.576 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573328+0000", 2024-09-18T04:38:09.576 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:09.576 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.576 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.576 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:09.576 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573328+0000", 2024-09-18T04:38:09.576 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:09.576 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.576 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.576 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:09.576 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573327+0000", 2024-09-18T04:38:09.576 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:09.577 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.577 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.577 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:09.577 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573331+0000", 2024-09-18T04:38:09.577 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9750000000000001e-06 2024-09-18T04:38:09.577 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.577 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.577 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:09.577 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573346+0000", 2024-09-18T04:38:09.577 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5469999999999999e-05 2024-09-18T04:38:09.577 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.577 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.577 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:09.578 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573370+0000", 2024-09-18T04:38:09.578 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3969999999999999e-05 2024-09-18T04:38:09.578 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.578 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.578 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:09.578 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.578536+0000", 2024-09-18T04:38:09.578 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0051659679999999999 2024-09-18T04:38:09.578 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.578 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.578 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:09.578 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.578689+0000", 2024-09-18T04:38:09.578 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015276299999999999 2024-09-18T04:38:09.578 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.579 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.579 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:09.579 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.578901+0000", 2024-09-18T04:38:09.579 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000211783 2024-09-18T04:38:09.579 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.579 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.579 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:09.579 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.583408+0000", 2024-09-18T04:38:09.579 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045071249999999998 2024-09-18T04:38:09.579 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.579 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.579 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:09.579 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.584645+0000", 2024-09-18T04:38:09.579 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001237007 2024-09-18T04:38:09.580 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.580 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.580 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:09.580 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.584703+0000", 2024-09-18T04:38:09.580 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.817e-05 2024-09-18T04:38:09.580 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.580 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.580 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:09.580 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.584725+0000", 2024-09-18T04:38:09.580 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2155e-05 2024-09-18T04:38:09.580 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:09.580 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:09.580 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:09.581 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.581 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.581 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:09.581 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.573382+0000", 2024-09-18T04:38:09.581 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 41.994358091999999, 2024-09-18T04:38:09.581 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012154974000000001, 2024-09-18T04:38:09.581 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:09.581 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:09.581 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:09.581 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:09.581 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:09.581 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 87 2024-09-18T04:38:09.582 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.582 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:09.582 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.582 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:09.582 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573382+0000", 2024-09-18T04:38:09.582 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:09.582 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.582 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.582 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:09.582 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573382+0000", 2024-09-18T04:38:09.582 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:09.582 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.582 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.583 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:09.583 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573382+0000", 2024-09-18T04:38:09.583 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:09.583 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.583 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.583 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:09.583 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573482+0000", 2024-09-18T04:38:09.583 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000100333 2024-09-18T04:38:09.583 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.583 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.583 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:09.583 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573492+0000", 2024-09-18T04:38:09.583 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5389999999999996e-06 2024-09-18T04:38:09.584 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.584 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.584 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:09.584 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573500+0000", 2024-09-18T04:38:09.584 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8929999999999994e-06 2024-09-18T04:38:09.584 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.584 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.584 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:09.584 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.579797+0000", 2024-09-18T04:38:09.584 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0062963239999999998 2024-09-18T04:38:09.584 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.584 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.584 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:09.585 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.579957+0000", 2024-09-18T04:38:09.585 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000160283 2024-09-18T04:38:09.585 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.585 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.585 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:09.585 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.580178+0000", 2024-09-18T04:38:09.585 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00022079200000000001 2024-09-18T04:38:09.585 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.585 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.585 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:09.585 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.583479+0000", 2024-09-18T04:38:09.585 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033007689999999998 2024-09-18T04:38:09.585 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.585 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.586 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:09.586 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585477+0000", 2024-09-18T04:38:09.586 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0019985670000000001 2024-09-18T04:38:09.586 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.586 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.586 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:09.586 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585517+0000", 2024-09-18T04:38:09.586 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9709999999999998e-05 2024-09-18T04:38:09.586 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.586 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.586 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:09.586 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585537+0000", 2024-09-18T04:38:09.586 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0326000000000002e-05 2024-09-18T04:38:09.587 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:09.587 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:09.587 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:09.587 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.587 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.587 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:09.587 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.573508+0000", 2024-09-18T04:38:09.587 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 41.994232386, 2024-09-18T04:38:09.587 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012068476, 2024-09-18T04:38:09.587 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:09.587 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:09.587 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:09.587 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:09.588 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:09.588 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 88 2024-09-18T04:38:09.588 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.588 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:09.588 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.588 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:09.588 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573508+0000", 2024-09-18T04:38:09.588 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:09.588 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.588 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.588 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:09.588 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573508+0000", 2024-09-18T04:38:09.588 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:09.588 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.589 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.589 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:09.589 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573507+0000", 2024-09-18T04:38:09.589 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:09.589 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.589 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.589 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:09.589 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573512+0000", 2024-09-18T04:38:09.589 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.2629999999999997e-06 2024-09-18T04:38:09.589 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.589 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.589 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:09.590 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573530+0000", 2024-09-18T04:38:09.590 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8507000000000001e-05 2024-09-18T04:38:09.590 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.590 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.590 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:09.590 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573537+0000", 2024-09-18T04:38:09.590 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.832e-06 2024-09-18T04:38:09.590 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.590 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.590 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:09.590 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.582486+0000", 2024-09-18T04:38:09.590 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0089494770000000008 2024-09-18T04:38:09.590 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.591 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.591 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:09.591 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.582630+0000", 2024-09-18T04:38:09.591 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000144005 2024-09-18T04:38:09.591 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.591 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.591 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:09.591 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.582842+0000", 2024-09-18T04:38:09.591 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00021189200000000001 2024-09-18T04:38:09.591 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.591 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.591 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:09.591 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.584777+0000", 2024-09-18T04:38:09.591 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001934753 2024-09-18T04:38:09.592 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.592 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.592 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:09.592 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585543+0000", 2024-09-18T04:38:09.592 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00076540099999999997 2024-09-18T04:38:09.592 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.592 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.592 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:09.592 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585561+0000", 2024-09-18T04:38:09.592 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8474999999999998e-05 2024-09-18T04:38:09.592 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.592 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.592 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:09.593 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585576+0000", 2024-09-18T04:38:09.593 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5427e-05 2024-09-18T04:38:09.593 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:09.593 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:09.593 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:09.593 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.593 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.593 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:09.593 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786394+0000", 2024-09-18T04:38:09.593 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 41.781345938999998, 2024-09-18T04:38:09.593 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011902085999999999, 2024-09-18T04:38:09.593 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:09.593 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:09.593 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:09.594 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:09.594 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:09.594 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 109 2024-09-18T04:38:09.594 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.594 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:09.594 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.594 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:09.594 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786394+0000", 2024-09-18T04:38:09.594 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:09.594 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.594 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.594 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:09.594 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786394+0000", 2024-09-18T04:38:09.594 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:09.595 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.595 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.595 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:09.595 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786394+0000", 2024-09-18T04:38:09.595 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:09.595 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.595 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.595 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:09.595 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786548+0000", 2024-09-18T04:38:09.595 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015463400000000001 2024-09-18T04:38:09.595 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.595 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.595 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:09.595 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786561+0000", 2024-09-18T04:38:09.596 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3093e-05 2024-09-18T04:38:09.596 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.596 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.596 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:09.596 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786571+0000", 2024-09-18T04:38:09.596 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4700000000000008e-06 2024-09-18T04:38:09.596 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.596 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.596 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:09.596 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.793367+0000", 2024-09-18T04:38:09.596 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0067966600000000004 2024-09-18T04:38:09.596 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.597 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.597 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:09.597 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.793526+0000", 2024-09-18T04:38:09.597 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015877 2024-09-18T04:38:09.597 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.597 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.597 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:09.597 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.793752+0000", 2024-09-18T04:38:09.597 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00022555200000000001 2024-09-18T04:38:09.597 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.597 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.597 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:09.597 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798125+0000", 2024-09-18T04:38:09.597 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0043733349999999999 2024-09-18T04:38:09.598 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.598 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.598 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:09.598 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798146+0000", 2024-09-18T04:38:09.598 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1063000000000001e-05 2024-09-18T04:38:09.598 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.598 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.598 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:09.598 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798232+0000", 2024-09-18T04:38:09.598 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5771000000000003e-05 2024-09-18T04:38:09.598 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.598 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.598 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:09.599 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798297+0000", 2024-09-18T04:38:09.599 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.4624999999999999e-05 2024-09-18T04:38:09.599 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:09.599 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:09.599 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:09.599 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.599 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.599 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:09.599 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786584+0000", 2024-09-18T04:38:09.599 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 41.781155859999998, 2024-09-18T04:38:09.599 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012013037000000001, 2024-09-18T04:38:09.599 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:09.600 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:09.600 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:09.600 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:09.600 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:09.600 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 110 2024-09-18T04:38:09.600 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.600 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:09.600 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.600 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:09.600 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786584+0000", 2024-09-18T04:38:09.600 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:09.600 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.600 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.601 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:09.601 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786584+0000", 2024-09-18T04:38:09.601 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:09.601 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.601 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.601 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:09.601 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786584+0000", 2024-09-18T04:38:09.601 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:09.601 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.601 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.601 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:09.601 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786590+0000", 2024-09-18T04:38:09.601 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6359999999999997e-06 2024-09-18T04:38:09.602 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.602 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.602 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:09.602 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786599+0000", 2024-09-18T04:38:09.602 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7369999999999995e-06 2024-09-18T04:38:09.602 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.602 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.602 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:09.602 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786606+0000", 2024-09-18T04:38:09.602 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1930000000000004e-06 2024-09-18T04:38:09.602 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.602 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.602 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:09.602 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.795018+0000", 2024-09-18T04:38:09.603 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0084115660000000005 2024-09-18T04:38:09.603 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.603 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.603 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:09.603 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.795184+0000", 2024-09-18T04:38:09.603 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000165833 2024-09-18T04:38:09.603 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.603 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.603 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:09.603 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.795414+0000", 2024-09-18T04:38:09.603 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00023088399999999999 2024-09-18T04:38:09.603 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.604 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.604 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:09.604 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798321+0000", 2024-09-18T04:38:09.604 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0029069539999999998 2024-09-18T04:38:09.604 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.604 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.604 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:09.604 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798404+0000", 2024-09-18T04:38:09.604 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2724000000000003e-05 2024-09-18T04:38:09.604 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.604 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.605 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:09.605 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798488+0000", 2024-09-18T04:38:09.605 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.3505999999999996e-05 2024-09-18T04:38:09.605 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.605 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.605 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:09.605 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798598+0000", 2024-09-18T04:38:09.605 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000109917 2024-09-18T04:38:09.605 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:09.605 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:09.605 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:09.605 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.605 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.606 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:09.606 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786613+0000", 2024-09-18T04:38:09.606 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 41.781127159999997, 2024-09-18T04:38:09.606 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014245760999999999, 2024-09-18T04:38:09.606 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:09.606 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:09.606 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:09.606 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:09.606 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:09.606 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 111 2024-09-18T04:38:09.606 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.606 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:09.606 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.607 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:09.607 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786613+0000", 2024-09-18T04:38:09.607 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:09.607 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.607 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.607 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:09.607 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786613+0000", 2024-09-18T04:38:09.607 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:09.607 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.607 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.607 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:09.607 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786612+0000", 2024-09-18T04:38:09.607 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:09.608 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.608 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.608 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:09.608 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786622+0000", 2024-09-18T04:38:09.608 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4099999999999997e-06 2024-09-18T04:38:09.608 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.608 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.608 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:09.608 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786630+0000", 2024-09-18T04:38:09.608 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4169999999999992e-06 2024-09-18T04:38:09.608 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.608 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.609 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:09.609 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786637+0000", 2024-09-18T04:38:09.609 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8290000000000002e-06 2024-09-18T04:38:09.609 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.609 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.609 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:09.609 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.797099+0000", 2024-09-18T04:38:09.609 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010461747 2024-09-18T04:38:09.609 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.609 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.609 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:09.609 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.797243+0000", 2024-09-18T04:38:09.609 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014388899999999999 2024-09-18T04:38:09.610 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.610 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.610 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:09.610 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.797463+0000", 2024-09-18T04:38:09.610 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000219912 2024-09-18T04:38:09.610 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.610 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.610 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:09.610 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800732+0000", 2024-09-18T04:38:09.610 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0032688719999999999 2024-09-18T04:38:09.610 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.610 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.610 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:09.611 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800751+0000", 2024-09-18T04:38:09.611 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9633999999999999e-05 2024-09-18T04:38:09.611 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.611 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.611 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:09.611 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800824+0000", 2024-09-18T04:38:09.611 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.2805999999999994e-05 2024-09-18T04:38:09.611 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.611 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.611 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:09.611 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800859+0000", 2024-09-18T04:38:09.611 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4952e-05 2024-09-18T04:38:09.611 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:09.612 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:09.612 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:09.612 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.612 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.612 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:09.612 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786649+0000", 2024-09-18T04:38:09.612 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 41.781091750999998, 2024-09-18T04:38:09.612 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.01613442, 2024-09-18T04:38:09.612 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:09.612 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:09.612 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:09.612 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:09.613 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:09.613 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 112 2024-09-18T04:38:09.613 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.613 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:09.613 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.613 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:09.613 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786649+0000", 2024-09-18T04:38:09.613 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:09.613 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.613 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.613 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:09.613 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786649+0000", 2024-09-18T04:38:09.613 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:09.614 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.614 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.614 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:09.614 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786648+0000", 2024-09-18T04:38:09.614 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:09.614 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.614 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.614 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:09.614 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786768+0000", 2024-09-18T04:38:09.614 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000120491 2024-09-18T04:38:09.614 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.614 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.614 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:09.615 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786795+0000", 2024-09-18T04:38:09.615 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6556000000000001e-05 2024-09-18T04:38:09.615 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.615 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.615 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:09.615 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786807+0000", 2024-09-18T04:38:09.615 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1739999999999999e-05 2024-09-18T04:38:09.615 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.615 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.615 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:09.615 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798683+0000", 2024-09-18T04:38:09.615 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011876070000000001 2024-09-18T04:38:09.615 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.616 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.616 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:09.616 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798867+0000", 2024-09-18T04:38:09.616 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000184103 2024-09-18T04:38:09.616 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.616 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.616 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:09.616 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.799082+0000", 2024-09-18T04:38:09.616 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00021476000000000001 2024-09-18T04:38:09.616 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.616 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.616 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:09.616 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802610+0000", 2024-09-18T04:38:09.616 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0035282209999999998 2024-09-18T04:38:09.617 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.617 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.617 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:09.617 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802706+0000", 2024-09-18T04:38:09.617 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5922999999999996e-05 2024-09-18T04:38:09.617 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.617 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.617 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:09.617 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802750+0000", 2024-09-18T04:38:09.617 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4511000000000001e-05 2024-09-18T04:38:09.617 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.617 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.617 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:09.617 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802783+0000", 2024-09-18T04:38:09.617 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2752000000000001e-05 2024-09-18T04:38:09.618 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:09.618 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:09.618 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:09.618 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.618 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.618 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:09.618 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786816+0000", 2024-09-18T04:38:09.618 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 41.780924683999999, 2024-09-18T04:38:09.618 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016197989999999999, 2024-09-18T04:38:09.618 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:09.618 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:09.618 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:09.618 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:09.619 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:09.619 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 113 2024-09-18T04:38:09.619 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.619 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:09.619 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.619 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:09.619 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786816+0000", 2024-09-18T04:38:09.619 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:09.619 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.619 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.619 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:09.619 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786816+0000", 2024-09-18T04:38:09.619 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:09.619 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.620 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.620 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:09.620 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786815+0000", 2024-09-18T04:38:09.620 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:09.620 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.620 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.620 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:09.620 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786821+0000", 2024-09-18T04:38:09.620 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9529999999999996e-06 2024-09-18T04:38:09.621 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.621 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.621 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:09.621 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786841+0000", 2024-09-18T04:38:09.621 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9743000000000001e-05 2024-09-18T04:38:09.621 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.621 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.621 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:09.621 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786848+0000", 2024-09-18T04:38:09.621 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4900000000000003e-06 2024-09-18T04:38:09.621 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.621 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.621 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:09.622 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.799747+0000", 2024-09-18T04:38:09.622 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012898902 2024-09-18T04:38:09.622 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.622 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.622 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:09.622 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.799898+0000", 2024-09-18T04:38:09.622 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000150877 2024-09-18T04:38:09.622 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.622 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.622 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:09.622 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800095+0000", 2024-09-18T04:38:09.622 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000197023 2024-09-18T04:38:09.622 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.623 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.623 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:09.623 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802829+0000", 2024-09-18T04:38:09.623 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0027343480000000002 2024-09-18T04:38:09.623 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.623 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.623 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:09.623 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802847+0000", 2024-09-18T04:38:09.623 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7283e-05 2024-09-18T04:38:09.623 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.623 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.623 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:09.623 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802974+0000", 2024-09-18T04:38:09.623 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012749199999999999 2024-09-18T04:38:09.624 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.624 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.624 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:09.624 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803014+0000", 2024-09-18T04:38:09.624 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9576000000000002e-05 2024-09-18T04:38:09.624 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:09.624 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:09.624 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:09.624 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.624 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.624 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:09.624 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:38:09.625 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 41.780839092999997, 2024-09-18T04:38:09.625 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017399218000000001, 2024-09-18T04:38:09.625 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:09.625 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:09.625 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:09.625 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:09.625 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:09.625 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 114 2024-09-18T04:38:09.625 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.625 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:09.625 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.626 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:09.626 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:38:09.626 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:09.626 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.626 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.626 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:09.626 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:38:09.626 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:09.626 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.626 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.626 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:09.626 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786900+0000", 2024-09-18T04:38:09.626 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:09.627 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.627 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.627 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:09.627 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786908+0000", 2024-09-18T04:38:09.627 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8199999999999997e-06 2024-09-18T04:38:09.627 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.627 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.627 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:09.627 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786925+0000", 2024-09-18T04:38:09.627 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6849000000000001e-05 2024-09-18T04:38:09.627 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.628 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.628 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:09.628 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786934+0000", 2024-09-18T04:38:09.628 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6299999999999993e-06 2024-09-18T04:38:09.628 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.628 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.628 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:09.628 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800926+0000", 2024-09-18T04:38:09.628 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013991632 2024-09-18T04:38:09.628 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.628 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.628 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:09.628 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801060+0000", 2024-09-18T04:38:09.629 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013378199999999999 2024-09-18T04:38:09.629 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.629 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.629 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:09.629 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801166+0000", 2024-09-18T04:38:09.629 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010607099999999999 2024-09-18T04:38:09.629 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.629 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.629 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:09.629 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803026+0000", 2024-09-18T04:38:09.629 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001860469 2024-09-18T04:38:09.629 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.629 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.630 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:09.630 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804245+0000", 2024-09-18T04:38:09.630 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0012185900000000001 2024-09-18T04:38:09.630 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.630 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.630 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:09.630 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804278+0000", 2024-09-18T04:38:09.630 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3272000000000003e-05 2024-09-18T04:38:09.630 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.630 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.630 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:09.630 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804300+0000", 2024-09-18T04:38:09.630 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2266e-05 2024-09-18T04:38:09.631 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:09.631 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:09.631 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:09.631 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.631 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.631 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:09.631 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787566+0000", 2024-09-18T04:38:09.631 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 41.780174133999999, 2024-09-18T04:38:09.631 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017124120999999999, 2024-09-18T04:38:09.631 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:09.631 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:09.631 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:09.632 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:09.632 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:09.632 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 115 2024-09-18T04:38:09.632 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.632 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:09.632 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.632 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:09.632 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787566+0000", 2024-09-18T04:38:09.632 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:09.632 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.632 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.632 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:09.632 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787566+0000", 2024-09-18T04:38:09.633 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:09.633 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.633 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.633 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:09.633 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787564+0000", 2024-09-18T04:38:09.633 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:09.633 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.633 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.633 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:09.633 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787751+0000", 2024-09-18T04:38:09.633 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000186873 2024-09-18T04:38:09.633 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.634 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.634 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:09.634 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787772+0000", 2024-09-18T04:38:09.634 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.126e-05 2024-09-18T04:38:09.634 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.634 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.634 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:09.634 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787794+0000", 2024-09-18T04:38:09.634 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1418999999999999e-05 2024-09-18T04:38:09.634 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.634 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.634 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:09.635 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801659+0000", 2024-09-18T04:38:09.635 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013865125000000001 2024-09-18T04:38:09.635 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.635 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.635 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:09.635 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801750+0000", 2024-09-18T04:38:09.635 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.1427000000000004e-05 2024-09-18T04:38:09.635 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.635 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.635 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:09.635 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801894+0000", 2024-09-18T04:38:09.636 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000143514 2024-09-18T04:38:09.636 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.636 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.636 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:09.636 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804313+0000", 2024-09-18T04:38:09.636 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0024190879999999998 2024-09-18T04:38:09.636 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.636 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.636 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:09.636 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804644+0000", 2024-09-18T04:38:09.636 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00033100799999999999 2024-09-18T04:38:09.637 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.637 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.637 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:09.637 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804672+0000", 2024-09-18T04:38:09.637 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7988e-05 2024-09-18T04:38:09.637 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.637 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.637 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:09.637 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804690+0000", 2024-09-18T04:38:09.637 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8298999999999999e-05 2024-09-18T04:38:09.638 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:09.638 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:09.638 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:09.638 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.638 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.638 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:09.638 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:38:09.638 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 41.779926195000002, 2024-09-18T04:38:09.638 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017521662, 2024-09-18T04:38:09.638 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:09.638 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:09.638 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:09.638 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:09.638 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:09.639 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 116 2024-09-18T04:38:09.639 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.639 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:09.639 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.639 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:09.639 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:38:09.639 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:09.639 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.639 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.639 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:09.639 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:38:09.639 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:09.639 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.640 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.640 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:09.640 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787813+0000", 2024-09-18T04:38:09.640 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:09.640 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.640 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.640 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:09.640 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787824+0000", 2024-09-18T04:38:09.640 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1157e-05 2024-09-18T04:38:09.640 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.640 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.640 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:09.641 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787840+0000", 2024-09-18T04:38:09.641 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6269e-05 2024-09-18T04:38:09.641 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.641 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.641 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:09.641 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787853+0000", 2024-09-18T04:38:09.641 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3156000000000001e-05 2024-09-18T04:38:09.641 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.641 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.641 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:09.641 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803110+0000", 2024-09-18T04:38:09.641 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015256736 2024-09-18T04:38:09.641 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.642 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.642 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:09.642 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803197+0000", 2024-09-18T04:38:09.642 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7195999999999997e-05 2024-09-18T04:38:09.642 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.642 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.642 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:09.642 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803308+0000", 2024-09-18T04:38:09.642 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111005 2024-09-18T04:38:09.642 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.642 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.642 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:09.642 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804316+0000", 2024-09-18T04:38:09.642 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010073949999999999 2024-09-18T04:38:09.643 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.643 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.643 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:09.643 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805232+0000", 2024-09-18T04:38:09.643 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091636300000000001 2024-09-18T04:38:09.643 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.643 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.643 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:09.643 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805277+0000", 2024-09-18T04:38:09.643 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4864000000000003e-05 2024-09-18T04:38:09.643 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.643 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.643 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:09.643 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805336+0000", 2024-09-18T04:38:09.644 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8968000000000002e-05 2024-09-18T04:38:09.644 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:09.644 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:09.644 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:09.644 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.644 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.644 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:09.644 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:38:09.644 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 41.779874139999997, 2024-09-18T04:38:09.644 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017983453, 2024-09-18T04:38:09.644 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:09.644 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:09.644 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:09.644 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:09.645 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:09.645 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 117 2024-09-18T04:38:09.645 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.645 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:09.645 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.645 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:09.645 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:38:09.645 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:09.645 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.645 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.645 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:09.645 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:38:09.645 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:09.645 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.646 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.646 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:09.646 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787865+0000", 2024-09-18T04:38:09.646 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:09.646 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.646 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.646 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:09.646 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787889+0000", 2024-09-18T04:38:09.646 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4244999999999999e-05 2024-09-18T04:38:09.646 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.646 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.646 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:09.647 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787905+0000", 2024-09-18T04:38:09.647 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5739999999999998e-05 2024-09-18T04:38:09.647 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.647 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.647 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:09.647 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787918+0000", 2024-09-18T04:38:09.647 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.252e-05 2024-09-18T04:38:09.647 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.647 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.647 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:09.647 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803693+0000", 2024-09-18T04:38:09.647 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015774872999999998 2024-09-18T04:38:09.647 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.647 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.648 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:09.648 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803775+0000", 2024-09-18T04:38:09.648 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2761000000000003e-05 2024-09-18T04:38:09.648 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.648 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.648 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:09.648 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803887+0000", 2024-09-18T04:38:09.648 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111708 2024-09-18T04:38:09.648 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.648 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.648 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:09.648 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804823+0000", 2024-09-18T04:38:09.649 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00093625100000000003 2024-09-18T04:38:09.649 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.649 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.649 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:09.649 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805804+0000", 2024-09-18T04:38:09.649 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00098114699999999992 2024-09-18T04:38:09.649 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.649 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.649 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:09.649 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805831+0000", 2024-09-18T04:38:09.649 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6401999999999999e-05 2024-09-18T04:38:09.649 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.649 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.650 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:09.650 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805850+0000", 2024-09-18T04:38:09.650 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8862e-05 2024-09-18T04:38:09.650 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:09.650 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:09.650 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:09.650 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.650 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.650 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:09.650 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.230830+0000", 2024-09-18T04:38:09.650 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 41.336910261, 2024-09-18T04:38:09.650 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011537146999999999, 2024-09-18T04:38:09.650 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:09.651 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:09.651 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:09.651 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:09.651 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:09.651 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 135 2024-09-18T04:38:09.651 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.651 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:09.651 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.651 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:09.651 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230830+0000", 2024-09-18T04:38:09.651 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:09.651 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.651 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.652 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:09.652 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230830+0000", 2024-09-18T04:38:09.652 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:09.652 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.652 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.652 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:09.652 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230829+0000", 2024-09-18T04:38:09.652 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:09.652 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.652 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.652 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:09.652 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230835+0000", 2024-09-18T04:38:09.652 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8660000000000002e-06 2024-09-18T04:38:09.653 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.653 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.653 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:09.653 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230844+0000", 2024-09-18T04:38:09.653 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9339999999999995e-06 2024-09-18T04:38:09.653 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.653 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.653 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:09.653 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230853+0000", 2024-09-18T04:38:09.653 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4419999999999998e-06 2024-09-18T04:38:09.653 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.653 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.653 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:09.654 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.237658+0000", 2024-09-18T04:38:09.654 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0068055310000000001 2024-09-18T04:38:09.654 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.654 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.654 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:09.654 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.237837+0000", 2024-09-18T04:38:09.654 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017943200000000001 2024-09-18T04:38:09.654 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.654 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.654 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:09.654 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.238112+0000", 2024-09-18T04:38:09.654 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00027488600000000001 2024-09-18T04:38:09.655 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.655 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.655 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:09.655 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.240505+0000", 2024-09-18T04:38:09.655 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0023930539999999999 2024-09-18T04:38:09.655 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.655 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.655 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:09.655 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242157+0000", 2024-09-18T04:38:09.655 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001651135 2024-09-18T04:38:09.655 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.655 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.655 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:09.655 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242281+0000", 2024-09-18T04:38:09.656 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012451300000000001 2024-09-18T04:38:09.656 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.656 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.656 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:09.656 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242367+0000", 2024-09-18T04:38:09.656 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6178000000000001e-05 2024-09-18T04:38:09.656 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:09.656 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:09.656 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:09.656 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.656 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.657 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:09.657 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.230858+0000", 2024-09-18T04:38:09.657 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 41.336882373000002, 2024-09-18T04:38:09.657 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013005821000000001, 2024-09-18T04:38:09.657 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:09.657 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:09.657 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:09.657 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:09.657 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:09.658 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 136 2024-09-18T04:38:09.658 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.658 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:09.658 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.658 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:09.658 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230858+0000", 2024-09-18T04:38:09.658 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:09.658 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.658 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.658 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:09.658 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230858+0000", 2024-09-18T04:38:09.658 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:09.658 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.659 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.659 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:09.659 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230858+0000", 2024-09-18T04:38:09.659 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:09.659 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.659 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.659 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:09.659 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230861+0000", 2024-09-18T04:38:09.659 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7459999999999998e-06 2024-09-18T04:38:09.659 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.659 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.659 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:09.659 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230877+0000", 2024-09-18T04:38:09.659 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5979000000000001e-05 2024-09-18T04:38:09.659 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.660 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.660 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:09.660 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230885+0000", 2024-09-18T04:38:09.660 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.5979999999999999e-06 2024-09-18T04:38:09.660 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.660 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.660 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:09.660 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.239023+0000", 2024-09-18T04:38:09.660 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0081380710000000002 2024-09-18T04:38:09.660 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.660 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.660 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:09.660 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.239216+0000", 2024-09-18T04:38:09.660 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019274799999999999 2024-09-18T04:38:09.661 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.661 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.661 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:09.661 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.239417+0000", 2024-09-18T04:38:09.661 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020093300000000001 2024-09-18T04:38:09.661 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.661 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.661 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:09.661 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242494+0000", 2024-09-18T04:38:09.661 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003077177 2024-09-18T04:38:09.661 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.662 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.662 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:09.662 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.243668+0000", 2024-09-18T04:38:09.662 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001174027 2024-09-18T04:38:09.662 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.662 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.662 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:09.662 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.243777+0000", 2024-09-18T04:38:09.662 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010955600000000001 2024-09-18T04:38:09.662 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.662 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.662 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:09.662 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.243864+0000", 2024-09-18T04:38:09.662 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6441999999999997e-05 2024-09-18T04:38:09.663 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:09.663 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:09.663 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:09.663 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.663 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.663 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:09.663 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.230901+0000", 2024-09-18T04:38:09.663 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 41.3368398, 2024-09-18T04:38:09.663 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014120882, 2024-09-18T04:38:09.663 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:09.663 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:09.663 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:09.663 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:09.663 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:09.664 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 137 2024-09-18T04:38:09.664 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.664 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:09.664 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.664 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:09.664 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230901+0000", 2024-09-18T04:38:09.664 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:09.664 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.664 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.664 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:09.664 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230901+0000", 2024-09-18T04:38:09.664 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:09.664 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.664 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.664 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:09.665 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230900+0000", 2024-09-18T04:38:09.665 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:09.665 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.665 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.665 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:09.665 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231002+0000", 2024-09-18T04:38:09.665 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000102392 2024-09-18T04:38:09.665 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.665 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.665 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:09.665 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231029+0000", 2024-09-18T04:38:09.666 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7166999999999999e-05 2024-09-18T04:38:09.666 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.666 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.666 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:09.666 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231036+0000", 2024-09-18T04:38:09.666 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7259999999999996e-06 2024-09-18T04:38:09.666 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.666 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.666 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:09.666 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.240562+0000", 2024-09-18T04:38:09.666 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0095257970000000008 2024-09-18T04:38:09.666 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.666 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.667 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:09.667 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.240730+0000", 2024-09-18T04:38:09.667 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016767099999999999 2024-09-18T04:38:09.667 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.667 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.667 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:09.667 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.240927+0000", 2024-09-18T04:38:09.667 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019711899999999999 2024-09-18T04:38:09.667 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.667 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.667 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:09.668 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244885+0000", 2024-09-18T04:38:09.668 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0039577340000000001 2024-09-18T04:38:09.668 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.668 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.668 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:09.668 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244899+0000", 2024-09-18T04:38:09.668 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4403000000000001e-05 2024-09-18T04:38:09.668 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.668 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.668 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:09.668 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244973+0000", 2024-09-18T04:38:09.668 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4177999999999994e-05 2024-09-18T04:38:09.669 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.669 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.669 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:09.669 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.245021+0000", 2024-09-18T04:38:09.669 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8344999999999999e-05 2024-09-18T04:38:09.669 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:09.669 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:09.669 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:09.669 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.669 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.669 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:09.670 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:38:09.670 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 41.336697008999998, 2024-09-18T04:38:09.670 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016528505999999998, 2024-09-18T04:38:09.670 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:09.670 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:09.670 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:09.670 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:09.670 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:09.670 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 138 2024-09-18T04:38:09.670 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.670 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:09.670 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.670 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:09.670 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:38:09.671 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:09.671 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.671 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.671 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:09.671 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:38:09.671 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:09.671 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.671 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.671 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:09.671 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:38:09.671 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:09.672 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.672 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.672 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:09.672 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231047+0000", 2024-09-18T04:38:09.672 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5399999999999997e-06 2024-09-18T04:38:09.672 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.672 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.672 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:09.672 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231068+0000", 2024-09-18T04:38:09.672 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0769000000000001e-05 2024-09-18T04:38:09.672 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.673 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.673 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:09.673 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231074+0000", 2024-09-18T04:38:09.673 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6180000000000003e-06 2024-09-18T04:38:09.673 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.673 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.673 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:09.673 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242556+0000", 2024-09-18T04:38:09.673 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011482654 2024-09-18T04:38:09.673 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.673 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.673 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:09.673 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242666+0000", 2024-09-18T04:38:09.674 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011003200000000001 2024-09-18T04:38:09.674 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.674 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.674 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:09.674 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242924+0000", 2024-09-18T04:38:09.674 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00025785899999999999 2024-09-18T04:38:09.674 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.674 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.674 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:09.674 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247263+0000", 2024-09-18T04:38:09.674 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0043387649999999996 2024-09-18T04:38:09.674 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.674 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.675 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:09.675 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247412+0000", 2024-09-18T04:38:09.675 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000149276 2024-09-18T04:38:09.675 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.675 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.675 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:09.675 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247522+0000", 2024-09-18T04:38:09.675 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000109788 2024-09-18T04:38:09.675 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.675 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.675 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:09.675 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247572+0000", 2024-09-18T04:38:09.675 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9808999999999997e-05 2024-09-18T04:38:09.675 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:09.676 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:09.676 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:09.676 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.676 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.676 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:09.676 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231083+0000", 2024-09-18T04:38:09.676 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 41.336657545999998, 2024-09-18T04:38:09.676 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0166731, 2024-09-18T04:38:09.676 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:09.676 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:09.676 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:09.676 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:09.676 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:09.677 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 139 2024-09-18T04:38:09.677 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.677 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:09.677 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.677 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:09.677 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231083+0000", 2024-09-18T04:38:09.677 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:09.677 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.677 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.677 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:09.677 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231083+0000", 2024-09-18T04:38:09.677 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:09.677 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.677 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.678 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:09.678 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231082+0000", 2024-09-18T04:38:09.678 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:09.678 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.678 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.678 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:09.678 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231090+0000", 2024-09-18T04:38:09.678 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1280000000000008e-06 2024-09-18T04:38:09.678 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.678 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.678 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:09.679 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231097+0000", 2024-09-18T04:38:09.679 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5119999999999997e-06 2024-09-18T04:38:09.679 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.679 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.679 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:09.679 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231106+0000", 2024-09-18T04:38:09.679 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9230000000000008e-06 2024-09-18T04:38:09.679 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.679 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.679 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:09.679 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.243977+0000", 2024-09-18T04:38:09.679 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012870754 2024-09-18T04:38:09.679 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.680 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.680 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:09.680 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244112+0000", 2024-09-18T04:38:09.680 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000135575 2024-09-18T04:38:09.680 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.680 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.680 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:09.680 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244265+0000", 2024-09-18T04:38:09.680 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015279900000000001 2024-09-18T04:38:09.680 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.680 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.680 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:09.681 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247649+0000", 2024-09-18T04:38:09.681 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003383896 2024-09-18T04:38:09.681 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.681 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.681 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:09.681 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247656+0000", 2024-09-18T04:38:09.681 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8920000000000003e-06 2024-09-18T04:38:09.681 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.681 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.681 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:09.681 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247692+0000", 2024-09-18T04:38:09.681 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6470000000000001e-05 2024-09-18T04:38:09.681 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.682 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.682 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:09.682 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247756+0000", 2024-09-18T04:38:09.682 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3676999999999994e-05 2024-09-18T04:38:09.682 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:09.682 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:09.682 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:09.682 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.682 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.682 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:09.682 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231190+0000", 2024-09-18T04:38:09.682 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 41.336550619999997, 2024-09-18T04:38:09.682 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016793556000000001, 2024-09-18T04:38:09.683 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:09.683 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:09.683 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:09.683 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:09.683 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:09.683 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 140 2024-09-18T04:38:09.683 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.683 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:09.683 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.683 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:09.683 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231190+0000", 2024-09-18T04:38:09.683 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:09.683 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.684 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.684 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:09.684 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231190+0000", 2024-09-18T04:38:09.684 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:09.684 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.684 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.684 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:09.684 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231189+0000", 2024-09-18T04:38:09.684 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:09.684 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.684 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.684 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:09.684 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231273+0000", 2024-09-18T04:38:09.685 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4445999999999998e-05 2024-09-18T04:38:09.685 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.685 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.685 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:09.685 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231282+0000", 2024-09-18T04:38:09.685 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8859999999999993e-06 2024-09-18T04:38:09.685 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.685 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.685 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:09.685 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231288+0000", 2024-09-18T04:38:09.685 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9819999999999996e-06 2024-09-18T04:38:09.685 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.686 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.686 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:09.686 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.245119+0000", 2024-09-18T04:38:09.686 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.01383121 2024-09-18T04:38:09.686 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.686 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.686 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:09.686 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.245247+0000", 2024-09-18T04:38:09.686 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012763299999999999 2024-09-18T04:38:09.686 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.686 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.686 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:09.686 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.245401+0000", 2024-09-18T04:38:09.687 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015399200000000001 2024-09-18T04:38:09.687 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.687 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.687 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:09.687 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247766+0000", 2024-09-18T04:38:09.687 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0023649510000000001 2024-09-18T04:38:09.687 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.687 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.687 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:09.687 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247904+0000", 2024-09-18T04:38:09.687 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013836499999999999 2024-09-18T04:38:09.687 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.687 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.687 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:09.688 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247950+0000", 2024-09-18T04:38:09.688 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5201000000000003e-05 2024-09-18T04:38:09.688 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.688 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.688 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:09.688 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247983+0000", 2024-09-18T04:38:09.688 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3686000000000002e-05 2024-09-18T04:38:09.688 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:09.688 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:09.688 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:09.688 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.688 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.688 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:09.689 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:38:09.689 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 41.336439876999997, 2024-09-18T04:38:09.689 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017344024, 2024-09-18T04:38:09.689 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:09.689 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:09.689 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:09.689 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:09.689 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:09.689 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 141 2024-09-18T04:38:09.689 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.689 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:09.689 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.689 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:09.690 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:38:09.690 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:09.690 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.690 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.690 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:09.690 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:38:09.690 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:09.690 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.690 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.690 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:09.690 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:38:09.690 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:09.690 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.691 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.691 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:09.691 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231309+0000", 2024-09-18T04:38:09.691 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5419999999999994e-06 2024-09-18T04:38:09.691 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.691 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.691 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:09.691 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231323+0000", 2024-09-18T04:38:09.691 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3108000000000001e-05 2024-09-18T04:38:09.691 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.691 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.691 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:09.691 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231340+0000", 2024-09-18T04:38:09.691 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.696e-05 2024-09-18T04:38:09.692 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.692 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.692 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:09.692 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.246143+0000", 2024-09-18T04:38:09.692 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014803320999999999 2024-09-18T04:38:09.692 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.692 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.692 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:09.692 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.246323+0000", 2024-09-18T04:38:09.692 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00018061600000000001 2024-09-18T04:38:09.692 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.692 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.693 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:09.693 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.246530+0000", 2024-09-18T04:38:09.693 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020625 2024-09-18T04:38:09.693 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.693 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.693 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:09.693 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248002+0000", 2024-09-18T04:38:09.693 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0014719379999999999 2024-09-18T04:38:09.693 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.693 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.693 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:09.693 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248582+0000", 2024-09-18T04:38:09.693 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00058020599999999999 2024-09-18T04:38:09.693 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.694 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.694 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:09.694 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248618+0000", 2024-09-18T04:38:09.694 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6559e-05 2024-09-18T04:38:09.694 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.694 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.694 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:09.694 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248645+0000", 2024-09-18T04:38:09.694 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6080000000000001e-05 2024-09-18T04:38:09.694 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:09.694 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:09.694 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:09.694 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.695 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.695 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-18T04:38:09.695 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.878463+0000", 2024-09-18T04:38:09.695 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 40.689277842999999, 2024-09-18T04:38:09.695 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010882697, 2024-09-18T04:38:09.695 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:09.695 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:09.695 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:09.695 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:09.695 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:09.695 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 159 2024-09-18T04:38:09.695 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.695 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:09.696 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.696 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:09.696 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878463+0000", 2024-09-18T04:38:09.696 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:09.696 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.696 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.696 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:09.696 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878463+0000", 2024-09-18T04:38:09.696 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:09.696 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.696 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.696 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:09.696 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878462+0000", 2024-09-18T04:38:09.696 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:09.697 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.697 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.697 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:09.697 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878467+0000", 2024-09-18T04:38:09.697 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4099999999999999e-06 2024-09-18T04:38:09.697 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.697 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.697 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:09.697 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878490+0000", 2024-09-18T04:38:09.697 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3159999999999998e-05 2024-09-18T04:38:09.697 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.697 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.697 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:09.698 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878511+0000", 2024-09-18T04:38:09.698 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1092000000000001e-05 2024-09-18T04:38:09.698 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.698 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.698 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:09.698 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.887163+0000", 2024-09-18T04:38:09.698 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.008651845 2024-09-18T04:38:09.698 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.698 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.698 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:09.698 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.887273+0000", 2024-09-18T04:38:09.698 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000109823 2024-09-18T04:38:09.698 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.699 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.699 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:09.699 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.887441+0000", 2024-09-18T04:38:09.699 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016777599999999999 2024-09-18T04:38:09.699 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.699 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.699 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:09.699 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.888634+0000", 2024-09-18T04:38:09.699 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001192839 2024-09-18T04:38:09.699 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.699 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.699 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:09.699 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.889153+0000", 2024-09-18T04:38:09.699 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00051953700000000004 2024-09-18T04:38:09.700 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.700 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.700 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:09.700 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.889234+0000", 2024-09-18T04:38:09.700 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0802000000000003e-05 2024-09-18T04:38:09.700 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:09.700 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:09.700 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:09.700 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.889345+0000", 2024-09-18T04:38:09.700 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011112000000000001 2024-09-18T04:38:09.700 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:09.700 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:09.700 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:09.701 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:09.701 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:09.701 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:09.702 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:38:09.703 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:09.699+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:09.763 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-18T04:38:09.763 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T04:38:09.763 INFO:tasks.thrashosds.thrasher:Reweighting osd 4 to 0.3929325963127023 2024-09-18T04:38:09.764 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 0.3929325963127023 2024-09-18T04:38:09.802 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:09.798+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:09.816 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:09.817 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:09.817 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:38:09.817 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:09.826 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:38:09.903 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:09.898+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:09.979 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:09.979 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:09.979 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:38:09.979 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:38:09.979 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:09.989 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:38:10.038 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:10.030+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:10.093 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:10.093 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:38:10.093 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:38:10.093 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [] 2024-09-18T04:38:10.093 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:10.103 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:38:10.104 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:10.098+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:10.205 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:10.198+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:10.208 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:10.208 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:10.208 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:38:10.209 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:10.218 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:38:10.306 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:10.303+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:10.325 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:10.325 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:10.325 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:38:10.325 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:38:10.325 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:10.334 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:38:10.407 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:10.403+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:10.445 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:10.445 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:38:10.445 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:38:10.446 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:38:10.446 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.446 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-18T04:38:10.446 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:38:10.446 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 35.970884462000001, 2024-09-18T04:38:10.446 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0095781189999999995, 2024-09-18T04:38:10.446 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:10.446 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:10.446 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:10.446 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:38:10.446 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:38:10.446 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 1 2024-09-18T04:38:10.446 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.446 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:10.447 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.447 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:10.447 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:38:10.447 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:10.447 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.447 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.447 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:10.447 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:38:10.447 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:10.447 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.447 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.447 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:10.447 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469862+0000", 2024-09-18T04:38:10.447 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:10.448 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.448 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.448 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:10.448 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469879+0000", 2024-09-18T04:38:10.448 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7431999999999998e-05 2024-09-18T04:38:10.448 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.448 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.448 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:10.448 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469896+0000", 2024-09-18T04:38:10.448 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6742999999999999e-05 2024-09-18T04:38:10.448 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.448 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.448 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:10.449 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469915+0000", 2024-09-18T04:38:10.449 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8991000000000002e-05 2024-09-18T04:38:10.449 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.449 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.449 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:10.449 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470397+0000", 2024-09-18T04:38:10.449 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048167799999999998 2024-09-18T04:38:10.449 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.449 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.449 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-09-18T04:38:10.449 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470412+0000", 2024-09-18T04:38:10.449 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5526e-05 2024-09-18T04:38:10.449 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.450 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.450 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:10.450 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473279+0000", 2024-09-18T04:38:10.450 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0028667469999999998 2024-09-18T04:38:10.450 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.450 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.450 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:10.450 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473441+0000", 2024-09-18T04:38:10.450 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000162082 2024-09-18T04:38:10.450 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.450 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.450 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:38:10.450 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473573+0000", 2024-09-18T04:38:10.451 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000132032 2024-09-18T04:38:10.451 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.451 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.451 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:10.451 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474826+0000", 2024-09-18T04:38:10.451 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001252717 2024-09-18T04:38:10.451 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.451 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.451 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:10.451 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479307+0000", 2024-09-18T04:38:10.451 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044813220000000003 2024-09-18T04:38:10.451 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.451 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.451 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:10.451 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479366+0000", 2024-09-18T04:38:10.452 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9024999999999998e-05 2024-09-18T04:38:10.452 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.452 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.452 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:10.452 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479443+0000", 2024-09-18T04:38:10.452 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6339000000000002e-05 2024-09-18T04:38:10.452 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:10.452 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:10.452 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:10.452 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.452 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.452 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.4214.0:1 2.3 e18/17 ondisk, result = 0)", 2024-09-18T04:38:10.452 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.474724+0000", 2024-09-18T04:38:10.452 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 35.966025098000003, 2024-09-18T04:38:10.452 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000105358, 2024-09-18T04:38:10.453 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:10.453 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:38:10.453 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:10.453 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.453 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:10.453 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474724+0000", 2024-09-18T04:38:10.453 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:10.453 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.453 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.453 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:10.453 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474724+0000", 2024-09-18T04:38:10.453 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:10.453 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.453 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.453 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:10.454 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474723+0000", 2024-09-18T04:38:10.454 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:10.454 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.454 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.454 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:10.454 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474731+0000", 2024-09-18T04:38:10.454 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2339999999999994e-06 2024-09-18T04:38:10.454 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.454 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.454 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:10.454 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474748+0000", 2024-09-18T04:38:10.454 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6603e-05 2024-09-18T04:38:10.454 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.454 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.455 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:10.455 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474760+0000", 2024-09-18T04:38:10.455 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2215000000000001e-05 2024-09-18T04:38:10.455 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.455 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.455 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:10.455 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474814+0000", 2024-09-18T04:38:10.455 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4267000000000001e-05 2024-09-18T04:38:10.455 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.455 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.455 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:10.455 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474821+0000", 2024-09-18T04:38:10.455 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0929999999999997e-06 2024-09-18T04:38:10.455 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.455 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.456 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:10.456 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474829+0000", 2024-09-18T04:38:10.456 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1200000000000002e-06 2024-09-18T04:38:10.456 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:10.456 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:10.456 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:10.456 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.456 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.456 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18)", 2024-09-18T04:38:10.456 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.482346+0000", 2024-09-18T04:38:10.456 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 35.958402665999998, 2024-09-18T04:38:10.456 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020435599999999999, 2024-09-18T04:38:10.456 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:10.456 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-09-18T04:38:10.457 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:10.457 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:38:10.457 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:38:10.457 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 2 2024-09-18T04:38:10.457 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.457 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:10.457 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.457 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:10.457 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482346+0000", 2024-09-18T04:38:10.457 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:10.457 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.457 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.457 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:10.457 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482346+0000", 2024-09-18T04:38:10.457 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:10.458 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.458 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.458 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:10.458 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482345+0000", 2024-09-18T04:38:10.458 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:10.458 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.458 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.458 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:10.458 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482360+0000", 2024-09-18T04:38:10.458 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4854999999999999e-05 2024-09-18T04:38:10.458 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.458 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.458 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:10.458 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482377+0000", 2024-09-18T04:38:10.459 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7785999999999999e-05 2024-09-18T04:38:10.459 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.459 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.459 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:10.459 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482387+0000", 2024-09-18T04:38:10.459 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6800000000000005e-06 2024-09-18T04:38:10.459 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.459 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.459 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:10.459 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482441+0000", 2024-09-18T04:38:10.459 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4046999999999999e-05 2024-09-18T04:38:10.459 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.459 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.459 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:10.460 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482551+0000", 2024-09-18T04:38:10.460 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000109468 2024-09-18T04:38:10.460 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:10.460 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:10.460 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:10.460 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.460 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.460 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18)", 2024-09-18T04:38:10.460 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.482853+0000", 2024-09-18T04:38:10.460 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 35.957896048000002, 2024-09-18T04:38:10.460 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00029815799999999997, 2024-09-18T04:38:10.460 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:10.460 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-09-18T04:38:10.460 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:10.460 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:38:10.461 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:38:10.461 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 3 2024-09-18T04:38:10.461 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.461 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:10.461 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.461 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:10.461 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482853+0000", 2024-09-18T04:38:10.461 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:10.461 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.461 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.461 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:10.461 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482853+0000", 2024-09-18T04:38:10.461 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:10.461 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.461 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.462 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:10.462 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482852+0000", 2024-09-18T04:38:10.462 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:10.462 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.462 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.462 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:10.462 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482859+0000", 2024-09-18T04:38:10.462 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0380000000000002e-06 2024-09-18T04:38:10.462 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.462 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.462 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:10.462 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482880+0000", 2024-09-18T04:38:10.462 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1243e-05 2024-09-18T04:38:10.462 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.462 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.463 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:10.463 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482896+0000", 2024-09-18T04:38:10.463 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5977e-05 2024-09-18T04:38:10.463 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.463 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.463 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:10.463 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482970+0000", 2024-09-18T04:38:10.463 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4603000000000004e-05 2024-09-18T04:38:10.463 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.463 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.463 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:10.463 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.483151+0000", 2024-09-18T04:38:10.463 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00018059799999999999 2024-09-18T04:38:10.463 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:10.463 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:10.463 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:10.464 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.464 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.464 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-18T04:38:10.464 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:35.468698+0000", 2024-09-18T04:38:10.464 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 34.972051211, 2024-09-18T04:38:10.464 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0039954129999999997, 2024-09-18T04:38:10.464 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:10.464 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:10.464 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:10.464 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:38:10.464 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:38:10.464 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 5 2024-09-18T04:38:10.464 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.464 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:10.464 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.465 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:10.465 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468698+0000", 2024-09-18T04:38:10.465 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:10.465 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.465 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.465 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:10.465 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468698+0000", 2024-09-18T04:38:10.465 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:10.465 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.465 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.465 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:10.465 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468696+0000", 2024-09-18T04:38:10.465 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:10.465 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.465 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.466 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:10.466 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468714+0000", 2024-09-18T04:38:10.466 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8743e-05 2024-09-18T04:38:10.466 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.466 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.466 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:10.466 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468748+0000", 2024-09-18T04:38:10.466 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3339e-05 2024-09-18T04:38:10.466 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.466 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.466 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:10.466 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468771+0000", 2024-09-18T04:38:10.466 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3119999999999999e-05 2024-09-18T04:38:10.467 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.467 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.467 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:10.467 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468922+0000", 2024-09-18T04:38:10.467 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000151383 2024-09-18T04:38:10.467 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.467 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.467 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:10.467 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.469278+0000", 2024-09-18T04:38:10.467 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00035628599999999998 2024-09-18T04:38:10.467 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.467 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.467 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:10.468 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.469629+0000", 2024-09-18T04:38:10.468 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00035014300000000003 2024-09-18T04:38:10.468 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.468 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.468 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:10.468 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.471620+0000", 2024-09-18T04:38:10.468 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001991278 2024-09-18T04:38:10.468 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.468 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.468 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:10.468 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472440+0000", 2024-09-18T04:38:10.468 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00081998599999999998 2024-09-18T04:38:10.468 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.469 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.469 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:10.469 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472592+0000", 2024-09-18T04:38:10.469 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015227400000000001 2024-09-18T04:38:10.469 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.469 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.469 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:10.469 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472693+0000", 2024-09-18T04:38:10.469 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000101074 2024-09-18T04:38:10.469 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:10.469 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:10.469 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:10.470 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.470 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.470 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.4214.0:5 2.2 e18/17 ondisk, result = 0)", 2024-09-18T04:38:10.470 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:35.472226+0000", 2024-09-18T04:38:10.470 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 34.968523310999998, 2024-09-18T04:38:10.470 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048505299999999999, 2024-09-18T04:38:10.470 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:10.470 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:38:10.470 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:10.470 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.470 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:10.470 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472226+0000", 2024-09-18T04:38:10.470 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:10.471 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.471 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.471 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:10.471 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472226+0000", 2024-09-18T04:38:10.471 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:10.471 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.471 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.471 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:10.471 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472224+0000", 2024-09-18T04:38:10.471 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:10.471 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.471 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.471 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:10.471 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472238+0000", 2024-09-18T04:38:10.472 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.468e-05 2024-09-18T04:38:10.472 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.472 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.472 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:10.472 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472272+0000", 2024-09-18T04:38:10.472 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3586e-05 2024-09-18T04:38:10.472 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.472 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.472 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:10.472 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472297+0000", 2024-09-18T04:38:10.472 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5282999999999998e-05 2024-09-18T04:38:10.472 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.472 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.472 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:10.473 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472396+0000", 2024-09-18T04:38:10.473 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.8394999999999996e-05 2024-09-18T04:38:10.473 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.473 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.473 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:10.473 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472421+0000", 2024-09-18T04:38:10.473 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5479000000000001e-05 2024-09-18T04:38:10.473 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.473 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.473 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:10.473 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472711+0000", 2024-09-18T04:38:10.473 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000289714 2024-09-18T04:38:10.473 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:10.474 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:10.474 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:10.474 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.474 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.474 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-18T04:38:10.474 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:38:10.474 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 33.957876042000002, 2024-09-18T04:38:10.474 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082699560000000002, 2024-09-18T04:38:10.474 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:10.474 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:10.474 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:10.474 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:38:10.474 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:38:10.474 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 7 2024-09-18T04:38:10.475 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.475 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:10.475 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.475 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:10.475 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:38:10.475 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:10.475 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.475 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.475 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:10.475 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:38:10.475 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:10.475 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.475 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.475 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:10.475 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482870+0000", 2024-09-18T04:38:10.476 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999967 2024-09-18T04:38:10.476 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.476 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.476 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:10.476 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482889+0000", 2024-09-18T04:38:10.476 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.965e-05 2024-09-18T04:38:10.476 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.476 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.476 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:10.476 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482938+0000", 2024-09-18T04:38:10.476 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8548999999999999e-05 2024-09-18T04:38:10.476 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.476 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.477 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:10.477 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482963+0000", 2024-09-18T04:38:10.477 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5075999999999998e-05 2024-09-18T04:38:10.477 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.477 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.477 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:10.477 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483121+0000", 2024-09-18T04:38:10.477 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015819400000000001 2024-09-18T04:38:10.477 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.477 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.477 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op must wait for map", 2024-09-18T04:38:10.477 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483138+0000", 2024-09-18T04:38:10.477 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7068999999999999e-05 2024-09-18T04:38:10.477 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.478 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.478 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:10.478 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488666+0000", 2024-09-18T04:38:10.478 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0055282049999999996 2024-09-18T04:38:10.478 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.478 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.478 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:10.478 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488783+0000", 2024-09-18T04:38:10.478 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000116737 2024-09-18T04:38:10.478 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.478 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.478 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:10.478 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488991+0000", 2024-09-18T04:38:10.478 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020781400000000001 2024-09-18T04:38:10.478 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.479 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.479 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:10.479 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490554+0000", 2024-09-18T04:38:10.479 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0015628370000000001 2024-09-18T04:38:10.479 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.479 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.479 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:10.479 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491019+0000", 2024-09-18T04:38:10.479 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000465608 2024-09-18T04:38:10.479 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.479 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.479 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:10.479 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491087+0000", 2024-09-18T04:38:10.479 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7988e-05 2024-09-18T04:38:10.479 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.480 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.480 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:10.480 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491143+0000", 2024-09-18T04:38:10.480 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5538999999999998e-05 2024-09-18T04:38:10.480 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:10.480 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:10.480 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:10.480 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.480 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.480 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.4214.0:7 2.2 e19/17 ondisk, result = 0)", 2024-09-18T04:38:10.480 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:36.490861+0000", 2024-09-18T04:38:10.480 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 33.949888420000001, 2024-09-18T04:38:10.480 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00029639400000000001, 2024-09-18T04:38:10.480 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:10.481 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:38:10.481 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:10.481 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.481 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:10.481 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490861+0000", 2024-09-18T04:38:10.481 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:10.481 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.481 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.481 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:10.481 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490861+0000", 2024-09-18T04:38:10.481 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:10.481 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.481 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.481 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:10.482 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490859+0000", 2024-09-18T04:38:10.482 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:10.482 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.482 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.482 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:10.482 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490867+0000", 2024-09-18T04:38:10.482 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.3089999999999996e-06 2024-09-18T04:38:10.482 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.482 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.482 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:10.482 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490909+0000", 2024-09-18T04:38:10.482 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1919000000000003e-05 2024-09-18T04:38:10.482 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.482 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.482 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:10.483 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490922+0000", 2024-09-18T04:38:10.483 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2836999999999999e-05 2024-09-18T04:38:10.483 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.483 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.483 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:10.483 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491003+0000", 2024-09-18T04:38:10.483 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0681999999999998e-05 2024-09-18T04:38:10.483 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.483 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.483 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:10.483 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491013+0000", 2024-09-18T04:38:10.483 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0220999999999999e-05 2024-09-18T04:38:10.483 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.483 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.483 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:10.483 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491157+0000", 2024-09-18T04:38:10.484 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014401799999999999 2024-09-18T04:38:10.484 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:10.484 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:10.484 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:10.484 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.484 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.484 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "MOSDPGPushReply(2.3 25/24 [PushReplyOp(2:c4c92e5a:::rbd_trash:head)])", 2024-09-18T04:38:10.484 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:51.680323+0000", 2024-09-18T04:38:10.484 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 18.760426044999999, 2024-09-18T04:38:10.484 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00054411499999999996, 2024-09-18T04:38:10.484 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:10.484 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-09-18T04:38:10.484 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:10.484 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.484 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:10.485 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680323+0000", 2024-09-18T04:38:10.485 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:10.485 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.485 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.485 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:10.485 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680323+0000", 2024-09-18T04:38:10.485 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:10.485 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.485 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.485 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:10.485 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680322+0000", 2024-09-18T04:38:10.485 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:10.485 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.485 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.485 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:10.485 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680333+0000", 2024-09-18T04:38:10.486 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1153e-05 2024-09-18T04:38:10.486 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.486 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.486 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:10.486 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680357+0000", 2024-09-18T04:38:10.486 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3252999999999999e-05 2024-09-18T04:38:10.486 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.486 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.486 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:10.486 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680378+0000", 2024-09-18T04:38:10.486 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1503000000000001e-05 2024-09-18T04:38:10.486 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.486 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.486 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:10.486 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680522+0000", 2024-09-18T04:38:10.486 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014410700000000001 2024-09-18T04:38:10.487 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:10.487 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:10.487 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:10.487 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680867+0000", 2024-09-18T04:38:10.487 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00034475899999999998 2024-09-18T04:38:10.487 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:10.487 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:10.487 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:10.487 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:10.487 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:10.487 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:10.488 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:38:10.534 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:10.531+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:10.605 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:10.606 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:10.606 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:38:10.606 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:10.609 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:10.602+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:10.616 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:38:10.710 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:10.706+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:10.724 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:10.724 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:10.724 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:38:10.724 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:38:10.724 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:10.734 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:38:10.772 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.4 to 0.392933 (6497) 2024-09-18T04:38:10.775 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-18T04:38:10.811 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:10.806+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:10.854 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:10.854 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:38:10.854 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:38:10.854 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:38:10.854 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.854 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:30 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'16, mlcod=15'14)", 2024-09-18T04:38:10.854 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.486330+0000", 2024-09-18T04:38:10.854 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 43.360714915000003, 2024-09-18T04:38:10.854 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026383700000000001, 2024-09-18T04:38:10.854 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:10.854 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:10.854 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:10.855 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.855 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:10.855 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486330+0000", 2024-09-18T04:38:10.855 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:10.855 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.855 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.855 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:10.855 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486330+0000", 2024-09-18T04:38:10.855 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:10.855 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.855 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.855 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:10.855 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486328+0000", 2024-09-18T04:38:10.855 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:10.856 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.856 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.856 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:10.856 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486342+0000", 2024-09-18T04:38:10.856 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3783e-05 2024-09-18T04:38:10.856 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.856 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.856 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:10.856 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486373+0000", 2024-09-18T04:38:10.856 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1362999999999998e-05 2024-09-18T04:38:10.856 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.856 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.856 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:10.856 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486387+0000", 2024-09-18T04:38:10.856 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3219e-05 2024-09-18T04:38:10.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.857 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.857 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:10.857 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486489+0000", 2024-09-18T04:38:10.857 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010232499999999999 2024-09-18T04:38:10.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.857 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.857 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:10.857 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486524+0000", 2024-09-18T04:38:10.857 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4972000000000003e-05 2024-09-18T04:38:10.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.857 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.857 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:10.858 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.488902+0000", 2024-09-18T04:38:10.858 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023776240000000001 2024-09-18T04:38:10.858 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.858 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.858 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:10.858 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.488968+0000", 2024-09-18T04:38:10.858 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6761000000000007e-05 2024-09-18T04:38:10.858 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:10.858 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:10.858 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:10.858 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.858 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.858 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:35 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'21, mlcod=15'17)", 2024-09-18T04:38:10.858 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.491817+0000", 2024-09-18T04:38:10.859 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 43.355227997, 2024-09-18T04:38:10.859 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023524739999999998, 2024-09-18T04:38:10.859 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:10.859 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:10.859 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:10.859 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.859 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:10.859 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491817+0000", 2024-09-18T04:38:10.859 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:10.859 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.859 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.859 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:10.859 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491817+0000", 2024-09-18T04:38:10.859 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:10.859 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.860 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.860 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:10.860 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491815+0000", 2024-09-18T04:38:10.860 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:10.860 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.860 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.860 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:10.860 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491890+0000", 2024-09-18T04:38:10.860 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4500999999999997e-05 2024-09-18T04:38:10.860 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.860 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.860 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:10.860 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491916+0000", 2024-09-18T04:38:10.860 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5666000000000001e-05 2024-09-18T04:38:10.861 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.861 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.861 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:10.861 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491928+0000", 2024-09-18T04:38:10.861 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2585999999999999e-05 2024-09-18T04:38:10.861 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.861 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.861 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:10.861 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.492000+0000", 2024-09-18T04:38:10.861 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2309000000000002e-05 2024-09-18T04:38:10.861 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.861 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.861 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:10.862 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.492027+0000", 2024-09-18T04:38:10.862 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6251e-05 2024-09-18T04:38:10.862 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.862 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.862 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:10.862 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.494114+0000", 2024-09-18T04:38:10.862 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020875260000000001 2024-09-18T04:38:10.862 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.862 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.862 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:10.862 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.494169+0000", 2024-09-18T04:38:10.862 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5145000000000001e-05 2024-09-18T04:38:10.862 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:10.862 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:10.862 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:10.863 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.863 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.863 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:57 1.0 e15/14 1:30306672:devicehealth::main.db.0000000000000000:head v 15'35, mlcod=15'33)", 2024-09-18T04:38:10.863 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.529997+0000", 2024-09-18T04:38:10.863 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 43.317048143000001, 2024-09-18T04:38:10.863 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002623869, 2024-09-18T04:38:10.863 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:10.863 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:10.863 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:10.863 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.863 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:10.863 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.529997+0000", 2024-09-18T04:38:10.863 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:10.864 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.864 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.864 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:10.864 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.529997+0000", 2024-09-18T04:38:10.864 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:10.864 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.864 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.864 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:10.864 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.529995+0000", 2024-09-18T04:38:10.865 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:10.865 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.865 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.865 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:10.865 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530116+0000", 2024-09-18T04:38:10.865 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000121141 2024-09-18T04:38:10.865 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.865 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.865 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:10.865 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530146+0000", 2024-09-18T04:38:10.865 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0423e-05 2024-09-18T04:38:10.865 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.865 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.865 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:10.866 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530159+0000", 2024-09-18T04:38:10.866 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2802999999999999e-05 2024-09-18T04:38:10.866 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.866 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.866 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:10.866 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530218+0000", 2024-09-18T04:38:10.866 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8461000000000002e-05 2024-09-18T04:38:10.866 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.866 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.866 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:10.866 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530253+0000", 2024-09-18T04:38:10.866 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4796999999999999e-05 2024-09-18T04:38:10.866 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.866 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.867 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:10.867 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.532559+0000", 2024-09-18T04:38:10.867 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002306939 2024-09-18T04:38:10.867 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.867 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.867 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:10.867 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.532621+0000", 2024-09-18T04:38:10.867 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1131999999999997e-05 2024-09-18T04:38:10.867 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:10.867 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:10.867 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:10.867 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.867 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.867 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:68 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'42, mlcod=15'39)", 2024-09-18T04:38:10.868 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.546404+0000", 2024-09-18T04:38:10.868 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 43.300640731999998, 2024-09-18T04:38:10.868 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024499769999999999, 2024-09-18T04:38:10.868 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:10.868 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:10.868 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:10.868 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.868 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:10.868 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546404+0000", 2024-09-18T04:38:10.868 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:10.868 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.868 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.868 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:10.869 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546404+0000", 2024-09-18T04:38:10.869 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:10.869 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.869 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.869 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:10.869 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546403+0000", 2024-09-18T04:38:10.869 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:10.869 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.869 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.869 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:10.869 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546416+0000", 2024-09-18T04:38:10.869 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.306e-05 2024-09-18T04:38:10.870 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.870 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.870 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:10.870 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546465+0000", 2024-09-18T04:38:10.870 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9441999999999999e-05 2024-09-18T04:38:10.870 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.870 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.870 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:10.870 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546480+0000", 2024-09-18T04:38:10.870 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4919999999999999e-05 2024-09-18T04:38:10.870 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.870 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.870 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:10.871 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546541+0000", 2024-09-18T04:38:10.871 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1085000000000002e-05 2024-09-18T04:38:10.871 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.871 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.871 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:10.871 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546569+0000", 2024-09-18T04:38:10.871 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7803000000000001e-05 2024-09-18T04:38:10.871 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.871 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.871 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:10.871 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.548725+0000", 2024-09-18T04:38:10.871 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021556629999999999 2024-09-18T04:38:10.871 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.872 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.872 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:10.872 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.548854+0000", 2024-09-18T04:38:10.872 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000129524 2024-09-18T04:38:10.872 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:10.872 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:10.872 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:10.872 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.872 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.872 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:69 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'43, mlcod=15'39)", 2024-09-18T04:38:10.872 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.547520+0000", 2024-09-18T04:38:10.872 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 43.299524656000003, 2024-09-18T04:38:10.873 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023506820000000002, 2024-09-18T04:38:10.873 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:10.873 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:10.873 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:10.873 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.873 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:10.873 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547520+0000", 2024-09-18T04:38:10.873 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:10.873 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.873 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.873 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:10.873 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547520+0000", 2024-09-18T04:38:10.873 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:10.874 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.874 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.874 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:10.874 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547519+0000", 2024-09-18T04:38:10.874 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:10.874 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.874 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.874 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:10.874 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547614+0000", 2024-09-18T04:38:10.874 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5235000000000006e-05 2024-09-18T04:38:10.874 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.874 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.874 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:10.875 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547640+0000", 2024-09-18T04:38:10.875 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5698999999999999e-05 2024-09-18T04:38:10.875 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.875 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.875 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:10.875 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547651+0000", 2024-09-18T04:38:10.875 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.134e-05 2024-09-18T04:38:10.875 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.875 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.875 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:10.875 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547805+0000", 2024-09-18T04:38:10.875 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015426099999999999 2024-09-18T04:38:10.876 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.876 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.876 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:10.876 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547844+0000", 2024-09-18T04:38:10.876 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8229999999999998e-05 2024-09-18T04:38:10.876 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.876 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.876 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:10.876 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.549810+0000", 2024-09-18T04:38:10.876 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0019660519999999998 2024-09-18T04:38:10.876 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.876 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.876 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:10.877 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.549871+0000", 2024-09-18T04:38:10.877 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1152e-05 2024-09-18T04:38:10.877 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:10.877 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:10.877 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:10.877 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.877 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.877 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:83 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'53, mlcod=15'50)", 2024-09-18T04:38:10.877 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.575989+0000", 2024-09-18T04:38:10.877 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 43.271055707999999, 2024-09-18T04:38:10.877 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023991310000000001, 2024-09-18T04:38:10.877 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:10.877 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:10.877 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:10.878 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.878 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:10.878 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.575989+0000", 2024-09-18T04:38:10.878 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:10.878 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.878 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.878 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:10.878 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.575989+0000", 2024-09-18T04:38:10.878 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:10.878 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.878 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.878 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:10.878 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.575988+0000", 2024-09-18T04:38:10.878 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:10.879 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.879 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.879 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:10.879 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576001+0000", 2024-09-18T04:38:10.879 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2751e-05 2024-09-18T04:38:10.879 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.879 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.879 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:10.879 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576023+0000", 2024-09-18T04:38:10.879 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2453999999999999e-05 2024-09-18T04:38:10.879 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.879 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.879 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:10.880 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576038+0000", 2024-09-18T04:38:10.880 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5125e-05 2024-09-18T04:38:10.880 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.880 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.880 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:10.880 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576161+0000", 2024-09-18T04:38:10.880 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000122543 2024-09-18T04:38:10.880 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.880 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.880 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:10.880 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576221+0000", 2024-09-18T04:38:10.880 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9948000000000003e-05 2024-09-18T04:38:10.880 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.880 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.881 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:10.881 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.578277+0000", 2024-09-18T04:38:10.881 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020567559999999999 2024-09-18T04:38:10.881 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.881 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.881 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:10.881 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.578388+0000", 2024-09-18T04:38:10.881 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011089100000000001 2024-09-18T04:38:10.881 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:10.881 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:10.881 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:10.881 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.881 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.881 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:86 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'56, mlcod=15'52)", 2024-09-18T04:38:10.882 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.580261+0000", 2024-09-18T04:38:10.882 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 43.266784072, 2024-09-18T04:38:10.882 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002418108, 2024-09-18T04:38:10.882 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:10.882 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:10.882 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:10.882 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.882 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:10.882 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580261+0000", 2024-09-18T04:38:10.882 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:10.882 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.882 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.882 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:10.882 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580261+0000", 2024-09-18T04:38:10.883 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:10.883 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.883 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.883 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:10.883 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580260+0000", 2024-09-18T04:38:10.883 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:10.883 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.883 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.883 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:10.883 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580269+0000", 2024-09-18T04:38:10.883 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5109999999999992e-06 2024-09-18T04:38:10.883 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.883 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.884 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:10.884 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580288+0000", 2024-09-18T04:38:10.884 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8783999999999998e-05 2024-09-18T04:38:10.884 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.884 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.884 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:10.884 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580296+0000", 2024-09-18T04:38:10.884 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3250000000000008e-06 2024-09-18T04:38:10.884 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.884 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.884 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:10.884 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580351+0000", 2024-09-18T04:38:10.884 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4132000000000003e-05 2024-09-18T04:38:10.884 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.885 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.885 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:10.885 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580381+0000", 2024-09-18T04:38:10.885 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0263000000000001e-05 2024-09-18T04:38:10.885 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.885 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.885 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:10.885 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.582615+0000", 2024-09-18T04:38:10.885 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0022339109999999999 2024-09-18T04:38:10.885 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.885 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.885 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:10.885 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.582679+0000", 2024-09-18T04:38:10.885 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4182000000000003e-05 2024-09-18T04:38:10.886 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:10.886 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:10.886 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:10.886 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.886 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.886 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:105 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'67, mlcod=15'64)", 2024-09-18T04:38:10.886 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.789089+0000", 2024-09-18T04:38:10.886 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 43.057955902000003, 2024-09-18T04:38:10.886 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024384229999999999, 2024-09-18T04:38:10.886 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:10.886 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:10.886 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:10.886 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.887 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:10.887 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789089+0000", 2024-09-18T04:38:10.887 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:10.887 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.887 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.887 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:10.887 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789089+0000", 2024-09-18T04:38:10.887 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:10.887 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.887 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.887 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:10.887 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789088+0000", 2024-09-18T04:38:10.887 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:10.888 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.888 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.888 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:10.888 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789107+0000", 2024-09-18T04:38:10.888 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9017000000000001e-05 2024-09-18T04:38:10.888 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.888 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.888 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:10.888 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789139+0000", 2024-09-18T04:38:10.888 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2261999999999997e-05 2024-09-18T04:38:10.888 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.888 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.888 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:10.889 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789153+0000", 2024-09-18T04:38:10.889 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4025999999999999e-05 2024-09-18T04:38:10.889 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.889 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.889 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:10.889 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789255+0000", 2024-09-18T04:38:10.889 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000101969 2024-09-18T04:38:10.889 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.889 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.889 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:10.889 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789314+0000", 2024-09-18T04:38:10.889 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8867999999999999e-05 2024-09-18T04:38:10.889 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.889 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.889 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:10.890 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.791453+0000", 2024-09-18T04:38:10.890 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002138938 2024-09-18T04:38:10.890 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.890 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.890 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:10.890 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.791527+0000", 2024-09-18T04:38:10.890 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4464999999999999e-05 2024-09-18T04:38:10.890 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:10.890 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:10.890 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:10.890 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.890 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.890 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:106 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'68, mlcod=15'64)", 2024-09-18T04:38:10.891 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.790295+0000", 2024-09-18T04:38:10.891 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 43.056750051000002, 2024-09-18T04:38:10.891 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024379390000000001, 2024-09-18T04:38:10.891 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:10.891 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:10.891 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:10.891 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.891 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:10.891 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790295+0000", 2024-09-18T04:38:10.891 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:10.891 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.891 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.891 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:10.892 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790295+0000", 2024-09-18T04:38:10.892 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:10.892 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.892 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.892 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:10.892 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790294+0000", 2024-09-18T04:38:10.892 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:10.892 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.892 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.892 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:10.892 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790390+0000", 2024-09-18T04:38:10.892 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6537000000000001e-05 2024-09-18T04:38:10.892 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.892 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.893 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:10.893 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790418+0000", 2024-09-18T04:38:10.893 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8218e-05 2024-09-18T04:38:10.893 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.893 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.893 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:10.893 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790431+0000", 2024-09-18T04:38:10.893 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2571999999999999e-05 2024-09-18T04:38:10.893 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.893 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.893 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:10.893 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790534+0000", 2024-09-18T04:38:10.893 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000103433 2024-09-18T04:38:10.893 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.894 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.894 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:10.894 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790565+0000", 2024-09-18T04:38:10.894 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0409999999999999e-05 2024-09-18T04:38:10.894 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.894 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.894 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:10.894 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.792651+0000", 2024-09-18T04:38:10.894 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020865250000000001 2024-09-18T04:38:10.894 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.894 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.894 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:10.895 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.792733+0000", 2024-09-18T04:38:10.895 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1477999999999995e-05 2024-09-18T04:38:10.895 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:10.895 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:10.895 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:10.895 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.895 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.895 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:109 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'71, mlcod=15'68)", 2024-09-18T04:38:10.895 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.795250+0000", 2024-09-18T04:38:10.895 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 43.051795036999998, 2024-09-18T04:38:10.895 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023990449999999998, 2024-09-18T04:38:10.895 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:10.895 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:10.895 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:10.896 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.896 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:10.896 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795250+0000", 2024-09-18T04:38:10.896 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:10.896 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.896 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.896 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:10.896 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795250+0000", 2024-09-18T04:38:10.896 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:10.896 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.896 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.896 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:10.896 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795248+0000", 2024-09-18T04:38:10.897 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:10.897 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.897 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.897 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:10.897 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795405+0000", 2024-09-18T04:38:10.897 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000157007 2024-09-18T04:38:10.897 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.897 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.897 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:10.897 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795422+0000", 2024-09-18T04:38:10.897 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7013000000000001e-05 2024-09-18T04:38:10.897 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.897 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.897 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:10.898 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795432+0000", 2024-09-18T04:38:10.898 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0448999999999999e-05 2024-09-18T04:38:10.898 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.898 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.898 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:10.898 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795534+0000", 2024-09-18T04:38:10.898 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000101195 2024-09-18T04:38:10.898 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.898 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.898 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:10.898 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795587+0000", 2024-09-18T04:38:10.898 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3356000000000002e-05 2024-09-18T04:38:10.898 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.899 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.899 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:10.899 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.797560+0000", 2024-09-18T04:38:10.899 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001973023 2024-09-18T04:38:10.899 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.899 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.899 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:10.899 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.797649+0000", 2024-09-18T04:38:10.899 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8864999999999997e-05 2024-09-18T04:38:10.899 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:10.899 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:10.899 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:10.899 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.900 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.900 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:115 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'77, mlcod=15'72)", 2024-09-18T04:38:10.900 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.803339+0000", 2024-09-18T04:38:10.900 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 43.043706319000002, 2024-09-18T04:38:10.900 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002385093, 2024-09-18T04:38:10.900 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:10.900 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:10.900 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:10.900 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.900 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:10.900 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803339+0000", 2024-09-18T04:38:10.900 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:10.900 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.900 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.901 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:10.901 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803339+0000", 2024-09-18T04:38:10.901 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:10.901 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.901 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.901 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:10.901 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803337+0000", 2024-09-18T04:38:10.901 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:10.901 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.901 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.901 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:10.901 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803475+0000", 2024-09-18T04:38:10.901 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013718199999999999 2024-09-18T04:38:10.902 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.902 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.902 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:10.902 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803507+0000", 2024-09-18T04:38:10.902 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2351000000000002e-05 2024-09-18T04:38:10.902 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.902 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.902 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:10.902 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803533+0000", 2024-09-18T04:38:10.902 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6010999999999999e-05 2024-09-18T04:38:10.902 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.902 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.902 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:10.903 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803609+0000", 2024-09-18T04:38:10.903 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.64e-05 2024-09-18T04:38:10.903 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.903 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.903 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:10.903 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803671+0000", 2024-09-18T04:38:10.903 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1467e-05 2024-09-18T04:38:10.903 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.903 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.903 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:10.903 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.805676+0000", 2024-09-18T04:38:10.903 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020047849999999998 2024-09-18T04:38:10.903 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.904 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.904 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:10.904 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.805724+0000", 2024-09-18T04:38:10.904 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8151e-05 2024-09-18T04:38:10.904 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:10.904 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:10.904 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:10.904 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.904 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.904 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:118 1.0 e15/14 1:30306672:devicehealth::main.db.0000000000000000:head v 15'80, mlcod=15'78)", 2024-09-18T04:38:10.904 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.807930+0000", 2024-09-18T04:38:10.904 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 43.039114816000001, 2024-09-18T04:38:10.905 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028002259999999998, 2024-09-18T04:38:10.905 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:10.905 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:10.905 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:10.905 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.905 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:10.905 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.807930+0000", 2024-09-18T04:38:10.905 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:10.905 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.905 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.905 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:10.905 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.807930+0000", 2024-09-18T04:38:10.905 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:10.906 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.906 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.906 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:10.906 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.807929+0000", 2024-09-18T04:38:10.906 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:10.906 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.906 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.906 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:10.906 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808024+0000", 2024-09-18T04:38:10.906 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5258000000000002e-05 2024-09-18T04:38:10.906 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.906 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.906 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:10.907 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808044+0000", 2024-09-18T04:38:10.907 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0203e-05 2024-09-18T04:38:10.907 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.907 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.907 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:10.907 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808053+0000", 2024-09-18T04:38:10.907 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2689999999999999e-06 2024-09-18T04:38:10.907 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.907 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.907 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:10.907 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808137+0000", 2024-09-18T04:38:10.907 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4441e-05 2024-09-18T04:38:10.907 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.908 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.908 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:10.908 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808172+0000", 2024-09-18T04:38:10.908 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4453999999999999e-05 2024-09-18T04:38:10.908 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.908 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.908 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:10.908 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.810663+0000", 2024-09-18T04:38:10.908 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024910269999999998 2024-09-18T04:38:10.908 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.908 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.908 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:10.908 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.810730+0000", 2024-09-18T04:38:10.909 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7658999999999997e-05 2024-09-18T04:38:10.909 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:10.909 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:10.909 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:10.909 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.909 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.909 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:129 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'87, mlcod=15'85)", 2024-09-18T04:38:10.909 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.231689+0000", 2024-09-18T04:38:10.909 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 42.615355803, 2024-09-18T04:38:10.909 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002376502, 2024-09-18T04:38:10.909 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:10.909 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:10.909 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:10.910 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.910 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:10.910 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231689+0000", 2024-09-18T04:38:10.910 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:10.910 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.910 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.910 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:10.910 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231689+0000", 2024-09-18T04:38:10.910 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:10.910 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.910 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.910 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:10.910 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231687+0000", 2024-09-18T04:38:10.911 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:10.911 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.911 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.911 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:10.911 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231702+0000", 2024-09-18T04:38:10.911 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4949999999999999e-05 2024-09-18T04:38:10.911 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.911 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.911 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:10.911 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231735+0000", 2024-09-18T04:38:10.911 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2276000000000001e-05 2024-09-18T04:38:10.911 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.911 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.911 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:10.912 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231750+0000", 2024-09-18T04:38:10.912 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5193e-05 2024-09-18T04:38:10.912 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.912 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.912 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:10.912 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231872+0000", 2024-09-18T04:38:10.912 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000122127 2024-09-18T04:38:10.912 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.912 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.912 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:10.912 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231936+0000", 2024-09-18T04:38:10.912 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4089000000000003e-05 2024-09-18T04:38:10.912 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.913 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.913 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:10.913 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234002+0000", 2024-09-18T04:38:10.913 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002065575 2024-09-18T04:38:10.913 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.913 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.913 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:10.913 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234066+0000", 2024-09-18T04:38:10.913 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3944999999999998e-05 2024-09-18T04:38:10.913 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:10.913 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:10.913 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:10.914 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.914 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.914 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:130 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'88, mlcod=15'85)", 2024-09-18T04:38:10.914 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.232853+0000", 2024-09-18T04:38:10.914 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 42.614192187999997, 2024-09-18T04:38:10.914 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027351739999999999, 2024-09-18T04:38:10.914 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:10.914 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:10.914 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:10.914 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.914 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:10.914 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232853+0000", 2024-09-18T04:38:10.914 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:10.914 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.915 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.915 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:10.915 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232853+0000", 2024-09-18T04:38:10.915 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:10.915 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.915 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.915 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:10.915 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232851+0000", 2024-09-18T04:38:10.915 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:10.915 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.915 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.915 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:10.915 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232866+0000", 2024-09-18T04:38:10.915 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4946000000000001e-05 2024-09-18T04:38:10.916 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.916 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.916 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:10.916 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232897+0000", 2024-09-18T04:38:10.916 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0982999999999998e-05 2024-09-18T04:38:10.916 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.916 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.916 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:10.916 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232915+0000", 2024-09-18T04:38:10.916 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8476000000000001e-05 2024-09-18T04:38:10.916 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.916 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.917 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:10.917 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.233029+0000", 2024-09-18T04:38:10.917 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011305400000000001 2024-09-18T04:38:10.917 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.917 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.917 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:10.917 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.233079+0000", 2024-09-18T04:38:10.917 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0668999999999999e-05 2024-09-18T04:38:10.917 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.917 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.917 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:10.917 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235534+0000", 2024-09-18T04:38:10.917 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024546030000000001 2024-09-18T04:38:10.917 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.918 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.918 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:10.918 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235588+0000", 2024-09-18T04:38:10.918 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4098999999999998e-05 2024-09-18T04:38:10.918 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:10.918 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:10.918 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:10.918 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.918 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.918 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:131 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'89, mlcod=15'85)", 2024-09-18T04:38:10.918 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.234089+0000", 2024-09-18T04:38:10.918 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 42.612956334000003, 2024-09-18T04:38:10.918 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026975609999999998, 2024-09-18T04:38:10.919 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:10.919 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:10.919 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:10.919 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.919 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:10.919 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234089+0000", 2024-09-18T04:38:10.919 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:10.919 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.919 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.919 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:10.919 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234089+0000", 2024-09-18T04:38:10.919 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:10.919 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.919 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.920 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:10.920 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234087+0000", 2024-09-18T04:38:10.920 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:10.920 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.920 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.920 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:10.920 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234145+0000", 2024-09-18T04:38:10.920 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8687999999999998e-05 2024-09-18T04:38:10.920 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.920 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.920 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:10.920 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234177+0000", 2024-09-18T04:38:10.920 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1526999999999998e-05 2024-09-18T04:38:10.921 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.921 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.921 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:10.921 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234190+0000", 2024-09-18T04:38:10.921 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2965999999999999e-05 2024-09-18T04:38:10.921 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.921 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.921 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:10.921 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234309+0000", 2024-09-18T04:38:10.921 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011875399999999999 2024-09-18T04:38:10.921 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.921 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.921 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:10.921 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234365+0000", 2024-09-18T04:38:10.922 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6436e-05 2024-09-18T04:38:10.922 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.922 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.922 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:10.922 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236675+0000", 2024-09-18T04:38:10.922 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023099520000000001 2024-09-18T04:38:10.922 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.922 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.922 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:10.922 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236786+0000", 2024-09-18T04:38:10.922 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000111025 2024-09-18T04:38:10.922 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:10.922 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:10.923 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:10.923 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.923 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.923 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:132 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'90, mlcod=15'85)", 2024-09-18T04:38:10.923 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.235300+0000", 2024-09-18T04:38:10.923 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 42.611744883999997, 2024-09-18T04:38:10.923 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025982599999999998, 2024-09-18T04:38:10.923 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:10.923 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:10.923 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:10.923 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.923 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:10.923 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235300+0000", 2024-09-18T04:38:10.923 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:10.924 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.924 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.924 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:10.924 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235300+0000", 2024-09-18T04:38:10.924 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:10.924 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.924 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.924 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:10.924 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235298+0000", 2024-09-18T04:38:10.924 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:10.924 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.924 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.924 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:10.924 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235318+0000", 2024-09-18T04:38:10.925 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9714000000000001e-05 2024-09-18T04:38:10.925 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.925 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.925 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:10.925 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235337+0000", 2024-09-18T04:38:10.925 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8892000000000001e-05 2024-09-18T04:38:10.925 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.925 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.925 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:10.925 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235348+0000", 2024-09-18T04:38:10.925 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.129e-05 2024-09-18T04:38:10.925 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.925 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.925 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:10.925 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235681+0000", 2024-09-18T04:38:10.926 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00033323000000000001 2024-09-18T04:38:10.926 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.926 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.926 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:10.926 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235733+0000", 2024-09-18T04:38:10.926 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1391999999999999e-05 2024-09-18T04:38:10.926 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.926 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.926 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:10.926 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237839+0000", 2024-09-18T04:38:10.926 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002105647 2024-09-18T04:38:10.926 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.926 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.927 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:10.927 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237898+0000", 2024-09-18T04:38:10.927 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9759000000000002e-05 2024-09-18T04:38:10.927 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:10.927 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:10.927 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:10.927 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.927 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.927 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:133 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'91, mlcod=15'86)", 2024-09-18T04:38:10.927 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.236920+0000", 2024-09-18T04:38:10.927 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 42.610124996000003, 2024-09-18T04:38:10.927 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024318410000000001, 2024-09-18T04:38:10.927 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:10.928 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:10.928 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:10.928 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.928 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:10.928 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236920+0000", 2024-09-18T04:38:10.928 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:10.928 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.928 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.928 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:10.928 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236920+0000", 2024-09-18T04:38:10.928 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:10.928 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.928 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.928 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:10.929 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236918+0000", 2024-09-18T04:38:10.929 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:10.929 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.929 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.929 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:10.929 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236931+0000", 2024-09-18T04:38:10.929 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2683999999999999e-05 2024-09-18T04:38:10.929 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.929 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.929 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:10.929 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236951+0000", 2024-09-18T04:38:10.929 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0292000000000002e-05 2024-09-18T04:38:10.930 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.930 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.930 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:10.930 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236962+0000", 2024-09-18T04:38:10.930 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0703000000000001e-05 2024-09-18T04:38:10.930 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.930 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.930 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:10.930 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237095+0000", 2024-09-18T04:38:10.930 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013312400000000001 2024-09-18T04:38:10.930 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.930 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.931 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:10.931 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237133+0000", 2024-09-18T04:38:10.931 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7496000000000001e-05 2024-09-18T04:38:10.931 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.931 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.931 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:10.931 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.239206+0000", 2024-09-18T04:38:10.931 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020730290000000001 2024-09-18T04:38:10.931 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.931 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.931 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:10.931 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.239352+0000", 2024-09-18T04:38:10.931 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014593000000000001 2024-09-18T04:38:10.932 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:10.932 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:10.932 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:10.932 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.932 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.932 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:134 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'92, mlcod=15'86)", 2024-09-18T04:38:10.932 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.237894+0000", 2024-09-18T04:38:10.932 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 42.609150974000002, 2024-09-18T04:38:10.932 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025299519999999998, 2024-09-18T04:38:10.932 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:10.932 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:10.932 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:10.932 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.933 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:10.933 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237894+0000", 2024-09-18T04:38:10.933 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:10.933 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.933 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.933 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:10.933 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237894+0000", 2024-09-18T04:38:10.933 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:10.933 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.933 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.933 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:10.933 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237892+0000", 2024-09-18T04:38:10.933 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:10.933 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.934 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.934 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:10.934 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237986+0000", 2024-09-18T04:38:10.934 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3341e-05 2024-09-18T04:38:10.934 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.934 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.934 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:10.934 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.238012+0000", 2024-09-18T04:38:10.934 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5843e-05 2024-09-18T04:38:10.934 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.934 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.934 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:10.934 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.238027+0000", 2024-09-18T04:38:10.935 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5469e-05 2024-09-18T04:38:10.935 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.935 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.935 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:10.935 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.238114+0000", 2024-09-18T04:38:10.935 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7483000000000002e-05 2024-09-18T04:38:10.935 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.935 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.935 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:10.935 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.238152+0000", 2024-09-18T04:38:10.935 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7565999999999999e-05 2024-09-18T04:38:10.935 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.935 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.936 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:10.936 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.240297+0000", 2024-09-18T04:38:10.936 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021447620000000001 2024-09-18T04:38:10.936 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.936 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.936 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:10.936 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.240424+0000", 2024-09-18T04:38:10.936 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012705400000000001 2024-09-18T04:38:10.936 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:10.936 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:10.936 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:10.936 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.936 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.937 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:139 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'97, mlcod=15'93)", 2024-09-18T04:38:10.937 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.245692+0000", 2024-09-18T04:38:10.937 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 42.601352833, 2024-09-18T04:38:10.937 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024260480000000001, 2024-09-18T04:38:10.937 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:10.937 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:10.937 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:10.937 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.937 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:10.937 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245692+0000", 2024-09-18T04:38:10.937 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:10.937 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.937 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.938 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:10.938 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245692+0000", 2024-09-18T04:38:10.938 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:10.938 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.938 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.938 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:10.938 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245690+0000", 2024-09-18T04:38:10.938 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:10.938 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.938 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.938 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:10.938 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245712+0000", 2024-09-18T04:38:10.938 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1916000000000002e-05 2024-09-18T04:38:10.939 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.939 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.939 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:10.939 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245741+0000", 2024-09-18T04:38:10.939 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8503000000000001e-05 2024-09-18T04:38:10.939 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.939 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.939 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:10.939 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245759+0000", 2024-09-18T04:38:10.939 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.872e-05 2024-09-18T04:38:10.939 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.939 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.939 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:10.939 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245859+0000", 2024-09-18T04:38:10.940 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9641000000000004e-05 2024-09-18T04:38:10.940 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.940 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.940 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:10.940 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245900+0000", 2024-09-18T04:38:10.940 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0589000000000001e-05 2024-09-18T04:38:10.940 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.940 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.940 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:10.940 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.247961+0000", 2024-09-18T04:38:10.940 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020619190000000002 2024-09-18T04:38:10.940 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.940 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.941 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:10.941 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.248118+0000", 2024-09-18T04:38:10.941 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015664300000000001 2024-09-18T04:38:10.941 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:10.941 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:10.941 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:10.941 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.941 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.941 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:144 1.0 e15/14 1:30306672:devicehealth::main.db.0000000000000000:head v 15'102, mlcod=15'98)", 2024-09-18T04:38:10.941 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.252149+0000", 2024-09-18T04:38:10.941 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 42.594895766999997, 2024-09-18T04:38:10.941 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024174959999999999, 2024-09-18T04:38:10.941 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:10.942 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:10.942 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:10.942 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.942 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:10.942 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252149+0000", 2024-09-18T04:38:10.942 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:10.942 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.942 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.942 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:10.942 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252149+0000", 2024-09-18T04:38:10.942 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:10.942 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.942 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.942 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:10.942 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252148+0000", 2024-09-18T04:38:10.943 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:10.943 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.943 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.943 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:10.943 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252225+0000", 2024-09-18T04:38:10.943 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6576999999999995e-05 2024-09-18T04:38:10.943 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.943 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.943 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:10.943 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252242+0000", 2024-09-18T04:38:10.943 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7740999999999998e-05 2024-09-18T04:38:10.944 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.944 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.944 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:10.944 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252254+0000", 2024-09-18T04:38:10.944 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.137e-05 2024-09-18T04:38:10.944 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.944 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.944 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:10.944 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252385+0000", 2024-09-18T04:38:10.944 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013091099999999999 2024-09-18T04:38:10.944 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.944 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.944 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:10.945 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252426+0000", 2024-09-18T04:38:10.945 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1535999999999997e-05 2024-09-18T04:38:10.945 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.945 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.945 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:10.945 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.254498+0000", 2024-09-18T04:38:10.945 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002071477 2024-09-18T04:38:10.945 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:10.945 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:10.945 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:10.945 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.254567+0000", 2024-09-18T04:38:10.946 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9014e-05 2024-09-18T04:38:10.946 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:10.946 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:10.946 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:10.946 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:10.946 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:10.946 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:10.948 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:38:10.948 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:10.943+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:11.049 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:11.046+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:11.056 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:11.056 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:11.056 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:38:11.056 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:11.066 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:38:11.150 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:11.147+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:11.166 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T04:38:11.169 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:11.169 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:11.169 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:38:11.169 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:38:11.169 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:11.180 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:38:11.250 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:11.247+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:11.298 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:38:11.318 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:38:11.347 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:11.348 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:11.351 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:11.346+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:11.354 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:38:11.359 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:38:11.376 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:38:11.402 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:38:11.424 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:38:11.427 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:38:11.427 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:38:11.427 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:38:11.427 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:38:11.427 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:38:11.427 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:38:11.452 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:11.446+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:11.466 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:11.466 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:11.476 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:38:11.553 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:11.546+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:11.583 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:11.583 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:11.594 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:38:11.654 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:11.651+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:11.699 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:11.700 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:11.709 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:38:11.754 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:11.750+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:11.778 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-09-18T04:38:11.827 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:11.827 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:11.837 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:38:11.854 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:11.851+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:11.941 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:11.941 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:11.951 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:38:11.955 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:11.950+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:12.053 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:12.054 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:12.056 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:12.051+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:12.063 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:38:12.157 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:12.155+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:12.215 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:12.215 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:12.224 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:38:12.258 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:12.255+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:12.328 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:12.328 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:12.337 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:38:12.382 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:12.378+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:12.443 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:12.443 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:12.453 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:38:12.459 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:12.455+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:12.560 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:12.560 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:12.560 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:12.554+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:12.570 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:38:12.661 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:12.654+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:12.723 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:12.723 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:12.732 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:38:12.762 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:12.759+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:12.841 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:12.842 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:12.851 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:38:12.901 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:12.895+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:12.963 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:12.958+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:12.968 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:12.969 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:12.979 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:38:13.064 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:13.058+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:13.090 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:13.090 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:13.100 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:38:13.165 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:13.159+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:13.204 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:13.204 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:13.214 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:38:13.265 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:13.259+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:13.366 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:13.362+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:13.369 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:13.370 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:13.379 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:38:13.439 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T04:38:13.484 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:13.485 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:13.485 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:13.479+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:13.495 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:38:13.568 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:13.562+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:13.579 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:38:13.601 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:38:13.624 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:38:13.650 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:13.650 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:38:13.650 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:38:13.650 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:38:13.650 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:13.650 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4214.0:5 2.2 e19/17 2:5cea7035:::rbd_info:head v 19'1, mlcod=0'0)", 2024-09-18T04:38:13.650 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:35.471260+0000", 2024-09-18T04:38:13.650 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 38.177647485000001, 2024-09-18T04:38:13.650 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021025950000000001, 2024-09-18T04:38:13.650 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:13.650 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:13.651 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:13.651 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:13.651 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:13.651 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471260+0000", 2024-09-18T04:38:13.651 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:13.651 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:13.651 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:13.651 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:13.651 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471260+0000", 2024-09-18T04:38:13.651 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:13.651 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:13.651 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:13.651 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:13.651 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471259+0000", 2024-09-18T04:38:13.652 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:13.652 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:13.652 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:13.652 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:13.652 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471275+0000", 2024-09-18T04:38:13.652 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6135999999999999e-05 2024-09-18T04:38:13.652 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:13.652 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:13.652 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:13.652 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471298+0000", 2024-09-18T04:38:13.652 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3632999999999999e-05 2024-09-18T04:38:13.652 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:13.653 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:13.653 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:13.653 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471320+0000", 2024-09-18T04:38:13.653 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1852999999999999e-05 2024-09-18T04:38:13.653 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:13.653 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:13.653 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:13.653 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471445+0000", 2024-09-18T04:38:13.653 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012461699999999999 2024-09-18T04:38:13.653 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:13.653 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:13.654 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:13.654 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471529+0000", 2024-09-18T04:38:13.654 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3875000000000006e-05 2024-09-18T04:38:13.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:13.654 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:13.654 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:13.654 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.473317+0000", 2024-09-18T04:38:13.654 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017886899999999999 2024-09-18T04:38:13.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:13.654 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:13.654 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:13.654 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.473362+0000", 2024-09-18T04:38:13.655 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4724000000000001e-05 2024-09-18T04:38:13.655 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:13.655 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:13.655 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:13.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:13.655 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:13.655 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4214.0:7 2.2 e20/17 2:5cea7035:::rbd_info:head v 20'2, mlcod=0'0)", 2024-09-18T04:38:13.655 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:36.490560+0000", 2024-09-18T04:38:13.655 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 37.158347345999999, 2024-09-18T04:38:13.655 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0015326020000000001, 2024-09-18T04:38:13.655 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:13.655 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:13.655 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:13.655 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:13.655 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:13.656 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490560+0000", 2024-09-18T04:38:13.656 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:13.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:13.656 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:13.656 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:13.656 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490560+0000", 2024-09-18T04:38:13.656 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:13.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:13.656 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:13.656 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:13.656 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490559+0000", 2024-09-18T04:38:13.656 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:13.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:13.657 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:13.657 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:13.657 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490568+0000", 2024-09-18T04:38:13.657 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5699999999999993e-06 2024-09-18T04:38:13.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:13.657 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:13.657 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:13.657 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490599+0000", 2024-09-18T04:38:13.657 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1566000000000002e-05 2024-09-18T04:38:13.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:13.657 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:13.657 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:13.658 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490609+0000", 2024-09-18T04:38:13.658 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.8700000000000004e-06 2024-09-18T04:38:13.658 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:13.658 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:13.658 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:13.658 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490708+0000", 2024-09-18T04:38:13.658 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9139000000000001e-05 2024-09-18T04:38:13.658 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:13.658 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:13.658 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:13.658 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490736+0000", 2024-09-18T04:38:13.658 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7617e-05 2024-09-18T04:38:13.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:13.659 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:13.659 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:13.659 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.492016+0000", 2024-09-18T04:38:13.659 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0012796579999999999 2024-09-18T04:38:13.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:13.659 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:13.659 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:13.659 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.492092+0000", 2024-09-18T04:38:13.659 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6571999999999998e-05 2024-09-18T04:38:13.659 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:13.659 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:13.659 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:13.659 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:13.660 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:13.660 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:13.660 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:38:13.661 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:38:13.668 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:13.662+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:13.675 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:38:13.716 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:38:13.718 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:38:13.718 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:38:13.718 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:38:13.718 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:38:13.718 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:38:13.718 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:38:13.770 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:13.762+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:13.781 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:13.781 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:13.781 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:38:13.781 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:13.790 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-18T04:38:13.792 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:38:13.871 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:13.867+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:13.891 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:13.891 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:13.892 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:38:13.892 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:38:13.892 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:13.901 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:38:13.972 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:13.967+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:14.000 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:14.000 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:38:14.000 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:38:14.000 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:38:14.000 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:14.000 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4214.0:1 2.3 e18/17 2:c4c92e5a:::rbd_trash:head v 18'1, mlcod=0'0)", 2024-09-18T04:38:14.000 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:34.475123+0000", 2024-09-18T04:38:14.000 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 39.523659207000001, 2024-09-18T04:38:14.000 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00081394599999999996, 2024-09-18T04:38:14.001 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:14.001 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:14.001 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:14.001 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:14.001 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:14.001 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475123+0000", 2024-09-18T04:38:14.001 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:14.001 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:14.001 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:14.001 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:14.001 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475123+0000", 2024-09-18T04:38:14.001 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:14.001 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:14.001 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:14.002 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:14.002 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475121+0000", 2024-09-18T04:38:14.002 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:14.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:14.002 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:14.002 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:14.002 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475149+0000", 2024-09-18T04:38:14.002 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7155999999999999e-05 2024-09-18T04:38:14.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:14.002 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:14.002 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:14.002 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475165+0000", 2024-09-18T04:38:14.002 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6160000000000001e-05 2024-09-18T04:38:14.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:14.003 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:14.003 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:14.003 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475183+0000", 2024-09-18T04:38:14.003 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7918e-05 2024-09-18T04:38:14.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:14.003 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:14.003 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:14.003 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475246+0000", 2024-09-18T04:38:14.003 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3176000000000006e-05 2024-09-18T04:38:14.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:14.003 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:14.003 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:14.003 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475279+0000", 2024-09-18T04:38:14.003 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3192999999999999e-05 2024-09-18T04:38:14.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:14.004 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:14.004 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:14.004 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475871+0000", 2024-09-18T04:38:14.004 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00059206099999999995 2024-09-18T04:38:14.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:14.004 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:14.004 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:14.004 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475937+0000", 2024-09-18T04:38:14.004 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5439999999999997e-05 2024-09-18T04:38:14.004 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:14.004 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:14.004 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:14.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:14.004 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:14.004 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 25/24 [PushOp(2:c4c92e5a:::rbd_trash:head, version: 18'1, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:c4c92e5a:::rbd_trash:head@18'1, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-18T04:38:14.005 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:51.680544+0000", 2024-09-18T04:38:14.005 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.318237365000002, 2024-09-18T04:38:14.005 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00050365200000000005, 2024-09-18T04:38:14.005 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:14.005 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:38:14.005 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:14.005 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:14.005 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:14.005 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680544+0000", 2024-09-18T04:38:14.005 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:14.005 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:14.005 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:14.005 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:14.006 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680544+0000", 2024-09-18T04:38:14.006 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:14.006 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:14.006 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:14.006 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:14.006 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680543+0000", 2024-09-18T04:38:14.006 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:14.006 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:14.006 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:14.006 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:14.006 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680571+0000", 2024-09-18T04:38:14.006 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7953000000000002e-05 2024-09-18T04:38:14.006 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:14.007 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:14.007 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:14.007 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680605+0000", 2024-09-18T04:38:14.007 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3933000000000002e-05 2024-09-18T04:38:14.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:14.007 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:14.007 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:14.007 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680636+0000", 2024-09-18T04:38:14.007 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1288999999999998e-05 2024-09-18T04:38:14.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:14.007 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:14.007 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:14.007 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680748+0000", 2024-09-18T04:38:14.007 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000111431 2024-09-18T04:38:14.008 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:14.008 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:14.008 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:14.008 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680758+0000", 2024-09-18T04:38:14.008 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0244999999999999e-05 2024-09-18T04:38:14.008 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:14.008 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:14.008 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:14.008 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.681048+0000", 2024-09-18T04:38:14.008 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00029003399999999999 2024-09-18T04:38:14.008 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:14.008 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:14.008 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:14.008 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:14.009 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:14.009 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:14.010 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:38:14.073 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:14.067+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:14.126 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:14.126 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:14.126 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:38:14.127 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:14.136 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:38:14.174 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:14.170+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:14.244 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:14.244 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:14.244 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:38:14.244 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:38:14.244 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:14.254 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:38:14.275 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:14.271+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:14.365 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:14.366 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:38:14.366 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:38:14.366 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:38:14.366 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.366 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:14.366 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.573328+0000", 2024-09-18T04:38:14.366 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 46.786242913000002, 2024-09-18T04:38:14.366 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011397898, 2024-09-18T04:38:14.366 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:14.366 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:14.366 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:14.366 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:14.366 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:14.367 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 86 2024-09-18T04:38:14.367 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.367 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:14.367 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.367 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:14.367 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573328+0000", 2024-09-18T04:38:14.367 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:14.367 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.367 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.367 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:14.367 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573328+0000", 2024-09-18T04:38:14.367 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:14.368 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.368 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.368 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:14.368 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573327+0000", 2024-09-18T04:38:14.368 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:14.368 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.368 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.368 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:14.368 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573331+0000", 2024-09-18T04:38:14.368 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9750000000000001e-06 2024-09-18T04:38:14.368 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.368 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.369 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:14.369 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573346+0000", 2024-09-18T04:38:14.369 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5469999999999999e-05 2024-09-18T04:38:14.369 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.369 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.369 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:14.369 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573370+0000", 2024-09-18T04:38:14.369 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3969999999999999e-05 2024-09-18T04:38:14.369 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.369 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.369 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:14.370 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.578536+0000", 2024-09-18T04:38:14.370 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0051659679999999999 2024-09-18T04:38:14.370 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.370 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.370 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:14.370 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.578689+0000", 2024-09-18T04:38:14.370 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015276299999999999 2024-09-18T04:38:14.370 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.370 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.370 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:14.370 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.578901+0000", 2024-09-18T04:38:14.370 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000211783 2024-09-18T04:38:14.370 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.371 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.371 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:14.371 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.583408+0000", 2024-09-18T04:38:14.371 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045071249999999998 2024-09-18T04:38:14.371 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.371 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.371 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:14.371 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.584645+0000", 2024-09-18T04:38:14.371 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001237007 2024-09-18T04:38:14.371 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.371 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.371 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:14.371 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.584703+0000", 2024-09-18T04:38:14.371 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.817e-05 2024-09-18T04:38:14.372 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.372 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.372 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:14.372 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.584725+0000", 2024-09-18T04:38:14.372 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2155e-05 2024-09-18T04:38:14.372 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:14.372 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:14.372 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:14.372 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.372 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.372 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:14.373 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.573382+0000", 2024-09-18T04:38:14.373 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 46.786188164999999, 2024-09-18T04:38:14.373 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012154974000000001, 2024-09-18T04:38:14.373 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:14.373 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:14.373 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:14.373 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:14.373 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:14.373 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 87 2024-09-18T04:38:14.373 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.373 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:14.373 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.373 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:14.373 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573382+0000", 2024-09-18T04:38:14.374 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:14.374 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.374 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.374 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:14.374 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573382+0000", 2024-09-18T04:38:14.374 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:14.374 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.374 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.374 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:14.374 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573382+0000", 2024-09-18T04:38:14.374 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:14.374 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.374 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.375 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:14.375 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573482+0000", 2024-09-18T04:38:14.375 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000100333 2024-09-18T04:38:14.375 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.375 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.375 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:14.375 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573492+0000", 2024-09-18T04:38:14.375 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5389999999999996e-06 2024-09-18T04:38:14.375 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.375 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.375 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:14.375 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573500+0000", 2024-09-18T04:38:14.375 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8929999999999994e-06 2024-09-18T04:38:14.376 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.376 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.376 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:14.376 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.579797+0000", 2024-09-18T04:38:14.376 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0062963239999999998 2024-09-18T04:38:14.376 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.376 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.376 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:14.376 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.579957+0000", 2024-09-18T04:38:14.376 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000160283 2024-09-18T04:38:14.376 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.376 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.376 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:14.377 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.580178+0000", 2024-09-18T04:38:14.377 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00022079200000000001 2024-09-18T04:38:14.377 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.377 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.377 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:14.377 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.583479+0000", 2024-09-18T04:38:14.377 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033007689999999998 2024-09-18T04:38:14.377 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.377 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.377 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:14.377 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585477+0000", 2024-09-18T04:38:14.377 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0019985670000000001 2024-09-18T04:38:14.377 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.378 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.378 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:14.378 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585517+0000", 2024-09-18T04:38:14.378 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9709999999999998e-05 2024-09-18T04:38:14.378 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.378 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.378 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:14.378 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585537+0000", 2024-09-18T04:38:14.378 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0326000000000002e-05 2024-09-18T04:38:14.378 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:14.378 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:14.378 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:14.378 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.379 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.379 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:14.379 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.573508+0000", 2024-09-18T04:38:14.379 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 46.786062459, 2024-09-18T04:38:14.379 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012068476, 2024-09-18T04:38:14.379 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:14.379 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:14.379 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:14.379 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:14.379 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:14.379 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 88 2024-09-18T04:38:14.379 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.379 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:14.379 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.380 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:14.380 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573508+0000", 2024-09-18T04:38:14.380 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:14.380 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.380 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.380 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:14.380 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573508+0000", 2024-09-18T04:38:14.380 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:14.380 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.380 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.380 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:14.380 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573507+0000", 2024-09-18T04:38:14.381 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:14.381 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.381 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.381 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:14.381 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573512+0000", 2024-09-18T04:38:14.381 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.2629999999999997e-06 2024-09-18T04:38:14.381 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.381 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.381 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:14.381 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573530+0000", 2024-09-18T04:38:14.381 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8507000000000001e-05 2024-09-18T04:38:14.381 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.382 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.382 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:14.382 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573537+0000", 2024-09-18T04:38:14.382 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.832e-06 2024-09-18T04:38:14.382 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.382 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.382 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:14.382 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.582486+0000", 2024-09-18T04:38:14.382 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0089494770000000008 2024-09-18T04:38:14.382 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.382 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.382 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:14.382 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.582630+0000", 2024-09-18T04:38:14.383 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000144005 2024-09-18T04:38:14.383 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.383 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.383 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:14.383 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.582842+0000", 2024-09-18T04:38:14.383 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00021189200000000001 2024-09-18T04:38:14.383 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.383 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.383 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:14.383 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.584777+0000", 2024-09-18T04:38:14.383 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001934753 2024-09-18T04:38:14.383 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.384 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.384 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:14.384 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585543+0000", 2024-09-18T04:38:14.384 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00076540099999999997 2024-09-18T04:38:14.384 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.384 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.384 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:14.384 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585561+0000", 2024-09-18T04:38:14.384 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8474999999999998e-05 2024-09-18T04:38:14.384 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.384 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.384 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:14.384 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585576+0000", 2024-09-18T04:38:14.385 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5427e-05 2024-09-18T04:38:14.385 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:14.385 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:14.385 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:14.385 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.385 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.385 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:14.385 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786394+0000", 2024-09-18T04:38:14.385 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 46.573176011999998, 2024-09-18T04:38:14.385 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011902085999999999, 2024-09-18T04:38:14.385 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:14.385 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:14.386 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:14.386 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:14.386 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:14.386 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 109 2024-09-18T04:38:14.386 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.386 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:14.386 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.386 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:14.386 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786394+0000", 2024-09-18T04:38:14.386 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:14.386 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.386 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.386 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:14.387 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786394+0000", 2024-09-18T04:38:14.387 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:14.387 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.387 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.387 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:14.387 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786394+0000", 2024-09-18T04:38:14.387 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:14.387 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.387 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.387 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:14.387 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786548+0000", 2024-09-18T04:38:14.387 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015463400000000001 2024-09-18T04:38:14.387 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.388 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.388 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:14.388 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786561+0000", 2024-09-18T04:38:14.388 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3093e-05 2024-09-18T04:38:14.388 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.388 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.388 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:14.388 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786571+0000", 2024-09-18T04:38:14.388 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4700000000000008e-06 2024-09-18T04:38:14.388 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.388 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.388 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:14.388 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.793367+0000", 2024-09-18T04:38:14.388 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0067966600000000004 2024-09-18T04:38:14.389 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.389 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.389 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:14.389 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.793526+0000", 2024-09-18T04:38:14.389 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015877 2024-09-18T04:38:14.389 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.389 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.389 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:14.389 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.793752+0000", 2024-09-18T04:38:14.389 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00022555200000000001 2024-09-18T04:38:14.389 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.389 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.389 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:14.390 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798125+0000", 2024-09-18T04:38:14.390 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0043733349999999999 2024-09-18T04:38:14.390 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.390 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.390 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:14.390 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798146+0000", 2024-09-18T04:38:14.390 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1063000000000001e-05 2024-09-18T04:38:14.390 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.390 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.390 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:14.390 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798232+0000", 2024-09-18T04:38:14.390 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5771000000000003e-05 2024-09-18T04:38:14.390 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.391 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.391 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:14.391 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798297+0000", 2024-09-18T04:38:14.391 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.4624999999999999e-05 2024-09-18T04:38:14.391 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:14.391 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:14.391 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:14.391 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.391 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.391 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:14.391 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786584+0000", 2024-09-18T04:38:14.392 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 46.572985932999998, 2024-09-18T04:38:14.392 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012013037000000001, 2024-09-18T04:38:14.392 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:14.392 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:14.392 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:14.392 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:14.392 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:14.392 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 110 2024-09-18T04:38:14.392 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.392 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:14.392 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.392 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:14.392 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786584+0000", 2024-09-18T04:38:14.393 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:14.393 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.393 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.393 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:14.393 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786584+0000", 2024-09-18T04:38:14.393 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:14.393 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.393 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.393 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:14.393 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786584+0000", 2024-09-18T04:38:14.393 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:14.393 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.393 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.393 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:14.394 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786590+0000", 2024-09-18T04:38:14.394 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6359999999999997e-06 2024-09-18T04:38:14.394 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.394 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.394 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:14.394 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786599+0000", 2024-09-18T04:38:14.394 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7369999999999995e-06 2024-09-18T04:38:14.394 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.394 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.395 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:14.395 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786606+0000", 2024-09-18T04:38:14.395 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1930000000000004e-06 2024-09-18T04:38:14.395 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.395 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.395 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:14.395 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.795018+0000", 2024-09-18T04:38:14.395 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0084115660000000005 2024-09-18T04:38:14.395 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.395 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.395 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:14.395 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.795184+0000", 2024-09-18T04:38:14.395 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000165833 2024-09-18T04:38:14.396 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.396 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.396 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:14.396 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.795414+0000", 2024-09-18T04:38:14.396 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00023088399999999999 2024-09-18T04:38:14.396 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.396 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.396 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:14.396 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798321+0000", 2024-09-18T04:38:14.396 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0029069539999999998 2024-09-18T04:38:14.396 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.396 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.397 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:14.397 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798404+0000", 2024-09-18T04:38:14.397 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2724000000000003e-05 2024-09-18T04:38:14.397 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.397 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.397 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:14.397 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798488+0000", 2024-09-18T04:38:14.397 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.3505999999999996e-05 2024-09-18T04:38:14.397 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.397 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.397 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:14.397 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798598+0000", 2024-09-18T04:38:14.397 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000109917 2024-09-18T04:38:14.398 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:14.398 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:14.398 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:14.398 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.398 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.398 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:14.398 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786613+0000", 2024-09-18T04:38:14.398 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 46.572957232999997, 2024-09-18T04:38:14.398 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014245760999999999, 2024-09-18T04:38:14.398 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:14.398 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:14.398 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:14.398 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:14.398 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:14.398 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 111 2024-09-18T04:38:14.399 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.399 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:14.399 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.399 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:14.399 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786613+0000", 2024-09-18T04:38:14.399 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:14.399 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.399 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.399 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:14.399 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786613+0000", 2024-09-18T04:38:14.399 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:14.399 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.399 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.399 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:14.400 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786612+0000", 2024-09-18T04:38:14.400 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:14.400 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.400 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.400 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:14.400 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786622+0000", 2024-09-18T04:38:14.400 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4099999999999997e-06 2024-09-18T04:38:14.400 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.400 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.400 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:14.400 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786630+0000", 2024-09-18T04:38:14.400 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4169999999999992e-06 2024-09-18T04:38:14.400 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.400 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.401 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:14.401 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786637+0000", 2024-09-18T04:38:14.401 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8290000000000002e-06 2024-09-18T04:38:14.401 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.401 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.401 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:14.401 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.797099+0000", 2024-09-18T04:38:14.401 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010461747 2024-09-18T04:38:14.401 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.401 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.401 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:14.401 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.797243+0000", 2024-09-18T04:38:14.401 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014388899999999999 2024-09-18T04:38:14.402 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.402 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.402 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:14.402 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.797463+0000", 2024-09-18T04:38:14.402 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000219912 2024-09-18T04:38:14.402 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.402 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.402 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:14.402 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800732+0000", 2024-09-18T04:38:14.402 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0032688719999999999 2024-09-18T04:38:14.402 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.402 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.403 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:14.403 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800751+0000", 2024-09-18T04:38:14.403 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9633999999999999e-05 2024-09-18T04:38:14.403 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.403 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.403 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:14.403 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800824+0000", 2024-09-18T04:38:14.403 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.2805999999999994e-05 2024-09-18T04:38:14.403 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.403 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.404 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:14.404 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800859+0000", 2024-09-18T04:38:14.404 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4952e-05 2024-09-18T04:38:14.404 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:14.404 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:14.404 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:14.404 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.404 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.404 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:14.404 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786649+0000", 2024-09-18T04:38:14.404 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 46.572921823999998, 2024-09-18T04:38:14.404 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.01613442, 2024-09-18T04:38:14.404 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:14.404 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:14.405 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:14.405 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:14.405 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:14.405 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 112 2024-09-18T04:38:14.405 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.405 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:14.405 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.405 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:14.405 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786649+0000", 2024-09-18T04:38:14.405 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:14.405 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.405 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.406 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:14.406 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786649+0000", 2024-09-18T04:38:14.406 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:14.406 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.406 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.406 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:14.406 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786648+0000", 2024-09-18T04:38:14.406 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:14.406 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.406 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.406 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:14.406 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786768+0000", 2024-09-18T04:38:14.406 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000120491 2024-09-18T04:38:14.406 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.406 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.407 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:14.407 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786795+0000", 2024-09-18T04:38:14.407 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6556000000000001e-05 2024-09-18T04:38:14.407 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.407 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.407 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:14.407 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786807+0000", 2024-09-18T04:38:14.407 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1739999999999999e-05 2024-09-18T04:38:14.407 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.407 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.407 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:14.407 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798683+0000", 2024-09-18T04:38:14.408 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011876070000000001 2024-09-18T04:38:14.408 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.408 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.408 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:14.408 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798867+0000", 2024-09-18T04:38:14.408 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000184103 2024-09-18T04:38:14.408 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.408 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.408 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:14.408 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.799082+0000", 2024-09-18T04:38:14.408 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00021476000000000001 2024-09-18T04:38:14.408 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.408 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.408 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:14.409 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802610+0000", 2024-09-18T04:38:14.409 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0035282209999999998 2024-09-18T04:38:14.409 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.409 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.409 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:14.409 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802706+0000", 2024-09-18T04:38:14.409 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5922999999999996e-05 2024-09-18T04:38:14.409 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.409 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.409 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:14.409 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802750+0000", 2024-09-18T04:38:14.409 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4511000000000001e-05 2024-09-18T04:38:14.409 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.410 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.410 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:14.410 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802783+0000", 2024-09-18T04:38:14.410 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2752000000000001e-05 2024-09-18T04:38:14.410 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:14.410 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:14.410 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:14.410 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.410 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.410 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:14.410 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786816+0000", 2024-09-18T04:38:14.410 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 46.572754756999998, 2024-09-18T04:38:14.410 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016197989999999999, 2024-09-18T04:38:14.411 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:14.411 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:14.411 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:14.411 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:14.411 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:14.411 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 113 2024-09-18T04:38:14.411 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.411 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:14.411 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.411 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:14.411 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786816+0000", 2024-09-18T04:38:14.412 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:14.412 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.412 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.412 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:14.412 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786816+0000", 2024-09-18T04:38:14.412 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:14.412 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.412 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.412 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:14.412 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786815+0000", 2024-09-18T04:38:14.412 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:14.412 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.413 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.413 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:14.413 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786821+0000", 2024-09-18T04:38:14.413 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9529999999999996e-06 2024-09-18T04:38:14.413 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.413 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.413 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:14.413 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786841+0000", 2024-09-18T04:38:14.413 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9743000000000001e-05 2024-09-18T04:38:14.413 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.413 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.413 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:14.413 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786848+0000", 2024-09-18T04:38:14.414 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4900000000000003e-06 2024-09-18T04:38:14.414 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.414 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.414 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:14.414 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.799747+0000", 2024-09-18T04:38:14.414 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012898902 2024-09-18T04:38:14.414 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.414 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.414 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:14.414 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.799898+0000", 2024-09-18T04:38:14.414 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000150877 2024-09-18T04:38:14.414 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.414 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.414 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:14.415 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800095+0000", 2024-09-18T04:38:14.415 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000197023 2024-09-18T04:38:14.415 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.415 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.415 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:14.415 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802829+0000", 2024-09-18T04:38:14.415 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0027343480000000002 2024-09-18T04:38:14.415 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.415 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.415 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:14.415 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802847+0000", 2024-09-18T04:38:14.415 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7283e-05 2024-09-18T04:38:14.415 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.415 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.416 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:14.416 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802974+0000", 2024-09-18T04:38:14.416 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012749199999999999 2024-09-18T04:38:14.416 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.416 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.416 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:14.416 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803014+0000", 2024-09-18T04:38:14.416 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9576000000000002e-05 2024-09-18T04:38:14.416 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:14.416 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:14.416 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:14.416 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.416 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.416 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:14.417 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:38:14.417 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 46.572669165999997, 2024-09-18T04:38:14.417 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017399218000000001, 2024-09-18T04:38:14.417 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:14.417 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:14.417 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:14.417 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:14.417 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:14.417 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 114 2024-09-18T04:38:14.417 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.417 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:14.417 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.417 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:14.417 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:38:14.418 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:14.418 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.418 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.418 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:14.418 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:38:14.418 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:14.418 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.418 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.418 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:14.418 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786900+0000", 2024-09-18T04:38:14.418 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:14.418 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.419 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.419 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:14.419 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786908+0000", 2024-09-18T04:38:14.419 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8199999999999997e-06 2024-09-18T04:38:14.419 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.419 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.419 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:14.419 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786925+0000", 2024-09-18T04:38:14.419 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6849000000000001e-05 2024-09-18T04:38:14.419 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.419 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.419 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:14.419 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786934+0000", 2024-09-18T04:38:14.419 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6299999999999993e-06 2024-09-18T04:38:14.419 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.420 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.420 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:14.420 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800926+0000", 2024-09-18T04:38:14.420 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013991632 2024-09-18T04:38:14.420 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.420 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.420 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:14.420 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801060+0000", 2024-09-18T04:38:14.420 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013378199999999999 2024-09-18T04:38:14.420 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.420 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.420 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:14.420 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801166+0000", 2024-09-18T04:38:14.421 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010607099999999999 2024-09-18T04:38:14.421 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.421 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.421 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:14.421 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803026+0000", 2024-09-18T04:38:14.421 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001860469 2024-09-18T04:38:14.421 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.421 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.421 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:14.421 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804245+0000", 2024-09-18T04:38:14.421 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0012185900000000001 2024-09-18T04:38:14.421 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.421 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.421 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:14.422 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804278+0000", 2024-09-18T04:38:14.422 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3272000000000003e-05 2024-09-18T04:38:14.422 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.422 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.422 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:14.422 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804300+0000", 2024-09-18T04:38:14.422 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2266e-05 2024-09-18T04:38:14.422 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:14.422 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:14.422 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:14.422 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.422 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.422 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:14.422 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787566+0000", 2024-09-18T04:38:14.422 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 46.572004206999999, 2024-09-18T04:38:14.423 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017124120999999999, 2024-09-18T04:38:14.423 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:14.423 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:14.423 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:14.423 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:14.423 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:14.423 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 115 2024-09-18T04:38:14.423 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.423 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:14.423 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.423 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:14.423 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787566+0000", 2024-09-18T04:38:14.423 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:14.423 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.424 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.424 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:14.424 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787566+0000", 2024-09-18T04:38:14.424 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:14.424 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.424 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.424 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:14.424 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787564+0000", 2024-09-18T04:38:14.424 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:14.424 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.424 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.424 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:14.424 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787751+0000", 2024-09-18T04:38:14.424 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000186873 2024-09-18T04:38:14.425 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.425 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.425 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:14.425 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787772+0000", 2024-09-18T04:38:14.425 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.126e-05 2024-09-18T04:38:14.425 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.425 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.425 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:14.425 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787794+0000", 2024-09-18T04:38:14.425 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1418999999999999e-05 2024-09-18T04:38:14.425 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.425 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.425 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:14.426 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801659+0000", 2024-09-18T04:38:14.426 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013865125000000001 2024-09-18T04:38:14.426 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.426 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.426 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:14.426 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801750+0000", 2024-09-18T04:38:14.426 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.1427000000000004e-05 2024-09-18T04:38:14.426 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.426 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.426 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:14.426 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801894+0000", 2024-09-18T04:38:14.426 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000143514 2024-09-18T04:38:14.426 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.426 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.427 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:14.427 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804313+0000", 2024-09-18T04:38:14.427 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0024190879999999998 2024-09-18T04:38:14.427 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.427 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.427 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:14.427 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804644+0000", 2024-09-18T04:38:14.427 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00033100799999999999 2024-09-18T04:38:14.427 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.427 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.427 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:14.427 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804672+0000", 2024-09-18T04:38:14.427 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7988e-05 2024-09-18T04:38:14.427 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.428 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.428 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:14.428 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804690+0000", 2024-09-18T04:38:14.428 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8298999999999999e-05 2024-09-18T04:38:14.428 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:14.428 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:14.428 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:14.428 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.428 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.428 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:14.428 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:38:14.428 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 46.571756268000001, 2024-09-18T04:38:14.428 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017521662, 2024-09-18T04:38:14.429 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:14.429 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:14.429 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:14.429 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:14.429 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:14.429 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 116 2024-09-18T04:38:14.429 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.429 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:14.429 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.429 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:14.429 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:38:14.429 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:14.429 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.430 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.430 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:14.430 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:38:14.430 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:14.430 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.430 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.430 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:14.430 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787813+0000", 2024-09-18T04:38:14.430 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:14.430 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.430 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.430 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:14.431 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787824+0000", 2024-09-18T04:38:14.431 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1157e-05 2024-09-18T04:38:14.431 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.431 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.431 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:14.431 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787840+0000", 2024-09-18T04:38:14.431 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6269e-05 2024-09-18T04:38:14.431 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.431 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.431 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:14.431 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787853+0000", 2024-09-18T04:38:14.431 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3156000000000001e-05 2024-09-18T04:38:14.431 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.431 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.432 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:14.432 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803110+0000", 2024-09-18T04:38:14.432 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015256736 2024-09-18T04:38:14.432 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.432 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.432 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:14.432 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803197+0000", 2024-09-18T04:38:14.432 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7195999999999997e-05 2024-09-18T04:38:14.432 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.432 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.432 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:14.432 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803308+0000", 2024-09-18T04:38:14.432 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111005 2024-09-18T04:38:14.432 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.433 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.433 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:14.433 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804316+0000", 2024-09-18T04:38:14.433 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010073949999999999 2024-09-18T04:38:14.433 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.433 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.433 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:14.433 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805232+0000", 2024-09-18T04:38:14.433 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091636300000000001 2024-09-18T04:38:14.433 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.433 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.433 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:14.433 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805277+0000", 2024-09-18T04:38:14.434 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4864000000000003e-05 2024-09-18T04:38:14.434 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.434 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.434 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:14.434 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805336+0000", 2024-09-18T04:38:14.434 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8968000000000002e-05 2024-09-18T04:38:14.434 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:14.434 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:14.434 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:14.434 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.434 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.434 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:14.434 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:38:14.435 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 46.571704212999997, 2024-09-18T04:38:14.435 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017983453, 2024-09-18T04:38:14.435 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:14.435 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:14.435 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:14.435 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:14.435 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:14.435 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 117 2024-09-18T04:38:14.435 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.435 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:14.435 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.435 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:14.435 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:38:14.435 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:14.436 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.436 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.436 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:14.436 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:38:14.436 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:14.436 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.436 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.436 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:14.436 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787865+0000", 2024-09-18T04:38:14.436 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:14.436 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.436 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.436 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:14.436 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787889+0000", 2024-09-18T04:38:14.437 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4244999999999999e-05 2024-09-18T04:38:14.437 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.437 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.437 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:14.437 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787905+0000", 2024-09-18T04:38:14.437 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5739999999999998e-05 2024-09-18T04:38:14.437 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.437 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.437 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:14.437 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787918+0000", 2024-09-18T04:38:14.437 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.252e-05 2024-09-18T04:38:14.437 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.437 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.438 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:14.438 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803693+0000", 2024-09-18T04:38:14.438 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015774872999999998 2024-09-18T04:38:14.438 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.438 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.438 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:14.438 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803775+0000", 2024-09-18T04:38:14.438 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2761000000000003e-05 2024-09-18T04:38:14.438 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.438 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.438 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:14.438 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803887+0000", 2024-09-18T04:38:14.438 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111708 2024-09-18T04:38:14.439 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.439 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.439 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:14.439 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804823+0000", 2024-09-18T04:38:14.439 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00093625100000000003 2024-09-18T04:38:14.439 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.439 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.439 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:14.439 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805804+0000", 2024-09-18T04:38:14.439 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00098114699999999992 2024-09-18T04:38:14.439 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.439 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.439 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:14.440 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805831+0000", 2024-09-18T04:38:14.440 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6401999999999999e-05 2024-09-18T04:38:14.440 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.440 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.440 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:14.440 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805850+0000", 2024-09-18T04:38:14.440 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8862e-05 2024-09-18T04:38:14.440 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:14.440 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:14.440 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:14.440 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.440 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.440 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:14.441 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.230830+0000", 2024-09-18T04:38:14.441 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 46.128740334, 2024-09-18T04:38:14.441 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011537146999999999, 2024-09-18T04:38:14.441 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:14.441 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:14.441 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:14.441 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:14.441 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:14.441 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 135 2024-09-18T04:38:14.441 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.441 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:14.441 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.441 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:14.441 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230830+0000", 2024-09-18T04:38:14.442 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:14.442 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.442 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.442 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:14.442 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230830+0000", 2024-09-18T04:38:14.442 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:14.442 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.442 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.442 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:14.442 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230829+0000", 2024-09-18T04:38:14.442 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:14.442 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.443 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.443 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:14.443 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230835+0000", 2024-09-18T04:38:14.443 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8660000000000002e-06 2024-09-18T04:38:14.443 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.443 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.443 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:14.443 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230844+0000", 2024-09-18T04:38:14.443 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9339999999999995e-06 2024-09-18T04:38:14.443 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.443 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.443 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:14.443 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230853+0000", 2024-09-18T04:38:14.443 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4419999999999998e-06 2024-09-18T04:38:14.444 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.444 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.444 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:14.444 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.237658+0000", 2024-09-18T04:38:14.444 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0068055310000000001 2024-09-18T04:38:14.444 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.444 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.444 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:14.444 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.237837+0000", 2024-09-18T04:38:14.444 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017943200000000001 2024-09-18T04:38:14.444 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.444 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.445 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:14.445 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.238112+0000", 2024-09-18T04:38:14.445 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00027488600000000001 2024-09-18T04:38:14.445 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.445 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.445 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:14.445 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.240505+0000", 2024-09-18T04:38:14.445 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0023930539999999999 2024-09-18T04:38:14.445 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.445 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.445 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:14.445 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242157+0000", 2024-09-18T04:38:14.445 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001651135 2024-09-18T04:38:14.445 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.446 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.446 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:14.446 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242281+0000", 2024-09-18T04:38:14.446 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012451300000000001 2024-09-18T04:38:14.446 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.446 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.446 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:14.446 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242367+0000", 2024-09-18T04:38:14.446 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6178000000000001e-05 2024-09-18T04:38:14.446 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:14.446 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:14.446 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:14.446 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.446 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.447 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:14.447 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.230858+0000", 2024-09-18T04:38:14.447 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 46.128712446000002, 2024-09-18T04:38:14.447 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013005821000000001, 2024-09-18T04:38:14.447 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:14.447 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:14.447 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:14.447 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:14.447 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:14.447 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 136 2024-09-18T04:38:14.447 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.447 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:14.447 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.447 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:14.448 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230858+0000", 2024-09-18T04:38:14.448 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:14.448 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.448 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.448 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:14.448 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230858+0000", 2024-09-18T04:38:14.448 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:14.448 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.448 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.448 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:14.448 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230858+0000", 2024-09-18T04:38:14.448 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:14.449 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.449 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.449 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:14.449 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230861+0000", 2024-09-18T04:38:14.449 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7459999999999998e-06 2024-09-18T04:38:14.449 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.449 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.449 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:14.449 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230877+0000", 2024-09-18T04:38:14.449 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5979000000000001e-05 2024-09-18T04:38:14.449 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.449 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.450 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:14.450 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230885+0000", 2024-09-18T04:38:14.450 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.5979999999999999e-06 2024-09-18T04:38:14.450 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.450 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.450 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:14.450 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.239023+0000", 2024-09-18T04:38:14.450 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0081380710000000002 2024-09-18T04:38:14.450 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.450 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.450 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:14.450 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.239216+0000", 2024-09-18T04:38:14.450 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019274799999999999 2024-09-18T04:38:14.451 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.451 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.451 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:14.451 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.239417+0000", 2024-09-18T04:38:14.451 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020093300000000001 2024-09-18T04:38:14.451 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.451 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.451 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:14.451 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242494+0000", 2024-09-18T04:38:14.451 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003077177 2024-09-18T04:38:14.451 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.451 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.451 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:14.452 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.243668+0000", 2024-09-18T04:38:14.452 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001174027 2024-09-18T04:38:14.452 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.452 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.452 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:14.452 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.243777+0000", 2024-09-18T04:38:14.452 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010955600000000001 2024-09-18T04:38:14.452 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.452 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.452 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:14.452 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.243864+0000", 2024-09-18T04:38:14.452 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6441999999999997e-05 2024-09-18T04:38:14.452 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:14.452 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:14.453 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:14.453 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.453 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.453 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:14.453 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.230901+0000", 2024-09-18T04:38:14.453 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 46.128669873, 2024-09-18T04:38:14.453 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014120882, 2024-09-18T04:38:14.453 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:14.453 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:14.453 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:14.453 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:14.453 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:14.453 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 137 2024-09-18T04:38:14.454 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.454 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:14.454 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.454 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:14.454 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230901+0000", 2024-09-18T04:38:14.454 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:14.454 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.454 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.454 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:14.454 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230901+0000", 2024-09-18T04:38:14.454 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:14.454 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.454 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.455 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:14.455 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230900+0000", 2024-09-18T04:38:14.455 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:14.455 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.455 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.455 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:14.455 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231002+0000", 2024-09-18T04:38:14.455 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000102392 2024-09-18T04:38:14.455 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.455 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.455 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:14.455 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231029+0000", 2024-09-18T04:38:14.455 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7166999999999999e-05 2024-09-18T04:38:14.456 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.456 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.456 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:14.456 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231036+0000", 2024-09-18T04:38:14.456 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7259999999999996e-06 2024-09-18T04:38:14.456 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.456 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.456 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:14.456 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.240562+0000", 2024-09-18T04:38:14.456 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0095257970000000008 2024-09-18T04:38:14.456 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.456 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.456 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:14.456 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.240730+0000", 2024-09-18T04:38:14.457 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016767099999999999 2024-09-18T04:38:14.457 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.457 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.457 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:14.457 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.240927+0000", 2024-09-18T04:38:14.457 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019711899999999999 2024-09-18T04:38:14.457 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.457 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.457 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:14.457 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244885+0000", 2024-09-18T04:38:14.457 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0039577340000000001 2024-09-18T04:38:14.457 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.457 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.458 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:14.458 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244899+0000", 2024-09-18T04:38:14.458 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4403000000000001e-05 2024-09-18T04:38:14.458 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.458 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.458 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:14.458 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244973+0000", 2024-09-18T04:38:14.458 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4177999999999994e-05 2024-09-18T04:38:14.458 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.458 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.458 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:14.458 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.245021+0000", 2024-09-18T04:38:14.459 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8344999999999999e-05 2024-09-18T04:38:14.459 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:14.459 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:14.459 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:14.459 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.459 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.459 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:14.459 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:38:14.459 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 46.128527081999998, 2024-09-18T04:38:14.459 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016528505999999998, 2024-09-18T04:38:14.459 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:14.459 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:14.459 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:14.459 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:14.460 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:14.460 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 138 2024-09-18T04:38:14.460 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.460 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:14.460 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.460 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:14.460 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:38:14.460 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:14.460 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.460 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.460 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:14.460 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:38:14.460 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:14.460 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.461 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.461 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:14.461 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:38:14.461 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:14.461 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.461 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.461 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:14.461 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231047+0000", 2024-09-18T04:38:14.461 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5399999999999997e-06 2024-09-18T04:38:14.461 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.461 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.461 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:14.462 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231068+0000", 2024-09-18T04:38:14.462 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0769000000000001e-05 2024-09-18T04:38:14.462 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.462 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.462 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:14.462 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231074+0000", 2024-09-18T04:38:14.462 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6180000000000003e-06 2024-09-18T04:38:14.462 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.462 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.462 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:14.462 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242556+0000", 2024-09-18T04:38:14.462 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011482654 2024-09-18T04:38:14.462 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.462 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.462 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:14.462 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242666+0000", 2024-09-18T04:38:14.463 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011003200000000001 2024-09-18T04:38:14.463 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.463 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.463 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:14.463 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242924+0000", 2024-09-18T04:38:14.463 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00025785899999999999 2024-09-18T04:38:14.463 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.463 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.463 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:14.463 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247263+0000", 2024-09-18T04:38:14.463 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0043387649999999996 2024-09-18T04:38:14.463 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.463 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.464 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:14.464 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247412+0000", 2024-09-18T04:38:14.464 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000149276 2024-09-18T04:38:14.464 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.464 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.464 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:14.464 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247522+0000", 2024-09-18T04:38:14.464 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000109788 2024-09-18T04:38:14.464 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.464 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.464 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:14.464 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247572+0000", 2024-09-18T04:38:14.464 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9808999999999997e-05 2024-09-18T04:38:14.465 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:14.465 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:14.465 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:14.465 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.465 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.465 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:14.465 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231083+0000", 2024-09-18T04:38:14.465 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 46.128487618999998, 2024-09-18T04:38:14.465 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0166731, 2024-09-18T04:38:14.465 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:14.465 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:14.465 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:14.466 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:14.466 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:14.466 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 139 2024-09-18T04:38:14.466 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.466 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:14.466 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.466 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:14.466 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231083+0000", 2024-09-18T04:38:14.466 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:14.466 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.466 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.466 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:14.466 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231083+0000", 2024-09-18T04:38:14.466 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:14.466 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.467 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.467 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:14.467 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231082+0000", 2024-09-18T04:38:14.467 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:14.467 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.467 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.467 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:14.467 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231090+0000", 2024-09-18T04:38:14.467 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1280000000000008e-06 2024-09-18T04:38:14.467 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.467 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.467 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:14.467 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231097+0000", 2024-09-18T04:38:14.467 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5119999999999997e-06 2024-09-18T04:38:14.468 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.468 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.468 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:14.468 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231106+0000", 2024-09-18T04:38:14.468 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9230000000000008e-06 2024-09-18T04:38:14.468 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.468 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.468 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:14.468 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.243977+0000", 2024-09-18T04:38:14.468 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012870754 2024-09-18T04:38:14.468 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.468 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.468 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:14.469 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244112+0000", 2024-09-18T04:38:14.469 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000135575 2024-09-18T04:38:14.469 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.469 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.469 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:14.469 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244265+0000", 2024-09-18T04:38:14.469 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015279900000000001 2024-09-18T04:38:14.469 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.469 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.469 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:14.469 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247649+0000", 2024-09-18T04:38:14.469 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003383896 2024-09-18T04:38:14.469 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.469 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.469 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:14.470 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247656+0000", 2024-09-18T04:38:14.470 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8920000000000003e-06 2024-09-18T04:38:14.470 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.470 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.470 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:14.470 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247692+0000", 2024-09-18T04:38:14.470 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6470000000000001e-05 2024-09-18T04:38:14.470 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.470 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.470 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:14.470 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247756+0000", 2024-09-18T04:38:14.470 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3676999999999994e-05 2024-09-18T04:38:14.470 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:14.471 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:14.471 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:14.471 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.471 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.471 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:14.471 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231190+0000", 2024-09-18T04:38:14.471 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 46.128380692999997, 2024-09-18T04:38:14.471 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016793556000000001, 2024-09-18T04:38:14.471 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:14.471 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:14.471 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:14.471 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:14.472 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:14.472 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 140 2024-09-18T04:38:14.472 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.472 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:14.472 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.472 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:14.472 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231190+0000", 2024-09-18T04:38:14.472 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:14.472 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.472 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.472 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:14.472 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231190+0000", 2024-09-18T04:38:14.473 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:14.473 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.473 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.473 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:14.473 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231189+0000", 2024-09-18T04:38:14.473 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:14.473 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.473 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.473 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:14.473 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231273+0000", 2024-09-18T04:38:14.473 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4445999999999998e-05 2024-09-18T04:38:14.473 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.473 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.473 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:14.474 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231282+0000", 2024-09-18T04:38:14.474 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8859999999999993e-06 2024-09-18T04:38:14.474 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.474 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.474 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:14.474 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231288+0000", 2024-09-18T04:38:14.474 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9819999999999996e-06 2024-09-18T04:38:14.474 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.474 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.474 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:14.474 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.245119+0000", 2024-09-18T04:38:14.474 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.01383121 2024-09-18T04:38:14.474 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.475 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.475 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:14.475 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.245247+0000", 2024-09-18T04:38:14.475 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012763299999999999 2024-09-18T04:38:14.475 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.475 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.475 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:14.475 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.245401+0000", 2024-09-18T04:38:14.475 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015399200000000001 2024-09-18T04:38:14.475 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.475 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.475 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:14.475 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247766+0000", 2024-09-18T04:38:14.475 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0023649510000000001 2024-09-18T04:38:14.476 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.476 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.476 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:14.476 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247904+0000", 2024-09-18T04:38:14.476 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013836499999999999 2024-09-18T04:38:14.476 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.476 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.476 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:14.476 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247950+0000", 2024-09-18T04:38:14.476 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5201000000000003e-05 2024-09-18T04:38:14.476 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.476 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.476 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:14.476 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247983+0000", 2024-09-18T04:38:14.477 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3686000000000002e-05 2024-09-18T04:38:14.477 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:14.477 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:14.477 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:14.477 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.477 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.477 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:14.477 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:38:14.477 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 46.128269950000004, 2024-09-18T04:38:14.477 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017344024, 2024-09-18T04:38:14.477 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:14.477 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:14.477 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:14.478 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:14.478 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:14.478 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 141 2024-09-18T04:38:14.478 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.478 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:14.478 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.478 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:14.478 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:38:14.478 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:14.478 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.478 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.478 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:14.478 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:38:14.479 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:14.479 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.479 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.479 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:14.479 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:38:14.479 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:14.479 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.479 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.479 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:14.479 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231309+0000", 2024-09-18T04:38:14.479 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5419999999999994e-06 2024-09-18T04:38:14.479 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.479 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.480 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:14.480 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231323+0000", 2024-09-18T04:38:14.480 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3108000000000001e-05 2024-09-18T04:38:14.480 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.480 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.480 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:14.480 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231340+0000", 2024-09-18T04:38:14.480 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.696e-05 2024-09-18T04:38:14.480 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.480 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.480 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:14.480 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.246143+0000", 2024-09-18T04:38:14.480 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014803320999999999 2024-09-18T04:38:14.480 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.481 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.481 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:14.481 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.246323+0000", 2024-09-18T04:38:14.481 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00018061600000000001 2024-09-18T04:38:14.481 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.481 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.481 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:14.481 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.246530+0000", 2024-09-18T04:38:14.481 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020625 2024-09-18T04:38:14.481 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.481 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.481 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:14.481 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248002+0000", 2024-09-18T04:38:14.482 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0014719379999999999 2024-09-18T04:38:14.482 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.482 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.482 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:14.482 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248582+0000", 2024-09-18T04:38:14.482 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00058020599999999999 2024-09-18T04:38:14.482 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.482 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.482 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:14.482 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248618+0000", 2024-09-18T04:38:14.482 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6559e-05 2024-09-18T04:38:14.482 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.482 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.483 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:14.483 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248645+0000", 2024-09-18T04:38:14.483 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6080000000000001e-05 2024-09-18T04:38:14.483 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:14.483 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:14.483 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:14.483 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.483 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.483 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-18T04:38:14.483 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.878463+0000", 2024-09-18T04:38:14.483 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 45.481107915999999, 2024-09-18T04:38:14.483 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010882697, 2024-09-18T04:38:14.483 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:14.484 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:14.484 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:14.484 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:14.484 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:14.484 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 159 2024-09-18T04:38:14.484 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.484 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:14.484 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.484 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:14.484 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878463+0000", 2024-09-18T04:38:14.484 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:14.484 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.484 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.485 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:14.485 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878463+0000", 2024-09-18T04:38:14.485 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:14.485 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.485 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.485 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:14.485 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878462+0000", 2024-09-18T04:38:14.485 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:14.485 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.485 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.485 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:14.485 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878467+0000", 2024-09-18T04:38:14.486 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4099999999999999e-06 2024-09-18T04:38:14.486 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.486 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.486 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:14.486 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878490+0000", 2024-09-18T04:38:14.486 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3159999999999998e-05 2024-09-18T04:38:14.486 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.486 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.486 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:14.486 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878511+0000", 2024-09-18T04:38:14.486 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1092000000000001e-05 2024-09-18T04:38:14.486 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.486 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.487 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:14.487 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.887163+0000", 2024-09-18T04:38:14.487 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.008651845 2024-09-18T04:38:14.487 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.487 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.487 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:14.487 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.887273+0000", 2024-09-18T04:38:14.487 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000109823 2024-09-18T04:38:14.487 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.487 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.487 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:14.487 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.887441+0000", 2024-09-18T04:38:14.487 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016777599999999999 2024-09-18T04:38:14.488 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.488 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.488 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:14.488 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.888634+0000", 2024-09-18T04:38:14.488 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001192839 2024-09-18T04:38:14.488 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.488 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.488 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:14.488 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.889153+0000", 2024-09-18T04:38:14.488 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00051953700000000004 2024-09-18T04:38:14.488 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.488 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.489 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:14.489 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.889234+0000", 2024-09-18T04:38:14.489 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0802000000000003e-05 2024-09-18T04:38:14.489 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:14.489 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:14.489 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:14.489 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.889345+0000", 2024-09-18T04:38:14.489 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011112000000000001 2024-09-18T04:38:14.489 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:14.489 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:14.489 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:14.489 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:14.489 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:14.490 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:14.491 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:38:14.492 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:14.486+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:14.592 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:14.587+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:14.653 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:14.653 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:14.653 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:38:14.653 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:14.663 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:38:14.693 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:14.686+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:14.768 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:14.768 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:14.768 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:38:14.768 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:38:14.768 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:14.778 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:38:14.793 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:14.791+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:14.805 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-09-18T04:38:14.880 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:14.880 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:38:14.880 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:38:14.880 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [] 2024-09-18T04:38:14.880 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:14.889 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:38:14.894 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:14.890+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:14.995 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:14.990+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:15.000 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:15.000 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:15.000 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:38:15.000 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:15.009 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:38:15.096 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:15.090+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:15.118 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:15.119 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:15.119 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:38:15.119 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:38:15.119 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:15.128 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:38:15.197 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:15.190+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:15.238 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:15.239 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:38:15.239 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:38:15.239 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:38:15.239 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.239 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-18T04:38:15.239 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:38:15.239 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 40.765528484000001, 2024-09-18T04:38:15.239 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0095781189999999995, 2024-09-18T04:38:15.239 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:15.239 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:15.239 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:15.239 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:38:15.239 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:38:15.240 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 1 2024-09-18T04:38:15.240 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.240 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:15.240 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.240 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:15.240 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:38:15.240 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:15.240 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.240 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.240 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:15.240 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:38:15.240 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:15.240 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.240 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.241 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:15.241 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469862+0000", 2024-09-18T04:38:15.241 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:15.241 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.241 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.241 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:15.241 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469879+0000", 2024-09-18T04:38:15.241 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7431999999999998e-05 2024-09-18T04:38:15.241 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.241 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.241 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:15.241 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469896+0000", 2024-09-18T04:38:15.241 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6742999999999999e-05 2024-09-18T04:38:15.241 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.242 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.242 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:15.242 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469915+0000", 2024-09-18T04:38:15.242 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8991000000000002e-05 2024-09-18T04:38:15.242 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.242 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.242 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:15.242 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470397+0000", 2024-09-18T04:38:15.242 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048167799999999998 2024-09-18T04:38:15.242 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.242 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.242 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-09-18T04:38:15.242 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470412+0000", 2024-09-18T04:38:15.243 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5526e-05 2024-09-18T04:38:15.243 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.243 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.243 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:15.243 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473279+0000", 2024-09-18T04:38:15.243 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0028667469999999998 2024-09-18T04:38:15.243 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.243 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.243 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:15.243 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473441+0000", 2024-09-18T04:38:15.243 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000162082 2024-09-18T04:38:15.243 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.243 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.243 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:38:15.244 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473573+0000", 2024-09-18T04:38:15.244 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000132032 2024-09-18T04:38:15.244 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.244 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.244 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:15.244 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474826+0000", 2024-09-18T04:38:15.244 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001252717 2024-09-18T04:38:15.244 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.244 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.244 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:15.244 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479307+0000", 2024-09-18T04:38:15.244 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044813220000000003 2024-09-18T04:38:15.244 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.244 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.245 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:15.245 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479366+0000", 2024-09-18T04:38:15.245 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9024999999999998e-05 2024-09-18T04:38:15.245 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.245 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.245 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:15.245 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479443+0000", 2024-09-18T04:38:15.245 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6339000000000002e-05 2024-09-18T04:38:15.245 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:15.245 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:15.245 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:15.245 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.245 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.245 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.4214.0:1 2.3 e18/17 ondisk, result = 0)", 2024-09-18T04:38:15.246 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.474724+0000", 2024-09-18T04:38:15.246 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 40.760669120000003, 2024-09-18T04:38:15.246 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000105358, 2024-09-18T04:38:15.246 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:15.246 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:38:15.246 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:15.246 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.246 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:15.246 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474724+0000", 2024-09-18T04:38:15.246 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:15.246 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.246 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.246 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:15.246 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474724+0000", 2024-09-18T04:38:15.246 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:15.247 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.247 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.247 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:15.247 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474723+0000", 2024-09-18T04:38:15.247 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:15.247 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.247 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.247 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:15.247 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474731+0000", 2024-09-18T04:38:15.247 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2339999999999994e-06 2024-09-18T04:38:15.247 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.247 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.247 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:15.247 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474748+0000", 2024-09-18T04:38:15.247 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6603e-05 2024-09-18T04:38:15.248 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.248 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.248 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:15.248 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474760+0000", 2024-09-18T04:38:15.248 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2215000000000001e-05 2024-09-18T04:38:15.248 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.248 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.248 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:15.248 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474814+0000", 2024-09-18T04:38:15.248 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4267000000000001e-05 2024-09-18T04:38:15.248 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.248 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.248 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:15.248 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474821+0000", 2024-09-18T04:38:15.248 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0929999999999997e-06 2024-09-18T04:38:15.249 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.249 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.249 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:15.249 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474829+0000", 2024-09-18T04:38:15.249 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1200000000000002e-06 2024-09-18T04:38:15.249 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:15.249 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:15.249 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:15.249 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.249 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.249 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18)", 2024-09-18T04:38:15.249 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.482346+0000", 2024-09-18T04:38:15.249 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 40.753046687999998, 2024-09-18T04:38:15.250 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020435599999999999, 2024-09-18T04:38:15.250 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:15.250 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-09-18T04:38:15.250 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:15.250 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:38:15.250 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:38:15.250 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 2 2024-09-18T04:38:15.250 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.250 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:15.250 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.250 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:15.250 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482346+0000", 2024-09-18T04:38:15.250 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:15.250 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.250 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.251 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:15.251 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482346+0000", 2024-09-18T04:38:15.251 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:15.251 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.251 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.251 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:15.251 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482345+0000", 2024-09-18T04:38:15.251 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:15.251 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.251 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.251 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:15.251 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482360+0000", 2024-09-18T04:38:15.251 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4854999999999999e-05 2024-09-18T04:38:15.251 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.252 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.252 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:15.252 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482377+0000", 2024-09-18T04:38:15.252 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7785999999999999e-05 2024-09-18T04:38:15.252 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.252 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.252 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:15.252 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482387+0000", 2024-09-18T04:38:15.252 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6800000000000005e-06 2024-09-18T04:38:15.252 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.252 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.252 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:15.252 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482441+0000", 2024-09-18T04:38:15.253 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4046999999999999e-05 2024-09-18T04:38:15.253 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.253 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.253 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:15.253 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482551+0000", 2024-09-18T04:38:15.253 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000109468 2024-09-18T04:38:15.253 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:15.253 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:15.253 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:15.253 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.253 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.253 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18)", 2024-09-18T04:38:15.253 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.482853+0000", 2024-09-18T04:38:15.253 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 40.752540070000002, 2024-09-18T04:38:15.253 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00029815799999999997, 2024-09-18T04:38:15.254 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:15.254 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-09-18T04:38:15.254 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:15.254 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:38:15.254 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:38:15.254 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 3 2024-09-18T04:38:15.254 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.254 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:15.254 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.254 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:15.254 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482853+0000", 2024-09-18T04:38:15.254 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:15.254 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.254 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.254 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:15.254 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482853+0000", 2024-09-18T04:38:15.255 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:15.255 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.255 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.255 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:15.255 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482852+0000", 2024-09-18T04:38:15.255 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:15.255 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.255 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.255 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:15.255 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482859+0000", 2024-09-18T04:38:15.255 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0380000000000002e-06 2024-09-18T04:38:15.255 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.255 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.255 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:15.255 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482880+0000", 2024-09-18T04:38:15.256 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1243e-05 2024-09-18T04:38:15.256 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.256 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.256 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:15.256 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482896+0000", 2024-09-18T04:38:15.256 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5977e-05 2024-09-18T04:38:15.256 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.256 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.256 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:15.256 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482970+0000", 2024-09-18T04:38:15.256 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4603000000000004e-05 2024-09-18T04:38:15.256 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.256 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.257 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:15.257 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.483151+0000", 2024-09-18T04:38:15.257 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00018059799999999999 2024-09-18T04:38:15.257 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:15.257 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:15.257 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:15.257 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.257 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.257 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-18T04:38:15.257 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:35.468698+0000", 2024-09-18T04:38:15.257 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 39.766695233, 2024-09-18T04:38:15.257 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0039954129999999997, 2024-09-18T04:38:15.257 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:15.257 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:15.257 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:15.258 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:38:15.258 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:38:15.258 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 5 2024-09-18T04:38:15.258 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.258 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:15.258 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.258 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:15.258 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468698+0000", 2024-09-18T04:38:15.258 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:15.258 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.258 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.258 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:15.258 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468698+0000", 2024-09-18T04:38:15.259 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:15.259 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.259 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.259 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:15.259 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468696+0000", 2024-09-18T04:38:15.259 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:15.259 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.259 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.259 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:15.259 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468714+0000", 2024-09-18T04:38:15.259 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8743e-05 2024-09-18T04:38:15.259 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.259 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.259 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:15.260 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468748+0000", 2024-09-18T04:38:15.260 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3339e-05 2024-09-18T04:38:15.260 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.260 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.260 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:15.260 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468771+0000", 2024-09-18T04:38:15.260 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3119999999999999e-05 2024-09-18T04:38:15.260 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.260 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.260 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:15.260 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468922+0000", 2024-09-18T04:38:15.260 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000151383 2024-09-18T04:38:15.260 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.261 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.261 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:15.261 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.469278+0000", 2024-09-18T04:38:15.261 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00035628599999999998 2024-09-18T04:38:15.261 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.261 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.261 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:15.261 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.469629+0000", 2024-09-18T04:38:15.261 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00035014300000000003 2024-09-18T04:38:15.261 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.261 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.261 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:15.261 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.471620+0000", 2024-09-18T04:38:15.262 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001991278 2024-09-18T04:38:15.262 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.262 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.262 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:15.262 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472440+0000", 2024-09-18T04:38:15.262 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00081998599999999998 2024-09-18T04:38:15.262 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.262 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.262 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:15.262 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472592+0000", 2024-09-18T04:38:15.262 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015227400000000001 2024-09-18T04:38:15.262 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.262 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.262 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:15.263 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472693+0000", 2024-09-18T04:38:15.263 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000101074 2024-09-18T04:38:15.263 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:15.263 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:15.263 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:15.263 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.263 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.263 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.4214.0:5 2.2 e18/17 ondisk, result = 0)", 2024-09-18T04:38:15.263 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:35.472226+0000", 2024-09-18T04:38:15.263 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 39.763167332999998, 2024-09-18T04:38:15.263 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048505299999999999, 2024-09-18T04:38:15.263 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:15.263 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:38:15.263 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:15.263 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.264 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:15.264 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472226+0000", 2024-09-18T04:38:15.264 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:15.264 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.264 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.264 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:15.264 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472226+0000", 2024-09-18T04:38:15.264 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:15.264 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.264 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.265 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:15.265 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472224+0000", 2024-09-18T04:38:15.265 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:15.265 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.265 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.265 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:15.265 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472238+0000", 2024-09-18T04:38:15.265 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.468e-05 2024-09-18T04:38:15.265 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.265 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.265 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:15.265 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472272+0000", 2024-09-18T04:38:15.265 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3586e-05 2024-09-18T04:38:15.265 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.266 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.266 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:15.266 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472297+0000", 2024-09-18T04:38:15.266 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5282999999999998e-05 2024-09-18T04:38:15.266 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.266 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.266 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:15.266 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472396+0000", 2024-09-18T04:38:15.266 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.8394999999999996e-05 2024-09-18T04:38:15.266 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.266 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.266 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:15.266 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472421+0000", 2024-09-18T04:38:15.266 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5479000000000001e-05 2024-09-18T04:38:15.267 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.267 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.267 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:15.267 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472711+0000", 2024-09-18T04:38:15.267 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000289714 2024-09-18T04:38:15.267 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:15.267 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:15.267 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:15.267 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.267 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.267 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-18T04:38:15.267 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:38:15.267 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 38.752520064000002, 2024-09-18T04:38:15.267 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082699560000000002, 2024-09-18T04:38:15.267 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:15.268 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:15.268 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:15.268 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:38:15.268 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:38:15.268 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 7 2024-09-18T04:38:15.268 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.268 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:15.268 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.268 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:15.268 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:38:15.268 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:15.268 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.268 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.268 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:15.269 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:38:15.269 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:15.269 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.269 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.269 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:15.269 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482870+0000", 2024-09-18T04:38:15.269 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999967 2024-09-18T04:38:15.269 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.269 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.269 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:15.269 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482889+0000", 2024-09-18T04:38:15.269 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.965e-05 2024-09-18T04:38:15.269 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.270 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.270 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:15.270 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482938+0000", 2024-09-18T04:38:15.270 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8548999999999999e-05 2024-09-18T04:38:15.270 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.270 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.270 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:15.270 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482963+0000", 2024-09-18T04:38:15.270 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5075999999999998e-05 2024-09-18T04:38:15.270 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.270 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.270 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:15.270 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483121+0000", 2024-09-18T04:38:15.271 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015819400000000001 2024-09-18T04:38:15.271 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.271 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.271 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op must wait for map", 2024-09-18T04:38:15.271 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483138+0000", 2024-09-18T04:38:15.271 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7068999999999999e-05 2024-09-18T04:38:15.271 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.271 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.271 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:15.271 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488666+0000", 2024-09-18T04:38:15.271 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0055282049999999996 2024-09-18T04:38:15.271 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.271 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.271 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:15.272 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488783+0000", 2024-09-18T04:38:15.272 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000116737 2024-09-18T04:38:15.272 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.272 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.272 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:15.272 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488991+0000", 2024-09-18T04:38:15.272 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020781400000000001 2024-09-18T04:38:15.272 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.272 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.272 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:15.272 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490554+0000", 2024-09-18T04:38:15.272 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0015628370000000001 2024-09-18T04:38:15.272 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.272 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.273 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:15.273 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491019+0000", 2024-09-18T04:38:15.273 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000465608 2024-09-18T04:38:15.273 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.273 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.273 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:15.273 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491087+0000", 2024-09-18T04:38:15.273 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7988e-05 2024-09-18T04:38:15.273 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.273 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.273 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:15.273 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491143+0000", 2024-09-18T04:38:15.273 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5538999999999998e-05 2024-09-18T04:38:15.274 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:15.274 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:15.274 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:15.274 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.274 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.274 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.4214.0:7 2.2 e19/17 ondisk, result = 0)", 2024-09-18T04:38:15.274 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:36.490861+0000", 2024-09-18T04:38:15.274 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 38.744532442000001, 2024-09-18T04:38:15.274 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00029639400000000001, 2024-09-18T04:38:15.274 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:15.274 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:38:15.274 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:15.274 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.275 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:15.275 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490861+0000", 2024-09-18T04:38:15.275 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:15.275 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.275 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.275 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:15.275 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490861+0000", 2024-09-18T04:38:15.275 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:15.275 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.275 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.275 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:15.275 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490859+0000", 2024-09-18T04:38:15.275 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:15.275 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.276 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.276 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:15.276 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490867+0000", 2024-09-18T04:38:15.276 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.3089999999999996e-06 2024-09-18T04:38:15.276 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.276 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.276 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:15.276 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490909+0000", 2024-09-18T04:38:15.276 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1919000000000003e-05 2024-09-18T04:38:15.276 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.276 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.276 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:15.276 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490922+0000", 2024-09-18T04:38:15.277 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2836999999999999e-05 2024-09-18T04:38:15.277 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.277 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.277 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:15.277 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491003+0000", 2024-09-18T04:38:15.277 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0681999999999998e-05 2024-09-18T04:38:15.277 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.277 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.277 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:15.277 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491013+0000", 2024-09-18T04:38:15.277 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0220999999999999e-05 2024-09-18T04:38:15.277 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.277 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.277 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:15.277 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491157+0000", 2024-09-18T04:38:15.278 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014401799999999999 2024-09-18T04:38:15.278 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:15.278 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:15.278 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:15.278 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.278 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.278 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "MOSDPGPushReply(2.3 25/24 [PushReplyOp(2:c4c92e5a:::rbd_trash:head)])", 2024-09-18T04:38:15.278 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:51.680323+0000", 2024-09-18T04:38:15.278 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 23.555070066999999, 2024-09-18T04:38:15.278 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00054411499999999996, 2024-09-18T04:38:15.278 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:15.278 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-09-18T04:38:15.278 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:15.278 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.278 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:15.278 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680323+0000", 2024-09-18T04:38:15.279 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:15.279 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.279 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.279 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:15.279 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680323+0000", 2024-09-18T04:38:15.279 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:15.279 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.279 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.279 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:15.279 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680322+0000", 2024-09-18T04:38:15.279 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:15.279 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.279 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.279 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:15.280 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680333+0000", 2024-09-18T04:38:15.280 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1153e-05 2024-09-18T04:38:15.280 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.280 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.280 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:15.280 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680357+0000", 2024-09-18T04:38:15.280 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3252999999999999e-05 2024-09-18T04:38:15.280 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.280 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.280 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:15.280 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680378+0000", 2024-09-18T04:38:15.280 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1503000000000001e-05 2024-09-18T04:38:15.280 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.280 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.281 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:15.281 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680522+0000", 2024-09-18T04:38:15.281 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014410700000000001 2024-09-18T04:38:15.281 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:15.281 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:15.281 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:15.281 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680867+0000", 2024-09-18T04:38:15.281 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00034475899999999998 2024-09-18T04:38:15.281 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:15.281 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:15.281 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:15.281 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:15.281 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:15.282 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:15.283 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:38:15.298 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:15.295+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:15.399 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:15.394+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:15.403 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:15.404 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:15.404 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:38:15.404 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:15.414 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:38:15.500 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:15.494+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:15.521 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:15.521 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:15.521 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:38:15.522 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:38:15.522 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:15.531 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:38:15.601 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:15.595+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:15.644 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:15.644 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:38:15.644 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:38:15.644 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:38:15.644 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.644 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:30 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'16, mlcod=15'14)", 2024-09-18T04:38:15.644 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.486330+0000", 2024-09-18T04:38:15.644 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 48.154993656000002, 2024-09-18T04:38:15.644 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026383700000000001, 2024-09-18T04:38:15.645 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:15.645 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:15.645 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:15.645 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.645 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:15.645 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486330+0000", 2024-09-18T04:38:15.645 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:15.645 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.645 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.645 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:15.645 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486330+0000", 2024-09-18T04:38:15.645 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:15.645 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.646 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.646 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:15.646 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486328+0000", 2024-09-18T04:38:15.646 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:15.646 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.646 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.646 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:15.646 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486342+0000", 2024-09-18T04:38:15.646 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3783e-05 2024-09-18T04:38:15.646 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.646 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.646 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:15.646 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486373+0000", 2024-09-18T04:38:15.647 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1362999999999998e-05 2024-09-18T04:38:15.647 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.647 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.647 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:15.647 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486387+0000", 2024-09-18T04:38:15.647 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3219e-05 2024-09-18T04:38:15.647 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.647 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.647 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:15.647 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486489+0000", 2024-09-18T04:38:15.647 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010232499999999999 2024-09-18T04:38:15.647 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.647 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.647 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:15.648 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486524+0000", 2024-09-18T04:38:15.648 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4972000000000003e-05 2024-09-18T04:38:15.648 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.648 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.648 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:15.648 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.488902+0000", 2024-09-18T04:38:15.648 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023776240000000001 2024-09-18T04:38:15.648 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.648 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.648 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:15.648 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.488968+0000", 2024-09-18T04:38:15.648 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6761000000000007e-05 2024-09-18T04:38:15.648 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:15.649 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:15.649 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:15.649 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.649 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.649 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:35 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'21, mlcod=15'17)", 2024-09-18T04:38:15.649 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.491817+0000", 2024-09-18T04:38:15.649 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 48.149506737999999, 2024-09-18T04:38:15.649 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023524739999999998, 2024-09-18T04:38:15.649 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:15.649 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:15.649 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:15.649 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.649 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:15.649 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491817+0000", 2024-09-18T04:38:15.650 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:15.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.650 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.650 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:15.650 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491817+0000", 2024-09-18T04:38:15.650 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:15.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.650 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.650 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:15.650 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491815+0000", 2024-09-18T04:38:15.650 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:15.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.650 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.651 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:15.651 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491890+0000", 2024-09-18T04:38:15.651 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4500999999999997e-05 2024-09-18T04:38:15.651 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.651 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.651 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:15.651 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491916+0000", 2024-09-18T04:38:15.651 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5666000000000001e-05 2024-09-18T04:38:15.651 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.651 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.651 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:15.651 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491928+0000", 2024-09-18T04:38:15.651 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2585999999999999e-05 2024-09-18T04:38:15.651 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.652 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.652 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:15.652 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.492000+0000", 2024-09-18T04:38:15.652 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2309000000000002e-05 2024-09-18T04:38:15.652 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.652 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.652 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:15.652 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.492027+0000", 2024-09-18T04:38:15.652 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6251e-05 2024-09-18T04:38:15.652 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.652 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.652 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:15.652 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.494114+0000", 2024-09-18T04:38:15.653 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020875260000000001 2024-09-18T04:38:15.653 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.653 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.653 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:15.653 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.494169+0000", 2024-09-18T04:38:15.653 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5145000000000001e-05 2024-09-18T04:38:15.653 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:15.653 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:15.653 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:15.653 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.653 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.653 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:57 1.0 e15/14 1:30306672:devicehealth::main.db.0000000000000000:head v 15'35, mlcod=15'33)", 2024-09-18T04:38:15.653 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.529997+0000", 2024-09-18T04:38:15.654 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 48.111326884, 2024-09-18T04:38:15.654 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002623869, 2024-09-18T04:38:15.654 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:15.654 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:15.654 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:15.654 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.654 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:15.654 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.529997+0000", 2024-09-18T04:38:15.654 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:15.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.654 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.654 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:15.654 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.529997+0000", 2024-09-18T04:38:15.654 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:15.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.655 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.655 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:15.655 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.529995+0000", 2024-09-18T04:38:15.655 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:15.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.655 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.655 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:15.655 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530116+0000", 2024-09-18T04:38:15.655 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000121141 2024-09-18T04:38:15.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.655 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.655 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:15.655 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530146+0000", 2024-09-18T04:38:15.656 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0423e-05 2024-09-18T04:38:15.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.656 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.656 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:15.656 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530159+0000", 2024-09-18T04:38:15.656 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2802999999999999e-05 2024-09-18T04:38:15.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.656 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.656 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:15.656 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530218+0000", 2024-09-18T04:38:15.656 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8461000000000002e-05 2024-09-18T04:38:15.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.656 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.656 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:15.656 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530253+0000", 2024-09-18T04:38:15.657 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4796999999999999e-05 2024-09-18T04:38:15.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.657 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.657 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:15.657 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.532559+0000", 2024-09-18T04:38:15.657 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002306939 2024-09-18T04:38:15.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.657 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.657 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:15.657 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.532621+0000", 2024-09-18T04:38:15.657 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1131999999999997e-05 2024-09-18T04:38:15.657 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:15.657 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:15.658 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:15.658 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.658 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.658 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:68 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'42, mlcod=15'39)", 2024-09-18T04:38:15.658 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.546404+0000", 2024-09-18T04:38:15.658 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 48.094919472999997, 2024-09-18T04:38:15.658 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024499769999999999, 2024-09-18T04:38:15.658 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:15.658 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:15.658 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:15.658 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.658 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:15.659 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546404+0000", 2024-09-18T04:38:15.659 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:15.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.659 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.659 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:15.659 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546404+0000", 2024-09-18T04:38:15.659 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:15.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.659 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.659 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:15.659 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546403+0000", 2024-09-18T04:38:15.659 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:15.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.659 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.660 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:15.660 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546416+0000", 2024-09-18T04:38:15.660 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.306e-05 2024-09-18T04:38:15.660 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.660 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.660 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:15.660 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546465+0000", 2024-09-18T04:38:15.660 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9441999999999999e-05 2024-09-18T04:38:15.660 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.660 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.660 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:15.660 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546480+0000", 2024-09-18T04:38:15.660 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4919999999999999e-05 2024-09-18T04:38:15.660 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.661 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.661 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:15.661 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546541+0000", 2024-09-18T04:38:15.661 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1085000000000002e-05 2024-09-18T04:38:15.661 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.661 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.661 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:15.661 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546569+0000", 2024-09-18T04:38:15.661 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7803000000000001e-05 2024-09-18T04:38:15.661 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.661 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.661 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:15.662 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.548725+0000", 2024-09-18T04:38:15.662 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021556629999999999 2024-09-18T04:38:15.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.662 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.662 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:15.662 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.548854+0000", 2024-09-18T04:38:15.662 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000129524 2024-09-18T04:38:15.662 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:15.662 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:15.662 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:15.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.662 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.662 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:69 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'43, mlcod=15'39)", 2024-09-18T04:38:15.662 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.547520+0000", 2024-09-18T04:38:15.663 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 48.093803397000002, 2024-09-18T04:38:15.663 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023506820000000002, 2024-09-18T04:38:15.663 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:15.663 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:15.663 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:15.663 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.663 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:15.663 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547520+0000", 2024-09-18T04:38:15.663 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:15.663 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.663 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.663 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:15.663 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547520+0000", 2024-09-18T04:38:15.663 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:15.664 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.664 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.664 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:15.664 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547519+0000", 2024-09-18T04:38:15.664 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:15.664 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.664 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.664 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:15.664 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547614+0000", 2024-09-18T04:38:15.664 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5235000000000006e-05 2024-09-18T04:38:15.664 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.664 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.664 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:15.664 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547640+0000", 2024-09-18T04:38:15.664 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5698999999999999e-05 2024-09-18T04:38:15.665 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.665 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.665 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:15.665 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547651+0000", 2024-09-18T04:38:15.665 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.134e-05 2024-09-18T04:38:15.665 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.665 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.665 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:15.665 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547805+0000", 2024-09-18T04:38:15.665 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015426099999999999 2024-09-18T04:38:15.665 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.665 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.665 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:15.666 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547844+0000", 2024-09-18T04:38:15.666 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8229999999999998e-05 2024-09-18T04:38:15.666 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.666 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.666 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:15.666 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.549810+0000", 2024-09-18T04:38:15.666 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0019660519999999998 2024-09-18T04:38:15.666 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.666 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.666 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:15.666 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.549871+0000", 2024-09-18T04:38:15.666 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1152e-05 2024-09-18T04:38:15.666 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:15.667 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:15.667 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:15.667 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.667 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.667 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:83 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'53, mlcod=15'50)", 2024-09-18T04:38:15.667 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.575989+0000", 2024-09-18T04:38:15.667 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 48.065334448999998, 2024-09-18T04:38:15.667 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023991310000000001, 2024-09-18T04:38:15.667 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:15.667 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:15.667 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:15.667 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.667 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:15.668 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.575989+0000", 2024-09-18T04:38:15.668 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:15.668 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.668 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.668 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:15.668 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.575989+0000", 2024-09-18T04:38:15.668 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:15.668 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.668 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.668 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:15.668 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.575988+0000", 2024-09-18T04:38:15.668 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:15.668 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.668 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.668 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:15.669 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576001+0000", 2024-09-18T04:38:15.669 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2751e-05 2024-09-18T04:38:15.669 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.669 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.669 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:15.669 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576023+0000", 2024-09-18T04:38:15.669 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2453999999999999e-05 2024-09-18T04:38:15.669 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.669 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.669 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:15.669 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576038+0000", 2024-09-18T04:38:15.669 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5125e-05 2024-09-18T04:38:15.669 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.669 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.670 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:15.670 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576161+0000", 2024-09-18T04:38:15.670 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000122543 2024-09-18T04:38:15.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.670 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.670 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:15.670 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576221+0000", 2024-09-18T04:38:15.670 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9948000000000003e-05 2024-09-18T04:38:15.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.670 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.670 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:15.670 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.578277+0000", 2024-09-18T04:38:15.670 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020567559999999999 2024-09-18T04:38:15.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.670 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.671 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:15.671 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.578388+0000", 2024-09-18T04:38:15.671 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011089100000000001 2024-09-18T04:38:15.671 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:15.671 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:15.671 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:15.671 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.671 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.671 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:86 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'56, mlcod=15'52)", 2024-09-18T04:38:15.671 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.580261+0000", 2024-09-18T04:38:15.671 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 48.061062812999999, 2024-09-18T04:38:15.671 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002418108, 2024-09-18T04:38:15.671 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:15.672 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:15.672 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:15.672 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.672 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:15.672 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580261+0000", 2024-09-18T04:38:15.672 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:15.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.672 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.672 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:15.672 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580261+0000", 2024-09-18T04:38:15.672 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:15.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.672 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.673 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:15.673 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580260+0000", 2024-09-18T04:38:15.673 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:15.673 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.673 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.673 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:15.673 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580269+0000", 2024-09-18T04:38:15.673 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5109999999999992e-06 2024-09-18T04:38:15.673 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.673 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.673 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:15.673 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580288+0000", 2024-09-18T04:38:15.673 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8783999999999998e-05 2024-09-18T04:38:15.673 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.674 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.674 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:15.674 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580296+0000", 2024-09-18T04:38:15.674 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3250000000000008e-06 2024-09-18T04:38:15.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.674 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.674 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:15.674 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580351+0000", 2024-09-18T04:38:15.674 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4132000000000003e-05 2024-09-18T04:38:15.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.674 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.674 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:15.674 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580381+0000", 2024-09-18T04:38:15.675 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0263000000000001e-05 2024-09-18T04:38:15.675 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.675 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.675 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:15.675 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.582615+0000", 2024-09-18T04:38:15.675 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0022339109999999999 2024-09-18T04:38:15.675 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.675 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.676 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:15.676 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.582679+0000", 2024-09-18T04:38:15.676 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4182000000000003e-05 2024-09-18T04:38:15.676 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:15.676 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:15.676 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:15.676 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.676 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.676 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:105 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'67, mlcod=15'64)", 2024-09-18T04:38:15.676 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.789089+0000", 2024-09-18T04:38:15.676 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 47.852234643000003, 2024-09-18T04:38:15.676 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024384229999999999, 2024-09-18T04:38:15.676 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:15.677 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:15.677 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:15.677 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.677 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:15.677 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789089+0000", 2024-09-18T04:38:15.677 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:15.677 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.677 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.677 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:15.677 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789089+0000", 2024-09-18T04:38:15.677 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:15.677 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.678 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.678 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:15.678 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789088+0000", 2024-09-18T04:38:15.678 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:15.678 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.678 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.678 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:15.678 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789107+0000", 2024-09-18T04:38:15.678 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9017000000000001e-05 2024-09-18T04:38:15.678 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.678 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.678 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:15.678 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789139+0000", 2024-09-18T04:38:15.678 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2261999999999997e-05 2024-09-18T04:38:15.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.679 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.679 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:15.679 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789153+0000", 2024-09-18T04:38:15.679 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4025999999999999e-05 2024-09-18T04:38:15.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.679 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.679 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:15.679 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789255+0000", 2024-09-18T04:38:15.679 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000101969 2024-09-18T04:38:15.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.679 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.679 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:15.679 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789314+0000", 2024-09-18T04:38:15.680 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8867999999999999e-05 2024-09-18T04:38:15.680 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.680 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.680 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:15.680 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.791453+0000", 2024-09-18T04:38:15.680 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002138938 2024-09-18T04:38:15.680 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.680 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.680 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:15.680 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.791527+0000", 2024-09-18T04:38:15.680 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4464999999999999e-05 2024-09-18T04:38:15.680 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:15.680 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:15.680 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:15.680 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.681 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.681 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:106 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'68, mlcod=15'64)", 2024-09-18T04:38:15.681 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.790295+0000", 2024-09-18T04:38:15.681 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 47.851028792000001, 2024-09-18T04:38:15.681 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024379390000000001, 2024-09-18T04:38:15.681 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:15.681 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:15.681 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:15.681 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.681 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:15.681 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790295+0000", 2024-09-18T04:38:15.681 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:15.681 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.682 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.682 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:15.682 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790295+0000", 2024-09-18T04:38:15.682 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:15.682 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.682 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.682 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:15.682 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790294+0000", 2024-09-18T04:38:15.682 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:15.682 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.682 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.682 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:15.682 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790390+0000", 2024-09-18T04:38:15.683 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6537000000000001e-05 2024-09-18T04:38:15.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.683 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.683 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:15.683 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790418+0000", 2024-09-18T04:38:15.683 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8218e-05 2024-09-18T04:38:15.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.683 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.683 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:15.683 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790431+0000", 2024-09-18T04:38:15.683 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2571999999999999e-05 2024-09-18T04:38:15.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.683 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.683 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:15.684 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790534+0000", 2024-09-18T04:38:15.684 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000103433 2024-09-18T04:38:15.684 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.684 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.684 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:15.684 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790565+0000", 2024-09-18T04:38:15.684 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0409999999999999e-05 2024-09-18T04:38:15.684 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.684 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.684 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:15.684 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.792651+0000", 2024-09-18T04:38:15.684 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020865250000000001 2024-09-18T04:38:15.684 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.684 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.685 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:15.685 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.792733+0000", 2024-09-18T04:38:15.685 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1477999999999995e-05 2024-09-18T04:38:15.685 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:15.685 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:15.685 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:15.685 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.685 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.685 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:109 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'71, mlcod=15'68)", 2024-09-18T04:38:15.685 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.795250+0000", 2024-09-18T04:38:15.685 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 47.846073777999997, 2024-09-18T04:38:15.685 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023990449999999998, 2024-09-18T04:38:15.685 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:15.686 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:15.686 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:15.686 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.686 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:15.686 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795250+0000", 2024-09-18T04:38:15.686 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:15.686 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.686 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.686 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:15.686 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795250+0000", 2024-09-18T04:38:15.686 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:15.686 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.686 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.686 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:15.686 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795248+0000", 2024-09-18T04:38:15.687 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:15.687 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.687 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.687 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:15.687 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795405+0000", 2024-09-18T04:38:15.687 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000157007 2024-09-18T04:38:15.687 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.687 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.687 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:15.687 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795422+0000", 2024-09-18T04:38:15.687 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7013000000000001e-05 2024-09-18T04:38:15.687 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.687 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.688 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:15.688 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795432+0000", 2024-09-18T04:38:15.688 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0448999999999999e-05 2024-09-18T04:38:15.688 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.688 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.688 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:15.688 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795534+0000", 2024-09-18T04:38:15.688 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000101195 2024-09-18T04:38:15.688 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.688 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.688 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:15.688 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795587+0000", 2024-09-18T04:38:15.688 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3356000000000002e-05 2024-09-18T04:38:15.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.689 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.689 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:15.689 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.797560+0000", 2024-09-18T04:38:15.689 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001973023 2024-09-18T04:38:15.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.689 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.689 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:15.689 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.797649+0000", 2024-09-18T04:38:15.689 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8864999999999997e-05 2024-09-18T04:38:15.689 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:15.689 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:15.689 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:15.690 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.690 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.690 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:115 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'77, mlcod=15'72)", 2024-09-18T04:38:15.690 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.803339+0000", 2024-09-18T04:38:15.690 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 47.837985060000001, 2024-09-18T04:38:15.690 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002385093, 2024-09-18T04:38:15.690 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:15.690 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:15.690 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:15.690 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.690 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:15.690 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803339+0000", 2024-09-18T04:38:15.690 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:15.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.691 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.691 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:15.691 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803339+0000", 2024-09-18T04:38:15.691 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:15.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.691 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.691 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:15.691 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803337+0000", 2024-09-18T04:38:15.691 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:15.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.691 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.691 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:15.691 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803475+0000", 2024-09-18T04:38:15.692 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013718199999999999 2024-09-18T04:38:15.692 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.692 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.692 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:15.692 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803507+0000", 2024-09-18T04:38:15.692 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2351000000000002e-05 2024-09-18T04:38:15.692 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.692 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.692 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:15.692 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803533+0000", 2024-09-18T04:38:15.692 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6010999999999999e-05 2024-09-18T04:38:15.692 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.692 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.692 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:15.693 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803609+0000", 2024-09-18T04:38:15.693 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.64e-05 2024-09-18T04:38:15.693 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.693 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.693 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:15.693 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803671+0000", 2024-09-18T04:38:15.693 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1467e-05 2024-09-18T04:38:15.693 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.693 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.693 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:15.693 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.805676+0000", 2024-09-18T04:38:15.693 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020047849999999998 2024-09-18T04:38:15.693 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.694 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.694 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:15.694 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.805724+0000", 2024-09-18T04:38:15.694 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8151e-05 2024-09-18T04:38:15.694 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:15.694 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:15.694 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:15.694 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.694 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.694 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:118 1.0 e15/14 1:30306672:devicehealth::main.db.0000000000000000:head v 15'80, mlcod=15'78)", 2024-09-18T04:38:15.694 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.807930+0000", 2024-09-18T04:38:15.694 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 47.833393557000001, 2024-09-18T04:38:15.694 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028002259999999998, 2024-09-18T04:38:15.694 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:15.695 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:15.695 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:15.695 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.695 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:15.695 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.807930+0000", 2024-09-18T04:38:15.695 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:15.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.695 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.695 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:15.695 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.807930+0000", 2024-09-18T04:38:15.695 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:15.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.695 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.695 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:15.696 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.807929+0000", 2024-09-18T04:38:15.696 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:15.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.696 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.696 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:15.696 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808024+0000", 2024-09-18T04:38:15.696 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5258000000000002e-05 2024-09-18T04:38:15.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.696 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.696 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:15.696 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808044+0000", 2024-09-18T04:38:15.696 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0203e-05 2024-09-18T04:38:15.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.697 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.697 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:15.697 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808053+0000", 2024-09-18T04:38:15.697 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2689999999999999e-06 2024-09-18T04:38:15.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.697 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.697 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:15.697 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808137+0000", 2024-09-18T04:38:15.697 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4441e-05 2024-09-18T04:38:15.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.697 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.697 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:15.697 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808172+0000", 2024-09-18T04:38:15.697 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4453999999999999e-05 2024-09-18T04:38:15.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.698 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.698 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:15.698 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.810663+0000", 2024-09-18T04:38:15.698 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024910269999999998 2024-09-18T04:38:15.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.698 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.698 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:15.698 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.810730+0000", 2024-09-18T04:38:15.698 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7658999999999997e-05 2024-09-18T04:38:15.698 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:15.698 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:15.698 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:15.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.699 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.699 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:129 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'87, mlcod=15'85)", 2024-09-18T04:38:15.699 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.231689+0000", 2024-09-18T04:38:15.699 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 47.409634543999999, 2024-09-18T04:38:15.699 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002376502, 2024-09-18T04:38:15.699 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:15.699 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:15.699 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:15.699 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.699 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:15.699 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231689+0000", 2024-09-18T04:38:15.699 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:15.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.700 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.700 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:15.700 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231689+0000", 2024-09-18T04:38:15.700 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:15.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.700 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.700 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:15.700 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231687+0000", 2024-09-18T04:38:15.700 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:15.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.700 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.700 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:15.700 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231702+0000", 2024-09-18T04:38:15.700 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4949999999999999e-05 2024-09-18T04:38:15.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.701 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.701 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:15.701 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231735+0000", 2024-09-18T04:38:15.701 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2276000000000001e-05 2024-09-18T04:38:15.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.701 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.701 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:15.701 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231750+0000", 2024-09-18T04:38:15.701 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5193e-05 2024-09-18T04:38:15.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.701 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.701 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:15.701 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231872+0000", 2024-09-18T04:38:15.701 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000122127 2024-09-18T04:38:15.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.702 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.702 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:15.702 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231936+0000", 2024-09-18T04:38:15.702 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4089000000000003e-05 2024-09-18T04:38:15.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.702 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.702 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:15.702 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234002+0000", 2024-09-18T04:38:15.702 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002065575 2024-09-18T04:38:15.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.702 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.702 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:15.702 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234066+0000", 2024-09-18T04:38:15.703 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3944999999999998e-05 2024-09-18T04:38:15.703 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:15.703 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:15.703 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:15.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.703 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.703 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:130 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'88, mlcod=15'85)", 2024-09-18T04:38:15.703 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.232853+0000", 2024-09-18T04:38:15.703 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 47.408470929000003, 2024-09-18T04:38:15.703 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027351739999999999, 2024-09-18T04:38:15.703 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:15.703 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:15.703 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:15.704 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.704 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:15.704 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232853+0000", 2024-09-18T04:38:15.704 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:15.704 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.704 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.704 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:15.704 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232853+0000", 2024-09-18T04:38:15.704 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:15.704 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.704 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.704 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:15.704 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232851+0000", 2024-09-18T04:38:15.704 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:15.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.705 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.705 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:15.705 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232866+0000", 2024-09-18T04:38:15.705 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4946000000000001e-05 2024-09-18T04:38:15.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.705 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.705 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:15.705 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232897+0000", 2024-09-18T04:38:15.705 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0982999999999998e-05 2024-09-18T04:38:15.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.705 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.705 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:15.705 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232915+0000", 2024-09-18T04:38:15.706 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8476000000000001e-05 2024-09-18T04:38:15.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.706 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.706 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:15.706 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.233029+0000", 2024-09-18T04:38:15.706 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011305400000000001 2024-09-18T04:38:15.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.706 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.706 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:15.706 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.233079+0000", 2024-09-18T04:38:15.706 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0668999999999999e-05 2024-09-18T04:38:15.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.707 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.707 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:15.707 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235534+0000", 2024-09-18T04:38:15.707 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024546030000000001 2024-09-18T04:38:15.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.707 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.707 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:15.707 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235588+0000", 2024-09-18T04:38:15.707 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4098999999999998e-05 2024-09-18T04:38:15.707 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:15.707 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:15.707 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:15.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.707 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.707 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:131 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'89, mlcod=15'85)", 2024-09-18T04:38:15.708 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.234089+0000", 2024-09-18T04:38:15.708 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 47.407235075000003, 2024-09-18T04:38:15.708 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026975609999999998, 2024-09-18T04:38:15.708 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:15.708 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:15.708 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:15.708 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.708 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:15.708 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234089+0000", 2024-09-18T04:38:15.708 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:15.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.708 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.708 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:15.709 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234089+0000", 2024-09-18T04:38:15.709 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:15.709 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.709 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.709 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:15.709 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234087+0000", 2024-09-18T04:38:15.709 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:15.709 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.709 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.709 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:15.709 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234145+0000", 2024-09-18T04:38:15.709 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8687999999999998e-05 2024-09-18T04:38:15.709 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.710 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.710 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:15.710 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234177+0000", 2024-09-18T04:38:15.710 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1526999999999998e-05 2024-09-18T04:38:15.710 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.710 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.710 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:15.710 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234190+0000", 2024-09-18T04:38:15.710 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2965999999999999e-05 2024-09-18T04:38:15.710 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.710 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.710 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:15.710 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234309+0000", 2024-09-18T04:38:15.710 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011875399999999999 2024-09-18T04:38:15.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.711 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.711 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:15.711 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234365+0000", 2024-09-18T04:38:15.711 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6436e-05 2024-09-18T04:38:15.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.711 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.711 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:15.711 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236675+0000", 2024-09-18T04:38:15.711 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023099520000000001 2024-09-18T04:38:15.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.711 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.711 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:15.712 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236786+0000", 2024-09-18T04:38:15.712 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000111025 2024-09-18T04:38:15.712 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:15.712 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:15.712 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:15.712 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.712 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.712 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:132 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'90, mlcod=15'85)", 2024-09-18T04:38:15.712 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.235300+0000", 2024-09-18T04:38:15.712 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 47.406023625000003, 2024-09-18T04:38:15.712 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025982599999999998, 2024-09-18T04:38:15.712 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:15.712 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:15.713 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:15.713 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.713 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:15.713 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235300+0000", 2024-09-18T04:38:15.713 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:15.713 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.713 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.713 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:15.713 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235300+0000", 2024-09-18T04:38:15.713 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:15.713 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.713 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.713 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:15.713 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235298+0000", 2024-09-18T04:38:15.714 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:15.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.714 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.714 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:15.714 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235318+0000", 2024-09-18T04:38:15.714 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9714000000000001e-05 2024-09-18T04:38:15.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.714 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.714 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:15.714 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235337+0000", 2024-09-18T04:38:15.714 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8892000000000001e-05 2024-09-18T04:38:15.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.714 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.715 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:15.715 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235348+0000", 2024-09-18T04:38:15.715 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.129e-05 2024-09-18T04:38:15.715 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.715 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.715 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:15.715 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235681+0000", 2024-09-18T04:38:15.715 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00033323000000000001 2024-09-18T04:38:15.715 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.715 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.715 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:15.715 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235733+0000", 2024-09-18T04:38:15.715 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1391999999999999e-05 2024-09-18T04:38:15.715 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.716 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.716 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:15.716 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237839+0000", 2024-09-18T04:38:15.716 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002105647 2024-09-18T04:38:15.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.716 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.716 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:15.716 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237898+0000", 2024-09-18T04:38:15.716 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9759000000000002e-05 2024-09-18T04:38:15.716 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:15.716 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:15.716 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:15.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.717 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.717 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:133 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'91, mlcod=15'86)", 2024-09-18T04:38:15.717 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.236920+0000", 2024-09-18T04:38:15.717 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 47.404403737000003, 2024-09-18T04:38:15.717 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024318410000000001, 2024-09-18T04:38:15.717 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:15.717 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:15.717 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:15.717 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.717 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:15.717 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236920+0000", 2024-09-18T04:38:15.717 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:15.717 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.717 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.718 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:15.718 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236920+0000", 2024-09-18T04:38:15.718 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:15.718 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.718 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.718 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:15.718 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236918+0000", 2024-09-18T04:38:15.718 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:15.718 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.718 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.718 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:15.718 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236931+0000", 2024-09-18T04:38:15.718 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2683999999999999e-05 2024-09-18T04:38:15.718 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.719 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.719 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:15.719 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236951+0000", 2024-09-18T04:38:15.719 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0292000000000002e-05 2024-09-18T04:38:15.719 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.719 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.719 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:15.719 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236962+0000", 2024-09-18T04:38:15.719 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0703000000000001e-05 2024-09-18T04:38:15.719 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.719 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.719 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:15.719 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237095+0000", 2024-09-18T04:38:15.719 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013312400000000001 2024-09-18T04:38:15.720 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.720 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.720 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:15.720 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237133+0000", 2024-09-18T04:38:15.720 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7496000000000001e-05 2024-09-18T04:38:15.720 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.720 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.720 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:15.720 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.239206+0000", 2024-09-18T04:38:15.720 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020730290000000001 2024-09-18T04:38:15.720 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.720 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.720 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:15.721 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.239352+0000", 2024-09-18T04:38:15.721 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014593000000000001 2024-09-18T04:38:15.721 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:15.721 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:15.721 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:15.721 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.721 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.721 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:134 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'92, mlcod=15'86)", 2024-09-18T04:38:15.721 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.237894+0000", 2024-09-18T04:38:15.721 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 47.403429715000001, 2024-09-18T04:38:15.721 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025299519999999998, 2024-09-18T04:38:15.721 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:15.721 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:15.722 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:15.722 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.722 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:15.722 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237894+0000", 2024-09-18T04:38:15.722 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:15.722 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.722 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.722 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:15.722 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237894+0000", 2024-09-18T04:38:15.722 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:15.722 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.722 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.722 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:15.722 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237892+0000", 2024-09-18T04:38:15.723 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:15.723 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.723 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.723 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:15.723 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237986+0000", 2024-09-18T04:38:15.723 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3341e-05 2024-09-18T04:38:15.723 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.723 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.723 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:15.723 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.238012+0000", 2024-09-18T04:38:15.723 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5843e-05 2024-09-18T04:38:15.723 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.723 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.723 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:15.724 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.238027+0000", 2024-09-18T04:38:15.724 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5469e-05 2024-09-18T04:38:15.724 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.724 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.724 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:15.724 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.238114+0000", 2024-09-18T04:38:15.724 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7483000000000002e-05 2024-09-18T04:38:15.724 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.724 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.724 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:15.724 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.238152+0000", 2024-09-18T04:38:15.724 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7565999999999999e-05 2024-09-18T04:38:15.724 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.725 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.725 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:15.725 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.240297+0000", 2024-09-18T04:38:15.725 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021447620000000001 2024-09-18T04:38:15.725 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.725 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.725 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:15.725 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.240424+0000", 2024-09-18T04:38:15.725 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012705400000000001 2024-09-18T04:38:15.725 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:15.725 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:15.725 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:15.725 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.725 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.726 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:139 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'97, mlcod=15'93)", 2024-09-18T04:38:15.726 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.245692+0000", 2024-09-18T04:38:15.726 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 47.395631573999999, 2024-09-18T04:38:15.726 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024260480000000001, 2024-09-18T04:38:15.726 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:15.726 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:15.726 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:15.726 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.726 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:15.726 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245692+0000", 2024-09-18T04:38:15.726 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:15.726 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.726 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.727 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:15.727 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245692+0000", 2024-09-18T04:38:15.727 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:15.727 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.727 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.727 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:15.727 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245690+0000", 2024-09-18T04:38:15.727 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:15.727 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.727 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.727 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:15.727 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245712+0000", 2024-09-18T04:38:15.727 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1916000000000002e-05 2024-09-18T04:38:15.727 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.728 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.728 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:15.728 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245741+0000", 2024-09-18T04:38:15.728 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8503000000000001e-05 2024-09-18T04:38:15.728 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.728 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.728 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:15.728 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245759+0000", 2024-09-18T04:38:15.728 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.872e-05 2024-09-18T04:38:15.728 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.728 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.728 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:15.728 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245859+0000", 2024-09-18T04:38:15.728 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9641000000000004e-05 2024-09-18T04:38:15.728 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.729 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.729 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:15.729 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245900+0000", 2024-09-18T04:38:15.729 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0589000000000001e-05 2024-09-18T04:38:15.729 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.729 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.729 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:15.729 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.247961+0000", 2024-09-18T04:38:15.729 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020619190000000002 2024-09-18T04:38:15.729 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.729 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.729 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:15.729 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.248118+0000", 2024-09-18T04:38:15.729 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015664300000000001 2024-09-18T04:38:15.729 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:15.730 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:15.730 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:15.730 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.730 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.730 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:144 1.0 e15/14 1:30306672:devicehealth::main.db.0000000000000000:head v 15'102, mlcod=15'98)", 2024-09-18T04:38:15.730 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.252149+0000", 2024-09-18T04:38:15.730 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 47.389174507999996, 2024-09-18T04:38:15.730 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024174959999999999, 2024-09-18T04:38:15.730 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:15.730 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:15.730 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:15.730 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.730 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:15.730 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252149+0000", 2024-09-18T04:38:15.730 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:15.731 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.731 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.731 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:15.731 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252149+0000", 2024-09-18T04:38:15.731 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:15.731 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.731 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.731 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:15.731 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252148+0000", 2024-09-18T04:38:15.731 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:15.731 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.731 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.731 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:15.731 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252225+0000", 2024-09-18T04:38:15.732 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6576999999999995e-05 2024-09-18T04:38:15.732 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.732 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.732 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:15.732 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252242+0000", 2024-09-18T04:38:15.732 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7740999999999998e-05 2024-09-18T04:38:15.732 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.732 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.732 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:15.732 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252254+0000", 2024-09-18T04:38:15.732 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.137e-05 2024-09-18T04:38:15.732 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.732 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.732 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:15.732 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252385+0000", 2024-09-18T04:38:15.733 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013091099999999999 2024-09-18T04:38:15.733 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.733 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.733 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:15.733 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252426+0000", 2024-09-18T04:38:15.733 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1535999999999997e-05 2024-09-18T04:38:15.733 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.733 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.733 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:15.733 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.254498+0000", 2024-09-18T04:38:15.733 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002071477 2024-09-18T04:38:15.733 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:15.733 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:15.733 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:15.734 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.254567+0000", 2024-09-18T04:38:15.734 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9014e-05 2024-09-18T04:38:15.734 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:15.734 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:15.734 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:15.734 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:15.734 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:15.734 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:15.735 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T04:38:15.736 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:38:15.736 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:15.731+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:15.788 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-18T04:38:15.788 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T04:38:15.788 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-09-18T04:38:15.788 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-18T04:38:15.788 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-18T04:38:15.788 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3 2024-09-18T04:38:15.836 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:15.831+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:15.889 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:15.889 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:15.889 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:38:15.889 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:15.899 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:38:15.905 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:15.905 INFO:teuthology.orchestra.run.smithi082.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-18T04:38:15.905 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:15.916 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:38:15.937 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:15.930+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:15.942 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:38:15.960 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:38:15.990 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:38:15.996 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:15.996 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:16.006 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:38:16.012 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:38:16.038 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:16.034+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:16.047 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:38:16.050 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:38:16.050 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:38:16.050 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:38:16.050 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:38:16.050 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:38:16.050 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:38:16.120 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:16.120 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:16.130 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:38:16.140 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:16.134+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:16.236 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:16.236 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:16.240 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:16.234+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:16.245 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:38:16.341 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:16.335+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:16.356 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:16.356 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:16.366 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:38:16.443 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:16.438+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:16.474 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:16.474 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:16.484 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:38:16.544 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:16.539+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:16.599 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:16.600 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:16.609 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:38:16.645 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:16.638+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:16.716 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:16.716 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:16.726 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:38:16.746 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:16.743+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:16.819 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-18T04:38:16.837 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:16.837 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:16.847 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:38:16.847 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:16.842+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:16.948 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:16.943+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:16.950 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:16.951 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:16.960 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:38:17.049 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:17.042+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:17.066 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:17.066 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:17.076 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:38:17.150 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:17.146+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:17.178 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:17.179 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:17.188 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:38:17.251 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:17.246+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:17.298 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:17.298 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:17.308 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:38:17.352 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:17.347+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:17.414 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:17.414 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:17.423 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:38:17.453 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:17.447+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:17.533 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:17.533 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:17.543 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:38:17.590 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:17.587+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:17.655 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:17.651+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:17.661 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:17.662 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:17.672 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:38:17.756 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:17.750+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:17.778 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:17.778 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:17.788 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:38:17.813 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-09-18T04:38:17.857 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:17.851+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:17.895 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:17.895 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:17.906 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:38:17.958 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:17.955+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:18.013 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:18.013 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:18.023 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:38:18.065 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T04:38:18.066 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:18.063+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:18.160 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:18.154+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:18.178 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:18.178 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:18.188 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:38:18.205 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:38:18.233 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:38:18.251 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:38:18.261 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:18.255+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:18.273 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:38:18.295 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:38:18.300 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:18.300 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:38:18.300 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:38:18.300 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:38:18.300 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:18.300 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4214.0:5 2.2 e19/17 2:5cea7035:::rbd_info:head v 19'1, mlcod=0'0)", 2024-09-18T04:38:18.300 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:35.471260+0000", 2024-09-18T04:38:18.300 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 42.827737356999997, 2024-09-18T04:38:18.301 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021025950000000001, 2024-09-18T04:38:18.301 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:18.301 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:18.301 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:18.301 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:18.301 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:18.301 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471260+0000", 2024-09-18T04:38:18.301 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:18.301 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:18.301 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:18.301 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:18.301 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471260+0000", 2024-09-18T04:38:18.301 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:18.302 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:18.302 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:18.302 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:18.302 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471259+0000", 2024-09-18T04:38:18.302 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:18.302 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:18.302 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:18.302 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:18.302 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471275+0000", 2024-09-18T04:38:18.302 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6135999999999999e-05 2024-09-18T04:38:18.302 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:18.302 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:18.302 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:18.303 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471298+0000", 2024-09-18T04:38:18.303 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3632999999999999e-05 2024-09-18T04:38:18.303 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:18.303 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:18.303 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:18.303 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471320+0000", 2024-09-18T04:38:18.303 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1852999999999999e-05 2024-09-18T04:38:18.303 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:18.303 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:18.303 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:18.303 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471445+0000", 2024-09-18T04:38:18.303 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012461699999999999 2024-09-18T04:38:18.303 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:18.304 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:18.304 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:18.304 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471529+0000", 2024-09-18T04:38:18.304 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3875000000000006e-05 2024-09-18T04:38:18.304 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:18.304 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:18.304 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:18.304 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.473317+0000", 2024-09-18T04:38:18.304 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017886899999999999 2024-09-18T04:38:18.304 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:18.304 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:18.304 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:18.305 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.473362+0000", 2024-09-18T04:38:18.305 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4724000000000001e-05 2024-09-18T04:38:18.305 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:18.305 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:18.305 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:18.305 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:18.305 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:18.305 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4214.0:7 2.2 e20/17 2:5cea7035:::rbd_info:head v 20'2, mlcod=0'0)", 2024-09-18T04:38:18.305 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:36.490560+0000", 2024-09-18T04:38:18.305 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 41.808437218000002, 2024-09-18T04:38:18.305 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0015326020000000001, 2024-09-18T04:38:18.305 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:18.305 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:18.306 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:18.306 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:18.306 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:18.306 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490560+0000", 2024-09-18T04:38:18.306 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:18.306 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:18.306 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:18.306 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:18.306 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490560+0000", 2024-09-18T04:38:18.306 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:18.306 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:18.306 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:18.306 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:18.306 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490559+0000", 2024-09-18T04:38:18.307 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:18.307 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:18.307 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:18.307 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:18.307 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490568+0000", 2024-09-18T04:38:18.307 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5699999999999993e-06 2024-09-18T04:38:18.307 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:18.307 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:18.307 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:18.307 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490599+0000", 2024-09-18T04:38:18.307 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1566000000000002e-05 2024-09-18T04:38:18.307 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:18.307 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:18.307 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:18.307 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490609+0000", 2024-09-18T04:38:18.308 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.8700000000000004e-06 2024-09-18T04:38:18.308 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:18.308 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:18.308 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:18.308 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490708+0000", 2024-09-18T04:38:18.308 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9139000000000001e-05 2024-09-18T04:38:18.308 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:18.308 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:18.308 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:18.308 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490736+0000", 2024-09-18T04:38:18.308 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7617e-05 2024-09-18T04:38:18.308 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:18.308 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:18.309 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:18.309 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.492016+0000", 2024-09-18T04:38:18.309 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0012796579999999999 2024-09-18T04:38:18.309 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:18.309 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:18.309 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:18.309 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.492092+0000", 2024-09-18T04:38:18.309 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6571999999999998e-05 2024-09-18T04:38:18.309 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:18.309 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:18.309 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:18.309 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:18.309 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:18.309 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:18.311 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:38:18.318 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:38:18.320 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:38:18.320 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:38:18.320 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:38:18.320 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:38:18.321 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:38:18.321 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:38:18.378 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:18.375+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:18.424 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:18.424 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:18.424 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:38:18.424 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:18.434 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:38:18.463 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:18.459+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:18.537 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:18.538 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:18.538 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:38:18.538 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:38:18.538 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:18.547 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:38:18.563 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:18.559+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:18.653 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:18.653 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:38:18.654 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:38:18.654 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:38:18.654 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:18.654 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4214.0:1 2.3 e18/17 2:c4c92e5a:::rbd_trash:head v 18'1, mlcod=0'0)", 2024-09-18T04:38:18.654 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:34.475123+0000", 2024-09-18T04:38:18.654 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 44.177122730999997, 2024-09-18T04:38:18.654 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00081394599999999996, 2024-09-18T04:38:18.654 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:18.654 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:18.654 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:18.654 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:18.654 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:18.654 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475123+0000", 2024-09-18T04:38:18.655 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:18.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:18.655 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:18.655 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:18.655 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475123+0000", 2024-09-18T04:38:18.655 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:18.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:18.655 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:18.655 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:18.655 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475121+0000", 2024-09-18T04:38:18.655 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:18.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:18.655 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:18.655 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:18.656 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475149+0000", 2024-09-18T04:38:18.656 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7155999999999999e-05 2024-09-18T04:38:18.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:18.656 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:18.656 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:18.656 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475165+0000", 2024-09-18T04:38:18.656 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6160000000000001e-05 2024-09-18T04:38:18.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:18.656 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:18.656 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:18.656 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475183+0000", 2024-09-18T04:38:18.656 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7918e-05 2024-09-18T04:38:18.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:18.656 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:18.657 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:18.657 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475246+0000", 2024-09-18T04:38:18.657 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3176000000000006e-05 2024-09-18T04:38:18.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:18.657 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:18.657 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:18.657 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475279+0000", 2024-09-18T04:38:18.657 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3192999999999999e-05 2024-09-18T04:38:18.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:18.657 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:18.657 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:18.657 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475871+0000", 2024-09-18T04:38:18.658 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00059206099999999995 2024-09-18T04:38:18.658 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:18.658 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:18.658 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:18.658 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475937+0000", 2024-09-18T04:38:18.658 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5439999999999997e-05 2024-09-18T04:38:18.658 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:18.658 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:18.658 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:18.658 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:18.658 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:18.658 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 25/24 [PushOp(2:c4c92e5a:::rbd_trash:head, version: 18'1, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:c4c92e5a:::rbd_trash:head@18'1, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-18T04:38:18.659 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:51.680544+0000", 2024-09-18T04:38:18.659 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 26.971700889000001, 2024-09-18T04:38:18.659 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00050365200000000005, 2024-09-18T04:38:18.659 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:18.659 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:38:18.659 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:18.659 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:18.659 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:18.659 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680544+0000", 2024-09-18T04:38:18.659 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:18.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:18.659 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:18.659 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:18.659 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680544+0000", 2024-09-18T04:38:18.660 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:18.660 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:18.660 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:18.660 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:18.660 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680543+0000", 2024-09-18T04:38:18.660 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:18.660 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:18.660 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:18.660 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:18.660 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680571+0000", 2024-09-18T04:38:18.660 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7953000000000002e-05 2024-09-18T04:38:18.660 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:18.660 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:18.660 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:18.661 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680605+0000", 2024-09-18T04:38:18.661 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3933000000000002e-05 2024-09-18T04:38:18.661 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:18.661 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:18.661 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:18.661 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680636+0000", 2024-09-18T04:38:18.661 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1288999999999998e-05 2024-09-18T04:38:18.661 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:18.661 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:18.661 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:18.661 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680748+0000", 2024-09-18T04:38:18.661 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000111431 2024-09-18T04:38:18.661 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:18.662 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:18.662 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:18.662 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680758+0000", 2024-09-18T04:38:18.662 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0244999999999999e-05 2024-09-18T04:38:18.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:18.662 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:18.662 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:18.662 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.681048+0000", 2024-09-18T04:38:18.662 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00029003399999999999 2024-09-18T04:38:18.662 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:18.662 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:18.662 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:18.662 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:18.662 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:18.663 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:18.664 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:38:18.665 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:18.658+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:18.765 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:18.758+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:18.780 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:18.781 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:18.781 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:38:18.781 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:18.791 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:38:18.866 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:18.862+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:18.898 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:18.898 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:18.898 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:38:18.899 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:38:18.899 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:18.908 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:38:18.967 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:18.962+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:19.015 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:19.015 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:38:19.015 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:38:19.015 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:38:19.015 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.015 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:19.016 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.573328+0000", 2024-09-18T04:38:19.016 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 51.436310740000003, 2024-09-18T04:38:19.016 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011397898, 2024-09-18T04:38:19.016 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:19.016 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:19.016 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:19.016 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:19.016 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:19.016 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 86 2024-09-18T04:38:19.016 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.016 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:19.016 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.016 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:19.016 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573328+0000", 2024-09-18T04:38:19.017 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.017 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.017 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.017 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:19.017 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573328+0000", 2024-09-18T04:38:19.017 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.017 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.017 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.017 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:19.017 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573327+0000", 2024-09-18T04:38:19.017 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:19.017 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.017 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.017 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:19.018 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573331+0000", 2024-09-18T04:38:19.018 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9750000000000001e-06 2024-09-18T04:38:19.018 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.018 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.018 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:19.018 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573346+0000", 2024-09-18T04:38:19.018 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5469999999999999e-05 2024-09-18T04:38:19.018 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.018 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.018 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:19.018 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573370+0000", 2024-09-18T04:38:19.018 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3969999999999999e-05 2024-09-18T04:38:19.018 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.018 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.018 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:19.019 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.578536+0000", 2024-09-18T04:38:19.019 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0051659679999999999 2024-09-18T04:38:19.019 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.019 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.019 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:19.019 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.578689+0000", 2024-09-18T04:38:19.019 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015276299999999999 2024-09-18T04:38:19.019 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.019 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.019 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:19.019 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.578901+0000", 2024-09-18T04:38:19.019 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000211783 2024-09-18T04:38:19.019 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.019 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.019 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:19.019 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.583408+0000", 2024-09-18T04:38:19.020 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045071249999999998 2024-09-18T04:38:19.020 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.020 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.020 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:19.020 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.584645+0000", 2024-09-18T04:38:19.020 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001237007 2024-09-18T04:38:19.020 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.020 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.020 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:19.020 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.584703+0000", 2024-09-18T04:38:19.020 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.817e-05 2024-09-18T04:38:19.020 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.020 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.020 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:19.021 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.584725+0000", 2024-09-18T04:38:19.021 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2155e-05 2024-09-18T04:38:19.021 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.021 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:19.021 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.021 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.021 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.021 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:19.021 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.573382+0000", 2024-09-18T04:38:19.021 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 51.436255992, 2024-09-18T04:38:19.021 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012154974000000001, 2024-09-18T04:38:19.021 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:19.021 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:19.021 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:19.022 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:19.022 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:19.022 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 87 2024-09-18T04:38:19.022 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.022 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:19.022 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.022 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:19.022 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573382+0000", 2024-09-18T04:38:19.022 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.022 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.022 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.022 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:19.022 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573382+0000", 2024-09-18T04:38:19.022 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.023 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.023 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.023 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:19.023 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573382+0000", 2024-09-18T04:38:19.023 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:19.023 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.023 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.023 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:19.023 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573482+0000", 2024-09-18T04:38:19.023 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000100333 2024-09-18T04:38:19.023 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.023 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.023 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:19.023 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573492+0000", 2024-09-18T04:38:19.024 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5389999999999996e-06 2024-09-18T04:38:19.024 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.024 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.024 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:19.024 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573500+0000", 2024-09-18T04:38:19.024 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8929999999999994e-06 2024-09-18T04:38:19.024 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.024 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.024 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:19.024 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.579797+0000", 2024-09-18T04:38:19.024 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0062963239999999998 2024-09-18T04:38:19.024 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.024 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.024 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:19.024 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.579957+0000", 2024-09-18T04:38:19.024 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000160283 2024-09-18T04:38:19.025 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.025 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.025 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:19.025 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.580178+0000", 2024-09-18T04:38:19.025 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00022079200000000001 2024-09-18T04:38:19.025 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.025 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.025 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:19.025 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.583479+0000", 2024-09-18T04:38:19.025 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033007689999999998 2024-09-18T04:38:19.025 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.025 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.025 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:19.025 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585477+0000", 2024-09-18T04:38:19.025 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0019985670000000001 2024-09-18T04:38:19.025 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.026 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.026 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:19.026 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585517+0000", 2024-09-18T04:38:19.026 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9709999999999998e-05 2024-09-18T04:38:19.026 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.026 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.026 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:19.026 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585537+0000", 2024-09-18T04:38:19.026 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0326000000000002e-05 2024-09-18T04:38:19.026 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.026 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:19.026 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.026 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.026 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.026 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:19.027 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.573508+0000", 2024-09-18T04:38:19.027 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 51.436130286000001, 2024-09-18T04:38:19.027 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012068476, 2024-09-18T04:38:19.027 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:19.027 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:19.027 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:19.027 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:19.027 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:19.027 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 88 2024-09-18T04:38:19.027 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.027 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:19.027 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.027 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:19.028 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573508+0000", 2024-09-18T04:38:19.028 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.028 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.028 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.028 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:19.028 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573508+0000", 2024-09-18T04:38:19.028 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.028 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.028 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.029 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:19.029 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573507+0000", 2024-09-18T04:38:19.029 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:19.029 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.029 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.029 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:19.029 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573512+0000", 2024-09-18T04:38:19.029 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.2629999999999997e-06 2024-09-18T04:38:19.029 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.029 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.029 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:19.029 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573530+0000", 2024-09-18T04:38:19.029 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8507000000000001e-05 2024-09-18T04:38:19.029 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.030 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.030 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:19.030 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573537+0000", 2024-09-18T04:38:19.030 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.832e-06 2024-09-18T04:38:19.030 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.030 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.030 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:19.030 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.582486+0000", 2024-09-18T04:38:19.030 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0089494770000000008 2024-09-18T04:38:19.030 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.030 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.030 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:19.030 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.582630+0000", 2024-09-18T04:38:19.030 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000144005 2024-09-18T04:38:19.030 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.030 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.031 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:19.031 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.582842+0000", 2024-09-18T04:38:19.031 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00021189200000000001 2024-09-18T04:38:19.031 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.031 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.031 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:19.031 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.584777+0000", 2024-09-18T04:38:19.031 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001934753 2024-09-18T04:38:19.031 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.031 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.031 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:19.031 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585543+0000", 2024-09-18T04:38:19.031 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00076540099999999997 2024-09-18T04:38:19.031 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.031 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.032 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:19.032 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585561+0000", 2024-09-18T04:38:19.032 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8474999999999998e-05 2024-09-18T04:38:19.032 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.032 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.032 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:19.032 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585576+0000", 2024-09-18T04:38:19.032 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5427e-05 2024-09-18T04:38:19.032 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.032 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:19.032 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.032 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.032 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.032 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:19.033 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786394+0000", 2024-09-18T04:38:19.033 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 51.223243838999998, 2024-09-18T04:38:19.033 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011902085999999999, 2024-09-18T04:38:19.033 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:19.033 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:19.033 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:19.033 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:19.033 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:19.033 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 109 2024-09-18T04:38:19.033 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.033 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:19.033 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.033 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:19.033 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786394+0000", 2024-09-18T04:38:19.034 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.034 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.034 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.034 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:19.034 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786394+0000", 2024-09-18T04:38:19.034 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.034 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.034 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.034 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:19.034 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786394+0000", 2024-09-18T04:38:19.034 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:19.034 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.034 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.034 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:19.035 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786548+0000", 2024-09-18T04:38:19.035 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015463400000000001 2024-09-18T04:38:19.035 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.035 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.035 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:19.035 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786561+0000", 2024-09-18T04:38:19.035 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3093e-05 2024-09-18T04:38:19.035 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.035 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.035 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:19.035 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786571+0000", 2024-09-18T04:38:19.035 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4700000000000008e-06 2024-09-18T04:38:19.035 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.036 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.036 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:19.036 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.793367+0000", 2024-09-18T04:38:19.036 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0067966600000000004 2024-09-18T04:38:19.036 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.036 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.036 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:19.036 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.793526+0000", 2024-09-18T04:38:19.036 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015877 2024-09-18T04:38:19.036 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.036 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.036 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:19.036 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.793752+0000", 2024-09-18T04:38:19.036 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00022555200000000001 2024-09-18T04:38:19.037 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.037 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.037 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:19.037 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798125+0000", 2024-09-18T04:38:19.037 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0043733349999999999 2024-09-18T04:38:19.037 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.037 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.037 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:19.037 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798146+0000", 2024-09-18T04:38:19.037 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1063000000000001e-05 2024-09-18T04:38:19.037 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.037 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.037 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:19.037 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798232+0000", 2024-09-18T04:38:19.038 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5771000000000003e-05 2024-09-18T04:38:19.038 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.038 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.038 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:19.038 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798297+0000", 2024-09-18T04:38:19.038 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.4624999999999999e-05 2024-09-18T04:38:19.038 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.038 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:19.038 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.038 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.038 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.038 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:19.038 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786584+0000", 2024-09-18T04:38:19.038 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 51.223053759999999, 2024-09-18T04:38:19.039 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012013037000000001, 2024-09-18T04:38:19.039 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:19.039 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:19.039 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:19.039 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:19.039 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:19.039 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 110 2024-09-18T04:38:19.039 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.039 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:19.039 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.039 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:19.039 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786584+0000", 2024-09-18T04:38:19.039 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.039 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.040 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.040 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:19.040 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786584+0000", 2024-09-18T04:38:19.040 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.040 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.040 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.040 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:19.040 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786584+0000", 2024-09-18T04:38:19.040 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:19.040 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.040 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.040 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:19.040 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786590+0000", 2024-09-18T04:38:19.040 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6359999999999997e-06 2024-09-18T04:38:19.040 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.040 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.041 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:19.041 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786599+0000", 2024-09-18T04:38:19.041 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7369999999999995e-06 2024-09-18T04:38:19.041 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.041 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.041 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:19.041 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786606+0000", 2024-09-18T04:38:19.041 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1930000000000004e-06 2024-09-18T04:38:19.041 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.041 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.041 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:19.041 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.795018+0000", 2024-09-18T04:38:19.041 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0084115660000000005 2024-09-18T04:38:19.041 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.042 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.042 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:19.042 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.795184+0000", 2024-09-18T04:38:19.042 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000165833 2024-09-18T04:38:19.042 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.042 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.042 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:19.042 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.795414+0000", 2024-09-18T04:38:19.042 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00023088399999999999 2024-09-18T04:38:19.042 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.042 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.042 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:19.042 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798321+0000", 2024-09-18T04:38:19.042 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0029069539999999998 2024-09-18T04:38:19.043 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.043 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.043 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:19.043 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798404+0000", 2024-09-18T04:38:19.043 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2724000000000003e-05 2024-09-18T04:38:19.043 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.043 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.043 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:19.043 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798488+0000", 2024-09-18T04:38:19.043 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.3505999999999996e-05 2024-09-18T04:38:19.043 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.043 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.043 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:19.044 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798598+0000", 2024-09-18T04:38:19.044 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000109917 2024-09-18T04:38:19.044 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.044 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:19.044 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.044 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.044 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.044 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:19.044 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786613+0000", 2024-09-18T04:38:19.044 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 51.223025059999998, 2024-09-18T04:38:19.044 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014245760999999999, 2024-09-18T04:38:19.044 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:19.044 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:19.044 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:19.044 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:19.045 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:19.045 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 111 2024-09-18T04:38:19.045 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.045 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:19.045 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.045 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:19.045 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786613+0000", 2024-09-18T04:38:19.045 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.045 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.045 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.045 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:19.045 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786613+0000", 2024-09-18T04:38:19.045 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.045 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.045 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.046 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:19.046 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786612+0000", 2024-09-18T04:38:19.046 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:19.046 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.046 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.046 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:19.046 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786622+0000", 2024-09-18T04:38:19.046 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4099999999999997e-06 2024-09-18T04:38:19.046 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.046 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.046 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:19.046 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786630+0000", 2024-09-18T04:38:19.046 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4169999999999992e-06 2024-09-18T04:38:19.047 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.047 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.047 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:19.047 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786637+0000", 2024-09-18T04:38:19.047 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8290000000000002e-06 2024-09-18T04:38:19.047 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.047 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.047 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:19.047 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.797099+0000", 2024-09-18T04:38:19.047 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010461747 2024-09-18T04:38:19.047 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.047 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.047 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:19.047 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.797243+0000", 2024-09-18T04:38:19.048 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014388899999999999 2024-09-18T04:38:19.048 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.048 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.048 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:19.048 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.797463+0000", 2024-09-18T04:38:19.048 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000219912 2024-09-18T04:38:19.048 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.048 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.048 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:19.048 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800732+0000", 2024-09-18T04:38:19.048 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0032688719999999999 2024-09-18T04:38:19.048 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.048 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.048 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:19.049 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800751+0000", 2024-09-18T04:38:19.049 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9633999999999999e-05 2024-09-18T04:38:19.049 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.049 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.049 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:19.049 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800824+0000", 2024-09-18T04:38:19.049 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.2805999999999994e-05 2024-09-18T04:38:19.049 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.049 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.049 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:19.049 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800859+0000", 2024-09-18T04:38:19.049 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4952e-05 2024-09-18T04:38:19.049 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.050 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:19.050 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.050 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.050 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.050 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:19.050 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786649+0000", 2024-09-18T04:38:19.050 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 51.222989650999999, 2024-09-18T04:38:19.050 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.01613442, 2024-09-18T04:38:19.050 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:19.050 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:19.050 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:19.050 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:19.050 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:19.050 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 112 2024-09-18T04:38:19.051 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.051 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:19.051 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.051 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:19.051 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786649+0000", 2024-09-18T04:38:19.051 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.051 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.051 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.051 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:19.051 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786649+0000", 2024-09-18T04:38:19.051 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.051 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.051 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.051 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:19.052 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786648+0000", 2024-09-18T04:38:19.052 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:19.052 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.052 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.052 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:19.052 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786768+0000", 2024-09-18T04:38:19.052 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000120491 2024-09-18T04:38:19.052 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.052 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.052 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:19.052 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786795+0000", 2024-09-18T04:38:19.052 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6556000000000001e-05 2024-09-18T04:38:19.052 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.052 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.053 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:19.053 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786807+0000", 2024-09-18T04:38:19.053 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1739999999999999e-05 2024-09-18T04:38:19.053 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.053 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.053 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:19.053 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798683+0000", 2024-09-18T04:38:19.053 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011876070000000001 2024-09-18T04:38:19.053 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.053 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.053 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:19.053 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798867+0000", 2024-09-18T04:38:19.053 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000184103 2024-09-18T04:38:19.053 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.054 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.054 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:19.054 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.799082+0000", 2024-09-18T04:38:19.054 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00021476000000000001 2024-09-18T04:38:19.054 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.054 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.054 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:19.054 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802610+0000", 2024-09-18T04:38:19.054 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0035282209999999998 2024-09-18T04:38:19.054 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.054 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.054 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:19.054 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802706+0000", 2024-09-18T04:38:19.054 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5922999999999996e-05 2024-09-18T04:38:19.055 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.055 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.055 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:19.055 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802750+0000", 2024-09-18T04:38:19.055 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4511000000000001e-05 2024-09-18T04:38:19.055 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.055 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.055 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:19.055 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802783+0000", 2024-09-18T04:38:19.055 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2752000000000001e-05 2024-09-18T04:38:19.055 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.055 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:19.055 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.055 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.056 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.056 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:19.056 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786816+0000", 2024-09-18T04:38:19.056 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 51.222822583999999, 2024-09-18T04:38:19.056 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016197989999999999, 2024-09-18T04:38:19.056 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:19.056 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:19.056 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:19.056 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:19.056 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:19.056 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 113 2024-09-18T04:38:19.056 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.056 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:19.056 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.057 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:19.057 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786816+0000", 2024-09-18T04:38:19.057 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.057 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.057 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.057 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:19.057 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786816+0000", 2024-09-18T04:38:19.057 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.057 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.057 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.057 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:19.057 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786815+0000", 2024-09-18T04:38:19.057 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:19.057 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.058 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.058 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:19.058 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786821+0000", 2024-09-18T04:38:19.058 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9529999999999996e-06 2024-09-18T04:38:19.058 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.058 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.058 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:19.058 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786841+0000", 2024-09-18T04:38:19.058 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9743000000000001e-05 2024-09-18T04:38:19.058 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.058 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.058 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:19.058 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786848+0000", 2024-09-18T04:38:19.058 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4900000000000003e-06 2024-09-18T04:38:19.059 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.059 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.059 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:19.059 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.799747+0000", 2024-09-18T04:38:19.059 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012898902 2024-09-18T04:38:19.059 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.059 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.059 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:19.059 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.799898+0000", 2024-09-18T04:38:19.059 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000150877 2024-09-18T04:38:19.059 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.059 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.059 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:19.059 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800095+0000", 2024-09-18T04:38:19.060 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000197023 2024-09-18T04:38:19.060 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.060 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.060 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:19.060 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802829+0000", 2024-09-18T04:38:19.060 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0027343480000000002 2024-09-18T04:38:19.060 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.060 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.060 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:19.060 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802847+0000", 2024-09-18T04:38:19.060 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7283e-05 2024-09-18T04:38:19.060 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.060 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.060 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:19.060 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802974+0000", 2024-09-18T04:38:19.061 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012749199999999999 2024-09-18T04:38:19.061 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.061 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.061 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:19.061 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803014+0000", 2024-09-18T04:38:19.061 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9576000000000002e-05 2024-09-18T04:38:19.061 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.061 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:19.061 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.061 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.061 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.061 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:19.061 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:38:19.061 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 51.222736992999998, 2024-09-18T04:38:19.061 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017399218000000001, 2024-09-18T04:38:19.062 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:19.062 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:19.062 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:19.062 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:19.062 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:19.062 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 114 2024-09-18T04:38:19.062 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.062 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:19.062 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.062 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:19.062 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:38:19.062 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.062 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.062 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.063 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:19.063 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:38:19.063 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.063 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.063 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.063 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:19.063 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786900+0000", 2024-09-18T04:38:19.063 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:19.063 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.063 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.063 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:19.063 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786908+0000", 2024-09-18T04:38:19.063 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8199999999999997e-06 2024-09-18T04:38:19.064 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.064 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.064 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:19.064 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786925+0000", 2024-09-18T04:38:19.064 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6849000000000001e-05 2024-09-18T04:38:19.064 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.064 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.064 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:19.064 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786934+0000", 2024-09-18T04:38:19.064 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6299999999999993e-06 2024-09-18T04:38:19.064 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.064 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.064 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:19.064 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800926+0000", 2024-09-18T04:38:19.065 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013991632 2024-09-18T04:38:19.065 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.065 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.065 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:19.065 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801060+0000", 2024-09-18T04:38:19.065 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013378199999999999 2024-09-18T04:38:19.065 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.065 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.065 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:19.065 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801166+0000", 2024-09-18T04:38:19.065 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010607099999999999 2024-09-18T04:38:19.065 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.065 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.065 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:19.066 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803026+0000", 2024-09-18T04:38:19.066 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001860469 2024-09-18T04:38:19.066 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.066 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.066 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:19.066 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804245+0000", 2024-09-18T04:38:19.066 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0012185900000000001 2024-09-18T04:38:19.066 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.066 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.066 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:19.066 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804278+0000", 2024-09-18T04:38:19.066 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3272000000000003e-05 2024-09-18T04:38:19.066 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.066 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.067 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:19.067 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804300+0000", 2024-09-18T04:38:19.067 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2266e-05 2024-09-18T04:38:19.067 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.067 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:19.067 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.067 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.067 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.067 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:19.067 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787566+0000", 2024-09-18T04:38:19.067 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 51.222072034, 2024-09-18T04:38:19.067 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017124120999999999, 2024-09-18T04:38:19.067 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:19.067 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:19.067 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:19.068 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:19.068 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:19.068 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 115 2024-09-18T04:38:19.068 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.068 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:19.068 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.068 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:19.068 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787566+0000", 2024-09-18T04:38:19.068 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.068 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.068 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.068 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:19.068 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787566+0000", 2024-09-18T04:38:19.068 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.069 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.069 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.069 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:19.069 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787564+0000", 2024-09-18T04:38:19.069 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:19.069 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.069 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.069 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:19.069 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787751+0000", 2024-09-18T04:38:19.069 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000186873 2024-09-18T04:38:19.069 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.069 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.069 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:19.069 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787772+0000", 2024-09-18T04:38:19.070 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.126e-05 2024-09-18T04:38:19.070 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.070 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.070 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:19.070 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787794+0000", 2024-09-18T04:38:19.070 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1418999999999999e-05 2024-09-18T04:38:19.070 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.070 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.070 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:19.070 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801659+0000", 2024-09-18T04:38:19.070 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013865125000000001 2024-09-18T04:38:19.070 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.070 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.071 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:19.071 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801750+0000", 2024-09-18T04:38:19.071 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.1427000000000004e-05 2024-09-18T04:38:19.071 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.071 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.071 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:19.071 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801894+0000", 2024-09-18T04:38:19.071 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000143514 2024-09-18T04:38:19.071 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.071 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.071 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:19.071 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804313+0000", 2024-09-18T04:38:19.071 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0024190879999999998 2024-09-18T04:38:19.071 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.071 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.072 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:19.072 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804644+0000", 2024-09-18T04:38:19.072 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00033100799999999999 2024-09-18T04:38:19.072 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.072 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.072 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:19.072 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804672+0000", 2024-09-18T04:38:19.072 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7988e-05 2024-09-18T04:38:19.072 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.072 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.072 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:19.072 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804690+0000", 2024-09-18T04:38:19.072 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8298999999999999e-05 2024-09-18T04:38:19.073 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.073 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:19.073 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.073 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.073 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.073 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:19.073 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:38:19.073 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 51.221824095000002, 2024-09-18T04:38:19.073 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017521662, 2024-09-18T04:38:19.073 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:19.073 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:19.073 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:19.073 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:19.074 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:19.074 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 116 2024-09-18T04:38:19.074 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.074 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:19.074 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.074 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:19.074 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:38:19.074 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.074 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.074 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.074 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:19.074 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:38:19.074 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.074 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.075 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.075 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:19.075 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787813+0000", 2024-09-18T04:38:19.075 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:19.075 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.075 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.075 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:19.075 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787824+0000", 2024-09-18T04:38:19.075 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1157e-05 2024-09-18T04:38:19.075 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.075 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.075 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:19.075 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787840+0000", 2024-09-18T04:38:19.075 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6269e-05 2024-09-18T04:38:19.076 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.076 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.076 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:19.076 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787853+0000", 2024-09-18T04:38:19.076 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3156000000000001e-05 2024-09-18T04:38:19.076 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.076 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.076 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:19.076 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803110+0000", 2024-09-18T04:38:19.076 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015256736 2024-09-18T04:38:19.076 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.076 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.076 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:19.077 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803197+0000", 2024-09-18T04:38:19.077 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7195999999999997e-05 2024-09-18T04:38:19.077 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.077 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.077 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:19.077 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803308+0000", 2024-09-18T04:38:19.077 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111005 2024-09-18T04:38:19.077 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.077 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.077 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:19.077 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804316+0000", 2024-09-18T04:38:19.077 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010073949999999999 2024-09-18T04:38:19.077 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.077 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.078 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:19.078 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805232+0000", 2024-09-18T04:38:19.078 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091636300000000001 2024-09-18T04:38:19.078 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.078 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.078 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:19.078 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805277+0000", 2024-09-18T04:38:19.078 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4864000000000003e-05 2024-09-18T04:38:19.078 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.078 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.078 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:19.078 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805336+0000", 2024-09-18T04:38:19.078 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8968000000000002e-05 2024-09-18T04:38:19.078 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.078 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:19.079 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.079 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.079 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.079 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:19.079 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:38:19.079 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 51.221772039999998, 2024-09-18T04:38:19.079 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017983453, 2024-09-18T04:38:19.079 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:19.079 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:19.079 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:19.079 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:19.079 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:19.079 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 117 2024-09-18T04:38:19.079 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.080 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:19.080 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.080 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:19.080 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:38:19.080 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.080 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.080 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.080 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:19.080 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:38:19.080 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.080 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.080 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.080 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:19.080 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787865+0000", 2024-09-18T04:38:19.080 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:19.081 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.081 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.081 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:19.081 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787889+0000", 2024-09-18T04:38:19.081 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4244999999999999e-05 2024-09-18T04:38:19.081 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.081 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.081 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:19.081 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787905+0000", 2024-09-18T04:38:19.081 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5739999999999998e-05 2024-09-18T04:38:19.081 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.081 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.081 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:19.081 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787918+0000", 2024-09-18T04:38:19.082 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.252e-05 2024-09-18T04:38:19.082 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.082 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.082 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:19.082 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803693+0000", 2024-09-18T04:38:19.082 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015774872999999998 2024-09-18T04:38:19.082 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.082 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.082 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:19.082 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803775+0000", 2024-09-18T04:38:19.082 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2761000000000003e-05 2024-09-18T04:38:19.082 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.082 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.082 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:19.082 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803887+0000", 2024-09-18T04:38:19.083 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111708 2024-09-18T04:38:19.083 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.083 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.083 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:19.083 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804823+0000", 2024-09-18T04:38:19.083 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00093625100000000003 2024-09-18T04:38:19.083 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.083 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.083 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:19.083 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805804+0000", 2024-09-18T04:38:19.083 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00098114699999999992 2024-09-18T04:38:19.083 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.083 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.083 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:19.084 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805831+0000", 2024-09-18T04:38:19.084 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6401999999999999e-05 2024-09-18T04:38:19.084 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.084 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.084 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:19.084 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805850+0000", 2024-09-18T04:38:19.084 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8862e-05 2024-09-18T04:38:19.084 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.084 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:19.084 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.084 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.084 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.084 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:19.085 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.230830+0000", 2024-09-18T04:38:19.085 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 50.778808161000001, 2024-09-18T04:38:19.085 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011537146999999999, 2024-09-18T04:38:19.085 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:19.085 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:19.085 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:19.085 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:19.085 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:19.085 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 135 2024-09-18T04:38:19.085 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.085 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:19.085 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.085 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:19.085 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230830+0000", 2024-09-18T04:38:19.085 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.085 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.086 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.086 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:19.086 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230830+0000", 2024-09-18T04:38:19.086 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.086 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.086 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.086 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:19.086 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230829+0000", 2024-09-18T04:38:19.086 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:19.086 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.086 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.086 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:19.086 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230835+0000", 2024-09-18T04:38:19.087 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8660000000000002e-06 2024-09-18T04:38:19.087 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.087 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.087 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:19.087 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230844+0000", 2024-09-18T04:38:19.087 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9339999999999995e-06 2024-09-18T04:38:19.087 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.087 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.087 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:19.087 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230853+0000", 2024-09-18T04:38:19.087 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4419999999999998e-06 2024-09-18T04:38:19.087 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.087 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.087 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:19.087 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.237658+0000", 2024-09-18T04:38:19.087 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0068055310000000001 2024-09-18T04:38:19.088 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.088 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.088 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:19.088 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.237837+0000", 2024-09-18T04:38:19.088 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017943200000000001 2024-09-18T04:38:19.088 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.088 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.088 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:19.088 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.238112+0000", 2024-09-18T04:38:19.088 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00027488600000000001 2024-09-18T04:38:19.088 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.088 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.088 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:19.089 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.240505+0000", 2024-09-18T04:38:19.089 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0023930539999999999 2024-09-18T04:38:19.089 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.089 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.089 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:19.089 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242157+0000", 2024-09-18T04:38:19.089 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001651135 2024-09-18T04:38:19.089 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.089 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.089 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:19.089 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242281+0000", 2024-09-18T04:38:19.089 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012451300000000001 2024-09-18T04:38:19.089 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.089 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.090 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:19.090 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242367+0000", 2024-09-18T04:38:19.090 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6178000000000001e-05 2024-09-18T04:38:19.090 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.090 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:19.090 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.090 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.090 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.090 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:19.090 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.230858+0000", 2024-09-18T04:38:19.090 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 50.778780273000002, 2024-09-18T04:38:19.090 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013005821000000001, 2024-09-18T04:38:19.090 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:19.090 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:19.091 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:19.091 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:19.091 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:19.091 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 136 2024-09-18T04:38:19.091 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.091 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:19.091 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.091 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:19.091 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230858+0000", 2024-09-18T04:38:19.091 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.091 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.091 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.091 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:19.091 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230858+0000", 2024-09-18T04:38:19.091 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.091 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.092 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.092 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:19.092 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230858+0000", 2024-09-18T04:38:19.092 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:19.092 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.092 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.092 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:19.092 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230861+0000", 2024-09-18T04:38:19.092 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7459999999999998e-06 2024-09-18T04:38:19.092 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.092 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.092 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:19.092 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230877+0000", 2024-09-18T04:38:19.093 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5979000000000001e-05 2024-09-18T04:38:19.093 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.093 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.093 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:19.093 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230885+0000", 2024-09-18T04:38:19.093 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.5979999999999999e-06 2024-09-18T04:38:19.093 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.093 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.093 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:19.093 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.239023+0000", 2024-09-18T04:38:19.093 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0081380710000000002 2024-09-18T04:38:19.093 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.093 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.093 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:19.094 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.239216+0000", 2024-09-18T04:38:19.094 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019274799999999999 2024-09-18T04:38:19.094 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.094 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.094 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:19.094 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.239417+0000", 2024-09-18T04:38:19.094 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020093300000000001 2024-09-18T04:38:19.094 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.094 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.094 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:19.094 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242494+0000", 2024-09-18T04:38:19.094 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003077177 2024-09-18T04:38:19.094 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.094 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.095 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:19.095 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.243668+0000", 2024-09-18T04:38:19.095 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001174027 2024-09-18T04:38:19.095 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.095 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.095 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:19.095 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.243777+0000", 2024-09-18T04:38:19.095 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010955600000000001 2024-09-18T04:38:19.095 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.095 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.095 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:19.095 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.243864+0000", 2024-09-18T04:38:19.095 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6441999999999997e-05 2024-09-18T04:38:19.095 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.096 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:19.096 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.096 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.096 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.096 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:19.096 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.230901+0000", 2024-09-18T04:38:19.096 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 50.778737700000001, 2024-09-18T04:38:19.096 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014120882, 2024-09-18T04:38:19.096 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:19.096 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:19.096 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:19.096 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:19.096 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:19.097 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 137 2024-09-18T04:38:19.097 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.097 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:19.097 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.097 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:19.097 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230901+0000", 2024-09-18T04:38:19.097 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.097 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.097 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.097 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:19.097 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230901+0000", 2024-09-18T04:38:19.097 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.097 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.097 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.098 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:19.098 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230900+0000", 2024-09-18T04:38:19.098 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:19.098 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.098 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.098 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:19.098 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231002+0000", 2024-09-18T04:38:19.098 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000102392 2024-09-18T04:38:19.098 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.098 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.098 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:19.098 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231029+0000", 2024-09-18T04:38:19.098 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7166999999999999e-05 2024-09-18T04:38:19.098 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.099 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.099 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:19.099 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231036+0000", 2024-09-18T04:38:19.099 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7259999999999996e-06 2024-09-18T04:38:19.099 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.099 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.099 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:19.099 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.240562+0000", 2024-09-18T04:38:19.099 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0095257970000000008 2024-09-18T04:38:19.099 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.099 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.099 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:19.099 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.240730+0000", 2024-09-18T04:38:19.099 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016767099999999999 2024-09-18T04:38:19.100 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.100 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.100 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:19.100 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.240927+0000", 2024-09-18T04:38:19.100 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019711899999999999 2024-09-18T04:38:19.100 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.100 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.100 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:19.100 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244885+0000", 2024-09-18T04:38:19.100 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0039577340000000001 2024-09-18T04:38:19.100 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.100 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.100 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:19.100 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244899+0000", 2024-09-18T04:38:19.101 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4403000000000001e-05 2024-09-18T04:38:19.101 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.101 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.101 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:19.101 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244973+0000", 2024-09-18T04:38:19.101 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4177999999999994e-05 2024-09-18T04:38:19.101 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.101 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.101 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:19.101 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.245021+0000", 2024-09-18T04:38:19.101 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8344999999999999e-05 2024-09-18T04:38:19.101 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.101 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:19.101 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.102 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.102 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.102 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:19.102 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:38:19.102 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 50.778594908999999, 2024-09-18T04:38:19.102 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016528505999999998, 2024-09-18T04:38:19.102 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:19.102 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:19.102 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:19.102 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:19.102 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:19.102 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 138 2024-09-18T04:38:19.102 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.102 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:19.103 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.103 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:19.103 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:38:19.103 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.103 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.103 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.103 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:19.103 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:38:19.103 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.103 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.103 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.103 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:19.103 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:38:19.103 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:19.104 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.104 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.104 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:19.104 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231047+0000", 2024-09-18T04:38:19.104 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5399999999999997e-06 2024-09-18T04:38:19.104 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.104 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.104 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:19.104 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231068+0000", 2024-09-18T04:38:19.104 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0769000000000001e-05 2024-09-18T04:38:19.104 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.104 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.104 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:19.104 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231074+0000", 2024-09-18T04:38:19.105 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6180000000000003e-06 2024-09-18T04:38:19.105 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.105 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.105 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:19.105 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242556+0000", 2024-09-18T04:38:19.105 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011482654 2024-09-18T04:38:19.105 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.105 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.105 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:19.105 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242666+0000", 2024-09-18T04:38:19.105 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011003200000000001 2024-09-18T04:38:19.105 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.105 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.105 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:19.106 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242924+0000", 2024-09-18T04:38:19.106 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00025785899999999999 2024-09-18T04:38:19.106 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.106 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.106 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:19.106 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247263+0000", 2024-09-18T04:38:19.106 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0043387649999999996 2024-09-18T04:38:19.106 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.106 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.106 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:19.106 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247412+0000", 2024-09-18T04:38:19.106 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000149276 2024-09-18T04:38:19.106 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.106 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.106 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:19.107 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247522+0000", 2024-09-18T04:38:19.107 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000109788 2024-09-18T04:38:19.107 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.107 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.107 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:19.107 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247572+0000", 2024-09-18T04:38:19.107 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9808999999999997e-05 2024-09-18T04:38:19.107 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.107 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:19.107 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.107 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.107 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.107 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:19.107 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231083+0000", 2024-09-18T04:38:19.108 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 50.778555445999999, 2024-09-18T04:38:19.108 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0166731, 2024-09-18T04:38:19.108 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:19.108 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:19.108 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:19.108 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:19.108 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:19.108 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 139 2024-09-18T04:38:19.108 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.108 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:19.108 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.108 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:19.108 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231083+0000", 2024-09-18T04:38:19.108 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.109 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.109 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.109 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:19.109 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231083+0000", 2024-09-18T04:38:19.109 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.109 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.109 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.109 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:19.109 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231082+0000", 2024-09-18T04:38:19.109 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:19.109 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.109 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.109 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:19.109 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231090+0000", 2024-09-18T04:38:19.110 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1280000000000008e-06 2024-09-18T04:38:19.110 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.110 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.110 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:19.110 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231097+0000", 2024-09-18T04:38:19.110 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5119999999999997e-06 2024-09-18T04:38:19.110 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.110 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.110 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:19.110 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231106+0000", 2024-09-18T04:38:19.110 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9230000000000008e-06 2024-09-18T04:38:19.110 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.110 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.110 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:19.111 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.243977+0000", 2024-09-18T04:38:19.111 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012870754 2024-09-18T04:38:19.111 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.111 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.111 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:19.111 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244112+0000", 2024-09-18T04:38:19.111 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000135575 2024-09-18T04:38:19.111 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.111 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.111 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:19.111 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244265+0000", 2024-09-18T04:38:19.111 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015279900000000001 2024-09-18T04:38:19.111 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.111 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.112 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:19.112 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247649+0000", 2024-09-18T04:38:19.112 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003383896 2024-09-18T04:38:19.112 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.112 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.112 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:19.112 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247656+0000", 2024-09-18T04:38:19.112 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8920000000000003e-06 2024-09-18T04:38:19.112 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.112 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.112 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:19.112 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247692+0000", 2024-09-18T04:38:19.112 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6470000000000001e-05 2024-09-18T04:38:19.112 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.113 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.113 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:19.113 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247756+0000", 2024-09-18T04:38:19.113 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3676999999999994e-05 2024-09-18T04:38:19.113 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.113 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:19.113 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.113 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.113 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.113 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:19.113 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231190+0000", 2024-09-18T04:38:19.113 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 50.778448519999998, 2024-09-18T04:38:19.113 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016793556000000001, 2024-09-18T04:38:19.113 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:19.113 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:19.114 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:19.114 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:19.114 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:19.114 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 140 2024-09-18T04:38:19.114 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.114 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:19.114 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.114 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:19.114 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231190+0000", 2024-09-18T04:38:19.114 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.114 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.114 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.114 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:19.114 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231190+0000", 2024-09-18T04:38:19.115 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.115 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.115 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.115 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:19.115 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231189+0000", 2024-09-18T04:38:19.115 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:19.115 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.115 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.115 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:19.115 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231273+0000", 2024-09-18T04:38:19.115 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4445999999999998e-05 2024-09-18T04:38:19.115 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.115 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.115 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:19.115 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231282+0000", 2024-09-18T04:38:19.116 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8859999999999993e-06 2024-09-18T04:38:19.116 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.116 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.116 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:19.116 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231288+0000", 2024-09-18T04:38:19.116 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9819999999999996e-06 2024-09-18T04:38:19.116 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.116 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.116 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:19.116 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.245119+0000", 2024-09-18T04:38:19.116 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.01383121 2024-09-18T04:38:19.116 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.116 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.116 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:19.117 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.245247+0000", 2024-09-18T04:38:19.117 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012763299999999999 2024-09-18T04:38:19.117 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.117 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.117 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:19.117 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.245401+0000", 2024-09-18T04:38:19.117 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015399200000000001 2024-09-18T04:38:19.117 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.117 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.117 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:19.117 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247766+0000", 2024-09-18T04:38:19.117 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0023649510000000001 2024-09-18T04:38:19.117 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.117 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.118 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:19.118 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247904+0000", 2024-09-18T04:38:19.118 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013836499999999999 2024-09-18T04:38:19.118 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.118 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.118 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:19.118 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247950+0000", 2024-09-18T04:38:19.118 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5201000000000003e-05 2024-09-18T04:38:19.118 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.118 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.118 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:19.118 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247983+0000", 2024-09-18T04:38:19.118 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3686000000000002e-05 2024-09-18T04:38:19.118 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.119 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:19.119 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.119 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.119 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.119 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:19.119 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:38:19.119 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 50.778337776999997, 2024-09-18T04:38:19.119 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017344024, 2024-09-18T04:38:19.119 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:19.119 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:19.119 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:19.119 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:19.119 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:19.119 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 141 2024-09-18T04:38:19.120 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.120 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:19.120 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.120 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:19.120 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:38:19.120 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.120 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.120 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.120 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:19.120 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:38:19.120 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.120 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.120 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.121 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:19.121 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:38:19.121 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:19.121 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.121 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.121 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:19.121 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231309+0000", 2024-09-18T04:38:19.121 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5419999999999994e-06 2024-09-18T04:38:19.121 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.121 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.121 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:19.121 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231323+0000", 2024-09-18T04:38:19.121 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3108000000000001e-05 2024-09-18T04:38:19.121 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.122 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.122 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:19.122 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231340+0000", 2024-09-18T04:38:19.122 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.696e-05 2024-09-18T04:38:19.122 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.122 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.122 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:19.122 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.246143+0000", 2024-09-18T04:38:19.122 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014803320999999999 2024-09-18T04:38:19.122 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.122 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.122 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:19.122 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.246323+0000", 2024-09-18T04:38:19.122 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00018061600000000001 2024-09-18T04:38:19.123 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.123 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.123 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:19.123 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.246530+0000", 2024-09-18T04:38:19.123 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020625 2024-09-18T04:38:19.123 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.123 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.123 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:19.123 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248002+0000", 2024-09-18T04:38:19.123 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0014719379999999999 2024-09-18T04:38:19.123 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.123 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.123 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:19.123 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248582+0000", 2024-09-18T04:38:19.124 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00058020599999999999 2024-09-18T04:38:19.124 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.124 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.124 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:19.124 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248618+0000", 2024-09-18T04:38:19.124 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6559e-05 2024-09-18T04:38:19.124 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.124 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.124 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:19.124 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248645+0000", 2024-09-18T04:38:19.124 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6080000000000001e-05 2024-09-18T04:38:19.124 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.124 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:19.125 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.125 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.125 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.125 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-18T04:38:19.125 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.878463+0000", 2024-09-18T04:38:19.125 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 50.131175743, 2024-09-18T04:38:19.125 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010882697, 2024-09-18T04:38:19.125 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:19.125 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:19.125 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:19.125 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:19.125 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:19.125 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 159 2024-09-18T04:38:19.125 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.126 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:19.126 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.126 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:19.126 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878463+0000", 2024-09-18T04:38:19.126 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.126 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.126 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.126 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:19.126 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878463+0000", 2024-09-18T04:38:19.126 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.126 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.126 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.126 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:19.126 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878462+0000", 2024-09-18T04:38:19.126 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:19.127 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.127 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.127 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:19.127 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878467+0000", 2024-09-18T04:38:19.127 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4099999999999999e-06 2024-09-18T04:38:19.127 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.127 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.127 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:19.127 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878490+0000", 2024-09-18T04:38:19.127 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3159999999999998e-05 2024-09-18T04:38:19.127 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.127 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.127 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:19.127 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878511+0000", 2024-09-18T04:38:19.127 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1092000000000001e-05 2024-09-18T04:38:19.128 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.128 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.128 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:19.128 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.887163+0000", 2024-09-18T04:38:19.128 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.008651845 2024-09-18T04:38:19.128 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.128 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.128 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:19.128 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.887273+0000", 2024-09-18T04:38:19.128 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000109823 2024-09-18T04:38:19.128 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.128 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.128 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:19.128 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.887441+0000", 2024-09-18T04:38:19.129 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016777599999999999 2024-09-18T04:38:19.129 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.129 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.129 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:19.129 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.888634+0000", 2024-09-18T04:38:19.129 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001192839 2024-09-18T04:38:19.129 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.129 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.129 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:19.129 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.889153+0000", 2024-09-18T04:38:19.129 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00051953700000000004 2024-09-18T04:38:19.129 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.129 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.129 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:19.130 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.889234+0000", 2024-09-18T04:38:19.130 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0802000000000003e-05 2024-09-18T04:38:19.130 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.130 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.130 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:19.130 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.889345+0000", 2024-09-18T04:38:19.130 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011112000000000001 2024-09-18T04:38:19.130 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.130 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:19.130 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.130 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.130 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:19.130 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:19.132 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:38:19.132 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:19.126+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:19.233 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:19.226+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:19.287 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:19.288 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:19.288 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:38:19.288 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:19.297 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:38:19.342 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:19.338+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:19.401 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:19.401 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:19.401 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:38:19.401 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:38:19.402 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:19.411 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:38:19.434 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:19.431+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:19.520 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:19.520 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:38:19.520 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:38:19.520 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [] 2024-09-18T04:38:19.520 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:19.529 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:38:19.535 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:19.531+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:19.636 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:19.630+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:19.637 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:19.637 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:19.637 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:38:19.637 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:19.647 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:38:19.737 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:19.730+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:19.751 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:19.751 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:19.751 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:38:19.751 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:38:19.751 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:19.761 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:38:19.827 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-18T04:38:19.874 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:19.871+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:19.913 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:19.913 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:38:19.913 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:38:19.913 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:38:19.913 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.913 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-18T04:38:19.913 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:38:19.913 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 45.439654044999997, 2024-09-18T04:38:19.913 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0095781189999999995, 2024-09-18T04:38:19.913 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:19.913 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:19.914 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:19.914 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:38:19.914 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:38:19.914 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 1 2024-09-18T04:38:19.914 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.914 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:19.914 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.914 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:19.914 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:38:19.914 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.914 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.914 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.914 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:19.914 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:38:19.914 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.914 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.915 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.915 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:19.915 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469862+0000", 2024-09-18T04:38:19.915 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:19.915 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.915 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.915 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:19.915 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469879+0000", 2024-09-18T04:38:19.915 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7431999999999998e-05 2024-09-18T04:38:19.915 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.915 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.915 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:19.915 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469896+0000", 2024-09-18T04:38:19.915 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6742999999999999e-05 2024-09-18T04:38:19.916 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.916 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.916 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:19.916 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469915+0000", 2024-09-18T04:38:19.916 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8991000000000002e-05 2024-09-18T04:38:19.916 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.916 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.916 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:19.916 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470397+0000", 2024-09-18T04:38:19.916 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048167799999999998 2024-09-18T04:38:19.916 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.916 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.916 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-09-18T04:38:19.917 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470412+0000", 2024-09-18T04:38:19.917 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5526e-05 2024-09-18T04:38:19.917 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.917 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.917 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:19.917 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473279+0000", 2024-09-18T04:38:19.917 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0028667469999999998 2024-09-18T04:38:19.917 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.918 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.918 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:19.918 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473441+0000", 2024-09-18T04:38:19.918 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000162082 2024-09-18T04:38:19.918 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.918 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.918 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:38:19.918 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473573+0000", 2024-09-18T04:38:19.918 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000132032 2024-09-18T04:38:19.918 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.918 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.918 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:19.918 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474826+0000", 2024-09-18T04:38:19.918 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001252717 2024-09-18T04:38:19.918 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.919 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.919 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:19.919 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479307+0000", 2024-09-18T04:38:19.919 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044813220000000003 2024-09-18T04:38:19.919 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.919 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.919 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:19.919 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479366+0000", 2024-09-18T04:38:19.919 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9024999999999998e-05 2024-09-18T04:38:19.919 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.919 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.919 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:19.919 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479443+0000", 2024-09-18T04:38:19.919 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6339000000000002e-05 2024-09-18T04:38:19.920 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.920 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:19.920 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.920 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.920 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.920 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.4214.0:1 2.3 e18/17 ondisk, result = 0)", 2024-09-18T04:38:19.920 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.474724+0000", 2024-09-18T04:38:19.920 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 45.434794681, 2024-09-18T04:38:19.920 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000105358, 2024-09-18T04:38:19.920 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:19.920 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:38:19.920 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:19.920 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.920 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:19.921 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474724+0000", 2024-09-18T04:38:19.921 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.921 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.921 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.921 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:19.921 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474724+0000", 2024-09-18T04:38:19.921 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.921 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.921 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.921 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:19.921 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474723+0000", 2024-09-18T04:38:19.921 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:19.921 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.921 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.922 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:19.922 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474731+0000", 2024-09-18T04:38:19.922 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2339999999999994e-06 2024-09-18T04:38:19.922 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.922 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.922 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:19.922 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474748+0000", 2024-09-18T04:38:19.922 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6603e-05 2024-09-18T04:38:19.922 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.922 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.922 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:19.922 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474760+0000", 2024-09-18T04:38:19.922 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2215000000000001e-05 2024-09-18T04:38:19.923 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.923 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.923 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:19.923 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474814+0000", 2024-09-18T04:38:19.923 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4267000000000001e-05 2024-09-18T04:38:19.923 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.923 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.923 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:19.923 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474821+0000", 2024-09-18T04:38:19.923 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0929999999999997e-06 2024-09-18T04:38:19.923 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.923 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.924 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:19.924 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474829+0000", 2024-09-18T04:38:19.924 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1200000000000002e-06 2024-09-18T04:38:19.924 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.924 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:19.924 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.924 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.924 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.924 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18)", 2024-09-18T04:38:19.924 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.482346+0000", 2024-09-18T04:38:19.924 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 45.427172249000002, 2024-09-18T04:38:19.924 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020435599999999999, 2024-09-18T04:38:19.924 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:19.925 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-09-18T04:38:19.925 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:19.925 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:38:19.925 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:38:19.925 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 2 2024-09-18T04:38:19.925 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.925 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:19.925 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.925 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:19.925 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482346+0000", 2024-09-18T04:38:19.925 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.925 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.925 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.926 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:19.926 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482346+0000", 2024-09-18T04:38:19.926 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.926 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.926 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.926 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:19.926 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482345+0000", 2024-09-18T04:38:19.926 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:19.926 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.926 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.926 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:19.926 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482360+0000", 2024-09-18T04:38:19.926 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4854999999999999e-05 2024-09-18T04:38:19.926 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.927 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.927 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:19.927 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482377+0000", 2024-09-18T04:38:19.927 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7785999999999999e-05 2024-09-18T04:38:19.927 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.927 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.927 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:19.927 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482387+0000", 2024-09-18T04:38:19.927 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6800000000000005e-06 2024-09-18T04:38:19.927 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.927 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.927 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:19.927 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482441+0000", 2024-09-18T04:38:19.928 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4046999999999999e-05 2024-09-18T04:38:19.928 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.928 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.928 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:19.928 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482551+0000", 2024-09-18T04:38:19.928 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000109468 2024-09-18T04:38:19.928 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.928 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:19.928 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.928 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.928 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.928 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18)", 2024-09-18T04:38:19.928 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.482853+0000", 2024-09-18T04:38:19.929 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 45.426665630999999, 2024-09-18T04:38:19.929 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00029815799999999997, 2024-09-18T04:38:19.929 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:19.929 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-09-18T04:38:19.929 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:19.929 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:38:19.929 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:38:19.929 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 3 2024-09-18T04:38:19.929 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.929 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:19.929 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.929 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:19.929 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482853+0000", 2024-09-18T04:38:19.929 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.930 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.930 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.930 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:19.930 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482853+0000", 2024-09-18T04:38:19.930 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.930 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.930 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.930 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:19.930 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482852+0000", 2024-09-18T04:38:19.930 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:19.930 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.930 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.930 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:19.931 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482859+0000", 2024-09-18T04:38:19.931 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0380000000000002e-06 2024-09-18T04:38:19.931 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.931 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.931 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:19.931 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482880+0000", 2024-09-18T04:38:19.931 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1243e-05 2024-09-18T04:38:19.931 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.931 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.931 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:19.931 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482896+0000", 2024-09-18T04:38:19.931 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5977e-05 2024-09-18T04:38:19.931 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.932 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.932 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:19.932 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482970+0000", 2024-09-18T04:38:19.932 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4603000000000004e-05 2024-09-18T04:38:19.932 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.932 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.932 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:19.932 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.483151+0000", 2024-09-18T04:38:19.932 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00018059799999999999 2024-09-18T04:38:19.932 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.932 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:19.932 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.932 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.933 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.933 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-18T04:38:19.933 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:35.468698+0000", 2024-09-18T04:38:19.933 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 44.440820793999997, 2024-09-18T04:38:19.933 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0039954129999999997, 2024-09-18T04:38:19.933 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:19.933 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:19.933 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:19.933 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:38:19.933 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:38:19.933 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 5 2024-09-18T04:38:19.933 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.933 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:19.933 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.934 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:19.934 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468698+0000", 2024-09-18T04:38:19.934 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.934 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.934 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.934 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:19.934 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468698+0000", 2024-09-18T04:38:19.934 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.934 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.934 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.934 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:19.934 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468696+0000", 2024-09-18T04:38:19.934 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:19.935 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.935 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.935 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:19.935 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468714+0000", 2024-09-18T04:38:19.935 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8743e-05 2024-09-18T04:38:19.935 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.935 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.935 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:19.935 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468748+0000", 2024-09-18T04:38:19.935 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3339e-05 2024-09-18T04:38:19.935 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.935 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.935 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:19.935 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468771+0000", 2024-09-18T04:38:19.936 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3119999999999999e-05 2024-09-18T04:38:19.936 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.936 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.936 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:19.936 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468922+0000", 2024-09-18T04:38:19.936 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000151383 2024-09-18T04:38:19.936 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.936 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.936 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:19.936 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.469278+0000", 2024-09-18T04:38:19.936 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00035628599999999998 2024-09-18T04:38:19.936 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.936 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.936 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:19.937 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.469629+0000", 2024-09-18T04:38:19.937 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00035014300000000003 2024-09-18T04:38:19.937 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.937 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.937 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:19.937 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.471620+0000", 2024-09-18T04:38:19.937 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001991278 2024-09-18T04:38:19.937 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.937 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.937 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:19.937 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472440+0000", 2024-09-18T04:38:19.937 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00081998599999999998 2024-09-18T04:38:19.937 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.937 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.938 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:19.938 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472592+0000", 2024-09-18T04:38:19.938 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015227400000000001 2024-09-18T04:38:19.938 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.938 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.938 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:19.938 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472693+0000", 2024-09-18T04:38:19.938 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000101074 2024-09-18T04:38:19.938 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.938 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:19.938 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.938 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.938 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.938 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.4214.0:5 2.2 e18/17 ondisk, result = 0)", 2024-09-18T04:38:19.938 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:35.472226+0000", 2024-09-18T04:38:19.939 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 44.437292894000002, 2024-09-18T04:38:19.939 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048505299999999999, 2024-09-18T04:38:19.939 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:19.939 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:38:19.939 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:19.939 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.939 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:19.939 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472226+0000", 2024-09-18T04:38:19.939 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.939 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.939 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.939 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:19.939 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472226+0000", 2024-09-18T04:38:19.939 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.939 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.939 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.940 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:19.940 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472224+0000", 2024-09-18T04:38:19.940 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:19.940 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.940 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.940 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:19.940 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472238+0000", 2024-09-18T04:38:19.940 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.468e-05 2024-09-18T04:38:19.940 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.940 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.940 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:19.940 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472272+0000", 2024-09-18T04:38:19.940 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3586e-05 2024-09-18T04:38:19.940 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.941 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.941 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:19.941 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472297+0000", 2024-09-18T04:38:19.941 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5282999999999998e-05 2024-09-18T04:38:19.941 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.941 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.941 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:19.941 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472396+0000", 2024-09-18T04:38:19.941 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.8394999999999996e-05 2024-09-18T04:38:19.941 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.941 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.941 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:19.941 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472421+0000", 2024-09-18T04:38:19.941 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5479000000000001e-05 2024-09-18T04:38:19.942 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.942 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.942 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:19.942 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472711+0000", 2024-09-18T04:38:19.942 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000289714 2024-09-18T04:38:19.942 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.942 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:19.942 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.942 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.942 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.942 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-18T04:38:19.942 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:38:19.942 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 43.426645624999999, 2024-09-18T04:38:19.942 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082699560000000002, 2024-09-18T04:38:19.943 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:19.943 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:19.943 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:19.943 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:38:19.943 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:38:19.943 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 7 2024-09-18T04:38:19.943 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.943 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:19.943 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.943 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:19.943 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:38:19.943 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.943 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.944 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.944 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:19.944 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:38:19.944 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.944 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.944 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.944 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:19.944 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482870+0000", 2024-09-18T04:38:19.944 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999967 2024-09-18T04:38:19.944 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.944 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.944 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:19.944 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482889+0000", 2024-09-18T04:38:19.945 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.965e-05 2024-09-18T04:38:19.945 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.945 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.945 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:19.945 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482938+0000", 2024-09-18T04:38:19.945 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8548999999999999e-05 2024-09-18T04:38:19.945 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.945 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.945 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:19.945 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482963+0000", 2024-09-18T04:38:19.945 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5075999999999998e-05 2024-09-18T04:38:19.945 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.945 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.945 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:19.946 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483121+0000", 2024-09-18T04:38:19.946 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015819400000000001 2024-09-18T04:38:19.946 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.946 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.946 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op must wait for map", 2024-09-18T04:38:19.946 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483138+0000", 2024-09-18T04:38:19.946 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7068999999999999e-05 2024-09-18T04:38:19.946 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.946 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.946 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:19.946 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488666+0000", 2024-09-18T04:38:19.946 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0055282049999999996 2024-09-18T04:38:19.946 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.946 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.946 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:19.947 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488783+0000", 2024-09-18T04:38:19.947 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000116737 2024-09-18T04:38:19.947 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.947 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.947 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:19.947 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488991+0000", 2024-09-18T04:38:19.947 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020781400000000001 2024-09-18T04:38:19.947 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.947 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.947 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:19.947 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490554+0000", 2024-09-18T04:38:19.947 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0015628370000000001 2024-09-18T04:38:19.947 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.947 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.948 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:19.948 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491019+0000", 2024-09-18T04:38:19.948 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000465608 2024-09-18T04:38:19.948 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.948 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.948 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:19.948 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491087+0000", 2024-09-18T04:38:19.948 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7988e-05 2024-09-18T04:38:19.948 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.948 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.948 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:19.948 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491143+0000", 2024-09-18T04:38:19.948 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5538999999999998e-05 2024-09-18T04:38:19.949 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.949 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:19.949 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.949 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.949 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.949 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.4214.0:7 2.2 e19/17 ondisk, result = 0)", 2024-09-18T04:38:19.949 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:36.490861+0000", 2024-09-18T04:38:19.949 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 43.418658002999997, 2024-09-18T04:38:19.949 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00029639400000000001, 2024-09-18T04:38:19.949 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:19.949 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:38:19.949 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:19.949 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.950 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:19.950 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490861+0000", 2024-09-18T04:38:19.950 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.950 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.950 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.950 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:19.950 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490861+0000", 2024-09-18T04:38:19.950 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.950 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.950 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.950 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:19.950 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490859+0000", 2024-09-18T04:38:19.950 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:19.950 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.951 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.951 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:19.951 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490867+0000", 2024-09-18T04:38:19.951 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.3089999999999996e-06 2024-09-18T04:38:19.951 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.951 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.951 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:19.951 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490909+0000", 2024-09-18T04:38:19.951 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1919000000000003e-05 2024-09-18T04:38:19.951 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.951 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.951 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:19.951 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490922+0000", 2024-09-18T04:38:19.951 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2836999999999999e-05 2024-09-18T04:38:19.952 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.952 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.952 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:19.952 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491003+0000", 2024-09-18T04:38:19.952 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0681999999999998e-05 2024-09-18T04:38:19.952 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.952 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.952 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:19.952 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491013+0000", 2024-09-18T04:38:19.952 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0220999999999999e-05 2024-09-18T04:38:19.952 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.952 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.952 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:19.952 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491157+0000", 2024-09-18T04:38:19.952 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014401799999999999 2024-09-18T04:38:19.953 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.953 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:19.953 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.953 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.953 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.953 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "MOSDPGPushReply(2.3 25/24 [PushReplyOp(2:c4c92e5a:::rbd_trash:head)])", 2024-09-18T04:38:19.953 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:51.680323+0000", 2024-09-18T04:38:19.953 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 28.229195627999999, 2024-09-18T04:38:19.953 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00054411499999999996, 2024-09-18T04:38:19.953 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:19.953 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-09-18T04:38:19.953 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:19.953 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.953 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:19.953 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680323+0000", 2024-09-18T04:38:19.954 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.954 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.954 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.954 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:19.954 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680323+0000", 2024-09-18T04:38:19.954 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:19.954 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.954 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.954 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:19.954 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680322+0000", 2024-09-18T04:38:19.954 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:19.954 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.954 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.954 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:19.954 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680333+0000", 2024-09-18T04:38:19.955 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1153e-05 2024-09-18T04:38:19.955 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.955 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.955 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:19.955 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680357+0000", 2024-09-18T04:38:19.955 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3252999999999999e-05 2024-09-18T04:38:19.955 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.955 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.955 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:19.955 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680378+0000", 2024-09-18T04:38:19.955 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1503000000000001e-05 2024-09-18T04:38:19.955 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.955 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.955 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:19.955 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680522+0000", 2024-09-18T04:38:19.955 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014410700000000001 2024-09-18T04:38:19.956 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:19.956 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:19.956 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:19.956 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680867+0000", 2024-09-18T04:38:19.956 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00034475899999999998 2024-09-18T04:38:19.956 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.956 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:19.956 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.956 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:19.956 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:19.956 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:19.958 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:19.950+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:19.958 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:38:20.059 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:20.055+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:20.063 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:20.063 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:20.063 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:38:20.063 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:20.072 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:38:20.159 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:20.155+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:20.227 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:20.227 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:20.227 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:38:20.227 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:38:20.227 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:20.237 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:38:20.282 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:20.279+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:20.333 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T04:38:20.348 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:20.348 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:38:20.348 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:38:20.349 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:38:20.349 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.349 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:30 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'16, mlcod=15'14)", 2024-09-18T04:38:20.349 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.486330+0000", 2024-09-18T04:38:20.349 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 52.858544733000002, 2024-09-18T04:38:20.349 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026383700000000001, 2024-09-18T04:38:20.349 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:20.349 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:20.349 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:20.349 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.349 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:20.349 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486330+0000", 2024-09-18T04:38:20.349 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:20.350 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.350 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.350 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:20.350 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486330+0000", 2024-09-18T04:38:20.350 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:20.350 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.350 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.350 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:20.350 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486328+0000", 2024-09-18T04:38:20.350 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:20.350 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.350 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.350 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:20.351 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486342+0000", 2024-09-18T04:38:20.351 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3783e-05 2024-09-18T04:38:20.351 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.351 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.351 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:20.351 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486373+0000", 2024-09-18T04:38:20.351 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1362999999999998e-05 2024-09-18T04:38:20.351 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.351 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.351 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:20.351 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486387+0000", 2024-09-18T04:38:20.351 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3219e-05 2024-09-18T04:38:20.352 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.352 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.352 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:20.352 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486489+0000", 2024-09-18T04:38:20.352 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010232499999999999 2024-09-18T04:38:20.352 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.352 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.352 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:20.352 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486524+0000", 2024-09-18T04:38:20.352 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4972000000000003e-05 2024-09-18T04:38:20.352 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.352 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.352 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:20.353 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.488902+0000", 2024-09-18T04:38:20.353 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023776240000000001 2024-09-18T04:38:20.353 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.353 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.353 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:20.353 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.488968+0000", 2024-09-18T04:38:20.353 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6761000000000007e-05 2024-09-18T04:38:20.353 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:20.353 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:20.353 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:20.353 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.353 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.353 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:35 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'21, mlcod=15'17)", 2024-09-18T04:38:20.353 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.491817+0000", 2024-09-18T04:38:20.354 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 52.853057815, 2024-09-18T04:38:20.354 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023524739999999998, 2024-09-18T04:38:20.354 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:20.354 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:20.354 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:20.354 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.354 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:20.354 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491817+0000", 2024-09-18T04:38:20.354 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:20.354 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.354 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.354 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:20.354 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491817+0000", 2024-09-18T04:38:20.354 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:20.354 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.354 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.355 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:20.355 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491815+0000", 2024-09-18T04:38:20.355 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:20.355 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.355 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.355 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:20.355 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491890+0000", 2024-09-18T04:38:20.355 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4500999999999997e-05 2024-09-18T04:38:20.355 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.355 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.355 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:20.355 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491916+0000", 2024-09-18T04:38:20.356 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5666000000000001e-05 2024-09-18T04:38:20.356 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.356 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.356 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:20.356 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491928+0000", 2024-09-18T04:38:20.356 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2585999999999999e-05 2024-09-18T04:38:20.356 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.356 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.356 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:20.356 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.492000+0000", 2024-09-18T04:38:20.356 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2309000000000002e-05 2024-09-18T04:38:20.356 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.356 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.357 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:20.357 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.492027+0000", 2024-09-18T04:38:20.357 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6251e-05 2024-09-18T04:38:20.357 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.357 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.357 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:20.357 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.494114+0000", 2024-09-18T04:38:20.357 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020875260000000001 2024-09-18T04:38:20.357 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.357 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.357 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:20.357 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.494169+0000", 2024-09-18T04:38:20.358 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5145000000000001e-05 2024-09-18T04:38:20.358 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:20.358 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:20.358 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:20.358 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.358 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.358 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:57 1.0 e15/14 1:30306672:devicehealth::main.db.0000000000000000:head v 15'35, mlcod=15'33)", 2024-09-18T04:38:20.358 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.529997+0000", 2024-09-18T04:38:20.358 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 52.814877961000001, 2024-09-18T04:38:20.358 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002623869, 2024-09-18T04:38:20.358 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:20.358 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:20.358 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:20.358 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.358 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:20.358 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.529997+0000", 2024-09-18T04:38:20.359 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:20.359 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.359 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.359 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:20.359 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.529997+0000", 2024-09-18T04:38:20.359 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:20.359 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.359 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.359 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:20.359 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.529995+0000", 2024-09-18T04:38:20.359 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:20.359 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.359 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.359 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:20.359 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530116+0000", 2024-09-18T04:38:20.360 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000121141 2024-09-18T04:38:20.360 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.360 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.360 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:20.360 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530146+0000", 2024-09-18T04:38:20.360 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0423e-05 2024-09-18T04:38:20.360 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.360 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.360 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:20.360 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530159+0000", 2024-09-18T04:38:20.360 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2802999999999999e-05 2024-09-18T04:38:20.360 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.360 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.360 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:20.361 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530218+0000", 2024-09-18T04:38:20.361 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8461000000000002e-05 2024-09-18T04:38:20.361 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.361 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.361 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:20.361 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530253+0000", 2024-09-18T04:38:20.361 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4796999999999999e-05 2024-09-18T04:38:20.361 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.362 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.362 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:20.362 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.532559+0000", 2024-09-18T04:38:20.362 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002306939 2024-09-18T04:38:20.362 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.362 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.362 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:20.362 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.532621+0000", 2024-09-18T04:38:20.362 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1131999999999997e-05 2024-09-18T04:38:20.363 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:20.363 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:20.363 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:20.363 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.363 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.363 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:68 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'42, mlcod=15'39)", 2024-09-18T04:38:20.363 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.546404+0000", 2024-09-18T04:38:20.363 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 52.798470549999998, 2024-09-18T04:38:20.363 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024499769999999999, 2024-09-18T04:38:20.363 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:20.363 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:20.364 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:20.364 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.364 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:20.364 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546404+0000", 2024-09-18T04:38:20.364 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:20.364 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.364 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.364 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:20.364 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546404+0000", 2024-09-18T04:38:20.364 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:20.364 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.364 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.364 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:20.364 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546403+0000", 2024-09-18T04:38:20.365 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:20.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.365 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.365 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:20.365 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546416+0000", 2024-09-18T04:38:20.365 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.306e-05 2024-09-18T04:38:20.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.365 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.365 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:20.365 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546465+0000", 2024-09-18T04:38:20.365 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9441999999999999e-05 2024-09-18T04:38:20.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.365 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.365 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:20.365 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546480+0000", 2024-09-18T04:38:20.366 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4919999999999999e-05 2024-09-18T04:38:20.366 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.366 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.366 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:20.366 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546541+0000", 2024-09-18T04:38:20.366 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1085000000000002e-05 2024-09-18T04:38:20.366 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.366 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.366 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:20.366 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546569+0000", 2024-09-18T04:38:20.366 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7803000000000001e-05 2024-09-18T04:38:20.366 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.366 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.366 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:20.366 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.548725+0000", 2024-09-18T04:38:20.366 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021556629999999999 2024-09-18T04:38:20.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.367 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.367 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:20.367 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.548854+0000", 2024-09-18T04:38:20.367 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000129524 2024-09-18T04:38:20.367 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:20.367 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:20.367 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:20.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.367 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.367 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:69 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'43, mlcod=15'39)", 2024-09-18T04:38:20.367 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.547520+0000", 2024-09-18T04:38:20.367 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 52.797354474000002, 2024-09-18T04:38:20.367 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023506820000000002, 2024-09-18T04:38:20.367 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:20.368 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:20.368 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:20.368 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.368 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:20.368 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547520+0000", 2024-09-18T04:38:20.368 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:20.368 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.368 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.368 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:20.368 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547520+0000", 2024-09-18T04:38:20.368 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:20.368 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.368 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.368 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:20.368 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547519+0000", 2024-09-18T04:38:20.368 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:20.369 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.369 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.369 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:20.369 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547614+0000", 2024-09-18T04:38:20.369 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5235000000000006e-05 2024-09-18T04:38:20.369 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.369 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.369 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:20.369 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547640+0000", 2024-09-18T04:38:20.369 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5698999999999999e-05 2024-09-18T04:38:20.369 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.369 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.369 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:20.370 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547651+0000", 2024-09-18T04:38:20.370 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.134e-05 2024-09-18T04:38:20.370 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.370 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.370 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:20.370 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547805+0000", 2024-09-18T04:38:20.370 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015426099999999999 2024-09-18T04:38:20.370 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.370 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.370 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:20.370 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547844+0000", 2024-09-18T04:38:20.370 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8229999999999998e-05 2024-09-18T04:38:20.370 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.370 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.371 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:20.371 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.549810+0000", 2024-09-18T04:38:20.371 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0019660519999999998 2024-09-18T04:38:20.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.371 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.371 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:20.371 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.549871+0000", 2024-09-18T04:38:20.371 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1152e-05 2024-09-18T04:38:20.371 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:20.371 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:20.371 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:20.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.371 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.371 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:83 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'53, mlcod=15'50)", 2024-09-18T04:38:20.371 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.575989+0000", 2024-09-18T04:38:20.372 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 52.768885525999998, 2024-09-18T04:38:20.372 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023991310000000001, 2024-09-18T04:38:20.372 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:20.372 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:20.372 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:20.372 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.372 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:20.372 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.575989+0000", 2024-09-18T04:38:20.372 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:20.372 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.372 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.372 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:20.372 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.575989+0000", 2024-09-18T04:38:20.372 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:20.372 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.373 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.373 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:20.373 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.575988+0000", 2024-09-18T04:38:20.373 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:20.373 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.373 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.373 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:20.373 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576001+0000", 2024-09-18T04:38:20.373 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2751e-05 2024-09-18T04:38:20.373 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.373 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.373 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:20.373 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576023+0000", 2024-09-18T04:38:20.374 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2453999999999999e-05 2024-09-18T04:38:20.374 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.374 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.374 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:20.374 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576038+0000", 2024-09-18T04:38:20.374 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5125e-05 2024-09-18T04:38:20.374 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.374 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.374 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:20.374 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576161+0000", 2024-09-18T04:38:20.374 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000122543 2024-09-18T04:38:20.374 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.374 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.374 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:20.374 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576221+0000", 2024-09-18T04:38:20.375 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9948000000000003e-05 2024-09-18T04:38:20.375 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.375 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.375 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:20.375 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.578277+0000", 2024-09-18T04:38:20.375 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020567559999999999 2024-09-18T04:38:20.375 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.375 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.375 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:20.375 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.578388+0000", 2024-09-18T04:38:20.375 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011089100000000001 2024-09-18T04:38:20.375 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:20.375 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:20.375 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:20.376 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.376 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.376 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:86 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'56, mlcod=15'52)", 2024-09-18T04:38:20.376 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.580261+0000", 2024-09-18T04:38:20.376 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 52.76461389, 2024-09-18T04:38:20.376 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002418108, 2024-09-18T04:38:20.376 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:20.376 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:20.376 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:20.376 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.376 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:20.376 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580261+0000", 2024-09-18T04:38:20.376 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:20.376 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.376 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.377 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:20.377 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580261+0000", 2024-09-18T04:38:20.377 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:20.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.377 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.377 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:20.377 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580260+0000", 2024-09-18T04:38:20.377 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:20.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.377 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.377 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:20.377 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580269+0000", 2024-09-18T04:38:20.377 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5109999999999992e-06 2024-09-18T04:38:20.378 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.378 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.378 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:20.378 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580288+0000", 2024-09-18T04:38:20.378 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8783999999999998e-05 2024-09-18T04:38:20.378 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.378 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.378 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:20.378 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580296+0000", 2024-09-18T04:38:20.378 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3250000000000008e-06 2024-09-18T04:38:20.378 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.378 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.378 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:20.378 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580351+0000", 2024-09-18T04:38:20.379 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4132000000000003e-05 2024-09-18T04:38:20.379 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.379 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.379 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:20.379 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580381+0000", 2024-09-18T04:38:20.379 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0263000000000001e-05 2024-09-18T04:38:20.379 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.379 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.379 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:20.379 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.582615+0000", 2024-09-18T04:38:20.379 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0022339109999999999 2024-09-18T04:38:20.379 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.379 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.379 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:20.379 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.582679+0000", 2024-09-18T04:38:20.379 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4182000000000003e-05 2024-09-18T04:38:20.380 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:20.380 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:20.380 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:20.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.380 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.380 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:105 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'67, mlcod=15'64)", 2024-09-18T04:38:20.380 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.789089+0000", 2024-09-18T04:38:20.380 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 52.555785720000003, 2024-09-18T04:38:20.380 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024384229999999999, 2024-09-18T04:38:20.380 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:20.380 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:20.380 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:20.380 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.380 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:20.380 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789089+0000", 2024-09-18T04:38:20.380 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:20.381 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.381 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.381 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:20.381 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789089+0000", 2024-09-18T04:38:20.381 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:20.381 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.381 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.381 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:20.381 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789088+0000", 2024-09-18T04:38:20.381 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:20.381 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.381 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.381 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:20.381 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789107+0000", 2024-09-18T04:38:20.382 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9017000000000001e-05 2024-09-18T04:38:20.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.382 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.382 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:20.382 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789139+0000", 2024-09-18T04:38:20.382 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2261999999999997e-05 2024-09-18T04:38:20.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.382 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.382 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:20.382 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789153+0000", 2024-09-18T04:38:20.382 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4025999999999999e-05 2024-09-18T04:38:20.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.382 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.382 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:20.383 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789255+0000", 2024-09-18T04:38:20.383 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000101969 2024-09-18T04:38:20.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.383 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.383 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:20.383 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789314+0000", 2024-09-18T04:38:20.383 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8867999999999999e-05 2024-09-18T04:38:20.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.383 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.383 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:20.383 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.791453+0000", 2024-09-18T04:38:20.383 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002138938 2024-09-18T04:38:20.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.383 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.384 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:20.384 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.791527+0000", 2024-09-18T04:38:20.384 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4464999999999999e-05 2024-09-18T04:38:20.384 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:20.384 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:20.384 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:20.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.384 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.384 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:106 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'68, mlcod=15'64)", 2024-09-18T04:38:20.384 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.790295+0000", 2024-09-18T04:38:20.384 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 52.554579869000001, 2024-09-18T04:38:20.384 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024379390000000001, 2024-09-18T04:38:20.384 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:20.384 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:20.384 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:20.385 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.385 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:20.385 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790295+0000", 2024-09-18T04:38:20.385 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:20.385 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.385 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.385 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:20.385 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790295+0000", 2024-09-18T04:38:20.385 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:20.385 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.385 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.385 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:20.386 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790294+0000", 2024-09-18T04:38:20.386 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:20.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.386 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.386 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:20.386 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790390+0000", 2024-09-18T04:38:20.386 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6537000000000001e-05 2024-09-18T04:38:20.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.386 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.386 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:20.386 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790418+0000", 2024-09-18T04:38:20.386 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8218e-05 2024-09-18T04:38:20.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.386 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.387 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:20.387 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790431+0000", 2024-09-18T04:38:20.387 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2571999999999999e-05 2024-09-18T04:38:20.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.387 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:20.387 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790534+0000", 2024-09-18T04:38:20.387 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000103433 2024-09-18T04:38:20.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.387 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:20.387 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790565+0000", 2024-09-18T04:38:20.387 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0409999999999999e-05 2024-09-18T04:38:20.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.388 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.388 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:20.388 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.792651+0000", 2024-09-18T04:38:20.388 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020865250000000001 2024-09-18T04:38:20.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.388 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.388 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:20.388 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.792733+0000", 2024-09-18T04:38:20.388 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1477999999999995e-05 2024-09-18T04:38:20.388 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:20.388 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:20.388 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:20.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.389 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.389 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:109 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'71, mlcod=15'68)", 2024-09-18T04:38:20.389 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.795250+0000", 2024-09-18T04:38:20.389 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 52.549624854999998, 2024-09-18T04:38:20.389 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023990449999999998, 2024-09-18T04:38:20.389 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:20.389 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:20.389 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:20.389 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.389 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:20.389 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795250+0000", 2024-09-18T04:38:20.389 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:20.389 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.389 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.389 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:20.389 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795250+0000", 2024-09-18T04:38:20.390 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:20.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.390 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.390 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:20.390 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795248+0000", 2024-09-18T04:38:20.390 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:20.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.390 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.390 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:20.390 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795405+0000", 2024-09-18T04:38:20.390 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000157007 2024-09-18T04:38:20.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.390 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.390 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:20.390 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795422+0000", 2024-09-18T04:38:20.390 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7013000000000001e-05 2024-09-18T04:38:20.391 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.391 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.391 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:20.391 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795432+0000", 2024-09-18T04:38:20.391 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0448999999999999e-05 2024-09-18T04:38:20.391 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.391 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.391 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:20.391 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795534+0000", 2024-09-18T04:38:20.391 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000101195 2024-09-18T04:38:20.391 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.391 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.391 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:20.391 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795587+0000", 2024-09-18T04:38:20.392 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3356000000000002e-05 2024-09-18T04:38:20.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.392 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.392 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:20.392 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.797560+0000", 2024-09-18T04:38:20.392 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001973023 2024-09-18T04:38:20.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.392 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.392 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:20.392 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.797649+0000", 2024-09-18T04:38:20.392 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8864999999999997e-05 2024-09-18T04:38:20.392 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:20.392 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:20.392 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:20.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.393 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.393 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:115 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'77, mlcod=15'72)", 2024-09-18T04:38:20.393 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.803339+0000", 2024-09-18T04:38:20.393 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 52.541536137000001, 2024-09-18T04:38:20.393 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002385093, 2024-09-18T04:38:20.393 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:20.393 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:20.393 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:20.393 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.393 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:20.393 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803339+0000", 2024-09-18T04:38:20.393 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:20.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.393 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.393 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:20.394 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803339+0000", 2024-09-18T04:38:20.394 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:20.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.394 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.394 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:20.394 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803337+0000", 2024-09-18T04:38:20.394 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:20.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.394 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.394 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:20.394 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803475+0000", 2024-09-18T04:38:20.394 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013718199999999999 2024-09-18T04:38:20.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.394 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.395 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:20.395 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803507+0000", 2024-09-18T04:38:20.395 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2351000000000002e-05 2024-09-18T04:38:20.395 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.395 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.395 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:20.395 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803533+0000", 2024-09-18T04:38:20.395 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6010999999999999e-05 2024-09-18T04:38:20.395 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.395 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.395 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:20.395 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803609+0000", 2024-09-18T04:38:20.395 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.64e-05 2024-09-18T04:38:20.395 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.395 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.396 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:20.396 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803671+0000", 2024-09-18T04:38:20.396 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1467e-05 2024-09-18T04:38:20.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.396 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.396 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:20.396 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.805676+0000", 2024-09-18T04:38:20.396 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020047849999999998 2024-09-18T04:38:20.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.396 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.396 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:20.396 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.805724+0000", 2024-09-18T04:38:20.396 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8151e-05 2024-09-18T04:38:20.396 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:20.397 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:20.397 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:20.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.397 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.397 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:118 1.0 e15/14 1:30306672:devicehealth::main.db.0000000000000000:head v 15'80, mlcod=15'78)", 2024-09-18T04:38:20.397 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.807930+0000", 2024-09-18T04:38:20.397 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 52.536944634000001, 2024-09-18T04:38:20.397 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028002259999999998, 2024-09-18T04:38:20.397 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:20.397 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:20.397 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:20.397 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.397 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:20.398 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.807930+0000", 2024-09-18T04:38:20.398 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:20.398 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.398 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.398 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:20.398 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.807930+0000", 2024-09-18T04:38:20.398 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:20.398 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.398 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.398 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:20.398 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.807929+0000", 2024-09-18T04:38:20.398 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:20.398 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.399 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.399 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:20.399 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808024+0000", 2024-09-18T04:38:20.399 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5258000000000002e-05 2024-09-18T04:38:20.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.399 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.399 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:20.399 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808044+0000", 2024-09-18T04:38:20.399 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0203e-05 2024-09-18T04:38:20.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.399 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.399 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:20.399 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808053+0000", 2024-09-18T04:38:20.399 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2689999999999999e-06 2024-09-18T04:38:20.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.400 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.400 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:20.400 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808137+0000", 2024-09-18T04:38:20.400 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4441e-05 2024-09-18T04:38:20.400 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.400 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.400 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:20.400 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808172+0000", 2024-09-18T04:38:20.400 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4453999999999999e-05 2024-09-18T04:38:20.400 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.400 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.401 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:20.401 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.810663+0000", 2024-09-18T04:38:20.401 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024910269999999998 2024-09-18T04:38:20.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.401 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.401 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:20.401 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.810730+0000", 2024-09-18T04:38:20.401 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7658999999999997e-05 2024-09-18T04:38:20.401 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:20.401 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:20.401 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:20.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.401 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.401 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:129 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'87, mlcod=15'85)", 2024-09-18T04:38:20.402 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.231689+0000", 2024-09-18T04:38:20.402 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 52.113185621, 2024-09-18T04:38:20.402 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002376502, 2024-09-18T04:38:20.402 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:20.402 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:20.402 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:20.402 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.402 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:20.402 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231689+0000", 2024-09-18T04:38:20.402 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:20.402 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.403 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.403 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:20.403 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231689+0000", 2024-09-18T04:38:20.403 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:20.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.403 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.403 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:20.403 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231687+0000", 2024-09-18T04:38:20.403 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:20.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.403 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.403 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:20.403 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231702+0000", 2024-09-18T04:38:20.403 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4949999999999999e-05 2024-09-18T04:38:20.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.403 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.404 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:20.404 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231735+0000", 2024-09-18T04:38:20.404 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2276000000000001e-05 2024-09-18T04:38:20.404 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.404 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.404 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:20.404 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231750+0000", 2024-09-18T04:38:20.404 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5193e-05 2024-09-18T04:38:20.404 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.404 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.404 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:20.404 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231872+0000", 2024-09-18T04:38:20.404 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000122127 2024-09-18T04:38:20.404 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.405 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.405 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:20.405 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231936+0000", 2024-09-18T04:38:20.405 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4089000000000003e-05 2024-09-18T04:38:20.405 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.405 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.405 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:20.405 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234002+0000", 2024-09-18T04:38:20.405 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002065575 2024-09-18T04:38:20.405 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.405 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.405 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:20.405 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234066+0000", 2024-09-18T04:38:20.406 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3944999999999998e-05 2024-09-18T04:38:20.406 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:20.406 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:20.406 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:20.406 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.406 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.406 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:130 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'88, mlcod=15'85)", 2024-09-18T04:38:20.406 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.232853+0000", 2024-09-18T04:38:20.406 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 52.112022005999997, 2024-09-18T04:38:20.406 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027351739999999999, 2024-09-18T04:38:20.406 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:20.406 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:20.406 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:20.406 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.407 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:20.407 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232853+0000", 2024-09-18T04:38:20.407 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:20.407 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.407 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.407 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:20.407 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232853+0000", 2024-09-18T04:38:20.407 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:20.407 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.407 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.407 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:20.407 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232851+0000", 2024-09-18T04:38:20.408 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:20.408 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.408 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.408 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:20.408 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232866+0000", 2024-09-18T04:38:20.408 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4946000000000001e-05 2024-09-18T04:38:20.408 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.408 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.408 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:20.409 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232897+0000", 2024-09-18T04:38:20.409 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0982999999999998e-05 2024-09-18T04:38:20.409 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.409 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.409 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:20.409 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232915+0000", 2024-09-18T04:38:20.409 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8476000000000001e-05 2024-09-18T04:38:20.409 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.409 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.409 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:20.409 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.233029+0000", 2024-09-18T04:38:20.409 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011305400000000001 2024-09-18T04:38:20.410 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.410 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.410 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:20.410 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.233079+0000", 2024-09-18T04:38:20.410 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0668999999999999e-05 2024-09-18T04:38:20.410 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.410 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.410 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:20.410 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235534+0000", 2024-09-18T04:38:20.410 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024546030000000001 2024-09-18T04:38:20.410 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.410 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.410 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:20.410 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235588+0000", 2024-09-18T04:38:20.410 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4098999999999998e-05 2024-09-18T04:38:20.411 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:20.411 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:20.411 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:20.411 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.411 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.411 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:131 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'89, mlcod=15'85)", 2024-09-18T04:38:20.411 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.234089+0000", 2024-09-18T04:38:20.411 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 52.110786152000003, 2024-09-18T04:38:20.411 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026975609999999998, 2024-09-18T04:38:20.411 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:20.411 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:20.411 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:20.411 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.411 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:20.411 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234089+0000", 2024-09-18T04:38:20.412 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:20.412 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.412 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.412 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:20.412 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234089+0000", 2024-09-18T04:38:20.412 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:20.412 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.412 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.412 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:20.412 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234087+0000", 2024-09-18T04:38:20.412 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:20.412 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.412 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.412 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:20.413 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234145+0000", 2024-09-18T04:38:20.413 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8687999999999998e-05 2024-09-18T04:38:20.413 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.413 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.413 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:20.413 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234177+0000", 2024-09-18T04:38:20.413 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1526999999999998e-05 2024-09-18T04:38:20.413 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.413 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.413 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:20.413 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234190+0000", 2024-09-18T04:38:20.413 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2965999999999999e-05 2024-09-18T04:38:20.413 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.414 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.414 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:20.414 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234309+0000", 2024-09-18T04:38:20.414 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011875399999999999 2024-09-18T04:38:20.414 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.414 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.414 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:20.414 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234365+0000", 2024-09-18T04:38:20.414 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6436e-05 2024-09-18T04:38:20.414 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.414 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.414 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:20.414 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236675+0000", 2024-09-18T04:38:20.414 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023099520000000001 2024-09-18T04:38:20.415 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.415 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.415 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:20.415 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236786+0000", 2024-09-18T04:38:20.415 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000111025 2024-09-18T04:38:20.415 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:20.415 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:20.415 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:20.415 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.415 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.415 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:132 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'90, mlcod=15'85)", 2024-09-18T04:38:20.415 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.235300+0000", 2024-09-18T04:38:20.415 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 52.109574702000003, 2024-09-18T04:38:20.415 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025982599999999998, 2024-09-18T04:38:20.415 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:20.416 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:20.416 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:20.416 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.416 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:20.416 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235300+0000", 2024-09-18T04:38:20.416 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:20.416 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.416 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.416 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:20.416 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235300+0000", 2024-09-18T04:38:20.416 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:20.416 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.417 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.417 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:20.417 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235298+0000", 2024-09-18T04:38:20.417 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:20.417 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.417 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.417 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:20.417 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235318+0000", 2024-09-18T04:38:20.417 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9714000000000001e-05 2024-09-18T04:38:20.417 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.417 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.417 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:20.417 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235337+0000", 2024-09-18T04:38:20.418 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8892000000000001e-05 2024-09-18T04:38:20.418 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.418 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.418 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:20.418 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235348+0000", 2024-09-18T04:38:20.418 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.129e-05 2024-09-18T04:38:20.418 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.418 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.418 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:20.418 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235681+0000", 2024-09-18T04:38:20.418 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00033323000000000001 2024-09-18T04:38:20.418 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.418 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.419 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:20.419 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235733+0000", 2024-09-18T04:38:20.419 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1391999999999999e-05 2024-09-18T04:38:20.419 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.419 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.419 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:20.419 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237839+0000", 2024-09-18T04:38:20.419 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002105647 2024-09-18T04:38:20.419 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.419 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.419 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:20.419 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237898+0000", 2024-09-18T04:38:20.419 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9759000000000002e-05 2024-09-18T04:38:20.420 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:20.420 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:20.420 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:20.420 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.420 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.420 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:133 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'91, mlcod=15'86)", 2024-09-18T04:38:20.420 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.236920+0000", 2024-09-18T04:38:20.420 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 52.107954814000003, 2024-09-18T04:38:20.420 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024318410000000001, 2024-09-18T04:38:20.420 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:20.420 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:20.420 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:20.420 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.420 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:20.421 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236920+0000", 2024-09-18T04:38:20.421 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:20.421 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.421 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.421 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:20.421 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236920+0000", 2024-09-18T04:38:20.421 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:20.421 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.421 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.421 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:20.421 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236918+0000", 2024-09-18T04:38:20.421 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:20.421 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.422 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.422 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:20.422 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236931+0000", 2024-09-18T04:38:20.422 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2683999999999999e-05 2024-09-18T04:38:20.422 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.422 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.422 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:20.422 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236951+0000", 2024-09-18T04:38:20.422 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0292000000000002e-05 2024-09-18T04:38:20.422 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.422 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.422 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:20.422 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236962+0000", 2024-09-18T04:38:20.422 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0703000000000001e-05 2024-09-18T04:38:20.422 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.423 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.423 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:20.423 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237095+0000", 2024-09-18T04:38:20.423 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013312400000000001 2024-09-18T04:38:20.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.423 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.423 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:20.423 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237133+0000", 2024-09-18T04:38:20.423 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7496000000000001e-05 2024-09-18T04:38:20.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.423 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.423 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:20.423 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.239206+0000", 2024-09-18T04:38:20.423 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020730290000000001 2024-09-18T04:38:20.424 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.424 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.424 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:20.424 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.239352+0000", 2024-09-18T04:38:20.424 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014593000000000001 2024-09-18T04:38:20.424 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:20.424 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:20.424 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:20.424 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.424 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.424 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:134 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'92, mlcod=15'86)", 2024-09-18T04:38:20.424 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.237894+0000", 2024-09-18T04:38:20.424 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 52.106980792000002, 2024-09-18T04:38:20.424 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025299519999999998, 2024-09-18T04:38:20.425 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:20.425 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:20.425 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:20.425 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.425 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:20.425 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237894+0000", 2024-09-18T04:38:20.425 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:20.425 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.425 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.426 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:20.426 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237894+0000", 2024-09-18T04:38:20.426 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:20.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.426 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.426 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:20.426 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237892+0000", 2024-09-18T04:38:20.426 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:20.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.426 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.426 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:20.427 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237986+0000", 2024-09-18T04:38:20.427 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3341e-05 2024-09-18T04:38:20.427 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.427 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.427 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:20.427 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.238012+0000", 2024-09-18T04:38:20.427 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5843e-05 2024-09-18T04:38:20.427 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.427 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.427 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:20.427 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.238027+0000", 2024-09-18T04:38:20.427 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5469e-05 2024-09-18T04:38:20.427 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.428 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.428 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:20.428 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.238114+0000", 2024-09-18T04:38:20.428 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7483000000000002e-05 2024-09-18T04:38:20.428 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.428 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.428 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:20.428 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.238152+0000", 2024-09-18T04:38:20.428 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7565999999999999e-05 2024-09-18T04:38:20.428 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.428 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.428 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:20.429 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.240297+0000", 2024-09-18T04:38:20.429 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021447620000000001 2024-09-18T04:38:20.429 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.429 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.429 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:20.429 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.240424+0000", 2024-09-18T04:38:20.429 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012705400000000001 2024-09-18T04:38:20.429 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:20.429 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:20.429 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:20.429 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.429 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.429 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:139 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'97, mlcod=15'93)", 2024-09-18T04:38:20.429 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.245692+0000", 2024-09-18T04:38:20.430 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 52.099182651, 2024-09-18T04:38:20.430 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024260480000000001, 2024-09-18T04:38:20.430 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:20.430 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:20.430 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:20.430 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.430 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:20.430 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245692+0000", 2024-09-18T04:38:20.430 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:20.430 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.430 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.430 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:20.430 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245692+0000", 2024-09-18T04:38:20.430 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:20.430 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.430 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.431 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:20.431 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245690+0000", 2024-09-18T04:38:20.431 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:20.431 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.431 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.431 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:20.431 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245712+0000", 2024-09-18T04:38:20.431 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1916000000000002e-05 2024-09-18T04:38:20.431 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.431 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.431 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:20.431 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245741+0000", 2024-09-18T04:38:20.431 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8503000000000001e-05 2024-09-18T04:38:20.431 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.432 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.432 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:20.432 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245759+0000", 2024-09-18T04:38:20.432 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.872e-05 2024-09-18T04:38:20.432 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.432 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.432 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:20.432 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245859+0000", 2024-09-18T04:38:20.432 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9641000000000004e-05 2024-09-18T04:38:20.432 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.432 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.432 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:20.432 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245900+0000", 2024-09-18T04:38:20.432 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0589000000000001e-05 2024-09-18T04:38:20.433 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.433 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.433 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:20.433 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.247961+0000", 2024-09-18T04:38:20.433 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020619190000000002 2024-09-18T04:38:20.433 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.433 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.433 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:20.433 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.248118+0000", 2024-09-18T04:38:20.433 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015664300000000001 2024-09-18T04:38:20.433 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:20.433 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:20.433 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:20.433 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.434 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.434 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:144 1.0 e15/14 1:30306672:devicehealth::main.db.0000000000000000:head v 15'102, mlcod=15'98)", 2024-09-18T04:38:20.434 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.252149+0000", 2024-09-18T04:38:20.434 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 52.092725584999997, 2024-09-18T04:38:20.434 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024174959999999999, 2024-09-18T04:38:20.434 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:20.434 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:20.434 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:20.434 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.434 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:20.434 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252149+0000", 2024-09-18T04:38:20.434 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:20.434 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.434 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.434 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:20.435 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252149+0000", 2024-09-18T04:38:20.435 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:20.435 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.435 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.435 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:20.435 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252148+0000", 2024-09-18T04:38:20.435 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:20.435 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.435 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.435 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:20.435 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252225+0000", 2024-09-18T04:38:20.435 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6576999999999995e-05 2024-09-18T04:38:20.435 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.436 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.436 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:20.436 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252242+0000", 2024-09-18T04:38:20.436 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7740999999999998e-05 2024-09-18T04:38:20.436 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.436 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.436 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:20.436 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252254+0000", 2024-09-18T04:38:20.436 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.137e-05 2024-09-18T04:38:20.436 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.436 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.436 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:20.436 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252385+0000", 2024-09-18T04:38:20.437 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013091099999999999 2024-09-18T04:38:20.437 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.437 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.437 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:20.437 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252426+0000", 2024-09-18T04:38:20.437 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1535999999999997e-05 2024-09-18T04:38:20.437 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.437 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.437 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:20.437 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.254498+0000", 2024-09-18T04:38:20.437 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002071477 2024-09-18T04:38:20.437 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:20.437 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:20.437 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:20.438 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.254567+0000", 2024-09-18T04:38:20.438 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9014e-05 2024-09-18T04:38:20.438 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:20.438 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:20.438 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:20.438 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:20.438 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:20.438 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:20.440 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:38:20.440 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:20.434+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:20.473 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:38:20.495 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:38:20.518 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:38:20.540 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:20.534+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:20.543 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:20.543 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:20.544 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:38:20.553 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:38:20.567 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:38:20.589 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:38:20.591 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:38:20.591 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:38:20.592 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:38:20.592 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:38:20.592 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:38:20.592 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:38:20.650 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:20.647+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:20.677 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:20.677 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:20.687 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:38:20.742 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:20.738+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:20.790 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:20.791 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:20.800 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:38:20.822 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-09-18T04:38:20.843 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:20.839+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:20.904 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:20.904 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:20.914 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:38:20.915 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-18T04:38:20.915 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T04:38:20.915 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-18T04:38:20.916 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-18T04:38:20.943 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:20.938+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:21.034 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:21.034 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:21.043 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:21.039+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:21.044 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:38:21.145 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:21.138+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:21.194 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:21.195 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:21.205 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:38:21.227 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:38:21.227 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:21.227 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 39, 2024-09-18T04:38:21.227 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "be6146ba-a991-4f6e-b359-99cbedc9f9b1", 2024-09-18T04:38:21.227 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-09-18T04:37:15.676634+0000", 2024-09-18T04:38:21.227 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-09-18T04:38:20.803684+0000", 2024-09-18T04:38:21.228 INFO:teuthology.orchestra.run.smithi028.stdout: "last_up_change": "2024-09-18T04:37:25.403175+0000", 2024-09-18T04:38:21.228 INFO:teuthology.orchestra.run.smithi028.stdout: "last_in_change": "2024-09-18T04:37:50.652169+0000", 2024-09-18T04:38:21.228 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-18T04:38:21.228 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_num": 5801984, 2024-09-18T04:38:21.228 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_set": [ 2024-09-18T04:38:21.228 INFO:teuthology.orchestra.run.smithi028.stdout: "noscrub", 2024-09-18T04:38:21.228 INFO:teuthology.orchestra.run.smithi028.stdout: "pglog_hardlimit", 2024-09-18T04:38:21.228 INFO:teuthology.orchestra.run.smithi028.stdout: "purged_snapdirs", 2024-09-18T04:38:21.228 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_deletes", 2024-09-18T04:38:21.228 INFO:teuthology.orchestra.run.smithi028.stdout: "sortbitwise" 2024-09-18T04:38:21.228 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-18T04:38:21.229 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_version": 5, 2024-09-18T04:38:21.229 INFO:teuthology.orchestra.run.smithi028.stdout: "full_ratio": 0.94999998807907104, 2024-09-18T04:38:21.229 INFO:teuthology.orchestra.run.smithi028.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-18T04:38:21.229 INFO:teuthology.orchestra.run.smithi028.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-18T04:38:21.229 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_snapshot": "", 2024-09-18T04:38:21.229 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_max": 2, 2024-09-18T04:38:21.229 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd": 6, 2024-09-18T04:38:21.229 INFO:teuthology.orchestra.run.smithi028.stdout: "require_min_compat_client": "luminous", 2024-09-18T04:38:21.229 INFO:teuthology.orchestra.run.smithi028.stdout: "min_compat_client": "luminous", 2024-09-18T04:38:21.229 INFO:teuthology.orchestra.run.smithi028.stdout: "require_osd_release": "reef", 2024-09-18T04:38:21.229 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_crimson": false, 2024-09-18T04:38:21.229 INFO:teuthology.orchestra.run.smithi028.stdout: "pools": [ 2024-09-18T04:38:21.230 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.230 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 1, 2024-09-18T04:38:21.230 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": ".mgr", 2024-09-18T04:38:21.230 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-18T04:37:25.654950+0000", 2024-09-18T04:38:21.230 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-18T04:38:21.230 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-09-18T04:38:21.230 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-18T04:38:21.230 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 2, 2024-09-18T04:38:21.230 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 1, 2024-09-18T04:38:21.230 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-18T04:38:21.230 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-18T04:38:21.230 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-18T04:38:21.230 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T04:38:21.231 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T04:38:21.231 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-18T04:38:21.231 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-18T04:38:21.231 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 1, 2024-09-18T04:38:21.231 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 1, 2024-09-18T04:38:21.231 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 1, 2024-09-18T04:38:21.231 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 1, 2024-09-18T04:38:21.231 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 1, 2024-09-18T04:38:21.231 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-18T04:38:21.231 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-18T04:38:21.231 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-18T04:38:21.231 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-18T04:38:21.232 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-18T04:38:21.232 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-18T04:38:21.232 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-18T04:38:21.232 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.232 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "16", 2024-09-18T04:38:21.232 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-18T04:38:21.232 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T04:38:21.232 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T04:38:21.232 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-18T04:38:21.232 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-18T04:38:21.232 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-09-18T04:38:21.232 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-09-18T04:38:21.233 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-18T04:38:21.233 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-18T04:38:21.233 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-18T04:38:21.233 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-18T04:38:21.233 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-18T04:38:21.233 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-18T04:38:21.233 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-18T04:38:21.233 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-18T04:38:21.233 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-18T04:38:21.233 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-18T04:38:21.233 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-18T04:38:21.234 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T04:38:21.234 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T04:38:21.234 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T04:38:21.234 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-18T04:38:21.234 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-18T04:38:21.234 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-18T04:38:21.234 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-18T04:38:21.234 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-18T04:38:21.234 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.234 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-18T04:38:21.234 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-18T04:38:21.234 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-18T04:38:21.234 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-18T04:38:21.234 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-18T04:38:21.234 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T04:38:21.234 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-18T04:38:21.235 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-18T04:38:21.235 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-18T04:38:21.235 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-18T04:38:21.235 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-18T04:38:21.235 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-09-18T04:38:21.235 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_max": 32, 2024-09-18T04:38:21.235 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_min": 1 2024-09-18T04:38:21.235 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.235 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-18T04:38:21.235 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": {} 2024-09-18T04:38:21.235 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.235 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-09-18T04:38:21.235 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 6.059999942779541, 2024-09-18T04:38:21.235 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 6.059999942779541, 2024-09-18T04:38:21.236 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.33000001311302185, 2024-09-18T04:38:21.236 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2, 2024-09-18T04:38:21.236 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2, 2024-09-18T04:38:21.236 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 1, 2024-09-18T04:38:21.236 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 1, 2024-09-18T04:38:21.236 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 1 2024-09-18T04:38:21.236 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:21.236 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.236 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.236 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 2, 2024-09-18T04:38:21.236 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "rbd", 2024-09-18T04:38:21.236 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-18T04:37:32.672462+0000", 2024-09-18T04:38:21.236 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 8193, 2024-09-18T04:38:21.236 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-18T04:38:21.236 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-18T04:38:21.237 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 2, 2024-09-18T04:38:21.237 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 1, 2024-09-18T04:38:21.237 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-18T04:38:21.237 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-18T04:38:21.237 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-18T04:38:21.237 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T04:38:21.237 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T04:38:21.237 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-18T04:38:21.237 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-18T04:38:21.237 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 8, 2024-09-18T04:38:21.237 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 8, 2024-09-18T04:38:21.237 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 8, 2024-09-18T04:38:21.237 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 8, 2024-09-18T04:38:21.237 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 8, 2024-09-18T04:38:21.238 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-18T04:38:21.238 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-18T04:38:21.238 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-18T04:38:21.238 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-18T04:38:21.238 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-18T04:38:21.238 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-18T04:38:21.238 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-18T04:38:21.238 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.238 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "20", 2024-09-18T04:38:21.238 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-18T04:38:21.238 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T04:38:21.238 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T04:38:21.238 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-18T04:38:21.238 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-18T04:38:21.239 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 3, 2024-09-18T04:38:21.239 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 20, 2024-09-18T04:38:21.239 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-18T04:38:21.239 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-18T04:38:21.239 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-18T04:38:21.239 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-18T04:38:21.239 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-18T04:38:21.239 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-18T04:38:21.239 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-18T04:38:21.239 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-18T04:38:21.239 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-18T04:38:21.239 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-18T04:38:21.239 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-18T04:38:21.239 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T04:38:21.239 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T04:38:21.240 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T04:38:21.240 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-18T04:38:21.240 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-18T04:38:21.240 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-18T04:38:21.240 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-18T04:38:21.240 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-18T04:38:21.240 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.240 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-18T04:38:21.240 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-18T04:38:21.240 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-18T04:38:21.240 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-18T04:38:21.240 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-18T04:38:21.240 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T04:38:21.240 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-18T04:38:21.240 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-18T04:38:21.241 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-18T04:38:21.241 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-18T04:38:21.241 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-18T04:38:21.241 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-09-18T04:38:21.241 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-18T04:38:21.241 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd": {} 2024-09-18T04:38:21.241 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.241 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-09-18T04:38:21.241 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.5099999904632568, 2024-09-18T04:38:21.241 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.5099999904632568, 2024-09-18T04:38:21.241 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.82999998331069946, 2024-09-18T04:38:21.241 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.25, 2024-09-18T04:38:21.241 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.25, 2024-09-18T04:38:21.241 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 1, 2024-09-18T04:38:21.241 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 1, 2024-09-18T04:38:21.242 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 1 2024-09-18T04:38:21.242 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:21.242 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:21.242 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-18T04:38:21.242 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-09-18T04:38:21.242 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.242 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-09-18T04:38:21.242 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "6177002d-72e6-4888-bcde-29b6a92d9dc4", 2024-09-18T04:38:21.242 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-18T04:38:21.242 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-18T04:38:21.242 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-18T04:38:21.242 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-18T04:38:21.242 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-18T04:38:21.242 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-18T04:38:21.242 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 13, 2024-09-18T04:38:21.243 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 0, 2024-09-18T04:38:21.243 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-18T04:38:21.243 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-18T04:38:21.243 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-18T04:38:21.243 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:21.243 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.243 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:21.243 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6801", 2024-09-18T04:38:21.243 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1554407851 2024-09-18T04:38:21.243 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.243 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.243 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:21.243 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6803", 2024-09-18T04:38:21.243 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1554407851 2024-09-18T04:38:21.244 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:21.244 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:21.244 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.244 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-18T04:38:21.244 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:21.244 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.244 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:21.244 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6805", 2024-09-18T04:38:21.244 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1554407851 2024-09-18T04:38:21.244 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.244 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.244 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:21.244 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6807", 2024-09-18T04:38:21.245 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1554407851 2024-09-18T04:38:21.245 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:21.245 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:21.245 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.245 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-18T04:38:21.245 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:21.245 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.245 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:21.245 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6813", 2024-09-18T04:38:21.245 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1554407851 2024-09-18T04:38:21.245 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.245 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.246 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:21.246 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6815", 2024-09-18T04:38:21.246 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1554407851 2024-09-18T04:38:21.246 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:21.246 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:21.246 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.246 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-18T04:38:21.246 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:21.246 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.246 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:21.246 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6809", 2024-09-18T04:38:21.246 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1554407851 2024-09-18T04:38:21.246 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.247 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.247 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:21.247 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6811", 2024-09-18T04:38:21.247 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1554407851 2024-09-18T04:38:21.247 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:21.247 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:21.247 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.247 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6803/1554407851", 2024-09-18T04:38:21.247 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6807/1554407851", 2024-09-18T04:38:21.247 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6815/1554407851", 2024-09-18T04:38:21.247 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6811/1554407851", 2024-09-18T04:38:21.247 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-18T04:38:21.247 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-18T04:38:21.247 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-18T04:38:21.248 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:21.248 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.248 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.248 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-09-18T04:38:21.248 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "4f62cb09-3457-418c-bb69-491d9b70914c", 2024-09-18T04:38:21.248 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-18T04:38:21.248 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-18T04:38:21.248 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-18T04:38:21.248 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-18T04:38:21.248 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-18T04:38:21.248 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-18T04:38:21.248 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 12, 2024-09-18T04:38:21.248 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 21, 2024-09-18T04:38:21.248 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-18T04:38:21.248 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-18T04:38:21.249 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-18T04:38:21.249 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:21.249 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.249 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:21.249 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6800", 2024-09-18T04:38:21.249 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 511141401 2024-09-18T04:38:21.249 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.249 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.249 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:21.249 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6802", 2024-09-18T04:38:21.249 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 511141401 2024-09-18T04:38:21.249 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:21.250 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:21.250 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.250 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-18T04:38:21.250 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:21.250 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.250 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:21.250 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6804", 2024-09-18T04:38:21.250 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 511141401 2024-09-18T04:38:21.250 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.250 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.250 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:21.250 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6806", 2024-09-18T04:38:21.250 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 511141401 2024-09-18T04:38:21.250 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:21.251 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:21.251 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.251 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-18T04:38:21.251 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:21.251 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.251 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:21.251 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6812", 2024-09-18T04:38:21.251 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 511141401 2024-09-18T04:38:21.251 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.251 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.251 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:21.251 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6814", 2024-09-18T04:38:21.251 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 511141401 2024-09-18T04:38:21.252 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:21.252 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:21.252 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.252 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-18T04:38:21.252 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:21.252 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.252 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:21.252 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6808", 2024-09-18T04:38:21.252 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 511141401 2024-09-18T04:38:21.252 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.252 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.252 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:21.252 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6810", 2024-09-18T04:38:21.252 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 511141401 2024-09-18T04:38:21.253 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:21.253 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:21.253 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.253 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6802/511141401", 2024-09-18T04:38:21.253 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6806/511141401", 2024-09-18T04:38:21.253 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6814/511141401", 2024-09-18T04:38:21.253 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6810/511141401", 2024-09-18T04:38:21.253 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-18T04:38:21.253 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-18T04:38:21.253 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-18T04:38:21.253 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:21.253 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.253 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.254 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-09-18T04:38:21.254 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "854fe031-3b87-4290-9c2f-3fb196fb806c", 2024-09-18T04:38:21.254 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-18T04:38:21.254 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-18T04:38:21.254 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-18T04:38:21.254 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-18T04:38:21.254 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-18T04:38:21.254 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-18T04:38:21.254 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 12, 2024-09-18T04:38:21.254 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 28, 2024-09-18T04:38:21.254 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-18T04:38:21.254 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-18T04:38:21.254 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-18T04:38:21.255 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:21.255 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.255 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:21.255 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6816", 2024-09-18T04:38:21.255 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2623426735 2024-09-18T04:38:21.255 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.255 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.255 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:21.255 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6817", 2024-09-18T04:38:21.255 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2623426735 2024-09-18T04:38:21.255 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:21.255 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:21.255 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.256 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-18T04:38:21.256 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:21.256 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.256 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:21.256 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6818", 2024-09-18T04:38:21.256 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2623426735 2024-09-18T04:38:21.256 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.256 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.256 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:21.256 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6819", 2024-09-18T04:38:21.256 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2623426735 2024-09-18T04:38:21.256 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:21.257 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:21.257 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.257 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-18T04:38:21.257 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:21.257 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.257 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:21.257 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6822", 2024-09-18T04:38:21.257 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2623426735 2024-09-18T04:38:21.257 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.257 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.257 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:21.257 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6823", 2024-09-18T04:38:21.258 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2623426735 2024-09-18T04:38:21.258 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:21.258 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:21.258 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.258 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-18T04:38:21.258 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:21.258 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.258 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:21.258 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6820", 2024-09-18T04:38:21.258 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2623426735 2024-09-18T04:38:21.258 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.258 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.258 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:21.258 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6821", 2024-09-18T04:38:21.258 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2623426735 2024-09-18T04:38:21.258 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:21.259 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:21.259 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.259 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6817/2623426735", 2024-09-18T04:38:21.259 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6819/2623426735", 2024-09-18T04:38:21.259 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6823/2623426735", 2024-09-18T04:38:21.259 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6821/2623426735", 2024-09-18T04:38:21.259 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-18T04:38:21.259 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-18T04:38:21.259 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-18T04:38:21.259 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:21.260 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.260 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.260 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-09-18T04:38:21.260 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "c2625f98-780b-4f86-bf36-a48d2a227756", 2024-09-18T04:38:21.260 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-18T04:38:21.260 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-18T04:38:21.260 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-18T04:38:21.260 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-18T04:38:21.260 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-18T04:38:21.260 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-18T04:38:21.260 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 12, 2024-09-18T04:38:21.260 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 32, 2024-09-18T04:38:21.260 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-18T04:38:21.260 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-18T04:38:21.261 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-18T04:38:21.261 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:21.261 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.261 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:21.261 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6808", 2024-09-18T04:38:21.261 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2746051132 2024-09-18T04:38:21.261 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.261 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.261 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:21.261 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6809", 2024-09-18T04:38:21.261 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2746051132 2024-09-18T04:38:21.262 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:21.262 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:21.262 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.262 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-18T04:38:21.262 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:21.262 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.262 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:21.262 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6810", 2024-09-18T04:38:21.262 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2746051132 2024-09-18T04:38:21.262 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.262 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.262 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:21.262 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6811", 2024-09-18T04:38:21.262 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2746051132 2024-09-18T04:38:21.262 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:21.263 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:21.263 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.263 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-18T04:38:21.263 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:21.263 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.263 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:21.263 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6814", 2024-09-18T04:38:21.263 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2746051132 2024-09-18T04:38:21.263 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.263 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.263 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:21.263 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6815", 2024-09-18T04:38:21.264 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2746051132 2024-09-18T04:38:21.264 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:21.264 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:21.264 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.264 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-18T04:38:21.264 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:21.264 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.264 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:21.264 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6812", 2024-09-18T04:38:21.264 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2746051132 2024-09-18T04:38:21.264 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.264 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.265 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:21.265 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6813", 2024-09-18T04:38:21.265 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2746051132 2024-09-18T04:38:21.265 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:21.265 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:21.265 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.265 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.82:6809/2746051132", 2024-09-18T04:38:21.265 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.82:6811/2746051132", 2024-09-18T04:38:21.265 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.82:6815/2746051132", 2024-09-18T04:38:21.265 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.82:6813/2746051132", 2024-09-18T04:38:21.265 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-18T04:38:21.265 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-18T04:38:21.265 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-18T04:38:21.266 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:21.266 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.266 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.266 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-09-18T04:38:21.266 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "deb22d86-4081-484d-96eb-9cb9d96b60bc", 2024-09-18T04:38:21.266 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-18T04:38:21.266 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-18T04:38:21.266 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0.3929290771484375, 2024-09-18T04:38:21.266 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-18T04:38:21.266 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-18T04:38:21.266 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-18T04:38:21.266 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 13, 2024-09-18T04:38:21.266 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 0, 2024-09-18T04:38:21.267 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-18T04:38:21.267 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-18T04:38:21.267 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-18T04:38:21.267 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:21.267 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.267 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:21.267 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6800", 2024-09-18T04:38:21.267 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2005712460 2024-09-18T04:38:21.267 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.267 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.267 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:21.267 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6801", 2024-09-18T04:38:21.267 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2005712460 2024-09-18T04:38:21.268 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:21.268 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:21.268 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.268 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-18T04:38:21.268 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:21.268 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.268 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:21.268 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6802", 2024-09-18T04:38:21.268 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2005712460 2024-09-18T04:38:21.268 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.268 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.268 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:21.269 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6803", 2024-09-18T04:38:21.269 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2005712460 2024-09-18T04:38:21.269 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:21.269 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:21.269 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.269 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-18T04:38:21.269 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:21.269 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.269 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:21.269 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6806", 2024-09-18T04:38:21.269 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2005712460 2024-09-18T04:38:21.269 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.269 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.269 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:21.270 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6807", 2024-09-18T04:38:21.270 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2005712460 2024-09-18T04:38:21.270 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:21.270 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:21.270 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.270 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-18T04:38:21.270 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:21.270 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.270 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:21.270 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6804", 2024-09-18T04:38:21.270 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2005712460 2024-09-18T04:38:21.270 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.270 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.270 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:21.271 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6805", 2024-09-18T04:38:21.271 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2005712460 2024-09-18T04:38:21.271 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:21.271 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:21.271 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.271 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.82:6801/2005712460", 2024-09-18T04:38:21.271 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.82:6803/2005712460", 2024-09-18T04:38:21.271 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.82:6807/2005712460", 2024-09-18T04:38:21.271 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.82:6805/2005712460", 2024-09-18T04:38:21.271 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-18T04:38:21.271 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-18T04:38:21.271 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-18T04:38:21.271 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:21.271 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.271 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.272 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-09-18T04:38:21.272 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "061d332c-f51b-403d-bb18-eba415c97e7f", 2024-09-18T04:38:21.272 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-18T04:38:21.272 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-18T04:38:21.272 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-18T04:38:21.272 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-18T04:38:21.272 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-18T04:38:21.272 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-18T04:38:21.272 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 13, 2024-09-18T04:38:21.272 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 24, 2024-09-18T04:38:21.272 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-18T04:38:21.272 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-18T04:38:21.272 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-18T04:38:21.272 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:21.272 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.273 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:21.273 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6816", 2024-09-18T04:38:21.273 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1267644232 2024-09-18T04:38:21.273 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.273 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.273 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:21.273 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6817", 2024-09-18T04:38:21.273 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1267644232 2024-09-18T04:38:21.273 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:21.273 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:21.273 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.273 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-18T04:38:21.273 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:21.274 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.274 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:21.274 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6818", 2024-09-18T04:38:21.274 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1267644232 2024-09-18T04:38:21.274 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.274 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.274 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:21.274 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6819", 2024-09-18T04:38:21.274 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1267644232 2024-09-18T04:38:21.274 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:21.274 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:21.274 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.275 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-18T04:38:21.275 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:21.275 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.275 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:21.275 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6822", 2024-09-18T04:38:21.275 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1267644232 2024-09-18T04:38:21.275 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.275 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.275 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:21.275 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6823", 2024-09-18T04:38:21.275 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1267644232 2024-09-18T04:38:21.275 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:21.275 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:21.276 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.276 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-18T04:38:21.276 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:21.276 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.276 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:21.276 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6820", 2024-09-18T04:38:21.276 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1267644232 2024-09-18T04:38:21.276 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.276 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.276 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:21.276 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6821", 2024-09-18T04:38:21.276 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1267644232 2024-09-18T04:38:21.276 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:21.276 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:21.277 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.277 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.82:6817/1267644232", 2024-09-18T04:38:21.277 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.82:6819/1267644232", 2024-09-18T04:38:21.277 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.82:6823/1267644232", 2024-09-18T04:38:21.277 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.82:6821/1267644232", 2024-09-18T04:38:21.277 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-18T04:38:21.277 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-18T04:38:21.277 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-18T04:38:21.277 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:21.277 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:21.277 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-18T04:38:21.277 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_xinfo": [ 2024-09-18T04:38:21.277 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.277 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-09-18T04:38:21.278 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-18T04:38:21.278 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-18T04:38:21.278 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-18T04:38:21.278 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138322906710015, 2024-09-18T04:38:21.278 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-18T04:38:21.278 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-18T04:37:22.192087+0000", 2024-09-18T04:38:21.278 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-18T04:38:21.278 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.278 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.278 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-09-18T04:38:21.278 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-18T04:38:21.278 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-18T04:38:21.278 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-18T04:38:21.279 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138322906710015, 2024-09-18T04:38:21.279 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-18T04:38:21.279 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-18T04:37:22.017833+0000", 2024-09-18T04:38:21.279 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-18T04:38:21.279 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.279 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.279 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-09-18T04:38:21.279 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-18T04:38:21.279 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-18T04:38:21.279 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-18T04:38:21.279 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138322906710015, 2024-09-18T04:38:21.279 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-18T04:38:21.279 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-18T04:37:22.755774+0000", 2024-09-18T04:38:21.280 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-18T04:38:21.280 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.280 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.280 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-09-18T04:38:21.280 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-18T04:38:21.280 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-18T04:38:21.280 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-18T04:38:21.280 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138322906710015, 2024-09-18T04:38:21.280 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-18T04:38:21.280 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-18T04:37:22.663785+0000", 2024-09-18T04:38:21.280 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-18T04:38:21.280 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.280 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.281 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-09-18T04:38:21.281 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-18T04:38:21.281 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-18T04:38:21.281 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-18T04:38:21.281 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138322906710015, 2024-09-18T04:38:21.281 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-18T04:38:21.281 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-18T04:37:22.531758+0000", 2024-09-18T04:38:21.281 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-18T04:38:21.281 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.281 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.281 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-09-18T04:38:21.281 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-18T04:38:21.281 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-18T04:38:21.282 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-18T04:38:21.282 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138322906710015, 2024-09-18T04:38:21.282 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-18T04:38:21.282 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-18T04:37:22.400774+0000", 2024-09-18T04:38:21.282 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-18T04:38:21.282 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:21.282 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-18T04:38:21.282 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap": [ 2024-09-18T04:38:21.282 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.282 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "2.5", 2024-09-18T04:38:21.282 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-09-18T04:38:21.282 INFO:teuthology.orchestra.run.smithi028.stdout: 2, 2024-09-18T04:38:21.282 INFO:teuthology.orchestra.run.smithi028.stdout: 0 2024-09-18T04:38:21.283 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:21.283 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.283 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:21.283 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "2.7", 2024-09-18T04:38:21.283 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-09-18T04:38:21.283 INFO:teuthology.orchestra.run.smithi028.stdout: 3, 2024-09-18T04:38:21.283 INFO:teuthology.orchestra.run.smithi028.stdout: 5 2024-09-18T04:38:21.283 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:21.283 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:21.283 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-18T04:38:21.283 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap_items": [], 2024-09-18T04:38:21.283 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap_primaries": [], 2024-09-18T04:38:21.283 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_temp": [], 2024-09-18T04:38:21.283 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_temp": [], 2024-09-18T04:38:21.283 INFO:teuthology.orchestra.run.smithi028.stdout: "blocklist": {}, 2024-09-18T04:38:21.284 INFO:teuthology.orchestra.run.smithi028.stdout: "range_blocklist": {}, 2024-09-18T04:38:21.284 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profiles": { 2024-09-18T04:38:21.284 INFO:teuthology.orchestra.run.smithi028.stdout: "default": { 2024-09-18T04:38:21.284 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-failure-domain": "osd", 2024-09-18T04:38:21.284 INFO:teuthology.orchestra.run.smithi028.stdout: "k": "2", 2024-09-18T04:38:21.284 INFO:teuthology.orchestra.run.smithi028.stdout: "m": "1", 2024-09-18T04:38:21.284 INFO:teuthology.orchestra.run.smithi028.stdout: "plugin": "jerasure", 2024-09-18T04:38:21.284 INFO:teuthology.orchestra.run.smithi028.stdout: "technique": "reed_sol_van" 2024-09-18T04:38:21.284 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:21.284 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:21.284 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps_queue": [], 2024-09-18T04:38:21.284 INFO:teuthology.orchestra.run.smithi028.stdout: "new_removed_snaps": [], 2024-09-18T04:38:21.285 INFO:teuthology.orchestra.run.smithi028.stdout: "new_purged_snaps": [], 2024-09-18T04:38:21.285 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_node_flags": {}, 2024-09-18T04:38:21.285 INFO:teuthology.orchestra.run.smithi028.stdout: "device_class_flags": {}, 2024-09-18T04:38:21.285 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": { 2024-09-18T04:38:21.285 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_enabled": false, 2024-09-18T04:38:21.285 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_bucket_count": 0, 2024-09-18T04:38:21.285 INFO:teuthology.orchestra.run.smithi028.stdout: "degraded_stretch_mode": 0, 2024-09-18T04:38:21.285 INFO:teuthology.orchestra.run.smithi028.stdout: "recovering_stretch_mode": 0, 2024-09-18T04:38:21.285 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_bucket": 0 2024-09-18T04:38:21.285 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:21.285 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:21.287 INFO:tasks.thrashosds.thrasher:j is {'epoch': 39, 'fsid': 'be6146ba-a991-4f6e-b359-99cbedc9f9b1', 'created': '2024-09-18T04:37:15.676634+0000', 'modified': '2024-09-18T04:38:20.803684+0000', 'last_up_change': '2024-09-18T04:37:25.403175+0000', 'last_in_change': '2024-09-18T04:37:50.652169+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-18T04:37:25.654950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-18T04:37:32.672462+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 20, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.5099999904632568, 'score_stable': 1.5099999904632568, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '6177002d-72e6-4888-bcde-29b6a92d9dc4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6801', 'nonce': 1554407851}, {'type': 'v1', 'addr': '172.21.15.28:6803', 'nonce': 1554407851}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6805', 'nonce': 1554407851}, {'type': 'v1', 'addr': '172.21.15.28:6807', 'nonce': 1554407851}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6813', 'nonce': 1554407851}, {'type': 'v1', 'addr': '172.21.15.28:6815', 'nonce': 1554407851}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6809', 'nonce': 1554407851}, {'type': 'v1', 'addr': '172.21.15.28:6811', 'nonce': 1554407851}]}, 'public_addr': '172.21.15.28:6803/1554407851', 'cluster_addr': '172.21.15.28:6807/1554407851', 'heartbeat_back_addr': '172.21.15.28:6815/1554407851', 'heartbeat_front_addr': '172.21.15.28:6811/1554407851', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '4f62cb09-3457-418c-bb69-491d9b70914c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 21, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6800', 'nonce': 511141401}, {'type': 'v1', 'addr': '172.21.15.28:6802', 'nonce': 511141401}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6804', 'nonce': 511141401}, {'type': 'v1', 'addr': '172.21.15.28:6806', 'nonce': 511141401}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6812', 'nonce': 511141401}, {'type': 'v1', 'addr': '172.21.15.28:6814', 'nonce': 511141401}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6808', 'nonce': 511141401}, {'type': 'v1', 'addr': '172.21.15.28:6810', 'nonce': 511141401}]}, 'public_addr': '172.21.15.28:6802/511141401', 'cluster_addr': '172.21.15.28:6806/511141401', 'heartbeat_back_addr': '172.21.15.28:6814/511141401', 'heartbeat_front_addr': '172.21.15.28:6810/511141401', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '854fe031-3b87-4290-9c2f-3fb196fb806c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 28, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6816', 'nonce': 2623426735}, {'type': 'v1', 'addr': '172.21.15.28:6817', 'nonce': 2623426735}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6818', 'nonce': 2623426735}, {'type': 'v1', 'addr': '172.21.15.28:6819', 'nonce': 2623426735}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6822', 'nonce': 2623426735}, {'type': 'v1', 'addr': '172.21.15.28:6823', 'nonce': 2623426735}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6820', 'nonce': 2623426735}, {'type': 'v1', 'addr': '172.21.15.28:6821', 'nonce': 2623426735}]}, 'public_addr': '172.21.15.28:6817/2623426735', 'cluster_addr': '172.21.15.28:6819/2623426735', 'heartbeat_back_addr': '172.21.15.28:6823/2623426735', 'heartbeat_front_addr': '172.21.15.28:6821/2623426735', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'c2625f98-780b-4f86-bf36-a48d2a227756', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 32, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6808', 'nonce': 2746051132}, {'type': 'v1', 'addr': '172.21.15.82:6809', 'nonce': 2746051132}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6810', 'nonce': 2746051132}, {'type': 'v1', 'addr': '172.21.15.82:6811', 'nonce': 2746051132}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6814', 'nonce': 2746051132}, {'type': 'v1', 'addr': '172.21.15.82:6815', 'nonce': 2746051132}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6812', 'nonce': 2746051132}, {'type': 'v1', 'addr': '172.21.15.82:6813', 'nonce': 2746051132}]}, 'public_addr': '172.21.15.82:6809/2746051132', 'cluster_addr': '172.21.15.82:6811/2746051132', 'heartbeat_back_addr': '172.21.15.82:6815/2746051132', 'heartbeat_front_addr': '172.21.15.82:6813/2746051132', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'deb22d86-4081-484d-96eb-9cb9d96b60bc', 'up': 1, 'in': 1, 'weight': 0.3929290771484375, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6800', 'nonce': 2005712460}, {'type': 'v1', 'addr': '172.21.15.82:6801', 'nonce': 2005712460}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6802', 'nonce': 2005712460}, {'type': 'v1', 'addr': '172.21.15.82:6803', 'nonce': 2005712460}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6806', 'nonce': 2005712460}, {'type': 'v1', 'addr': '172.21.15.82:6807', 'nonce': 2005712460}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6804', 'nonce': 2005712460}, {'type': 'v1', 'addr': '172.21.15.82:6805', 'nonce': 2005712460}]}, 'public_addr': '172.21.15.82:6801/2005712460', 'cluster_addr': '172.21.15.82:6803/2005712460', 'heartbeat_back_addr': '172.21.15.82:6807/2005712460', 'heartbeat_front_addr': '172.21.15.82:6805/2005712460', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '061d332c-f51b-403d-bb18-eba415c97e7f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 24, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6816', 'nonce': 1267644232}, {'type': 'v1', 'addr': '172.21.15.82:6817', 'nonce': 1267644232}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6818', 'nonce': 1267644232}, {'type': 'v1', 'addr': '172.21.15.82:6819', 'nonce': 1267644232}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6822', 'nonce': 1267644232}, {'type': 'v1', 'addr': '172.21.15.82:6823', 'nonce': 1267644232}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6820', 'nonce': 1267644232}, {'type': 'v1', 'addr': '172.21.15.82:6821', 'nonce': 1267644232}]}, 'public_addr': '172.21.15.82:6817/1267644232', 'cluster_addr': '172.21.15.82:6819/1267644232', 'heartbeat_back_addr': '172.21.15.82:6823/1267644232', 'heartbeat_front_addr': '172.21.15.82:6821/1267644232', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T04:37:22.192087+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T04:37:22.017833+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T04:37:22.755774+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T04:37:22.663785+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T04:37:22.531758+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T04:37:22.400774+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.5', 'osds': [2, 0]}, {'pgid': '2.7', 'osds': [3, 5]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-18T04:38:21.287 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T04:38:21.288 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:21.282+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:21.321 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:21.321 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:21.331 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:38:21.387 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:21.383+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:21.434 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:21.435 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:21.444 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:38:21.488 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:21.483+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:21.550 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:21.550 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:21.559 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:38:21.588 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:21.582+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:21.599 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:38:21.603 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-18T04:38:21.614 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":61,"stamp":"2024-09-18T04:38:20.817007+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":5,"kb":566231040,"kb_used":185104,"kb_used_data":2500,"kb_used_omap":9,"kb_used_meta":182518,"kb_avail":566045936,"statfs":{"total":579820584960,"available":579631038464,"internally_reserved":0,"allocated":2560000,"data_stored":2036176,"data_compressed":66040,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":9561,"internal_metadata":186899111},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.214155"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":41,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T04:38:15.223987+0000","last_change":"2024-09-18T04:38:05.754820+0000","last_active":"2024-09-18T04:38:15.223987+0000","last_peered":"2024-09-18T04:38:15.223987+0000","last_clean":"2024-09-18T04:38:15.223987+0000","last_became_active":"2024-09-18T04:38:05.753962+0000","last_became_peered":"2024-09-18T04:38:05.753962+0000","last_unstale":"2024-09-18T04:38:15.223987+0000","last_undegraded":"2024-09-18T04:38:15.223987+0000","last_fullsized":"2024-09-18T04:38:15.223987+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:53:52.224098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047343899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":33,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T04:38:14.798024+0000","last_change":"2024-09-18T04:37:36.494193+0000","last_active":"2024-09-18T04:38:14.798024+0000","last_peered":"2024-09-18T04:38:14.798024+0000","last_clean":"2024-09-18T04:38:14.798024+0000","last_became_active":"2024-09-18T04:37:34.472940+0000","last_became_peered":"2024-09-18T04:37:34.472940+0000","last_unstale":"2024-09-18T04:38:14.798024+0000","last_undegraded":"2024-09-18T04:38:14.798024+0000","last_fullsized":"2024-09-18T04:38:14.798024+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:18:29.266815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070553900000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T04:38:14.798313+0000","last_change":"2024-09-18T04:37:59.284861+0000","last_active":"2024-09-18T04:38:14.798313+0000","last_peered":"2024-09-18T04:38:14.798313+0000","last_clean":"2024-09-18T04:38:14.798313+0000","last_became_active":"2024-09-18T04:37:59.283691+0000","last_became_peered":"2024-09-18T04:37:59.283691+0000","last_unstale":"2024-09-18T04:38:14.798313+0000","last_undegraded":"2024-09-18T04:38:14.798313+0000","last_fullsized":"2024-09-18T04:38:14.798313+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:19:36.536686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":33,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T04:38:14.798066+0000","last_change":"2024-09-18T04:37:36.494228+0000","last_active":"2024-09-18T04:38:14.798066+0000","last_peered":"2024-09-18T04:38:14.798066+0000","last_clean":"2024-09-18T04:38:14.798066+0000","last_became_active":"2024-09-18T04:37:34.471265+0000","last_became_peered":"2024-09-18T04:37:34.471265+0000","last_unstale":"2024-09-18T04:38:14.798066+0000","last_undegraded":"2024-09-18T04:38:14.798066+0000","last_fullsized":"2024-09-18T04:38:14.798066+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:23:05.748076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081514499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"20'2","reported_seq":34,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T04:38:11.770166+0000","last_change":"2024-09-18T04:37:36.490461+0000","last_active":"2024-09-18T04:38:11.770166+0000","last_peered":"2024-09-18T04:38:11.770166+0000","last_clean":"2024-09-18T04:38:11.770166+0000","last_became_active":"2024-09-18T04:37:34.473088+0000","last_became_peered":"2024-09-18T04:37:34.473088+0000","last_unstale":"2024-09-18T04:38:11.770166+0000","last_undegraded":"2024-09-18T04:38:11.770166+0000","last_fullsized":"2024-09-18T04:38:11.770166+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:17:57.352397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001879932,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T04:38:14.798265+0000","last_change":"2024-09-18T04:37:51.679120+0000","last_active":"2024-09-18T04:38:14.798265+0000","last_peered":"2024-09-18T04:38:14.798265+0000","last_clean":"2024-09-18T04:38:14.798265+0000","last_became_active":"2024-09-18T04:37:51.678711+0000","last_became_peered":"2024-09-18T04:37:51.678711+0000","last_unstale":"2024-09-18T04:38:14.798265+0000","last_undegraded":"2024-09-18T04:38:14.798265+0000","last_fullsized":"2024-09-18T04:38:14.798265+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T04:46:09.459905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":33,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T04:38:15.223880+0000","last_change":"2024-09-18T04:37:36.484950+0000","last_active":"2024-09-18T04:38:15.223880+0000","last_peered":"2024-09-18T04:38:15.223880+0000","last_clean":"2024-09-18T04:38:15.223880+0000","last_became_active":"2024-09-18T04:37:34.470841+0000","last_became_peered":"2024-09-18T04:37:34.470841+0000","last_unstale":"2024-09-18T04:38:15.223880+0000","last_undegraded":"2024-09-18T04:38:15.223880+0000","last_fullsized":"2024-09-18T04:38:15.223880+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:16:34.528783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090513900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"18'1","reported_seq":56,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T04:38:11.770143+0000","last_change":"2024-09-18T04:37:51.681357+0000","last_active":"2024-09-18T04:38:11.770143+0000","last_peered":"2024-09-18T04:38:11.770143+0000","last_clean":"2024-09-18T04:38:11.770143+0000","last_became_active":"2024-09-18T04:37:51.676131+0000","last_became_peered":"2024-09-18T04:37:51.676131+0000","last_unstale":"2024-09-18T04:38:11.770143+0000","last_undegraded":"2024-09-18T04:38:11.770143+0000","last_fullsized":"2024-09-18T04:38:11.770143+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:45:22.605009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018293420000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":181,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T04:38:15.223899+0000","last_change":"2024-09-18T04:37:27.423069+0000","last_active":"2024-09-18T04:38:15.223899+0000","last_peered":"2024-09-18T04:38:15.223899+0000","last_clean":"2024-09-18T04:38:15.223899+0000","last_became_active":"2024-09-18T04:37:27.422350+0000","last_became_peered":"2024-09-18T04:37:27.422350+0000","last_unstale":"2024-09-18T04:38:15.223899+0000","last_undegraded":"2024-09-18T04:38:15.223899+0000","last_fullsized":"2024-09-18T04:38:15.223899+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:26.406292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:26.406292+0000","last_clean_scrub_stamp":"2024-09-18T04:37:26.406292+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:31:45.852149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":66040,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":13,"seq":55834574862,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27412,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":401408,"data_stored":140284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85699999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98099999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75900000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93200000000000005}]}]},{"osd":4,"up_from":13,"seq":55834574862,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27408,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344432,"statfs":{"total":96636764160,"available":96608698368,"internally_reserved":0,"allocated":397312,"data_stored":140265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82499999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86399999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574863,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35664,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94336176,"statfs":{"total":96636764160,"available":96600244224,"internally_reserved":0,"allocated":462848,"data_stored":730633,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95599999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97999999999999998}]}]},{"osd":3,"up_from":12,"seq":51539607567,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31584,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340256,"statfs":{"total":96636764160,"available":96604422144,"internally_reserved":0,"allocated":475136,"data_stored":735237,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.665}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4510000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.577}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78000000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83099999999999996}]}]},{"osd":2,"up_from":12,"seq":51539607565,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31516,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340324,"statfs":{"total":96636764160,"available":96604491776,"internally_reserved":0,"allocated":409600,"data_stored":144869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.135}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0409999999999999}]}]},{"osd":1,"up_from":12,"seq":51539607567,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31520,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340320,"statfs":{"total":96636764160,"available":96604487680,"internally_reserved":0,"allocated":413696,"data_stored":144888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.497}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79800000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T04:38:21.615 INFO:tasks.thrashosds.thrasher:Setting 2.1 to [4, 5, 2, 1] 2024-09-18T04:38:21.615 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.1', '4', '5', '2', '1'] 2024-09-18T04:38:21.616 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 2.1 4 5 2 1 2024-09-18T04:38:21.663 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:21.663 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:21.673 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:38:21.714 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:21.710+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:21.789 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:21.783+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:21.826 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:21.826 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:21.836 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:38:21.889 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:21.882+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:21.939 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:21.940 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:21.949 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:38:21.994 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:21.990+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:22.058 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:22.058 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:22.067 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:38:22.091 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:22.087+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:22.177 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:22.177 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:22.187 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:38:22.191 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:22.186+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:22.289 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:22.289 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:22.292 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:22.287+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:22.299 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:38:22.394 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:22.386+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:22.414 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:22.414 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:22.424 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:38:22.495 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:22.490+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:22.533 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:22.533 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:22.543 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:38:22.596 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:22.591+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:22.603 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T04:38:22.647 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:22.647 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:22.657 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:38:22.698 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:22.695+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:22.739 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:38:22.760 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:38:22.778 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:38:22.797 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:22.790+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:22.801 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:38:22.805 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:22.805 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:22.805 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:38:22.806 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:22.815 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:38:22.825 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:38:22.834 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-18T04:38:22.835 INFO:teuthology.orchestra.run.smithi028.stderr:set 2.1 pg_upmap_items mapping to [4->5,2->1] 2024-09-18T04:38:22.857 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:38:22.859 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:38:22.859 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:38:22.860 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:38:22.860 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:38:22.860 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:38:22.860 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:38:22.898 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:22.895+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:22.916 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:22.917 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:22.917 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:38:22.917 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:38:22.917 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:22.927 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:38:22.999 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:22.994+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:23.088 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:23.088 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:38:23.088 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:38:23.088 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:38:23.088 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:23.088 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4214.0:5 2.2 e19/17 2:5cea7035:::rbd_info:head v 19'1, mlcod=0'0)", 2024-09-18T04:38:23.088 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:35.471260+0000", 2024-09-18T04:38:23.088 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 47.615392858999996, 2024-09-18T04:38:23.089 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021025950000000001, 2024-09-18T04:38:23.089 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:23.089 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:23.089 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:23.089 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:23.089 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:23.089 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471260+0000", 2024-09-18T04:38:23.089 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:23.089 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:23.089 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:23.089 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:23.089 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471260+0000", 2024-09-18T04:38:23.089 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:23.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:23.090 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:23.090 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:23.090 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471259+0000", 2024-09-18T04:38:23.090 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:23.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:23.090 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:23.090 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:23.090 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471275+0000", 2024-09-18T04:38:23.090 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6135999999999999e-05 2024-09-18T04:38:23.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:23.090 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:23.091 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:23.091 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471298+0000", 2024-09-18T04:38:23.091 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3632999999999999e-05 2024-09-18T04:38:23.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:23.091 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:23.091 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:23.091 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471320+0000", 2024-09-18T04:38:23.091 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1852999999999999e-05 2024-09-18T04:38:23.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:23.091 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:23.091 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:23.091 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471445+0000", 2024-09-18T04:38:23.091 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012461699999999999 2024-09-18T04:38:23.092 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:23.092 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:23.092 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:23.092 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471529+0000", 2024-09-18T04:38:23.092 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3875000000000006e-05 2024-09-18T04:38:23.092 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:23.092 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:23.092 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:23.092 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.473317+0000", 2024-09-18T04:38:23.092 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017886899999999999 2024-09-18T04:38:23.092 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:23.092 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:23.092 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:23.093 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.473362+0000", 2024-09-18T04:38:23.093 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4724000000000001e-05 2024-09-18T04:38:23.093 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:23.093 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:23.093 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:23.093 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:23.093 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:23.093 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4214.0:7 2.2 e20/17 2:5cea7035:::rbd_info:head v 20'2, mlcod=0'0)", 2024-09-18T04:38:23.093 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:36.490560+0000", 2024-09-18T04:38:23.093 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 46.596092720000001, 2024-09-18T04:38:23.093 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0015326020000000001, 2024-09-18T04:38:23.093 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:23.093 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:23.094 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:23.094 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:23.094 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:23.094 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490560+0000", 2024-09-18T04:38:23.094 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:23.094 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:23.094 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:23.094 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:23.094 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490560+0000", 2024-09-18T04:38:23.094 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:23.094 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:23.094 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:23.094 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:23.095 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490559+0000", 2024-09-18T04:38:23.095 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:23.095 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:23.095 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:23.095 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:23.095 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490568+0000", 2024-09-18T04:38:23.095 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5699999999999993e-06 2024-09-18T04:38:23.095 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:23.095 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:23.095 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:23.095 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490599+0000", 2024-09-18T04:38:23.095 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1566000000000002e-05 2024-09-18T04:38:23.095 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:23.096 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:23.096 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:23.096 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490609+0000", 2024-09-18T04:38:23.096 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.8700000000000004e-06 2024-09-18T04:38:23.096 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:23.096 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:23.096 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:23.096 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490708+0000", 2024-09-18T04:38:23.096 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9139000000000001e-05 2024-09-18T04:38:23.096 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:23.096 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:23.096 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:23.096 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490736+0000", 2024-09-18T04:38:23.096 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7617e-05 2024-09-18T04:38:23.097 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:23.097 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:23.097 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:23.097 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.492016+0000", 2024-09-18T04:38:23.097 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0012796579999999999 2024-09-18T04:38:23.097 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:23.097 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:23.097 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:23.097 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.492092+0000", 2024-09-18T04:38:23.097 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6571999999999998e-05 2024-09-18T04:38:23.097 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:23.098 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:23.098 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:23.098 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:23.098 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:23.098 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:23.099 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:38:23.100 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:23.094+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:23.201 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:23.195+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:23.204 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:23.204 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:23.204 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:38:23.205 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:23.214 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:38:23.302 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:23.298+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:23.324 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:23.324 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:23.324 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:38:23.325 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:38:23.325 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:23.334 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:38:23.403 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:23.399+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:23.447 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:23.447 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:38:23.447 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:38:23.447 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:38:23.447 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:23.447 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4214.0:1 2.3 e18/17 2:c4c92e5a:::rbd_trash:head v 18'1, mlcod=0'0)", 2024-09-18T04:38:23.447 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:34.475123+0000", 2024-09-18T04:38:23.448 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 48.970812641000002, 2024-09-18T04:38:23.448 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00081394599999999996, 2024-09-18T04:38:23.448 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:23.448 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:23.448 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:23.448 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:23.448 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:23.448 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475123+0000", 2024-09-18T04:38:23.448 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:23.448 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:23.448 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:23.448 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:23.448 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475123+0000", 2024-09-18T04:38:23.449 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:23.449 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:23.449 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:23.449 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:23.449 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475121+0000", 2024-09-18T04:38:23.449 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:23.449 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:23.449 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:23.449 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:23.449 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475149+0000", 2024-09-18T04:38:23.449 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7155999999999999e-05 2024-09-18T04:38:23.449 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:23.449 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:23.450 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:23.450 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475165+0000", 2024-09-18T04:38:23.450 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6160000000000001e-05 2024-09-18T04:38:23.450 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:23.450 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:23.450 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:23.450 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475183+0000", 2024-09-18T04:38:23.450 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7918e-05 2024-09-18T04:38:23.450 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:23.450 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:23.450 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:23.450 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475246+0000", 2024-09-18T04:38:23.450 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3176000000000006e-05 2024-09-18T04:38:23.451 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:23.451 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:23.451 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:23.451 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475279+0000", 2024-09-18T04:38:23.451 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3192999999999999e-05 2024-09-18T04:38:23.451 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:23.451 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:23.451 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:23.451 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475871+0000", 2024-09-18T04:38:23.451 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00059206099999999995 2024-09-18T04:38:23.451 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:23.451 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:23.451 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:23.452 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475937+0000", 2024-09-18T04:38:23.452 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5439999999999997e-05 2024-09-18T04:38:23.452 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:23.452 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:23.452 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:23.452 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:23.452 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:23.452 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 25/24 [PushOp(2:c4c92e5a:::rbd_trash:head, version: 18'1, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:c4c92e5a:::rbd_trash:head@18'1, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-18T04:38:23.452 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:51.680544+0000", 2024-09-18T04:38:23.452 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 31.765390798999999, 2024-09-18T04:38:23.452 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00050365200000000005, 2024-09-18T04:38:23.452 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:23.452 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:38:23.452 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:23.452 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:23.453 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:23.453 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680544+0000", 2024-09-18T04:38:23.453 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:23.453 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:23.453 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:23.453 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:23.453 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680544+0000", 2024-09-18T04:38:23.453 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:23.453 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:23.453 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:23.453 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:23.453 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680543+0000", 2024-09-18T04:38:23.453 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:23.453 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:23.454 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:23.454 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:23.454 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680571+0000", 2024-09-18T04:38:23.454 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7953000000000002e-05 2024-09-18T04:38:23.454 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:23.454 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:23.454 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:23.454 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680605+0000", 2024-09-18T04:38:23.454 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3933000000000002e-05 2024-09-18T04:38:23.454 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:23.454 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:23.454 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:23.454 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680636+0000", 2024-09-18T04:38:23.454 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1288999999999998e-05 2024-09-18T04:38:23.454 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:23.455 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:23.455 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:23.455 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680748+0000", 2024-09-18T04:38:23.455 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000111431 2024-09-18T04:38:23.455 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:23.455 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:23.455 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:23.455 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680758+0000", 2024-09-18T04:38:23.455 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0244999999999999e-05 2024-09-18T04:38:23.455 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:23.455 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:23.455 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:23.455 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.681048+0000", 2024-09-18T04:38:23.455 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00029003399999999999 2024-09-18T04:38:23.456 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:23.456 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:23.456 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:23.456 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:23.456 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:23.456 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:23.457 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:38:23.504 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:23.499+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:23.576 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:23.576 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:23.576 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:38:23.576 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:23.586 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:38:23.604 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:23.599+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:23.693 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:23.693 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:23.694 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:38:23.694 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:38:23.694 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:23.703 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:38:23.704 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:23.698+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:23.806 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:23.802+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:23.813 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:23.813 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:38:23.813 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:38:23.813 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:38:23.814 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.814 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:23.814 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.573328+0000", 2024-09-18T04:38:23.814 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 56.236281146000003, 2024-09-18T04:38:23.814 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011397898, 2024-09-18T04:38:23.814 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:23.814 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:23.814 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:23.814 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:23.814 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:23.814 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 86 2024-09-18T04:38:23.814 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.814 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:23.814 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.814 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:23.815 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573328+0000", 2024-09-18T04:38:23.815 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:23.815 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.815 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.815 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:23.815 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573328+0000", 2024-09-18T04:38:23.815 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:23.815 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.815 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.815 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:23.815 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573327+0000", 2024-09-18T04:38:23.815 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:23.815 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.815 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.816 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:23.816 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573331+0000", 2024-09-18T04:38:23.816 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9750000000000001e-06 2024-09-18T04:38:23.816 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.816 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.816 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:23.816 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573346+0000", 2024-09-18T04:38:23.816 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5469999999999999e-05 2024-09-18T04:38:23.816 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.816 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.817 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:23.817 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573370+0000", 2024-09-18T04:38:23.817 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3969999999999999e-05 2024-09-18T04:38:23.817 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.817 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.817 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:23.817 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.578536+0000", 2024-09-18T04:38:23.817 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0051659679999999999 2024-09-18T04:38:23.817 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.817 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.817 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:23.817 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.578689+0000", 2024-09-18T04:38:23.817 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015276299999999999 2024-09-18T04:38:23.818 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.818 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.818 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:23.818 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.578901+0000", 2024-09-18T04:38:23.818 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000211783 2024-09-18T04:38:23.818 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.818 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.818 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:23.818 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.583408+0000", 2024-09-18T04:38:23.818 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045071249999999998 2024-09-18T04:38:23.818 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.818 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.818 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:23.818 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.584645+0000", 2024-09-18T04:38:23.819 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001237007 2024-09-18T04:38:23.819 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.819 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.819 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:23.819 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.584703+0000", 2024-09-18T04:38:23.819 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.817e-05 2024-09-18T04:38:23.819 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.819 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.819 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:23.819 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.584725+0000", 2024-09-18T04:38:23.819 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2155e-05 2024-09-18T04:38:23.819 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:23.819 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:23.819 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:23.819 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.820 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.820 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:23.820 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.573382+0000", 2024-09-18T04:38:23.820 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 56.236226397999999, 2024-09-18T04:38:23.820 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012154974000000001, 2024-09-18T04:38:23.820 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:23.820 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:23.820 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:23.820 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:23.820 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:23.820 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 87 2024-09-18T04:38:23.820 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.820 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:23.821 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.821 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:23.821 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573382+0000", 2024-09-18T04:38:23.821 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:23.821 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.821 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.821 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:23.821 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573382+0000", 2024-09-18T04:38:23.821 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:23.822 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.822 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.822 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:23.822 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573382+0000", 2024-09-18T04:38:23.822 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:23.822 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.822 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.822 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:23.822 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573482+0000", 2024-09-18T04:38:23.822 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000100333 2024-09-18T04:38:23.822 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.822 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.822 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:23.822 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573492+0000", 2024-09-18T04:38:23.823 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5389999999999996e-06 2024-09-18T04:38:23.823 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.823 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.823 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:23.823 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573500+0000", 2024-09-18T04:38:23.823 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8929999999999994e-06 2024-09-18T04:38:23.823 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.823 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.823 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:23.823 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.579797+0000", 2024-09-18T04:38:23.823 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0062963239999999998 2024-09-18T04:38:23.823 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.823 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.823 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:23.823 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.579957+0000", 2024-09-18T04:38:23.823 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000160283 2024-09-18T04:38:23.824 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.824 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.824 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:23.824 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.580178+0000", 2024-09-18T04:38:23.824 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00022079200000000001 2024-09-18T04:38:23.824 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.824 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.824 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:23.824 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.583479+0000", 2024-09-18T04:38:23.824 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033007689999999998 2024-09-18T04:38:23.824 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.824 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.824 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:23.824 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585477+0000", 2024-09-18T04:38:23.825 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0019985670000000001 2024-09-18T04:38:23.825 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.825 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.825 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:23.825 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585517+0000", 2024-09-18T04:38:23.825 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9709999999999998e-05 2024-09-18T04:38:23.825 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.825 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.825 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:23.825 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585537+0000", 2024-09-18T04:38:23.825 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0326000000000002e-05 2024-09-18T04:38:23.825 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:23.825 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:23.825 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:23.825 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.826 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.826 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:23.826 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.573508+0000", 2024-09-18T04:38:23.826 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 56.236100692000001, 2024-09-18T04:38:23.826 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012068476, 2024-09-18T04:38:23.826 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:23.826 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:23.826 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:23.826 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:23.826 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:23.826 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 88 2024-09-18T04:38:23.826 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.826 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:23.826 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.827 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:23.827 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573508+0000", 2024-09-18T04:38:23.827 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:23.827 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.827 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.827 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:23.827 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573508+0000", 2024-09-18T04:38:23.827 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:23.827 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.827 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.827 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:23.827 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573507+0000", 2024-09-18T04:38:23.827 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:23.827 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.828 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.828 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:23.828 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573512+0000", 2024-09-18T04:38:23.828 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.2629999999999997e-06 2024-09-18T04:38:23.828 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.828 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.828 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:23.828 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573530+0000", 2024-09-18T04:38:23.828 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8507000000000001e-05 2024-09-18T04:38:23.828 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.828 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.828 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:23.828 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573537+0000", 2024-09-18T04:38:23.828 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.832e-06 2024-09-18T04:38:23.829 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.829 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.829 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:23.829 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.582486+0000", 2024-09-18T04:38:23.829 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0089494770000000008 2024-09-18T04:38:23.829 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.829 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.829 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:23.829 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.582630+0000", 2024-09-18T04:38:23.829 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000144005 2024-09-18T04:38:23.829 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.829 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.829 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:23.830 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.582842+0000", 2024-09-18T04:38:23.830 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00021189200000000001 2024-09-18T04:38:23.830 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.830 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.830 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:23.830 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.584777+0000", 2024-09-18T04:38:23.830 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001934753 2024-09-18T04:38:23.830 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.830 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.830 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:23.830 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585543+0000", 2024-09-18T04:38:23.830 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00076540099999999997 2024-09-18T04:38:23.830 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.830 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.831 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:23.831 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585561+0000", 2024-09-18T04:38:23.831 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8474999999999998e-05 2024-09-18T04:38:23.831 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.831 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.831 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:23.831 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585576+0000", 2024-09-18T04:38:23.831 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5427e-05 2024-09-18T04:38:23.831 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:23.831 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:23.831 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:23.831 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.831 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.831 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:23.832 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786394+0000", 2024-09-18T04:38:23.832 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 56.023214244999998, 2024-09-18T04:38:23.832 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011902085999999999, 2024-09-18T04:38:23.832 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:23.832 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:23.832 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:23.832 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:23.832 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:23.832 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 109 2024-09-18T04:38:23.832 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.832 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:23.832 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.832 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:23.833 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786394+0000", 2024-09-18T04:38:23.833 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:23.833 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.833 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.833 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:23.833 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786394+0000", 2024-09-18T04:38:23.833 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:23.833 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.833 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.834 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:23.834 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786394+0000", 2024-09-18T04:38:23.834 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:23.834 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.834 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.834 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:23.834 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786548+0000", 2024-09-18T04:38:23.834 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015463400000000001 2024-09-18T04:38:23.834 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.834 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.834 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:23.834 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786561+0000", 2024-09-18T04:38:23.834 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3093e-05 2024-09-18T04:38:23.834 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.835 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.835 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:23.835 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786571+0000", 2024-09-18T04:38:23.835 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4700000000000008e-06 2024-09-18T04:38:23.835 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.835 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.835 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:23.835 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.793367+0000", 2024-09-18T04:38:23.835 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0067966600000000004 2024-09-18T04:38:23.835 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.835 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.835 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:23.835 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.793526+0000", 2024-09-18T04:38:23.836 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015877 2024-09-18T04:38:23.836 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.836 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.836 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:23.836 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.793752+0000", 2024-09-18T04:38:23.836 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00022555200000000001 2024-09-18T04:38:23.836 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.836 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.836 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:23.836 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798125+0000", 2024-09-18T04:38:23.836 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0043733349999999999 2024-09-18T04:38:23.836 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.837 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.837 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:23.837 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798146+0000", 2024-09-18T04:38:23.837 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1063000000000001e-05 2024-09-18T04:38:23.837 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.837 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.837 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:23.837 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798232+0000", 2024-09-18T04:38:23.837 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5771000000000003e-05 2024-09-18T04:38:23.837 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.837 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.837 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:23.837 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798297+0000", 2024-09-18T04:38:23.838 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.4624999999999999e-05 2024-09-18T04:38:23.838 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:23.838 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:23.838 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:23.838 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.838 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.838 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:23.838 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786584+0000", 2024-09-18T04:38:23.838 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 56.023024165999999, 2024-09-18T04:38:23.838 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012013037000000001, 2024-09-18T04:38:23.838 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:23.838 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:23.838 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:23.838 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:23.839 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:23.839 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 110 2024-09-18T04:38:23.839 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.839 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:23.839 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.839 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:23.839 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786584+0000", 2024-09-18T04:38:23.839 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:23.839 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.839 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.839 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:23.839 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786584+0000", 2024-09-18T04:38:23.839 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:23.839 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.839 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.840 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:23.840 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786584+0000", 2024-09-18T04:38:23.840 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:23.840 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.840 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.840 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:23.840 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786590+0000", 2024-09-18T04:38:23.840 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6359999999999997e-06 2024-09-18T04:38:23.840 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.840 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.840 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:23.840 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786599+0000", 2024-09-18T04:38:23.840 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7369999999999995e-06 2024-09-18T04:38:23.841 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.841 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.841 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:23.841 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786606+0000", 2024-09-18T04:38:23.841 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1930000000000004e-06 2024-09-18T04:38:23.841 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.841 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.841 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:23.841 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.795018+0000", 2024-09-18T04:38:23.841 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0084115660000000005 2024-09-18T04:38:23.841 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.841 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.841 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:23.841 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.795184+0000", 2024-09-18T04:38:23.841 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000165833 2024-09-18T04:38:23.842 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.842 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.842 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:23.842 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.795414+0000", 2024-09-18T04:38:23.842 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00023088399999999999 2024-09-18T04:38:23.842 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.842 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.842 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:23.842 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798321+0000", 2024-09-18T04:38:23.842 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0029069539999999998 2024-09-18T04:38:23.842 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.842 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.842 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:23.843 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798404+0000", 2024-09-18T04:38:23.843 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2724000000000003e-05 2024-09-18T04:38:23.843 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.843 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.843 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:23.843 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798488+0000", 2024-09-18T04:38:23.843 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.3505999999999996e-05 2024-09-18T04:38:23.843 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.843 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.843 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:23.843 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798598+0000", 2024-09-18T04:38:23.843 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000109917 2024-09-18T04:38:23.843 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:23.843 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:23.844 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:23.844 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.844 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.844 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:23.844 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786613+0000", 2024-09-18T04:38:23.844 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 56.022995465999998, 2024-09-18T04:38:23.844 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014245760999999999, 2024-09-18T04:38:23.844 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:23.844 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:23.844 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:23.844 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:23.844 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:23.844 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 111 2024-09-18T04:38:23.845 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.845 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:23.845 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.845 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:23.845 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786613+0000", 2024-09-18T04:38:23.845 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:23.845 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.845 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.845 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:23.845 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786613+0000", 2024-09-18T04:38:23.845 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:23.845 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.846 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.846 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:23.846 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786612+0000", 2024-09-18T04:38:23.846 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:23.846 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.846 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.846 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:23.846 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786622+0000", 2024-09-18T04:38:23.846 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4099999999999997e-06 2024-09-18T04:38:23.846 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.846 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.846 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:23.846 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786630+0000", 2024-09-18T04:38:23.846 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4169999999999992e-06 2024-09-18T04:38:23.847 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.847 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.847 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:23.847 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786637+0000", 2024-09-18T04:38:23.847 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8290000000000002e-06 2024-09-18T04:38:23.847 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.847 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.847 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:23.847 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.797099+0000", 2024-09-18T04:38:23.847 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010461747 2024-09-18T04:38:23.847 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.847 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.847 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:23.847 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.797243+0000", 2024-09-18T04:38:23.847 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014388899999999999 2024-09-18T04:38:23.847 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.848 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.848 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:23.848 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.797463+0000", 2024-09-18T04:38:23.848 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000219912 2024-09-18T04:38:23.848 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.848 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.848 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:23.848 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800732+0000", 2024-09-18T04:38:23.848 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0032688719999999999 2024-09-18T04:38:23.848 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.848 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.848 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:23.848 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800751+0000", 2024-09-18T04:38:23.848 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9633999999999999e-05 2024-09-18T04:38:23.849 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.849 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.849 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:23.849 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800824+0000", 2024-09-18T04:38:23.849 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.2805999999999994e-05 2024-09-18T04:38:23.849 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.849 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.849 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:23.849 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800859+0000", 2024-09-18T04:38:23.849 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4952e-05 2024-09-18T04:38:23.849 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:23.849 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:23.849 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:23.849 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.849 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.850 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:23.850 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786649+0000", 2024-09-18T04:38:23.850 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 56.022960056999999, 2024-09-18T04:38:23.850 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.01613442, 2024-09-18T04:38:23.850 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:23.850 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:23.850 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:23.850 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:23.850 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:23.850 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 112 2024-09-18T04:38:23.850 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.850 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:23.850 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.851 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:23.851 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786649+0000", 2024-09-18T04:38:23.851 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:23.851 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.851 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.851 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:23.851 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786649+0000", 2024-09-18T04:38:23.851 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:23.851 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.851 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.851 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:23.851 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786648+0000", 2024-09-18T04:38:23.851 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:23.851 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.851 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.852 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:23.852 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786768+0000", 2024-09-18T04:38:23.852 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000120491 2024-09-18T04:38:23.852 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.852 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.852 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:23.852 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786795+0000", 2024-09-18T04:38:23.852 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6556000000000001e-05 2024-09-18T04:38:23.852 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.852 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.852 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:23.852 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786807+0000", 2024-09-18T04:38:23.852 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1739999999999999e-05 2024-09-18T04:38:23.852 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.852 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.853 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:23.853 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798683+0000", 2024-09-18T04:38:23.853 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011876070000000001 2024-09-18T04:38:23.853 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.853 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.853 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:23.853 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798867+0000", 2024-09-18T04:38:23.853 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000184103 2024-09-18T04:38:23.853 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.853 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.853 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:23.853 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.799082+0000", 2024-09-18T04:38:23.853 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00021476000000000001 2024-09-18T04:38:23.854 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.854 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.854 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:23.854 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802610+0000", 2024-09-18T04:38:23.854 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0035282209999999998 2024-09-18T04:38:23.854 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.854 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.854 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:23.854 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802706+0000", 2024-09-18T04:38:23.854 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5922999999999996e-05 2024-09-18T04:38:23.854 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.854 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.854 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:23.854 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802750+0000", 2024-09-18T04:38:23.855 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4511000000000001e-05 2024-09-18T04:38:23.855 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.855 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.855 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:23.855 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802783+0000", 2024-09-18T04:38:23.855 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2752000000000001e-05 2024-09-18T04:38:23.855 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:23.855 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:23.855 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:23.855 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.855 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.855 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:23.855 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786816+0000", 2024-09-18T04:38:23.855 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 56.022792989999999, 2024-09-18T04:38:23.856 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016197989999999999, 2024-09-18T04:38:23.856 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:23.856 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:23.856 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:23.856 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:23.856 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:23.856 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 113 2024-09-18T04:38:23.856 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.856 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:23.856 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.856 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:23.856 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786816+0000", 2024-09-18T04:38:23.857 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:23.857 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.857 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.857 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:23.857 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786816+0000", 2024-09-18T04:38:23.857 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:23.857 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.857 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.857 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:23.857 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786815+0000", 2024-09-18T04:38:23.857 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:23.857 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.857 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.858 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:23.858 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786821+0000", 2024-09-18T04:38:23.858 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9529999999999996e-06 2024-09-18T04:38:23.858 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.858 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.858 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:23.858 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786841+0000", 2024-09-18T04:38:23.858 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9743000000000001e-05 2024-09-18T04:38:23.858 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.858 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.858 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:23.858 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786848+0000", 2024-09-18T04:38:23.858 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4900000000000003e-06 2024-09-18T04:38:23.858 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.859 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.859 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:23.859 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.799747+0000", 2024-09-18T04:38:23.859 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012898902 2024-09-18T04:38:23.859 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.859 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.859 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:23.859 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.799898+0000", 2024-09-18T04:38:23.859 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000150877 2024-09-18T04:38:23.859 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.859 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.859 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:23.859 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800095+0000", 2024-09-18T04:38:23.859 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000197023 2024-09-18T04:38:23.860 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.860 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.860 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:23.860 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802829+0000", 2024-09-18T04:38:23.860 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0027343480000000002 2024-09-18T04:38:23.860 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.860 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.860 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:23.860 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802847+0000", 2024-09-18T04:38:23.860 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7283e-05 2024-09-18T04:38:23.860 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.860 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.860 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:23.860 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802974+0000", 2024-09-18T04:38:23.861 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012749199999999999 2024-09-18T04:38:23.861 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.861 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.861 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:23.861 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803014+0000", 2024-09-18T04:38:23.861 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9576000000000002e-05 2024-09-18T04:38:23.861 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:23.861 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:23.861 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:23.861 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.861 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.861 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:23.861 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:38:23.862 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 56.022707398999998, 2024-09-18T04:38:23.862 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017399218000000001, 2024-09-18T04:38:23.862 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:23.862 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:23.862 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:23.862 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:23.862 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:23.862 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 114 2024-09-18T04:38:23.862 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.862 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:23.862 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.862 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:23.862 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:38:23.862 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:23.863 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.863 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.863 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:23.863 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:38:23.863 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:23.863 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.863 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.863 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:23.863 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786900+0000", 2024-09-18T04:38:23.863 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:23.863 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.863 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.863 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:23.863 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786908+0000", 2024-09-18T04:38:23.864 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8199999999999997e-06 2024-09-18T04:38:23.864 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.864 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.864 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:23.864 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786925+0000", 2024-09-18T04:38:23.864 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6849000000000001e-05 2024-09-18T04:38:23.864 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.864 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.864 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:23.864 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786934+0000", 2024-09-18T04:38:23.864 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6299999999999993e-06 2024-09-18T04:38:23.864 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.864 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.865 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:23.865 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800926+0000", 2024-09-18T04:38:23.865 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013991632 2024-09-18T04:38:23.865 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.865 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.865 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:23.865 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801060+0000", 2024-09-18T04:38:23.865 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013378199999999999 2024-09-18T04:38:23.865 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.865 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.865 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:23.865 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801166+0000", 2024-09-18T04:38:23.865 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010607099999999999 2024-09-18T04:38:23.865 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.866 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.866 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:23.866 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803026+0000", 2024-09-18T04:38:23.866 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001860469 2024-09-18T04:38:23.866 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.866 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.866 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:23.866 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804245+0000", 2024-09-18T04:38:23.866 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0012185900000000001 2024-09-18T04:38:23.866 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.866 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.866 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:23.866 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804278+0000", 2024-09-18T04:38:23.866 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3272000000000003e-05 2024-09-18T04:38:23.867 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.867 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.867 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:23.867 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804300+0000", 2024-09-18T04:38:23.867 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2266e-05 2024-09-18T04:38:23.867 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:23.867 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:23.867 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:23.867 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.867 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.867 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:23.867 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787566+0000", 2024-09-18T04:38:23.867 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 56.02204244, 2024-09-18T04:38:23.867 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017124120999999999, 2024-09-18T04:38:23.868 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:23.868 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:23.868 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:23.868 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:23.868 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:23.868 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 115 2024-09-18T04:38:23.868 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.868 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:23.868 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.868 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:23.868 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787566+0000", 2024-09-18T04:38:23.868 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:23.868 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.868 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.869 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:23.869 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787566+0000", 2024-09-18T04:38:23.869 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:23.869 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.869 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.869 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:23.869 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787564+0000", 2024-09-18T04:38:23.869 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:23.869 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.869 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.870 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:23.870 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787751+0000", 2024-09-18T04:38:23.870 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000186873 2024-09-18T04:38:23.870 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.870 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.870 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:23.870 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787772+0000", 2024-09-18T04:38:23.870 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.126e-05 2024-09-18T04:38:23.870 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.870 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.870 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:23.870 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787794+0000", 2024-09-18T04:38:23.870 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1418999999999999e-05 2024-09-18T04:38:23.870 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.871 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.871 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:23.871 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801659+0000", 2024-09-18T04:38:23.871 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013865125000000001 2024-09-18T04:38:23.871 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.871 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.871 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:23.871 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801750+0000", 2024-09-18T04:38:23.871 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.1427000000000004e-05 2024-09-18T04:38:23.871 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.871 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.871 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:23.871 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801894+0000", 2024-09-18T04:38:23.871 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000143514 2024-09-18T04:38:23.871 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.872 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.872 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:23.872 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804313+0000", 2024-09-18T04:38:23.872 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0024190879999999998 2024-09-18T04:38:23.872 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.872 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.872 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:23.872 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804644+0000", 2024-09-18T04:38:23.872 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00033100799999999999 2024-09-18T04:38:23.872 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.872 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.872 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:23.872 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804672+0000", 2024-09-18T04:38:23.873 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7988e-05 2024-09-18T04:38:23.873 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.873 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.873 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:23.873 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804690+0000", 2024-09-18T04:38:23.873 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8298999999999999e-05 2024-09-18T04:38:23.873 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:23.873 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:23.873 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:23.873 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.873 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.873 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:23.873 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:38:23.873 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 56.021794501000002, 2024-09-18T04:38:23.874 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017521662, 2024-09-18T04:38:23.874 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:23.874 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:23.874 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:23.874 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:23.874 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:23.874 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 116 2024-09-18T04:38:23.874 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.874 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:23.874 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.874 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:23.874 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:38:23.874 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:23.874 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.875 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.875 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:23.875 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:38:23.875 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:23.875 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.875 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.875 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:23.875 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787813+0000", 2024-09-18T04:38:23.875 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:23.875 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.875 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.875 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:23.875 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787824+0000", 2024-09-18T04:38:23.875 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1157e-05 2024-09-18T04:38:23.876 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.876 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.876 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:23.876 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787840+0000", 2024-09-18T04:38:23.876 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6269e-05 2024-09-18T04:38:23.876 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.876 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.876 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:23.876 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787853+0000", 2024-09-18T04:38:23.876 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3156000000000001e-05 2024-09-18T04:38:23.876 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.876 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.876 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:23.876 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803110+0000", 2024-09-18T04:38:23.876 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015256736 2024-09-18T04:38:23.877 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.877 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.877 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:23.877 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803197+0000", 2024-09-18T04:38:23.877 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7195999999999997e-05 2024-09-18T04:38:23.877 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.877 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.877 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:23.877 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803308+0000", 2024-09-18T04:38:23.877 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111005 2024-09-18T04:38:23.877 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.877 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.877 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:23.878 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804316+0000", 2024-09-18T04:38:23.878 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010073949999999999 2024-09-18T04:38:23.878 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.878 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.878 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:23.878 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805232+0000", 2024-09-18T04:38:23.878 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091636300000000001 2024-09-18T04:38:23.878 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.878 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.878 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:23.879 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805277+0000", 2024-09-18T04:38:23.879 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4864000000000003e-05 2024-09-18T04:38:23.879 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.879 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.879 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:23.879 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805336+0000", 2024-09-18T04:38:23.879 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8968000000000002e-05 2024-09-18T04:38:23.879 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:23.879 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:23.879 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:23.879 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.879 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.879 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:23.879 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:38:23.880 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 56.021742445999998, 2024-09-18T04:38:23.880 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017983453, 2024-09-18T04:38:23.880 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:23.880 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:23.880 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:23.880 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:23.880 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:23.880 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 117 2024-09-18T04:38:23.880 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.880 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:23.880 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.880 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:23.880 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:38:23.880 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:23.881 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.881 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.881 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:23.881 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:38:23.881 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:23.881 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.881 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.881 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:23.881 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787865+0000", 2024-09-18T04:38:23.881 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:23.881 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.881 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.881 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:23.881 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787889+0000", 2024-09-18T04:38:23.882 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4244999999999999e-05 2024-09-18T04:38:23.882 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.882 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.882 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:23.882 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787905+0000", 2024-09-18T04:38:23.882 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5739999999999998e-05 2024-09-18T04:38:23.882 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.882 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.882 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:23.882 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787918+0000", 2024-09-18T04:38:23.882 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.252e-05 2024-09-18T04:38:23.882 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.882 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.882 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:23.883 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803693+0000", 2024-09-18T04:38:23.883 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015774872999999998 2024-09-18T04:38:23.883 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.883 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.883 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:23.883 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803775+0000", 2024-09-18T04:38:23.883 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2761000000000003e-05 2024-09-18T04:38:23.883 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.883 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.883 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:23.883 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803887+0000", 2024-09-18T04:38:23.883 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111708 2024-09-18T04:38:23.883 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.883 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.883 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:23.883 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804823+0000", 2024-09-18T04:38:23.884 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00093625100000000003 2024-09-18T04:38:23.884 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.884 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.884 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:23.884 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805804+0000", 2024-09-18T04:38:23.884 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00098114699999999992 2024-09-18T04:38:23.884 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.884 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.884 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:23.884 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805831+0000", 2024-09-18T04:38:23.884 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6401999999999999e-05 2024-09-18T04:38:23.884 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.884 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.884 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:23.885 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805850+0000", 2024-09-18T04:38:23.885 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8862e-05 2024-09-18T04:38:23.885 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:23.885 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:23.885 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:23.885 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.885 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.885 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:23.885 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.230830+0000", 2024-09-18T04:38:23.885 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 55.578778567000001, 2024-09-18T04:38:23.885 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011537146999999999, 2024-09-18T04:38:23.885 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:23.885 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:23.886 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:23.886 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:23.886 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:23.886 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 135 2024-09-18T04:38:23.886 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.886 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:23.886 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.886 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:23.886 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230830+0000", 2024-09-18T04:38:23.886 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:23.886 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.886 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.886 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:23.886 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230830+0000", 2024-09-18T04:38:23.886 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:23.887 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.887 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.887 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:23.887 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230829+0000", 2024-09-18T04:38:23.887 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:23.887 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.887 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.887 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:23.887 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230835+0000", 2024-09-18T04:38:23.887 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8660000000000002e-06 2024-09-18T04:38:23.887 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.887 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.887 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:23.888 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230844+0000", 2024-09-18T04:38:23.888 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9339999999999995e-06 2024-09-18T04:38:23.888 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.888 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.888 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:23.888 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230853+0000", 2024-09-18T04:38:23.888 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4419999999999998e-06 2024-09-18T04:38:23.888 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.888 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.888 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:23.888 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.237658+0000", 2024-09-18T04:38:23.888 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0068055310000000001 2024-09-18T04:38:23.888 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.888 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.889 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:23.889 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.237837+0000", 2024-09-18T04:38:23.889 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017943200000000001 2024-09-18T04:38:23.889 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.889 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.889 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:23.889 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.238112+0000", 2024-09-18T04:38:23.889 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00027488600000000001 2024-09-18T04:38:23.889 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.889 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.889 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:23.889 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.240505+0000", 2024-09-18T04:38:23.889 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0023930539999999999 2024-09-18T04:38:23.890 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.890 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.890 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:23.890 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242157+0000", 2024-09-18T04:38:23.890 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001651135 2024-09-18T04:38:23.890 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.890 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.890 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:23.890 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242281+0000", 2024-09-18T04:38:23.890 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012451300000000001 2024-09-18T04:38:23.890 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.890 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.890 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:23.890 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242367+0000", 2024-09-18T04:38:23.891 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6178000000000001e-05 2024-09-18T04:38:23.891 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:23.891 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:23.891 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:23.891 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.891 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.891 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:23.891 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.230858+0000", 2024-09-18T04:38:23.891 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 55.578750679000002, 2024-09-18T04:38:23.891 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013005821000000001, 2024-09-18T04:38:23.891 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:23.891 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:23.891 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:23.891 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:23.891 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:23.892 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 136 2024-09-18T04:38:23.892 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.892 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:23.892 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.892 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:23.892 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230858+0000", 2024-09-18T04:38:23.892 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:23.892 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.892 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.892 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:23.892 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230858+0000", 2024-09-18T04:38:23.892 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:23.892 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.893 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.893 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:23.893 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230858+0000", 2024-09-18T04:38:23.893 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:23.893 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.893 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.893 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:23.893 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230861+0000", 2024-09-18T04:38:23.893 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7459999999999998e-06 2024-09-18T04:38:23.893 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.893 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.893 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:23.893 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230877+0000", 2024-09-18T04:38:23.893 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5979000000000001e-05 2024-09-18T04:38:23.894 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.894 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.894 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:23.894 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230885+0000", 2024-09-18T04:38:23.894 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.5979999999999999e-06 2024-09-18T04:38:23.894 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.894 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.894 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:23.894 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.239023+0000", 2024-09-18T04:38:23.894 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0081380710000000002 2024-09-18T04:38:23.894 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.894 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.894 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:23.895 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.239216+0000", 2024-09-18T04:38:23.895 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019274799999999999 2024-09-18T04:38:23.895 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.895 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.895 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:23.895 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.239417+0000", 2024-09-18T04:38:23.895 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020093300000000001 2024-09-18T04:38:23.895 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.895 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.895 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:23.895 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242494+0000", 2024-09-18T04:38:23.895 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003077177 2024-09-18T04:38:23.895 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.895 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.895 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:23.896 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.243668+0000", 2024-09-18T04:38:23.896 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001174027 2024-09-18T04:38:23.896 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.896 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.896 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:23.896 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.243777+0000", 2024-09-18T04:38:23.896 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010955600000000001 2024-09-18T04:38:23.896 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.896 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.896 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:23.896 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.243864+0000", 2024-09-18T04:38:23.896 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6441999999999997e-05 2024-09-18T04:38:23.896 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:23.896 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:23.897 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:23.897 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.897 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.897 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:23.897 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.230901+0000", 2024-09-18T04:38:23.897 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 55.578708106000001, 2024-09-18T04:38:23.897 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014120882, 2024-09-18T04:38:23.897 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:23.897 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:23.897 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:23.897 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:23.897 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:23.897 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 137 2024-09-18T04:38:23.897 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.898 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:23.898 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.898 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:23.898 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230901+0000", 2024-09-18T04:38:23.898 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:23.898 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.898 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.898 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:23.898 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230901+0000", 2024-09-18T04:38:23.898 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:23.898 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.898 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.898 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:23.899 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230900+0000", 2024-09-18T04:38:23.899 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:23.899 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.899 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.899 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:23.899 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231002+0000", 2024-09-18T04:38:23.899 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000102392 2024-09-18T04:38:23.899 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.899 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.899 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:23.899 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231029+0000", 2024-09-18T04:38:23.899 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7166999999999999e-05 2024-09-18T04:38:23.899 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.899 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.899 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:23.900 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231036+0000", 2024-09-18T04:38:23.900 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7259999999999996e-06 2024-09-18T04:38:23.900 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.900 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.900 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:23.900 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.240562+0000", 2024-09-18T04:38:23.900 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0095257970000000008 2024-09-18T04:38:23.900 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.900 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.900 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:23.900 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.240730+0000", 2024-09-18T04:38:23.900 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016767099999999999 2024-09-18T04:38:23.900 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.901 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.901 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:23.901 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.240927+0000", 2024-09-18T04:38:23.901 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019711899999999999 2024-09-18T04:38:23.901 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.901 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.901 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:23.901 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244885+0000", 2024-09-18T04:38:23.901 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0039577340000000001 2024-09-18T04:38:23.901 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.901 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.901 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:23.901 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244899+0000", 2024-09-18T04:38:23.901 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4403000000000001e-05 2024-09-18T04:38:23.901 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.902 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.902 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:23.902 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244973+0000", 2024-09-18T04:38:23.902 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4177999999999994e-05 2024-09-18T04:38:23.902 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.902 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.902 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:23.902 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.245021+0000", 2024-09-18T04:38:23.902 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8344999999999999e-05 2024-09-18T04:38:23.902 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:23.902 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:23.902 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:23.902 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.902 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.903 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:23.903 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:38:23.903 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 55.578565314999999, 2024-09-18T04:38:23.903 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016528505999999998, 2024-09-18T04:38:23.903 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:23.903 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:23.903 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:23.903 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:23.903 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:23.903 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 138 2024-09-18T04:38:23.903 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.903 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:23.903 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.903 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:23.904 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:38:23.904 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:23.904 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.904 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.904 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:23.904 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:38:23.904 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:23.904 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.904 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.904 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:23.904 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:38:23.904 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:23.904 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.904 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.904 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:23.905 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231047+0000", 2024-09-18T04:38:23.905 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5399999999999997e-06 2024-09-18T04:38:23.905 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.905 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.905 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:23.905 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231068+0000", 2024-09-18T04:38:23.905 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0769000000000001e-05 2024-09-18T04:38:23.905 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.905 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.905 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:23.905 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231074+0000", 2024-09-18T04:38:23.905 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6180000000000003e-06 2024-09-18T04:38:23.905 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.906 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.906 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:23.906 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242556+0000", 2024-09-18T04:38:23.906 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011482654 2024-09-18T04:38:23.906 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.906 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.906 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:23.906 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242666+0000", 2024-09-18T04:38:23.906 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011003200000000001 2024-09-18T04:38:23.906 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.906 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.906 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:23.906 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242924+0000", 2024-09-18T04:38:23.906 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00025785899999999999 2024-09-18T04:38:23.906 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.907 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.907 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:23.907 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247263+0000", 2024-09-18T04:38:23.907 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0043387649999999996 2024-09-18T04:38:23.907 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.907 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.907 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:23.907 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247412+0000", 2024-09-18T04:38:23.907 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000149276 2024-09-18T04:38:23.907 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.907 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.907 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:23.907 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247522+0000", 2024-09-18T04:38:23.907 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000109788 2024-09-18T04:38:23.907 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.908 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.908 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:23.908 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247572+0000", 2024-09-18T04:38:23.908 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9808999999999997e-05 2024-09-18T04:38:23.908 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:23.908 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:23.908 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:23.908 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.908 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.908 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:23.908 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231083+0000", 2024-09-18T04:38:23.908 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 55.578525851999999, 2024-09-18T04:38:23.908 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0166731, 2024-09-18T04:38:23.908 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:23.909 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:23.909 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:23.909 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:23.909 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:23.909 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 139 2024-09-18T04:38:23.909 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.909 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:23.909 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.909 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:23.909 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231083+0000", 2024-09-18T04:38:23.909 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:23.909 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.909 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.909 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:23.910 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231083+0000", 2024-09-18T04:38:23.910 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:23.910 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.910 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.910 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:23.910 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231082+0000", 2024-09-18T04:38:23.910 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:23.910 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.910 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.910 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:23.910 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231090+0000", 2024-09-18T04:38:23.910 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1280000000000008e-06 2024-09-18T04:38:23.910 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.910 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.911 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:23.911 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231097+0000", 2024-09-18T04:38:23.911 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5119999999999997e-06 2024-09-18T04:38:23.911 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.911 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.911 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:23.911 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231106+0000", 2024-09-18T04:38:23.911 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9230000000000008e-06 2024-09-18T04:38:23.911 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.911 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.911 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:23.911 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.243977+0000", 2024-09-18T04:38:23.911 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012870754 2024-09-18T04:38:23.911 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.912 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.912 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:23.912 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244112+0000", 2024-09-18T04:38:23.912 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000135575 2024-09-18T04:38:23.912 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.912 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.912 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:23.912 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244265+0000", 2024-09-18T04:38:23.912 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015279900000000001 2024-09-18T04:38:23.912 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.912 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.912 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:23.912 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247649+0000", 2024-09-18T04:38:23.912 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003383896 2024-09-18T04:38:23.913 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.913 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.913 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:23.913 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247656+0000", 2024-09-18T04:38:23.913 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8920000000000003e-06 2024-09-18T04:38:23.913 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.913 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.913 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:23.913 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247692+0000", 2024-09-18T04:38:23.913 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6470000000000001e-05 2024-09-18T04:38:23.913 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.913 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.913 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:23.913 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247756+0000", 2024-09-18T04:38:23.913 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3676999999999994e-05 2024-09-18T04:38:23.914 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:23.914 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:23.914 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:23.914 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.914 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.914 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:23.914 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231190+0000", 2024-09-18T04:38:23.914 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 55.578418925999998, 2024-09-18T04:38:23.914 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016793556000000001, 2024-09-18T04:38:23.914 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:23.914 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:23.914 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:23.914 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:23.915 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:23.915 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 140 2024-09-18T04:38:23.915 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.915 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:23.915 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.915 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:23.915 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231190+0000", 2024-09-18T04:38:23.915 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:23.915 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.915 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.915 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:23.915 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231190+0000", 2024-09-18T04:38:23.915 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:23.915 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.915 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.916 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:23.916 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231189+0000", 2024-09-18T04:38:23.916 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:23.916 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.916 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.916 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:23.916 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231273+0000", 2024-09-18T04:38:23.916 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4445999999999998e-05 2024-09-18T04:38:23.916 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.916 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.916 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:23.916 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231282+0000", 2024-09-18T04:38:23.916 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8859999999999993e-06 2024-09-18T04:38:23.917 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.917 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.917 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:23.917 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231288+0000", 2024-09-18T04:38:23.917 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9819999999999996e-06 2024-09-18T04:38:23.917 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.917 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.917 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:23.917 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.245119+0000", 2024-09-18T04:38:23.917 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.01383121 2024-09-18T04:38:23.917 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.917 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.917 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:23.917 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.245247+0000", 2024-09-18T04:38:23.918 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012763299999999999 2024-09-18T04:38:23.918 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.918 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.918 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:23.918 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.245401+0000", 2024-09-18T04:38:23.918 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015399200000000001 2024-09-18T04:38:23.918 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.918 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.918 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:23.918 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247766+0000", 2024-09-18T04:38:23.918 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0023649510000000001 2024-09-18T04:38:23.918 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.918 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.918 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:23.919 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247904+0000", 2024-09-18T04:38:23.919 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013836499999999999 2024-09-18T04:38:23.919 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.919 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.919 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:23.919 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247950+0000", 2024-09-18T04:38:23.919 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5201000000000003e-05 2024-09-18T04:38:23.919 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.919 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.919 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:23.919 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247983+0000", 2024-09-18T04:38:23.919 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3686000000000002e-05 2024-09-18T04:38:23.919 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:23.919 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:23.920 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:23.920 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.920 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.920 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:23.920 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:38:23.920 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 55.578308182999997, 2024-09-18T04:38:23.920 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017344024, 2024-09-18T04:38:23.920 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:23.920 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:23.920 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:23.920 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:23.920 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:23.920 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 141 2024-09-18T04:38:23.920 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.920 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:23.921 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.921 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:23.921 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:38:23.921 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:23.921 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.921 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.921 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:23.921 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:38:23.921 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:23.922 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.922 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.922 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:23.922 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:38:23.922 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:23.922 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.922 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.922 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:23.922 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231309+0000", 2024-09-18T04:38:23.922 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5419999999999994e-06 2024-09-18T04:38:23.922 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.922 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.922 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:23.922 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231323+0000", 2024-09-18T04:38:23.923 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3108000000000001e-05 2024-09-18T04:38:23.923 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.923 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.923 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:23.923 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231340+0000", 2024-09-18T04:38:23.923 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.696e-05 2024-09-18T04:38:23.923 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.923 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.923 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:23.923 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.246143+0000", 2024-09-18T04:38:23.923 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014803320999999999 2024-09-18T04:38:23.923 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.923 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.923 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:23.924 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.246323+0000", 2024-09-18T04:38:23.924 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00018061600000000001 2024-09-18T04:38:23.924 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.924 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.924 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:23.924 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.246530+0000", 2024-09-18T04:38:23.924 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020625 2024-09-18T04:38:23.924 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.924 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.924 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:23.924 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248002+0000", 2024-09-18T04:38:23.924 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0014719379999999999 2024-09-18T04:38:23.924 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.924 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.924 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:23.925 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248582+0000", 2024-09-18T04:38:23.925 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00058020599999999999 2024-09-18T04:38:23.925 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.925 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.925 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:23.925 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248618+0000", 2024-09-18T04:38:23.925 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6559e-05 2024-09-18T04:38:23.925 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.925 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.925 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:23.925 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248645+0000", 2024-09-18T04:38:23.925 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6080000000000001e-05 2024-09-18T04:38:23.925 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:23.925 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:23.926 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:23.926 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.926 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.926 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-18T04:38:23.926 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.878463+0000", 2024-09-18T04:38:23.926 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 54.931146149, 2024-09-18T04:38:23.926 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010882697, 2024-09-18T04:38:23.926 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:23.926 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:23.926 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:23.926 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:23.926 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:23.926 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 159 2024-09-18T04:38:23.927 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.927 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:23.927 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.927 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:23.927 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878463+0000", 2024-09-18T04:38:23.927 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:23.927 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.927 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.927 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:23.927 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878463+0000", 2024-09-18T04:38:23.927 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:23.927 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.927 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.927 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:23.927 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878462+0000", 2024-09-18T04:38:23.928 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:23.928 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.928 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.928 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:23.928 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878467+0000", 2024-09-18T04:38:23.928 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4099999999999999e-06 2024-09-18T04:38:23.928 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.928 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.928 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:23.928 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878490+0000", 2024-09-18T04:38:23.928 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3159999999999998e-05 2024-09-18T04:38:23.928 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.928 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.928 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:23.929 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878511+0000", 2024-09-18T04:38:23.929 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1092000000000001e-05 2024-09-18T04:38:23.929 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.929 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.929 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:23.929 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.887163+0000", 2024-09-18T04:38:23.929 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.008651845 2024-09-18T04:38:23.929 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.929 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.929 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:23.929 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.887273+0000", 2024-09-18T04:38:23.929 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000109823 2024-09-18T04:38:23.929 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.929 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.930 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:23.930 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.887441+0000", 2024-09-18T04:38:23.930 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016777599999999999 2024-09-18T04:38:23.930 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.930 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.930 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:23.930 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.888634+0000", 2024-09-18T04:38:23.930 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001192839 2024-09-18T04:38:23.930 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.930 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.930 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:23.930 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.889153+0000", 2024-09-18T04:38:23.930 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00051953700000000004 2024-09-18T04:38:23.930 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.930 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.931 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:23.931 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.889234+0000", 2024-09-18T04:38:23.931 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0802000000000003e-05 2024-09-18T04:38:23.931 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:23.931 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:23.931 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:23.931 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.889345+0000", 2024-09-18T04:38:23.931 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011112000000000001 2024-09-18T04:38:23.931 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:23.931 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:23.931 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:23.931 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:23.931 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:23.931 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:23.933 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-09-18T04:38:23.934 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:38:23.934 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:23.926+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:24.034 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:24.030+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:24.091 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:24.092 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:24.092 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:38:24.092 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:24.101 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:38:24.135 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:24.131+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:24.210 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:24.210 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:24.210 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:38:24.210 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:38:24.210 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:24.220 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:38:24.236 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:24.231+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:24.329 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:24.329 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:38:24.329 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:38:24.329 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [] 2024-09-18T04:38:24.329 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:24.337 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:24.330+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:24.339 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:38:24.438 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:24.434+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:24.442 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:24.442 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:24.442 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:38:24.442 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:24.452 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:38:24.540 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:24.534+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:24.554 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:24.554 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:24.554 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:38:24.555 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:38:24.555 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:24.564 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:38:24.641 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:24.634+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:24.675 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:24.676 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:38:24.676 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:38:24.676 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:38:24.676 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.676 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-18T04:38:24.676 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:38:24.676 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 50.202368086, 2024-09-18T04:38:24.676 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0095781189999999995, 2024-09-18T04:38:24.676 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:24.676 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:24.676 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:24.676 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:38:24.677 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:38:24.677 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 1 2024-09-18T04:38:24.677 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.677 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:24.677 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.677 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:24.677 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:38:24.677 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:24.677 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.677 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.677 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:24.677 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:38:24.677 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:24.677 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.678 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.678 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:24.678 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469862+0000", 2024-09-18T04:38:24.678 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:24.678 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.678 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.678 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:24.678 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469879+0000", 2024-09-18T04:38:24.678 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7431999999999998e-05 2024-09-18T04:38:24.678 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.678 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.678 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:24.678 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469896+0000", 2024-09-18T04:38:24.679 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6742999999999999e-05 2024-09-18T04:38:24.679 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.679 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.679 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:24.679 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469915+0000", 2024-09-18T04:38:24.679 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8991000000000002e-05 2024-09-18T04:38:24.679 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.679 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.679 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:24.679 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470397+0000", 2024-09-18T04:38:24.679 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048167799999999998 2024-09-18T04:38:24.679 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.679 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.679 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-09-18T04:38:24.680 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470412+0000", 2024-09-18T04:38:24.680 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5526e-05 2024-09-18T04:38:24.680 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.680 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.680 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:24.680 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473279+0000", 2024-09-18T04:38:24.680 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0028667469999999998 2024-09-18T04:38:24.680 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.680 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.680 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:24.680 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473441+0000", 2024-09-18T04:38:24.681 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000162082 2024-09-18T04:38:24.681 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.681 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.681 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:38:24.681 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473573+0000", 2024-09-18T04:38:24.681 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000132032 2024-09-18T04:38:24.681 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.681 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.681 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:24.681 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474826+0000", 2024-09-18T04:38:24.681 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001252717 2024-09-18T04:38:24.681 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.681 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.682 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:24.682 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479307+0000", 2024-09-18T04:38:24.682 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044813220000000003 2024-09-18T04:38:24.682 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.682 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.682 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:24.682 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479366+0000", 2024-09-18T04:38:24.682 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9024999999999998e-05 2024-09-18T04:38:24.682 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.682 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.682 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:24.682 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479443+0000", 2024-09-18T04:38:24.682 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6339000000000002e-05 2024-09-18T04:38:24.682 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:24.683 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:24.683 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:24.683 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.683 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.683 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.4214.0:1 2.3 e18/17 ondisk, result = 0)", 2024-09-18T04:38:24.683 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.474724+0000", 2024-09-18T04:38:24.683 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 50.197508722000002, 2024-09-18T04:38:24.683 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000105358, 2024-09-18T04:38:24.683 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:24.683 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:38:24.683 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:24.683 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.683 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:24.683 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474724+0000", 2024-09-18T04:38:24.684 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:24.684 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.684 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.684 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:24.684 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474724+0000", 2024-09-18T04:38:24.684 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:24.684 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.684 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.684 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:24.684 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474723+0000", 2024-09-18T04:38:24.684 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:24.684 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.684 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.684 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:24.685 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474731+0000", 2024-09-18T04:38:24.685 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2339999999999994e-06 2024-09-18T04:38:24.685 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.685 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.685 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:24.685 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474748+0000", 2024-09-18T04:38:24.685 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6603e-05 2024-09-18T04:38:24.685 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.685 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.685 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:24.685 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474760+0000", 2024-09-18T04:38:24.685 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2215000000000001e-05 2024-09-18T04:38:24.685 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.686 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.686 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:24.686 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474814+0000", 2024-09-18T04:38:24.686 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4267000000000001e-05 2024-09-18T04:38:24.686 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.686 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.686 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:24.686 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474821+0000", 2024-09-18T04:38:24.686 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0929999999999997e-06 2024-09-18T04:38:24.686 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.686 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.686 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:24.686 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474829+0000", 2024-09-18T04:38:24.687 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1200000000000002e-06 2024-09-18T04:38:24.687 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:24.687 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:24.687 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:24.687 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.687 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.687 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18)", 2024-09-18T04:38:24.687 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.482346+0000", 2024-09-18T04:38:24.687 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 50.189886289999997, 2024-09-18T04:38:24.687 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020435599999999999, 2024-09-18T04:38:24.687 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:24.687 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-09-18T04:38:24.687 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:24.687 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:38:24.688 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:38:24.688 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 2 2024-09-18T04:38:24.688 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.688 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:24.688 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.688 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:24.688 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482346+0000", 2024-09-18T04:38:24.688 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:24.688 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.688 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.688 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:24.688 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482346+0000", 2024-09-18T04:38:24.689 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:24.689 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.689 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.689 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:24.689 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482345+0000", 2024-09-18T04:38:24.689 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:24.689 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.689 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.689 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:24.689 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482360+0000", 2024-09-18T04:38:24.689 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4854999999999999e-05 2024-09-18T04:38:24.689 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.689 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.690 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:24.690 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482377+0000", 2024-09-18T04:38:24.690 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7785999999999999e-05 2024-09-18T04:38:24.690 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.690 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.690 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:24.690 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482387+0000", 2024-09-18T04:38:24.690 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6800000000000005e-06 2024-09-18T04:38:24.690 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.690 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.690 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:24.690 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482441+0000", 2024-09-18T04:38:24.690 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4046999999999999e-05 2024-09-18T04:38:24.690 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.691 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.691 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:24.691 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482551+0000", 2024-09-18T04:38:24.691 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000109468 2024-09-18T04:38:24.691 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:24.691 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:24.691 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:24.691 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.691 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.691 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18)", 2024-09-18T04:38:24.691 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.482853+0000", 2024-09-18T04:38:24.691 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 50.189379672000001, 2024-09-18T04:38:24.691 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00029815799999999997, 2024-09-18T04:38:24.692 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:24.692 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-09-18T04:38:24.692 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:24.692 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:38:24.692 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:38:24.692 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 3 2024-09-18T04:38:24.692 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.692 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:24.692 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.692 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:24.692 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482853+0000", 2024-09-18T04:38:24.692 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:24.692 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.692 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.693 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:24.693 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482853+0000", 2024-09-18T04:38:24.693 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:24.693 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.693 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.693 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:24.693 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482852+0000", 2024-09-18T04:38:24.693 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:24.693 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.693 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.693 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:24.693 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482859+0000", 2024-09-18T04:38:24.693 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0380000000000002e-06 2024-09-18T04:38:24.694 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.694 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.694 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:24.694 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482880+0000", 2024-09-18T04:38:24.694 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1243e-05 2024-09-18T04:38:24.694 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.694 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.694 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:24.694 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482896+0000", 2024-09-18T04:38:24.694 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5977e-05 2024-09-18T04:38:24.694 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.694 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.694 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:24.695 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482970+0000", 2024-09-18T04:38:24.695 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4603000000000004e-05 2024-09-18T04:38:24.695 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.695 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.695 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:24.695 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.483151+0000", 2024-09-18T04:38:24.695 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00018059799999999999 2024-09-18T04:38:24.695 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:24.695 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:24.695 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:24.696 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.696 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.696 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-18T04:38:24.696 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:35.468698+0000", 2024-09-18T04:38:24.696 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 49.203534834999999, 2024-09-18T04:38:24.696 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0039954129999999997, 2024-09-18T04:38:24.696 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:24.696 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:24.696 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:24.696 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:38:24.696 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:38:24.696 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 5 2024-09-18T04:38:24.697 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.697 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:24.697 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.697 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:24.697 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468698+0000", 2024-09-18T04:38:24.697 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:24.697 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.697 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.697 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:24.697 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468698+0000", 2024-09-18T04:38:24.697 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:24.697 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.697 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.698 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:24.698 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468696+0000", 2024-09-18T04:38:24.698 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:24.698 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.698 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.698 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:24.698 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468714+0000", 2024-09-18T04:38:24.698 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8743e-05 2024-09-18T04:38:24.698 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.698 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.698 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:24.698 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468748+0000", 2024-09-18T04:38:24.698 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3339e-05 2024-09-18T04:38:24.698 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.699 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.699 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:24.699 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468771+0000", 2024-09-18T04:38:24.699 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3119999999999999e-05 2024-09-18T04:38:24.699 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.699 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.699 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:24.699 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468922+0000", 2024-09-18T04:38:24.699 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000151383 2024-09-18T04:38:24.699 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.699 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.699 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:24.699 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.469278+0000", 2024-09-18T04:38:24.700 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00035628599999999998 2024-09-18T04:38:24.700 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.700 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.700 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:24.700 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.469629+0000", 2024-09-18T04:38:24.700 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00035014300000000003 2024-09-18T04:38:24.700 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.700 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.700 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:24.700 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.471620+0000", 2024-09-18T04:38:24.700 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001991278 2024-09-18T04:38:24.700 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.700 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.701 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:24.701 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472440+0000", 2024-09-18T04:38:24.701 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00081998599999999998 2024-09-18T04:38:24.701 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.701 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.701 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:24.701 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472592+0000", 2024-09-18T04:38:24.701 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015227400000000001 2024-09-18T04:38:24.701 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.701 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.701 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:24.701 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472693+0000", 2024-09-18T04:38:24.702 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000101074 2024-09-18T04:38:24.702 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:24.702 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:24.702 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:24.702 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.702 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.702 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.4214.0:5 2.2 e18/17 ondisk, result = 0)", 2024-09-18T04:38:24.702 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:35.472226+0000", 2024-09-18T04:38:24.702 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 49.200006934999998, 2024-09-18T04:38:24.702 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048505299999999999, 2024-09-18T04:38:24.702 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:24.702 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:38:24.702 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:24.702 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.703 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:24.703 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472226+0000", 2024-09-18T04:38:24.703 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:24.703 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.703 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.703 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:24.703 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472226+0000", 2024-09-18T04:38:24.703 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:24.703 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.703 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.703 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:24.703 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472224+0000", 2024-09-18T04:38:24.703 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:24.704 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.704 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.704 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:24.704 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472238+0000", 2024-09-18T04:38:24.704 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.468e-05 2024-09-18T04:38:24.704 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.704 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.704 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:24.704 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472272+0000", 2024-09-18T04:38:24.704 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3586e-05 2024-09-18T04:38:24.704 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.704 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.704 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:24.705 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472297+0000", 2024-09-18T04:38:24.705 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5282999999999998e-05 2024-09-18T04:38:24.705 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.705 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.705 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:24.705 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472396+0000", 2024-09-18T04:38:24.705 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.8394999999999996e-05 2024-09-18T04:38:24.705 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.705 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.705 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:24.705 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472421+0000", 2024-09-18T04:38:24.705 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5479000000000001e-05 2024-09-18T04:38:24.705 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.706 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.706 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:24.706 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472711+0000", 2024-09-18T04:38:24.706 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000289714 2024-09-18T04:38:24.706 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:24.706 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:24.706 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:24.706 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.706 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.706 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-18T04:38:24.706 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:38:24.706 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 48.189359666000001, 2024-09-18T04:38:24.707 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082699560000000002, 2024-09-18T04:38:24.707 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:24.707 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:24.707 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:24.707 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:38:24.707 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:38:24.707 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 7 2024-09-18T04:38:24.707 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.707 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:24.707 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.707 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:24.707 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:38:24.707 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:24.707 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.708 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.708 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:24.708 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:38:24.708 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:24.708 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.708 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.708 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:24.708 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482870+0000", 2024-09-18T04:38:24.708 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999967 2024-09-18T04:38:24.708 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.708 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.708 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:24.708 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482889+0000", 2024-09-18T04:38:24.708 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.965e-05 2024-09-18T04:38:24.709 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.709 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.709 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:24.709 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482938+0000", 2024-09-18T04:38:24.709 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8548999999999999e-05 2024-09-18T04:38:24.709 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.709 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.709 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:24.709 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482963+0000", 2024-09-18T04:38:24.709 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5075999999999998e-05 2024-09-18T04:38:24.709 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.709 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.709 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:24.710 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483121+0000", 2024-09-18T04:38:24.710 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015819400000000001 2024-09-18T04:38:24.710 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.710 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.710 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op must wait for map", 2024-09-18T04:38:24.710 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483138+0000", 2024-09-18T04:38:24.710 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7068999999999999e-05 2024-09-18T04:38:24.710 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.710 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.710 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:24.710 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488666+0000", 2024-09-18T04:38:24.710 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0055282049999999996 2024-09-18T04:38:24.710 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.710 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.710 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:24.711 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488783+0000", 2024-09-18T04:38:24.711 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000116737 2024-09-18T04:38:24.711 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.711 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.711 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:24.711 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488991+0000", 2024-09-18T04:38:24.711 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020781400000000001 2024-09-18T04:38:24.711 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.711 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.711 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:24.711 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490554+0000", 2024-09-18T04:38:24.711 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0015628370000000001 2024-09-18T04:38:24.711 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.711 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.712 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:24.712 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491019+0000", 2024-09-18T04:38:24.712 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000465608 2024-09-18T04:38:24.712 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.712 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.712 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:24.712 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491087+0000", 2024-09-18T04:38:24.712 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7988e-05 2024-09-18T04:38:24.712 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.712 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.712 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:24.712 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491143+0000", 2024-09-18T04:38:24.712 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5538999999999998e-05 2024-09-18T04:38:24.713 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:24.713 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:24.713 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:24.713 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.713 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.713 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.4214.0:7 2.2 e19/17 ondisk, result = 0)", 2024-09-18T04:38:24.713 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:36.490861+0000", 2024-09-18T04:38:24.713 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 48.181372044, 2024-09-18T04:38:24.713 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00029639400000000001, 2024-09-18T04:38:24.713 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:24.713 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:38:24.713 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:24.713 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.713 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:24.714 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490861+0000", 2024-09-18T04:38:24.714 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:24.714 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.714 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.714 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:24.714 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490861+0000", 2024-09-18T04:38:24.714 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:24.714 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.714 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.714 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:24.714 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490859+0000", 2024-09-18T04:38:24.714 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:24.714 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.714 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.715 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:24.715 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490867+0000", 2024-09-18T04:38:24.715 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.3089999999999996e-06 2024-09-18T04:38:24.715 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.715 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.715 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:24.715 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490909+0000", 2024-09-18T04:38:24.715 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1919000000000003e-05 2024-09-18T04:38:24.715 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.715 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.715 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:24.715 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490922+0000", 2024-09-18T04:38:24.715 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2836999999999999e-05 2024-09-18T04:38:24.715 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.715 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.715 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:24.716 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491003+0000", 2024-09-18T04:38:24.716 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0681999999999998e-05 2024-09-18T04:38:24.716 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.716 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.716 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:24.716 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491013+0000", 2024-09-18T04:38:24.716 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0220999999999999e-05 2024-09-18T04:38:24.716 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.716 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.716 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:24.716 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491157+0000", 2024-09-18T04:38:24.716 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014401799999999999 2024-09-18T04:38:24.716 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:24.716 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:24.717 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:24.717 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.717 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.717 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "MOSDPGPushReply(2.3 25/24 [PushReplyOp(2:c4c92e5a:::rbd_trash:head)])", 2024-09-18T04:38:24.717 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:51.680323+0000", 2024-09-18T04:38:24.717 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 32.991909669000002, 2024-09-18T04:38:24.717 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00054411499999999996, 2024-09-18T04:38:24.717 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:24.717 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-09-18T04:38:24.717 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:24.717 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.717 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:24.717 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680323+0000", 2024-09-18T04:38:24.717 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:24.718 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.718 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.718 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:24.718 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680323+0000", 2024-09-18T04:38:24.718 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:24.718 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.718 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.718 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:24.718 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680322+0000", 2024-09-18T04:38:24.718 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:24.719 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.719 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.719 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:24.719 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680333+0000", 2024-09-18T04:38:24.719 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1153e-05 2024-09-18T04:38:24.719 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.719 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.719 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:24.719 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680357+0000", 2024-09-18T04:38:24.719 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3252999999999999e-05 2024-09-18T04:38:24.719 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.719 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.719 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:24.720 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680378+0000", 2024-09-18T04:38:24.720 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1503000000000001e-05 2024-09-18T04:38:24.720 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.720 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.720 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:24.720 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680522+0000", 2024-09-18T04:38:24.720 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014410700000000001 2024-09-18T04:38:24.720 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:24.720 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:24.720 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:24.720 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680867+0000", 2024-09-18T04:38:24.720 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00034475899999999998 2024-09-18T04:38:24.720 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:24.720 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:24.721 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:24.721 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:24.721 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:24.721 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:24.722 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:38:24.770 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:24.766+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:24.838 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:24.838 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:24.838 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:38:24.838 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:24.842 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:24.838+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:24.849 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:38:24.872 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T04:38:24.944 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:24.938+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:24.958 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:24.958 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:24.958 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:38:24.959 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:38:24.959 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:24.969 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:38:25.038 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:38:25.045 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:25.038+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:25.058 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:38:25.071 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:25.071 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:25.077 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:38:25.082 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:38:25.099 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:38:25.122 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:38:25.144 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:38:25.145 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:38:25.146 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:38:25.146 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:38:25.146 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:38:25.146 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:38:25.146 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:38:25.147 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:25.142+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:25.185 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:25.185 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:25.195 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:38:25.247 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:25.242+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:25.348 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:25.348 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:25.348 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:25.342+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:25.358 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:38:25.449 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:25.442+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:25.470 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:25.470 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:25.480 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:38:25.550 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:25.546+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:25.593 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:25.593 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:25.600 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:38:25.651 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:25.646+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:25.708 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:25.708 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:25.718 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:38:25.751 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:25.746+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:25.827 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:25.828 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:25.837 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:38:25.882 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:25.878+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:25.935 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-18T04:38:25.953 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:25.946+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:25.955 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:25.955 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:25.965 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:38:26.054 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:26.050+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:26.073 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:26.073 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:26.083 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:38:26.155 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:26.150+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:26.235 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:26.235 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:26.245 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:38:26.290 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:26.286+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:26.353 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:26.353 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:26.357 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:26.354+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:26.363 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:38:26.458 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:26.454+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:26.472 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:26.472 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:26.482 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:38:26.558 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:26.554+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:26.590 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:26.590 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:26.600 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:38:26.659 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:26.654+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:26.753 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:26.753 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:26.759 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:26.754+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:26.763 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:38:26.860 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:26.854+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:26.862 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-09-18T04:38:26.878 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:26.879 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:26.889 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:38:26.961 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:26.954+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:27.038 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:27.038 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:27.048 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:38:27.062 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:27.058+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:27.158 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T04:38:27.172 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:27.172 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:27.173 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:27.166+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:27.183 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:38:27.264 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:27.258+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:27.298 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:38:27.321 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:38:27.341 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:38:27.342 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:27.342 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:27.342 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:38:27.342 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:38:27.342 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:27.352 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:38:27.362 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:38:27.365 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:27.358+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:27.403 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:38:27.429 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:38:27.431 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:38:27.431 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:38:27.431 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:38:27.431 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:38:27.431 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:38:27.431 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:38:27.465 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:27.458+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:27.466 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:27.466 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:38:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:38:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:38:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:30 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'16, mlcod=15'14)", 2024-09-18T04:38:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.486330+0000", 2024-09-18T04:38:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 59.976325803999998, 2024-09-18T04:38:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026383700000000001, 2024-09-18T04:38:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486330+0000", 2024-09-18T04:38:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486330+0000", 2024-09-18T04:38:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486328+0000", 2024-09-18T04:38:27.469 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:27.469 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.469 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.469 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:27.469 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486342+0000", 2024-09-18T04:38:27.469 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3783e-05 2024-09-18T04:38:27.469 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.469 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.469 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:27.470 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486373+0000", 2024-09-18T04:38:27.470 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1362999999999998e-05 2024-09-18T04:38:27.470 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.470 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.470 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:27.470 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486387+0000", 2024-09-18T04:38:27.470 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3219e-05 2024-09-18T04:38:27.470 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.470 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.470 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:27.470 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486489+0000", 2024-09-18T04:38:27.471 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010232499999999999 2024-09-18T04:38:27.471 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.471 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.471 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:27.471 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486524+0000", 2024-09-18T04:38:27.471 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4972000000000003e-05 2024-09-18T04:38:27.471 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.471 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.471 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:27.471 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.488902+0000", 2024-09-18T04:38:27.471 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023776240000000001 2024-09-18T04:38:27.471 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.471 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.471 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:27.472 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.488968+0000", 2024-09-18T04:38:27.472 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6761000000000007e-05 2024-09-18T04:38:27.472 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:27.472 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:27.472 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:27.472 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.472 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.472 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:35 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'21, mlcod=15'17)", 2024-09-18T04:38:27.472 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.491817+0000", 2024-09-18T04:38:27.472 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 59.970838886000003, 2024-09-18T04:38:27.472 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023524739999999998, 2024-09-18T04:38:27.472 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:27.473 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:27.473 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:27.473 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.473 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:27.473 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491817+0000", 2024-09-18T04:38:27.473 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:27.473 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.473 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.473 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:27.473 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491817+0000", 2024-09-18T04:38:27.473 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:27.473 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.474 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.474 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:27.474 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491815+0000", 2024-09-18T04:38:27.474 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:27.474 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.474 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.474 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:27.474 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491890+0000", 2024-09-18T04:38:27.474 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4500999999999997e-05 2024-09-18T04:38:27.474 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.474 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.474 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:27.474 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491916+0000", 2024-09-18T04:38:27.475 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5666000000000001e-05 2024-09-18T04:38:27.475 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.475 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.475 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:27.475 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491928+0000", 2024-09-18T04:38:27.475 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2585999999999999e-05 2024-09-18T04:38:27.475 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.475 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.475 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:27.475 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.492000+0000", 2024-09-18T04:38:27.475 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2309000000000002e-05 2024-09-18T04:38:27.475 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.476 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.476 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:27.476 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.492027+0000", 2024-09-18T04:38:27.476 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6251e-05 2024-09-18T04:38:27.476 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.476 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.476 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:27.476 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.494114+0000", 2024-09-18T04:38:27.476 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020875260000000001 2024-09-18T04:38:27.476 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.476 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.476 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:27.476 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.494169+0000", 2024-09-18T04:38:27.477 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5145000000000001e-05 2024-09-18T04:38:27.477 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:27.477 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:27.477 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:27.477 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.477 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.477 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:57 1.0 e15/14 1:30306672:devicehealth::main.db.0000000000000000:head v 15'35, mlcod=15'33)", 2024-09-18T04:38:27.477 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.529997+0000", 2024-09-18T04:38:27.477 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 59.932659031999997, 2024-09-18T04:38:27.477 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002623869, 2024-09-18T04:38:27.477 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:27.477 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:27.478 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:27.478 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.478 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:27.478 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.529997+0000", 2024-09-18T04:38:27.478 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:27.478 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.478 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.478 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:27.478 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.529997+0000", 2024-09-18T04:38:27.478 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:27.478 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.478 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.478 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:27.479 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.529995+0000", 2024-09-18T04:38:27.479 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:27.479 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.479 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.479 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:27.479 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530116+0000", 2024-09-18T04:38:27.479 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000121141 2024-09-18T04:38:27.479 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.479 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.479 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:27.479 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530146+0000", 2024-09-18T04:38:27.479 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0423e-05 2024-09-18T04:38:27.480 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.480 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.480 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:27.480 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530159+0000", 2024-09-18T04:38:27.480 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2802999999999999e-05 2024-09-18T04:38:27.480 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.480 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.480 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:27.481 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530218+0000", 2024-09-18T04:38:27.481 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8461000000000002e-05 2024-09-18T04:38:27.481 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.481 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.481 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:27.481 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530253+0000", 2024-09-18T04:38:27.481 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4796999999999999e-05 2024-09-18T04:38:27.481 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.481 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.481 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:27.481 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.532559+0000", 2024-09-18T04:38:27.481 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002306939 2024-09-18T04:38:27.482 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.482 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.482 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:27.482 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.532621+0000", 2024-09-18T04:38:27.482 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1131999999999997e-05 2024-09-18T04:38:27.482 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:27.482 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:27.482 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:27.482 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.482 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.482 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:68 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'42, mlcod=15'39)", 2024-09-18T04:38:27.482 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.546404+0000", 2024-09-18T04:38:27.482 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 59.916251621000001, 2024-09-18T04:38:27.483 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024499769999999999, 2024-09-18T04:38:27.483 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:27.483 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:27.483 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:27.483 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.483 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:27.483 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546404+0000", 2024-09-18T04:38:27.483 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:27.483 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.483 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.483 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:27.483 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546404+0000", 2024-09-18T04:38:27.483 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:27.484 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.484 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.484 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:27.484 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546403+0000", 2024-09-18T04:38:27.484 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:27.484 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.484 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.484 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:27.484 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546416+0000", 2024-09-18T04:38:27.484 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.306e-05 2024-09-18T04:38:27.484 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.484 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.485 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:27.485 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546465+0000", 2024-09-18T04:38:27.485 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9441999999999999e-05 2024-09-18T04:38:27.485 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.485 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.485 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:27.485 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546480+0000", 2024-09-18T04:38:27.485 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4919999999999999e-05 2024-09-18T04:38:27.485 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.485 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.485 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:27.485 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546541+0000", 2024-09-18T04:38:27.485 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1085000000000002e-05 2024-09-18T04:38:27.485 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.486 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.486 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:27.486 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546569+0000", 2024-09-18T04:38:27.486 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7803000000000001e-05 2024-09-18T04:38:27.486 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.486 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.486 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:27.487 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.548725+0000", 2024-09-18T04:38:27.487 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021556629999999999 2024-09-18T04:38:27.487 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.487 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.487 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:27.487 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.548854+0000", 2024-09-18T04:38:27.487 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000129524 2024-09-18T04:38:27.487 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:27.487 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:27.487 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:27.487 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.488 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.488 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:69 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'43, mlcod=15'39)", 2024-09-18T04:38:27.488 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.547520+0000", 2024-09-18T04:38:27.488 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 59.915135544999998, 2024-09-18T04:38:27.488 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023506820000000002, 2024-09-18T04:38:27.488 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:27.488 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:27.488 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:27.488 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.488 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:27.488 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547520+0000", 2024-09-18T04:38:27.488 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:27.488 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.489 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.489 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:27.489 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547520+0000", 2024-09-18T04:38:27.489 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:27.489 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.489 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.489 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:27.489 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547519+0000", 2024-09-18T04:38:27.489 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:27.489 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.490 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.490 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:27.490 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547614+0000", 2024-09-18T04:38:27.490 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5235000000000006e-05 2024-09-18T04:38:27.490 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.490 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.490 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:27.490 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547640+0000", 2024-09-18T04:38:27.490 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5698999999999999e-05 2024-09-18T04:38:27.490 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.490 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.490 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:27.491 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547651+0000", 2024-09-18T04:38:27.491 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.134e-05 2024-09-18T04:38:27.491 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.491 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.491 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:27.491 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547805+0000", 2024-09-18T04:38:27.491 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015426099999999999 2024-09-18T04:38:27.491 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.491 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.491 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:27.491 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547844+0000", 2024-09-18T04:38:27.491 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8229999999999998e-05 2024-09-18T04:38:27.491 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.491 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.492 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:27.492 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.549810+0000", 2024-09-18T04:38:27.492 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0019660519999999998 2024-09-18T04:38:27.492 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.492 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.492 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:27.492 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.549871+0000", 2024-09-18T04:38:27.492 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1152e-05 2024-09-18T04:38:27.492 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:27.492 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:27.492 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:27.492 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.492 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.493 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:83 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'53, mlcod=15'50)", 2024-09-18T04:38:27.493 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.575989+0000", 2024-09-18T04:38:27.493 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 59.886666597000001, 2024-09-18T04:38:27.493 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023991310000000001, 2024-09-18T04:38:27.493 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:27.493 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:27.493 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:27.493 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.493 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:27.493 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.575989+0000", 2024-09-18T04:38:27.493 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:27.493 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.494 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.494 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:27.494 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.575989+0000", 2024-09-18T04:38:27.494 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:27.494 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.494 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.494 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:27.494 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.575988+0000", 2024-09-18T04:38:27.494 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:27.494 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.494 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.494 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:27.494 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576001+0000", 2024-09-18T04:38:27.494 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2751e-05 2024-09-18T04:38:27.495 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.495 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.495 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:27.495 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576023+0000", 2024-09-18T04:38:27.495 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2453999999999999e-05 2024-09-18T04:38:27.495 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.495 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.495 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:27.495 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576038+0000", 2024-09-18T04:38:27.495 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5125e-05 2024-09-18T04:38:27.495 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.495 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.495 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:27.495 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576161+0000", 2024-09-18T04:38:27.496 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000122543 2024-09-18T04:38:27.496 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.496 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.496 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:27.496 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576221+0000", 2024-09-18T04:38:27.496 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9948000000000003e-05 2024-09-18T04:38:27.496 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.496 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.496 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:27.496 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.578277+0000", 2024-09-18T04:38:27.496 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020567559999999999 2024-09-18T04:38:27.496 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.496 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.497 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:27.497 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.578388+0000", 2024-09-18T04:38:27.497 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011089100000000001 2024-09-18T04:38:27.497 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:27.497 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:27.497 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:27.497 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.497 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.497 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:86 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'56, mlcod=15'52)", 2024-09-18T04:38:27.497 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.580261+0000", 2024-09-18T04:38:27.497 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 59.882394961000003, 2024-09-18T04:38:27.497 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002418108, 2024-09-18T04:38:27.497 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:27.498 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:27.498 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:27.498 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.498 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:27.498 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580261+0000", 2024-09-18T04:38:27.498 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:27.498 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.498 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.498 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:27.498 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580261+0000", 2024-09-18T04:38:27.498 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:27.498 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.498 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.498 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:27.499 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580260+0000", 2024-09-18T04:38:27.499 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:27.499 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.499 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.499 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:27.499 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580269+0000", 2024-09-18T04:38:27.499 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5109999999999992e-06 2024-09-18T04:38:27.499 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.499 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.499 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:27.499 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580288+0000", 2024-09-18T04:38:27.499 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8783999999999998e-05 2024-09-18T04:38:27.499 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.499 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.500 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:27.500 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580296+0000", 2024-09-18T04:38:27.500 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3250000000000008e-06 2024-09-18T04:38:27.500 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.500 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.500 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:27.500 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580351+0000", 2024-09-18T04:38:27.500 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4132000000000003e-05 2024-09-18T04:38:27.500 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.500 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.500 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:27.500 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580381+0000", 2024-09-18T04:38:27.500 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0263000000000001e-05 2024-09-18T04:38:27.501 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.501 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.501 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:27.501 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.582615+0000", 2024-09-18T04:38:27.501 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0022339109999999999 2024-09-18T04:38:27.501 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.501 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.501 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:27.501 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.582679+0000", 2024-09-18T04:38:27.501 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4182000000000003e-05 2024-09-18T04:38:27.501 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:27.501 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:27.501 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:27.501 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.502 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.502 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:105 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'67, mlcod=15'64)", 2024-09-18T04:38:27.502 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.789089+0000", 2024-09-18T04:38:27.502 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 59.673566790999999, 2024-09-18T04:38:27.502 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024384229999999999, 2024-09-18T04:38:27.502 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:27.502 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:27.502 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:27.502 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.502 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:27.502 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789089+0000", 2024-09-18T04:38:27.502 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:27.502 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.503 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.503 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:27.503 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789089+0000", 2024-09-18T04:38:27.503 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:27.503 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.503 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.503 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:27.503 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789088+0000", 2024-09-18T04:38:27.503 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:27.503 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.503 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.503 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:27.503 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789107+0000", 2024-09-18T04:38:27.503 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9017000000000001e-05 2024-09-18T04:38:27.504 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.504 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.504 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:27.504 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789139+0000", 2024-09-18T04:38:27.504 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2261999999999997e-05 2024-09-18T04:38:27.504 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.504 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.504 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:27.504 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789153+0000", 2024-09-18T04:38:27.504 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4025999999999999e-05 2024-09-18T04:38:27.504 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.504 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.504 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:27.505 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789255+0000", 2024-09-18T04:38:27.505 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000101969 2024-09-18T04:38:27.505 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.505 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.505 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:27.505 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789314+0000", 2024-09-18T04:38:27.505 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8867999999999999e-05 2024-09-18T04:38:27.505 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.505 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.505 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:27.505 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.791453+0000", 2024-09-18T04:38:27.505 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002138938 2024-09-18T04:38:27.505 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.506 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.506 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:27.506 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.791527+0000", 2024-09-18T04:38:27.506 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4464999999999999e-05 2024-09-18T04:38:27.506 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:27.506 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:27.506 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:27.506 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.506 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.506 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:106 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'68, mlcod=15'64)", 2024-09-18T04:38:27.506 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.790295+0000", 2024-09-18T04:38:27.506 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 59.672360939999997, 2024-09-18T04:38:27.506 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024379390000000001, 2024-09-18T04:38:27.506 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:27.507 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:27.507 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:27.507 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.507 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:27.507 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790295+0000", 2024-09-18T04:38:27.507 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:27.507 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.507 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.507 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:27.507 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790295+0000", 2024-09-18T04:38:27.507 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:27.507 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.507 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.507 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:27.508 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790294+0000", 2024-09-18T04:38:27.508 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:27.508 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.508 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.508 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:27.508 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790390+0000", 2024-09-18T04:38:27.508 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6537000000000001e-05 2024-09-18T04:38:27.508 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.508 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.508 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:27.508 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790418+0000", 2024-09-18T04:38:27.508 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8218e-05 2024-09-18T04:38:27.508 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.508 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.509 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:27.509 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790431+0000", 2024-09-18T04:38:27.509 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2571999999999999e-05 2024-09-18T04:38:27.509 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.509 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.509 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:27.509 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790534+0000", 2024-09-18T04:38:27.509 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000103433 2024-09-18T04:38:27.509 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.509 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.509 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:27.509 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790565+0000", 2024-09-18T04:38:27.509 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0409999999999999e-05 2024-09-18T04:38:27.509 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.510 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.510 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:27.510 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.792651+0000", 2024-09-18T04:38:27.510 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020865250000000001 2024-09-18T04:38:27.510 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.510 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.510 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:27.510 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.792733+0000", 2024-09-18T04:38:27.510 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1477999999999995e-05 2024-09-18T04:38:27.510 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:27.510 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:27.510 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:27.510 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.510 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.511 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:109 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'71, mlcod=15'68)", 2024-09-18T04:38:27.511 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.795250+0000", 2024-09-18T04:38:27.511 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 59.667405926000001, 2024-09-18T04:38:27.511 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023990449999999998, 2024-09-18T04:38:27.511 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:27.511 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:27.511 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:27.511 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.511 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:27.511 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795250+0000", 2024-09-18T04:38:27.511 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:27.511 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.511 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.512 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:27.512 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795250+0000", 2024-09-18T04:38:27.512 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:27.512 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.512 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.512 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:27.512 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795248+0000", 2024-09-18T04:38:27.512 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:27.512 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.512 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.512 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:27.512 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795405+0000", 2024-09-18T04:38:27.512 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000157007 2024-09-18T04:38:27.512 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.513 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.513 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:27.513 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795422+0000", 2024-09-18T04:38:27.513 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7013000000000001e-05 2024-09-18T04:38:27.513 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.513 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.513 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:27.513 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795432+0000", 2024-09-18T04:38:27.513 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0448999999999999e-05 2024-09-18T04:38:27.513 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.513 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.513 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:27.514 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795534+0000", 2024-09-18T04:38:27.514 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000101195 2024-09-18T04:38:27.514 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.514 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.514 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:27.514 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795587+0000", 2024-09-18T04:38:27.514 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3356000000000002e-05 2024-09-18T04:38:27.514 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.514 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.514 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:27.514 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.797560+0000", 2024-09-18T04:38:27.514 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001973023 2024-09-18T04:38:27.514 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.515 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.515 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:27.515 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.797649+0000", 2024-09-18T04:38:27.515 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8864999999999997e-05 2024-09-18T04:38:27.515 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:27.515 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:27.515 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:27.515 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.515 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.515 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:115 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'77, mlcod=15'72)", 2024-09-18T04:38:27.515 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.803339+0000", 2024-09-18T04:38:27.515 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 59.659317207999997, 2024-09-18T04:38:27.515 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002385093, 2024-09-18T04:38:27.515 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:27.516 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:27.516 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:27.516 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.516 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:27.516 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803339+0000", 2024-09-18T04:38:27.516 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:27.516 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.516 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.516 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:27.516 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803339+0000", 2024-09-18T04:38:27.516 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:27.516 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.516 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.517 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:27.517 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803337+0000", 2024-09-18T04:38:27.517 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:27.517 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.517 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.517 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:27.517 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803475+0000", 2024-09-18T04:38:27.517 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013718199999999999 2024-09-18T04:38:27.517 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.517 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.517 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:27.517 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803507+0000", 2024-09-18T04:38:27.517 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2351000000000002e-05 2024-09-18T04:38:27.518 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.518 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.518 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:27.518 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803533+0000", 2024-09-18T04:38:27.518 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6010999999999999e-05 2024-09-18T04:38:27.518 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.518 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.518 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:27.518 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803609+0000", 2024-09-18T04:38:27.518 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.64e-05 2024-09-18T04:38:27.518 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.518 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.518 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:27.519 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803671+0000", 2024-09-18T04:38:27.519 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1467e-05 2024-09-18T04:38:27.519 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.519 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.519 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:27.519 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.805676+0000", 2024-09-18T04:38:27.519 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020047849999999998 2024-09-18T04:38:27.519 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.519 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.519 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:27.519 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.805724+0000", 2024-09-18T04:38:27.519 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8151e-05 2024-09-18T04:38:27.519 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:27.520 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:27.520 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:27.520 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.520 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.520 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:118 1.0 e15/14 1:30306672:devicehealth::main.db.0000000000000000:head v 15'80, mlcod=15'78)", 2024-09-18T04:38:27.520 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.807930+0000", 2024-09-18T04:38:27.520 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 59.654725704999997, 2024-09-18T04:38:27.520 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028002259999999998, 2024-09-18T04:38:27.520 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:27.520 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:27.520 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:27.520 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.520 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:27.521 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.807930+0000", 2024-09-18T04:38:27.521 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:27.521 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.521 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.521 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:27.521 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.807930+0000", 2024-09-18T04:38:27.521 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:27.521 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.521 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.521 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:27.521 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.807929+0000", 2024-09-18T04:38:27.521 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:27.521 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.521 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.522 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:27.522 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808024+0000", 2024-09-18T04:38:27.522 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5258000000000002e-05 2024-09-18T04:38:27.522 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.522 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.522 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:27.522 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808044+0000", 2024-09-18T04:38:27.522 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0203e-05 2024-09-18T04:38:27.522 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.522 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.522 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:27.522 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808053+0000", 2024-09-18T04:38:27.522 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2689999999999999e-06 2024-09-18T04:38:27.522 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.523 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.523 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:27.523 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808137+0000", 2024-09-18T04:38:27.523 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4441e-05 2024-09-18T04:38:27.523 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.523 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.523 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:27.523 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808172+0000", 2024-09-18T04:38:27.523 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4453999999999999e-05 2024-09-18T04:38:27.523 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.523 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.523 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:27.523 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.810663+0000", 2024-09-18T04:38:27.523 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024910269999999998 2024-09-18T04:38:27.524 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.524 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.524 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:27.524 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.810730+0000", 2024-09-18T04:38:27.524 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7658999999999997e-05 2024-09-18T04:38:27.524 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:27.524 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:27.524 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:27.524 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.524 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.524 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:129 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'87, mlcod=15'85)", 2024-09-18T04:38:27.524 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.231689+0000", 2024-09-18T04:38:27.524 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 59.230966692000003, 2024-09-18T04:38:27.525 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002376502, 2024-09-18T04:38:27.525 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:27.525 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:27.525 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:27.525 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.525 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:27.525 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231689+0000", 2024-09-18T04:38:27.525 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:27.525 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.525 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.525 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:27.525 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231689+0000", 2024-09-18T04:38:27.525 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:27.525 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.526 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.526 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:27.526 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231687+0000", 2024-09-18T04:38:27.526 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:27.526 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.526 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.526 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:27.526 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231702+0000", 2024-09-18T04:38:27.526 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4949999999999999e-05 2024-09-18T04:38:27.526 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.526 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.526 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:27.526 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231735+0000", 2024-09-18T04:38:27.526 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2276000000000001e-05 2024-09-18T04:38:27.527 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.527 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.527 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:27.527 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231750+0000", 2024-09-18T04:38:27.527 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5193e-05 2024-09-18T04:38:27.527 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.527 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.527 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:27.527 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231872+0000", 2024-09-18T04:38:27.527 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000122127 2024-09-18T04:38:27.527 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.527 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.527 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:27.527 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231936+0000", 2024-09-18T04:38:27.528 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4089000000000003e-05 2024-09-18T04:38:27.528 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.528 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.528 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:27.528 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234002+0000", 2024-09-18T04:38:27.528 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002065575 2024-09-18T04:38:27.528 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.528 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.528 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:27.528 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234066+0000", 2024-09-18T04:38:27.528 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3944999999999998e-05 2024-09-18T04:38:27.528 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:27.528 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:27.529 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:27.529 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.529 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.529 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:130 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'88, mlcod=15'85)", 2024-09-18T04:38:27.529 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.232853+0000", 2024-09-18T04:38:27.529 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 59.229803077, 2024-09-18T04:38:27.529 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027351739999999999, 2024-09-18T04:38:27.529 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:27.529 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:27.529 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:27.529 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.529 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:27.529 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232853+0000", 2024-09-18T04:38:27.529 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:27.530 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.530 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.530 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:27.530 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232853+0000", 2024-09-18T04:38:27.530 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:27.530 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.530 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.530 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:27.530 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232851+0000", 2024-09-18T04:38:27.530 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:27.530 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.530 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.530 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:27.530 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232866+0000", 2024-09-18T04:38:27.531 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4946000000000001e-05 2024-09-18T04:38:27.531 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.531 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.531 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:27.531 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232897+0000", 2024-09-18T04:38:27.531 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0982999999999998e-05 2024-09-18T04:38:27.531 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.531 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.531 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:27.531 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232915+0000", 2024-09-18T04:38:27.531 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8476000000000001e-05 2024-09-18T04:38:27.531 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.531 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.532 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:27.532 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.233029+0000", 2024-09-18T04:38:27.532 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011305400000000001 2024-09-18T04:38:27.532 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.532 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.532 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:27.532 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.233079+0000", 2024-09-18T04:38:27.532 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0668999999999999e-05 2024-09-18T04:38:27.532 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.532 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.532 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:27.532 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235534+0000", 2024-09-18T04:38:27.532 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024546030000000001 2024-09-18T04:38:27.533 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.533 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.533 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:27.533 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235588+0000", 2024-09-18T04:38:27.533 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4098999999999998e-05 2024-09-18T04:38:27.533 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:27.533 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:27.533 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:27.533 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.533 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.533 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:131 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'89, mlcod=15'85)", 2024-09-18T04:38:27.533 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.234089+0000", 2024-09-18T04:38:27.534 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 59.228567222999999, 2024-09-18T04:38:27.534 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026975609999999998, 2024-09-18T04:38:27.534 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:27.534 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:27.534 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:27.534 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.534 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:27.534 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234089+0000", 2024-09-18T04:38:27.534 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:27.534 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.534 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.534 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:27.534 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234089+0000", 2024-09-18T04:38:27.535 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:27.535 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.535 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.535 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:27.535 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234087+0000", 2024-09-18T04:38:27.535 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:27.535 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.535 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.535 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:27.535 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234145+0000", 2024-09-18T04:38:27.535 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8687999999999998e-05 2024-09-18T04:38:27.535 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.535 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.535 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:27.536 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234177+0000", 2024-09-18T04:38:27.536 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1526999999999998e-05 2024-09-18T04:38:27.536 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.536 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.536 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:27.536 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234190+0000", 2024-09-18T04:38:27.536 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2965999999999999e-05 2024-09-18T04:38:27.536 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.536 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.536 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:27.536 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234309+0000", 2024-09-18T04:38:27.536 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011875399999999999 2024-09-18T04:38:27.536 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.537 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.537 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:27.537 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234365+0000", 2024-09-18T04:38:27.537 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6436e-05 2024-09-18T04:38:27.537 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.537 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.537 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:27.537 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236675+0000", 2024-09-18T04:38:27.537 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023099520000000001 2024-09-18T04:38:27.537 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.537 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.537 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:27.537 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236786+0000", 2024-09-18T04:38:27.538 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000111025 2024-09-18T04:38:27.538 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:27.538 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:27.538 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:27.538 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.538 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.538 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:132 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'90, mlcod=15'85)", 2024-09-18T04:38:27.538 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.235300+0000", 2024-09-18T04:38:27.538 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 59.227355772999999, 2024-09-18T04:38:27.538 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025982599999999998, 2024-09-18T04:38:27.538 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:27.538 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:27.538 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:27.539 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.539 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:27.539 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235300+0000", 2024-09-18T04:38:27.539 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:27.539 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.539 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.539 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:27.539 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235300+0000", 2024-09-18T04:38:27.539 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:27.539 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.539 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.539 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:27.539 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235298+0000", 2024-09-18T04:38:27.539 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:27.540 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.540 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.540 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:27.540 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235318+0000", 2024-09-18T04:38:27.540 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9714000000000001e-05 2024-09-18T04:38:27.540 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.540 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.540 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:27.540 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235337+0000", 2024-09-18T04:38:27.540 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8892000000000001e-05 2024-09-18T04:38:27.540 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.540 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.540 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:27.541 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235348+0000", 2024-09-18T04:38:27.541 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.129e-05 2024-09-18T04:38:27.541 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.541 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.541 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:27.541 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235681+0000", 2024-09-18T04:38:27.541 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00033323000000000001 2024-09-18T04:38:27.541 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.541 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.541 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:27.541 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235733+0000", 2024-09-18T04:38:27.541 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1391999999999999e-05 2024-09-18T04:38:27.541 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.541 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.542 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:27.542 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237839+0000", 2024-09-18T04:38:27.542 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002105647 2024-09-18T04:38:27.542 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.542 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.542 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:27.542 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237898+0000", 2024-09-18T04:38:27.542 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9759000000000002e-05 2024-09-18T04:38:27.542 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:27.542 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:27.542 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:27.542 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.542 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.543 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:133 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'91, mlcod=15'86)", 2024-09-18T04:38:27.543 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.236920+0000", 2024-09-18T04:38:27.543 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 59.225735884999999, 2024-09-18T04:38:27.543 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024318410000000001, 2024-09-18T04:38:27.543 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:27.543 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:27.543 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:27.543 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.543 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:27.543 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236920+0000", 2024-09-18T04:38:27.543 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:27.543 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.543 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.544 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:27.544 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236920+0000", 2024-09-18T04:38:27.544 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:27.544 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.544 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.544 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:27.544 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236918+0000", 2024-09-18T04:38:27.544 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:27.544 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.544 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.544 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:27.544 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236931+0000", 2024-09-18T04:38:27.544 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2683999999999999e-05 2024-09-18T04:38:27.544 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.545 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.545 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:27.545 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236951+0000", 2024-09-18T04:38:27.545 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0292000000000002e-05 2024-09-18T04:38:27.545 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.545 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.545 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:27.545 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236962+0000", 2024-09-18T04:38:27.545 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0703000000000001e-05 2024-09-18T04:38:27.545 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.545 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.545 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:27.545 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237095+0000", 2024-09-18T04:38:27.545 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013312400000000001 2024-09-18T04:38:27.546 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.546 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.546 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:27.546 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237133+0000", 2024-09-18T04:38:27.546 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7496000000000001e-05 2024-09-18T04:38:27.546 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.546 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.546 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:27.546 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.239206+0000", 2024-09-18T04:38:27.546 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020730290000000001 2024-09-18T04:38:27.546 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.546 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.546 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:27.547 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.239352+0000", 2024-09-18T04:38:27.547 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014593000000000001 2024-09-18T04:38:27.547 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:27.547 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:27.547 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:27.547 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.547 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.547 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:134 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'92, mlcod=15'86)", 2024-09-18T04:38:27.547 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.237894+0000", 2024-09-18T04:38:27.547 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 59.224761862999998, 2024-09-18T04:38:27.547 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025299519999999998, 2024-09-18T04:38:27.547 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:27.547 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:27.547 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:27.548 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.548 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:27.548 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237894+0000", 2024-09-18T04:38:27.548 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:27.548 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.548 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.548 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:27.548 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237894+0000", 2024-09-18T04:38:27.548 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:27.548 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.548 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.548 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:27.548 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237892+0000", 2024-09-18T04:38:27.549 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:27.549 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.549 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.549 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:27.549 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237986+0000", 2024-09-18T04:38:27.549 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3341e-05 2024-09-18T04:38:27.549 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.549 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.549 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:27.549 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.238012+0000", 2024-09-18T04:38:27.549 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5843e-05 2024-09-18T04:38:27.549 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.549 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.549 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:27.550 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.238027+0000", 2024-09-18T04:38:27.550 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5469e-05 2024-09-18T04:38:27.550 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.550 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.550 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:27.550 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.238114+0000", 2024-09-18T04:38:27.550 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7483000000000002e-05 2024-09-18T04:38:27.550 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.550 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.550 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:27.550 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.238152+0000", 2024-09-18T04:38:27.550 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7565999999999999e-05 2024-09-18T04:38:27.550 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.550 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.551 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:27.551 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.240297+0000", 2024-09-18T04:38:27.551 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021447620000000001 2024-09-18T04:38:27.551 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.551 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.551 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:27.551 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.240424+0000", 2024-09-18T04:38:27.551 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012705400000000001 2024-09-18T04:38:27.551 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:27.551 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:27.551 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:27.551 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.551 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.552 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:139 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'97, mlcod=15'93)", 2024-09-18T04:38:27.552 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.245692+0000", 2024-09-18T04:38:27.552 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 59.216963722000003, 2024-09-18T04:38:27.552 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024260480000000001, 2024-09-18T04:38:27.552 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:27.552 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:27.552 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:27.552 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.552 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:27.552 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245692+0000", 2024-09-18T04:38:27.552 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:27.552 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.552 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.553 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:27.553 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245692+0000", 2024-09-18T04:38:27.553 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:27.553 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.553 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.553 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:27.553 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245690+0000", 2024-09-18T04:38:27.553 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:27.553 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.553 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.553 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:27.553 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245712+0000", 2024-09-18T04:38:27.553 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1916000000000002e-05 2024-09-18T04:38:27.554 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.554 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.554 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:27.554 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245741+0000", 2024-09-18T04:38:27.554 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8503000000000001e-05 2024-09-18T04:38:27.554 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.554 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.554 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:27.554 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245759+0000", 2024-09-18T04:38:27.554 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.872e-05 2024-09-18T04:38:27.554 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.554 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.554 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:27.554 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245859+0000", 2024-09-18T04:38:27.555 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9641000000000004e-05 2024-09-18T04:38:27.555 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.555 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.555 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:27.555 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245900+0000", 2024-09-18T04:38:27.555 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0589000000000001e-05 2024-09-18T04:38:27.555 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.555 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.555 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:27.555 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.247961+0000", 2024-09-18T04:38:27.555 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020619190000000002 2024-09-18T04:38:27.555 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.555 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.556 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:27.556 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.248118+0000", 2024-09-18T04:38:27.556 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015664300000000001 2024-09-18T04:38:27.556 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:27.556 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:27.556 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:27.556 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.556 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.556 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:144 1.0 e15/14 1:30306672:devicehealth::main.db.0000000000000000:head v 15'102, mlcod=15'98)", 2024-09-18T04:38:27.556 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.252149+0000", 2024-09-18T04:38:27.556 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 59.210506656, 2024-09-18T04:38:27.556 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024174959999999999, 2024-09-18T04:38:27.556 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:27.557 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:27.557 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:27.557 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.557 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:27.557 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252149+0000", 2024-09-18T04:38:27.557 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:27.557 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.557 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.557 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:27.557 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252149+0000", 2024-09-18T04:38:27.557 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:27.557 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.557 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.557 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:27.558 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252148+0000", 2024-09-18T04:38:27.558 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:27.558 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.558 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.558 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:27.558 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252225+0000", 2024-09-18T04:38:27.558 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6576999999999995e-05 2024-09-18T04:38:27.558 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.558 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.558 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:27.558 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252242+0000", 2024-09-18T04:38:27.558 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7740999999999998e-05 2024-09-18T04:38:27.558 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.558 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.559 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:27.559 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252254+0000", 2024-09-18T04:38:27.559 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.137e-05 2024-09-18T04:38:27.559 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.559 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.559 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:27.559 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252385+0000", 2024-09-18T04:38:27.559 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013091099999999999 2024-09-18T04:38:27.559 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.559 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.559 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:27.559 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252426+0000", 2024-09-18T04:38:27.559 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1535999999999997e-05 2024-09-18T04:38:27.559 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.560 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.560 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:27.560 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.254498+0000", 2024-09-18T04:38:27.560 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002071477 2024-09-18T04:38:27.560 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.560 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.560 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:27.560 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.254567+0000", 2024-09-18T04:38:27.560 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9014e-05 2024-09-18T04:38:27.560 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:27.560 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:27.560 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:27.560 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:27.561 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:27.561 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:27.562 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:38:27.606 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:27.602+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:27.667 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:27.662+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:27.720 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:27.720 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:27.720 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:38:27.720 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:27.730 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:38:27.767 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:27.762+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:27.841 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:27.841 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:27.841 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:38:27.842 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:38:27.842 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:27.849 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:38:27.850 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-18T04:38:27.850 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-18T04:38:27.868 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:27.862+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:27.955 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:27.956 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:38:27.956 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:38:27.956 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:38:27.956 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.956 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4214.0:5 2.2 e19/17 2:5cea7035:::rbd_info:head v 19'1, mlcod=0'0)", 2024-09-18T04:38:27.956 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:35.471260+0000", 2024-09-18T04:38:27.956 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 52.483344242000001, 2024-09-18T04:38:27.956 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021025950000000001, 2024-09-18T04:38:27.956 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:27.956 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:27.956 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:27.956 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.956 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:27.957 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471260+0000", 2024-09-18T04:38:27.957 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:27.957 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.957 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.957 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:27.957 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471260+0000", 2024-09-18T04:38:27.957 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:27.957 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.957 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.957 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:27.957 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471259+0000", 2024-09-18T04:38:27.957 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:27.958 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.958 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.958 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:27.958 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471275+0000", 2024-09-18T04:38:27.958 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6135999999999999e-05 2024-09-18T04:38:27.958 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.958 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.958 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:27.958 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471298+0000", 2024-09-18T04:38:27.958 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3632999999999999e-05 2024-09-18T04:38:27.958 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.958 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.958 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:27.958 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471320+0000", 2024-09-18T04:38:27.958 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1852999999999999e-05 2024-09-18T04:38:27.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.959 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.959 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:27.959 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471445+0000", 2024-09-18T04:38:27.959 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012461699999999999 2024-09-18T04:38:27.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.959 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.959 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:27.959 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471529+0000", 2024-09-18T04:38:27.959 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3875000000000006e-05 2024-09-18T04:38:27.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.959 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.959 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:27.959 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.473317+0000", 2024-09-18T04:38:27.959 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017886899999999999 2024-09-18T04:38:27.960 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.960 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.960 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:27.960 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.473362+0000", 2024-09-18T04:38:27.960 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4724000000000001e-05 2024-09-18T04:38:27.960 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:27.960 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:27.960 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:27.960 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.960 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.960 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4214.0:7 2.2 e20/17 2:5cea7035:::rbd_info:head v 20'2, mlcod=0'0)", 2024-09-18T04:38:27.960 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:36.490560+0000", 2024-09-18T04:38:27.960 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 51.464044102999999, 2024-09-18T04:38:27.960 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0015326020000000001, 2024-09-18T04:38:27.961 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:27.961 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:27.961 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:27.961 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.961 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:27.961 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490560+0000", 2024-09-18T04:38:27.961 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:27.961 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.961 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.961 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:27.961 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490560+0000", 2024-09-18T04:38:27.961 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:27.961 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.961 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.962 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:27.962 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490559+0000", 2024-09-18T04:38:27.962 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:27.962 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.962 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.962 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:27.962 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490568+0000", 2024-09-18T04:38:27.962 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5699999999999993e-06 2024-09-18T04:38:27.962 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.962 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.962 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:27.962 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490599+0000", 2024-09-18T04:38:27.962 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1566000000000002e-05 2024-09-18T04:38:27.962 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.962 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.963 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:27.963 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490609+0000", 2024-09-18T04:38:27.963 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.8700000000000004e-06 2024-09-18T04:38:27.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.963 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.963 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:27.963 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490708+0000", 2024-09-18T04:38:27.963 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9139000000000001e-05 2024-09-18T04:38:27.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.963 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.963 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:27.963 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490736+0000", 2024-09-18T04:38:27.963 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7617e-05 2024-09-18T04:38:27.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.964 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.964 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:27.964 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.492016+0000", 2024-09-18T04:38:27.964 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0012796579999999999 2024-09-18T04:38:27.964 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:27.964 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:27.964 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:27.964 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.492092+0000", 2024-09-18T04:38:27.964 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6571999999999998e-05 2024-09-18T04:38:27.964 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:27.964 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:27.964 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:27.964 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:27.964 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:27.964 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:27.965 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:38:27.968 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:27.962+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:28.069 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:28.069 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:28.069 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:38:28.070 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:28.070 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:28.062+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:28.078 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:38:28.170 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:28.166+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:28.180 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:28.180 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:28.180 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:38:28.180 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:38:28.180 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:28.191 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:38:28.271 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:28.266+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:28.297 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:28.297 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:38:28.297 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:38:28.297 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:38:28.297 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:28.297 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4214.0:1 2.3 e18/17 2:c4c92e5a:::rbd_trash:head v 18'1, mlcod=0'0)", 2024-09-18T04:38:28.297 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:34.475123+0000", 2024-09-18T04:38:28.297 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 53.820668327, 2024-09-18T04:38:28.297 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00081394599999999996, 2024-09-18T04:38:28.298 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:28.298 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:28.298 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:28.298 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:28.298 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:28.298 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475123+0000", 2024-09-18T04:38:28.298 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:28.298 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:28.298 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:28.298 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:28.298 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475123+0000", 2024-09-18T04:38:28.298 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:28.298 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:28.298 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:28.298 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:28.298 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475121+0000", 2024-09-18T04:38:28.299 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:28.299 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:28.299 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:28.299 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:28.299 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475149+0000", 2024-09-18T04:38:28.299 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7155999999999999e-05 2024-09-18T04:38:28.299 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:28.299 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:28.299 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:28.299 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475165+0000", 2024-09-18T04:38:28.299 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6160000000000001e-05 2024-09-18T04:38:28.299 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:28.299 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:28.299 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:28.300 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475183+0000", 2024-09-18T04:38:28.300 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7918e-05 2024-09-18T04:38:28.300 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:28.300 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:28.300 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:28.300 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475246+0000", 2024-09-18T04:38:28.300 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3176000000000006e-05 2024-09-18T04:38:28.300 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:28.300 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:28.300 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:28.300 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475279+0000", 2024-09-18T04:38:28.300 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3192999999999999e-05 2024-09-18T04:38:28.300 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:28.300 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:28.301 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:28.301 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475871+0000", 2024-09-18T04:38:28.301 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00059206099999999995 2024-09-18T04:38:28.301 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:28.301 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:28.301 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:28.301 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475937+0000", 2024-09-18T04:38:28.301 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5439999999999997e-05 2024-09-18T04:38:28.301 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:28.301 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:28.301 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:28.301 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:28.301 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:28.301 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 25/24 [PushOp(2:c4c92e5a:::rbd_trash:head, version: 18'1, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:c4c92e5a:::rbd_trash:head@18'1, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-18T04:38:28.302 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:51.680544+0000", 2024-09-18T04:38:28.302 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 36.615246485, 2024-09-18T04:38:28.302 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00050365200000000005, 2024-09-18T04:38:28.302 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:28.302 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:38:28.302 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:28.302 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:28.302 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:28.302 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680544+0000", 2024-09-18T04:38:28.302 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:28.302 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:28.302 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:28.302 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:28.302 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680544+0000", 2024-09-18T04:38:28.303 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:28.303 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:28.303 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:28.303 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:28.303 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680543+0000", 2024-09-18T04:38:28.303 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:28.303 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:28.303 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:28.303 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:28.303 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680571+0000", 2024-09-18T04:38:28.303 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7953000000000002e-05 2024-09-18T04:38:28.303 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:28.303 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:28.303 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:28.303 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680605+0000", 2024-09-18T04:38:28.304 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3933000000000002e-05 2024-09-18T04:38:28.304 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:28.304 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:28.304 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:28.304 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680636+0000", 2024-09-18T04:38:28.304 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1288999999999998e-05 2024-09-18T04:38:28.304 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:28.304 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:28.304 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:28.304 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680748+0000", 2024-09-18T04:38:28.304 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000111431 2024-09-18T04:38:28.305 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:28.305 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:28.305 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:28.305 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680758+0000", 2024-09-18T04:38:28.305 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0244999999999999e-05 2024-09-18T04:38:28.305 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:28.305 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:28.305 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:28.305 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.681048+0000", 2024-09-18T04:38:28.305 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00029003399999999999 2024-09-18T04:38:28.305 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:28.305 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:28.305 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:28.305 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:28.306 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:28.306 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:28.308 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:38:28.372 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:28.366+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:28.425 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:28.425 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:28.425 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:38:28.425 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:28.435 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:38:28.473 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:28.466+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:28.543 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:28.543 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:28.543 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:38:28.543 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:38:28.543 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:28.552 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:38:28.574 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:28.570+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:28.665 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:28.666 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:38:28.666 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:38:28.666 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:38:28.666 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.666 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:28.666 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.573328+0000", 2024-09-18T04:38:28.666 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 61.086682179, 2024-09-18T04:38:28.666 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011397898, 2024-09-18T04:38:28.666 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:28.666 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:28.666 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:28.666 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:28.667 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:28.667 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 86 2024-09-18T04:38:28.667 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.667 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:28.667 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.667 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:28.667 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573328+0000", 2024-09-18T04:38:28.667 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:28.667 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.667 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.667 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:28.667 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573328+0000", 2024-09-18T04:38:28.667 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:28.667 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.667 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.668 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:28.668 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573327+0000", 2024-09-18T04:38:28.668 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:28.668 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.668 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.668 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:28.668 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573331+0000", 2024-09-18T04:38:28.668 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9750000000000001e-06 2024-09-18T04:38:28.668 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.668 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.668 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:28.668 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573346+0000", 2024-09-18T04:38:28.668 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5469999999999999e-05 2024-09-18T04:38:28.668 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.668 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.669 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:28.669 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573370+0000", 2024-09-18T04:38:28.669 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3969999999999999e-05 2024-09-18T04:38:28.669 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.669 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.669 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:28.669 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.578536+0000", 2024-09-18T04:38:28.669 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0051659679999999999 2024-09-18T04:38:28.669 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.669 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.669 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:28.669 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.578689+0000", 2024-09-18T04:38:28.669 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015276299999999999 2024-09-18T04:38:28.670 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.670 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.670 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:28.670 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.578901+0000", 2024-09-18T04:38:28.670 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000211783 2024-09-18T04:38:28.670 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.670 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.670 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:28.670 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.583408+0000", 2024-09-18T04:38:28.670 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045071249999999998 2024-09-18T04:38:28.670 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.670 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.670 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:28.670 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.584645+0000", 2024-09-18T04:38:28.671 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001237007 2024-09-18T04:38:28.671 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.671 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.671 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:28.671 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.584703+0000", 2024-09-18T04:38:28.671 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.817e-05 2024-09-18T04:38:28.671 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.671 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.671 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:28.671 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.584725+0000", 2024-09-18T04:38:28.671 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2155e-05 2024-09-18T04:38:28.671 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:28.671 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:28.671 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:28.671 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.672 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.672 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:28.672 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.573382+0000", 2024-09-18T04:38:28.672 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 61.086627430999997, 2024-09-18T04:38:28.672 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012154974000000001, 2024-09-18T04:38:28.672 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:28.672 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:28.672 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:28.672 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:28.672 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:28.672 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 87 2024-09-18T04:38:28.672 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.672 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:28.672 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.673 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:28.673 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573382+0000", 2024-09-18T04:38:28.673 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:28.673 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.673 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.673 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:28.673 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573382+0000", 2024-09-18T04:38:28.673 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:28.673 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.673 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.673 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:28.673 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573382+0000", 2024-09-18T04:38:28.673 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:28.674 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.674 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.674 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:28.674 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573482+0000", 2024-09-18T04:38:28.674 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000100333 2024-09-18T04:38:28.674 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.674 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.674 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:28.674 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573492+0000", 2024-09-18T04:38:28.674 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5389999999999996e-06 2024-09-18T04:38:28.674 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.674 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.674 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:28.674 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573500+0000", 2024-09-18T04:38:28.675 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8929999999999994e-06 2024-09-18T04:38:28.675 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.675 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.675 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:28.675 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.579797+0000", 2024-09-18T04:38:28.675 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0062963239999999998 2024-09-18T04:38:28.675 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.675 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.675 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:28.675 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.579957+0000", 2024-09-18T04:38:28.675 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000160283 2024-09-18T04:38:28.675 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.675 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.675 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:28.675 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.580178+0000", 2024-09-18T04:38:28.675 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00022079200000000001 2024-09-18T04:38:28.676 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.676 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.676 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:28.676 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.583479+0000", 2024-09-18T04:38:28.676 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033007689999999998 2024-09-18T04:38:28.676 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.676 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.676 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:28.676 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585477+0000", 2024-09-18T04:38:28.676 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0019985670000000001 2024-09-18T04:38:28.676 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.676 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.676 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:28.676 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585517+0000", 2024-09-18T04:38:28.676 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9709999999999998e-05 2024-09-18T04:38:28.677 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.677 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.677 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:28.677 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585537+0000", 2024-09-18T04:38:28.677 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0326000000000002e-05 2024-09-18T04:38:28.677 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:28.677 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:28.677 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:28.677 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.677 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.677 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:28.677 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.573508+0000", 2024-09-18T04:38:28.677 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 61.086501724999998, 2024-09-18T04:38:28.677 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012068476, 2024-09-18T04:38:28.678 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:28.678 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:28.678 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:28.678 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:28.678 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:28.678 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 88 2024-09-18T04:38:28.678 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.678 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:28.678 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.678 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:28.678 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573508+0000", 2024-09-18T04:38:28.678 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:28.678 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.678 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.678 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:28.678 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573508+0000", 2024-09-18T04:38:28.679 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:28.679 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.679 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.679 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:28.679 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573507+0000", 2024-09-18T04:38:28.679 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:28.679 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.679 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.679 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:28.679 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573512+0000", 2024-09-18T04:38:28.679 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.2629999999999997e-06 2024-09-18T04:38:28.679 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.679 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.679 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:28.680 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573530+0000", 2024-09-18T04:38:28.680 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8507000000000001e-05 2024-09-18T04:38:28.680 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.680 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.680 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:28.680 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573537+0000", 2024-09-18T04:38:28.680 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.832e-06 2024-09-18T04:38:28.680 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.680 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.680 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:28.680 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.582486+0000", 2024-09-18T04:38:28.680 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0089494770000000008 2024-09-18T04:38:28.680 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.680 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.680 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:28.681 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.582630+0000", 2024-09-18T04:38:28.681 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000144005 2024-09-18T04:38:28.681 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.681 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.681 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:28.681 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.582842+0000", 2024-09-18T04:38:28.681 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00021189200000000001 2024-09-18T04:38:28.681 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.681 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.681 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:28.681 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.584777+0000", 2024-09-18T04:38:28.681 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001934753 2024-09-18T04:38:28.681 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.682 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.682 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:28.682 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585543+0000", 2024-09-18T04:38:28.682 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00076540099999999997 2024-09-18T04:38:28.682 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.682 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.682 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:28.682 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585561+0000", 2024-09-18T04:38:28.682 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8474999999999998e-05 2024-09-18T04:38:28.682 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.682 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.682 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:28.682 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585576+0000", 2024-09-18T04:38:28.682 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5427e-05 2024-09-18T04:38:28.682 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:28.683 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:28.683 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:28.683 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.683 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.683 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:28.683 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786394+0000", 2024-09-18T04:38:28.683 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 60.873615278000003, 2024-09-18T04:38:28.683 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011902085999999999, 2024-09-18T04:38:28.683 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:28.683 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:28.683 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:28.683 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:28.683 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:28.683 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 109 2024-09-18T04:38:28.683 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.683 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:28.684 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.684 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:28.684 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786394+0000", 2024-09-18T04:38:28.684 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:28.684 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.684 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.684 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:28.684 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786394+0000", 2024-09-18T04:38:28.684 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:28.684 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.684 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.684 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:28.684 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786394+0000", 2024-09-18T04:38:28.684 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:28.685 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.685 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.685 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:28.685 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786548+0000", 2024-09-18T04:38:28.685 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015463400000000001 2024-09-18T04:38:28.685 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.685 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.685 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:28.685 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786561+0000", 2024-09-18T04:38:28.685 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3093e-05 2024-09-18T04:38:28.685 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.685 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.685 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:28.685 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786571+0000", 2024-09-18T04:38:28.686 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4700000000000008e-06 2024-09-18T04:38:28.686 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.686 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.686 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:28.686 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.793367+0000", 2024-09-18T04:38:28.686 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0067966600000000004 2024-09-18T04:38:28.686 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.686 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.686 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:28.686 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.793526+0000", 2024-09-18T04:38:28.686 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015877 2024-09-18T04:38:28.686 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.686 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.687 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:28.687 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.793752+0000", 2024-09-18T04:38:28.687 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00022555200000000001 2024-09-18T04:38:28.687 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.687 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.687 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:28.687 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798125+0000", 2024-09-18T04:38:28.687 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0043733349999999999 2024-09-18T04:38:28.687 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.687 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.687 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:28.687 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798146+0000", 2024-09-18T04:38:28.687 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1063000000000001e-05 2024-09-18T04:38:28.687 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.688 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.688 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:28.688 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798232+0000", 2024-09-18T04:38:28.688 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5771000000000003e-05 2024-09-18T04:38:28.688 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.688 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.688 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:28.688 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798297+0000", 2024-09-18T04:38:28.688 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.4624999999999999e-05 2024-09-18T04:38:28.688 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:28.688 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:28.688 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:28.688 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.688 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.689 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:28.689 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786584+0000", 2024-09-18T04:38:28.689 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 60.873425199000003, 2024-09-18T04:38:28.689 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012013037000000001, 2024-09-18T04:38:28.689 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:28.689 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:28.689 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:28.689 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:28.689 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:28.689 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 110 2024-09-18T04:38:28.689 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.689 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:28.689 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.690 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:28.690 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786584+0000", 2024-09-18T04:38:28.690 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:28.690 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.690 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.690 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:28.690 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786584+0000", 2024-09-18T04:38:28.690 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:28.690 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.690 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.690 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:28.690 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786584+0000", 2024-09-18T04:38:28.690 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:28.690 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.691 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.691 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:28.691 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786590+0000", 2024-09-18T04:38:28.691 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6359999999999997e-06 2024-09-18T04:38:28.691 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.691 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.691 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:28.691 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786599+0000", 2024-09-18T04:38:28.691 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7369999999999995e-06 2024-09-18T04:38:28.691 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.691 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.691 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:28.691 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786606+0000", 2024-09-18T04:38:28.691 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1930000000000004e-06 2024-09-18T04:38:28.691 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.692 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.692 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:28.692 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.795018+0000", 2024-09-18T04:38:28.692 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0084115660000000005 2024-09-18T04:38:28.692 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.692 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.692 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:28.692 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.795184+0000", 2024-09-18T04:38:28.692 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000165833 2024-09-18T04:38:28.692 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.692 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.692 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:28.692 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.795414+0000", 2024-09-18T04:38:28.692 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00023088399999999999 2024-09-18T04:38:28.692 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.693 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.693 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:28.693 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798321+0000", 2024-09-18T04:38:28.693 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0029069539999999998 2024-09-18T04:38:28.693 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.693 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.693 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:28.693 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798404+0000", 2024-09-18T04:38:28.693 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2724000000000003e-05 2024-09-18T04:38:28.693 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.693 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.693 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:28.693 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798488+0000", 2024-09-18T04:38:28.694 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.3505999999999996e-05 2024-09-18T04:38:28.694 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.694 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.694 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:28.694 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798598+0000", 2024-09-18T04:38:28.694 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000109917 2024-09-18T04:38:28.694 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:28.694 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:28.694 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:28.694 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.694 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.694 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:28.694 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786613+0000", 2024-09-18T04:38:28.694 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 60.873396499000002, 2024-09-18T04:38:28.694 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014245760999999999, 2024-09-18T04:38:28.695 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:28.695 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:28.695 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:28.695 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:28.695 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:28.695 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 111 2024-09-18T04:38:28.695 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.695 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:28.695 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.695 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:28.695 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786613+0000", 2024-09-18T04:38:28.695 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:28.695 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.695 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.696 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:28.696 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786613+0000", 2024-09-18T04:38:28.696 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:28.696 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.696 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.696 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:28.696 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786612+0000", 2024-09-18T04:38:28.696 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:28.696 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.696 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.696 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:28.696 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786622+0000", 2024-09-18T04:38:28.696 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4099999999999997e-06 2024-09-18T04:38:28.696 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.696 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.697 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:28.697 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786630+0000", 2024-09-18T04:38:28.697 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4169999999999992e-06 2024-09-18T04:38:28.697 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.697 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.697 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:28.697 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786637+0000", 2024-09-18T04:38:28.697 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8290000000000002e-06 2024-09-18T04:38:28.697 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.697 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.697 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:28.697 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.797099+0000", 2024-09-18T04:38:28.697 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010461747 2024-09-18T04:38:28.697 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.698 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.698 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:28.698 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.797243+0000", 2024-09-18T04:38:28.698 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014388899999999999 2024-09-18T04:38:28.698 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.698 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.698 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:28.698 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.797463+0000", 2024-09-18T04:38:28.698 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000219912 2024-09-18T04:38:28.698 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.698 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.698 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:28.698 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800732+0000", 2024-09-18T04:38:28.698 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0032688719999999999 2024-09-18T04:38:28.698 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.699 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.699 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:28.699 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800751+0000", 2024-09-18T04:38:28.699 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9633999999999999e-05 2024-09-18T04:38:28.699 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.699 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.699 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:28.699 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800824+0000", 2024-09-18T04:38:28.699 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.2805999999999994e-05 2024-09-18T04:38:28.699 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.699 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.699 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:28.699 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800859+0000", 2024-09-18T04:38:28.699 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4952e-05 2024-09-18T04:38:28.699 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:28.700 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:28.700 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:28.700 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.700 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.700 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:28.700 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786649+0000", 2024-09-18T04:38:28.700 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 60.873361090000003, 2024-09-18T04:38:28.700 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.01613442, 2024-09-18T04:38:28.700 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:28.700 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:28.700 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:28.700 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:28.700 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:28.700 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 112 2024-09-18T04:38:28.700 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.701 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:28.701 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.701 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:28.701 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786649+0000", 2024-09-18T04:38:28.701 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:28.701 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.701 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.701 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:28.701 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786649+0000", 2024-09-18T04:38:28.701 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:28.702 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.702 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.702 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:28.702 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786648+0000", 2024-09-18T04:38:28.702 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:28.702 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.702 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.702 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:28.702 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786768+0000", 2024-09-18T04:38:28.702 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000120491 2024-09-18T04:38:28.702 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.702 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.702 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:28.702 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786795+0000", 2024-09-18T04:38:28.702 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6556000000000001e-05 2024-09-18T04:38:28.702 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.703 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.703 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:28.703 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786807+0000", 2024-09-18T04:38:28.703 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1739999999999999e-05 2024-09-18T04:38:28.703 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.703 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.703 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:28.703 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798683+0000", 2024-09-18T04:38:28.703 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011876070000000001 2024-09-18T04:38:28.703 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.703 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.703 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:28.703 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798867+0000", 2024-09-18T04:38:28.703 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000184103 2024-09-18T04:38:28.703 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.703 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.704 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:28.704 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.799082+0000", 2024-09-18T04:38:28.704 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00021476000000000001 2024-09-18T04:38:28.704 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.704 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.704 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:28.704 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802610+0000", 2024-09-18T04:38:28.704 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0035282209999999998 2024-09-18T04:38:28.704 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.704 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.704 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:28.704 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802706+0000", 2024-09-18T04:38:28.704 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5922999999999996e-05 2024-09-18T04:38:28.704 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.705 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.705 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:28.705 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802750+0000", 2024-09-18T04:38:28.705 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4511000000000001e-05 2024-09-18T04:38:28.705 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.705 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.705 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:28.705 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802783+0000", 2024-09-18T04:38:28.705 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2752000000000001e-05 2024-09-18T04:38:28.705 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:28.705 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:28.705 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:28.705 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.705 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.705 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:28.706 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786816+0000", 2024-09-18T04:38:28.706 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 60.873194023000003, 2024-09-18T04:38:28.706 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016197989999999999, 2024-09-18T04:38:28.706 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:28.706 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:28.706 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:28.706 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:28.706 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:28.706 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 113 2024-09-18T04:38:28.706 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.706 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:28.706 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.707 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:28.707 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786816+0000", 2024-09-18T04:38:28.707 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:28.707 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.707 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.707 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:28.707 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786816+0000", 2024-09-18T04:38:28.707 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:28.707 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.707 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.707 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:28.707 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786815+0000", 2024-09-18T04:38:28.707 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:28.707 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.707 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.708 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:28.708 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786821+0000", 2024-09-18T04:38:28.708 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9529999999999996e-06 2024-09-18T04:38:28.708 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.708 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.708 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:28.708 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786841+0000", 2024-09-18T04:38:28.708 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9743000000000001e-05 2024-09-18T04:38:28.708 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.708 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.708 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:28.708 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786848+0000", 2024-09-18T04:38:28.708 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4900000000000003e-06 2024-09-18T04:38:28.708 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.709 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.709 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:28.709 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.799747+0000", 2024-09-18T04:38:28.709 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012898902 2024-09-18T04:38:28.709 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.709 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.709 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:28.709 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.799898+0000", 2024-09-18T04:38:28.709 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000150877 2024-09-18T04:38:28.709 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.709 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.709 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:28.709 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800095+0000", 2024-09-18T04:38:28.709 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000197023 2024-09-18T04:38:28.710 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.710 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.710 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:28.710 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802829+0000", 2024-09-18T04:38:28.710 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0027343480000000002 2024-09-18T04:38:28.710 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.710 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.710 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:28.710 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802847+0000", 2024-09-18T04:38:28.710 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7283e-05 2024-09-18T04:38:28.710 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.710 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.710 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:28.710 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802974+0000", 2024-09-18T04:38:28.710 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012749199999999999 2024-09-18T04:38:28.710 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.711 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.711 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:28.711 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803014+0000", 2024-09-18T04:38:28.711 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9576000000000002e-05 2024-09-18T04:38:28.711 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:28.711 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:28.711 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:28.711 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.711 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.711 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:28.711 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:38:28.711 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 60.873108432000002, 2024-09-18T04:38:28.711 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017399218000000001, 2024-09-18T04:38:28.711 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:28.711 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:28.712 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:28.712 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:28.712 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:28.712 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 114 2024-09-18T04:38:28.712 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.712 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:28.712 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.712 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:28.712 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:38:28.712 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:28.712 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.712 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.712 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:28.713 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:38:28.713 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:28.713 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.713 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.713 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:28.713 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786900+0000", 2024-09-18T04:38:28.713 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:28.713 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.713 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.713 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:28.713 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786908+0000", 2024-09-18T04:38:28.713 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8199999999999997e-06 2024-09-18T04:38:28.713 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.713 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.714 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:28.714 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786925+0000", 2024-09-18T04:38:28.714 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6849000000000001e-05 2024-09-18T04:38:28.714 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.714 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.714 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:28.714 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786934+0000", 2024-09-18T04:38:28.714 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6299999999999993e-06 2024-09-18T04:38:28.714 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.714 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.714 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:28.714 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800926+0000", 2024-09-18T04:38:28.714 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013991632 2024-09-18T04:38:28.714 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.715 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.715 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:28.715 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801060+0000", 2024-09-18T04:38:28.715 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013378199999999999 2024-09-18T04:38:28.715 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.715 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.715 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:28.715 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801166+0000", 2024-09-18T04:38:28.715 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010607099999999999 2024-09-18T04:38:28.715 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.715 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.715 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:28.715 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803026+0000", 2024-09-18T04:38:28.715 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001860469 2024-09-18T04:38:28.715 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.716 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.716 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:28.716 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804245+0000", 2024-09-18T04:38:28.716 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0012185900000000001 2024-09-18T04:38:28.716 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.716 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.716 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:28.716 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804278+0000", 2024-09-18T04:38:28.716 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3272000000000003e-05 2024-09-18T04:38:28.716 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.716 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.716 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:28.716 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804300+0000", 2024-09-18T04:38:28.716 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2266e-05 2024-09-18T04:38:28.717 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:28.717 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:28.717 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:28.717 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.717 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.717 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:28.717 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787566+0000", 2024-09-18T04:38:28.717 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 60.872443472999997, 2024-09-18T04:38:28.717 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017124120999999999, 2024-09-18T04:38:28.717 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:28.717 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:28.717 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:28.717 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:28.717 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:28.718 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 115 2024-09-18T04:38:28.718 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.718 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:28.718 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.718 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:28.718 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787566+0000", 2024-09-18T04:38:28.718 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:28.718 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.718 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.718 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:28.718 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787566+0000", 2024-09-18T04:38:28.718 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:28.718 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.718 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.718 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:28.719 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787564+0000", 2024-09-18T04:38:28.719 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:28.719 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.719 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.719 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:28.719 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787751+0000", 2024-09-18T04:38:28.719 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000186873 2024-09-18T04:38:28.719 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.719 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.719 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:28.719 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787772+0000", 2024-09-18T04:38:28.719 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.126e-05 2024-09-18T04:38:28.719 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.719 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.720 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:28.720 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787794+0000", 2024-09-18T04:38:28.720 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1418999999999999e-05 2024-09-18T04:38:28.720 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.720 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.720 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:28.720 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801659+0000", 2024-09-18T04:38:28.720 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013865125000000001 2024-09-18T04:38:28.720 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.720 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.720 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:28.720 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801750+0000", 2024-09-18T04:38:28.720 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.1427000000000004e-05 2024-09-18T04:38:28.720 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.720 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.721 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:28.721 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801894+0000", 2024-09-18T04:38:28.721 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000143514 2024-09-18T04:38:28.721 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.721 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.721 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:28.721 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804313+0000", 2024-09-18T04:38:28.721 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0024190879999999998 2024-09-18T04:38:28.721 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.721 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.721 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:28.721 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804644+0000", 2024-09-18T04:38:28.721 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00033100799999999999 2024-09-18T04:38:28.722 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.722 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.722 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:28.722 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804672+0000", 2024-09-18T04:38:28.722 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7988e-05 2024-09-18T04:38:28.722 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.722 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.722 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:28.722 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804690+0000", 2024-09-18T04:38:28.722 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8298999999999999e-05 2024-09-18T04:38:28.722 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:28.722 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:28.722 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:28.722 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.723 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.723 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:28.723 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:38:28.723 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 60.872195533999999, 2024-09-18T04:38:28.723 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017521662, 2024-09-18T04:38:28.723 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:28.723 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:28.723 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:28.723 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:28.723 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:28.723 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 116 2024-09-18T04:38:28.723 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.723 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:28.723 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.723 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:28.724 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:38:28.724 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:28.724 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.724 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.724 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:28.724 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:38:28.724 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:28.724 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.724 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.724 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:28.724 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787813+0000", 2024-09-18T04:38:28.724 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:28.724 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.724 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.725 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:28.725 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787824+0000", 2024-09-18T04:38:28.725 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1157e-05 2024-09-18T04:38:28.725 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.725 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.725 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:28.725 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787840+0000", 2024-09-18T04:38:28.725 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6269e-05 2024-09-18T04:38:28.725 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.725 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.725 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:28.725 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787853+0000", 2024-09-18T04:38:28.725 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3156000000000001e-05 2024-09-18T04:38:28.726 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.726 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.726 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:28.726 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803110+0000", 2024-09-18T04:38:28.726 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015256736 2024-09-18T04:38:28.726 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.726 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.726 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:28.726 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803197+0000", 2024-09-18T04:38:28.726 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7195999999999997e-05 2024-09-18T04:38:28.726 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.726 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.727 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:28.727 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803308+0000", 2024-09-18T04:38:28.727 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111005 2024-09-18T04:38:28.727 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.727 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.727 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:28.727 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804316+0000", 2024-09-18T04:38:28.727 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010073949999999999 2024-09-18T04:38:28.727 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.727 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.727 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:28.727 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805232+0000", 2024-09-18T04:38:28.727 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091636300000000001 2024-09-18T04:38:28.727 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.728 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.728 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:28.728 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805277+0000", 2024-09-18T04:38:28.728 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4864000000000003e-05 2024-09-18T04:38:28.728 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.728 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.728 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:28.728 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805336+0000", 2024-09-18T04:38:28.728 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8968000000000002e-05 2024-09-18T04:38:28.728 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:28.728 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:28.728 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:28.728 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.728 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.728 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:28.729 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:38:28.729 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 60.872143479000002, 2024-09-18T04:38:28.729 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017983453, 2024-09-18T04:38:28.729 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:28.729 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:28.729 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:28.729 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:28.729 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:28.729 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 117 2024-09-18T04:38:28.729 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.729 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:28.729 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.729 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:28.729 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:38:28.729 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:28.730 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.730 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.730 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:28.730 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:38:28.730 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:28.730 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.730 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.730 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:28.730 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787865+0000", 2024-09-18T04:38:28.730 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:28.730 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.730 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.730 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:28.730 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787889+0000", 2024-09-18T04:38:28.730 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4244999999999999e-05 2024-09-18T04:38:28.730 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.731 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.731 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:28.731 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787905+0000", 2024-09-18T04:38:28.731 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5739999999999998e-05 2024-09-18T04:38:28.731 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.731 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.731 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:28.731 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787918+0000", 2024-09-18T04:38:28.731 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.252e-05 2024-09-18T04:38:28.731 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.732 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.732 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:28.732 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803693+0000", 2024-09-18T04:38:28.732 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015774872999999998 2024-09-18T04:38:28.732 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.732 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.732 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:28.732 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803775+0000", 2024-09-18T04:38:28.732 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2761000000000003e-05 2024-09-18T04:38:28.732 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.732 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.732 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:28.732 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803887+0000", 2024-09-18T04:38:28.732 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111708 2024-09-18T04:38:28.733 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.733 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.733 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:28.733 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804823+0000", 2024-09-18T04:38:28.733 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00093625100000000003 2024-09-18T04:38:28.733 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.733 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.733 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:28.733 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805804+0000", 2024-09-18T04:38:28.733 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00098114699999999992 2024-09-18T04:38:28.733 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.733 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.733 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:28.733 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805831+0000", 2024-09-18T04:38:28.734 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6401999999999999e-05 2024-09-18T04:38:28.734 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.734 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.734 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:28.734 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805850+0000", 2024-09-18T04:38:28.734 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8862e-05 2024-09-18T04:38:28.734 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:28.734 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:28.734 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:28.734 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.734 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.734 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:28.734 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.230830+0000", 2024-09-18T04:38:28.734 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 60.429179599999998, 2024-09-18T04:38:28.735 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011537146999999999, 2024-09-18T04:38:28.735 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:28.735 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:28.735 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:28.735 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:28.735 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:28.735 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 135 2024-09-18T04:38:28.735 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.735 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:28.735 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.735 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:28.735 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230830+0000", 2024-09-18T04:38:28.735 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:28.735 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.736 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.736 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:28.736 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230830+0000", 2024-09-18T04:38:28.736 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:28.736 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.736 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.736 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:28.736 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230829+0000", 2024-09-18T04:38:28.736 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:28.736 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.736 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.736 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:28.736 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230835+0000", 2024-09-18T04:38:28.736 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8660000000000002e-06 2024-09-18T04:38:28.736 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.737 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.737 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:28.737 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230844+0000", 2024-09-18T04:38:28.737 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9339999999999995e-06 2024-09-18T04:38:28.737 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.737 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.737 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:28.737 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230853+0000", 2024-09-18T04:38:28.737 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4419999999999998e-06 2024-09-18T04:38:28.737 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.737 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.737 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:28.737 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.237658+0000", 2024-09-18T04:38:28.737 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0068055310000000001 2024-09-18T04:38:28.737 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.737 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.738 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:28.738 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.237837+0000", 2024-09-18T04:38:28.738 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017943200000000001 2024-09-18T04:38:28.738 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.738 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.738 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:28.738 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.238112+0000", 2024-09-18T04:38:28.738 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00027488600000000001 2024-09-18T04:38:28.738 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.738 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.738 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:28.738 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.240505+0000", 2024-09-18T04:38:28.738 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0023930539999999999 2024-09-18T04:38:28.738 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.738 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.739 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:28.739 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242157+0000", 2024-09-18T04:38:28.739 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001651135 2024-09-18T04:38:28.739 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.739 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.739 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:28.739 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242281+0000", 2024-09-18T04:38:28.739 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012451300000000001 2024-09-18T04:38:28.739 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.739 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.739 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:28.739 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242367+0000", 2024-09-18T04:38:28.739 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6178000000000001e-05 2024-09-18T04:38:28.739 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:28.740 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:28.740 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:28.740 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.740 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.740 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:28.740 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.230858+0000", 2024-09-18T04:38:28.740 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 60.429151711999999, 2024-09-18T04:38:28.740 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013005821000000001, 2024-09-18T04:38:28.740 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:28.740 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:28.740 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:28.740 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:28.740 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:28.740 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 136 2024-09-18T04:38:28.741 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.741 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:28.741 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.741 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:28.741 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230858+0000", 2024-09-18T04:38:28.741 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:28.741 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.741 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.741 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:28.741 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230858+0000", 2024-09-18T04:38:28.741 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:28.741 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.741 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.741 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:28.742 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230858+0000", 2024-09-18T04:38:28.742 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:28.742 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.742 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.742 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:28.742 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230861+0000", 2024-09-18T04:38:28.742 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7459999999999998e-06 2024-09-18T04:38:28.742 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.742 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.742 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:28.742 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230877+0000", 2024-09-18T04:38:28.742 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5979000000000001e-05 2024-09-18T04:38:28.742 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.742 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.742 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:28.743 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230885+0000", 2024-09-18T04:38:28.743 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.5979999999999999e-06 2024-09-18T04:38:28.743 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.743 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.743 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:28.743 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.239023+0000", 2024-09-18T04:38:28.743 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0081380710000000002 2024-09-18T04:38:28.743 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.743 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.743 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:28.743 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.239216+0000", 2024-09-18T04:38:28.743 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019274799999999999 2024-09-18T04:38:28.743 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.743 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.743 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:28.744 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.239417+0000", 2024-09-18T04:38:28.744 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020093300000000001 2024-09-18T04:38:28.744 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.744 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.744 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:28.744 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242494+0000", 2024-09-18T04:38:28.744 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003077177 2024-09-18T04:38:28.744 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.744 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.744 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:28.744 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.243668+0000", 2024-09-18T04:38:28.744 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001174027 2024-09-18T04:38:28.744 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.745 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.745 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:28.745 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.243777+0000", 2024-09-18T04:38:28.745 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010955600000000001 2024-09-18T04:38:28.745 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.745 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.745 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:28.745 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.243864+0000", 2024-09-18T04:38:28.745 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6441999999999997e-05 2024-09-18T04:38:28.745 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:28.745 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:28.745 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:28.746 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.746 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.746 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:28.746 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.230901+0000", 2024-09-18T04:38:28.746 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 60.429109138999998, 2024-09-18T04:38:28.746 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014120882, 2024-09-18T04:38:28.746 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:28.746 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:28.746 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:28.746 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:28.746 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:28.746 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 137 2024-09-18T04:38:28.746 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.746 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:28.747 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.747 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:28.747 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230901+0000", 2024-09-18T04:38:28.747 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:28.747 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.747 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.747 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:28.747 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230901+0000", 2024-09-18T04:38:28.747 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:28.747 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.747 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.747 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:28.747 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230900+0000", 2024-09-18T04:38:28.748 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:28.748 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.748 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.748 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:28.748 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231002+0000", 2024-09-18T04:38:28.748 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000102392 2024-09-18T04:38:28.748 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.748 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.748 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:28.748 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231029+0000", 2024-09-18T04:38:28.748 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7166999999999999e-05 2024-09-18T04:38:28.748 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.748 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.748 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:28.749 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231036+0000", 2024-09-18T04:38:28.749 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7259999999999996e-06 2024-09-18T04:38:28.749 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.749 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.749 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:28.749 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.240562+0000", 2024-09-18T04:38:28.749 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0095257970000000008 2024-09-18T04:38:28.749 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.749 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.749 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:28.749 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.240730+0000", 2024-09-18T04:38:28.749 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016767099999999999 2024-09-18T04:38:28.749 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.749 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.749 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:28.749 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.240927+0000", 2024-09-18T04:38:28.750 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019711899999999999 2024-09-18T04:38:28.750 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.750 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.750 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:28.750 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244885+0000", 2024-09-18T04:38:28.750 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0039577340000000001 2024-09-18T04:38:28.750 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.750 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.750 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:28.750 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244899+0000", 2024-09-18T04:38:28.750 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4403000000000001e-05 2024-09-18T04:38:28.750 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.750 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.750 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:28.750 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244973+0000", 2024-09-18T04:38:28.751 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4177999999999994e-05 2024-09-18T04:38:28.751 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.751 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.751 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:28.751 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.245021+0000", 2024-09-18T04:38:28.751 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8344999999999999e-05 2024-09-18T04:38:28.751 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:28.751 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:28.751 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:28.751 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.751 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.751 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:28.751 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:38:28.751 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 60.428966348000003, 2024-09-18T04:38:28.752 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016528505999999998, 2024-09-18T04:38:28.752 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:28.752 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:28.752 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:28.752 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:28.752 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:28.752 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 138 2024-09-18T04:38:28.752 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.752 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:28.752 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.752 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:28.752 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:38:28.752 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:28.752 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.752 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.753 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:28.753 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:38:28.753 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:28.753 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.753 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.753 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:28.753 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:38:28.753 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:28.753 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.753 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.753 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:28.753 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231047+0000", 2024-09-18T04:38:28.753 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5399999999999997e-06 2024-09-18T04:38:28.753 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.754 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.754 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:28.754 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231068+0000", 2024-09-18T04:38:28.754 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0769000000000001e-05 2024-09-18T04:38:28.754 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.754 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.754 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:28.754 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231074+0000", 2024-09-18T04:38:28.754 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6180000000000003e-06 2024-09-18T04:38:28.754 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.754 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.754 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:28.754 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242556+0000", 2024-09-18T04:38:28.754 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011482654 2024-09-18T04:38:28.754 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.755 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.755 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:28.755 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242666+0000", 2024-09-18T04:38:28.755 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011003200000000001 2024-09-18T04:38:28.755 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.755 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.755 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:28.755 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242924+0000", 2024-09-18T04:38:28.755 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00025785899999999999 2024-09-18T04:38:28.755 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.755 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.755 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:28.755 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247263+0000", 2024-09-18T04:38:28.755 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0043387649999999996 2024-09-18T04:38:28.756 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.756 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.756 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:28.756 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247412+0000", 2024-09-18T04:38:28.756 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000149276 2024-09-18T04:38:28.756 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.756 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.756 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:28.756 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247522+0000", 2024-09-18T04:38:28.756 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000109788 2024-09-18T04:38:28.756 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.756 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.756 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:28.756 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247572+0000", 2024-09-18T04:38:28.756 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9808999999999997e-05 2024-09-18T04:38:28.756 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:28.757 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:28.757 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:28.757 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.757 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.757 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:28.757 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231083+0000", 2024-09-18T04:38:28.757 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 60.428926885000003, 2024-09-18T04:38:28.757 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0166731, 2024-09-18T04:38:28.757 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:28.757 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:28.757 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:28.757 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:28.757 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:28.758 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 139 2024-09-18T04:38:28.758 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.758 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:28.758 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.758 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:28.758 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231083+0000", 2024-09-18T04:38:28.758 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:28.758 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.758 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.758 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:28.758 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231083+0000", 2024-09-18T04:38:28.758 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:28.758 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.758 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.758 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:28.758 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231082+0000", 2024-09-18T04:38:28.758 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:28.759 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.759 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.759 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:28.759 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231090+0000", 2024-09-18T04:38:28.759 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1280000000000008e-06 2024-09-18T04:38:28.759 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.759 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.759 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:28.759 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231097+0000", 2024-09-18T04:38:28.759 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5119999999999997e-06 2024-09-18T04:38:28.759 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.759 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.759 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:28.760 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231106+0000", 2024-09-18T04:38:28.760 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9230000000000008e-06 2024-09-18T04:38:28.760 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.760 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.760 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:28.760 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.243977+0000", 2024-09-18T04:38:28.760 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012870754 2024-09-18T04:38:28.760 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.760 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.760 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:28.760 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244112+0000", 2024-09-18T04:38:28.760 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000135575 2024-09-18T04:38:28.760 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.760 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.760 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:28.761 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244265+0000", 2024-09-18T04:38:28.761 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015279900000000001 2024-09-18T04:38:28.761 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.761 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.761 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:28.761 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247649+0000", 2024-09-18T04:38:28.761 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003383896 2024-09-18T04:38:28.761 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.761 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.761 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:28.761 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247656+0000", 2024-09-18T04:38:28.761 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8920000000000003e-06 2024-09-18T04:38:28.761 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.762 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.762 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:28.762 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247692+0000", 2024-09-18T04:38:28.762 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6470000000000001e-05 2024-09-18T04:38:28.762 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.762 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.762 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:28.762 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247756+0000", 2024-09-18T04:38:28.762 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3676999999999994e-05 2024-09-18T04:38:28.762 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:28.762 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:28.762 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:28.762 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.762 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.762 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:28.763 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231190+0000", 2024-09-18T04:38:28.763 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 60.428819959000002, 2024-09-18T04:38:28.763 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016793556000000001, 2024-09-18T04:38:28.763 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:28.763 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:28.763 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:28.763 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:28.763 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:28.763 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 140 2024-09-18T04:38:28.763 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.763 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:28.763 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.763 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:28.763 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231190+0000", 2024-09-18T04:38:28.764 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:28.764 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.764 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.764 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:28.764 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231190+0000", 2024-09-18T04:38:28.764 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:28.764 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.764 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.764 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:28.764 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231189+0000", 2024-09-18T04:38:28.764 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:28.764 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.764 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.764 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:28.765 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231273+0000", 2024-09-18T04:38:28.765 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4445999999999998e-05 2024-09-18T04:38:28.765 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.765 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.765 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:28.765 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231282+0000", 2024-09-18T04:38:28.765 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8859999999999993e-06 2024-09-18T04:38:28.765 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.765 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.765 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:28.765 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231288+0000", 2024-09-18T04:38:28.765 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9819999999999996e-06 2024-09-18T04:38:28.765 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.765 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.766 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:28.766 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.245119+0000", 2024-09-18T04:38:28.766 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.01383121 2024-09-18T04:38:28.766 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.766 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.766 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:28.766 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.245247+0000", 2024-09-18T04:38:28.766 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012763299999999999 2024-09-18T04:38:28.766 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.766 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.766 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:28.766 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.245401+0000", 2024-09-18T04:38:28.766 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015399200000000001 2024-09-18T04:38:28.766 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.767 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.767 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:28.767 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247766+0000", 2024-09-18T04:38:28.767 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0023649510000000001 2024-09-18T04:38:28.767 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.767 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.767 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:28.767 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247904+0000", 2024-09-18T04:38:28.767 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013836499999999999 2024-09-18T04:38:28.767 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.767 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.767 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:28.767 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247950+0000", 2024-09-18T04:38:28.767 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5201000000000003e-05 2024-09-18T04:38:28.768 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.768 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.768 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:28.768 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247983+0000", 2024-09-18T04:38:28.768 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3686000000000002e-05 2024-09-18T04:38:28.768 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:28.768 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:28.768 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:28.768 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.768 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.768 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:28.768 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:38:28.768 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 60.428709216000001, 2024-09-18T04:38:28.768 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017344024, 2024-09-18T04:38:28.769 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:28.769 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:28.769 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:28.769 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:28.769 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:28.769 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 141 2024-09-18T04:38:28.769 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.769 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:28.769 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.769 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:28.769 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:38:28.769 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:28.769 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.769 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.770 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:28.770 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:38:28.770 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:28.770 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.770 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.770 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:28.770 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:38:28.770 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:28.770 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.770 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.770 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:28.770 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231309+0000", 2024-09-18T04:38:28.770 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5419999999999994e-06 2024-09-18T04:38:28.770 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.771 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.771 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:28.771 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231323+0000", 2024-09-18T04:38:28.771 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3108000000000001e-05 2024-09-18T04:38:28.771 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.771 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.771 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:28.771 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231340+0000", 2024-09-18T04:38:28.771 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.696e-05 2024-09-18T04:38:28.771 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.771 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.771 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:28.771 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.246143+0000", 2024-09-18T04:38:28.771 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014803320999999999 2024-09-18T04:38:28.772 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.772 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.772 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:28.772 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.246323+0000", 2024-09-18T04:38:28.772 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00018061600000000001 2024-09-18T04:38:28.772 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.772 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.772 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:28.772 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.246530+0000", 2024-09-18T04:38:28.772 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020625 2024-09-18T04:38:28.772 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.772 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.772 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:28.773 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248002+0000", 2024-09-18T04:38:28.773 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0014719379999999999 2024-09-18T04:38:28.773 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.773 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.773 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:28.773 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248582+0000", 2024-09-18T04:38:28.773 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00058020599999999999 2024-09-18T04:38:28.773 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.773 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.773 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:28.773 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248618+0000", 2024-09-18T04:38:28.773 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6559e-05 2024-09-18T04:38:28.773 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.773 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.774 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:28.774 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248645+0000", 2024-09-18T04:38:28.774 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6080000000000001e-05 2024-09-18T04:38:28.774 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:28.774 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:28.774 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:28.774 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.774 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.774 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-18T04:38:28.774 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.878463+0000", 2024-09-18T04:38:28.774 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 59.781547181999997, 2024-09-18T04:38:28.774 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010882697, 2024-09-18T04:38:28.774 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:28.774 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:28.774 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:28.775 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:28.775 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:28.775 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 159 2024-09-18T04:38:28.775 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.775 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:28.775 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.775 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:28.775 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878463+0000", 2024-09-18T04:38:28.775 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:28.775 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.775 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.775 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:28.775 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878463+0000", 2024-09-18T04:38:28.775 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:28.775 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.776 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.776 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:28.776 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878462+0000", 2024-09-18T04:38:28.776 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:28.776 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.776 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.776 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:28.776 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878467+0000", 2024-09-18T04:38:28.776 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4099999999999999e-06 2024-09-18T04:38:28.776 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.776 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.776 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:28.776 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878490+0000", 2024-09-18T04:38:28.776 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3159999999999998e-05 2024-09-18T04:38:28.777 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.777 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.777 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:28.777 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878511+0000", 2024-09-18T04:38:28.777 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1092000000000001e-05 2024-09-18T04:38:28.777 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.777 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.777 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:28.777 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.887163+0000", 2024-09-18T04:38:28.777 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.008651845 2024-09-18T04:38:28.777 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.777 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.777 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:28.777 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.887273+0000", 2024-09-18T04:38:28.778 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000109823 2024-09-18T04:38:28.778 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.778 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.778 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:28.778 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.887441+0000", 2024-09-18T04:38:28.778 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016777599999999999 2024-09-18T04:38:28.778 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.778 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.778 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:28.778 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.888634+0000", 2024-09-18T04:38:28.778 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001192839 2024-09-18T04:38:28.778 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.778 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.778 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:28.778 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.889153+0000", 2024-09-18T04:38:28.778 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00051953700000000004 2024-09-18T04:38:28.779 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.779 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.779 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:28.779 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.889234+0000", 2024-09-18T04:38:28.779 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0802000000000003e-05 2024-09-18T04:38:28.779 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:28.779 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:28.779 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:28.779 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.889345+0000", 2024-09-18T04:38:28.779 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011112000000000001 2024-09-18T04:38:28.779 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:28.779 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:28.779 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:28.779 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:28.779 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:28.779 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:28.781 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:38:28.781 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:28.778+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:28.871 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.0 to 1 (10000) 2024-09-18T04:38:28.877 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-18T04:38:28.884 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-18T04:38:28.885 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:28.878+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:28.897 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:28.898 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:28.898 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:38:28.898 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:28.907 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:38:28.982 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:28.978+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:29.009 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:29.009 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:29.009 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:38:29.009 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:38:29.009 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:29.019 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:38:29.084 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:29.078+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:29.125 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:29.126 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:38:29.126 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:38:29.126 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [] 2024-09-18T04:38:29.126 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:29.136 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:38:29.185 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:29.178+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:29.238 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:29.238 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:29.238 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:38:29.238 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:29.248 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:38:29.286 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:29.282+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:29.351 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:29.351 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:29.351 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:38:29.351 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:38:29.351 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:29.361 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:38:29.387 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:29.382+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:29.446 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T04:38:29.490 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:29.486+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:29.512 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:29.512 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:38:29.512 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:38:29.512 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:38:29.512 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.512 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-18T04:38:29.512 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:38:29.512 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 55.039550527999999, 2024-09-18T04:38:29.512 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0095781189999999995, 2024-09-18T04:38:29.513 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:29.513 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:29.513 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:29.513 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:38:29.513 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:38:29.513 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 1 2024-09-18T04:38:29.513 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.513 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:29.513 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.513 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:29.513 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:38:29.513 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:29.513 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.514 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.514 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:29.514 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:38:29.514 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:29.514 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.514 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.514 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:29.514 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469862+0000", 2024-09-18T04:38:29.514 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:29.514 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.514 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.514 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:29.514 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469879+0000", 2024-09-18T04:38:29.514 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7431999999999998e-05 2024-09-18T04:38:29.515 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.515 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.515 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:29.515 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469896+0000", 2024-09-18T04:38:29.515 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6742999999999999e-05 2024-09-18T04:38:29.515 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.515 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.515 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:29.515 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469915+0000", 2024-09-18T04:38:29.515 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8991000000000002e-05 2024-09-18T04:38:29.515 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.515 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.515 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:29.516 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470397+0000", 2024-09-18T04:38:29.516 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048167799999999998 2024-09-18T04:38:29.516 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.516 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.516 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-09-18T04:38:29.516 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470412+0000", 2024-09-18T04:38:29.516 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5526e-05 2024-09-18T04:38:29.516 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.516 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.516 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:29.516 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473279+0000", 2024-09-18T04:38:29.516 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0028667469999999998 2024-09-18T04:38:29.516 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.517 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.517 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:29.517 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473441+0000", 2024-09-18T04:38:29.517 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000162082 2024-09-18T04:38:29.517 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.517 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.517 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:38:29.517 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473573+0000", 2024-09-18T04:38:29.517 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000132032 2024-09-18T04:38:29.517 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.517 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.517 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:29.518 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474826+0000", 2024-09-18T04:38:29.518 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001252717 2024-09-18T04:38:29.518 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.518 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.518 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:29.518 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479307+0000", 2024-09-18T04:38:29.518 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044813220000000003 2024-09-18T04:38:29.518 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.518 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.518 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:29.518 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479366+0000", 2024-09-18T04:38:29.518 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9024999999999998e-05 2024-09-18T04:38:29.518 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.519 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.519 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:29.519 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479443+0000", 2024-09-18T04:38:29.519 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6339000000000002e-05 2024-09-18T04:38:29.519 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:29.519 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:29.519 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:29.519 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.519 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.519 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.4214.0:1 2.3 e18/17 ondisk, result = 0)", 2024-09-18T04:38:29.519 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.474724+0000", 2024-09-18T04:38:29.519 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 55.034691164000002, 2024-09-18T04:38:29.519 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000105358, 2024-09-18T04:38:29.519 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:29.520 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:38:29.520 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:29.520 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.520 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:29.520 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474724+0000", 2024-09-18T04:38:29.520 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:29.520 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.520 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.520 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:29.520 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474724+0000", 2024-09-18T04:38:29.520 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:29.520 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.520 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.521 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:29.521 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474723+0000", 2024-09-18T04:38:29.521 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:29.521 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.521 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.521 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:29.521 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474731+0000", 2024-09-18T04:38:29.521 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2339999999999994e-06 2024-09-18T04:38:29.521 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.521 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.521 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:29.521 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474748+0000", 2024-09-18T04:38:29.521 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6603e-05 2024-09-18T04:38:29.522 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.522 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.522 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:29.522 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474760+0000", 2024-09-18T04:38:29.522 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2215000000000001e-05 2024-09-18T04:38:29.522 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.522 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.522 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:29.522 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474814+0000", 2024-09-18T04:38:29.522 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4267000000000001e-05 2024-09-18T04:38:29.522 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.522 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.522 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:29.522 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474821+0000", 2024-09-18T04:38:29.523 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0929999999999997e-06 2024-09-18T04:38:29.523 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.523 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.523 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:29.523 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474829+0000", 2024-09-18T04:38:29.523 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1200000000000002e-06 2024-09-18T04:38:29.523 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:29.523 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:29.523 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:29.523 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.523 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.523 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18)", 2024-09-18T04:38:29.523 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.482346+0000", 2024-09-18T04:38:29.523 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 55.027068731999996, 2024-09-18T04:38:29.524 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020435599999999999, 2024-09-18T04:38:29.524 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:29.524 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-09-18T04:38:29.524 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:29.524 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:38:29.524 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:38:29.524 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 2 2024-09-18T04:38:29.524 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.524 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:29.524 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.524 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:29.524 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482346+0000", 2024-09-18T04:38:29.524 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:29.524 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.525 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.525 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:29.525 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482346+0000", 2024-09-18T04:38:29.525 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:29.525 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.525 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.525 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:29.525 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482345+0000", 2024-09-18T04:38:29.525 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:29.525 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.525 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.525 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:29.525 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482360+0000", 2024-09-18T04:38:29.525 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4854999999999999e-05 2024-09-18T04:38:29.526 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.526 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.526 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:29.526 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482377+0000", 2024-09-18T04:38:29.526 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7785999999999999e-05 2024-09-18T04:38:29.526 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.526 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.526 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:29.526 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482387+0000", 2024-09-18T04:38:29.526 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6800000000000005e-06 2024-09-18T04:38:29.526 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.526 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.526 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:29.527 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482441+0000", 2024-09-18T04:38:29.527 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4046999999999999e-05 2024-09-18T04:38:29.527 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.527 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.527 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:29.527 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482551+0000", 2024-09-18T04:38:29.527 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000109468 2024-09-18T04:38:29.527 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:29.527 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:29.527 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:29.527 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.527 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.527 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18)", 2024-09-18T04:38:29.528 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.482853+0000", 2024-09-18T04:38:29.528 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 55.026562114000001, 2024-09-18T04:38:29.528 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00029815799999999997, 2024-09-18T04:38:29.528 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:29.528 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-09-18T04:38:29.528 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:29.528 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:38:29.528 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:38:29.528 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 3 2024-09-18T04:38:29.528 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.529 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:29.529 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.529 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:29.529 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482853+0000", 2024-09-18T04:38:29.529 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:29.529 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.529 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.529 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:29.529 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482853+0000", 2024-09-18T04:38:29.529 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:29.529 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.529 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.529 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:29.529 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482852+0000", 2024-09-18T04:38:29.530 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:29.530 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.530 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.530 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:29.530 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482859+0000", 2024-09-18T04:38:29.530 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0380000000000002e-06 2024-09-18T04:38:29.530 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.530 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.530 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:29.530 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482880+0000", 2024-09-18T04:38:29.530 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1243e-05 2024-09-18T04:38:29.530 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.530 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.531 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:29.531 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482896+0000", 2024-09-18T04:38:29.531 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5977e-05 2024-09-18T04:38:29.531 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.531 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.531 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:29.531 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482970+0000", 2024-09-18T04:38:29.531 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4603000000000004e-05 2024-09-18T04:38:29.531 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.531 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.531 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:29.531 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.483151+0000", 2024-09-18T04:38:29.531 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00018059799999999999 2024-09-18T04:38:29.531 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:29.532 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:29.532 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:29.532 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.532 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.532 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-18T04:38:29.532 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:35.468698+0000", 2024-09-18T04:38:29.532 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 54.040717276999999, 2024-09-18T04:38:29.532 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0039954129999999997, 2024-09-18T04:38:29.532 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:29.532 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:29.532 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:29.532 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:38:29.532 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:38:29.533 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 5 2024-09-18T04:38:29.533 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.533 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:29.533 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.533 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:29.533 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468698+0000", 2024-09-18T04:38:29.533 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:29.533 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.533 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.533 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:29.533 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468698+0000", 2024-09-18T04:38:29.533 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:29.533 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.534 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.534 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:29.534 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468696+0000", 2024-09-18T04:38:29.534 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:29.534 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.534 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.534 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:29.534 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468714+0000", 2024-09-18T04:38:29.534 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8743e-05 2024-09-18T04:38:29.534 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.534 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.534 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:29.534 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468748+0000", 2024-09-18T04:38:29.535 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3339e-05 2024-09-18T04:38:29.535 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.535 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.535 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:29.535 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468771+0000", 2024-09-18T04:38:29.535 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3119999999999999e-05 2024-09-18T04:38:29.535 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.535 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.535 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:29.535 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468922+0000", 2024-09-18T04:38:29.535 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000151383 2024-09-18T04:38:29.535 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.535 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.536 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:29.536 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.469278+0000", 2024-09-18T04:38:29.536 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00035628599999999998 2024-09-18T04:38:29.536 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.536 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.536 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:29.536 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.469629+0000", 2024-09-18T04:38:29.536 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00035014300000000003 2024-09-18T04:38:29.536 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.536 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.536 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:29.536 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.471620+0000", 2024-09-18T04:38:29.536 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001991278 2024-09-18T04:38:29.536 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.537 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.537 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:29.537 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472440+0000", 2024-09-18T04:38:29.537 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00081998599999999998 2024-09-18T04:38:29.537 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.537 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.537 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:29.537 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472592+0000", 2024-09-18T04:38:29.537 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015227400000000001 2024-09-18T04:38:29.537 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.537 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.537 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:29.537 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472693+0000", 2024-09-18T04:38:29.538 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000101074 2024-09-18T04:38:29.538 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:29.538 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:29.538 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:29.538 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.538 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.538 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.4214.0:5 2.2 e18/17 ondisk, result = 0)", 2024-09-18T04:38:29.538 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:35.472226+0000", 2024-09-18T04:38:29.538 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 54.037189376999997, 2024-09-18T04:38:29.538 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048505299999999999, 2024-09-18T04:38:29.538 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:29.538 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:38:29.539 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:29.539 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.539 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:29.539 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472226+0000", 2024-09-18T04:38:29.539 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:29.539 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.539 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.539 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:29.539 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472226+0000", 2024-09-18T04:38:29.539 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:29.539 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.539 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.539 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:29.540 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472224+0000", 2024-09-18T04:38:29.540 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:29.540 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.540 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.540 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:29.540 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472238+0000", 2024-09-18T04:38:29.540 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.468e-05 2024-09-18T04:38:29.540 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.540 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.540 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:29.540 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472272+0000", 2024-09-18T04:38:29.540 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3586e-05 2024-09-18T04:38:29.540 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.540 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.541 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:29.541 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472297+0000", 2024-09-18T04:38:29.541 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5282999999999998e-05 2024-09-18T04:38:29.541 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.541 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.541 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:29.541 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472396+0000", 2024-09-18T04:38:29.541 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.8394999999999996e-05 2024-09-18T04:38:29.541 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.541 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.541 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:29.541 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472421+0000", 2024-09-18T04:38:29.541 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5479000000000001e-05 2024-09-18T04:38:29.542 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.542 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.542 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:29.542 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472711+0000", 2024-09-18T04:38:29.542 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000289714 2024-09-18T04:38:29.542 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:29.542 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:29.542 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:29.542 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.542 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.542 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-18T04:38:29.542 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:38:29.542 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 53.026542108000001, 2024-09-18T04:38:29.542 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082699560000000002, 2024-09-18T04:38:29.543 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:29.543 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:29.543 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:29.543 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:38:29.543 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:38:29.543 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 7 2024-09-18T04:38:29.543 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.543 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:29.543 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.543 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:29.543 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:38:29.543 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:29.543 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.543 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.543 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:29.544 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:38:29.544 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:29.544 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.544 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.544 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:29.544 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482870+0000", 2024-09-18T04:38:29.544 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999967 2024-09-18T04:38:29.544 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.544 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.544 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:29.544 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482889+0000", 2024-09-18T04:38:29.544 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.965e-05 2024-09-18T04:38:29.544 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.544 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.545 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:29.545 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482938+0000", 2024-09-18T04:38:29.545 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8548999999999999e-05 2024-09-18T04:38:29.545 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.545 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.545 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:29.545 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482963+0000", 2024-09-18T04:38:29.545 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5075999999999998e-05 2024-09-18T04:38:29.545 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.545 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.545 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:29.545 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483121+0000", 2024-09-18T04:38:29.545 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015819400000000001 2024-09-18T04:38:29.545 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.546 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.546 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op must wait for map", 2024-09-18T04:38:29.546 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483138+0000", 2024-09-18T04:38:29.546 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7068999999999999e-05 2024-09-18T04:38:29.546 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.546 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.546 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:29.546 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488666+0000", 2024-09-18T04:38:29.546 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0055282049999999996 2024-09-18T04:38:29.546 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.546 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.546 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:29.546 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488783+0000", 2024-09-18T04:38:29.546 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000116737 2024-09-18T04:38:29.547 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.547 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.547 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:29.547 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488991+0000", 2024-09-18T04:38:29.547 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020781400000000001 2024-09-18T04:38:29.547 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.547 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.547 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:29.547 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490554+0000", 2024-09-18T04:38:29.547 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0015628370000000001 2024-09-18T04:38:29.547 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.547 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.547 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:29.547 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491019+0000", 2024-09-18T04:38:29.548 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000465608 2024-09-18T04:38:29.548 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.548 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.548 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:29.548 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491087+0000", 2024-09-18T04:38:29.548 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7988e-05 2024-09-18T04:38:29.548 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.548 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.548 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:29.548 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491143+0000", 2024-09-18T04:38:29.548 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5538999999999998e-05 2024-09-18T04:38:29.548 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:29.548 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:29.549 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:29.549 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.549 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.549 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.4214.0:7 2.2 e19/17 ondisk, result = 0)", 2024-09-18T04:38:29.549 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:36.490861+0000", 2024-09-18T04:38:29.549 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 53.018554485999999, 2024-09-18T04:38:29.549 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00029639400000000001, 2024-09-18T04:38:29.549 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:29.549 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:38:29.549 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:29.549 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.549 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:29.549 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490861+0000", 2024-09-18T04:38:29.549 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:29.550 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.550 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.550 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:29.550 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490861+0000", 2024-09-18T04:38:29.550 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:29.550 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.550 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.550 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:29.550 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490859+0000", 2024-09-18T04:38:29.550 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:29.550 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.550 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.550 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:29.550 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490867+0000", 2024-09-18T04:38:29.550 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.3089999999999996e-06 2024-09-18T04:38:29.551 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.551 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.551 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:29.551 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490909+0000", 2024-09-18T04:38:29.551 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1919000000000003e-05 2024-09-18T04:38:29.551 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.551 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.551 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:29.551 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490922+0000", 2024-09-18T04:38:29.551 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2836999999999999e-05 2024-09-18T04:38:29.551 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.551 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.551 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:29.552 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491003+0000", 2024-09-18T04:38:29.552 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0681999999999998e-05 2024-09-18T04:38:29.552 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.552 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.552 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:29.552 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491013+0000", 2024-09-18T04:38:29.552 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0220999999999999e-05 2024-09-18T04:38:29.552 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.552 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.552 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:29.552 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491157+0000", 2024-09-18T04:38:29.552 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014401799999999999 2024-09-18T04:38:29.552 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:29.552 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:29.552 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:29.553 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.553 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.553 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "MOSDPGPushReply(2.3 25/24 [PushReplyOp(2:c4c92e5a:::rbd_trash:head)])", 2024-09-18T04:38:29.553 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:51.680323+0000", 2024-09-18T04:38:29.553 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 37.829092111000001, 2024-09-18T04:38:29.553 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00054411499999999996, 2024-09-18T04:38:29.553 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:29.553 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-09-18T04:38:29.553 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:29.553 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.553 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:29.553 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680323+0000", 2024-09-18T04:38:29.553 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:29.554 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.554 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.554 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:29.554 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680323+0000", 2024-09-18T04:38:29.554 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:29.554 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.554 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.554 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:29.554 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680322+0000", 2024-09-18T04:38:29.554 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:29.554 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.554 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.554 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:29.554 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680333+0000", 2024-09-18T04:38:29.555 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1153e-05 2024-09-18T04:38:29.555 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.555 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.555 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:29.555 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680357+0000", 2024-09-18T04:38:29.555 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3252999999999999e-05 2024-09-18T04:38:29.555 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.555 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.555 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:29.555 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680378+0000", 2024-09-18T04:38:29.555 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1503000000000001e-05 2024-09-18T04:38:29.555 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.555 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.555 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:29.556 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680522+0000", 2024-09-18T04:38:29.556 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014410700000000001 2024-09-18T04:38:29.556 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:29.556 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:29.556 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:29.556 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680867+0000", 2024-09-18T04:38:29.556 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00034475899999999998 2024-09-18T04:38:29.556 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:29.556 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:29.556 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:29.556 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:29.556 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:29.556 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:29.557 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:38:29.589 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:29.582+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:29.592 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:38:29.619 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:38:29.637 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:38:29.659 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:38:29.668 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:29.668 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:29.678 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:38:29.684 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:38:29.690 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:29.686+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:29.714 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:38:29.716 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:38:29.716 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:38:29.716 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:38:29.716 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:38:29.716 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:38:29.716 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:38:29.791 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:29.786+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:29.792 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:29.792 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:29.802 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:38:29.882 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-09-18T04:38:29.882 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.1 to 1 (10000) 2024-09-18T04:38:29.892 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:29.886+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:29.893 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-18T04:38:29.909 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:29.909 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:29.920 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:38:29.993 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:29.986+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:30.024 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:30.024 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:30.033 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:38:30.094 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:30.090+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:30.193 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:30.193 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:30.194 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:30.190+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:30.203 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:38:30.295 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:30.290+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:30.355 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:30.355 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:30.365 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:38:30.410 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:30.406+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:30.478 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:30.478 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:30.488 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:38:30.497 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:30.490+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:30.594 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:30.594 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:30.598 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:30.594+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:30.603 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:38:30.699 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:30.694+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:30.710 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:30.711 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:30.721 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:38:30.800 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:30.794+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:30.837 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:30.837 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:30.848 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:38:30.883 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.3 to 1 (10000) 2024-09-18T04:38:30.897 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-18T04:38:30.905 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:30.894+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:30.952 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:30.952 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:30.962 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:38:31.010 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:31.006+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:31.065 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:31.065 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:31.075 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:38:31.104 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:31.098+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:31.179 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:31.179 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:31.189 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:38:31.205 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:31.198+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:31.297 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:31.297 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:31.306 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:31.302+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:31.307 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:38:31.406 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:31.402+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:31.415 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:31.415 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:31.425 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:38:31.508 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:31.502+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:31.529 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:31.529 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:31.538 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:38:31.609 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:31.602+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:31.646 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:31.646 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:31.655 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:38:31.709 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:31.702+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:31.731 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T04:38:31.761 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:31.761 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:31.771 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:38:31.809 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:31.806+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:31.890 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.4 to 1 (10000) 2024-09-18T04:38:31.890 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:38:31.891 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-18T04:38:31.894 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:31.894 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:31.894 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:38:31.894 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:31.905 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-18T04:38:31.909 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:38:31.909 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:38:31.910 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:31.906+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:31.931 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:38:31.958 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:38:31.982 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:38:32.005 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:38:32.007 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:38:32.007 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:38:32.007 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:38:32.007 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:38:32.007 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:38:32.007 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:38:32.011 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:32.006+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:32.015 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:32.015 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:32.015 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:38:32.016 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:38:32.016 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:32.029 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:38:32.111 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:32.106+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:32.183 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:32.183 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:38:32.183 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:38:32.183 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:38:32.183 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.183 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:30 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'16, mlcod=15'14)", 2024-09-18T04:38:32.183 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.486330+0000", 2024-09-18T04:38:32.183 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 64.692611327999998, 2024-09-18T04:38:32.183 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026383700000000001, 2024-09-18T04:38:32.183 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:32.184 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:32.184 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:32.184 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.184 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:32.184 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486330+0000", 2024-09-18T04:38:32.184 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:32.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.184 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.184 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:32.184 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486330+0000", 2024-09-18T04:38:32.184 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:32.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.184 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.184 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:32.185 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486328+0000", 2024-09-18T04:38:32.185 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:32.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.185 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.185 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:32.185 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486342+0000", 2024-09-18T04:38:32.185 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3783e-05 2024-09-18T04:38:32.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.185 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.185 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:32.185 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486373+0000", 2024-09-18T04:38:32.185 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1362999999999998e-05 2024-09-18T04:38:32.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.185 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.186 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:32.186 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486387+0000", 2024-09-18T04:38:32.186 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3219e-05 2024-09-18T04:38:32.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.186 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.186 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:32.186 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486489+0000", 2024-09-18T04:38:32.186 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010232499999999999 2024-09-18T04:38:32.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.186 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.186 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:32.186 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486524+0000", 2024-09-18T04:38:32.186 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4972000000000003e-05 2024-09-18T04:38:32.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.186 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.187 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:32.187 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.488902+0000", 2024-09-18T04:38:32.187 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023776240000000001 2024-09-18T04:38:32.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.187 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.187 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:32.187 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.488968+0000", 2024-09-18T04:38:32.187 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6761000000000007e-05 2024-09-18T04:38:32.187 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:32.187 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:32.187 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:32.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.187 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.187 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:35 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'21, mlcod=15'17)", 2024-09-18T04:38:32.188 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.491817+0000", 2024-09-18T04:38:32.188 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 64.687124409999996, 2024-09-18T04:38:32.188 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023524739999999998, 2024-09-18T04:38:32.188 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:32.188 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:32.188 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:32.188 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.188 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:32.188 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491817+0000", 2024-09-18T04:38:32.188 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:32.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.188 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.188 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:32.189 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491817+0000", 2024-09-18T04:38:32.189 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:32.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.189 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.189 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:32.189 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491815+0000", 2024-09-18T04:38:32.189 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:32.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.189 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.189 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:32.189 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491890+0000", 2024-09-18T04:38:32.189 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4500999999999997e-05 2024-09-18T04:38:32.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.190 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.190 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:32.190 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491916+0000", 2024-09-18T04:38:32.190 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5666000000000001e-05 2024-09-18T04:38:32.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.190 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.190 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:32.190 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491928+0000", 2024-09-18T04:38:32.190 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2585999999999999e-05 2024-09-18T04:38:32.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.190 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.190 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:32.190 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.492000+0000", 2024-09-18T04:38:32.190 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2309000000000002e-05 2024-09-18T04:38:32.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.191 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.191 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:32.191 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.492027+0000", 2024-09-18T04:38:32.191 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6251e-05 2024-09-18T04:38:32.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.191 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.191 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:32.191 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.494114+0000", 2024-09-18T04:38:32.191 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020875260000000001 2024-09-18T04:38:32.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.191 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.191 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:32.191 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.494169+0000", 2024-09-18T04:38:32.191 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5145000000000001e-05 2024-09-18T04:38:32.192 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:32.192 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:32.192 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:32.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.192 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.192 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:57 1.0 e15/14 1:30306672:devicehealth::main.db.0000000000000000:head v 15'35, mlcod=15'33)", 2024-09-18T04:38:32.192 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.529997+0000", 2024-09-18T04:38:32.192 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 64.648944556000004, 2024-09-18T04:38:32.192 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002623869, 2024-09-18T04:38:32.192 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:32.192 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:32.192 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:32.192 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.193 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:32.193 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.529997+0000", 2024-09-18T04:38:32.193 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:32.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.193 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.193 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:32.193 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.529997+0000", 2024-09-18T04:38:32.193 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:32.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.193 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.193 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:32.193 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.529995+0000", 2024-09-18T04:38:32.193 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:32.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.193 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.194 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:32.194 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530116+0000", 2024-09-18T04:38:32.194 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000121141 2024-09-18T04:38:32.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.194 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.194 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:32.194 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530146+0000", 2024-09-18T04:38:32.194 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0423e-05 2024-09-18T04:38:32.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.194 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.194 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:32.194 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530159+0000", 2024-09-18T04:38:32.194 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2802999999999999e-05 2024-09-18T04:38:32.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.195 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.195 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:32.195 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530218+0000", 2024-09-18T04:38:32.195 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8461000000000002e-05 2024-09-18T04:38:32.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.195 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.195 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:32.195 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530253+0000", 2024-09-18T04:38:32.195 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4796999999999999e-05 2024-09-18T04:38:32.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.195 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.195 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:32.195 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.532559+0000", 2024-09-18T04:38:32.195 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002306939 2024-09-18T04:38:32.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.196 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.196 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:32.196 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.532621+0000", 2024-09-18T04:38:32.196 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1131999999999997e-05 2024-09-18T04:38:32.196 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:32.196 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:32.196 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:32.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.196 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.196 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:68 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'42, mlcod=15'39)", 2024-09-18T04:38:32.196 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.546404+0000", 2024-09-18T04:38:32.196 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 64.632537145000001, 2024-09-18T04:38:32.196 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024499769999999999, 2024-09-18T04:38:32.196 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:32.197 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:32.197 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:32.197 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.197 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:32.197 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546404+0000", 2024-09-18T04:38:32.197 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:32.197 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.197 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.197 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:32.198 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546404+0000", 2024-09-18T04:38:32.198 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:32.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.198 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.198 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:32.198 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546403+0000", 2024-09-18T04:38:32.198 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:32.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.198 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.198 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:32.198 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546416+0000", 2024-09-18T04:38:32.198 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.306e-05 2024-09-18T04:38:32.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.198 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.199 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:32.199 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546465+0000", 2024-09-18T04:38:32.199 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9441999999999999e-05 2024-09-18T04:38:32.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.199 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.199 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:32.199 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546480+0000", 2024-09-18T04:38:32.199 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4919999999999999e-05 2024-09-18T04:38:32.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.199 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.199 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:32.199 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546541+0000", 2024-09-18T04:38:32.199 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1085000000000002e-05 2024-09-18T04:38:32.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.199 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.200 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:32.200 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546569+0000", 2024-09-18T04:38:32.200 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7803000000000001e-05 2024-09-18T04:38:32.200 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.200 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.200 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:32.200 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.548725+0000", 2024-09-18T04:38:32.200 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021556629999999999 2024-09-18T04:38:32.200 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.201 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.201 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:32.201 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.548854+0000", 2024-09-18T04:38:32.201 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000129524 2024-09-18T04:38:32.201 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:32.201 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:32.201 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:32.201 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.201 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.201 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:69 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'43, mlcod=15'39)", 2024-09-18T04:38:32.201 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.547520+0000", 2024-09-18T04:38:32.201 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 64.631421068999998, 2024-09-18T04:38:32.201 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023506820000000002, 2024-09-18T04:38:32.201 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:32.202 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:32.202 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:32.202 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.202 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:32.202 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547520+0000", 2024-09-18T04:38:32.202 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:32.202 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.202 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.202 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:32.202 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547520+0000", 2024-09-18T04:38:32.202 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:32.202 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.202 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.203 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:32.203 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547519+0000", 2024-09-18T04:38:32.203 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:32.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.203 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.203 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:32.203 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547614+0000", 2024-09-18T04:38:32.203 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5235000000000006e-05 2024-09-18T04:38:32.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.203 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.203 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:32.203 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547640+0000", 2024-09-18T04:38:32.203 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5698999999999999e-05 2024-09-18T04:38:32.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.204 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.204 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:32.204 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547651+0000", 2024-09-18T04:38:32.204 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.134e-05 2024-09-18T04:38:32.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.204 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.204 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:32.204 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547805+0000", 2024-09-18T04:38:32.204 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015426099999999999 2024-09-18T04:38:32.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.204 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.204 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:32.204 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547844+0000", 2024-09-18T04:38:32.204 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8229999999999998e-05 2024-09-18T04:38:32.205 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.205 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.205 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:32.205 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.549810+0000", 2024-09-18T04:38:32.205 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0019660519999999998 2024-09-18T04:38:32.205 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.205 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.205 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:32.205 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.549871+0000", 2024-09-18T04:38:32.205 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1152e-05 2024-09-18T04:38:32.205 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:32.205 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:32.205 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:32.206 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.206 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.206 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:83 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'53, mlcod=15'50)", 2024-09-18T04:38:32.206 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.575989+0000", 2024-09-18T04:38:32.206 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 64.602952121000001, 2024-09-18T04:38:32.206 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023991310000000001, 2024-09-18T04:38:32.206 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:32.206 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:32.206 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:32.206 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.206 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:32.206 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.575989+0000", 2024-09-18T04:38:32.206 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:32.206 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.206 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.207 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:32.207 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.575989+0000", 2024-09-18T04:38:32.207 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:32.207 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.207 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.207 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:32.207 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.575988+0000", 2024-09-18T04:38:32.207 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:32.207 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.207 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.207 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:32.207 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576001+0000", 2024-09-18T04:38:32.207 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2751e-05 2024-09-18T04:38:32.207 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.208 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.208 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:32.208 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576023+0000", 2024-09-18T04:38:32.208 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2453999999999999e-05 2024-09-18T04:38:32.208 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.208 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.208 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:32.208 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576038+0000", 2024-09-18T04:38:32.208 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5125e-05 2024-09-18T04:38:32.208 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.208 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.208 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:32.208 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576161+0000", 2024-09-18T04:38:32.208 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000122543 2024-09-18T04:38:32.209 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.209 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.209 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:32.209 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576221+0000", 2024-09-18T04:38:32.209 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9948000000000003e-05 2024-09-18T04:38:32.209 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.209 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.209 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:32.209 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.578277+0000", 2024-09-18T04:38:32.209 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020567559999999999 2024-09-18T04:38:32.209 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.209 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.209 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:32.209 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.578388+0000", 2024-09-18T04:38:32.209 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011089100000000001 2024-09-18T04:38:32.210 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:32.210 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:32.210 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:32.210 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.210 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.210 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:86 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'56, mlcod=15'52)", 2024-09-18T04:38:32.210 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.580261+0000", 2024-09-18T04:38:32.210 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 64.598680485000003, 2024-09-18T04:38:32.210 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002418108, 2024-09-18T04:38:32.210 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:32.210 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:32.210 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:32.210 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.210 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:32.211 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580261+0000", 2024-09-18T04:38:32.211 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:32.211 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.211 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.211 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:32.211 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580261+0000", 2024-09-18T04:38:32.211 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:32.211 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.211 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.211 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:32.211 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580260+0000", 2024-09-18T04:38:32.211 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:32.211 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.211 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.212 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:32.212 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580269+0000", 2024-09-18T04:38:32.212 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5109999999999992e-06 2024-09-18T04:38:32.212 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.212 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.212 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:32.212 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580288+0000", 2024-09-18T04:38:32.212 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8783999999999998e-05 2024-09-18T04:38:32.212 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.212 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.212 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:32.212 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580296+0000", 2024-09-18T04:38:32.212 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3250000000000008e-06 2024-09-18T04:38:32.212 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.213 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.213 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:32.213 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580351+0000", 2024-09-18T04:38:32.213 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4132000000000003e-05 2024-09-18T04:38:32.213 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.213 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.213 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:32.213 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580381+0000", 2024-09-18T04:38:32.213 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0263000000000001e-05 2024-09-18T04:38:32.213 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.213 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.213 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:32.213 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.582615+0000", 2024-09-18T04:38:32.213 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0022339109999999999 2024-09-18T04:38:32.214 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.214 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.214 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:32.214 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.582679+0000", 2024-09-18T04:38:32.214 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4182000000000003e-05 2024-09-18T04:38:32.214 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:32.214 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:32.214 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:32.214 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.214 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.214 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:105 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'67, mlcod=15'64)", 2024-09-18T04:38:32.214 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.789089+0000", 2024-09-18T04:38:32.214 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 64.389852314999999, 2024-09-18T04:38:32.214 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024384229999999999, 2024-09-18T04:38:32.214 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:32.215 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:32.215 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:32.215 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.215 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:32.215 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789089+0000", 2024-09-18T04:38:32.215 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:32.215 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.215 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.215 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:32.215 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789089+0000", 2024-09-18T04:38:32.215 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:32.215 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.215 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.215 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:32.216 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789088+0000", 2024-09-18T04:38:32.216 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:32.216 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.216 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.216 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:32.216 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789107+0000", 2024-09-18T04:38:32.216 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9017000000000001e-05 2024-09-18T04:38:32.216 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.216 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.216 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:32.216 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789139+0000", 2024-09-18T04:38:32.216 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2261999999999997e-05 2024-09-18T04:38:32.216 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.216 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.217 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:32.217 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789153+0000", 2024-09-18T04:38:32.217 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4025999999999999e-05 2024-09-18T04:38:32.217 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.217 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.217 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:32.217 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789255+0000", 2024-09-18T04:38:32.217 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000101969 2024-09-18T04:38:32.217 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.217 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.217 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:32.217 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789314+0000", 2024-09-18T04:38:32.217 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8867999999999999e-05 2024-09-18T04:38:32.217 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.217 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.218 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:32.218 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.791453+0000", 2024-09-18T04:38:32.218 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002138938 2024-09-18T04:38:32.218 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.218 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.218 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:32.218 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.791527+0000", 2024-09-18T04:38:32.218 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4464999999999999e-05 2024-09-18T04:38:32.218 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:32.218 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:32.218 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:32.218 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.218 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.219 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:106 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'68, mlcod=15'64)", 2024-09-18T04:38:32.219 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.790295+0000", 2024-09-18T04:38:32.219 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 64.388646464000004, 2024-09-18T04:38:32.219 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024379390000000001, 2024-09-18T04:38:32.219 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:32.219 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:32.219 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:32.219 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.219 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:32.219 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790295+0000", 2024-09-18T04:38:32.219 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:32.219 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.219 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.219 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:32.220 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790295+0000", 2024-09-18T04:38:32.220 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:32.220 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.220 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.220 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:32.220 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790294+0000", 2024-09-18T04:38:32.220 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:32.220 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.220 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.220 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:32.220 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790390+0000", 2024-09-18T04:38:32.220 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6537000000000001e-05 2024-09-18T04:38:32.220 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.220 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.221 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:32.221 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790418+0000", 2024-09-18T04:38:32.221 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8218e-05 2024-09-18T04:38:32.221 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.221 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.221 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:32.221 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790431+0000", 2024-09-18T04:38:32.221 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2571999999999999e-05 2024-09-18T04:38:32.221 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.221 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.221 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:32.221 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790534+0000", 2024-09-18T04:38:32.221 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000103433 2024-09-18T04:38:32.221 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.222 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.222 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:32.222 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790565+0000", 2024-09-18T04:38:32.222 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0409999999999999e-05 2024-09-18T04:38:32.222 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.222 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.222 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:32.222 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.792651+0000", 2024-09-18T04:38:32.222 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020865250000000001 2024-09-18T04:38:32.222 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.222 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.222 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:32.222 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.792733+0000", 2024-09-18T04:38:32.222 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1477999999999995e-05 2024-09-18T04:38:32.223 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:32.223 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:32.223 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:32.223 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.223 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.223 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:109 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'71, mlcod=15'68)", 2024-09-18T04:38:32.223 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.795250+0000", 2024-09-18T04:38:32.223 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 64.383691450000001, 2024-09-18T04:38:32.223 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023990449999999998, 2024-09-18T04:38:32.223 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:32.223 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:32.223 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:32.223 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.223 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:32.224 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795250+0000", 2024-09-18T04:38:32.224 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:32.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.224 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.224 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:32.224 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795250+0000", 2024-09-18T04:38:32.224 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:32.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.224 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.224 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:32.224 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795248+0000", 2024-09-18T04:38:32.224 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:32.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.225 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.225 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:32.225 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795405+0000", 2024-09-18T04:38:32.225 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000157007 2024-09-18T04:38:32.225 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.225 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.225 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:32.225 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795422+0000", 2024-09-18T04:38:32.225 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7013000000000001e-05 2024-09-18T04:38:32.225 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.225 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.225 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:32.225 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795432+0000", 2024-09-18T04:38:32.225 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0448999999999999e-05 2024-09-18T04:38:32.226 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.226 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.226 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:32.226 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795534+0000", 2024-09-18T04:38:32.226 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000101195 2024-09-18T04:38:32.226 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.226 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.226 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:32.226 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795587+0000", 2024-09-18T04:38:32.226 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3356000000000002e-05 2024-09-18T04:38:32.226 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.226 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.226 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:32.226 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.797560+0000", 2024-09-18T04:38:32.226 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001973023 2024-09-18T04:38:32.227 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.227 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.227 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:32.227 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.797649+0000", 2024-09-18T04:38:32.227 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8864999999999997e-05 2024-09-18T04:38:32.227 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:32.227 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:32.227 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:32.227 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.227 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.227 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:115 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'77, mlcod=15'72)", 2024-09-18T04:38:32.227 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.803339+0000", 2024-09-18T04:38:32.227 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 64.375602732000004, 2024-09-18T04:38:32.227 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002385093, 2024-09-18T04:38:32.228 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:32.228 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:32.228 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:32.228 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.228 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:32.228 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803339+0000", 2024-09-18T04:38:32.228 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:32.228 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.228 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.228 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:32.228 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803339+0000", 2024-09-18T04:38:32.228 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:32.228 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.228 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.229 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:32.229 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803337+0000", 2024-09-18T04:38:32.229 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:32.229 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.229 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.229 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:32.229 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803475+0000", 2024-09-18T04:38:32.229 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013718199999999999 2024-09-18T04:38:32.229 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.229 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.229 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:32.229 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803507+0000", 2024-09-18T04:38:32.229 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2351000000000002e-05 2024-09-18T04:38:32.229 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.230 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.230 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:32.230 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803533+0000", 2024-09-18T04:38:32.230 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6010999999999999e-05 2024-09-18T04:38:32.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.230 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.230 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:32.230 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803609+0000", 2024-09-18T04:38:32.230 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.64e-05 2024-09-18T04:38:32.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.230 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.230 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:32.230 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803671+0000", 2024-09-18T04:38:32.230 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1467e-05 2024-09-18T04:38:32.231 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.231 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.231 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:32.231 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.805676+0000", 2024-09-18T04:38:32.231 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020047849999999998 2024-09-18T04:38:32.231 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.231 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.231 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:32.231 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.805724+0000", 2024-09-18T04:38:32.231 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8151e-05 2024-09-18T04:38:32.231 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:32.231 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:32.231 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:32.231 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.231 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.232 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:118 1.0 e15/14 1:30306672:devicehealth::main.db.0000000000000000:head v 15'80, mlcod=15'78)", 2024-09-18T04:38:32.232 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.807930+0000", 2024-09-18T04:38:32.232 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 64.371011229000004, 2024-09-18T04:38:32.232 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028002259999999998, 2024-09-18T04:38:32.232 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:32.232 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:32.232 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:32.232 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.232 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:32.232 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.807930+0000", 2024-09-18T04:38:32.232 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:32.232 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.232 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.233 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:32.233 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.807930+0000", 2024-09-18T04:38:32.233 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:32.233 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.233 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.233 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:32.233 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.807929+0000", 2024-09-18T04:38:32.233 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:32.233 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.233 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.233 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:32.233 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808024+0000", 2024-09-18T04:38:32.233 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5258000000000002e-05 2024-09-18T04:38:32.234 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.234 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.234 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:32.234 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808044+0000", 2024-09-18T04:38:32.234 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0203e-05 2024-09-18T04:38:32.234 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.234 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.234 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:32.234 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808053+0000", 2024-09-18T04:38:32.234 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2689999999999999e-06 2024-09-18T04:38:32.234 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.234 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.234 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:32.234 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808137+0000", 2024-09-18T04:38:32.235 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4441e-05 2024-09-18T04:38:32.235 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.235 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.235 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:32.235 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808172+0000", 2024-09-18T04:38:32.235 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4453999999999999e-05 2024-09-18T04:38:32.235 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.235 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.235 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:32.235 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.810663+0000", 2024-09-18T04:38:32.235 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024910269999999998 2024-09-18T04:38:32.235 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.235 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.235 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:32.235 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.810730+0000", 2024-09-18T04:38:32.236 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7658999999999997e-05 2024-09-18T04:38:32.236 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:32.236 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:32.236 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:32.236 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.236 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.236 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:129 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'87, mlcod=15'85)", 2024-09-18T04:38:32.236 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.231689+0000", 2024-09-18T04:38:32.236 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 63.947252216000003, 2024-09-18T04:38:32.236 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002376502, 2024-09-18T04:38:32.236 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:32.236 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:32.236 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:32.236 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.237 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:32.237 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231689+0000", 2024-09-18T04:38:32.237 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:32.237 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.237 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.237 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:32.237 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231689+0000", 2024-09-18T04:38:32.237 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:32.237 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.237 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.237 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:32.237 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231687+0000", 2024-09-18T04:38:32.237 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:32.237 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.238 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.238 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:32.238 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231702+0000", 2024-09-18T04:38:32.238 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4949999999999999e-05 2024-09-18T04:38:32.238 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.238 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.238 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:32.238 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231735+0000", 2024-09-18T04:38:32.238 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2276000000000001e-05 2024-09-18T04:38:32.238 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.238 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.238 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:32.238 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231750+0000", 2024-09-18T04:38:32.238 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5193e-05 2024-09-18T04:38:32.239 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.239 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.239 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:32.239 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231872+0000", 2024-09-18T04:38:32.239 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000122127 2024-09-18T04:38:32.239 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.239 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.239 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:32.239 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231936+0000", 2024-09-18T04:38:32.239 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4089000000000003e-05 2024-09-18T04:38:32.239 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.239 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.239 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:32.240 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234002+0000", 2024-09-18T04:38:32.240 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002065575 2024-09-18T04:38:32.240 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.240 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.240 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:32.240 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234066+0000", 2024-09-18T04:38:32.240 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3944999999999998e-05 2024-09-18T04:38:32.240 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:32.240 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:32.240 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:32.240 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.240 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.240 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:130 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'88, mlcod=15'85)", 2024-09-18T04:38:32.240 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.232853+0000", 2024-09-18T04:38:32.241 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 63.946088601, 2024-09-18T04:38:32.241 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027351739999999999, 2024-09-18T04:38:32.241 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:32.241 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:32.241 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:32.241 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.241 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:32.241 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232853+0000", 2024-09-18T04:38:32.241 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:32.241 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.241 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.241 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:32.241 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232853+0000", 2024-09-18T04:38:32.241 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:32.242 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.242 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.242 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:32.242 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232851+0000", 2024-09-18T04:38:32.242 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:32.242 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.242 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.242 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:32.242 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232866+0000", 2024-09-18T04:38:32.242 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4946000000000001e-05 2024-09-18T04:38:32.242 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.242 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.242 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:32.242 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232897+0000", 2024-09-18T04:38:32.243 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0982999999999998e-05 2024-09-18T04:38:32.243 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.243 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.243 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:32.243 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232915+0000", 2024-09-18T04:38:32.243 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8476000000000001e-05 2024-09-18T04:38:32.243 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.243 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.243 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:32.243 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.233029+0000", 2024-09-18T04:38:32.243 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011305400000000001 2024-09-18T04:38:32.243 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.243 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.244 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:32.244 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.233079+0000", 2024-09-18T04:38:32.244 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0668999999999999e-05 2024-09-18T04:38:32.244 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.244 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.244 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:32.244 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235534+0000", 2024-09-18T04:38:32.244 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024546030000000001 2024-09-18T04:38:32.244 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.244 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.244 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:32.244 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235588+0000", 2024-09-18T04:38:32.244 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4098999999999998e-05 2024-09-18T04:38:32.245 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:32.245 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:32.245 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:32.245 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.245 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.245 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:131 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'89, mlcod=15'85)", 2024-09-18T04:38:32.245 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.234089+0000", 2024-09-18T04:38:32.245 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 63.944852746999999, 2024-09-18T04:38:32.245 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026975609999999998, 2024-09-18T04:38:32.245 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:32.245 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:32.245 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:32.245 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.245 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:32.246 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234089+0000", 2024-09-18T04:38:32.246 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:32.246 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.246 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.246 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:32.246 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234089+0000", 2024-09-18T04:38:32.246 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:32.246 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.246 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.246 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:32.246 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234087+0000", 2024-09-18T04:38:32.246 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:32.246 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.246 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.247 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:32.247 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234145+0000", 2024-09-18T04:38:32.247 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8687999999999998e-05 2024-09-18T04:38:32.247 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.247 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.247 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:32.247 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234177+0000", 2024-09-18T04:38:32.247 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1526999999999998e-05 2024-09-18T04:38:32.247 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.247 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.247 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:32.247 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234190+0000", 2024-09-18T04:38:32.247 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2965999999999999e-05 2024-09-18T04:38:32.248 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.248 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.248 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:32.248 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234309+0000", 2024-09-18T04:38:32.248 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011875399999999999 2024-09-18T04:38:32.248 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.248 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.248 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:32.248 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234365+0000", 2024-09-18T04:38:32.248 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6436e-05 2024-09-18T04:38:32.248 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.248 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.248 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:32.248 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236675+0000", 2024-09-18T04:38:32.249 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023099520000000001 2024-09-18T04:38:32.249 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.249 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.249 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:32.249 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236786+0000", 2024-09-18T04:38:32.249 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000111025 2024-09-18T04:38:32.249 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:32.249 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:32.249 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:32.249 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.249 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.249 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:132 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'90, mlcod=15'85)", 2024-09-18T04:38:32.249 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.235300+0000", 2024-09-18T04:38:32.249 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 63.943641296999999, 2024-09-18T04:38:32.250 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025982599999999998, 2024-09-18T04:38:32.250 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:32.250 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:32.250 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:32.250 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.250 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:32.250 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235300+0000", 2024-09-18T04:38:32.250 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:32.250 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.250 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.250 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:32.250 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235300+0000", 2024-09-18T04:38:32.250 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:32.250 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.251 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.251 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:32.251 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235298+0000", 2024-09-18T04:38:32.251 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:32.251 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.251 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.251 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:32.251 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235318+0000", 2024-09-18T04:38:32.251 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9714000000000001e-05 2024-09-18T04:38:32.251 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.251 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.251 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:32.251 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235337+0000", 2024-09-18T04:38:32.251 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8892000000000001e-05 2024-09-18T04:38:32.252 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.252 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.252 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:32.252 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235348+0000", 2024-09-18T04:38:32.252 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.129e-05 2024-09-18T04:38:32.252 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.252 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.252 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:32.252 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235681+0000", 2024-09-18T04:38:32.252 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00033323000000000001 2024-09-18T04:38:32.252 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.252 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.252 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:32.252 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235733+0000", 2024-09-18T04:38:32.253 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1391999999999999e-05 2024-09-18T04:38:32.253 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.253 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.253 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:32.253 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237839+0000", 2024-09-18T04:38:32.253 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002105647 2024-09-18T04:38:32.253 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.253 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.253 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:32.253 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237898+0000", 2024-09-18T04:38:32.253 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9759000000000002e-05 2024-09-18T04:38:32.253 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:32.253 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:32.253 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:32.254 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.254 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.254 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:133 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'91, mlcod=15'86)", 2024-09-18T04:38:32.254 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.236920+0000", 2024-09-18T04:38:32.254 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 63.942021408999999, 2024-09-18T04:38:32.254 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024318410000000001, 2024-09-18T04:38:32.254 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:32.254 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:32.254 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:32.254 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.254 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:32.254 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236920+0000", 2024-09-18T04:38:32.254 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:32.254 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.255 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.255 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:32.255 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236920+0000", 2024-09-18T04:38:32.255 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:32.255 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.255 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.255 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:32.255 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236918+0000", 2024-09-18T04:38:32.255 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:32.255 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.255 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.255 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:32.255 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236931+0000", 2024-09-18T04:38:32.255 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2683999999999999e-05 2024-09-18T04:38:32.256 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.256 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.256 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:32.256 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236951+0000", 2024-09-18T04:38:32.256 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0292000000000002e-05 2024-09-18T04:38:32.256 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.256 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.256 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:32.256 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236962+0000", 2024-09-18T04:38:32.256 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0703000000000001e-05 2024-09-18T04:38:32.256 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.256 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.256 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:32.257 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237095+0000", 2024-09-18T04:38:32.257 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013312400000000001 2024-09-18T04:38:32.257 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.257 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.257 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:32.257 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237133+0000", 2024-09-18T04:38:32.257 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7496000000000001e-05 2024-09-18T04:38:32.257 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.257 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.257 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:32.257 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.239206+0000", 2024-09-18T04:38:32.257 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020730290000000001 2024-09-18T04:38:32.257 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.258 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.258 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:32.258 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.239352+0000", 2024-09-18T04:38:32.258 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014593000000000001 2024-09-18T04:38:32.258 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:32.258 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:32.258 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:32.258 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.258 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.258 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:134 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'92, mlcod=15'86)", 2024-09-18T04:38:32.258 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.237894+0000", 2024-09-18T04:38:32.258 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 63.941047386999998, 2024-09-18T04:38:32.258 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025299519999999998, 2024-09-18T04:38:32.258 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:32.259 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:32.259 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:32.259 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.259 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:32.259 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237894+0000", 2024-09-18T04:38:32.259 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:32.259 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.259 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.259 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:32.259 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237894+0000", 2024-09-18T04:38:32.259 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:32.259 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.259 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.259 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:32.260 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237892+0000", 2024-09-18T04:38:32.260 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:32.260 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.260 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.260 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:32.260 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237986+0000", 2024-09-18T04:38:32.260 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3341e-05 2024-09-18T04:38:32.260 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.260 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.260 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:32.260 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.238012+0000", 2024-09-18T04:38:32.260 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5843e-05 2024-09-18T04:38:32.260 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.261 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.261 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:32.261 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.238027+0000", 2024-09-18T04:38:32.261 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5469e-05 2024-09-18T04:38:32.261 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.261 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.261 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:32.261 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.238114+0000", 2024-09-18T04:38:32.261 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7483000000000002e-05 2024-09-18T04:38:32.261 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.261 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.261 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:32.261 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.238152+0000", 2024-09-18T04:38:32.262 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7565999999999999e-05 2024-09-18T04:38:32.262 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.262 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.262 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:32.262 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.240297+0000", 2024-09-18T04:38:32.262 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021447620000000001 2024-09-18T04:38:32.262 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.262 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.262 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:32.262 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.240424+0000", 2024-09-18T04:38:32.262 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012705400000000001 2024-09-18T04:38:32.262 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:32.262 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:32.262 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:32.263 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.263 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.263 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:139 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'97, mlcod=15'93)", 2024-09-18T04:38:32.263 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.245692+0000", 2024-09-18T04:38:32.263 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 63.933249246000003, 2024-09-18T04:38:32.263 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024260480000000001, 2024-09-18T04:38:32.263 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:32.263 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:32.263 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:32.263 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.263 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:32.263 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245692+0000", 2024-09-18T04:38:32.263 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:32.264 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.264 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.264 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:32.264 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245692+0000", 2024-09-18T04:38:32.264 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:32.264 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.264 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.264 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:32.264 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245690+0000", 2024-09-18T04:38:32.265 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:32.265 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.265 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.265 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:32.265 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245712+0000", 2024-09-18T04:38:32.265 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1916000000000002e-05 2024-09-18T04:38:32.265 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.265 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.265 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:32.265 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245741+0000", 2024-09-18T04:38:32.265 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8503000000000001e-05 2024-09-18T04:38:32.265 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.265 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.265 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:32.265 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245759+0000", 2024-09-18T04:38:32.266 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.872e-05 2024-09-18T04:38:32.266 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.266 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.266 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:32.266 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245859+0000", 2024-09-18T04:38:32.266 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9641000000000004e-05 2024-09-18T04:38:32.266 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.266 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.266 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:32.266 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245900+0000", 2024-09-18T04:38:32.266 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0589000000000001e-05 2024-09-18T04:38:32.266 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.266 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.267 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:32.267 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.247961+0000", 2024-09-18T04:38:32.267 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020619190000000002 2024-09-18T04:38:32.267 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.267 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.267 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:32.267 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.248118+0000", 2024-09-18T04:38:32.267 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015664300000000001 2024-09-18T04:38:32.267 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:32.267 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:32.267 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:32.267 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.267 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.267 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:144 1.0 e15/14 1:30306672:devicehealth::main.db.0000000000000000:head v 15'102, mlcod=15'98)", 2024-09-18T04:38:32.268 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.252149+0000", 2024-09-18T04:38:32.268 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 63.92679218, 2024-09-18T04:38:32.268 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024174959999999999, 2024-09-18T04:38:32.268 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:32.268 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:32.268 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:32.268 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.268 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:32.268 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252149+0000", 2024-09-18T04:38:32.268 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:32.268 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.268 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.268 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:32.268 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252149+0000", 2024-09-18T04:38:32.269 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:32.269 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.269 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.269 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:32.269 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252148+0000", 2024-09-18T04:38:32.269 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:32.269 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.269 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.269 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:32.269 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252225+0000", 2024-09-18T04:38:32.269 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6576999999999995e-05 2024-09-18T04:38:32.269 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.269 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.269 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:32.269 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252242+0000", 2024-09-18T04:38:32.269 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7740999999999998e-05 2024-09-18T04:38:32.270 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.270 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.270 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:32.270 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252254+0000", 2024-09-18T04:38:32.270 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.137e-05 2024-09-18T04:38:32.270 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.270 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.270 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:32.270 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252385+0000", 2024-09-18T04:38:32.270 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013091099999999999 2024-09-18T04:38:32.270 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.270 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.270 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:32.270 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252426+0000", 2024-09-18T04:38:32.270 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1535999999999997e-05 2024-09-18T04:38:32.270 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.271 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.271 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:32.271 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.254498+0000", 2024-09-18T04:38:32.271 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002071477 2024-09-18T04:38:32.271 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.271 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.271 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:32.271 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.254567+0000", 2024-09-18T04:38:32.271 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9014e-05 2024-09-18T04:38:32.271 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:32.271 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:32.271 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:32.271 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:32.271 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:32.271 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:32.273 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:38:32.274 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:32.266+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:32.374 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:32.370+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:32.385 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:32.385 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:32.385 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:38:32.385 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:32.394 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:38:32.475 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:32.470+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:32.503 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:32.503 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:32.503 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:38:32.503 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:38:32.503 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:32.513 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:38:32.576 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:32.570+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:32.622 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:32.622 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:38:32.622 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:38:32.622 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:38:32.622 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.622 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4214.0:5 2.2 e19/17 2:5cea7035:::rbd_info:head v 19'1, mlcod=0'0)", 2024-09-18T04:38:32.623 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:35.471260+0000", 2024-09-18T04:38:32.623 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 57.149596152000001, 2024-09-18T04:38:32.623 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021025950000000001, 2024-09-18T04:38:32.623 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:32.623 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:32.623 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:32.623 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.623 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:32.623 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471260+0000", 2024-09-18T04:38:32.623 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:32.623 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.623 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.623 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:32.623 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471260+0000", 2024-09-18T04:38:32.624 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:32.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.624 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.624 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:32.624 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471259+0000", 2024-09-18T04:38:32.624 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:32.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.624 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.624 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:32.624 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471275+0000", 2024-09-18T04:38:32.624 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6135999999999999e-05 2024-09-18T04:38:32.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.624 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.625 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:32.625 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471298+0000", 2024-09-18T04:38:32.625 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3632999999999999e-05 2024-09-18T04:38:32.625 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.625 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.625 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:32.625 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471320+0000", 2024-09-18T04:38:32.625 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1852999999999999e-05 2024-09-18T04:38:32.625 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.625 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.625 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:32.625 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471445+0000", 2024-09-18T04:38:32.625 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012461699999999999 2024-09-18T04:38:32.625 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.626 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.626 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:32.626 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471529+0000", 2024-09-18T04:38:32.626 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3875000000000006e-05 2024-09-18T04:38:32.626 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.626 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.626 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:32.626 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.473317+0000", 2024-09-18T04:38:32.626 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017886899999999999 2024-09-18T04:38:32.626 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.626 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.626 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:32.626 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.473362+0000", 2024-09-18T04:38:32.626 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4724000000000001e-05 2024-09-18T04:38:32.627 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:32.627 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:32.627 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:32.627 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.627 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.627 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4214.0:7 2.2 e20/17 2:5cea7035:::rbd_info:head v 20'2, mlcod=0'0)", 2024-09-18T04:38:32.627 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:36.490560+0000", 2024-09-18T04:38:32.627 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 56.130296012999999, 2024-09-18T04:38:32.627 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0015326020000000001, 2024-09-18T04:38:32.627 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:32.627 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:32.627 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:32.627 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.627 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:32.628 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490560+0000", 2024-09-18T04:38:32.628 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:32.628 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.628 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.628 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:32.628 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490560+0000", 2024-09-18T04:38:32.628 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:32.628 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.628 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.628 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:32.628 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490559+0000", 2024-09-18T04:38:32.628 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:32.628 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.629 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.629 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:32.629 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490568+0000", 2024-09-18T04:38:32.629 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5699999999999993e-06 2024-09-18T04:38:32.629 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.629 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.629 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:32.629 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490599+0000", 2024-09-18T04:38:32.629 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1566000000000002e-05 2024-09-18T04:38:32.629 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.629 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.629 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:32.629 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490609+0000", 2024-09-18T04:38:32.630 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.8700000000000004e-06 2024-09-18T04:38:32.630 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.630 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.630 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:32.630 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490708+0000", 2024-09-18T04:38:32.630 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9139000000000001e-05 2024-09-18T04:38:32.630 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.630 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.630 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:32.630 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490736+0000", 2024-09-18T04:38:32.630 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7617e-05 2024-09-18T04:38:32.630 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.630 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.630 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:32.631 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.492016+0000", 2024-09-18T04:38:32.631 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0012796579999999999 2024-09-18T04:38:32.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:32.631 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:32.631 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:32.631 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.492092+0000", 2024-09-18T04:38:32.631 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6571999999999998e-05 2024-09-18T04:38:32.631 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:32.631 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:32.631 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:32.631 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:32.631 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:32.631 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:32.632 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:38:32.676 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:32.670+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:32.739 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:32.739 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:32.739 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:38:32.740 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:32.749 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:38:32.794 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:32.790+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:32.877 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:32.870+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:32.903 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-09-18T04:38:32.904 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.5 to 1 (10000) 2024-09-18T04:38:32.908 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:32.908 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:32.908 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:38:32.908 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:38:32.908 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:32.918 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:38:32.920 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-18T04:38:32.979 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:32.974+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:33.019 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:33.019 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:38:33.019 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:38:33.019 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:38:33.019 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:33.019 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4214.0:1 2.3 e18/17 2:c4c92e5a:::rbd_trash:head v 18'1, mlcod=0'0)", 2024-09-18T04:38:33.019 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:34.475123+0000", 2024-09-18T04:38:33.020 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 58.542432830999999, 2024-09-18T04:38:33.020 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00081394599999999996, 2024-09-18T04:38:33.020 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:33.020 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:33.020 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:33.020 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:33.020 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:33.020 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475123+0000", 2024-09-18T04:38:33.020 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:33.020 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:33.020 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:33.020 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:33.020 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475123+0000", 2024-09-18T04:38:33.021 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:33.021 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:33.021 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:33.021 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:33.021 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475121+0000", 2024-09-18T04:38:33.021 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:33.021 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:33.021 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:33.021 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:33.021 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475149+0000", 2024-09-18T04:38:33.021 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7155999999999999e-05 2024-09-18T04:38:33.021 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:33.021 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:33.022 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:33.022 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475165+0000", 2024-09-18T04:38:33.022 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6160000000000001e-05 2024-09-18T04:38:33.022 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:33.022 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:33.022 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:33.022 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475183+0000", 2024-09-18T04:38:33.022 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7918e-05 2024-09-18T04:38:33.022 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:33.022 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:33.022 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:33.022 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475246+0000", 2024-09-18T04:38:33.022 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3176000000000006e-05 2024-09-18T04:38:33.023 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:33.023 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:33.023 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:33.023 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475279+0000", 2024-09-18T04:38:33.023 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3192999999999999e-05 2024-09-18T04:38:33.023 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:33.023 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:33.023 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:33.023 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475871+0000", 2024-09-18T04:38:33.023 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00059206099999999995 2024-09-18T04:38:33.023 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:33.023 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:33.023 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:33.023 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475937+0000", 2024-09-18T04:38:33.024 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5439999999999997e-05 2024-09-18T04:38:33.024 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:33.024 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:33.024 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:33.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:33.024 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:33.024 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 25/24 [PushOp(2:c4c92e5a:::rbd_trash:head, version: 18'1, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:c4c92e5a:::rbd_trash:head@18'1, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-18T04:38:33.024 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:51.680544+0000", 2024-09-18T04:38:33.024 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 41.337010988999999, 2024-09-18T04:38:33.024 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00050365200000000005, 2024-09-18T04:38:33.024 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:33.024 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:38:33.024 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:33.025 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:33.025 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:33.025 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680544+0000", 2024-09-18T04:38:33.025 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:33.025 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:33.025 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:33.025 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:33.025 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680544+0000", 2024-09-18T04:38:33.025 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:33.025 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:33.025 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:33.025 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:33.025 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680543+0000", 2024-09-18T04:38:33.025 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:33.026 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:33.026 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:33.026 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:33.026 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680571+0000", 2024-09-18T04:38:33.026 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7953000000000002e-05 2024-09-18T04:38:33.026 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:33.026 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:33.026 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:33.026 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680605+0000", 2024-09-18T04:38:33.026 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3933000000000002e-05 2024-09-18T04:38:33.026 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:33.026 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:33.026 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:33.026 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680636+0000", 2024-09-18T04:38:33.027 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1288999999999998e-05 2024-09-18T04:38:33.027 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:33.027 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:33.027 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:33.027 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680748+0000", 2024-09-18T04:38:33.027 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000111431 2024-09-18T04:38:33.027 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:33.027 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:33.027 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:33.027 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680758+0000", 2024-09-18T04:38:33.028 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0244999999999999e-05 2024-09-18T04:38:33.028 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:33.028 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:33.028 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:33.028 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.681048+0000", 2024-09-18T04:38:33.028 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00029003399999999999 2024-09-18T04:38:33.028 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:33.028 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:33.028 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:33.028 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:33.028 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:33.028 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:33.031 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:38:33.079 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:33.074+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:33.146 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:33.146 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:33.146 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:38:33.146 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:33.156 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:38:33.180 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:33.174+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:33.267 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:33.267 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:33.267 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:38:33.267 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:38:33.267 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:33.276 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:38:33.281 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:33.274+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:33.382 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:33.378+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:33.385 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:33.385 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:38:33.385 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:38:33.385 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:38:33.386 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.386 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:33.386 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.573328+0000", 2024-09-18T04:38:33.386 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 65.806389807000002, 2024-09-18T04:38:33.386 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011397898, 2024-09-18T04:38:33.386 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:33.386 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:33.386 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:33.386 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:33.386 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:33.386 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 86 2024-09-18T04:38:33.386 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.386 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:33.386 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.387 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:33.387 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573328+0000", 2024-09-18T04:38:33.387 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:33.387 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.387 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.387 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:33.387 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573328+0000", 2024-09-18T04:38:33.387 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:33.387 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.387 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.387 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:33.387 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573327+0000", 2024-09-18T04:38:33.387 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:33.388 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.388 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.388 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:33.388 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573331+0000", 2024-09-18T04:38:33.388 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9750000000000001e-06 2024-09-18T04:38:33.388 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.388 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.388 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:33.388 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573346+0000", 2024-09-18T04:38:33.388 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5469999999999999e-05 2024-09-18T04:38:33.388 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.388 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.388 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:33.388 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573370+0000", 2024-09-18T04:38:33.389 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3969999999999999e-05 2024-09-18T04:38:33.389 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.389 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.389 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:33.389 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.578536+0000", 2024-09-18T04:38:33.389 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0051659679999999999 2024-09-18T04:38:33.389 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.389 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.389 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:33.389 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.578689+0000", 2024-09-18T04:38:33.389 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015276299999999999 2024-09-18T04:38:33.389 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.389 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.390 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:33.390 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.578901+0000", 2024-09-18T04:38:33.390 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000211783 2024-09-18T04:38:33.390 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.390 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.390 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:33.390 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.583408+0000", 2024-09-18T04:38:33.390 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045071249999999998 2024-09-18T04:38:33.390 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.390 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.390 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:33.390 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.584645+0000", 2024-09-18T04:38:33.390 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001237007 2024-09-18T04:38:33.391 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.391 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.391 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:33.391 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.584703+0000", 2024-09-18T04:38:33.391 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.817e-05 2024-09-18T04:38:33.391 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.391 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.391 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:33.391 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.584725+0000", 2024-09-18T04:38:33.391 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2155e-05 2024-09-18T04:38:33.391 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:33.391 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:33.391 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:33.391 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.391 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.392 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:33.392 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.573382+0000", 2024-09-18T04:38:33.392 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 65.806335059000006, 2024-09-18T04:38:33.392 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012154974000000001, 2024-09-18T04:38:33.392 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:33.392 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:33.392 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:33.392 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:33.392 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:33.392 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 87 2024-09-18T04:38:33.392 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.392 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:33.392 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.393 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:33.393 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573382+0000", 2024-09-18T04:38:33.393 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:33.393 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.393 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.393 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:33.393 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573382+0000", 2024-09-18T04:38:33.393 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:33.393 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.393 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.393 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:33.393 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573382+0000", 2024-09-18T04:38:33.393 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:33.394 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.394 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.394 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:33.394 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573482+0000", 2024-09-18T04:38:33.394 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000100333 2024-09-18T04:38:33.394 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.394 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.394 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:33.394 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573492+0000", 2024-09-18T04:38:33.394 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5389999999999996e-06 2024-09-18T04:38:33.394 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.394 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.394 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:33.394 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573500+0000", 2024-09-18T04:38:33.395 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8929999999999994e-06 2024-09-18T04:38:33.395 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.395 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.395 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:33.395 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.579797+0000", 2024-09-18T04:38:33.395 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0062963239999999998 2024-09-18T04:38:33.395 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.395 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.395 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:33.395 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.579957+0000", 2024-09-18T04:38:33.395 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000160283 2024-09-18T04:38:33.395 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.395 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.396 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:33.396 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.580178+0000", 2024-09-18T04:38:33.396 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00022079200000000001 2024-09-18T04:38:33.396 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.396 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.396 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:33.396 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.583479+0000", 2024-09-18T04:38:33.396 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033007689999999998 2024-09-18T04:38:33.396 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.396 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.396 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:33.396 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585477+0000", 2024-09-18T04:38:33.396 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0019985670000000001 2024-09-18T04:38:33.396 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.397 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.397 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:33.397 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585517+0000", 2024-09-18T04:38:33.397 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9709999999999998e-05 2024-09-18T04:38:33.397 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.397 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.397 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:33.397 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585537+0000", 2024-09-18T04:38:33.397 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0326000000000002e-05 2024-09-18T04:38:33.397 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:33.397 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:33.397 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:33.397 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.397 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.398 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:33.398 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.573508+0000", 2024-09-18T04:38:33.398 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 65.806209353, 2024-09-18T04:38:33.398 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012068476, 2024-09-18T04:38:33.398 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:33.398 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:33.398 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:33.398 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:33.398 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:33.398 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 88 2024-09-18T04:38:33.398 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.398 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:33.398 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.399 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:33.399 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573508+0000", 2024-09-18T04:38:33.399 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:33.399 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.399 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.399 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:33.399 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573508+0000", 2024-09-18T04:38:33.399 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:33.399 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.399 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.399 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:33.399 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573507+0000", 2024-09-18T04:38:33.399 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:33.399 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.400 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.400 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:33.400 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573512+0000", 2024-09-18T04:38:33.400 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.2629999999999997e-06 2024-09-18T04:38:33.400 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.400 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.400 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:33.400 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573530+0000", 2024-09-18T04:38:33.400 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8507000000000001e-05 2024-09-18T04:38:33.400 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.400 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.400 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:33.401 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573537+0000", 2024-09-18T04:38:33.401 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.832e-06 2024-09-18T04:38:33.401 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.401 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.401 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:33.401 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.582486+0000", 2024-09-18T04:38:33.401 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0089494770000000008 2024-09-18T04:38:33.401 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.401 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.401 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:33.401 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.582630+0000", 2024-09-18T04:38:33.401 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000144005 2024-09-18T04:38:33.401 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.401 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.402 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:33.402 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.582842+0000", 2024-09-18T04:38:33.402 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00021189200000000001 2024-09-18T04:38:33.402 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.402 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.402 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:33.402 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.584777+0000", 2024-09-18T04:38:33.402 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001934753 2024-09-18T04:38:33.402 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.402 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.402 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:33.402 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585543+0000", 2024-09-18T04:38:33.402 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00076540099999999997 2024-09-18T04:38:33.402 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.403 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.403 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:33.403 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585561+0000", 2024-09-18T04:38:33.403 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8474999999999998e-05 2024-09-18T04:38:33.403 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.403 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.403 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:33.403 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585576+0000", 2024-09-18T04:38:33.403 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5427e-05 2024-09-18T04:38:33.403 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:33.403 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:33.403 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:33.403 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.403 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.404 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:33.404 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786394+0000", 2024-09-18T04:38:33.404 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 65.593322905999997, 2024-09-18T04:38:33.404 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011902085999999999, 2024-09-18T04:38:33.404 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:33.404 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:33.404 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:33.404 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:33.404 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:33.404 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 109 2024-09-18T04:38:33.404 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.405 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:33.405 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.405 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:33.405 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786394+0000", 2024-09-18T04:38:33.405 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:33.405 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.405 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.405 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:33.405 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786394+0000", 2024-09-18T04:38:33.405 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:33.405 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.405 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.406 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:33.406 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786394+0000", 2024-09-18T04:38:33.406 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:33.406 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.406 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.406 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:33.406 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786548+0000", 2024-09-18T04:38:33.406 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015463400000000001 2024-09-18T04:38:33.406 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.406 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.406 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:33.406 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786561+0000", 2024-09-18T04:38:33.406 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3093e-05 2024-09-18T04:38:33.406 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.407 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.407 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:33.407 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786571+0000", 2024-09-18T04:38:33.407 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4700000000000008e-06 2024-09-18T04:38:33.407 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.407 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.407 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:33.407 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.793367+0000", 2024-09-18T04:38:33.407 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0067966600000000004 2024-09-18T04:38:33.407 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.407 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.407 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:33.407 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.793526+0000", 2024-09-18T04:38:33.408 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015877 2024-09-18T04:38:33.408 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.408 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.408 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:33.408 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.793752+0000", 2024-09-18T04:38:33.408 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00022555200000000001 2024-09-18T04:38:33.408 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.408 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.408 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:33.408 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798125+0000", 2024-09-18T04:38:33.408 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0043733349999999999 2024-09-18T04:38:33.408 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.408 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.408 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:33.409 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798146+0000", 2024-09-18T04:38:33.409 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1063000000000001e-05 2024-09-18T04:38:33.409 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.409 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.409 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:33.409 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798232+0000", 2024-09-18T04:38:33.409 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5771000000000003e-05 2024-09-18T04:38:33.409 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.409 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.409 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:33.409 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798297+0000", 2024-09-18T04:38:33.409 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.4624999999999999e-05 2024-09-18T04:38:33.409 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:33.409 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:33.410 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:33.410 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.410 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.410 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:33.410 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786584+0000", 2024-09-18T04:38:33.410 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 65.593132827000005, 2024-09-18T04:38:33.410 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012013037000000001, 2024-09-18T04:38:33.410 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:33.410 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:33.410 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:33.410 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:33.410 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:33.410 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 110 2024-09-18T04:38:33.410 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.411 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:33.411 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.411 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:33.411 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786584+0000", 2024-09-18T04:38:33.411 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:33.411 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.411 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.411 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:33.411 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786584+0000", 2024-09-18T04:38:33.411 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:33.411 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.411 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.411 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:33.411 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786584+0000", 2024-09-18T04:38:33.412 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:33.412 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.412 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.412 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:33.412 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786590+0000", 2024-09-18T04:38:33.412 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6359999999999997e-06 2024-09-18T04:38:33.412 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.412 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.412 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:33.412 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786599+0000", 2024-09-18T04:38:33.412 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7369999999999995e-06 2024-09-18T04:38:33.412 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.412 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.412 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:33.413 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786606+0000", 2024-09-18T04:38:33.413 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1930000000000004e-06 2024-09-18T04:38:33.413 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.413 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.413 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:33.413 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.795018+0000", 2024-09-18T04:38:33.413 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0084115660000000005 2024-09-18T04:38:33.413 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.413 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.413 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:33.413 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.795184+0000", 2024-09-18T04:38:33.413 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000165833 2024-09-18T04:38:33.413 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.413 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.414 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:33.414 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.795414+0000", 2024-09-18T04:38:33.414 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00023088399999999999 2024-09-18T04:38:33.414 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.414 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.414 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:33.414 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798321+0000", 2024-09-18T04:38:33.414 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0029069539999999998 2024-09-18T04:38:33.414 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.414 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.414 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:33.414 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798404+0000", 2024-09-18T04:38:33.414 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2724000000000003e-05 2024-09-18T04:38:33.414 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.415 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.415 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:33.415 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798488+0000", 2024-09-18T04:38:33.415 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.3505999999999996e-05 2024-09-18T04:38:33.415 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.415 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.415 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:33.415 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798598+0000", 2024-09-18T04:38:33.415 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000109917 2024-09-18T04:38:33.415 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:33.415 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:33.415 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:33.415 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.415 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.416 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:33.416 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786613+0000", 2024-09-18T04:38:33.416 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 65.593104127000004, 2024-09-18T04:38:33.416 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014245760999999999, 2024-09-18T04:38:33.416 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:33.416 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:33.416 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:33.416 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:33.416 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:33.416 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 111 2024-09-18T04:38:33.416 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.416 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:33.416 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.416 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:33.416 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786613+0000", 2024-09-18T04:38:33.417 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:33.417 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.417 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.417 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:33.417 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786613+0000", 2024-09-18T04:38:33.417 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:33.417 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.417 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.417 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:33.417 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786612+0000", 2024-09-18T04:38:33.417 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:33.417 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.417 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.417 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:33.418 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786622+0000", 2024-09-18T04:38:33.418 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4099999999999997e-06 2024-09-18T04:38:33.418 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.418 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.418 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:33.418 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786630+0000", 2024-09-18T04:38:33.418 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4169999999999992e-06 2024-09-18T04:38:33.418 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.418 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.418 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:33.418 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786637+0000", 2024-09-18T04:38:33.418 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8290000000000002e-06 2024-09-18T04:38:33.418 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.418 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.419 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:33.419 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.797099+0000", 2024-09-18T04:38:33.419 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010461747 2024-09-18T04:38:33.419 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.419 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.419 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:33.419 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.797243+0000", 2024-09-18T04:38:33.419 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014388899999999999 2024-09-18T04:38:33.419 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.419 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.419 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:33.419 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.797463+0000", 2024-09-18T04:38:33.419 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000219912 2024-09-18T04:38:33.420 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.420 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.420 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:33.420 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800732+0000", 2024-09-18T04:38:33.420 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0032688719999999999 2024-09-18T04:38:33.420 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.420 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.420 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:33.420 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800751+0000", 2024-09-18T04:38:33.420 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9633999999999999e-05 2024-09-18T04:38:33.420 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.420 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.420 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:33.420 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800824+0000", 2024-09-18T04:38:33.421 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.2805999999999994e-05 2024-09-18T04:38:33.421 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.421 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.421 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:33.421 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800859+0000", 2024-09-18T04:38:33.421 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4952e-05 2024-09-18T04:38:33.421 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:33.421 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:33.421 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:33.421 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.421 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.421 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:33.421 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786649+0000", 2024-09-18T04:38:33.421 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 65.593068717999998, 2024-09-18T04:38:33.422 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.01613442, 2024-09-18T04:38:33.422 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:33.422 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:33.422 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:33.422 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:33.422 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:33.422 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 112 2024-09-18T04:38:33.422 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.422 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:33.422 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.422 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:33.422 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786649+0000", 2024-09-18T04:38:33.422 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:33.422 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.422 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.423 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:33.423 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786649+0000", 2024-09-18T04:38:33.423 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:33.423 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.423 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.423 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:33.423 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786648+0000", 2024-09-18T04:38:33.423 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:33.423 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.423 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.423 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:33.423 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786768+0000", 2024-09-18T04:38:33.423 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000120491 2024-09-18T04:38:33.423 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.423 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.424 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:33.424 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786795+0000", 2024-09-18T04:38:33.424 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6556000000000001e-05 2024-09-18T04:38:33.424 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.424 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.424 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:33.424 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786807+0000", 2024-09-18T04:38:33.424 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1739999999999999e-05 2024-09-18T04:38:33.424 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.424 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.424 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:33.424 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798683+0000", 2024-09-18T04:38:33.424 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011876070000000001 2024-09-18T04:38:33.425 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.425 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.425 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:33.425 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798867+0000", 2024-09-18T04:38:33.425 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000184103 2024-09-18T04:38:33.425 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.425 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.425 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:33.425 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.799082+0000", 2024-09-18T04:38:33.425 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00021476000000000001 2024-09-18T04:38:33.425 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.425 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.425 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:33.426 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802610+0000", 2024-09-18T04:38:33.426 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0035282209999999998 2024-09-18T04:38:33.426 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.426 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.426 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:33.426 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802706+0000", 2024-09-18T04:38:33.426 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5922999999999996e-05 2024-09-18T04:38:33.426 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.426 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.426 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:33.426 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802750+0000", 2024-09-18T04:38:33.426 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4511000000000001e-05 2024-09-18T04:38:33.426 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.426 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.427 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:33.427 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802783+0000", 2024-09-18T04:38:33.427 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2752000000000001e-05 2024-09-18T04:38:33.427 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:33.427 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:33.427 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:33.427 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.427 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.427 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:33.427 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786816+0000", 2024-09-18T04:38:33.427 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 65.592901651000005, 2024-09-18T04:38:33.427 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016197989999999999, 2024-09-18T04:38:33.428 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:33.428 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:33.428 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:33.428 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:33.428 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:33.428 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 113 2024-09-18T04:38:33.428 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.428 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:33.428 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.428 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:33.428 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786816+0000", 2024-09-18T04:38:33.428 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:33.428 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.429 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.429 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:33.429 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786816+0000", 2024-09-18T04:38:33.429 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:33.429 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.429 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.429 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:33.429 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786815+0000", 2024-09-18T04:38:33.429 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:33.429 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.429 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.429 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:33.429 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786821+0000", 2024-09-18T04:38:33.430 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9529999999999996e-06 2024-09-18T04:38:33.430 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.430 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.430 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:33.430 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786841+0000", 2024-09-18T04:38:33.430 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9743000000000001e-05 2024-09-18T04:38:33.430 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.430 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.430 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:33.430 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786848+0000", 2024-09-18T04:38:33.430 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4900000000000003e-06 2024-09-18T04:38:33.430 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.430 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.430 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:33.431 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.799747+0000", 2024-09-18T04:38:33.431 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012898902 2024-09-18T04:38:33.431 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.431 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.431 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:33.431 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.799898+0000", 2024-09-18T04:38:33.431 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000150877 2024-09-18T04:38:33.431 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.431 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.431 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:33.431 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800095+0000", 2024-09-18T04:38:33.431 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000197023 2024-09-18T04:38:33.431 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.431 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.432 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:33.432 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802829+0000", 2024-09-18T04:38:33.432 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0027343480000000002 2024-09-18T04:38:33.432 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.432 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.432 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:33.432 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802847+0000", 2024-09-18T04:38:33.432 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7283e-05 2024-09-18T04:38:33.432 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.432 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.432 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:33.432 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802974+0000", 2024-09-18T04:38:33.432 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012749199999999999 2024-09-18T04:38:33.432 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.433 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.433 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:33.433 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803014+0000", 2024-09-18T04:38:33.433 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9576000000000002e-05 2024-09-18T04:38:33.433 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:33.433 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:33.433 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:33.433 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.433 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.433 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:33.433 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:38:33.433 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 65.592816060000004, 2024-09-18T04:38:33.433 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017399218000000001, 2024-09-18T04:38:33.434 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:33.434 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:33.434 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:33.434 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:33.434 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:33.434 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 114 2024-09-18T04:38:33.434 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.434 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:33.434 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.434 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:33.434 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:38:33.434 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:33.434 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.434 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.434 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:33.435 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:38:33.435 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:33.435 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.435 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.435 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:33.435 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786900+0000", 2024-09-18T04:38:33.435 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:33.435 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.435 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.435 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:33.435 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786908+0000", 2024-09-18T04:38:33.435 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8199999999999997e-06 2024-09-18T04:38:33.435 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.435 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.435 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:33.436 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786925+0000", 2024-09-18T04:38:33.436 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6849000000000001e-05 2024-09-18T04:38:33.436 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.436 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.436 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:33.436 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786934+0000", 2024-09-18T04:38:33.436 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6299999999999993e-06 2024-09-18T04:38:33.436 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.436 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.436 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:33.436 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800926+0000", 2024-09-18T04:38:33.436 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013991632 2024-09-18T04:38:33.436 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.436 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.436 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:33.437 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801060+0000", 2024-09-18T04:38:33.437 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013378199999999999 2024-09-18T04:38:33.437 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.437 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.437 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:33.437 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801166+0000", 2024-09-18T04:38:33.437 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010607099999999999 2024-09-18T04:38:33.437 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.437 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.437 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:33.437 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803026+0000", 2024-09-18T04:38:33.437 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001860469 2024-09-18T04:38:33.437 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.438 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.438 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:33.438 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804245+0000", 2024-09-18T04:38:33.438 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0012185900000000001 2024-09-18T04:38:33.438 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.438 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.438 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:33.438 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804278+0000", 2024-09-18T04:38:33.438 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3272000000000003e-05 2024-09-18T04:38:33.438 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.438 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.438 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:33.439 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804300+0000", 2024-09-18T04:38:33.439 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2266e-05 2024-09-18T04:38:33.439 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:33.439 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:33.439 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:33.439 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.439 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.439 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:33.439 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787566+0000", 2024-09-18T04:38:33.439 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 65.592151100999999, 2024-09-18T04:38:33.439 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017124120999999999, 2024-09-18T04:38:33.439 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:33.439 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:33.439 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:33.440 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:33.440 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:33.440 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 115 2024-09-18T04:38:33.440 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.440 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:33.440 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.440 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:33.440 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787566+0000", 2024-09-18T04:38:33.440 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:33.440 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.440 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.440 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:33.441 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787566+0000", 2024-09-18T04:38:33.441 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:33.441 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.441 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.441 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:33.441 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787564+0000", 2024-09-18T04:38:33.441 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:33.441 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.441 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.441 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:33.441 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787751+0000", 2024-09-18T04:38:33.441 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000186873 2024-09-18T04:38:33.441 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.441 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.442 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:33.442 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787772+0000", 2024-09-18T04:38:33.442 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.126e-05 2024-09-18T04:38:33.442 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.442 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.442 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:33.442 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787794+0000", 2024-09-18T04:38:33.442 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1418999999999999e-05 2024-09-18T04:38:33.442 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.442 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.442 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:33.442 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801659+0000", 2024-09-18T04:38:33.442 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013865125000000001 2024-09-18T04:38:33.443 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.443 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.443 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:33.443 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801750+0000", 2024-09-18T04:38:33.443 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.1427000000000004e-05 2024-09-18T04:38:33.443 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.443 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.443 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:33.443 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801894+0000", 2024-09-18T04:38:33.443 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000143514 2024-09-18T04:38:33.443 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.443 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.443 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:33.443 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804313+0000", 2024-09-18T04:38:33.443 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0024190879999999998 2024-09-18T04:38:33.444 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.444 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.444 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:33.444 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804644+0000", 2024-09-18T04:38:33.444 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00033100799999999999 2024-09-18T04:38:33.444 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.444 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.444 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:33.444 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804672+0000", 2024-09-18T04:38:33.444 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7988e-05 2024-09-18T04:38:33.444 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.444 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.444 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:33.444 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804690+0000", 2024-09-18T04:38:33.445 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8298999999999999e-05 2024-09-18T04:38:33.445 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:33.445 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:33.445 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:33.445 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.445 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.445 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:33.445 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:38:33.445 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 65.591903161999994, 2024-09-18T04:38:33.445 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017521662, 2024-09-18T04:38:33.445 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:33.445 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:33.445 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:33.445 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:33.446 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:33.446 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 116 2024-09-18T04:38:33.446 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.446 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:33.446 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.446 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:33.446 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:38:33.446 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:33.446 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.446 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.446 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:33.446 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:38:33.446 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:33.447 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.447 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.447 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:33.447 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787813+0000", 2024-09-18T04:38:33.447 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:33.447 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.447 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.447 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:33.447 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787824+0000", 2024-09-18T04:38:33.447 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1157e-05 2024-09-18T04:38:33.447 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.447 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.447 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:33.447 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787840+0000", 2024-09-18T04:38:33.447 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6269e-05 2024-09-18T04:38:33.448 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.448 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.448 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:33.448 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787853+0000", 2024-09-18T04:38:33.448 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3156000000000001e-05 2024-09-18T04:38:33.448 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.448 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.448 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:33.448 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803110+0000", 2024-09-18T04:38:33.448 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015256736 2024-09-18T04:38:33.448 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.448 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.448 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:33.449 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803197+0000", 2024-09-18T04:38:33.449 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7195999999999997e-05 2024-09-18T04:38:33.449 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.449 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.449 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:33.449 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803308+0000", 2024-09-18T04:38:33.449 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111005 2024-09-18T04:38:33.449 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.449 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.449 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:33.449 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804316+0000", 2024-09-18T04:38:33.449 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010073949999999999 2024-09-18T04:38:33.449 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.449 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.450 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:33.450 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805232+0000", 2024-09-18T04:38:33.450 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091636300000000001 2024-09-18T04:38:33.450 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.450 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.450 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:33.450 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805277+0000", 2024-09-18T04:38:33.450 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4864000000000003e-05 2024-09-18T04:38:33.450 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.450 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.450 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:33.450 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805336+0000", 2024-09-18T04:38:33.450 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8968000000000002e-05 2024-09-18T04:38:33.450 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:33.450 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:33.451 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:33.451 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.451 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.451 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:33.451 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:38:33.451 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 65.591851106999997, 2024-09-18T04:38:33.451 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017983453, 2024-09-18T04:38:33.451 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:33.451 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:33.451 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:33.451 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:33.451 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:33.451 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 117 2024-09-18T04:38:33.451 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.452 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:33.452 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.452 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:33.452 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:38:33.452 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:33.452 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.452 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.452 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:33.452 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:38:33.452 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:33.452 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.452 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.452 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:33.453 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787865+0000", 2024-09-18T04:38:33.453 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:33.453 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.453 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.453 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:33.453 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787889+0000", 2024-09-18T04:38:33.453 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4244999999999999e-05 2024-09-18T04:38:33.453 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.453 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.453 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:33.453 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787905+0000", 2024-09-18T04:38:33.453 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5739999999999998e-05 2024-09-18T04:38:33.453 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.453 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.454 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:33.454 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787918+0000", 2024-09-18T04:38:33.454 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.252e-05 2024-09-18T04:38:33.454 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.454 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.454 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:33.454 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803693+0000", 2024-09-18T04:38:33.454 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015774872999999998 2024-09-18T04:38:33.454 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.454 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.454 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:33.454 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803775+0000", 2024-09-18T04:38:33.454 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2761000000000003e-05 2024-09-18T04:38:33.454 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.454 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.455 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:33.455 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803887+0000", 2024-09-18T04:38:33.455 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111708 2024-09-18T04:38:33.455 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.455 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.455 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:33.455 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804823+0000", 2024-09-18T04:38:33.455 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00093625100000000003 2024-09-18T04:38:33.455 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.455 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.455 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:33.455 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805804+0000", 2024-09-18T04:38:33.455 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00098114699999999992 2024-09-18T04:38:33.456 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.456 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.456 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:33.456 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805831+0000", 2024-09-18T04:38:33.456 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6401999999999999e-05 2024-09-18T04:38:33.456 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.456 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.456 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:33.456 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805850+0000", 2024-09-18T04:38:33.456 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8862e-05 2024-09-18T04:38:33.456 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:33.456 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:33.456 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:33.457 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.457 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.457 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:33.457 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.230830+0000", 2024-09-18T04:38:33.457 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 65.148887228000007, 2024-09-18T04:38:33.457 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011537146999999999, 2024-09-18T04:38:33.457 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:33.457 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:33.457 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:33.457 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:33.457 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:33.457 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 135 2024-09-18T04:38:33.457 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.457 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:33.458 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.458 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:33.458 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230830+0000", 2024-09-18T04:38:33.458 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:33.458 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.458 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.458 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:33.458 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230830+0000", 2024-09-18T04:38:33.458 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:33.458 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.458 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.458 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:33.458 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230829+0000", 2024-09-18T04:38:33.458 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:33.459 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.459 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.459 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:33.459 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230835+0000", 2024-09-18T04:38:33.459 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8660000000000002e-06 2024-09-18T04:38:33.459 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.459 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.459 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:33.459 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230844+0000", 2024-09-18T04:38:33.459 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9339999999999995e-06 2024-09-18T04:38:33.459 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.459 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.459 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:33.459 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230853+0000", 2024-09-18T04:38:33.460 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4419999999999998e-06 2024-09-18T04:38:33.460 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.460 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.460 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:33.460 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.237658+0000", 2024-09-18T04:38:33.460 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0068055310000000001 2024-09-18T04:38:33.460 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.460 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.460 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:33.460 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.237837+0000", 2024-09-18T04:38:33.460 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017943200000000001 2024-09-18T04:38:33.460 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.460 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.461 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:33.461 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.238112+0000", 2024-09-18T04:38:33.461 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00027488600000000001 2024-09-18T04:38:33.461 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.461 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.461 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:33.461 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.240505+0000", 2024-09-18T04:38:33.461 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0023930539999999999 2024-09-18T04:38:33.461 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.461 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.461 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:33.461 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242157+0000", 2024-09-18T04:38:33.461 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001651135 2024-09-18T04:38:33.461 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.462 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.462 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:33.462 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242281+0000", 2024-09-18T04:38:33.462 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012451300000000001 2024-09-18T04:38:33.462 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.462 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.462 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:33.462 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242367+0000", 2024-09-18T04:38:33.462 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6178000000000001e-05 2024-09-18T04:38:33.462 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:33.462 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:33.462 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:33.462 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.462 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.463 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:33.463 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.230858+0000", 2024-09-18T04:38:33.463 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 65.148859340000001, 2024-09-18T04:38:33.463 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013005821000000001, 2024-09-18T04:38:33.463 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:33.463 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:33.463 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:33.463 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:33.463 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:33.463 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 136 2024-09-18T04:38:33.463 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.463 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:33.463 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.463 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:33.464 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230858+0000", 2024-09-18T04:38:33.464 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:33.464 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.464 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.464 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:33.464 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230858+0000", 2024-09-18T04:38:33.464 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:33.464 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.464 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.464 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:33.464 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230858+0000", 2024-09-18T04:38:33.464 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:33.464 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.465 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.465 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:33.465 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230861+0000", 2024-09-18T04:38:33.465 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7459999999999998e-06 2024-09-18T04:38:33.465 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.465 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.465 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:33.465 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230877+0000", 2024-09-18T04:38:33.465 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5979000000000001e-05 2024-09-18T04:38:33.465 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.465 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.465 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:33.465 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230885+0000", 2024-09-18T04:38:33.465 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.5979999999999999e-06 2024-09-18T04:38:33.466 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.466 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.466 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:33.466 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.239023+0000", 2024-09-18T04:38:33.466 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0081380710000000002 2024-09-18T04:38:33.466 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.466 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.466 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:33.466 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.239216+0000", 2024-09-18T04:38:33.466 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019274799999999999 2024-09-18T04:38:33.466 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.466 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.466 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:33.466 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.239417+0000", 2024-09-18T04:38:33.467 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020093300000000001 2024-09-18T04:38:33.467 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.467 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.467 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:33.467 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242494+0000", 2024-09-18T04:38:33.467 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003077177 2024-09-18T04:38:33.467 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.467 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.467 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:33.467 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.243668+0000", 2024-09-18T04:38:33.467 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001174027 2024-09-18T04:38:33.467 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.467 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.467 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:33.467 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.243777+0000", 2024-09-18T04:38:33.468 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010955600000000001 2024-09-18T04:38:33.468 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.468 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.468 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:33.468 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.243864+0000", 2024-09-18T04:38:33.468 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6441999999999997e-05 2024-09-18T04:38:33.468 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:33.468 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:33.468 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:33.468 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.468 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.468 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:33.468 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.230901+0000", 2024-09-18T04:38:33.468 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 65.148816767, 2024-09-18T04:38:33.469 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014120882, 2024-09-18T04:38:33.469 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:33.469 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:33.469 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:33.469 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:33.469 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:33.469 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 137 2024-09-18T04:38:33.469 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.469 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:33.469 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.469 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:33.469 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230901+0000", 2024-09-18T04:38:33.469 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:33.470 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.470 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.470 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:33.470 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230901+0000", 2024-09-18T04:38:33.470 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:33.470 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.470 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.470 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:33.470 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230900+0000", 2024-09-18T04:38:33.470 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:33.470 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.470 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.470 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:33.470 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231002+0000", 2024-09-18T04:38:33.471 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000102392 2024-09-18T04:38:33.471 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.471 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.471 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:33.471 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231029+0000", 2024-09-18T04:38:33.471 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7166999999999999e-05 2024-09-18T04:38:33.471 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.471 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.471 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:33.471 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231036+0000", 2024-09-18T04:38:33.471 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7259999999999996e-06 2024-09-18T04:38:33.471 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.471 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.471 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:33.472 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.240562+0000", 2024-09-18T04:38:33.472 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0095257970000000008 2024-09-18T04:38:33.472 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.472 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.472 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:33.472 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.240730+0000", 2024-09-18T04:38:33.472 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016767099999999999 2024-09-18T04:38:33.472 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.472 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.472 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:33.472 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.240927+0000", 2024-09-18T04:38:33.472 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019711899999999999 2024-09-18T04:38:33.472 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.473 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.473 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:33.473 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244885+0000", 2024-09-18T04:38:33.473 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0039577340000000001 2024-09-18T04:38:33.473 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.473 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.473 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:33.473 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244899+0000", 2024-09-18T04:38:33.473 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4403000000000001e-05 2024-09-18T04:38:33.473 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.473 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.473 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:33.473 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244973+0000", 2024-09-18T04:38:33.474 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4177999999999994e-05 2024-09-18T04:38:33.474 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.474 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.474 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:33.474 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.245021+0000", 2024-09-18T04:38:33.474 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8344999999999999e-05 2024-09-18T04:38:33.474 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:33.474 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:33.474 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:33.474 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.474 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.474 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:33.474 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:38:33.474 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 65.148673975999998, 2024-09-18T04:38:33.475 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016528505999999998, 2024-09-18T04:38:33.475 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:33.475 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:33.475 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:33.475 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:33.475 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:33.475 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 138 2024-09-18T04:38:33.475 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.475 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:33.475 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.475 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:33.475 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:38:33.475 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:33.475 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.476 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.476 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:33.476 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:38:33.476 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:33.476 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.476 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.476 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:33.476 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:38:33.476 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:33.476 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.476 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.476 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:33.476 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231047+0000", 2024-09-18T04:38:33.476 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5399999999999997e-06 2024-09-18T04:38:33.477 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.477 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.477 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:33.477 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231068+0000", 2024-09-18T04:38:33.477 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0769000000000001e-05 2024-09-18T04:38:33.477 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.477 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.477 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:33.477 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231074+0000", 2024-09-18T04:38:33.477 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6180000000000003e-06 2024-09-18T04:38:33.477 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.477 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.477 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:33.477 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242556+0000", 2024-09-18T04:38:33.478 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011482654 2024-09-18T04:38:33.478 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.478 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.478 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:33.478 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242666+0000", 2024-09-18T04:38:33.478 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011003200000000001 2024-09-18T04:38:33.478 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.478 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.478 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:33.478 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242924+0000", 2024-09-18T04:38:33.478 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00025785899999999999 2024-09-18T04:38:33.478 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.478 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.478 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:33.479 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247263+0000", 2024-09-18T04:38:33.479 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0043387649999999996 2024-09-18T04:38:33.479 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.479 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.479 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:33.479 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247412+0000", 2024-09-18T04:38:33.479 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000149276 2024-09-18T04:38:33.479 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.479 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.479 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:33.479 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247522+0000", 2024-09-18T04:38:33.479 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000109788 2024-09-18T04:38:33.479 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.479 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.479 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:33.480 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247572+0000", 2024-09-18T04:38:33.480 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9808999999999997e-05 2024-09-18T04:38:33.480 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:33.480 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:33.480 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:33.480 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.480 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.480 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:33.480 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231083+0000", 2024-09-18T04:38:33.480 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 65.148634513000005, 2024-09-18T04:38:33.480 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0166731, 2024-09-18T04:38:33.480 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:33.480 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:33.481 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:33.481 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:33.481 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:33.481 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 139 2024-09-18T04:38:33.481 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.481 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:33.481 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.481 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:33.481 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231083+0000", 2024-09-18T04:38:33.481 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:33.481 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.481 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.481 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:33.481 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231083+0000", 2024-09-18T04:38:33.482 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:33.482 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.482 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.482 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:33.482 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231082+0000", 2024-09-18T04:38:33.482 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:33.482 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.482 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.482 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:33.482 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231090+0000", 2024-09-18T04:38:33.482 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1280000000000008e-06 2024-09-18T04:38:33.482 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.482 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.482 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:33.483 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231097+0000", 2024-09-18T04:38:33.483 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5119999999999997e-06 2024-09-18T04:38:33.483 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.483 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.483 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:33.483 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231106+0000", 2024-09-18T04:38:33.483 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9230000000000008e-06 2024-09-18T04:38:33.483 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.483 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.483 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:33.483 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.243977+0000", 2024-09-18T04:38:33.483 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012870754 2024-09-18T04:38:33.483 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.483 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.484 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:33.484 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244112+0000", 2024-09-18T04:38:33.484 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000135575 2024-09-18T04:38:33.484 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.484 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.484 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:33.484 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244265+0000", 2024-09-18T04:38:33.484 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015279900000000001 2024-09-18T04:38:33.484 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.484 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.484 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:33.484 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247649+0000", 2024-09-18T04:38:33.484 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003383896 2024-09-18T04:38:33.484 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.485 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.485 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:33.485 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247656+0000", 2024-09-18T04:38:33.485 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8920000000000003e-06 2024-09-18T04:38:33.485 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.485 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.485 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:33.485 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247692+0000", 2024-09-18T04:38:33.485 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6470000000000001e-05 2024-09-18T04:38:33.485 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.485 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.485 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:33.485 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247756+0000", 2024-09-18T04:38:33.485 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3676999999999994e-05 2024-09-18T04:38:33.486 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:33.486 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:33.486 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:33.486 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.486 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.486 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:33.486 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231190+0000", 2024-09-18T04:38:33.486 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 65.148527587000004, 2024-09-18T04:38:33.486 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016793556000000001, 2024-09-18T04:38:33.486 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:33.486 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:33.486 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:33.486 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:33.486 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:33.487 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 140 2024-09-18T04:38:33.487 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.487 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:33.487 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.487 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:33.487 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231190+0000", 2024-09-18T04:38:33.487 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:33.487 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.487 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.487 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:33.487 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231190+0000", 2024-09-18T04:38:33.487 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:33.487 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.487 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.488 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:33.488 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231189+0000", 2024-09-18T04:38:33.488 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:33.488 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.488 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.488 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:33.488 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231273+0000", 2024-09-18T04:38:33.488 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4445999999999998e-05 2024-09-18T04:38:33.488 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.488 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.488 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:33.488 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231282+0000", 2024-09-18T04:38:33.488 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8859999999999993e-06 2024-09-18T04:38:33.488 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.489 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.489 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:33.489 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231288+0000", 2024-09-18T04:38:33.489 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9819999999999996e-06 2024-09-18T04:38:33.489 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.489 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.489 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:33.489 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.245119+0000", 2024-09-18T04:38:33.489 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.01383121 2024-09-18T04:38:33.489 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.489 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.489 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:33.489 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.245247+0000", 2024-09-18T04:38:33.489 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012763299999999999 2024-09-18T04:38:33.490 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.490 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.490 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:33.490 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.245401+0000", 2024-09-18T04:38:33.490 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015399200000000001 2024-09-18T04:38:33.490 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.490 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.490 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:33.490 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247766+0000", 2024-09-18T04:38:33.490 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0023649510000000001 2024-09-18T04:38:33.490 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.490 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.490 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:33.490 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247904+0000", 2024-09-18T04:38:33.491 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013836499999999999 2024-09-18T04:38:33.491 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.491 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.491 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:33.491 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247950+0000", 2024-09-18T04:38:33.491 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5201000000000003e-05 2024-09-18T04:38:33.491 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.491 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.491 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:33.491 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247983+0000", 2024-09-18T04:38:33.491 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3686000000000002e-05 2024-09-18T04:38:33.491 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:33.491 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:33.491 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:33.492 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.492 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.492 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:33.492 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:38:33.492 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 65.148416843999996, 2024-09-18T04:38:33.492 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017344024, 2024-09-18T04:38:33.492 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:33.492 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:33.492 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:33.492 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:33.492 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:33.492 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 141 2024-09-18T04:38:33.492 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.492 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:33.493 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.493 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:33.493 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:38:33.493 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:33.493 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.493 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.493 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:33.493 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:38:33.493 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:33.493 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.493 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.493 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:33.493 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:38:33.493 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:33.494 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.494 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.494 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:33.494 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231309+0000", 2024-09-18T04:38:33.494 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5419999999999994e-06 2024-09-18T04:38:33.494 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.494 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.494 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:33.494 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231323+0000", 2024-09-18T04:38:33.494 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3108000000000001e-05 2024-09-18T04:38:33.494 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.494 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.494 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:33.494 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231340+0000", 2024-09-18T04:38:33.495 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.696e-05 2024-09-18T04:38:33.495 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.495 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.495 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:33.495 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.246143+0000", 2024-09-18T04:38:33.495 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014803320999999999 2024-09-18T04:38:33.495 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.495 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.495 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:33.495 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.246323+0000", 2024-09-18T04:38:33.495 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00018061600000000001 2024-09-18T04:38:33.495 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.495 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.495 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:33.496 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.246530+0000", 2024-09-18T04:38:33.496 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020625 2024-09-18T04:38:33.496 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.496 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.496 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:33.496 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248002+0000", 2024-09-18T04:38:33.496 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0014719379999999999 2024-09-18T04:38:33.496 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.496 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.496 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:33.496 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248582+0000", 2024-09-18T04:38:33.496 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00058020599999999999 2024-09-18T04:38:33.496 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.496 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.497 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:33.497 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248618+0000", 2024-09-18T04:38:33.497 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6559e-05 2024-09-18T04:38:33.497 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.497 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.497 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:33.497 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248645+0000", 2024-09-18T04:38:33.497 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6080000000000001e-05 2024-09-18T04:38:33.497 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:33.497 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:33.497 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:33.497 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.497 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.498 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-18T04:38:33.498 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.878463+0000", 2024-09-18T04:38:33.498 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 64.501254810000006, 2024-09-18T04:38:33.498 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010882697, 2024-09-18T04:38:33.498 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:33.498 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:33.498 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:33.498 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:33.498 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:33.498 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 159 2024-09-18T04:38:33.498 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.498 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:33.498 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.498 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:33.499 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878463+0000", 2024-09-18T04:38:33.499 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:33.499 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.499 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.499 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:33.499 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878463+0000", 2024-09-18T04:38:33.499 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:33.499 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.499 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.499 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:33.499 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878462+0000", 2024-09-18T04:38:33.499 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:33.499 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.499 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.500 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:33.500 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878467+0000", 2024-09-18T04:38:33.500 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4099999999999999e-06 2024-09-18T04:38:33.500 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.500 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.500 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:33.500 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878490+0000", 2024-09-18T04:38:33.500 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3159999999999998e-05 2024-09-18T04:38:33.500 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.500 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.500 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:33.500 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878511+0000", 2024-09-18T04:38:33.500 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1092000000000001e-05 2024-09-18T04:38:33.500 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.501 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.501 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:33.501 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.887163+0000", 2024-09-18T04:38:33.501 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.008651845 2024-09-18T04:38:33.501 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.501 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.501 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:33.501 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.887273+0000", 2024-09-18T04:38:33.501 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000109823 2024-09-18T04:38:33.501 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.501 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.501 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:33.501 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.887441+0000", 2024-09-18T04:38:33.501 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016777599999999999 2024-09-18T04:38:33.502 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.502 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.502 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:33.502 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.888634+0000", 2024-09-18T04:38:33.502 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001192839 2024-09-18T04:38:33.502 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.502 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.502 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:33.502 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.889153+0000", 2024-09-18T04:38:33.502 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00051953700000000004 2024-09-18T04:38:33.502 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.502 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.502 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:33.502 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.889234+0000", 2024-09-18T04:38:33.503 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0802000000000003e-05 2024-09-18T04:38:33.503 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:33.503 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:33.503 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:33.503 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.889345+0000", 2024-09-18T04:38:33.503 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011112000000000001 2024-09-18T04:38:33.503 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:33.503 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:33.503 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:33.503 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:33.503 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:33.503 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:33.505 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:38:33.505 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:33.498+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:33.606 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:33.602+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:33.661 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:33.661 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:33.661 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:38:33.661 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:33.671 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:38:33.706 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:33.702+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:33.776 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:33.776 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:33.776 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:38:33.776 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:38:33.776 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:33.786 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:38:33.807 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:33.802+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:33.890 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:33.890 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:38:33.891 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:38:33.891 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [] 2024-09-18T04:38:33.891 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:33.900 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:38:33.908 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.2 to 1 (10000) 2024-09-18T04:38:33.922 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-18T04:38:33.923 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T04:38:33.942 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:33.938+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:34.009 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:34.002+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:34.020 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T04:38:34.053 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:34.053 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:34.053 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:38:34.053 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:34.062 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:38:34.109 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:34.102+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:34.164 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:38:34.167 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:34.168 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:34.168 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:38:34.168 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:38:34.168 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:34.177 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:38:34.186 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:38:34.207 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:38:34.218 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:34.214+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:34.228 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:38:34.251 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:38:34.274 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:38:34.276 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:38:34.276 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:38:34.276 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:38:34.276 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:38:34.276 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:38:34.276 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:38:34.278 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:38:34.283 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-18T04:38:34.295 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":77,"stamp":"2024-09-18T04:38:33.901917+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":5,"kb":566231040,"kb_used":185508,"kb_used_data":2908,"kb_used_omap":9,"kb_used_meta":182518,"kb_avail":566045532,"statfs":{"total":579820584960,"available":579630624768,"internally_reserved":0,"allocated":2977792,"data_stored":2193667,"data_compressed":66040,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":9561,"internal_metadata":186899111},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.032998"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":47,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T04:38:29.848426+0000","last_change":"2024-09-18T04:38:05.754820+0000","last_active":"2024-09-18T04:38:29.848426+0000","last_peered":"2024-09-18T04:38:29.848426+0000","last_clean":"2024-09-18T04:38:29.848426+0000","last_became_active":"2024-09-18T04:38:05.753962+0000","last_became_peered":"2024-09-18T04:38:05.753962+0000","last_unstale":"2024-09-18T04:38:29.848426+0000","last_undegraded":"2024-09-18T04:38:29.848426+0000","last_fullsized":"2024-09-18T04:38:29.848426+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:53:52.224098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047343899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":38,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T04:38:26.855304+0000","last_change":"2024-09-18T04:37:36.494193+0000","last_active":"2024-09-18T04:38:26.855304+0000","last_peered":"2024-09-18T04:38:26.855304+0000","last_clean":"2024-09-18T04:38:26.855304+0000","last_became_active":"2024-09-18T04:37:34.472940+0000","last_became_peered":"2024-09-18T04:37:34.472940+0000","last_unstale":"2024-09-18T04:38:26.855304+0000","last_undegraded":"2024-09-18T04:38:26.855304+0000","last_fullsized":"2024-09-18T04:38:26.855304+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:18:29.266815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070553900000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":25,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T04:38:28.865700+0000","last_change":"2024-09-18T04:37:59.284861+0000","last_active":"2024-09-18T04:38:28.865700+0000","last_peered":"2024-09-18T04:38:28.865700+0000","last_clean":"2024-09-18T04:38:28.865700+0000","last_became_active":"2024-09-18T04:37:59.283691+0000","last_became_peered":"2024-09-18T04:37:59.283691+0000","last_unstale":"2024-09-18T04:38:28.865700+0000","last_undegraded":"2024-09-18T04:38:28.865700+0000","last_fullsized":"2024-09-18T04:38:28.865700+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:19:36.536686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":38,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T04:38:26.855424+0000","last_change":"2024-09-18T04:37:36.494228+0000","last_active":"2024-09-18T04:38:26.855424+0000","last_peered":"2024-09-18T04:38:26.855424+0000","last_clean":"2024-09-18T04:38:26.855424+0000","last_became_active":"2024-09-18T04:37:34.471265+0000","last_became_peered":"2024-09-18T04:37:34.471265+0000","last_unstale":"2024-09-18T04:38:26.855424+0000","last_undegraded":"2024-09-18T04:38:26.855424+0000","last_fullsized":"2024-09-18T04:38:26.855424+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:23:05.748076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081514499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"20'2","reported_seq":41,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T04:38:29.944201+0000","last_change":"2024-09-18T04:37:36.490461+0000","last_active":"2024-09-18T04:38:29.944201+0000","last_peered":"2024-09-18T04:38:29.944201+0000","last_clean":"2024-09-18T04:38:29.944201+0000","last_became_active":"2024-09-18T04:37:34.473088+0000","last_became_peered":"2024-09-18T04:37:34.473088+0000","last_unstale":"2024-09-18T04:38:29.944201+0000","last_undegraded":"2024-09-18T04:38:29.944201+0000","last_fullsized":"2024-09-18T04:38:29.944201+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:17:57.352397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001879932,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":29,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T04:38:28.865689+0000","last_change":"2024-09-18T04:37:51.679120+0000","last_active":"2024-09-18T04:38:28.865689+0000","last_peered":"2024-09-18T04:38:28.865689+0000","last_clean":"2024-09-18T04:38:28.865689+0000","last_became_active":"2024-09-18T04:37:51.678711+0000","last_became_peered":"2024-09-18T04:37:51.678711+0000","last_unstale":"2024-09-18T04:38:28.865689+0000","last_undegraded":"2024-09-18T04:38:28.865689+0000","last_fullsized":"2024-09-18T04:38:28.865689+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T04:46:09.459905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":39,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T04:38:29.848530+0000","last_change":"2024-09-18T04:37:36.484950+0000","last_active":"2024-09-18T04:38:29.848530+0000","last_peered":"2024-09-18T04:38:29.848530+0000","last_clean":"2024-09-18T04:38:29.848530+0000","last_became_active":"2024-09-18T04:37:34.470841+0000","last_became_peered":"2024-09-18T04:37:34.470841+0000","last_unstale":"2024-09-18T04:38:29.848530+0000","last_undegraded":"2024-09-18T04:38:29.848530+0000","last_fullsized":"2024-09-18T04:38:29.848530+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:16:34.528783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090513900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"18'1","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T04:38:29.944151+0000","last_change":"2024-09-18T04:37:51.681357+0000","last_active":"2024-09-18T04:38:29.944151+0000","last_peered":"2024-09-18T04:38:29.944151+0000","last_clean":"2024-09-18T04:38:29.944151+0000","last_became_active":"2024-09-18T04:37:51.676131+0000","last_became_peered":"2024-09-18T04:37:51.676131+0000","last_unstale":"2024-09-18T04:38:29.944151+0000","last_undegraded":"2024-09-18T04:38:29.944151+0000","last_fullsized":"2024-09-18T04:38:29.944151+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:45:22.605009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018293420000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":187,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T04:38:29.848504+0000","last_change":"2024-09-18T04:37:27.423069+0000","last_active":"2024-09-18T04:38:29.848504+0000","last_peered":"2024-09-18T04:38:29.848504+0000","last_clean":"2024-09-18T04:38:29.848504+0000","last_became_active":"2024-09-18T04:37:27.422350+0000","last_became_peered":"2024-09-18T04:37:27.422350+0000","last_unstale":"2024-09-18T04:38:29.848504+0000","last_undegraded":"2024-09-18T04:38:29.848504+0000","last_fullsized":"2024-09-18T04:38:29.848504+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:26.406292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:26.406292+0000","last_clean_scrub_stamp":"2024-09-18T04:37:26.406292+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:31:45.852149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":66040,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":13,"seq":55834574865,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27496,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344344,"statfs":{"total":96636764160,"available":96608608256,"internally_reserved":0,"allocated":487424,"data_stored":172702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:38:26 2024","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.873,"15min":0.873},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.5249999999999999,"5min":1.5249999999999999,"15min":1.5249999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.369,"5min":1.369,"15min":1.369},"last":0.61899999999999999}]},{"osd":1,"last update":"Wed Sep 18 04:38:26 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.66500000000000004}]},{"osd":2,"last update":"Wed Sep 18 04:38:26 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.4930000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.64900000000000002}]},{"osd":3,"last update":"Wed Sep 18 04:38:26 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.22,"5min":1.22,"15min":1.22},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.109,"5min":1.109,"15min":1.109},"last":0.495}]},{"osd":4,"last update":"Wed Sep 18 04:38:26 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.121,"5min":1.121,"15min":1.121},"last":0.68500000000000005}]}]},{"osd":4,"up_from":13,"seq":55834574865,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27492,"kb_used_data":472,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344348,"statfs":{"total":96636764160,"available":96608612352,"internally_reserved":0,"allocated":483328,"data_stored":172683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.175,"5min":1.175,"15min":1.175},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.258,"5min":1.258,"15min":1.258},"last":0.85599999999999998}]},{"osd":1,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.97599999999999998}]},{"osd":2,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.93899999999999995},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.619,"5min":1.619,"15min":1.619},"last":0.78400000000000003}]},{"osd":3,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.73699999999999999}]},{"osd":5,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":1.0649999999999999},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.214,"5min":1.214,"15min":1.214},"last":1.0149999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574866,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35748,"kb_used_data":536,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94336092,"statfs":{"total":96636764160,"available":96600158208,"internally_reserved":0,"allocated":548864,"data_stored":763051,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 04:38:28 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.113,"5min":1.113,"15min":1.113},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.59899999999999998}]},{"osd":2,"last update":"Wed Sep 18 04:38:28 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.8180000000000001,"5min":1.8180000000000001,"15min":1.8180000000000001},"last":0.76800000000000002}]},{"osd":3,"last update":"Wed Sep 18 04:38:28 2024","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":1.169,"5min":1.169,"15min":1.169},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.97699999999999998}]},{"osd":4,"last update":"Wed Sep 18 04:38:28 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":1.0600000000000001},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":1.2529999999999999}]},{"osd":5,"last update":"Wed Sep 18 04:38:28 2024","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.8919999999999999,"5min":1.8919999999999999,"15min":1.8919999999999999},"last":1.1479999999999999},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":1.224,"5min":1.224,"15min":1.224},"last":1.3140000000000001}]}]},{"osd":3,"up_from":12,"seq":51539607569,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31628,"kb_used_data":512,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340212,"statfs":{"total":96636764160,"available":96604377088,"internally_reserved":0,"allocated":524288,"data_stored":753769,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":2.665,"5min":2.665,"15min":2.665},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.64200000000000002}]},{"osd":1,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":2.4510000000000001,"5min":2.4510000000000001,"15min":2.4510000000000001},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.59599999999999997}]},{"osd":2,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":2.577,"5min":2.577,"15min":2.577},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.78000000000000003}]},{"osd":4,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.80600000000000005}]},{"osd":5,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.85899999999999999}]}]},{"osd":2,"up_from":12,"seq":51539607567,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31576,"kb_used_data":460,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340264,"statfs":{"total":96636764160,"available":96604430336,"internally_reserved":0,"allocated":471040,"data_stored":168042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.74399999999999999}]},{"osd":1,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.624},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.52400000000000002}]},{"osd":3,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.84299999999999997}]},{"osd":4,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.879,"5min":0.879,"15min":0.879},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.76900000000000002}]},{"osd":5,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.93799999999999994}]}]},{"osd":1,"up_from":12,"seq":51539607569,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31568,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340272,"statfs":{"total":96636764160,"available":96604438528,"internally_reserved":0,"allocated":462848,"data_stored":163420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.435},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.48799999999999999}]},{"osd":2,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.327,"5min":1.327,"15min":1.327},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.40899999999999997}]},{"osd":3,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.121,"5min":1.121,"15min":1.121},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.873,"5min":0.873,"15min":0.873},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.71699999999999997}]},{"osd":4,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":0.89900000000000002},{"interface":"front","average":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.117,"5min":1.117,"15min":1.117},"last":0.99299999999999999}]},{"osd":5,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":1.3859999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.80000000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T04:38:34.297 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T04:38:34.310 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:34.306+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:34.324 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:34.324 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:34.335 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:38:34.411 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:34.406+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:34.448 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:34.448 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:34.458 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:38:34.512 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:34.506+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:34.572 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:34.572 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:34.582 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:38:34.612 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:38:34.614 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:34.610+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:34.615 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-18T04:38:34.626 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":77,"stamp":"2024-09-18T04:38:33.901917+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":5,"kb":566231040,"kb_used":185508,"kb_used_data":2908,"kb_used_omap":9,"kb_used_meta":182518,"kb_avail":566045532,"statfs":{"total":579820584960,"available":579630624768,"internally_reserved":0,"allocated":2977792,"data_stored":2193667,"data_compressed":66040,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":9561,"internal_metadata":186899111},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.032998"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":47,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T04:38:29.848426+0000","last_change":"2024-09-18T04:38:05.754820+0000","last_active":"2024-09-18T04:38:29.848426+0000","last_peered":"2024-09-18T04:38:29.848426+0000","last_clean":"2024-09-18T04:38:29.848426+0000","last_became_active":"2024-09-18T04:38:05.753962+0000","last_became_peered":"2024-09-18T04:38:05.753962+0000","last_unstale":"2024-09-18T04:38:29.848426+0000","last_undegraded":"2024-09-18T04:38:29.848426+0000","last_fullsized":"2024-09-18T04:38:29.848426+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:53:52.224098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047343899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":38,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T04:38:26.855304+0000","last_change":"2024-09-18T04:37:36.494193+0000","last_active":"2024-09-18T04:38:26.855304+0000","last_peered":"2024-09-18T04:38:26.855304+0000","last_clean":"2024-09-18T04:38:26.855304+0000","last_became_active":"2024-09-18T04:37:34.472940+0000","last_became_peered":"2024-09-18T04:37:34.472940+0000","last_unstale":"2024-09-18T04:38:26.855304+0000","last_undegraded":"2024-09-18T04:38:26.855304+0000","last_fullsized":"2024-09-18T04:38:26.855304+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:18:29.266815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070553900000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":25,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T04:38:28.865700+0000","last_change":"2024-09-18T04:37:59.284861+0000","last_active":"2024-09-18T04:38:28.865700+0000","last_peered":"2024-09-18T04:38:28.865700+0000","last_clean":"2024-09-18T04:38:28.865700+0000","last_became_active":"2024-09-18T04:37:59.283691+0000","last_became_peered":"2024-09-18T04:37:59.283691+0000","last_unstale":"2024-09-18T04:38:28.865700+0000","last_undegraded":"2024-09-18T04:38:28.865700+0000","last_fullsized":"2024-09-18T04:38:28.865700+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:19:36.536686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":38,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T04:38:26.855424+0000","last_change":"2024-09-18T04:37:36.494228+0000","last_active":"2024-09-18T04:38:26.855424+0000","last_peered":"2024-09-18T04:38:26.855424+0000","last_clean":"2024-09-18T04:38:26.855424+0000","last_became_active":"2024-09-18T04:37:34.471265+0000","last_became_peered":"2024-09-18T04:37:34.471265+0000","last_unstale":"2024-09-18T04:38:26.855424+0000","last_undegraded":"2024-09-18T04:38:26.855424+0000","last_fullsized":"2024-09-18T04:38:26.855424+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:23:05.748076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081514499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"20'2","reported_seq":41,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T04:38:29.944201+0000","last_change":"2024-09-18T04:37:36.490461+0000","last_active":"2024-09-18T04:38:29.944201+0000","last_peered":"2024-09-18T04:38:29.944201+0000","last_clean":"2024-09-18T04:38:29.944201+0000","last_became_active":"2024-09-18T04:37:34.473088+0000","last_became_peered":"2024-09-18T04:37:34.473088+0000","last_unstale":"2024-09-18T04:38:29.944201+0000","last_undegraded":"2024-09-18T04:38:29.944201+0000","last_fullsized":"2024-09-18T04:38:29.944201+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:17:57.352397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001879932,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":29,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T04:38:28.865689+0000","last_change":"2024-09-18T04:37:51.679120+0000","last_active":"2024-09-18T04:38:28.865689+0000","last_peered":"2024-09-18T04:38:28.865689+0000","last_clean":"2024-09-18T04:38:28.865689+0000","last_became_active":"2024-09-18T04:37:51.678711+0000","last_became_peered":"2024-09-18T04:37:51.678711+0000","last_unstale":"2024-09-18T04:38:28.865689+0000","last_undegraded":"2024-09-18T04:38:28.865689+0000","last_fullsized":"2024-09-18T04:38:28.865689+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T04:46:09.459905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":39,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T04:38:29.848530+0000","last_change":"2024-09-18T04:37:36.484950+0000","last_active":"2024-09-18T04:38:29.848530+0000","last_peered":"2024-09-18T04:38:29.848530+0000","last_clean":"2024-09-18T04:38:29.848530+0000","last_became_active":"2024-09-18T04:37:34.470841+0000","last_became_peered":"2024-09-18T04:37:34.470841+0000","last_unstale":"2024-09-18T04:38:29.848530+0000","last_undegraded":"2024-09-18T04:38:29.848530+0000","last_fullsized":"2024-09-18T04:38:29.848530+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:16:34.528783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090513900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"18'1","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T04:38:29.944151+0000","last_change":"2024-09-18T04:37:51.681357+0000","last_active":"2024-09-18T04:38:29.944151+0000","last_peered":"2024-09-18T04:38:29.944151+0000","last_clean":"2024-09-18T04:38:29.944151+0000","last_became_active":"2024-09-18T04:37:51.676131+0000","last_became_peered":"2024-09-18T04:37:51.676131+0000","last_unstale":"2024-09-18T04:38:29.944151+0000","last_undegraded":"2024-09-18T04:38:29.944151+0000","last_fullsized":"2024-09-18T04:38:29.944151+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:45:22.605009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018293420000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":187,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T04:38:29.848504+0000","last_change":"2024-09-18T04:37:27.423069+0000","last_active":"2024-09-18T04:38:29.848504+0000","last_peered":"2024-09-18T04:38:29.848504+0000","last_clean":"2024-09-18T04:38:29.848504+0000","last_became_active":"2024-09-18T04:37:27.422350+0000","last_became_peered":"2024-09-18T04:37:27.422350+0000","last_unstale":"2024-09-18T04:38:29.848504+0000","last_undegraded":"2024-09-18T04:38:29.848504+0000","last_fullsized":"2024-09-18T04:38:29.848504+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:26.406292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:26.406292+0000","last_clean_scrub_stamp":"2024-09-18T04:37:26.406292+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:31:45.852149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":66040,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":13,"seq":55834574865,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27496,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344344,"statfs":{"total":96636764160,"available":96608608256,"internally_reserved":0,"allocated":487424,"data_stored":172702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:38:26 2024","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.873,"15min":0.873},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.5249999999999999,"5min":1.5249999999999999,"15min":1.5249999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.369,"5min":1.369,"15min":1.369},"last":0.61899999999999999}]},{"osd":1,"last update":"Wed Sep 18 04:38:26 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.66500000000000004}]},{"osd":2,"last update":"Wed Sep 18 04:38:26 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.4930000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.64900000000000002}]},{"osd":3,"last update":"Wed Sep 18 04:38:26 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.22,"5min":1.22,"15min":1.22},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.109,"5min":1.109,"15min":1.109},"last":0.495}]},{"osd":4,"last update":"Wed Sep 18 04:38:26 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.121,"5min":1.121,"15min":1.121},"last":0.68500000000000005}]}]},{"osd":4,"up_from":13,"seq":55834574865,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27492,"kb_used_data":472,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344348,"statfs":{"total":96636764160,"available":96608612352,"internally_reserved":0,"allocated":483328,"data_stored":172683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.175,"5min":1.175,"15min":1.175},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.258,"5min":1.258,"15min":1.258},"last":0.85599999999999998}]},{"osd":1,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.97599999999999998}]},{"osd":2,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.93899999999999995},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.619,"5min":1.619,"15min":1.619},"last":0.78400000000000003}]},{"osd":3,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.73699999999999999}]},{"osd":5,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":1.0649999999999999},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.214,"5min":1.214,"15min":1.214},"last":1.0149999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574866,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35748,"kb_used_data":536,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94336092,"statfs":{"total":96636764160,"available":96600158208,"internally_reserved":0,"allocated":548864,"data_stored":763051,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 04:38:28 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.113,"5min":1.113,"15min":1.113},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.59899999999999998}]},{"osd":2,"last update":"Wed Sep 18 04:38:28 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.8180000000000001,"5min":1.8180000000000001,"15min":1.8180000000000001},"last":0.76800000000000002}]},{"osd":3,"last update":"Wed Sep 18 04:38:28 2024","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":1.169,"5min":1.169,"15min":1.169},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.97699999999999998}]},{"osd":4,"last update":"Wed Sep 18 04:38:28 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":1.0600000000000001},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":1.2529999999999999}]},{"osd":5,"last update":"Wed Sep 18 04:38:28 2024","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.8919999999999999,"5min":1.8919999999999999,"15min":1.8919999999999999},"last":1.1479999999999999},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":1.224,"5min":1.224,"15min":1.224},"last":1.3140000000000001}]}]},{"osd":3,"up_from":12,"seq":51539607569,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31628,"kb_used_data":512,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340212,"statfs":{"total":96636764160,"available":96604377088,"internally_reserved":0,"allocated":524288,"data_stored":753769,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":2.665,"5min":2.665,"15min":2.665},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.64200000000000002}]},{"osd":1,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":2.4510000000000001,"5min":2.4510000000000001,"15min":2.4510000000000001},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.59599999999999997}]},{"osd":2,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":2.577,"5min":2.577,"15min":2.577},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.78000000000000003}]},{"osd":4,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.80600000000000005}]},{"osd":5,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.85899999999999999}]}]},{"osd":2,"up_from":12,"seq":51539607567,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31576,"kb_used_data":460,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340264,"statfs":{"total":96636764160,"available":96604430336,"internally_reserved":0,"allocated":471040,"data_stored":168042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.74399999999999999}]},{"osd":1,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.624},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.52400000000000002}]},{"osd":3,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.84299999999999997}]},{"osd":4,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.879,"5min":0.879,"15min":0.879},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.76900000000000002}]},{"osd":5,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.93799999999999994}]}]},{"osd":1,"up_from":12,"seq":51539607569,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31568,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340272,"statfs":{"total":96636764160,"available":96604438528,"internally_reserved":0,"allocated":462848,"data_stored":163420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.435},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.48799999999999999}]},{"osd":2,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.327,"5min":1.327,"15min":1.327},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.40899999999999997}]},{"osd":3,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.121,"5min":1.121,"15min":1.121},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.873,"5min":0.873,"15min":0.873},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.71699999999999997}]},{"osd":4,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":0.89900000000000002},{"interface":"front","average":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.117,"5min":1.117,"15min":1.117},"last":0.99299999999999999}]},{"osd":5,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":1.3859999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.80000000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T04:38:34.628 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-18T04:38:34.628 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T04:38:34.628 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-09-18T04:38:34.628 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-18T04:38:34.628 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-18T04:38:34.629 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config set filestore_inject_stall 3 2024-09-18T04:38:34.714 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:34.710+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:34.717 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:34.717 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:34.727 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:38:34.779 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:34.779 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-18T04:38:34.779 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:34.815 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:34.810+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:34.830 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:34.830 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:34.840 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:38:34.916 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:34.910+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:34.920 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-18T04:38:35.010 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:35.010 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:35.017 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:35.010+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:35.020 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:38:35.118 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:35.114+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:35.181 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:35.181 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:35.191 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:38:35.219 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:35.214+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:35.297 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:35.297 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:35.307 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:38:35.320 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:35.314+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:35.410 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:35.411 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:35.421 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:38:35.421 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:35.414+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:35.521 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:35.514+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:35.526 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:35.526 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:35.536 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:38:35.622 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:35.614+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:35.652 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:35.652 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:35.663 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:38:35.722 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:35.718+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:35.771 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:35.772 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:35.781 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:38:35.823 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:35.818+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:35.888 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:35.888 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:35.898 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:38:35.920 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-09-18T04:38:35.942 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:35.938+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:36.004 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:36.004 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:36.014 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:38:36.024 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:36.018+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:36.121 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:36.122 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:36.125 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:36.118+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:36.131 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:38:36.226 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:36.222+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:36.240 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:36.240 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:36.250 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:38:36.291 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T04:38:36.327 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:36.322+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:36.406 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:36.406 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:36.416 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:38:36.427 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:36.422+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:36.451 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:38:36.472 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:38:36.493 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:38:36.516 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:38:36.522 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:36.522 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:36.529 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:36.522+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:36.532 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:38:36.537 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:38:36.559 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:38:36.561 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:38:36.562 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:38:36.562 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:38:36.562 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:38:36.562 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:38:36.562 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:38:36.632 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:36.626+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:36.704 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:36.705 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:38:36.705 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:38:36.705 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:38:36.705 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.705 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-18T04:38:36.705 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:38:36.705 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 62.168155910000003, 2024-09-18T04:38:36.705 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0095781189999999995, 2024-09-18T04:38:36.705 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:36.705 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:36.705 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:36.706 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:38:36.706 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:38:36.706 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 1 2024-09-18T04:38:36.706 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.706 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:36.706 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.706 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:36.706 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:38:36.706 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:36.706 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.706 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.706 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:36.706 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:38:36.706 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:36.707 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.707 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.707 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:36.707 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469862+0000", 2024-09-18T04:38:36.707 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:36.707 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.707 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.707 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:36.707 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469879+0000", 2024-09-18T04:38:36.707 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7431999999999998e-05 2024-09-18T04:38:36.707 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.707 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.707 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:36.708 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469896+0000", 2024-09-18T04:38:36.708 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6742999999999999e-05 2024-09-18T04:38:36.708 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.708 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.708 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:36.708 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469915+0000", 2024-09-18T04:38:36.708 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8991000000000002e-05 2024-09-18T04:38:36.708 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.708 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.708 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:36.708 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470397+0000", 2024-09-18T04:38:36.708 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048167799999999998 2024-09-18T04:38:36.708 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.708 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.709 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-09-18T04:38:36.709 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470412+0000", 2024-09-18T04:38:36.709 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5526e-05 2024-09-18T04:38:36.709 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.709 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.709 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:36.709 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473279+0000", 2024-09-18T04:38:36.709 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0028667469999999998 2024-09-18T04:38:36.709 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.709 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.709 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:36.709 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473441+0000", 2024-09-18T04:38:36.709 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000162082 2024-09-18T04:38:36.710 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.710 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.710 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:38:36.710 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473573+0000", 2024-09-18T04:38:36.710 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000132032 2024-09-18T04:38:36.710 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.710 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.710 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:36.710 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474826+0000", 2024-09-18T04:38:36.710 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001252717 2024-09-18T04:38:36.710 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.710 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.710 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:36.710 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479307+0000", 2024-09-18T04:38:36.711 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044813220000000003 2024-09-18T04:38:36.711 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.711 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.711 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:36.711 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479366+0000", 2024-09-18T04:38:36.711 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9024999999999998e-05 2024-09-18T04:38:36.711 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.711 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.711 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:36.711 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479443+0000", 2024-09-18T04:38:36.711 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6339000000000002e-05 2024-09-18T04:38:36.711 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:36.711 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:36.711 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:36.712 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.712 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.712 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.4214.0:1 2.3 e18/17 ondisk, result = 0)", 2024-09-18T04:38:36.712 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.474724+0000", 2024-09-18T04:38:36.712 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 62.163296545999998, 2024-09-18T04:38:36.712 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000105358, 2024-09-18T04:38:36.712 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:36.712 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:38:36.712 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:36.712 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.712 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:36.712 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474724+0000", 2024-09-18T04:38:36.712 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:36.713 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.713 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.713 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:36.713 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474724+0000", 2024-09-18T04:38:36.713 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:36.713 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.713 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.713 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:36.713 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474723+0000", 2024-09-18T04:38:36.713 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:36.713 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.713 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.713 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:36.713 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474731+0000", 2024-09-18T04:38:36.714 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2339999999999994e-06 2024-09-18T04:38:36.714 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.714 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.714 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:36.714 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474748+0000", 2024-09-18T04:38:36.714 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6603e-05 2024-09-18T04:38:36.714 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.714 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.714 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:36.714 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474760+0000", 2024-09-18T04:38:36.714 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2215000000000001e-05 2024-09-18T04:38:36.714 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.714 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.715 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:36.715 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474814+0000", 2024-09-18T04:38:36.715 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4267000000000001e-05 2024-09-18T04:38:36.715 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.715 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.715 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:36.715 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474821+0000", 2024-09-18T04:38:36.715 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0929999999999997e-06 2024-09-18T04:38:36.715 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.715 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.715 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:36.715 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474829+0000", 2024-09-18T04:38:36.715 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1200000000000002e-06 2024-09-18T04:38:36.715 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:36.716 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:36.716 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:36.716 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.716 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.716 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18)", 2024-09-18T04:38:36.716 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.482346+0000", 2024-09-18T04:38:36.716 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 62.155674114, 2024-09-18T04:38:36.716 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020435599999999999, 2024-09-18T04:38:36.716 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:36.716 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-09-18T04:38:36.716 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:36.716 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:38:36.717 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:38:36.717 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 2 2024-09-18T04:38:36.717 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.717 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:36.717 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.717 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:36.717 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482346+0000", 2024-09-18T04:38:36.717 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:36.717 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.717 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.717 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:36.717 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482346+0000", 2024-09-18T04:38:36.718 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:36.718 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.718 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.718 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:36.718 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482345+0000", 2024-09-18T04:38:36.718 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:36.718 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.718 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.718 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:36.718 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482360+0000", 2024-09-18T04:38:36.718 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4854999999999999e-05 2024-09-18T04:38:36.718 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.718 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.718 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:36.719 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482377+0000", 2024-09-18T04:38:36.719 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7785999999999999e-05 2024-09-18T04:38:36.719 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.719 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.719 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:36.719 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482387+0000", 2024-09-18T04:38:36.719 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6800000000000005e-06 2024-09-18T04:38:36.719 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.719 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.719 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:36.719 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482441+0000", 2024-09-18T04:38:36.719 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4046999999999999e-05 2024-09-18T04:38:36.720 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.720 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.720 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:36.720 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482551+0000", 2024-09-18T04:38:36.720 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000109468 2024-09-18T04:38:36.720 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:36.720 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:36.720 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:36.720 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.720 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.720 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18)", 2024-09-18T04:38:36.720 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.482853+0000", 2024-09-18T04:38:36.721 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 62.155167495999997, 2024-09-18T04:38:36.721 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00029815799999999997, 2024-09-18T04:38:36.721 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:36.721 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-09-18T04:38:36.721 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:36.721 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:38:36.721 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:38:36.721 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 3 2024-09-18T04:38:36.721 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.721 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:36.721 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.722 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:36.722 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482853+0000", 2024-09-18T04:38:36.722 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:36.722 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.722 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.722 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:36.722 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482853+0000", 2024-09-18T04:38:36.722 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:36.722 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.722 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.722 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:36.722 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482852+0000", 2024-09-18T04:38:36.722 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:36.723 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.723 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.723 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:36.723 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482859+0000", 2024-09-18T04:38:36.723 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0380000000000002e-06 2024-09-18T04:38:36.723 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.723 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.723 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:36.723 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482880+0000", 2024-09-18T04:38:36.723 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1243e-05 2024-09-18T04:38:36.723 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.723 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.724 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:36.724 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482896+0000", 2024-09-18T04:38:36.724 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5977e-05 2024-09-18T04:38:36.724 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.724 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.724 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:36.724 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482970+0000", 2024-09-18T04:38:36.724 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4603000000000004e-05 2024-09-18T04:38:36.724 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.724 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.724 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:36.724 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.483151+0000", 2024-09-18T04:38:36.725 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00018059799999999999 2024-09-18T04:38:36.725 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:36.725 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:36.725 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:36.725 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.725 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.725 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-18T04:38:36.725 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:35.468698+0000", 2024-09-18T04:38:36.725 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 61.169322659000002, 2024-09-18T04:38:36.725 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0039954129999999997, 2024-09-18T04:38:36.725 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:36.725 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:36.725 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:36.726 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:38:36.726 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:38:36.726 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 5 2024-09-18T04:38:36.726 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.726 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:36.726 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.726 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:36.726 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468698+0000", 2024-09-18T04:38:36.726 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:36.726 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.726 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.726 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:36.726 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468698+0000", 2024-09-18T04:38:36.726 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:36.726 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.727 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.727 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:36.727 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468696+0000", 2024-09-18T04:38:36.727 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:36.727 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.727 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.727 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:36.727 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468714+0000", 2024-09-18T04:38:36.727 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8743e-05 2024-09-18T04:38:36.727 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.727 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.727 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:36.727 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468748+0000", 2024-09-18T04:38:36.728 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3339e-05 2024-09-18T04:38:36.728 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.728 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.728 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:36.728 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468771+0000", 2024-09-18T04:38:36.728 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3119999999999999e-05 2024-09-18T04:38:36.728 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.728 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.728 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:36.728 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468922+0000", 2024-09-18T04:38:36.728 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000151383 2024-09-18T04:38:36.728 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.728 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.729 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:36.729 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.469278+0000", 2024-09-18T04:38:36.729 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00035628599999999998 2024-09-18T04:38:36.729 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.729 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.729 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:36.729 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.469629+0000", 2024-09-18T04:38:36.729 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00035014300000000003 2024-09-18T04:38:36.729 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.729 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.729 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:36.729 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.471620+0000", 2024-09-18T04:38:36.729 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001991278 2024-09-18T04:38:36.729 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.730 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.730 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:36.730 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472440+0000", 2024-09-18T04:38:36.730 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00081998599999999998 2024-09-18T04:38:36.730 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.730 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.730 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:36.730 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472592+0000", 2024-09-18T04:38:36.730 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015227400000000001 2024-09-18T04:38:36.730 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.730 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.730 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:36.730 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472693+0000", 2024-09-18T04:38:36.731 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000101074 2024-09-18T04:38:36.731 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:36.731 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:36.731 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:36.731 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.731 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.731 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.4214.0:5 2.2 e18/17 ondisk, result = 0)", 2024-09-18T04:38:36.731 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:35.472226+0000", 2024-09-18T04:38:36.731 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 61.165794759000001, 2024-09-18T04:38:36.731 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048505299999999999, 2024-09-18T04:38:36.731 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:36.731 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:38:36.731 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:36.731 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.732 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:36.732 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472226+0000", 2024-09-18T04:38:36.732 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:36.732 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.732 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.732 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:36.732 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472226+0000", 2024-09-18T04:38:36.732 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:36.732 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.732 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.732 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:36.732 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472224+0000", 2024-09-18T04:38:36.733 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:36.733 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.733 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.733 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:36.733 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472238+0000", 2024-09-18T04:38:36.733 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.468e-05 2024-09-18T04:38:36.733 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.733 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.733 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:36.733 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472272+0000", 2024-09-18T04:38:36.736 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3586e-05 2024-09-18T04:38:36.736 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.736 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.736 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:36.736 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472297+0000", 2024-09-18T04:38:36.736 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5282999999999998e-05 2024-09-18T04:38:36.737 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.737 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.737 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:36.737 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472396+0000", 2024-09-18T04:38:36.737 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.8394999999999996e-05 2024-09-18T04:38:36.737 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.737 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.737 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:36.738 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472421+0000", 2024-09-18T04:38:36.738 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5479000000000001e-05 2024-09-18T04:38:36.738 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.738 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.738 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:36.738 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472711+0000", 2024-09-18T04:38:36.738 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000289714 2024-09-18T04:38:36.738 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:36.738 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:36.738 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:36.739 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.739 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.739 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-18T04:38:36.739 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:38:36.739 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 60.155147489999997, 2024-09-18T04:38:36.739 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082699560000000002, 2024-09-18T04:38:36.739 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:36.739 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:36.739 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:36.739 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:38:36.740 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:38:36.740 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 7 2024-09-18T04:38:36.740 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.740 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:36.740 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.740 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:36.740 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:38:36.740 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:36.740 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.740 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.740 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:36.741 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:38:36.741 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:36.741 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.741 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.741 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:36.741 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482870+0000", 2024-09-18T04:38:36.741 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999967 2024-09-18T04:38:36.741 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.741 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.742 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:36.742 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482889+0000", 2024-09-18T04:38:36.742 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.965e-05 2024-09-18T04:38:36.742 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.742 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.742 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:36.742 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482938+0000", 2024-09-18T04:38:36.742 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8548999999999999e-05 2024-09-18T04:38:36.742 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.742 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.743 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:36.743 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482963+0000", 2024-09-18T04:38:36.743 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5075999999999998e-05 2024-09-18T04:38:36.743 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.743 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.743 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:36.743 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483121+0000", 2024-09-18T04:38:36.743 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015819400000000001 2024-09-18T04:38:36.744 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.744 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.744 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op must wait for map", 2024-09-18T04:38:36.744 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483138+0000", 2024-09-18T04:38:36.744 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7068999999999999e-05 2024-09-18T04:38:36.744 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.744 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.744 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:36.744 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488666+0000", 2024-09-18T04:38:36.745 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0055282049999999996 2024-09-18T04:38:36.745 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.745 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.745 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:36.745 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488783+0000", 2024-09-18T04:38:36.745 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000116737 2024-09-18T04:38:36.745 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.745 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.745 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:36.746 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488991+0000", 2024-09-18T04:38:36.746 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020781400000000001 2024-09-18T04:38:36.746 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.746 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.746 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:36.746 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490554+0000", 2024-09-18T04:38:36.746 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0015628370000000001 2024-09-18T04:38:36.746 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.746 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.747 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:36.747 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491019+0000", 2024-09-18T04:38:36.747 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000465608 2024-09-18T04:38:36.747 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.747 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.747 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:36.747 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491087+0000", 2024-09-18T04:38:36.747 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7988e-05 2024-09-18T04:38:36.747 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.748 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.748 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:36.748 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491143+0000", 2024-09-18T04:38:36.748 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5538999999999998e-05 2024-09-18T04:38:36.748 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:36.748 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:36.748 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:36.748 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.748 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.749 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.4214.0:7 2.2 e19/17 ondisk, result = 0)", 2024-09-18T04:38:36.749 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:36.490861+0000", 2024-09-18T04:38:36.749 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 60.147159868000003, 2024-09-18T04:38:36.749 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00029639400000000001, 2024-09-18T04:38:36.749 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:36.749 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:38:36.749 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:36.749 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.750 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:36.750 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490861+0000", 2024-09-18T04:38:36.750 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:36.750 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.750 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.750 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:36.750 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490861+0000", 2024-09-18T04:38:36.750 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:36.750 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.750 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.751 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:36.751 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490859+0000", 2024-09-18T04:38:36.751 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:36.751 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.751 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.751 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:36.751 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490867+0000", 2024-09-18T04:38:36.751 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.3089999999999996e-06 2024-09-18T04:38:36.751 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.752 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.752 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:36.752 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490909+0000", 2024-09-18T04:38:36.752 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1919000000000003e-05 2024-09-18T04:38:36.752 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.752 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.752 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:36.752 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490922+0000", 2024-09-18T04:38:36.752 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2836999999999999e-05 2024-09-18T04:38:36.753 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.753 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.753 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:36.753 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491003+0000", 2024-09-18T04:38:36.753 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0681999999999998e-05 2024-09-18T04:38:36.753 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.753 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.753 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:36.753 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491013+0000", 2024-09-18T04:38:36.753 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0220999999999999e-05 2024-09-18T04:38:36.754 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.754 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.754 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:36.754 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491157+0000", 2024-09-18T04:38:36.754 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014401799999999999 2024-09-18T04:38:36.754 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:36.754 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:36.754 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:36.754 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.754 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.755 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "MOSDPGPushReply(2.3 25/24 [PushReplyOp(2:c4c92e5a:::rbd_trash:head)])", 2024-09-18T04:38:36.755 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:51.680323+0000", 2024-09-18T04:38:36.755 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 44.957697492999998, 2024-09-18T04:38:36.755 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00054411499999999996, 2024-09-18T04:38:36.755 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:36.755 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-09-18T04:38:36.755 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:36.755 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.755 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:36.755 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680323+0000", 2024-09-18T04:38:36.756 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:36.756 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.756 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.756 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:36.756 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680323+0000", 2024-09-18T04:38:36.756 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:36.756 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.756 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.756 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:36.757 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680322+0000", 2024-09-18T04:38:36.757 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:36.757 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.757 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.757 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:36.757 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680333+0000", 2024-09-18T04:38:36.757 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1153e-05 2024-09-18T04:38:36.757 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.757 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.757 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:36.757 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680357+0000", 2024-09-18T04:38:36.758 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3252999999999999e-05 2024-09-18T04:38:36.758 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.758 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.758 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:36.758 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680378+0000", 2024-09-18T04:38:36.758 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1503000000000001e-05 2024-09-18T04:38:36.758 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.758 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.758 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:36.758 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680522+0000", 2024-09-18T04:38:36.758 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014410700000000001 2024-09-18T04:38:36.758 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:36.759 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:36.759 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:36.759 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680867+0000", 2024-09-18T04:38:36.759 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00034475899999999998 2024-09-18T04:38:36.759 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:36.759 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:36.759 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:36.759 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:36.759 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:36.759 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:36.760 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:38:36.761 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:36.758+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:36.862 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:36.858+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:36.877 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:36.877 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:36.878 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:38:36.878 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:36.888 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:38:36.964 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:36.958+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:37.002 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:37.002 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:37.002 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:38:37.002 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:38:37.002 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:37.012 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:38:37.064 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:37.058+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:37.165 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:37.158+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:37.168 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:37.168 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:38:37.169 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:38:37.169 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:38:37.169 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.169 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:30 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'16, mlcod=15'14)", 2024-09-18T04:38:37.169 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.486330+0000", 2024-09-18T04:38:37.169 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 69.677813404999995, 2024-09-18T04:38:37.169 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026383700000000001, 2024-09-18T04:38:37.169 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:37.169 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:37.169 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:37.169 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.169 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:37.170 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486330+0000", 2024-09-18T04:38:37.170 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:37.170 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.170 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.170 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:37.170 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486330+0000", 2024-09-18T04:38:37.170 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:37.170 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.170 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.170 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:37.170 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486328+0000", 2024-09-18T04:38:37.170 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:37.170 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.171 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.171 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:37.171 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486342+0000", 2024-09-18T04:38:37.171 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3783e-05 2024-09-18T04:38:37.171 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.171 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.171 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:37.171 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486373+0000", 2024-09-18T04:38:37.171 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1362999999999998e-05 2024-09-18T04:38:37.171 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.171 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.171 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:37.171 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486387+0000", 2024-09-18T04:38:37.171 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3219e-05 2024-09-18T04:38:37.172 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.172 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.172 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:37.172 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486489+0000", 2024-09-18T04:38:37.172 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010232499999999999 2024-09-18T04:38:37.172 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.172 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.172 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:37.172 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486524+0000", 2024-09-18T04:38:37.172 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4972000000000003e-05 2024-09-18T04:38:37.172 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.172 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.172 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:37.173 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.488902+0000", 2024-09-18T04:38:37.173 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023776240000000001 2024-09-18T04:38:37.173 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.173 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.173 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:37.173 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.488968+0000", 2024-09-18T04:38:37.173 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6761000000000007e-05 2024-09-18T04:38:37.173 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:37.173 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:37.173 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:37.173 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.173 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.173 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:35 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'21, mlcod=15'17)", 2024-09-18T04:38:37.174 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.491817+0000", 2024-09-18T04:38:37.174 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 69.672326487000007, 2024-09-18T04:38:37.174 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023524739999999998, 2024-09-18T04:38:37.174 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:37.174 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:37.174 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:37.174 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.174 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:37.174 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491817+0000", 2024-09-18T04:38:37.174 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:37.174 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.174 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.174 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:37.175 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491817+0000", 2024-09-18T04:38:37.175 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:37.175 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.175 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.175 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:37.175 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491815+0000", 2024-09-18T04:38:37.175 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:37.175 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.175 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.175 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:37.175 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491890+0000", 2024-09-18T04:38:37.175 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4500999999999997e-05 2024-09-18T04:38:37.175 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.176 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.176 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:37.176 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491916+0000", 2024-09-18T04:38:37.176 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5666000000000001e-05 2024-09-18T04:38:37.176 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.176 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.176 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:37.176 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491928+0000", 2024-09-18T04:38:37.176 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2585999999999999e-05 2024-09-18T04:38:37.176 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.176 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.176 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:37.177 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.492000+0000", 2024-09-18T04:38:37.177 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2309000000000002e-05 2024-09-18T04:38:37.177 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.177 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.177 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:37.177 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.492027+0000", 2024-09-18T04:38:37.177 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6251e-05 2024-09-18T04:38:37.177 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.177 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.178 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:37.178 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.494114+0000", 2024-09-18T04:38:37.178 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020875260000000001 2024-09-18T04:38:37.178 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.178 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.178 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:37.178 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.494169+0000", 2024-09-18T04:38:37.178 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5145000000000001e-05 2024-09-18T04:38:37.178 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:37.178 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:37.178 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:37.179 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.179 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.179 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:57 1.0 e15/14 1:30306672:devicehealth::main.db.0000000000000000:head v 15'35, mlcod=15'33)", 2024-09-18T04:38:37.179 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.529997+0000", 2024-09-18T04:38:37.179 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 69.634146633, 2024-09-18T04:38:37.179 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002623869, 2024-09-18T04:38:37.179 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:37.179 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:37.179 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:37.179 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.179 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:37.179 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.529997+0000", 2024-09-18T04:38:37.180 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:37.180 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.180 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.180 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:37.180 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.529997+0000", 2024-09-18T04:38:37.180 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:37.180 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.180 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.180 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:37.180 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.529995+0000", 2024-09-18T04:38:37.180 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:37.180 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.181 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.181 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:37.181 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530116+0000", 2024-09-18T04:38:37.181 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000121141 2024-09-18T04:38:37.181 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.181 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.181 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:37.181 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530146+0000", 2024-09-18T04:38:37.181 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0423e-05 2024-09-18T04:38:37.181 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.181 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.181 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:37.181 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530159+0000", 2024-09-18T04:38:37.182 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2802999999999999e-05 2024-09-18T04:38:37.182 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.182 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.182 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:37.182 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530218+0000", 2024-09-18T04:38:37.182 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8461000000000002e-05 2024-09-18T04:38:37.182 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.182 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.182 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:37.182 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530253+0000", 2024-09-18T04:38:37.182 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4796999999999999e-05 2024-09-18T04:38:37.182 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.182 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.183 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:37.183 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.532559+0000", 2024-09-18T04:38:37.183 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002306939 2024-09-18T04:38:37.183 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.183 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.183 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:37.183 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.532621+0000", 2024-09-18T04:38:37.183 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1131999999999997e-05 2024-09-18T04:38:37.183 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:37.183 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:37.183 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:37.183 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.183 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.183 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:68 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'42, mlcod=15'39)", 2024-09-18T04:38:37.184 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.546404+0000", 2024-09-18T04:38:37.184 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 69.617739221999997, 2024-09-18T04:38:37.184 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024499769999999999, 2024-09-18T04:38:37.184 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:37.184 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:37.184 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:37.184 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.184 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:37.184 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546404+0000", 2024-09-18T04:38:37.184 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:37.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.184 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.184 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:37.185 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546404+0000", 2024-09-18T04:38:37.185 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:37.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.185 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.185 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:37.185 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546403+0000", 2024-09-18T04:38:37.185 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:37.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.185 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.185 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:37.185 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546416+0000", 2024-09-18T04:38:37.185 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.306e-05 2024-09-18T04:38:37.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.185 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.186 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:37.186 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546465+0000", 2024-09-18T04:38:37.186 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9441999999999999e-05 2024-09-18T04:38:37.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.186 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.186 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:37.186 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546480+0000", 2024-09-18T04:38:37.186 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4919999999999999e-05 2024-09-18T04:38:37.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.186 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.186 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:37.186 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546541+0000", 2024-09-18T04:38:37.186 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1085000000000002e-05 2024-09-18T04:38:37.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.187 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.187 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:37.187 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546569+0000", 2024-09-18T04:38:37.187 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7803000000000001e-05 2024-09-18T04:38:37.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.187 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.187 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:37.187 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.548725+0000", 2024-09-18T04:38:37.187 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021556629999999999 2024-09-18T04:38:37.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.187 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.187 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:37.187 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.548854+0000", 2024-09-18T04:38:37.188 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000129524 2024-09-18T04:38:37.188 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:37.188 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:37.188 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:37.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.188 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.188 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:69 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'43, mlcod=15'39)", 2024-09-18T04:38:37.188 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.547520+0000", 2024-09-18T04:38:37.188 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 69.616623145999995, 2024-09-18T04:38:37.188 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023506820000000002, 2024-09-18T04:38:37.188 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:37.188 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:37.188 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:37.188 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.189 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:37.189 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547520+0000", 2024-09-18T04:38:37.189 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:37.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.189 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.189 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:37.189 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547520+0000", 2024-09-18T04:38:37.189 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:37.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.189 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.189 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:37.189 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547519+0000", 2024-09-18T04:38:37.190 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:37.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.190 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.190 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:37.190 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547614+0000", 2024-09-18T04:38:37.190 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5235000000000006e-05 2024-09-18T04:38:37.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.190 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.190 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:37.190 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547640+0000", 2024-09-18T04:38:37.190 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5698999999999999e-05 2024-09-18T04:38:37.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.190 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.190 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:37.190 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547651+0000", 2024-09-18T04:38:37.191 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.134e-05 2024-09-18T04:38:37.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.191 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.191 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:37.191 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547805+0000", 2024-09-18T04:38:37.191 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015426099999999999 2024-09-18T04:38:37.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.191 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.191 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:37.191 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547844+0000", 2024-09-18T04:38:37.191 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8229999999999998e-05 2024-09-18T04:38:37.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.191 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.191 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:37.192 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.549810+0000", 2024-09-18T04:38:37.192 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0019660519999999998 2024-09-18T04:38:37.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.192 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.192 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:37.192 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.549871+0000", 2024-09-18T04:38:37.192 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1152e-05 2024-09-18T04:38:37.192 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:37.192 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:37.192 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:37.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.192 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.192 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:83 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'53, mlcod=15'50)", 2024-09-18T04:38:37.193 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.575989+0000", 2024-09-18T04:38:37.193 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 69.588154197999998, 2024-09-18T04:38:37.193 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023991310000000001, 2024-09-18T04:38:37.193 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:37.193 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:37.193 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:37.193 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.193 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:37.193 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.575989+0000", 2024-09-18T04:38:37.193 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:37.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.193 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.193 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:37.194 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.575989+0000", 2024-09-18T04:38:37.194 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:37.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.194 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.194 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:37.194 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.575988+0000", 2024-09-18T04:38:37.194 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:37.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.194 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.194 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:37.194 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576001+0000", 2024-09-18T04:38:37.194 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2751e-05 2024-09-18T04:38:37.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.194 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.195 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:37.195 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576023+0000", 2024-09-18T04:38:37.195 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2453999999999999e-05 2024-09-18T04:38:37.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.195 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.195 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:37.195 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576038+0000", 2024-09-18T04:38:37.195 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5125e-05 2024-09-18T04:38:37.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.195 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.195 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:37.195 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576161+0000", 2024-09-18T04:38:37.195 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000122543 2024-09-18T04:38:37.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.196 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.196 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:37.196 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576221+0000", 2024-09-18T04:38:37.196 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9948000000000003e-05 2024-09-18T04:38:37.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.196 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.196 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:37.196 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.578277+0000", 2024-09-18T04:38:37.196 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020567559999999999 2024-09-18T04:38:37.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.196 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.197 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:37.197 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.578388+0000", 2024-09-18T04:38:37.197 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011089100000000001 2024-09-18T04:38:37.197 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:37.197 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:37.197 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:37.197 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.197 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.197 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:86 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'56, mlcod=15'52)", 2024-09-18T04:38:37.197 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.580261+0000", 2024-09-18T04:38:37.197 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 69.583882561999999, 2024-09-18T04:38:37.198 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002418108, 2024-09-18T04:38:37.198 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:37.198 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:37.198 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:37.198 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.198 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:37.198 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580261+0000", 2024-09-18T04:38:37.198 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:37.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.198 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.198 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:37.198 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580261+0000", 2024-09-18T04:38:37.198 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:37.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.199 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.199 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:37.199 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580260+0000", 2024-09-18T04:38:37.199 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:37.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.199 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.199 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:37.199 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580269+0000", 2024-09-18T04:38:37.199 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5109999999999992e-06 2024-09-18T04:38:37.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.200 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.200 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:37.200 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580288+0000", 2024-09-18T04:38:37.200 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8783999999999998e-05 2024-09-18T04:38:37.200 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.200 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.200 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:37.200 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580296+0000", 2024-09-18T04:38:37.200 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3250000000000008e-06 2024-09-18T04:38:37.200 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.200 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.200 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:37.200 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580351+0000", 2024-09-18T04:38:37.201 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4132000000000003e-05 2024-09-18T04:38:37.201 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.201 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.201 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:37.201 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580381+0000", 2024-09-18T04:38:37.201 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0263000000000001e-05 2024-09-18T04:38:37.201 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.201 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.201 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:37.201 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.582615+0000", 2024-09-18T04:38:37.201 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0022339109999999999 2024-09-18T04:38:37.201 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.201 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.202 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:37.202 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.582679+0000", 2024-09-18T04:38:37.202 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4182000000000003e-05 2024-09-18T04:38:37.202 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:37.202 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:37.202 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:37.202 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.202 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.202 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:105 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'67, mlcod=15'64)", 2024-09-18T04:38:37.202 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.789089+0000", 2024-09-18T04:38:37.202 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 69.375054391999996, 2024-09-18T04:38:37.202 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024384229999999999, 2024-09-18T04:38:37.202 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:37.203 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:37.203 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:37.203 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.203 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:37.203 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789089+0000", 2024-09-18T04:38:37.203 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:37.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.203 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.203 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:37.203 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789089+0000", 2024-09-18T04:38:37.203 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:37.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.203 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.204 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:37.204 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789088+0000", 2024-09-18T04:38:37.204 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:37.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.204 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.204 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:37.204 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789107+0000", 2024-09-18T04:38:37.204 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9017000000000001e-05 2024-09-18T04:38:37.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.204 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.204 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:37.204 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789139+0000", 2024-09-18T04:38:37.204 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2261999999999997e-05 2024-09-18T04:38:37.205 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.205 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.205 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:37.205 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789153+0000", 2024-09-18T04:38:37.205 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4025999999999999e-05 2024-09-18T04:38:37.205 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.205 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.205 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:37.205 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789255+0000", 2024-09-18T04:38:37.205 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000101969 2024-09-18T04:38:37.205 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.205 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.205 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:37.205 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789314+0000", 2024-09-18T04:38:37.206 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8867999999999999e-05 2024-09-18T04:38:37.206 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.206 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.206 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:37.206 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.791453+0000", 2024-09-18T04:38:37.206 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002138938 2024-09-18T04:38:37.206 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.206 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.206 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:37.206 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.791527+0000", 2024-09-18T04:38:37.206 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4464999999999999e-05 2024-09-18T04:38:37.206 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:37.206 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:37.207 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:37.207 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.207 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.207 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:106 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'68, mlcod=15'64)", 2024-09-18T04:38:37.207 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.790295+0000", 2024-09-18T04:38:37.207 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 69.373848541000001, 2024-09-18T04:38:37.207 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024379390000000001, 2024-09-18T04:38:37.207 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:37.207 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:37.207 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:37.207 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.207 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:37.207 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790295+0000", 2024-09-18T04:38:37.207 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:37.208 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.208 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.208 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:37.208 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790295+0000", 2024-09-18T04:38:37.208 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:37.208 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.208 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.208 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:37.208 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790294+0000", 2024-09-18T04:38:37.208 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:37.208 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.208 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.208 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:37.209 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790390+0000", 2024-09-18T04:38:37.209 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6537000000000001e-05 2024-09-18T04:38:37.209 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.209 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.209 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:37.209 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790418+0000", 2024-09-18T04:38:37.209 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8218e-05 2024-09-18T04:38:37.209 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.209 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.209 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:37.209 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790431+0000", 2024-09-18T04:38:37.209 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2571999999999999e-05 2024-09-18T04:38:37.209 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.210 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.210 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:37.210 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790534+0000", 2024-09-18T04:38:37.210 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000103433 2024-09-18T04:38:37.210 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.210 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.210 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:37.210 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790565+0000", 2024-09-18T04:38:37.210 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0409999999999999e-05 2024-09-18T04:38:37.210 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.210 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.210 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:37.210 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.792651+0000", 2024-09-18T04:38:37.210 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020865250000000001 2024-09-18T04:38:37.211 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.211 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.211 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:37.211 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.792733+0000", 2024-09-18T04:38:37.211 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1477999999999995e-05 2024-09-18T04:38:37.211 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:37.211 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:37.211 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:37.211 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.211 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.211 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:109 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'71, mlcod=15'68)", 2024-09-18T04:38:37.211 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.795250+0000", 2024-09-18T04:38:37.211 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 69.368893526999997, 2024-09-18T04:38:37.212 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023990449999999998, 2024-09-18T04:38:37.212 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:37.212 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:37.212 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:37.212 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.212 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:37.212 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795250+0000", 2024-09-18T04:38:37.212 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:37.212 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.212 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.212 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:37.212 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795250+0000", 2024-09-18T04:38:37.212 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:37.212 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.213 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.213 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:37.213 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795248+0000", 2024-09-18T04:38:37.213 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:37.213 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.213 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.213 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:37.213 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795405+0000", 2024-09-18T04:38:37.213 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000157007 2024-09-18T04:38:37.213 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.213 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.213 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:37.213 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795422+0000", 2024-09-18T04:38:37.214 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7013000000000001e-05 2024-09-18T04:38:37.214 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.214 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.214 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:37.214 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795432+0000", 2024-09-18T04:38:37.214 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0448999999999999e-05 2024-09-18T04:38:37.214 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.214 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.214 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:37.214 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795534+0000", 2024-09-18T04:38:37.214 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000101195 2024-09-18T04:38:37.214 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.214 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.214 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:37.215 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795587+0000", 2024-09-18T04:38:37.215 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3356000000000002e-05 2024-09-18T04:38:37.215 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.215 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.215 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:37.215 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.797560+0000", 2024-09-18T04:38:37.215 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001973023 2024-09-18T04:38:37.215 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.215 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.215 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:37.215 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.797649+0000", 2024-09-18T04:38:37.215 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8864999999999997e-05 2024-09-18T04:38:37.215 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:37.215 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:37.216 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:37.216 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.216 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.216 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:115 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'77, mlcod=15'72)", 2024-09-18T04:38:37.216 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.803339+0000", 2024-09-18T04:38:37.216 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 69.360804809000001, 2024-09-18T04:38:37.216 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002385093, 2024-09-18T04:38:37.216 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:37.216 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:37.216 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:37.216 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.216 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:37.216 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803339+0000", 2024-09-18T04:38:37.216 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:37.217 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.217 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.217 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:37.217 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803339+0000", 2024-09-18T04:38:37.217 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:37.217 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.217 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.217 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:37.217 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803337+0000", 2024-09-18T04:38:37.217 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:37.217 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.217 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.217 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:37.218 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803475+0000", 2024-09-18T04:38:37.218 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013718199999999999 2024-09-18T04:38:37.218 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.218 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.218 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:37.218 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803507+0000", 2024-09-18T04:38:37.218 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2351000000000002e-05 2024-09-18T04:38:37.218 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.218 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.218 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:37.218 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803533+0000", 2024-09-18T04:38:37.218 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6010999999999999e-05 2024-09-18T04:38:37.218 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.219 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.219 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:37.219 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803609+0000", 2024-09-18T04:38:37.219 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.64e-05 2024-09-18T04:38:37.219 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.219 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.219 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:37.219 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803671+0000", 2024-09-18T04:38:37.219 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1467e-05 2024-09-18T04:38:37.219 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.219 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.219 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:37.220 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.805676+0000", 2024-09-18T04:38:37.220 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020047849999999998 2024-09-18T04:38:37.220 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.220 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.220 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:37.220 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.805724+0000", 2024-09-18T04:38:37.220 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8151e-05 2024-09-18T04:38:37.220 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:37.220 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:37.220 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:37.220 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.220 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.221 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:118 1.0 e15/14 1:30306672:devicehealth::main.db.0000000000000000:head v 15'80, mlcod=15'78)", 2024-09-18T04:38:37.221 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.807930+0000", 2024-09-18T04:38:37.221 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 69.356213306000001, 2024-09-18T04:38:37.221 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028002259999999998, 2024-09-18T04:38:37.221 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:37.221 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:37.221 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:37.221 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.221 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:37.221 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.807930+0000", 2024-09-18T04:38:37.221 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:37.221 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.221 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.222 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:37.222 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.807930+0000", 2024-09-18T04:38:37.222 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:37.222 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.222 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.222 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:37.222 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.807929+0000", 2024-09-18T04:38:37.222 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:37.222 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.222 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.222 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:37.222 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808024+0000", 2024-09-18T04:38:37.222 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5258000000000002e-05 2024-09-18T04:38:37.223 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.223 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.223 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:37.223 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808044+0000", 2024-09-18T04:38:37.223 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0203e-05 2024-09-18T04:38:37.223 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.223 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.223 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:37.223 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808053+0000", 2024-09-18T04:38:37.223 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2689999999999999e-06 2024-09-18T04:38:37.223 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.223 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.223 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:37.223 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808137+0000", 2024-09-18T04:38:37.224 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4441e-05 2024-09-18T04:38:37.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.224 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.224 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:37.224 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808172+0000", 2024-09-18T04:38:37.224 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4453999999999999e-05 2024-09-18T04:38:37.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.224 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.224 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:37.224 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.810663+0000", 2024-09-18T04:38:37.224 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024910269999999998 2024-09-18T04:38:37.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.224 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.225 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:37.225 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.810730+0000", 2024-09-18T04:38:37.225 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7658999999999997e-05 2024-09-18T04:38:37.225 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:37.225 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:37.225 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:37.225 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.225 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.225 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:129 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'87, mlcod=15'85)", 2024-09-18T04:38:37.225 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.231689+0000", 2024-09-18T04:38:37.225 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 68.932454293000006, 2024-09-18T04:38:37.225 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002376502, 2024-09-18T04:38:37.225 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:37.226 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:37.226 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:37.226 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.226 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:37.226 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231689+0000", 2024-09-18T04:38:37.226 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:37.226 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.226 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.226 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:37.226 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231689+0000", 2024-09-18T04:38:37.226 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:37.226 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.226 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.227 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:37.227 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231687+0000", 2024-09-18T04:38:37.227 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:37.227 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.227 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.227 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:37.227 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231702+0000", 2024-09-18T04:38:37.227 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4949999999999999e-05 2024-09-18T04:38:37.227 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.227 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.227 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:37.227 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231735+0000", 2024-09-18T04:38:37.227 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2276000000000001e-05 2024-09-18T04:38:37.227 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.228 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.228 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:37.228 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231750+0000", 2024-09-18T04:38:37.228 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5193e-05 2024-09-18T04:38:37.228 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.228 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.228 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:37.228 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231872+0000", 2024-09-18T04:38:37.228 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000122127 2024-09-18T04:38:37.228 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.228 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.229 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:37.229 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231936+0000", 2024-09-18T04:38:37.229 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4089000000000003e-05 2024-09-18T04:38:37.229 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.229 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.229 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:37.229 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234002+0000", 2024-09-18T04:38:37.229 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002065575 2024-09-18T04:38:37.229 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.229 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.229 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:37.229 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234066+0000", 2024-09-18T04:38:37.230 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3944999999999998e-05 2024-09-18T04:38:37.230 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:37.230 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:37.230 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:37.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.230 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.230 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:130 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'88, mlcod=15'85)", 2024-09-18T04:38:37.230 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.232853+0000", 2024-09-18T04:38:37.230 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 68.931290677999996, 2024-09-18T04:38:37.230 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027351739999999999, 2024-09-18T04:38:37.230 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:37.230 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:37.230 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:37.230 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.231 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:37.231 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232853+0000", 2024-09-18T04:38:37.231 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:37.231 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.231 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.231 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:37.231 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232853+0000", 2024-09-18T04:38:37.231 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:37.231 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.231 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.231 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:37.231 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232851+0000", 2024-09-18T04:38:37.232 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:37.232 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.232 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.232 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:37.232 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232866+0000", 2024-09-18T04:38:37.232 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4946000000000001e-05 2024-09-18T04:38:37.232 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.232 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.232 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:37.232 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232897+0000", 2024-09-18T04:38:37.232 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0982999999999998e-05 2024-09-18T04:38:37.232 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.232 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.233 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:37.233 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232915+0000", 2024-09-18T04:38:37.233 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8476000000000001e-05 2024-09-18T04:38:37.233 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.233 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.233 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:37.233 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.233029+0000", 2024-09-18T04:38:37.233 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011305400000000001 2024-09-18T04:38:37.233 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.233 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.233 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:37.234 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.233079+0000", 2024-09-18T04:38:37.234 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0668999999999999e-05 2024-09-18T04:38:37.234 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.234 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.234 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:37.234 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235534+0000", 2024-09-18T04:38:37.234 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024546030000000001 2024-09-18T04:38:37.234 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.234 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.234 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:37.234 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235588+0000", 2024-09-18T04:38:37.234 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4098999999999998e-05 2024-09-18T04:38:37.235 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:37.235 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:37.235 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:37.235 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.235 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.235 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:131 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'89, mlcod=15'85)", 2024-09-18T04:38:37.235 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.234089+0000", 2024-09-18T04:38:37.235 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 68.930054823999996, 2024-09-18T04:38:37.235 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026975609999999998, 2024-09-18T04:38:37.235 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:37.235 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:37.235 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:37.235 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.236 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:37.236 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234089+0000", 2024-09-18T04:38:37.236 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:37.236 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.236 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.236 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:37.236 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234089+0000", 2024-09-18T04:38:37.236 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:37.236 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.236 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.236 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:37.236 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234087+0000", 2024-09-18T04:38:37.236 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:37.236 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.237 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.237 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:37.237 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234145+0000", 2024-09-18T04:38:37.237 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8687999999999998e-05 2024-09-18T04:38:37.237 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.237 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.237 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:37.237 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234177+0000", 2024-09-18T04:38:37.237 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1526999999999998e-05 2024-09-18T04:38:37.237 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.237 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.237 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:37.237 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234190+0000", 2024-09-18T04:38:37.238 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2965999999999999e-05 2024-09-18T04:38:37.238 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.238 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.238 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:37.238 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234309+0000", 2024-09-18T04:38:37.238 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011875399999999999 2024-09-18T04:38:37.238 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.238 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.238 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:37.238 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234365+0000", 2024-09-18T04:38:37.238 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6436e-05 2024-09-18T04:38:37.238 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.238 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.238 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:37.239 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236675+0000", 2024-09-18T04:38:37.239 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023099520000000001 2024-09-18T04:38:37.239 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.239 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.239 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:37.239 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236786+0000", 2024-09-18T04:38:37.239 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000111025 2024-09-18T04:38:37.239 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:37.239 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:37.239 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:37.239 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.239 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.239 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:132 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'90, mlcod=15'85)", 2024-09-18T04:38:37.240 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.235300+0000", 2024-09-18T04:38:37.240 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 68.928843373999996, 2024-09-18T04:38:37.240 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025982599999999998, 2024-09-18T04:38:37.240 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:37.240 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:37.240 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:37.240 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.240 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:37.240 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235300+0000", 2024-09-18T04:38:37.240 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:37.240 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.240 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.240 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:37.241 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235300+0000", 2024-09-18T04:38:37.241 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:37.241 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.241 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.241 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:37.241 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235298+0000", 2024-09-18T04:38:37.241 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:37.241 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.241 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.241 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:37.241 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235318+0000", 2024-09-18T04:38:37.241 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9714000000000001e-05 2024-09-18T04:38:37.241 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.241 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.242 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:37.242 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235337+0000", 2024-09-18T04:38:37.242 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8892000000000001e-05 2024-09-18T04:38:37.242 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.242 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.242 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:37.242 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235348+0000", 2024-09-18T04:38:37.242 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.129e-05 2024-09-18T04:38:37.242 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.242 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.242 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:37.242 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235681+0000", 2024-09-18T04:38:37.242 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00033323000000000001 2024-09-18T04:38:37.243 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.243 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.243 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:37.243 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235733+0000", 2024-09-18T04:38:37.243 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1391999999999999e-05 2024-09-18T04:38:37.243 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.243 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.243 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:37.243 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237839+0000", 2024-09-18T04:38:37.243 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002105647 2024-09-18T04:38:37.243 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.243 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.243 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:37.243 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237898+0000", 2024-09-18T04:38:37.244 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9759000000000002e-05 2024-09-18T04:38:37.244 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:37.244 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:37.244 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:37.244 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.244 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.244 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:133 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'91, mlcod=15'86)", 2024-09-18T04:38:37.244 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.236920+0000", 2024-09-18T04:38:37.244 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 68.927223486000003, 2024-09-18T04:38:37.244 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024318410000000001, 2024-09-18T04:38:37.244 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:37.244 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:37.244 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:37.245 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.245 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:37.245 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236920+0000", 2024-09-18T04:38:37.245 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:37.245 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.245 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.245 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:37.245 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236920+0000", 2024-09-18T04:38:37.245 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:37.245 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.245 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.245 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:37.245 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236918+0000", 2024-09-18T04:38:37.246 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:37.246 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.246 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.246 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:37.246 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236931+0000", 2024-09-18T04:38:37.246 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2683999999999999e-05 2024-09-18T04:38:37.246 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.246 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.246 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:37.246 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236951+0000", 2024-09-18T04:38:37.246 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0292000000000002e-05 2024-09-18T04:38:37.246 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.246 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.247 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:37.247 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236962+0000", 2024-09-18T04:38:37.247 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0703000000000001e-05 2024-09-18T04:38:37.247 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.247 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.247 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:37.247 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237095+0000", 2024-09-18T04:38:37.247 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013312400000000001 2024-09-18T04:38:37.247 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.247 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.247 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:37.247 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237133+0000", 2024-09-18T04:38:37.247 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7496000000000001e-05 2024-09-18T04:38:37.247 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.248 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.248 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:37.248 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.239206+0000", 2024-09-18T04:38:37.248 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020730290000000001 2024-09-18T04:38:37.248 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.248 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.248 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:37.248 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.239352+0000", 2024-09-18T04:38:37.248 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014593000000000001 2024-09-18T04:38:37.248 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:37.248 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:37.248 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:37.248 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.248 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.249 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:134 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'92, mlcod=15'86)", 2024-09-18T04:38:37.249 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.237894+0000", 2024-09-18T04:38:37.249 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 68.926249463999994, 2024-09-18T04:38:37.249 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025299519999999998, 2024-09-18T04:38:37.249 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:37.249 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:37.249 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:37.249 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.249 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237894+0000", 2024-09-18T04:38:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237894+0000", 2024-09-18T04:38:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237892+0000", 2024-09-18T04:38:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:37.253 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.253 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.253 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:37.253 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237986+0000", 2024-09-18T04:38:37.253 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3341e-05 2024-09-18T04:38:37.253 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.253 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.253 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:37.253 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.238012+0000", 2024-09-18T04:38:37.253 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5843e-05 2024-09-18T04:38:37.253 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.253 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.254 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:37.254 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.238027+0000", 2024-09-18T04:38:37.254 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5469e-05 2024-09-18T04:38:37.254 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.254 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.254 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:37.254 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.238114+0000", 2024-09-18T04:38:37.254 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7483000000000002e-05 2024-09-18T04:38:37.254 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.254 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.254 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:37.254 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.238152+0000", 2024-09-18T04:38:37.254 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7565999999999999e-05 2024-09-18T04:38:37.255 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.255 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.255 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:37.255 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.240297+0000", 2024-09-18T04:38:37.255 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021447620000000001 2024-09-18T04:38:37.255 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.255 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.255 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:37.255 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.240424+0000", 2024-09-18T04:38:37.255 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012705400000000001 2024-09-18T04:38:37.255 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:37.255 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:37.255 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:37.255 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.256 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.256 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:139 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'97, mlcod=15'93)", 2024-09-18T04:38:37.256 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.245692+0000", 2024-09-18T04:38:37.256 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 68.918451322999999, 2024-09-18T04:38:37.256 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024260480000000001, 2024-09-18T04:38:37.256 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:37.256 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:37.256 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:37.256 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.256 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:37.256 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245692+0000", 2024-09-18T04:38:37.256 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:37.256 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.257 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.257 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:37.257 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245692+0000", 2024-09-18T04:38:37.257 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:37.257 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.257 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.257 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:37.257 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245690+0000", 2024-09-18T04:38:37.257 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:37.257 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.257 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.257 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:37.257 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245712+0000", 2024-09-18T04:38:37.257 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1916000000000002e-05 2024-09-18T04:38:37.258 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.258 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.258 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:37.258 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245741+0000", 2024-09-18T04:38:37.258 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8503000000000001e-05 2024-09-18T04:38:37.258 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.258 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.258 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:37.258 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245759+0000", 2024-09-18T04:38:37.258 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.872e-05 2024-09-18T04:38:37.258 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.258 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.258 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:37.259 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245859+0000", 2024-09-18T04:38:37.259 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9641000000000004e-05 2024-09-18T04:38:37.259 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.259 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.259 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:37.259 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245900+0000", 2024-09-18T04:38:37.259 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0589000000000001e-05 2024-09-18T04:38:37.259 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.259 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.259 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:37.259 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.247961+0000", 2024-09-18T04:38:37.259 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020619190000000002 2024-09-18T04:38:37.259 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.259 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.259 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.248118+0000", 2024-09-18T04:38:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015664300000000001 2024-09-18T04:38:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:144 1.0 e15/14 1:30306672:devicehealth::main.db.0000000000000000:head v 15'102, mlcod=15'98)", 2024-09-18T04:38:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.252149+0000", 2024-09-18T04:38:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 68.911994257000003, 2024-09-18T04:38:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024174959999999999, 2024-09-18T04:38:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:37.261 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.261 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:37.261 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252149+0000", 2024-09-18T04:38:37.261 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:37.261 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.261 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.261 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:37.261 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252149+0000", 2024-09-18T04:38:37.261 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:37.261 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.261 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.261 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:37.261 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252148+0000", 2024-09-18T04:38:37.262 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:37.262 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.262 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.262 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:37.262 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252225+0000", 2024-09-18T04:38:37.262 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6576999999999995e-05 2024-09-18T04:38:37.262 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.262 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.262 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:37.262 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252242+0000", 2024-09-18T04:38:37.262 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7740999999999998e-05 2024-09-18T04:38:37.262 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.262 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.263 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:37.263 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252254+0000", 2024-09-18T04:38:37.263 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.137e-05 2024-09-18T04:38:37.263 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.263 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.263 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:37.263 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252385+0000", 2024-09-18T04:38:37.263 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013091099999999999 2024-09-18T04:38:37.263 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.263 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.263 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:37.263 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252426+0000", 2024-09-18T04:38:37.263 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1535999999999997e-05 2024-09-18T04:38:37.263 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.264 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.264 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:37.264 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.254498+0000", 2024-09-18T04:38:37.264 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002071477 2024-09-18T04:38:37.264 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.264 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.264 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:37.264 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.254567+0000", 2024-09-18T04:38:37.264 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9014e-05 2024-09-18T04:38:37.264 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:37.264 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:37.264 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:37.264 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:37.264 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:37.265 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:37.266 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:38:37.267 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:37.262+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:37.367 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:37.362+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:37.425 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:37.425 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:37.425 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:38:37.426 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:37.435 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:38:37.468 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:37.462+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:37.546 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:37.546 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:37.546 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:38:37.546 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:38:37.547 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:37.556 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:38:37.602 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:37.598+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:37.669 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:37.670 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:38:37.670 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:38:37.670 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:38:37.670 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.670 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4214.0:5 2.2 e19/17 2:5cea7035:::rbd_info:head v 19'1, mlcod=0'0)", 2024-09-18T04:38:37.670 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:35.471260+0000", 2024-09-18T04:38:37.670 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 62.196436542999997, 2024-09-18T04:38:37.670 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021025950000000001, 2024-09-18T04:38:37.670 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:37.670 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:37.670 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:37.670 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.671 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:37.671 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471260+0000", 2024-09-18T04:38:37.671 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:37.671 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.671 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.671 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:37.671 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471260+0000", 2024-09-18T04:38:37.671 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:37.671 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.671 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.671 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:37.671 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471259+0000", 2024-09-18T04:38:37.671 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:37.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.672 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.672 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:37.672 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471275+0000", 2024-09-18T04:38:37.672 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6135999999999999e-05 2024-09-18T04:38:37.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.672 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.672 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:37.672 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471298+0000", 2024-09-18T04:38:37.672 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3632999999999999e-05 2024-09-18T04:38:37.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.673 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.673 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:37.673 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471320+0000", 2024-09-18T04:38:37.673 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1852999999999999e-05 2024-09-18T04:38:37.673 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.673 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.673 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:37.673 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471445+0000", 2024-09-18T04:38:37.673 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012461699999999999 2024-09-18T04:38:37.673 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.673 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.673 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:37.674 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471529+0000", 2024-09-18T04:38:37.674 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3875000000000006e-05 2024-09-18T04:38:37.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.674 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.674 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:37.674 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.473317+0000", 2024-09-18T04:38:37.674 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017886899999999999 2024-09-18T04:38:37.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.675 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.675 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:37.675 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.473362+0000", 2024-09-18T04:38:37.675 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4724000000000001e-05 2024-09-18T04:38:37.675 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:37.675 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:37.675 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:37.675 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.676 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.676 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4214.0:7 2.2 e20/17 2:5cea7035:::rbd_info:head v 20'2, mlcod=0'0)", 2024-09-18T04:38:37.676 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:36.490560+0000", 2024-09-18T04:38:37.676 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 61.177136404000002, 2024-09-18T04:38:37.676 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0015326020000000001, 2024-09-18T04:38:37.676 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:37.676 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:37.676 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:37.676 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.676 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:37.676 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490560+0000", 2024-09-18T04:38:37.676 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:37.676 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.677 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.677 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:37.677 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490560+0000", 2024-09-18T04:38:37.677 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:37.677 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.677 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.677 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:37.677 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490559+0000", 2024-09-18T04:38:37.677 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:37.677 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.677 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.677 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:37.677 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490568+0000", 2024-09-18T04:38:37.678 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5699999999999993e-06 2024-09-18T04:38:37.678 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.678 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.678 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:37.678 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490599+0000", 2024-09-18T04:38:37.678 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1566000000000002e-05 2024-09-18T04:38:37.678 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.678 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.678 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:37.678 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490609+0000", 2024-09-18T04:38:37.678 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.8700000000000004e-06 2024-09-18T04:38:37.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.679 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.679 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:37.679 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490708+0000", 2024-09-18T04:38:37.679 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9139000000000001e-05 2024-09-18T04:38:37.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.679 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.679 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:37.680 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.490736+0000", 2024-09-18T04:38:37.680 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7617e-05 2024-09-18T04:38:37.680 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.680 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.680 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:37.680 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.492016+0000", 2024-09-18T04:38:37.680 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0012796579999999999 2024-09-18T04:38:37.680 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:37.680 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:37.680 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:37.680 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:36.492092+0000", 2024-09-18T04:38:37.680 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6571999999999998e-05 2024-09-18T04:38:37.681 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:37.681 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:37.681 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:37.681 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:37.681 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:37.681 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:37.682 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:37.662+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:37.683 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:38:37.771 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:37.766+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:37.795 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:37.796 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:37.796 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:38:37.796 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:37.806 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:38:37.872 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:37.866+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:37.934 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-18T04:38:37.967 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:37.967 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:37.968 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:38:37.968 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:38:37.968 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:37.972 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:37.966+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:37.982 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:38:38.073 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:38.066+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:38.139 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:38.139 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:38:38.139 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:38:38.139 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:38:38.139 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:38.140 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4214.0:1 2.3 e18/17 2:c4c92e5a:::rbd_trash:head v 18'1, mlcod=0'0)", 2024-09-18T04:38:38.140 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:34.475123+0000", 2024-09-18T04:38:38.140 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 63.662863588, 2024-09-18T04:38:38.140 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00081394599999999996, 2024-09-18T04:38:38.140 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:38.140 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:38.140 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:38.140 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:38.140 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:38.140 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475123+0000", 2024-09-18T04:38:38.140 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:38.140 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:38.140 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:38.141 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:38.141 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475123+0000", 2024-09-18T04:38:38.141 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:38.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:38.141 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:38.141 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:38.141 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475121+0000", 2024-09-18T04:38:38.141 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:38.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:38.141 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:38.141 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:38.141 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475149+0000", 2024-09-18T04:38:38.141 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7155999999999999e-05 2024-09-18T04:38:38.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:38.142 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:38.142 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:38.142 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475165+0000", 2024-09-18T04:38:38.142 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6160000000000001e-05 2024-09-18T04:38:38.142 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:38.142 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:38.142 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:38.142 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475183+0000", 2024-09-18T04:38:38.142 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7918e-05 2024-09-18T04:38:38.142 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:38.142 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:38.142 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:38.143 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475246+0000", 2024-09-18T04:38:38.143 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3176000000000006e-05 2024-09-18T04:38:38.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:38.143 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:38.143 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:38.143 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475279+0000", 2024-09-18T04:38:38.143 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3192999999999999e-05 2024-09-18T04:38:38.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:38.143 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:38.143 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:38.143 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475871+0000", 2024-09-18T04:38:38.143 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00059206099999999995 2024-09-18T04:38:38.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:38.143 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:38.144 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:38.144 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:34.475937+0000", 2024-09-18T04:38:38.144 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5439999999999997e-05 2024-09-18T04:38:38.144 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:38.144 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:38.144 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:38.144 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:38.144 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:38.144 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 25/24 [PushOp(2:c4c92e5a:::rbd_trash:head, version: 18'1, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:c4c92e5a:::rbd_trash:head@18'1, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-18T04:38:38.144 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:51.680544+0000", 2024-09-18T04:38:38.144 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 46.457441746000001, 2024-09-18T04:38:38.144 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00050365200000000005, 2024-09-18T04:38:38.144 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:38.144 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:38:38.145 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:38.145 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:38.145 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:38.145 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680544+0000", 2024-09-18T04:38:38.145 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:38.145 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:38.145 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:38.145 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:38.145 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680544+0000", 2024-09-18T04:38:38.145 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:38.145 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:38.145 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:38.145 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:38.146 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680543+0000", 2024-09-18T04:38:38.146 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:38.146 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:38.146 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:38.146 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:38.146 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680571+0000", 2024-09-18T04:38:38.146 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7953000000000002e-05 2024-09-18T04:38:38.146 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:38.146 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:38.146 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:38.146 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680605+0000", 2024-09-18T04:38:38.146 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3933000000000002e-05 2024-09-18T04:38:38.146 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:38.146 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:38.147 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:38.147 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680636+0000", 2024-09-18T04:38:38.147 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1288999999999998e-05 2024-09-18T04:38:38.147 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:38.147 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:38.147 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:38.147 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680748+0000", 2024-09-18T04:38:38.147 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000111431 2024-09-18T04:38:38.147 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:38.147 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:38.147 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:38.147 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.680758+0000", 2024-09-18T04:38:38.147 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0244999999999999e-05 2024-09-18T04:38:38.147 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:38.148 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:38.148 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:38.148 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:51.681048+0000", 2024-09-18T04:38:38.148 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00029003399999999999 2024-09-18T04:38:38.148 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:38.148 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:38.148 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:38.148 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:38.148 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:38.148 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:38.149 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:38:38.194 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:38.190+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:38.264 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:38.264 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:38.264 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:38:38.265 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:38.274 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:38:38.275 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:38.270+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:38.377 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:38.374+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:38.378 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:38.378 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:38.378 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:38:38.378 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:38:38.378 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:38.387 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:38:38.478 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:38.474+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:38.501 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:38.501 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:38:38.501 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:38:38.501 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:38:38.501 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.501 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:38.501 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.573328+0000", 2024-09-18T04:38:38.501 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 70.921477639000003, 2024-09-18T04:38:38.501 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011397898, 2024-09-18T04:38:38.502 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:38.502 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:38.502 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:38.502 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:38.502 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:38.502 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 86 2024-09-18T04:38:38.502 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.502 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:38.502 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.502 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:38.502 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573328+0000", 2024-09-18T04:38:38.503 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:38.503 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.503 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.503 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:38.503 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573328+0000", 2024-09-18T04:38:38.503 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:38.503 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.503 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.503 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:38.503 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573327+0000", 2024-09-18T04:38:38.503 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:38.504 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.504 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.504 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:38.504 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573331+0000", 2024-09-18T04:38:38.504 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9750000000000001e-06 2024-09-18T04:38:38.504 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.504 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.504 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:38.504 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573346+0000", 2024-09-18T04:38:38.504 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5469999999999999e-05 2024-09-18T04:38:38.504 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.505 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.505 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:38.505 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573370+0000", 2024-09-18T04:38:38.505 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3969999999999999e-05 2024-09-18T04:38:38.505 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.505 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.505 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:38.505 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.578536+0000", 2024-09-18T04:38:38.505 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0051659679999999999 2024-09-18T04:38:38.505 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.505 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.506 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:38.506 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.578689+0000", 2024-09-18T04:38:38.506 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015276299999999999 2024-09-18T04:38:38.506 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.506 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.506 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:38.506 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.578901+0000", 2024-09-18T04:38:38.506 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000211783 2024-09-18T04:38:38.506 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.506 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.506 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:38.506 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.583408+0000", 2024-09-18T04:38:38.507 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045071249999999998 2024-09-18T04:38:38.507 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.507 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.507 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:38.507 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.584645+0000", 2024-09-18T04:38:38.507 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001237007 2024-09-18T04:38:38.507 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.507 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.507 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:38.507 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.584703+0000", 2024-09-18T04:38:38.507 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.817e-05 2024-09-18T04:38:38.508 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.508 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.508 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:38.508 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.584725+0000", 2024-09-18T04:38:38.508 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2155e-05 2024-09-18T04:38:38.508 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:38.508 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:38.508 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:38.508 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.508 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.508 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:38.508 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.573382+0000", 2024-09-18T04:38:38.509 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 70.921422891000006, 2024-09-18T04:38:38.509 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012154974000000001, 2024-09-18T04:38:38.509 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:38.509 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:38.509 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:38.509 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:38.509 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:38.509 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 87 2024-09-18T04:38:38.509 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.509 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:38.509 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.510 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:38.510 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573382+0000", 2024-09-18T04:38:38.510 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:38.510 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.510 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.510 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:38.510 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573382+0000", 2024-09-18T04:38:38.510 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:38.510 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.510 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.510 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:38.510 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573382+0000", 2024-09-18T04:38:38.511 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:38.511 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.511 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.511 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:38.511 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573482+0000", 2024-09-18T04:38:38.511 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000100333 2024-09-18T04:38:38.511 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.511 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.511 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:38.511 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573492+0000", 2024-09-18T04:38:38.511 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5389999999999996e-06 2024-09-18T04:38:38.512 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.512 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.512 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:38.512 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573500+0000", 2024-09-18T04:38:38.512 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8929999999999994e-06 2024-09-18T04:38:38.512 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.512 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.512 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:38.512 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.579797+0000", 2024-09-18T04:38:38.512 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0062963239999999998 2024-09-18T04:38:38.512 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.513 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.513 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:38.513 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.579957+0000", 2024-09-18T04:38:38.513 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000160283 2024-09-18T04:38:38.513 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.513 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.513 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:38.513 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.580178+0000", 2024-09-18T04:38:38.513 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00022079200000000001 2024-09-18T04:38:38.513 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.513 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.514 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:38.514 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.583479+0000", 2024-09-18T04:38:38.514 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033007689999999998 2024-09-18T04:38:38.514 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.514 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.514 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:38.514 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585477+0000", 2024-09-18T04:38:38.514 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0019985670000000001 2024-09-18T04:38:38.514 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.514 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.514 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:38.514 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585517+0000", 2024-09-18T04:38:38.515 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9709999999999998e-05 2024-09-18T04:38:38.515 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.515 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.515 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:38.515 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585537+0000", 2024-09-18T04:38:38.515 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0326000000000002e-05 2024-09-18T04:38:38.515 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:38.515 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:38.515 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:38.515 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.515 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.515 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:38.516 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.573508+0000", 2024-09-18T04:38:38.516 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 70.921297185, 2024-09-18T04:38:38.516 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012068476, 2024-09-18T04:38:38.516 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:38.516 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:38.516 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:38.516 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:38.516 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:38.516 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 88 2024-09-18T04:38:38.516 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.516 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:38.517 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.517 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:38.517 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573508+0000", 2024-09-18T04:38:38.517 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:38.517 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.517 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.517 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:38.517 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573508+0000", 2024-09-18T04:38:38.517 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:38.517 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.517 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.517 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:38.518 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573507+0000", 2024-09-18T04:38:38.518 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:38.518 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.518 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.518 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:38.518 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573512+0000", 2024-09-18T04:38:38.518 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.2629999999999997e-06 2024-09-18T04:38:38.518 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.518 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.518 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:38.518 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573530+0000", 2024-09-18T04:38:38.519 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8507000000000001e-05 2024-09-18T04:38:38.519 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.519 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.519 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:38.519 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.573537+0000", 2024-09-18T04:38:38.519 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.832e-06 2024-09-18T04:38:38.519 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.519 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.519 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:38.519 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.582486+0000", 2024-09-18T04:38:38.519 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0089494770000000008 2024-09-18T04:38:38.520 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.520 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.520 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:38.520 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.582630+0000", 2024-09-18T04:38:38.520 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000144005 2024-09-18T04:38:38.520 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.520 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.520 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:38.520 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.582842+0000", 2024-09-18T04:38:38.520 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00021189200000000001 2024-09-18T04:38:38.520 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.521 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.521 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:38.521 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.584777+0000", 2024-09-18T04:38:38.521 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001934753 2024-09-18T04:38:38.521 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.521 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.521 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:38.521 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585543+0000", 2024-09-18T04:38:38.521 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00076540099999999997 2024-09-18T04:38:38.521 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.522 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.522 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:38.522 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585561+0000", 2024-09-18T04:38:38.522 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8474999999999998e-05 2024-09-18T04:38:38.522 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.522 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.522 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:38.522 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.585576+0000", 2024-09-18T04:38:38.522 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5427e-05 2024-09-18T04:38:38.522 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:38.522 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:38.523 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:38.523 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.523 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.523 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:38.523 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786394+0000", 2024-09-18T04:38:38.523 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 70.708410737999998, 2024-09-18T04:38:38.523 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011902085999999999, 2024-09-18T04:38:38.523 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:38.523 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:38.523 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:38.523 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:38.524 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:38.524 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 109 2024-09-18T04:38:38.524 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.524 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:38.524 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.524 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:38.524 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786394+0000", 2024-09-18T04:38:38.524 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:38.524 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.524 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.524 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:38.525 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786394+0000", 2024-09-18T04:38:38.525 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:38.525 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.525 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.525 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:38.525 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786394+0000", 2024-09-18T04:38:38.525 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:38.525 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.525 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.525 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:38.525 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786548+0000", 2024-09-18T04:38:38.526 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015463400000000001 2024-09-18T04:38:38.526 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.526 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.526 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:38.526 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786561+0000", 2024-09-18T04:38:38.526 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3093e-05 2024-09-18T04:38:38.526 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.526 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.526 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:38.526 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786571+0000", 2024-09-18T04:38:38.526 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4700000000000008e-06 2024-09-18T04:38:38.527 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.527 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.527 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:38.527 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.793367+0000", 2024-09-18T04:38:38.527 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0067966600000000004 2024-09-18T04:38:38.527 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.527 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.527 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:38.527 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.793526+0000", 2024-09-18T04:38:38.527 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015877 2024-09-18T04:38:38.527 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.528 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.528 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:38.528 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.793752+0000", 2024-09-18T04:38:38.528 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00022555200000000001 2024-09-18T04:38:38.528 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.528 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.528 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:38.528 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798125+0000", 2024-09-18T04:38:38.528 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0043733349999999999 2024-09-18T04:38:38.528 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.529 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.529 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:38.529 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798146+0000", 2024-09-18T04:38:38.529 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1063000000000001e-05 2024-09-18T04:38:38.529 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.529 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.529 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:38.529 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798232+0000", 2024-09-18T04:38:38.529 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5771000000000003e-05 2024-09-18T04:38:38.529 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.529 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.530 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:38.530 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798297+0000", 2024-09-18T04:38:38.530 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.4624999999999999e-05 2024-09-18T04:38:38.530 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:38.530 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:38.530 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:38.530 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.530 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.530 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:38.530 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786584+0000", 2024-09-18T04:38:38.530 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 70.708220659000006, 2024-09-18T04:38:38.530 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012013037000000001, 2024-09-18T04:38:38.531 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:38.531 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:38.531 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:38.531 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:38.531 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:38.531 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 110 2024-09-18T04:38:38.531 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.531 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:38.531 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.531 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:38.531 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786584+0000", 2024-09-18T04:38:38.532 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:38.532 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.532 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.532 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:38.532 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786584+0000", 2024-09-18T04:38:38.532 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:38.532 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.532 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.532 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:38.532 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786584+0000", 2024-09-18T04:38:38.532 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:38.533 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.533 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.533 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:38.533 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786590+0000", 2024-09-18T04:38:38.533 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6359999999999997e-06 2024-09-18T04:38:38.533 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.533 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.533 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:38.533 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786599+0000", 2024-09-18T04:38:38.533 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7369999999999995e-06 2024-09-18T04:38:38.533 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.534 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.534 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:38.534 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786606+0000", 2024-09-18T04:38:38.534 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1930000000000004e-06 2024-09-18T04:38:38.534 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.534 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.534 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:38.534 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.795018+0000", 2024-09-18T04:38:38.534 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0084115660000000005 2024-09-18T04:38:38.534 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.534 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.535 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:38.535 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.795184+0000", 2024-09-18T04:38:38.535 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000165833 2024-09-18T04:38:38.535 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.535 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.535 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:38.535 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.795414+0000", 2024-09-18T04:38:38.535 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00023088399999999999 2024-09-18T04:38:38.535 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.535 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.535 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:38.536 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798321+0000", 2024-09-18T04:38:38.536 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0029069539999999998 2024-09-18T04:38:38.536 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.536 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.536 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:38.536 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798404+0000", 2024-09-18T04:38:38.536 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2724000000000003e-05 2024-09-18T04:38:38.536 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.536 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.536 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:38.537 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798488+0000", 2024-09-18T04:38:38.537 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.3505999999999996e-05 2024-09-18T04:38:38.537 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.537 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.537 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:38.537 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798598+0000", 2024-09-18T04:38:38.537 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000109917 2024-09-18T04:38:38.537 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:38.537 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:38.537 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:38.537 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.538 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.538 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:38.538 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786613+0000", 2024-09-18T04:38:38.538 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 70.708191959000004, 2024-09-18T04:38:38.538 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014245760999999999, 2024-09-18T04:38:38.538 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:38.538 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:38.538 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:38.538 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:38.538 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:38.538 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 111 2024-09-18T04:38:38.538 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.539 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:38.539 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.539 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:38.539 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786613+0000", 2024-09-18T04:38:38.539 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:38.539 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.539 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.539 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:38.539 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786613+0000", 2024-09-18T04:38:38.539 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:38.539 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.539 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.540 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:38.540 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786612+0000", 2024-09-18T04:38:38.540 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:38.540 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.540 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.540 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:38.540 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786622+0000", 2024-09-18T04:38:38.540 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4099999999999997e-06 2024-09-18T04:38:38.540 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.540 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.540 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:38.541 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786630+0000", 2024-09-18T04:38:38.541 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4169999999999992e-06 2024-09-18T04:38:38.541 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.541 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.541 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:38.541 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786637+0000", 2024-09-18T04:38:38.541 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8290000000000002e-06 2024-09-18T04:38:38.541 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.541 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.541 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:38.541 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.797099+0000", 2024-09-18T04:38:38.541 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010461747 2024-09-18T04:38:38.542 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.542 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.542 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:38.542 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.797243+0000", 2024-09-18T04:38:38.542 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014388899999999999 2024-09-18T04:38:38.542 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.542 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.542 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:38.542 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.797463+0000", 2024-09-18T04:38:38.542 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000219912 2024-09-18T04:38:38.542 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.543 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.543 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:38.543 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800732+0000", 2024-09-18T04:38:38.543 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0032688719999999999 2024-09-18T04:38:38.543 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.543 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.543 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:38.543 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800751+0000", 2024-09-18T04:38:38.543 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9633999999999999e-05 2024-09-18T04:38:38.543 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.543 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.544 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:38.544 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800824+0000", 2024-09-18T04:38:38.544 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.2805999999999994e-05 2024-09-18T04:38:38.544 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.544 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.544 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:38.544 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800859+0000", 2024-09-18T04:38:38.544 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4952e-05 2024-09-18T04:38:38.544 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:38.544 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:38.544 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:38.545 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.545 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.545 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:38.545 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786649+0000", 2024-09-18T04:38:38.545 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 70.708156549999998, 2024-09-18T04:38:38.545 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.01613442, 2024-09-18T04:38:38.545 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:38.545 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:38.545 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:38.545 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:38.545 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:38.546 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 112 2024-09-18T04:38:38.546 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.546 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:38.546 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.546 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:38.546 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786649+0000", 2024-09-18T04:38:38.546 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:38.546 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.546 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.546 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:38.546 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786649+0000", 2024-09-18T04:38:38.546 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:38.547 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.547 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.547 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:38.547 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786648+0000", 2024-09-18T04:38:38.547 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:38.547 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.547 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.547 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:38.547 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786768+0000", 2024-09-18T04:38:38.547 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000120491 2024-09-18T04:38:38.547 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.547 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.548 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:38.548 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786795+0000", 2024-09-18T04:38:38.548 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6556000000000001e-05 2024-09-18T04:38:38.548 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.548 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.548 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:38.548 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786807+0000", 2024-09-18T04:38:38.548 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1739999999999999e-05 2024-09-18T04:38:38.548 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.548 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.548 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:38.549 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798683+0000", 2024-09-18T04:38:38.549 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011876070000000001 2024-09-18T04:38:38.549 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.549 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.549 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:38.549 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.798867+0000", 2024-09-18T04:38:38.549 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000184103 2024-09-18T04:38:38.549 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.549 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.549 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:38.549 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.799082+0000", 2024-09-18T04:38:38.549 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00021476000000000001 2024-09-18T04:38:38.550 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.550 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.550 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:38.550 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802610+0000", 2024-09-18T04:38:38.550 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0035282209999999998 2024-09-18T04:38:38.550 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.550 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.550 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:38.550 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802706+0000", 2024-09-18T04:38:38.550 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5922999999999996e-05 2024-09-18T04:38:38.550 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.551 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.551 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:38.551 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802750+0000", 2024-09-18T04:38:38.551 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4511000000000001e-05 2024-09-18T04:38:38.551 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.551 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.551 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:38.551 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802783+0000", 2024-09-18T04:38:38.551 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2752000000000001e-05 2024-09-18T04:38:38.551 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:38.551 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:38.551 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:38.552 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.552 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.552 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:38.552 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786816+0000", 2024-09-18T04:38:38.552 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 70.707989483000006, 2024-09-18T04:38:38.552 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016197989999999999, 2024-09-18T04:38:38.552 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:38.552 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:38.552 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:38.552 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:38.552 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:38.553 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 113 2024-09-18T04:38:38.553 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.553 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:38.553 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.553 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:38.553 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786816+0000", 2024-09-18T04:38:38.553 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:38.553 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.553 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.553 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:38.553 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786816+0000", 2024-09-18T04:38:38.554 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:38.554 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.554 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.554 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:38.554 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786815+0000", 2024-09-18T04:38:38.554 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:38.554 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.554 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.554 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:38.554 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786821+0000", 2024-09-18T04:38:38.554 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9529999999999996e-06 2024-09-18T04:38:38.554 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.555 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.555 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:38.555 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786841+0000", 2024-09-18T04:38:38.555 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9743000000000001e-05 2024-09-18T04:38:38.555 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.555 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.555 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:38.555 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786848+0000", 2024-09-18T04:38:38.555 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4900000000000003e-06 2024-09-18T04:38:38.555 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.555 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.556 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:38.556 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.799747+0000", 2024-09-18T04:38:38.556 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012898902 2024-09-18T04:38:38.556 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.556 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.556 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:38.556 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.799898+0000", 2024-09-18T04:38:38.556 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000150877 2024-09-18T04:38:38.556 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.556 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.556 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:38.557 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800095+0000", 2024-09-18T04:38:38.557 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000197023 2024-09-18T04:38:38.557 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.557 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.557 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:38.557 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802829+0000", 2024-09-18T04:38:38.557 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0027343480000000002 2024-09-18T04:38:38.557 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.557 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.557 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:38.557 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802847+0000", 2024-09-18T04:38:38.557 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7283e-05 2024-09-18T04:38:38.558 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.558 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.558 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:38.558 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.802974+0000", 2024-09-18T04:38:38.558 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012749199999999999 2024-09-18T04:38:38.558 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.558 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.558 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:38.558 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803014+0000", 2024-09-18T04:38:38.558 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9576000000000002e-05 2024-09-18T04:38:38.558 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:38.559 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:38.559 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:38.559 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.559 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.559 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:38.559 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:38:38.559 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 70.707903892000004, 2024-09-18T04:38:38.559 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017399218000000001, 2024-09-18T04:38:38.559 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:38.560 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:38.560 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:38.560 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:38.560 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:38.560 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 114 2024-09-18T04:38:38.560 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.560 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:38.560 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.560 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:38.560 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:38:38.560 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:38.560 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.561 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.561 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:38.561 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:38:38.561 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:38.561 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.561 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.561 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:38.561 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786900+0000", 2024-09-18T04:38:38.561 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:38.562 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.562 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.562 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:38.562 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786908+0000", 2024-09-18T04:38:38.562 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8199999999999997e-06 2024-09-18T04:38:38.562 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.562 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.562 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:38.562 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786925+0000", 2024-09-18T04:38:38.562 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6849000000000001e-05 2024-09-18T04:38:38.563 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.563 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.563 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:38.563 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786934+0000", 2024-09-18T04:38:38.563 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6299999999999993e-06 2024-09-18T04:38:38.563 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.563 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.563 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:38.563 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800926+0000", 2024-09-18T04:38:38.563 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013991632 2024-09-18T04:38:38.563 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.563 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.564 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:38.564 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801060+0000", 2024-09-18T04:38:38.564 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013378199999999999 2024-09-18T04:38:38.564 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.564 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.564 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:38.564 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801166+0000", 2024-09-18T04:38:38.564 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010607099999999999 2024-09-18T04:38:38.564 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.564 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.564 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:38.565 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803026+0000", 2024-09-18T04:38:38.565 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001860469 2024-09-18T04:38:38.565 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.565 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.565 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:38.565 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804245+0000", 2024-09-18T04:38:38.565 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0012185900000000001 2024-09-18T04:38:38.565 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.565 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.565 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:38.565 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804278+0000", 2024-09-18T04:38:38.565 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3272000000000003e-05 2024-09-18T04:38:38.566 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.566 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.566 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:38.566 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804300+0000", 2024-09-18T04:38:38.566 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2266e-05 2024-09-18T04:38:38.566 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:38.566 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:38.566 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:38.566 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.566 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.566 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:38.567 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787566+0000", 2024-09-18T04:38:38.567 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 70.707238932999999, 2024-09-18T04:38:38.567 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017124120999999999, 2024-09-18T04:38:38.567 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:38.567 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:38.567 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:38.567 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:38.567 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:38.567 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 115 2024-09-18T04:38:38.567 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.567 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:38.568 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.568 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:38.568 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787566+0000", 2024-09-18T04:38:38.568 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:38.568 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.568 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.568 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:38.568 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787566+0000", 2024-09-18T04:38:38.568 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:38.568 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.568 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.569 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:38.569 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787564+0000", 2024-09-18T04:38:38.569 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:38.569 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.569 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.569 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:38.569 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787751+0000", 2024-09-18T04:38:38.569 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000186873 2024-09-18T04:38:38.569 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.569 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.569 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:38.570 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787772+0000", 2024-09-18T04:38:38.570 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.126e-05 2024-09-18T04:38:38.570 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.570 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.570 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:38.570 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787794+0000", 2024-09-18T04:38:38.570 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1418999999999999e-05 2024-09-18T04:38:38.570 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.570 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.570 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:38.570 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801659+0000", 2024-09-18T04:38:38.571 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013865125000000001 2024-09-18T04:38:38.571 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.571 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.571 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:38.571 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801750+0000", 2024-09-18T04:38:38.571 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.1427000000000004e-05 2024-09-18T04:38:38.571 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.571 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.571 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:38.571 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801894+0000", 2024-09-18T04:38:38.571 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000143514 2024-09-18T04:38:38.572 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.572 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.572 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:38.572 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804313+0000", 2024-09-18T04:38:38.572 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0024190879999999998 2024-09-18T04:38:38.572 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.572 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.572 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:38.572 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804644+0000", 2024-09-18T04:38:38.572 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00033100799999999999 2024-09-18T04:38:38.572 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.572 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.573 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:38.573 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804672+0000", 2024-09-18T04:38:38.573 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7988e-05 2024-09-18T04:38:38.573 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.573 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.573 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:38.573 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804690+0000", 2024-09-18T04:38:38.573 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8298999999999999e-05 2024-09-18T04:38:38.573 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:38.573 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:38.573 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:38.574 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.574 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.574 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:38.574 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:38:38.574 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 70.706990993999995, 2024-09-18T04:38:38.574 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017521662, 2024-09-18T04:38:38.574 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:38.574 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:38.574 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:38.574 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:38.574 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:38.575 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 116 2024-09-18T04:38:38.575 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.575 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:38.575 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.575 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:38.575 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:38:38.575 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:38.575 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.575 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.575 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:38.576 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:38:38.576 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:38.576 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.576 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.576 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:38.576 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787813+0000", 2024-09-18T04:38:38.576 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:38.576 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.576 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.576 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:38.576 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787824+0000", 2024-09-18T04:38:38.576 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1157e-05 2024-09-18T04:38:38.577 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.577 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.577 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:38.577 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787840+0000", 2024-09-18T04:38:38.577 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6269e-05 2024-09-18T04:38:38.577 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.577 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.577 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:38.577 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787853+0000", 2024-09-18T04:38:38.577 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3156000000000001e-05 2024-09-18T04:38:38.577 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.577 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.578 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:38.578 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803110+0000", 2024-09-18T04:38:38.578 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015256736 2024-09-18T04:38:38.578 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.578 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.578 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:38.578 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803197+0000", 2024-09-18T04:38:38.578 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7195999999999997e-05 2024-09-18T04:38:38.578 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.578 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.578 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:38.578 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803308+0000", 2024-09-18T04:38:38.579 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111005 2024-09-18T04:38:38.579 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.579 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.579 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:38.579 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804316+0000", 2024-09-18T04:38:38.579 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010073949999999999 2024-09-18T04:38:38.579 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.579 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.579 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:38.579 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805232+0000", 2024-09-18T04:38:38.579 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091636300000000001 2024-09-18T04:38:38.579 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.580 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.580 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:38.580 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805277+0000", 2024-09-18T04:38:38.580 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4864000000000003e-05 2024-09-18T04:38:38.580 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.580 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.580 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:38.580 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805336+0000", 2024-09-18T04:38:38.580 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8968000000000002e-05 2024-09-18T04:38:38.580 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:38.580 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:38.581 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:38.581 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.581 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.581 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:38.581 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:38:38.581 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 70.706938938999997, 2024-09-18T04:38:38.581 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017983453, 2024-09-18T04:38:38.581 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:38.581 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:38.581 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:38.581 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:38.581 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:38.582 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 117 2024-09-18T04:38:38.582 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.582 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:38.582 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.582 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:38.582 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:38:38.582 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:38.582 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.582 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.582 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:38.582 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:38:38.582 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:38.583 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.583 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.583 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:38.583 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787865+0000", 2024-09-18T04:38:38.583 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:38.583 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.583 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.583 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:38.583 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787889+0000", 2024-09-18T04:38:38.583 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4244999999999999e-05 2024-09-18T04:38:38.583 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.584 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.584 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:38.584 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787905+0000", 2024-09-18T04:38:38.584 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5739999999999998e-05 2024-09-18T04:38:38.584 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.584 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.584 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:38.584 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787918+0000", 2024-09-18T04:38:38.584 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.252e-05 2024-09-18T04:38:38.584 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.584 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.584 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:38.585 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803693+0000", 2024-09-18T04:38:38.585 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015774872999999998 2024-09-18T04:38:38.585 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.585 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.585 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:38.585 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803775+0000", 2024-09-18T04:38:38.585 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2761000000000003e-05 2024-09-18T04:38:38.585 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.585 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.585 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:38.585 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803887+0000", 2024-09-18T04:38:38.586 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111708 2024-09-18T04:38:38.586 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.586 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.586 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:38.586 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804823+0000", 2024-09-18T04:38:38.586 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00093625100000000003 2024-09-18T04:38:38.586 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.586 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.586 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:38.586 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805804+0000", 2024-09-18T04:38:38.586 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00098114699999999992 2024-09-18T04:38:38.587 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.587 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.587 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:38.587 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805831+0000", 2024-09-18T04:38:38.587 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6401999999999999e-05 2024-09-18T04:38:38.587 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.587 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.587 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:38.587 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805850+0000", 2024-09-18T04:38:38.587 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8862e-05 2024-09-18T04:38:38.587 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:38.587 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:38.588 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:38.588 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.588 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.588 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:38.588 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.230830+0000", 2024-09-18T04:38:38.588 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 70.263975060000007, 2024-09-18T04:38:38.588 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011537146999999999, 2024-09-18T04:38:38.588 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:38.588 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:38.588 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:38.588 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:38.588 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:38.589 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 135 2024-09-18T04:38:38.589 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.589 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:38.589 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.589 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:38.589 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230830+0000", 2024-09-18T04:38:38.589 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:38.589 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.589 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.589 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:38.589 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230830+0000", 2024-09-18T04:38:38.590 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:38.590 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.590 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.590 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:38.590 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230829+0000", 2024-09-18T04:38:38.590 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:38.590 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.590 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.590 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:38.590 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230835+0000", 2024-09-18T04:38:38.590 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8660000000000002e-06 2024-09-18T04:38:38.590 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.591 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.591 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:38.591 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230844+0000", 2024-09-18T04:38:38.591 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9339999999999995e-06 2024-09-18T04:38:38.591 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.591 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.591 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:38.591 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230853+0000", 2024-09-18T04:38:38.591 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4419999999999998e-06 2024-09-18T04:38:38.591 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.591 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.592 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:38.592 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.237658+0000", 2024-09-18T04:38:38.592 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0068055310000000001 2024-09-18T04:38:38.592 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.592 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.592 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:38.592 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.237837+0000", 2024-09-18T04:38:38.592 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017943200000000001 2024-09-18T04:38:38.592 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.592 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.592 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:38.593 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.238112+0000", 2024-09-18T04:38:38.593 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00027488600000000001 2024-09-18T04:38:38.593 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.593 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.593 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:38.593 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.240505+0000", 2024-09-18T04:38:38.593 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0023930539999999999 2024-09-18T04:38:38.593 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.593 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.593 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:38.593 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242157+0000", 2024-09-18T04:38:38.593 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001651135 2024-09-18T04:38:38.594 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.594 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.594 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:38.594 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242281+0000", 2024-09-18T04:38:38.594 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012451300000000001 2024-09-18T04:38:38.594 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.594 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.594 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:38.594 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242367+0000", 2024-09-18T04:38:38.594 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6178000000000001e-05 2024-09-18T04:38:38.594 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:38.594 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:38.595 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:38.595 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.595 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.595 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:38.595 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.230858+0000", 2024-09-18T04:38:38.595 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 70.263947172000002, 2024-09-18T04:38:38.595 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013005821000000001, 2024-09-18T04:38:38.595 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:38.595 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:38.595 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:38.595 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:38.596 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:38.596 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 136 2024-09-18T04:38:38.596 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.596 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:38.596 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.596 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:38.596 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230858+0000", 2024-09-18T04:38:38.596 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:38.596 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.596 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.596 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:38.597 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230858+0000", 2024-09-18T04:38:38.597 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:38.597 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.597 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.597 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:38.597 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230858+0000", 2024-09-18T04:38:38.597 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:38.597 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.597 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.597 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:38.597 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230861+0000", 2024-09-18T04:38:38.598 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7459999999999998e-06 2024-09-18T04:38:38.598 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.598 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.598 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:38.598 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230877+0000", 2024-09-18T04:38:38.598 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5979000000000001e-05 2024-09-18T04:38:38.598 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.598 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.598 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:38.598 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230885+0000", 2024-09-18T04:38:38.598 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.5979999999999999e-06 2024-09-18T04:38:38.599 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.599 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.599 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:38.599 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.239023+0000", 2024-09-18T04:38:38.599 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0081380710000000002 2024-09-18T04:38:38.599 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.599 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.599 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:38.599 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.239216+0000", 2024-09-18T04:38:38.599 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019274799999999999 2024-09-18T04:38:38.599 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.600 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.600 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:38.600 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.239417+0000", 2024-09-18T04:38:38.600 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020093300000000001 2024-09-18T04:38:38.600 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.600 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.600 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:38.600 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242494+0000", 2024-09-18T04:38:38.600 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003077177 2024-09-18T04:38:38.600 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.600 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.600 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:38.601 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.243668+0000", 2024-09-18T04:38:38.601 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001174027 2024-09-18T04:38:38.601 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.601 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.601 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:38.601 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.243777+0000", 2024-09-18T04:38:38.601 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010955600000000001 2024-09-18T04:38:38.601 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.601 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.601 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:38.601 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.243864+0000", 2024-09-18T04:38:38.602 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6441999999999997e-05 2024-09-18T04:38:38.602 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:38.602 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:38.602 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:38.602 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.602 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.602 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:38.602 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.230901+0000", 2024-09-18T04:38:38.602 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 70.263904599, 2024-09-18T04:38:38.602 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014120882, 2024-09-18T04:38:38.602 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:38.603 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:38.603 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:38.603 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:38.603 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:38.603 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 137 2024-09-18T04:38:38.603 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.603 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:38.603 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.603 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:38.603 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230901+0000", 2024-09-18T04:38:38.603 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:38.604 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.604 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.604 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:38.604 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230901+0000", 2024-09-18T04:38:38.604 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:38.604 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.604 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.604 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:38.604 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.230900+0000", 2024-09-18T04:38:38.604 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:38.604 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.604 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.605 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:38.605 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231002+0000", 2024-09-18T04:38:38.605 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000102392 2024-09-18T04:38:38.605 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.605 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.605 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:38.605 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231029+0000", 2024-09-18T04:38:38.605 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7166999999999999e-05 2024-09-18T04:38:38.605 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.605 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.605 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:38.605 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231036+0000", 2024-09-18T04:38:38.606 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7259999999999996e-06 2024-09-18T04:38:38.606 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.606 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.606 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:38.606 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.240562+0000", 2024-09-18T04:38:38.606 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0095257970000000008 2024-09-18T04:38:38.606 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.606 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.606 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:38.606 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.240730+0000", 2024-09-18T04:38:38.606 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016767099999999999 2024-09-18T04:38:38.607 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.607 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.607 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:38.607 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.240927+0000", 2024-09-18T04:38:38.607 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019711899999999999 2024-09-18T04:38:38.607 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.607 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.607 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:38.607 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244885+0000", 2024-09-18T04:38:38.607 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0039577340000000001 2024-09-18T04:38:38.607 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.608 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.608 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:38.608 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244899+0000", 2024-09-18T04:38:38.608 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4403000000000001e-05 2024-09-18T04:38:38.608 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.608 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.608 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:38.608 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244973+0000", 2024-09-18T04:38:38.608 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4177999999999994e-05 2024-09-18T04:38:38.608 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.608 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.608 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:38.609 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.245021+0000", 2024-09-18T04:38:38.609 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8344999999999999e-05 2024-09-18T04:38:38.609 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:38.609 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:38.609 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:38.609 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.609 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.609 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:38.609 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:38:38.609 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 70.263761807999998, 2024-09-18T04:38:38.609 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016528505999999998, 2024-09-18T04:38:38.609 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:38.610 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:38.610 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:38.610 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:38.610 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:38.610 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 138 2024-09-18T04:38:38.610 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.610 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:38.610 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.610 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:38.610 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:38:38.610 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:38.611 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.611 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.611 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:38.611 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:38:38.611 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:38.611 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.611 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.611 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:38.611 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:38:38.611 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:38.611 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.612 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.612 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:38.612 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231047+0000", 2024-09-18T04:38:38.612 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5399999999999997e-06 2024-09-18T04:38:38.612 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.612 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.612 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:38.612 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231068+0000", 2024-09-18T04:38:38.612 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0769000000000001e-05 2024-09-18T04:38:38.612 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.612 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.612 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:38.613 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231074+0000", 2024-09-18T04:38:38.613 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6180000000000003e-06 2024-09-18T04:38:38.613 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.613 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.613 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:38.613 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242556+0000", 2024-09-18T04:38:38.613 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011482654 2024-09-18T04:38:38.613 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.613 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.613 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:38.613 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242666+0000", 2024-09-18T04:38:38.613 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011003200000000001 2024-09-18T04:38:38.614 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.614 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.614 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:38.614 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242924+0000", 2024-09-18T04:38:38.614 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00025785899999999999 2024-09-18T04:38:38.614 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.614 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.614 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:38.614 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247263+0000", 2024-09-18T04:38:38.614 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0043387649999999996 2024-09-18T04:38:38.614 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.614 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.615 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:38.615 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247412+0000", 2024-09-18T04:38:38.615 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000149276 2024-09-18T04:38:38.615 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.615 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.615 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:38.615 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247522+0000", 2024-09-18T04:38:38.615 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000109788 2024-09-18T04:38:38.615 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.615 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.615 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:38.616 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247572+0000", 2024-09-18T04:38:38.616 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9808999999999997e-05 2024-09-18T04:38:38.616 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:38.616 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:38.616 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:38.616 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.616 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.616 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:38.616 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231083+0000", 2024-09-18T04:38:38.616 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 70.263722345000005, 2024-09-18T04:38:38.617 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0166731, 2024-09-18T04:38:38.617 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:38.617 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:38.617 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:38.617 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:38.617 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:38.617 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 139 2024-09-18T04:38:38.617 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.617 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:38.617 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.617 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:38.617 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231083+0000", 2024-09-18T04:38:38.618 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:38.618 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.618 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.618 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:38.618 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231083+0000", 2024-09-18T04:38:38.618 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:38.618 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.618 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.618 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:38.618 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231082+0000", 2024-09-18T04:38:38.618 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:38.618 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.619 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.619 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:38.619 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231090+0000", 2024-09-18T04:38:38.619 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1280000000000008e-06 2024-09-18T04:38:38.619 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.619 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.619 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:38.619 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231097+0000", 2024-09-18T04:38:38.619 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5119999999999997e-06 2024-09-18T04:38:38.619 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.620 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.620 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:38.620 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231106+0000", 2024-09-18T04:38:38.620 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9230000000000008e-06 2024-09-18T04:38:38.620 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.620 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.620 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:38.620 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.243977+0000", 2024-09-18T04:38:38.620 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012870754 2024-09-18T04:38:38.620 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.620 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.620 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:38.621 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244112+0000", 2024-09-18T04:38:38.621 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000135575 2024-09-18T04:38:38.621 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.621 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.621 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:38.621 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244265+0000", 2024-09-18T04:38:38.621 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015279900000000001 2024-09-18T04:38:38.621 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.621 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.621 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:38.621 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247649+0000", 2024-09-18T04:38:38.621 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003383896 2024-09-18T04:38:38.622 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.622 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.622 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:38.622 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247656+0000", 2024-09-18T04:38:38.622 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8920000000000003e-06 2024-09-18T04:38:38.622 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.622 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.622 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:38.622 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247692+0000", 2024-09-18T04:38:38.622 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6470000000000001e-05 2024-09-18T04:38:38.622 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.622 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.623 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:38.623 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247756+0000", 2024-09-18T04:38:38.623 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3676999999999994e-05 2024-09-18T04:38:38.623 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:38.623 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:38.623 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:38.623 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.623 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.623 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:38.623 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231190+0000", 2024-09-18T04:38:38.623 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 70.263615419000004, 2024-09-18T04:38:38.624 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016793556000000001, 2024-09-18T04:38:38.624 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:38.624 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:38.624 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:38.624 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:38.624 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:38.624 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 140 2024-09-18T04:38:38.624 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.624 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:38.624 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.624 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:38.624 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231190+0000", 2024-09-18T04:38:38.625 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:38.625 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.625 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.625 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:38.625 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231190+0000", 2024-09-18T04:38:38.625 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:38.625 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.625 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.625 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:38.625 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231189+0000", 2024-09-18T04:38:38.625 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:38.625 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.626 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.626 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:38.626 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231273+0000", 2024-09-18T04:38:38.626 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4445999999999998e-05 2024-09-18T04:38:38.626 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.626 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.626 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:38.626 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231282+0000", 2024-09-18T04:38:38.626 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8859999999999993e-06 2024-09-18T04:38:38.626 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.626 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.626 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:38.627 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231288+0000", 2024-09-18T04:38:38.627 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9819999999999996e-06 2024-09-18T04:38:38.627 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.627 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.627 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:38.627 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.245119+0000", 2024-09-18T04:38:38.627 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.01383121 2024-09-18T04:38:38.627 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.627 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.627 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:38.627 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.245247+0000", 2024-09-18T04:38:38.628 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012763299999999999 2024-09-18T04:38:38.628 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.628 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.628 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:38.628 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.245401+0000", 2024-09-18T04:38:38.628 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015399200000000001 2024-09-18T04:38:38.628 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.628 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.628 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:38.628 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247766+0000", 2024-09-18T04:38:38.628 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0023649510000000001 2024-09-18T04:38:38.629 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.629 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.629 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:38.629 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247904+0000", 2024-09-18T04:38:38.629 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013836499999999999 2024-09-18T04:38:38.629 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.629 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.629 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:38.629 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247950+0000", 2024-09-18T04:38:38.629 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5201000000000003e-05 2024-09-18T04:38:38.629 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.629 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.630 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:38.630 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247983+0000", 2024-09-18T04:38:38.630 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3686000000000002e-05 2024-09-18T04:38:38.630 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:38.630 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:38.630 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:38.630 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.630 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.630 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:38.630 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:38:38.630 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 70.263504675999997, 2024-09-18T04:38:38.630 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017344024, 2024-09-18T04:38:38.631 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:38.631 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:38.631 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:38.631 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:38.631 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:38.631 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 141 2024-09-18T04:38:38.631 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.631 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:38.631 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.631 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:38.631 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:38:38.632 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:38.632 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.632 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.632 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:38.632 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:38:38.632 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:38.632 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.632 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.632 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:38.632 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:38:38.632 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:38.632 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.633 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.633 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:38.633 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231309+0000", 2024-09-18T04:38:38.633 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5419999999999994e-06 2024-09-18T04:38:38.633 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.633 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.633 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:38.633 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231323+0000", 2024-09-18T04:38:38.633 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3108000000000001e-05 2024-09-18T04:38:38.633 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.633 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.634 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:38.634 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231340+0000", 2024-09-18T04:38:38.634 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.696e-05 2024-09-18T04:38:38.634 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.634 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.634 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:38.634 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.246143+0000", 2024-09-18T04:38:38.634 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014803320999999999 2024-09-18T04:38:38.634 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.634 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.634 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:38.634 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.246323+0000", 2024-09-18T04:38:38.635 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00018061600000000001 2024-09-18T04:38:38.635 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.635 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.635 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:38.635 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.246530+0000", 2024-09-18T04:38:38.635 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020625 2024-09-18T04:38:38.635 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.635 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.635 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:38.635 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248002+0000", 2024-09-18T04:38:38.635 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0014719379999999999 2024-09-18T04:38:38.636 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.636 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.636 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:38.636 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248582+0000", 2024-09-18T04:38:38.636 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00058020599999999999 2024-09-18T04:38:38.636 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.636 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.636 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:38.636 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248618+0000", 2024-09-18T04:38:38.636 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6559e-05 2024-09-18T04:38:38.636 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.637 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.637 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:38.637 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248645+0000", 2024-09-18T04:38:38.637 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6080000000000001e-05 2024-09-18T04:38:38.637 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:38.637 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:38.637 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:38.637 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.637 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.637 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-18T04:38:38.637 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.878463+0000", 2024-09-18T04:38:38.638 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 69.616342642000006, 2024-09-18T04:38:38.638 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010882697, 2024-09-18T04:38:38.638 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:38.638 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:38.638 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:38.638 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:38.638 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:38.638 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 159 2024-09-18T04:38:38.638 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.638 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:38.638 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.638 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:38.639 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878463+0000", 2024-09-18T04:38:38.639 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:38.639 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.639 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.639 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:38.639 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878463+0000", 2024-09-18T04:38:38.639 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:38.639 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.639 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.639 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:38.639 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878462+0000", 2024-09-18T04:38:38.639 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:38.640 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.640 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.640 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:38.640 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878467+0000", 2024-09-18T04:38:38.640 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4099999999999999e-06 2024-09-18T04:38:38.640 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.640 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.640 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:38.640 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878490+0000", 2024-09-18T04:38:38.640 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3159999999999998e-05 2024-09-18T04:38:38.640 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.641 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.641 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:38.641 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.878511+0000", 2024-09-18T04:38:38.641 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1092000000000001e-05 2024-09-18T04:38:38.641 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.641 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.641 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:38.641 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.887163+0000", 2024-09-18T04:38:38.641 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.008651845 2024-09-18T04:38:38.641 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.641 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.641 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:38.641 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.887273+0000", 2024-09-18T04:38:38.642 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000109823 2024-09-18T04:38:38.642 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.642 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.642 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:38.642 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.887441+0000", 2024-09-18T04:38:38.642 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016777599999999999 2024-09-18T04:38:38.642 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.642 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.642 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:38.642 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.888634+0000", 2024-09-18T04:38:38.642 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001192839 2024-09-18T04:38:38.642 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.643 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.643 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:38.643 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.889153+0000", 2024-09-18T04:38:38.643 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00051953700000000004 2024-09-18T04:38:38.643 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.643 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.643 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:38.643 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.889234+0000", 2024-09-18T04:38:38.643 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0802000000000003e-05 2024-09-18T04:38:38.643 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:38.643 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:38.643 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:38.644 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.889345+0000", 2024-09-18T04:38:38.644 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011112000000000001 2024-09-18T04:38:38.644 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:38.644 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:38.644 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:38.644 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:38.644 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:38.644 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:38.644 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T04:38:38.647 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:38:38.647 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:38.642+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:38.747 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:38.742+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:38.797 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:38:38.808 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:38.808 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:38.809 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:38:38.809 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:38.819 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:38:38.820 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:38:38.842 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:38:38.848 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:38.842+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:38.865 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:38:38.889 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:38:38.909 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:38:38.911 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:38:38.912 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:38:38.912 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:38:38.912 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:38:38.912 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:38:38.912 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:38:38.922 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:38.922 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:38.928 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-09-18T04:38:38.933 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:38:38.978 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:38.974+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:39.037 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:39.037 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:39.046 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:38:39.051 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:39.046+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:39.151 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:39.150+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:39.155 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:39.155 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:39.165 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:38:39.253 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:39.246+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:39.274 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:39.274 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:39.284 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:38:39.353 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:39.346+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:39.386 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:39.386 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:39.396 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:38:39.454 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:39.450+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:39.514 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:39.515 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:39.525 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:38:39.555 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:39.550+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:39.636 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:39.636 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:39.646 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:38:39.694 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:39.690+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:39.755 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:39.755 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:39.756 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:39.750+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:39.765 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:38:39.791 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-18T04:38:39.791 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T04:38:39.791 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-09-18T04:38:39.791 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-18T04:38:39.791 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-18T04:38:39.792 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3 2024-09-18T04:38:39.858 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:39.854+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:39.877 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:39.877 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:39.887 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:38:39.914 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:39.914 INFO:teuthology.orchestra.run.smithi082.stdout: "success": "" 2024-09-18T04:38:39.914 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:39.959 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:39.954+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:40.053 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:40.053 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:40.059 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:40.054+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:40.064 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:38:40.160 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:40.154+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:40.177 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:40.177 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:40.187 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:38:40.261 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:40.254+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:40.295 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:40.296 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:40.306 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:38:40.362 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:40.358+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:40.414 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:40.414 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:40.424 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:38:40.463 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:40.458+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:40.528 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:40.528 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:40.538 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:38:40.564 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:40.558+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:40.658 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:40.658 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:40.665 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:40.658+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:40.669 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:38:40.700 INFO:tasks.workunit.client.0.smithi115.stderr:Note: switching to 'a477b1894d5c5a613f23260ca0e6b96d79652ba7'. 2024-09-18T04:38:40.700 INFO:tasks.workunit.client.0.smithi115.stderr: 2024-09-18T04:38:40.700 INFO:tasks.workunit.client.0.smithi115.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-18T04:38:40.700 INFO:tasks.workunit.client.0.smithi115.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-18T04:38:40.700 INFO:tasks.workunit.client.0.smithi115.stderr:state without impacting any branches by switching back to a branch. 2024-09-18T04:38:40.700 INFO:tasks.workunit.client.0.smithi115.stderr: 2024-09-18T04:38:40.700 INFO:tasks.workunit.client.0.smithi115.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-18T04:38:40.700 INFO:tasks.workunit.client.0.smithi115.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-18T04:38:40.700 INFO:tasks.workunit.client.0.smithi115.stderr: 2024-09-18T04:38:40.700 INFO:tasks.workunit.client.0.smithi115.stderr: git switch -c 2024-09-18T04:38:40.700 INFO:tasks.workunit.client.0.smithi115.stderr: 2024-09-18T04:38:40.701 INFO:tasks.workunit.client.0.smithi115.stderr:Or undo this operation with: 2024-09-18T04:38:40.701 INFO:tasks.workunit.client.0.smithi115.stderr: 2024-09-18T04:38:40.701 INFO:tasks.workunit.client.0.smithi115.stderr: git switch - 2024-09-18T04:38:40.701 INFO:tasks.workunit.client.0.smithi115.stderr: 2024-09-18T04:38:40.701 INFO:tasks.workunit.client.0.smithi115.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-18T04:38:40.701 INFO:tasks.workunit.client.0.smithi115.stderr: 2024-09-18T04:38:40.701 INFO:tasks.workunit.client.0.smithi115.stderr:HEAD is now at a477b1894d5 Merge pull request #59761 from rhcs-dashboard/reef-hosts-selection-show-only-10-entries-fix 2024-09-18T04:38:40.706 DEBUG:teuthology.orchestra.run.smithi115:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-18T04:38:40.734 INFO:tasks.workunit.client.0.smithi115.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-18T04:38:40.737 INFO:tasks.workunit.client.0.smithi115.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-18T04:38:40.737 INFO:tasks.workunit.client.0.smithi115.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-18T04:38:40.766 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:40.762+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:40.773 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:40.773 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:40.783 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:38:40.868 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:40.866+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:40.887 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:40.887 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:40.897 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:38:40.924 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T04:38:40.940 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-18T04:38:40.970 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:40.966+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:41.007 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:41.008 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:41.018 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:38:41.069 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:41.062+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:41.075 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:38:41.095 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:38:41.120 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:38:41.125 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:41.125 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:41.135 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:38:41.142 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:38:41.160 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:38:41.170 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:41.166+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:41.182 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:38:41.185 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:38:41.185 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:38:41.185 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:38:41.185 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:38:41.185 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:38:41.185 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:38:41.246 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:41.246 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:38:41.246 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:38:41.246 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [] 2024-09-18T04:38:41.246 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:41.255 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:38:41.270 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:41.266+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:41.362 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:41.362 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:41.362 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:38:41.362 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:41.371 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:41.366+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:41.372 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:38:41.472 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:41.466+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:41.526 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:41.526 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:41.526 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:38:41.526 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:38:41.526 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:41.536 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:38:41.540 INFO:tasks.workunit.client.0.smithi115.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-18T04:38:41.572 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:41.566+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:41.631 INFO:tasks.workunit.client.0.smithi115.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-18T04:38:41.648 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:41.648 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:38:41.648 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:38:41.648 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:38:41.648 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.648 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-18T04:38:41.648 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:38:41.649 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 67.174970439000006, 2024-09-18T04:38:41.649 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0095781189999999995, 2024-09-18T04:38:41.649 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:41.649 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:41.649 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:41.649 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:38:41.649 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:38:41.649 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 1 2024-09-18T04:38:41.649 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.649 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:41.649 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.649 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:41.649 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:38:41.649 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:41.650 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.650 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.650 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:41.650 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:38:41.650 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:41.650 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.650 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.650 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:41.650 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469862+0000", 2024-09-18T04:38:41.650 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:41.650 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.650 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.650 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:41.650 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469879+0000", 2024-09-18T04:38:41.651 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7431999999999998e-05 2024-09-18T04:38:41.651 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.651 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.651 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:41.651 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469896+0000", 2024-09-18T04:38:41.651 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6742999999999999e-05 2024-09-18T04:38:41.651 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.651 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.651 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:41.651 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469915+0000", 2024-09-18T04:38:41.651 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8991000000000002e-05 2024-09-18T04:38:41.651 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.651 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.651 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:41.652 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470397+0000", 2024-09-18T04:38:41.652 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048167799999999998 2024-09-18T04:38:41.652 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.652 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.652 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-09-18T04:38:41.652 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470412+0000", 2024-09-18T04:38:41.652 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5526e-05 2024-09-18T04:38:41.652 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.652 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.652 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:41.652 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473279+0000", 2024-09-18T04:38:41.652 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0028667469999999998 2024-09-18T04:38:41.653 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.653 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.653 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:41.653 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473441+0000", 2024-09-18T04:38:41.653 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000162082 2024-09-18T04:38:41.653 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.653 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.653 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:38:41.653 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473573+0000", 2024-09-18T04:38:41.653 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000132032 2024-09-18T04:38:41.653 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.653 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.653 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:41.653 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474826+0000", 2024-09-18T04:38:41.654 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001252717 2024-09-18T04:38:41.654 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.654 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.654 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:41.654 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479307+0000", 2024-09-18T04:38:41.654 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044813220000000003 2024-09-18T04:38:41.654 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.654 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.654 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:41.654 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479366+0000", 2024-09-18T04:38:41.654 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9024999999999998e-05 2024-09-18T04:38:41.654 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.654 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.654 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:41.655 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479443+0000", 2024-09-18T04:38:41.655 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6339000000000002e-05 2024-09-18T04:38:41.655 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:41.655 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:41.655 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:41.655 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.655 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.655 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.4214.0:1 2.3 e18/17 ondisk, result = 0)", 2024-09-18T04:38:41.655 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.474724+0000", 2024-09-18T04:38:41.655 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 67.170111074999994, 2024-09-18T04:38:41.655 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000105358, 2024-09-18T04:38:41.655 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:41.655 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:38:41.655 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:41.656 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.656 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:41.656 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474724+0000", 2024-09-18T04:38:41.656 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:41.656 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.656 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.656 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:41.656 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474724+0000", 2024-09-18T04:38:41.656 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:41.656 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.656 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.656 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:41.656 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474723+0000", 2024-09-18T04:38:41.656 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:41.657 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.657 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.657 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:41.657 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474731+0000", 2024-09-18T04:38:41.657 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2339999999999994e-06 2024-09-18T04:38:41.657 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.657 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.657 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:41.657 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474748+0000", 2024-09-18T04:38:41.657 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6603e-05 2024-09-18T04:38:41.657 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.657 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.657 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:41.657 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474760+0000", 2024-09-18T04:38:41.658 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2215000000000001e-05 2024-09-18T04:38:41.658 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.658 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.658 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:41.658 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474814+0000", 2024-09-18T04:38:41.658 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4267000000000001e-05 2024-09-18T04:38:41.658 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.658 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.658 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:41.658 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474821+0000", 2024-09-18T04:38:41.658 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0929999999999997e-06 2024-09-18T04:38:41.658 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.658 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.659 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:41.659 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474829+0000", 2024-09-18T04:38:41.659 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1200000000000002e-06 2024-09-18T04:38:41.659 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:41.659 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:41.659 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:41.659 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.659 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.659 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18)", 2024-09-18T04:38:41.659 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.482346+0000", 2024-09-18T04:38:41.659 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 67.162488643000003, 2024-09-18T04:38:41.659 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020435599999999999, 2024-09-18T04:38:41.659 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:41.659 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-09-18T04:38:41.660 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:41.660 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:38:41.660 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:38:41.660 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 2 2024-09-18T04:38:41.660 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.660 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:41.660 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.660 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:41.660 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482346+0000", 2024-09-18T04:38:41.660 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:41.660 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.660 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.660 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:41.660 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482346+0000", 2024-09-18T04:38:41.661 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:41.661 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.661 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.661 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:41.661 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482345+0000", 2024-09-18T04:38:41.661 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:41.661 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.661 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.661 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:41.661 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482360+0000", 2024-09-18T04:38:41.661 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4854999999999999e-05 2024-09-18T04:38:41.661 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.661 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.661 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:41.662 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482377+0000", 2024-09-18T04:38:41.662 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7785999999999999e-05 2024-09-18T04:38:41.662 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.662 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.662 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:41.662 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482387+0000", 2024-09-18T04:38:41.662 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6800000000000005e-06 2024-09-18T04:38:41.662 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.662 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.662 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:41.662 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482441+0000", 2024-09-18T04:38:41.662 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4046999999999999e-05 2024-09-18T04:38:41.662 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.662 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.663 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:41.663 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482551+0000", 2024-09-18T04:38:41.663 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000109468 2024-09-18T04:38:41.663 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:41.663 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:41.663 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:41.663 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.663 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.663 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18)", 2024-09-18T04:38:41.663 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.482853+0000", 2024-09-18T04:38:41.663 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 67.161982025, 2024-09-18T04:38:41.663 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00029815799999999997, 2024-09-18T04:38:41.663 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:41.663 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-09-18T04:38:41.663 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:41.664 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:38:41.664 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:38:41.664 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 3 2024-09-18T04:38:41.664 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.664 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:41.664 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.664 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:41.664 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482853+0000", 2024-09-18T04:38:41.664 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:41.664 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.664 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.664 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:41.664 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482853+0000", 2024-09-18T04:38:41.664 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:41.665 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.665 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.665 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:41.665 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482852+0000", 2024-09-18T04:38:41.665 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:41.665 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.665 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.665 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:41.665 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482859+0000", 2024-09-18T04:38:41.665 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0380000000000002e-06 2024-09-18T04:38:41.665 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.665 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.665 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:41.665 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482880+0000", 2024-09-18T04:38:41.666 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1243e-05 2024-09-18T04:38:41.666 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.666 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.666 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:41.666 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482896+0000", 2024-09-18T04:38:41.666 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5977e-05 2024-09-18T04:38:41.666 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.666 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.666 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:41.666 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.482970+0000", 2024-09-18T04:38:41.666 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4603000000000004e-05 2024-09-18T04:38:41.666 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.666 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.666 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:41.667 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.483151+0000", 2024-09-18T04:38:41.667 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00018059799999999999 2024-09-18T04:38:41.667 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:41.667 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:41.667 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:41.667 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.667 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.667 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-18T04:38:41.667 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:35.468698+0000", 2024-09-18T04:38:41.667 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 66.176137187999998, 2024-09-18T04:38:41.667 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0039954129999999997, 2024-09-18T04:38:41.667 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:41.667 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:41.667 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:41.668 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:38:41.668 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:38:41.668 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 5 2024-09-18T04:38:41.668 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.668 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:41.668 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.668 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:41.668 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468698+0000", 2024-09-18T04:38:41.668 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:41.668 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.668 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.668 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:41.668 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468698+0000", 2024-09-18T04:38:41.668 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:41.669 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.669 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.669 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:41.669 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468696+0000", 2024-09-18T04:38:41.669 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:41.669 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.669 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.669 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:41.669 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468714+0000", 2024-09-18T04:38:41.669 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8743e-05 2024-09-18T04:38:41.669 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.669 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.669 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:41.670 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468748+0000", 2024-09-18T04:38:41.670 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3339e-05 2024-09-18T04:38:41.670 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.670 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.670 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:41.670 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468771+0000", 2024-09-18T04:38:41.670 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3119999999999999e-05 2024-09-18T04:38:41.670 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.670 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.670 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:41.670 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468922+0000", 2024-09-18T04:38:41.670 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000151383 2024-09-18T04:38:41.670 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.670 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.670 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:41.671 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.469278+0000", 2024-09-18T04:38:41.671 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00035628599999999998 2024-09-18T04:38:41.671 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.671 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.671 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:41.671 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.469629+0000", 2024-09-18T04:38:41.671 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00035014300000000003 2024-09-18T04:38:41.671 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.671 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.671 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:41.671 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.471620+0000", 2024-09-18T04:38:41.671 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001991278 2024-09-18T04:38:41.671 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.671 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.672 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:41.672 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472440+0000", 2024-09-18T04:38:41.672 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00081998599999999998 2024-09-18T04:38:41.672 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.672 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.672 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:41.672 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472592+0000", 2024-09-18T04:38:41.672 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015227400000000001 2024-09-18T04:38:41.672 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.672 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.672 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:41.672 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472693+0000", 2024-09-18T04:38:41.672 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000101074 2024-09-18T04:38:41.672 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:41.673 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:41.673 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:41.673 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.673 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.673 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.4214.0:5 2.2 e18/17 ondisk, result = 0)", 2024-09-18T04:38:41.673 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:35.472226+0000", 2024-09-18T04:38:41.673 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 66.172609288000004, 2024-09-18T04:38:41.673 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048505299999999999, 2024-09-18T04:38:41.673 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:41.673 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:38:41.673 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:41.673 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.673 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:41.673 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472226+0000", 2024-09-18T04:38:41.674 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:41.674 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.674 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.674 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:41.674 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472226+0000", 2024-09-18T04:38:41.674 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:41.674 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.674 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.674 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:41.674 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472224+0000", 2024-09-18T04:38:41.674 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:41.674 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.674 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.674 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:41.674 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472238+0000", 2024-09-18T04:38:41.675 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.468e-05 2024-09-18T04:38:41.675 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.675 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.675 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:41.675 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472272+0000", 2024-09-18T04:38:41.675 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3586e-05 2024-09-18T04:38:41.675 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.675 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.675 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:41.675 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472297+0000", 2024-09-18T04:38:41.675 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5282999999999998e-05 2024-09-18T04:38:41.675 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.675 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.675 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:41.676 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472396+0000", 2024-09-18T04:38:41.676 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.8394999999999996e-05 2024-09-18T04:38:41.676 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.676 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.676 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:41.676 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472421+0000", 2024-09-18T04:38:41.676 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5479000000000001e-05 2024-09-18T04:38:41.676 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.676 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.676 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:41.676 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472711+0000", 2024-09-18T04:38:41.676 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000289714 2024-09-18T04:38:41.676 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:41.676 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:41.677 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:41.677 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.677 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.677 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-18T04:38:41.677 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:38:41.677 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 65.161962019000001, 2024-09-18T04:38:41.677 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082699560000000002, 2024-09-18T04:38:41.677 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:41.677 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:41.677 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:41.677 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:38:41.677 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:38:41.677 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 7 2024-09-18T04:38:41.677 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.677 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:41.678 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.678 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:41.678 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:38:41.678 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:41.678 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.678 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.678 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:41.678 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:38:41.678 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:41.678 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.678 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.678 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:41.678 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482870+0000", 2024-09-18T04:38:41.678 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999967 2024-09-18T04:38:41.679 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.679 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.679 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:41.679 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482889+0000", 2024-09-18T04:38:41.679 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.965e-05 2024-09-18T04:38:41.679 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.679 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.679 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:41.679 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482938+0000", 2024-09-18T04:38:41.679 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8548999999999999e-05 2024-09-18T04:38:41.679 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.679 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.679 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:41.679 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482963+0000", 2024-09-18T04:38:41.680 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5075999999999998e-05 2024-09-18T04:38:41.680 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.680 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.680 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:41.680 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483121+0000", 2024-09-18T04:38:41.680 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015819400000000001 2024-09-18T04:38:41.680 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.680 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.680 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op must wait for map", 2024-09-18T04:38:41.680 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483138+0000", 2024-09-18T04:38:41.680 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7068999999999999e-05 2024-09-18T04:38:41.680 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.680 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.680 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:41.681 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488666+0000", 2024-09-18T04:38:41.681 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0055282049999999996 2024-09-18T04:38:41.681 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.681 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.681 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:41.681 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488783+0000", 2024-09-18T04:38:41.681 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000116737 2024-09-18T04:38:41.681 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.681 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.681 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:41.681 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488991+0000", 2024-09-18T04:38:41.681 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020781400000000001 2024-09-18T04:38:41.681 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.681 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.682 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:41.682 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490554+0000", 2024-09-18T04:38:41.682 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0015628370000000001 2024-09-18T04:38:41.682 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.682 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.682 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:41.682 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491019+0000", 2024-09-18T04:38:41.682 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000465608 2024-09-18T04:38:41.682 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.682 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.682 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:41.682 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491087+0000", 2024-09-18T04:38:41.682 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7988e-05 2024-09-18T04:38:41.682 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.682 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.683 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:41.683 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491143+0000", 2024-09-18T04:38:41.683 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5538999999999998e-05 2024-09-18T04:38:41.683 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:41.683 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:41.683 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:41.683 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.683 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.683 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.4214.0:7 2.2 e19/17 ondisk, result = 0)", 2024-09-18T04:38:41.683 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:36.490861+0000", 2024-09-18T04:38:41.683 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 65.153974396999999, 2024-09-18T04:38:41.683 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00029639400000000001, 2024-09-18T04:38:41.683 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:41.683 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:38:41.684 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:41.684 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.684 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:41.684 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490861+0000", 2024-09-18T04:38:41.684 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:41.684 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.684 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.684 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:41.684 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490861+0000", 2024-09-18T04:38:41.684 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:41.684 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.684 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.684 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:41.684 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490859+0000", 2024-09-18T04:38:41.685 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:41.685 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.685 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.685 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:41.685 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490867+0000", 2024-09-18T04:38:41.685 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.3089999999999996e-06 2024-09-18T04:38:41.685 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.685 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.685 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:41.685 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490909+0000", 2024-09-18T04:38:41.685 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1919000000000003e-05 2024-09-18T04:38:41.685 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.685 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.685 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:41.686 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490922+0000", 2024-09-18T04:38:41.686 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2836999999999999e-05 2024-09-18T04:38:41.686 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.686 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.686 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:41.686 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491003+0000", 2024-09-18T04:38:41.686 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0681999999999998e-05 2024-09-18T04:38:41.686 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.686 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.686 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:41.686 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491013+0000", 2024-09-18T04:38:41.686 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0220999999999999e-05 2024-09-18T04:38:41.686 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.686 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.686 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:41.687 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491157+0000", 2024-09-18T04:38:41.687 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014401799999999999 2024-09-18T04:38:41.687 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:41.687 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:41.687 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:41.687 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.687 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.687 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "MOSDPGPushReply(2.3 25/24 [PushReplyOp(2:c4c92e5a:::rbd_trash:head)])", 2024-09-18T04:38:41.687 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:51.680323+0000", 2024-09-18T04:38:41.687 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 49.964512022000001, 2024-09-18T04:38:41.687 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00054411499999999996, 2024-09-18T04:38:41.687 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:41.687 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-09-18T04:38:41.687 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:41.688 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.688 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:41.688 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680323+0000", 2024-09-18T04:38:41.688 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:41.688 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.688 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.688 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:41.688 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680323+0000", 2024-09-18T04:38:41.688 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:41.688 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.688 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.688 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:41.688 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680322+0000", 2024-09-18T04:38:41.688 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:41.688 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.689 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.689 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:41.689 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680333+0000", 2024-09-18T04:38:41.689 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1153e-05 2024-09-18T04:38:41.689 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.689 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.689 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:41.689 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680357+0000", 2024-09-18T04:38:41.689 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3252999999999999e-05 2024-09-18T04:38:41.689 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.689 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.689 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:41.689 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680378+0000", 2024-09-18T04:38:41.689 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1503000000000001e-05 2024-09-18T04:38:41.690 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.690 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.690 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:41.690 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680522+0000", 2024-09-18T04:38:41.690 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014410700000000001 2024-09-18T04:38:41.690 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:41.690 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:41.690 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:41.690 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:51.680867+0000", 2024-09-18T04:38:41.690 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00034475899999999998 2024-09-18T04:38:41.690 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:41.690 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:41.690 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:41.690 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:41.691 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:41.691 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:41.692 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:41.686+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:41.692 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:38:41.696 INFO:tasks.workunit.client.0.smithi115.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-18T04:38:41.698 INFO:tasks.workunit.client.0.smithi115.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-18T04:38:41.698 INFO:tasks.workunit.client.0.smithi115.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-18T04:38:41.761 INFO:tasks.workunit.client.0.smithi115.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-18T04:38:41.768 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-18T04:38:41.768 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-18T04:38:41.792 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:41.790+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:41.807 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:41.807 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:41.807 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:38:41.807 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:41.818 INFO:tasks.workunit:Running workunits matching rbd/rbd-nbd.sh on client.0... 2024-09-18T04:38:41.819 INFO:tasks.workunit:Running workunit rbd/rbd-nbd.sh... 2024-09-18T04:38:41.819 DEBUG:teuthology.orchestra.run.smithi115:workunit test rbd/rbd-nbd.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a477b1894d5c5a613f23260ca0e6b96d79652ba7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd-nbd.sh 2024-09-18T04:38:41.820 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:38:41.834 INFO:tasks.workunit.client.0.smithi115.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd-nbd.sh 2024-09-18T04:38:41.836 INFO:tasks.workunit.client.0.smithi115.stderr:+ . /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/../../standalone/ceph-helpers.sh 2024-09-18T04:38:41.836 INFO:tasks.workunit.client.0.smithi115.stderr:++ TIMEOUT=300 2024-09-18T04:38:41.836 INFO:tasks.workunit.client.0.smithi115.stderr:++ WAIT_FOR_CLEAN_TIMEOUT=90 2024-09-18T04:38:41.836 INFO:tasks.workunit.client.0.smithi115.stderr:++ MAX_TIMEOUT=15 2024-09-18T04:38:41.837 INFO:tasks.workunit.client.0.smithi115.stderr:++ PG_NUM=4 2024-09-18T04:38:41.837 INFO:tasks.workunit.client.0.smithi115.stderr:++ TMPDIR=/tmp 2024-09-18T04:38:41.837 INFO:tasks.workunit.client.0.smithi115.stderr:++ CEPH_BUILD_VIRTUALENV=/tmp 2024-09-18T04:38:41.837 INFO:tasks.workunit.client.0.smithi115.stderr:++ TESTDIR=/home/ubuntu/cephtest 2024-09-18T04:38:41.837 INFO:tasks.workunit.client.0.smithi115.stderr:++ type xmlstarlet 2024-09-18T04:38:41.837 INFO:tasks.workunit.client.0.smithi115.stderr:++ XMLSTARLET=xmlstarlet 2024-09-18T04:38:41.837 INFO:tasks.workunit.client.0.smithi115.stderr:+++ uname 2024-09-18T04:38:41.838 INFO:tasks.workunit.client.0.smithi115.stderr:++ '[' Linux = FreeBSD ']' 2024-09-18T04:38:41.838 INFO:tasks.workunit.client.0.smithi115.stderr:++ SED=sed 2024-09-18T04:38:41.838 INFO:tasks.workunit.client.0.smithi115.stderr:++ AWK=awk 2024-09-18T04:38:41.839 INFO:tasks.workunit.client.0.smithi115.stderr:+++ stty -a 2024-09-18T04:38:41.839 INFO:tasks.workunit.client.0.smithi115.stderr:+++ head -1 2024-09-18T04:38:41.840 INFO:tasks.workunit.client.0.smithi115.stderr:+++ sed -e 's/.*columns \([0-9]*\).*/\1/' 2024-09-18T04:38:41.848 INFO:tasks.workunit.client.0.smithi115.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-18T04:38:41.848 INFO:tasks.workunit.client.0.smithi115.stderr:++ termwidth= 2024-09-18T04:38:41.849 INFO:tasks.workunit.client.0.smithi115.stderr:++ '[' -n '' -a '' '!=' 0 ']' 2024-09-18T04:38:41.849 INFO:tasks.workunit.client.0.smithi115.stderr:++ DIFFCOLOPTS='-y ' 2024-09-18T04:38:41.849 INFO:tasks.workunit.client.0.smithi115.stderr:++ KERNCORE=kernel.core_pattern 2024-09-18T04:38:41.849 INFO:tasks.workunit.client.0.smithi115.stderr:++ EXTRA_OPTS= 2024-09-18T04:38:41.857 INFO:tasks.workunit.client.0.smithi115.stderr:++ test '' = TESTS 2024-09-18T04:38:41.858 INFO:tasks.workunit.client.0.smithi115.stderr:+ POOL=rbd 2024-09-18T04:38:41.858 INFO:tasks.workunit.client.0.smithi115.stderr:+ ANOTHER_POOL=new_default_pool17342 2024-09-18T04:38:41.858 INFO:tasks.workunit.client.0.smithi115.stderr:+ NS=ns 2024-09-18T04:38:41.858 INFO:tasks.workunit.client.0.smithi115.stderr:+ IMAGE=testrbdnbd17342 2024-09-18T04:38:41.858 INFO:tasks.workunit.client.0.smithi115.stderr:+ SIZE=64 2024-09-18T04:38:41.858 INFO:tasks.workunit.client.0.smithi115.stderr:+ DATA= 2024-09-18T04:38:41.858 INFO:tasks.workunit.client.0.smithi115.stderr:+ DEV= 2024-09-18T04:38:41.859 INFO:tasks.workunit.client.0.smithi115.stderr:+ setup 2024-09-18T04:38:41.859 INFO:tasks.workunit.client.0.smithi115.stderr:+ local ns x 2024-09-18T04:38:41.859 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' -e CMakeCache.txt ']' 2024-09-18T04:38:41.859 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo echo test sudo 2024-09-18T04:38:41.859 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:38:41.859 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:38:41.860 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:38:41.860 INFO:tasks.workunit.client.0.smithi115.stderr:++ which echo 2024-09-18T04:38:41.862 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/bin/echo 2024-09-18T04:38:41.862 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:38:41.862 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/bin/echo test sudo 2024-09-18T04:38:41.872 INFO:tasks.workunit.client.0.smithi115.stdout:test sudo 2024-09-18T04:38:41.873 INFO:tasks.workunit.client.0.smithi115.stderr:+ trap cleanup INT TERM EXIT 2024-09-18T04:38:41.874 INFO:tasks.workunit.client.0.smithi115.stderr:++ mktemp -d 2024-09-18T04:38:41.875 INFO:tasks.workunit.client.0.smithi115.stderr:+ TEMPDIR=/tmp/tmp.WJMNJxIdAN 2024-09-18T04:38:41.875 INFO:tasks.workunit.client.0.smithi115.stderr:+ DATA=/tmp/tmp.WJMNJxIdAN/data 2024-09-18T04:38:41.875 INFO:tasks.workunit.client.0.smithi115.stderr:+ dd if=/dev/urandom of=/tmp/tmp.WJMNJxIdAN/data bs=1M count=64 2024-09-18T04:38:41.893 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:41.886+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:41.924 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:41.924 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:41.924 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:38:41.924 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:38:41.924 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:41.936 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:38:41.940 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-09-18T04:38:41.994 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:41.990+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:42.041 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:42.041 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:38:42.041 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:38:42.041 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:38:42.041 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.041 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:30 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'16, mlcod=15'14)", 2024-09-18T04:38:42.041 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.486330+0000", 2024-09-18T04:38:42.041 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 74.552008710999999, 2024-09-18T04:38:42.042 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026383700000000001, 2024-09-18T04:38:42.042 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:42.042 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:42.042 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:42.042 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.042 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:42.042 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486330+0000", 2024-09-18T04:38:42.042 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.042 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.042 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:42.042 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486330+0000", 2024-09-18T04:38:42.042 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.043 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.043 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:42.043 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486328+0000", 2024-09-18T04:38:42.043 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:42.043 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.043 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.043 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:42.043 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486342+0000", 2024-09-18T04:38:42.043 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3783e-05 2024-09-18T04:38:42.043 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.043 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.043 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:42.043 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486373+0000", 2024-09-18T04:38:42.043 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1362999999999998e-05 2024-09-18T04:38:42.044 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.044 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.044 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:42.044 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486387+0000", 2024-09-18T04:38:42.044 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3219e-05 2024-09-18T04:38:42.044 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.044 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.044 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:42.044 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486489+0000", 2024-09-18T04:38:42.044 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010232499999999999 2024-09-18T04:38:42.044 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.044 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.044 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:42.044 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486524+0000", 2024-09-18T04:38:42.045 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4972000000000003e-05 2024-09-18T04:38:42.045 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.045 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.045 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:42.045 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.488902+0000", 2024-09-18T04:38:42.045 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023776240000000001 2024-09-18T04:38:42.045 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.045 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.045 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:42.045 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.488968+0000", 2024-09-18T04:38:42.045 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6761000000000007e-05 2024-09-18T04:38:42.045 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.045 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:42.046 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.046 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.046 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.046 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:35 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'21, mlcod=15'17)", 2024-09-18T04:38:42.046 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.491817+0000", 2024-09-18T04:38:42.046 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 74.546521792999997, 2024-09-18T04:38:42.046 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023524739999999998, 2024-09-18T04:38:42.046 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:42.046 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:42.046 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:42.046 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.046 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:42.046 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491817+0000", 2024-09-18T04:38:42.047 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.047 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.047 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.047 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:42.047 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491817+0000", 2024-09-18T04:38:42.047 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.047 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.047 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.047 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:42.047 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491815+0000", 2024-09-18T04:38:42.047 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:42.047 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.047 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.047 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:42.048 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491890+0000", 2024-09-18T04:38:42.048 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4500999999999997e-05 2024-09-18T04:38:42.048 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.048 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.048 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:42.048 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491916+0000", 2024-09-18T04:38:42.048 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5666000000000001e-05 2024-09-18T04:38:42.048 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.048 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.048 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:42.048 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.491928+0000", 2024-09-18T04:38:42.048 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2585999999999999e-05 2024-09-18T04:38:42.048 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.049 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.049 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:42.049 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.492000+0000", 2024-09-18T04:38:42.049 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2309000000000002e-05 2024-09-18T04:38:42.049 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.049 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.049 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:42.049 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.492027+0000", 2024-09-18T04:38:42.049 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6251e-05 2024-09-18T04:38:42.049 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.049 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.049 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:42.049 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.494114+0000", 2024-09-18T04:38:42.049 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020875260000000001 2024-09-18T04:38:42.050 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.050 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.050 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:42.050 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.494169+0000", 2024-09-18T04:38:42.050 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5145000000000001e-05 2024-09-18T04:38:42.050 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.050 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:42.050 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.050 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.050 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.050 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:57 1.0 e15/14 1:30306672:devicehealth::main.db.0000000000000000:head v 15'35, mlcod=15'33)", 2024-09-18T04:38:42.050 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.529997+0000", 2024-09-18T04:38:42.050 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 74.508341939000005, 2024-09-18T04:38:42.050 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002623869, 2024-09-18T04:38:42.051 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:42.051 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:42.051 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:42.051 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.051 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:42.051 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.529997+0000", 2024-09-18T04:38:42.051 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.051 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.051 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.051 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:42.051 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.529997+0000", 2024-09-18T04:38:42.051 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.051 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.051 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.052 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:42.052 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.529995+0000", 2024-09-18T04:38:42.052 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:42.052 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.052 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.052 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:42.052 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530116+0000", 2024-09-18T04:38:42.052 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000121141 2024-09-18T04:38:42.052 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.052 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.052 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:42.052 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530146+0000", 2024-09-18T04:38:42.052 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0423e-05 2024-09-18T04:38:42.052 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.053 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.053 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:42.053 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530159+0000", 2024-09-18T04:38:42.053 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2802999999999999e-05 2024-09-18T04:38:42.053 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.053 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.053 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:42.053 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530218+0000", 2024-09-18T04:38:42.053 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8461000000000002e-05 2024-09-18T04:38:42.053 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.053 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.053 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:42.053 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530253+0000", 2024-09-18T04:38:42.053 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4796999999999999e-05 2024-09-18T04:38:42.054 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.054 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.054 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:42.054 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.532559+0000", 2024-09-18T04:38:42.054 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002306939 2024-09-18T04:38:42.054 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.054 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.054 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:42.054 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.532621+0000", 2024-09-18T04:38:42.054 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1131999999999997e-05 2024-09-18T04:38:42.054 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.054 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:42.054 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.054 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.055 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.055 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:68 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'42, mlcod=15'39)", 2024-09-18T04:38:42.055 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.546404+0000", 2024-09-18T04:38:42.055 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 74.491934528000002, 2024-09-18T04:38:42.055 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024499769999999999, 2024-09-18T04:38:42.055 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:42.055 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:42.055 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:42.055 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.055 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:42.055 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546404+0000", 2024-09-18T04:38:42.055 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.055 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.055 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.056 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:42.056 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546404+0000", 2024-09-18T04:38:42.056 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.056 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.056 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.056 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:42.056 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546403+0000", 2024-09-18T04:38:42.056 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:42.056 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.056 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.056 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:42.056 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546416+0000", 2024-09-18T04:38:42.056 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.306e-05 2024-09-18T04:38:42.057 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.057 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.057 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:42.057 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546465+0000", 2024-09-18T04:38:42.057 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9441999999999999e-05 2024-09-18T04:38:42.057 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.057 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.057 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:42.057 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546480+0000", 2024-09-18T04:38:42.057 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4919999999999999e-05 2024-09-18T04:38:42.057 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.057 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.057 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:42.057 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546541+0000", 2024-09-18T04:38:42.058 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1085000000000002e-05 2024-09-18T04:38:42.058 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.058 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.058 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:42.058 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546569+0000", 2024-09-18T04:38:42.058 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7803000000000001e-05 2024-09-18T04:38:42.058 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.058 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.058 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:42.058 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.548725+0000", 2024-09-18T04:38:42.058 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021556629999999999 2024-09-18T04:38:42.058 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.058 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.059 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:42.059 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.548854+0000", 2024-09-18T04:38:42.059 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000129524 2024-09-18T04:38:42.059 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.059 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:42.059 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.059 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.059 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.059 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:69 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'43, mlcod=15'39)", 2024-09-18T04:38:42.059 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.547520+0000", 2024-09-18T04:38:42.059 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 74.490818451999999, 2024-09-18T04:38:42.059 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023506820000000002, 2024-09-18T04:38:42.059 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:42.059 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:42.059 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:42.060 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.060 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:42.060 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547520+0000", 2024-09-18T04:38:42.060 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.060 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.060 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.060 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:42.060 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547520+0000", 2024-09-18T04:38:42.060 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.061 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.061 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.061 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:42.061 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547519+0000", 2024-09-18T04:38:42.061 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:42.061 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.061 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.061 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:42.061 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547614+0000", 2024-09-18T04:38:42.061 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5235000000000006e-05 2024-09-18T04:38:42.061 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.061 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.061 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:42.062 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547640+0000", 2024-09-18T04:38:42.062 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5698999999999999e-05 2024-09-18T04:38:42.062 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.062 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.062 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:42.062 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547651+0000", 2024-09-18T04:38:42.062 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.134e-05 2024-09-18T04:38:42.062 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.062 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.062 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:42.062 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547805+0000", 2024-09-18T04:38:42.062 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015426099999999999 2024-09-18T04:38:42.062 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.062 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.063 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:42.063 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.547844+0000", 2024-09-18T04:38:42.063 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8229999999999998e-05 2024-09-18T04:38:42.063 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.063 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.063 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:42.063 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.549810+0000", 2024-09-18T04:38:42.063 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0019660519999999998 2024-09-18T04:38:42.063 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.063 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.063 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:42.063 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.549871+0000", 2024-09-18T04:38:42.063 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1152e-05 2024-09-18T04:38:42.064 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.064 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:42.064 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.064 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.064 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.064 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:83 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'53, mlcod=15'50)", 2024-09-18T04:38:42.064 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.575989+0000", 2024-09-18T04:38:42.064 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 74.462349504000002, 2024-09-18T04:38:42.064 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023991310000000001, 2024-09-18T04:38:42.064 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:42.064 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:42.064 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:42.064 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.065 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:42.065 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.575989+0000", 2024-09-18T04:38:42.065 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.065 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.065 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.065 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:42.065 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.575989+0000", 2024-09-18T04:38:42.065 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.065 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.065 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.065 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:42.065 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.575988+0000", 2024-09-18T04:38:42.065 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:42.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.066 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.066 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:42.066 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576001+0000", 2024-09-18T04:38:42.066 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2751e-05 2024-09-18T04:38:42.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.066 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.066 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:42.066 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576023+0000", 2024-09-18T04:38:42.066 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2453999999999999e-05 2024-09-18T04:38:42.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.066 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.066 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:42.066 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576038+0000", 2024-09-18T04:38:42.066 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5125e-05 2024-09-18T04:38:42.067 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.067 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.067 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:42.067 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576161+0000", 2024-09-18T04:38:42.067 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000122543 2024-09-18T04:38:42.067 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.067 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.067 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:42.067 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.576221+0000", 2024-09-18T04:38:42.067 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9948000000000003e-05 2024-09-18T04:38:42.067 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.067 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.067 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:42.068 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.578277+0000", 2024-09-18T04:38:42.068 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020567559999999999 2024-09-18T04:38:42.068 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.068 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.068 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:42.068 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.578388+0000", 2024-09-18T04:38:42.068 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011089100000000001 2024-09-18T04:38:42.068 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.068 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:42.068 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.068 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.068 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.068 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:86 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'56, mlcod=15'52)", 2024-09-18T04:38:42.068 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.580261+0000", 2024-09-18T04:38:42.069 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 74.458077868000004, 2024-09-18T04:38:42.069 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002418108, 2024-09-18T04:38:42.069 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:42.069 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:42.069 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:42.069 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.069 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:42.069 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580261+0000", 2024-09-18T04:38:42.069 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.069 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.069 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.069 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:42.069 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580261+0000", 2024-09-18T04:38:42.070 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.070 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.070 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.070 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:42.070 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580260+0000", 2024-09-18T04:38:42.070 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:42.070 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.070 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.070 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:42.070 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580269+0000", 2024-09-18T04:38:42.070 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5109999999999992e-06 2024-09-18T04:38:42.070 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.070 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.071 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:42.071 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580288+0000", 2024-09-18T04:38:42.071 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8783999999999998e-05 2024-09-18T04:38:42.071 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.071 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.071 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:42.071 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580296+0000", 2024-09-18T04:38:42.071 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3250000000000008e-06 2024-09-18T04:38:42.071 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.071 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.071 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:42.071 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580351+0000", 2024-09-18T04:38:42.071 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4132000000000003e-05 2024-09-18T04:38:42.071 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.072 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.072 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:42.072 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.580381+0000", 2024-09-18T04:38:42.072 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0263000000000001e-05 2024-09-18T04:38:42.072 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.072 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.072 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:42.072 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.582615+0000", 2024-09-18T04:38:42.072 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0022339109999999999 2024-09-18T04:38:42.072 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.072 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.072 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:42.072 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.582679+0000", 2024-09-18T04:38:42.072 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4182000000000003e-05 2024-09-18T04:38:42.073 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.073 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:42.073 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.073 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.073 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.073 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:105 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'67, mlcod=15'64)", 2024-09-18T04:38:42.073 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.789089+0000", 2024-09-18T04:38:42.073 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 74.249249698, 2024-09-18T04:38:42.073 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024384229999999999, 2024-09-18T04:38:42.073 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:42.073 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:42.073 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:42.073 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.073 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:42.074 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789089+0000", 2024-09-18T04:38:42.074 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.074 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.074 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.074 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:42.074 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789089+0000", 2024-09-18T04:38:42.074 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.074 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.074 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.074 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:42.074 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789088+0000", 2024-09-18T04:38:42.074 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:42.074 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.075 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.075 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:42.075 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789107+0000", 2024-09-18T04:38:42.075 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9017000000000001e-05 2024-09-18T04:38:42.075 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.075 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.075 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:42.075 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789139+0000", 2024-09-18T04:38:42.075 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2261999999999997e-05 2024-09-18T04:38:42.075 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.075 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.075 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:42.075 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789153+0000", 2024-09-18T04:38:42.075 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4025999999999999e-05 2024-09-18T04:38:42.075 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.076 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.076 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:42.076 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789255+0000", 2024-09-18T04:38:42.076 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000101969 2024-09-18T04:38:42.076 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.076 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.076 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:42.076 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789314+0000", 2024-09-18T04:38:42.076 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8867999999999999e-05 2024-09-18T04:38:42.076 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.076 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.076 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:42.076 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.791453+0000", 2024-09-18T04:38:42.077 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002138938 2024-09-18T04:38:42.077 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.077 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.077 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:42.077 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.791527+0000", 2024-09-18T04:38:42.077 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4464999999999999e-05 2024-09-18T04:38:42.077 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.077 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:42.077 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.077 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.077 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.077 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:106 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'68, mlcod=15'64)", 2024-09-18T04:38:42.077 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.790295+0000", 2024-09-18T04:38:42.077 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 74.248043847000005, 2024-09-18T04:38:42.078 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024379390000000001, 2024-09-18T04:38:42.078 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:42.078 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:42.078 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:42.078 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.078 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:42.078 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790295+0000", 2024-09-18T04:38:42.078 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.078 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.078 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.078 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:42.078 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790295+0000", 2024-09-18T04:38:42.078 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.079 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.079 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.079 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:42.079 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790294+0000", 2024-09-18T04:38:42.079 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:42.079 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.079 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.079 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:42.079 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790390+0000", 2024-09-18T04:38:42.079 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6537000000000001e-05 2024-09-18T04:38:42.079 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.079 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.079 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:42.079 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790418+0000", 2024-09-18T04:38:42.080 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8218e-05 2024-09-18T04:38:42.080 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.080 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.080 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:42.080 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790431+0000", 2024-09-18T04:38:42.080 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2571999999999999e-05 2024-09-18T04:38:42.080 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.080 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.080 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:42.080 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790534+0000", 2024-09-18T04:38:42.080 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000103433 2024-09-18T04:38:42.080 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.080 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.080 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:42.081 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.790565+0000", 2024-09-18T04:38:42.081 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0409999999999999e-05 2024-09-18T04:38:42.081 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.081 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.081 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:42.081 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.792651+0000", 2024-09-18T04:38:42.081 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020865250000000001 2024-09-18T04:38:42.081 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.081 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.081 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:42.081 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.792733+0000", 2024-09-18T04:38:42.081 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1477999999999995e-05 2024-09-18T04:38:42.081 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.082 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:42.082 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.082 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.082 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.082 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:109 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'71, mlcod=15'68)", 2024-09-18T04:38:42.082 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.795250+0000", 2024-09-18T04:38:42.082 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 74.243088833000002, 2024-09-18T04:38:42.082 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023990449999999998, 2024-09-18T04:38:42.082 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:42.082 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:42.082 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:42.082 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.082 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:42.083 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795250+0000", 2024-09-18T04:38:42.083 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.083 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.083 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.083 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:42.083 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795250+0000", 2024-09-18T04:38:42.083 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.083 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.083 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.083 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:42.083 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795248+0000", 2024-09-18T04:38:42.083 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:42.083 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.083 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.084 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:42.084 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795405+0000", 2024-09-18T04:38:42.084 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000157007 2024-09-18T04:38:42.084 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.084 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.084 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:42.084 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795422+0000", 2024-09-18T04:38:42.084 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7013000000000001e-05 2024-09-18T04:38:42.084 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.084 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.084 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:42.084 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795432+0000", 2024-09-18T04:38:42.084 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0448999999999999e-05 2024-09-18T04:38:42.084 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.084 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.085 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:42.085 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795534+0000", 2024-09-18T04:38:42.085 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000101195 2024-09-18T04:38:42.085 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.085 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.085 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:42.085 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.795587+0000", 2024-09-18T04:38:42.085 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3356000000000002e-05 2024-09-18T04:38:42.085 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.085 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.085 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:42.085 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.797560+0000", 2024-09-18T04:38:42.085 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001973023 2024-09-18T04:38:42.086 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.086 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.086 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:42.086 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.797649+0000", 2024-09-18T04:38:42.086 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8864999999999997e-05 2024-09-18T04:38:42.086 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.086 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:42.086 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.086 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.086 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.086 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:115 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'77, mlcod=15'72)", 2024-09-18T04:38:42.086 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.803339+0000", 2024-09-18T04:38:42.086 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 74.235000115000005, 2024-09-18T04:38:42.086 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002385093, 2024-09-18T04:38:42.087 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:42.087 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:42.087 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:42.087 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.087 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:42.087 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803339+0000", 2024-09-18T04:38:42.087 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.087 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.087 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:42.087 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803339+0000", 2024-09-18T04:38:42.087 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.087 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.087 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:42.088 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803337+0000", 2024-09-18T04:38:42.088 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:42.088 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.088 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.088 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:42.088 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803475+0000", 2024-09-18T04:38:42.088 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013718199999999999 2024-09-18T04:38:42.088 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.088 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.088 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:42.088 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803507+0000", 2024-09-18T04:38:42.088 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2351000000000002e-05 2024-09-18T04:38:42.088 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.089 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.089 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:42.089 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803533+0000", 2024-09-18T04:38:42.089 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6010999999999999e-05 2024-09-18T04:38:42.089 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.089 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.089 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:42.089 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803609+0000", 2024-09-18T04:38:42.089 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.64e-05 2024-09-18T04:38:42.089 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.089 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.089 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:42.089 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.803671+0000", 2024-09-18T04:38:42.089 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1467e-05 2024-09-18T04:38:42.089 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.090 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.090 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:42.090 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.805676+0000", 2024-09-18T04:38:42.090 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020047849999999998 2024-09-18T04:38:42.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.090 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.090 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:42.090 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.805724+0000", 2024-09-18T04:38:42.090 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8151e-05 2024-09-18T04:38:42.090 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.090 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:42.090 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.091 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.091 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:118 1.0 e15/14 1:30306672:devicehealth::main.db.0000000000000000:head v 15'80, mlcod=15'78)", 2024-09-18T04:38:42.091 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.807930+0000", 2024-09-18T04:38:42.091 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 74.230408612000005, 2024-09-18T04:38:42.091 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028002259999999998, 2024-09-18T04:38:42.091 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:42.091 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:42.091 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:42.091 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.091 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:42.091 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.807930+0000", 2024-09-18T04:38:42.091 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.092 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.092 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:42.092 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.807930+0000", 2024-09-18T04:38:42.092 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.092 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.092 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.092 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:42.092 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.807929+0000", 2024-09-18T04:38:42.092 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:42.092 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.092 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.092 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:42.092 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808024+0000", 2024-09-18T04:38:42.092 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5258000000000002e-05 2024-09-18T04:38:42.093 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.093 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.093 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:42.093 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808044+0000", 2024-09-18T04:38:42.093 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0203e-05 2024-09-18T04:38:42.093 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.093 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.093 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:42.093 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808053+0000", 2024-09-18T04:38:42.093 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2689999999999999e-06 2024-09-18T04:38:42.093 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.093 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.093 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:42.094 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808137+0000", 2024-09-18T04:38:42.094 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4441e-05 2024-09-18T04:38:42.094 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.094 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.094 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:42.094 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808172+0000", 2024-09-18T04:38:42.094 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4453999999999999e-05 2024-09-18T04:38:42.094 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.094 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.094 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:42.094 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.810663+0000", 2024-09-18T04:38:42.094 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024910269999999998 2024-09-18T04:38:42.094 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.094 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.094 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:42.095 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.810730+0000", 2024-09-18T04:38:42.095 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7658999999999997e-05 2024-09-18T04:38:42.095 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.095 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:42.095 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.095 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.095 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.095 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:129 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'87, mlcod=15'85)", 2024-09-18T04:38:42.095 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.231689+0000", 2024-09-18T04:38:42.095 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 73.806649598999996, 2024-09-18T04:38:42.096 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002376502, 2024-09-18T04:38:42.096 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:42.096 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:42.096 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:42.096 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.096 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:42.096 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231689+0000", 2024-09-18T04:38:42.096 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.096 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.096 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.096 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:42.096 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231689+0000", 2024-09-18T04:38:42.096 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.097 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.097 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.097 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:42.097 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231687+0000", 2024-09-18T04:38:42.097 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:42.097 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.097 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.097 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:42.097 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231702+0000", 2024-09-18T04:38:42.097 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4949999999999999e-05 2024-09-18T04:38:42.097 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.097 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.097 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:42.097 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231735+0000", 2024-09-18T04:38:42.098 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2276000000000001e-05 2024-09-18T04:38:42.098 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.098 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.098 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:42.098 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231750+0000", 2024-09-18T04:38:42.098 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5193e-05 2024-09-18T04:38:42.098 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.098 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.098 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:42.098 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231872+0000", 2024-09-18T04:38:42.098 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000122127 2024-09-18T04:38:42.098 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.098 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.099 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:42.099 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.231936+0000", 2024-09-18T04:38:42.099 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4089000000000003e-05 2024-09-18T04:38:42.099 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.099 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.099 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:42.099 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234002+0000", 2024-09-18T04:38:42.099 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002065575 2024-09-18T04:38:42.099 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.099 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.099 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:42.099 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234066+0000", 2024-09-18T04:38:42.099 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3944999999999998e-05 2024-09-18T04:38:42.099 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.100 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:42.100 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.100 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.100 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.100 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:130 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'88, mlcod=15'85)", 2024-09-18T04:38:42.100 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.232853+0000", 2024-09-18T04:38:42.100 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 73.805485984000001, 2024-09-18T04:38:42.100 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027351739999999999, 2024-09-18T04:38:42.100 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:42.100 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:42.100 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:42.100 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.100 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:42.101 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232853+0000", 2024-09-18T04:38:42.101 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.101 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.101 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.101 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:42.101 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232853+0000", 2024-09-18T04:38:42.101 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.101 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.101 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.101 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:42.101 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232851+0000", 2024-09-18T04:38:42.101 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:42.101 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.102 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.102 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:42.102 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232866+0000", 2024-09-18T04:38:42.102 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4946000000000001e-05 2024-09-18T04:38:42.102 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.102 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.102 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:42.102 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232897+0000", 2024-09-18T04:38:42.102 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0982999999999998e-05 2024-09-18T04:38:42.102 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.102 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.102 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:42.102 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232915+0000", 2024-09-18T04:38:42.102 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8476000000000001e-05 2024-09-18T04:38:42.103 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.103 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.103 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:42.103 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.233029+0000", 2024-09-18T04:38:42.103 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011305400000000001 2024-09-18T04:38:42.103 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.103 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.103 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:42.103 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.233079+0000", 2024-09-18T04:38:42.103 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0668999999999999e-05 2024-09-18T04:38:42.103 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.103 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.103 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:42.104 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235534+0000", 2024-09-18T04:38:42.104 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024546030000000001 2024-09-18T04:38:42.104 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.104 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.104 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:42.104 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235588+0000", 2024-09-18T04:38:42.104 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4098999999999998e-05 2024-09-18T04:38:42.104 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.104 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:42.104 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.104 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.104 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.104 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:131 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'89, mlcod=15'85)", 2024-09-18T04:38:42.105 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.234089+0000", 2024-09-18T04:38:42.105 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 73.80425013, 2024-09-18T04:38:42.105 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026975609999999998, 2024-09-18T04:38:42.105 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:42.105 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:42.105 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:42.105 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.105 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:42.105 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234089+0000", 2024-09-18T04:38:42.105 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.105 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.105 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.105 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:42.105 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234089+0000", 2024-09-18T04:38:42.106 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.106 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.106 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.106 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:42.106 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234087+0000", 2024-09-18T04:38:42.106 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:42.106 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.106 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.106 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:42.106 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234145+0000", 2024-09-18T04:38:42.106 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8687999999999998e-05 2024-09-18T04:38:42.106 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.106 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.106 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:42.107 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234177+0000", 2024-09-18T04:38:42.107 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1526999999999998e-05 2024-09-18T04:38:42.107 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.107 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.107 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:42.107 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234190+0000", 2024-09-18T04:38:42.107 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2965999999999999e-05 2024-09-18T04:38:42.107 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.107 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.107 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:42.107 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234309+0000", 2024-09-18T04:38:42.107 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011875399999999999 2024-09-18T04:38:42.107 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.107 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.108 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:42.108 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234365+0000", 2024-09-18T04:38:42.108 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6436e-05 2024-09-18T04:38:42.108 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.108 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.108 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:42.108 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236675+0000", 2024-09-18T04:38:42.108 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023099520000000001 2024-09-18T04:38:42.108 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.108 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.108 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:42.108 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236786+0000", 2024-09-18T04:38:42.108 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000111025 2024-09-18T04:38:42.108 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.109 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:42.109 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.109 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.109 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.109 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:132 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'90, mlcod=15'85)", 2024-09-18T04:38:42.109 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.235300+0000", 2024-09-18T04:38:42.109 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 73.80303868, 2024-09-18T04:38:42.109 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025982599999999998, 2024-09-18T04:38:42.109 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:42.109 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:42.109 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:42.109 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.109 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:42.109 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235300+0000", 2024-09-18T04:38:42.110 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.110 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.110 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.110 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:42.110 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235300+0000", 2024-09-18T04:38:42.110 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.110 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.110 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.110 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:42.110 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235298+0000", 2024-09-18T04:38:42.110 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:42.110 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.110 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.110 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:42.110 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235318+0000", 2024-09-18T04:38:42.111 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9714000000000001e-05 2024-09-18T04:38:42.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.111 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.111 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:42.111 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235337+0000", 2024-09-18T04:38:42.111 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8892000000000001e-05 2024-09-18T04:38:42.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.111 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.111 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:42.111 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235348+0000", 2024-09-18T04:38:42.111 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.129e-05 2024-09-18T04:38:42.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.111 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.111 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:42.112 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235681+0000", 2024-09-18T04:38:42.112 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00033323000000000001 2024-09-18T04:38:42.112 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.112 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.112 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:42.112 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235733+0000", 2024-09-18T04:38:42.112 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1391999999999999e-05 2024-09-18T04:38:42.112 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.112 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.112 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:42.112 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237839+0000", 2024-09-18T04:38:42.112 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002105647 2024-09-18T04:38:42.112 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.113 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.113 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:42.113 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237898+0000", 2024-09-18T04:38:42.113 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9759000000000002e-05 2024-09-18T04:38:42.113 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.113 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:42.113 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.113 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.113 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.113 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:133 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'91, mlcod=15'86)", 2024-09-18T04:38:42.113 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.236920+0000", 2024-09-18T04:38:42.113 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 73.801418792000007, 2024-09-18T04:38:42.113 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024318410000000001, 2024-09-18T04:38:42.113 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:42.114 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:42.114 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:42.114 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.114 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:42.114 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236920+0000", 2024-09-18T04:38:42.114 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.114 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.114 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.114 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:42.114 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236920+0000", 2024-09-18T04:38:42.114 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.114 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.114 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.114 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:42.115 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236918+0000", 2024-09-18T04:38:42.115 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:42.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.115 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.115 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:42.115 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236931+0000", 2024-09-18T04:38:42.115 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2683999999999999e-05 2024-09-18T04:38:42.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.115 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.115 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:42.115 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236951+0000", 2024-09-18T04:38:42.115 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0292000000000002e-05 2024-09-18T04:38:42.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.115 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.116 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:42.116 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236962+0000", 2024-09-18T04:38:42.116 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0703000000000001e-05 2024-09-18T04:38:42.116 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.116 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.116 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:42.116 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237095+0000", 2024-09-18T04:38:42.116 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013312400000000001 2024-09-18T04:38:42.116 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.116 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.116 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:42.116 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237133+0000", 2024-09-18T04:38:42.116 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7496000000000001e-05 2024-09-18T04:38:42.117 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.117 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.117 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:42.117 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.239206+0000", 2024-09-18T04:38:42.117 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020730290000000001 2024-09-18T04:38:42.117 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.117 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.117 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:42.117 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.239352+0000", 2024-09-18T04:38:42.117 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014593000000000001 2024-09-18T04:38:42.117 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.117 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:42.117 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.118 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.118 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.118 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:134 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'92, mlcod=15'86)", 2024-09-18T04:38:42.118 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.237894+0000", 2024-09-18T04:38:42.118 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 73.800444769999999, 2024-09-18T04:38:42.118 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025299519999999998, 2024-09-18T04:38:42.118 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:42.118 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:42.118 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:42.118 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.118 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:42.118 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237894+0000", 2024-09-18T04:38:42.118 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.118 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.118 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.119 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:42.119 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237894+0000", 2024-09-18T04:38:42.119 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.119 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.119 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:42.119 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237892+0000", 2024-09-18T04:38:42.119 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:42.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.119 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.119 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:42.119 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237986+0000", 2024-09-18T04:38:42.119 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3341e-05 2024-09-18T04:38:42.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.120 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.120 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:42.120 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.238012+0000", 2024-09-18T04:38:42.120 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5843e-05 2024-09-18T04:38:42.120 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.120 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.120 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:42.120 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.238027+0000", 2024-09-18T04:38:42.120 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5469e-05 2024-09-18T04:38:42.120 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.120 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.120 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:42.120 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.238114+0000", 2024-09-18T04:38:42.120 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7483000000000002e-05 2024-09-18T04:38:42.121 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.121 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.121 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:42.121 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.238152+0000", 2024-09-18T04:38:42.121 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7565999999999999e-05 2024-09-18T04:38:42.121 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.121 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.121 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:42.121 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.240297+0000", 2024-09-18T04:38:42.121 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021447620000000001 2024-09-18T04:38:42.121 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.121 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.121 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:42.121 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.240424+0000", 2024-09-18T04:38:42.122 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012705400000000001 2024-09-18T04:38:42.122 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.122 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:42.122 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.122 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.122 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.122 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:139 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'97, mlcod=15'93)", 2024-09-18T04:38:42.122 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.245692+0000", 2024-09-18T04:38:42.122 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 73.792646629000004, 2024-09-18T04:38:42.122 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024260480000000001, 2024-09-18T04:38:42.122 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:42.122 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:42.122 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:42.122 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.123 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:42.123 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245692+0000", 2024-09-18T04:38:42.123 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.123 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.123 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.123 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:42.123 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245692+0000", 2024-09-18T04:38:42.123 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.123 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.123 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.123 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:42.123 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245690+0000", 2024-09-18T04:38:42.123 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:42.123 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.123 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.124 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:42.124 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245712+0000", 2024-09-18T04:38:42.124 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1916000000000002e-05 2024-09-18T04:38:42.124 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.124 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.124 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:42.124 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245741+0000", 2024-09-18T04:38:42.124 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8503000000000001e-05 2024-09-18T04:38:42.124 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.124 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.124 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:42.124 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245759+0000", 2024-09-18T04:38:42.124 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.872e-05 2024-09-18T04:38:42.125 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.125 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.125 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:42.125 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245859+0000", 2024-09-18T04:38:42.125 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9641000000000004e-05 2024-09-18T04:38:42.125 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.125 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.125 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:42.125 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.245900+0000", 2024-09-18T04:38:42.125 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0589000000000001e-05 2024-09-18T04:38:42.125 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.125 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.125 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:42.125 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.247961+0000", 2024-09-18T04:38:42.125 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020619190000000002 2024-09-18T04:38:42.126 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.126 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.126 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:42.126 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.248118+0000", 2024-09-18T04:38:42.126 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015664300000000001 2024-09-18T04:38:42.126 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.126 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:42.126 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.126 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.126 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.126 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:144 1.0 e15/14 1:30306672:devicehealth::main.db.0000000000000000:head v 15'102, mlcod=15'98)", 2024-09-18T04:38:42.126 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.252149+0000", 2024-09-18T04:38:42.126 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 73.786189562999994, 2024-09-18T04:38:42.126 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024174959999999999, 2024-09-18T04:38:42.127 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:42.127 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:42.127 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:42.127 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.127 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:42.127 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252149+0000", 2024-09-18T04:38:42.127 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.127 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.127 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.127 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:42.127 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252149+0000", 2024-09-18T04:38:42.127 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.127 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.127 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.128 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:42.128 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252148+0000", 2024-09-18T04:38:42.128 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:42.128 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.128 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.128 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:42.128 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252225+0000", 2024-09-18T04:38:42.128 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6576999999999995e-05 2024-09-18T04:38:42.128 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.128 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.128 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:42.128 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252242+0000", 2024-09-18T04:38:42.128 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7740999999999998e-05 2024-09-18T04:38:42.128 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.129 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.129 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:42.129 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252254+0000", 2024-09-18T04:38:42.129 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.137e-05 2024-09-18T04:38:42.129 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.129 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.129 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:42.129 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252385+0000", 2024-09-18T04:38:42.129 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013091099999999999 2024-09-18T04:38:42.129 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.129 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.129 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:42.129 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.252426+0000", 2024-09-18T04:38:42.129 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1535999999999997e-05 2024-09-18T04:38:42.129 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.130 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.130 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:42.130 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.254498+0000", 2024-09-18T04:38:42.130 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002071477 2024-09-18T04:38:42.130 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.130 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.130 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:42.130 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.254567+0000", 2024-09-18T04:38:42.130 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9014e-05 2024-09-18T04:38:42.130 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.130 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:42.130 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.130 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.130 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:42.131 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:42.132 INFO:tasks.workunit.client.0.smithi115.stderr:64+0 records in 2024-09-18T04:38:42.132 INFO:tasks.workunit.client.0.smithi115.stderr:64+0 records out 2024-09-18T04:38:42.132 INFO:tasks.workunit.client.0.smithi115.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.190795 s, 352 MB/s 2024-09-18T04:38:42.132 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd namespace create rbd/ns 2024-09-18T04:38:42.133 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:38:42.133 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:42.126+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:42.139 INFO:tasks.workunit.client.0.smithi115.stderr:+ for ns in '' ${NS} 2024-09-18T04:38:42.139 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd --dest-pool rbd --dest-namespace '' --no-progress import /tmp/tmp.WJMNJxIdAN/data testrbdnbd17342 2024-09-18T04:38:42.232 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:42.226+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:42.246 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:42.246 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:42.247 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:38:42.247 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:42.257 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:38:42.338 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:42.330+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:42.397 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:42.398 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:42.398 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:38:42.398 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:38:42.398 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:42.404 INFO:tasks.workunit.client.0.smithi115.stderr:+ for ns in '' ${NS} 2024-09-18T04:38:42.404 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd --dest-pool rbd --dest-namespace ns --no-progress import /tmp/tmp.WJMNJxIdAN/data testrbdnbd17342 2024-09-18T04:38:42.407 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:38:42.434 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:42.430+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:42.515 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:42.515 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:38:42.515 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:38:42.515 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:38:42.515 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.515 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4214.0:5 2.2 e19/17 2:5cea7035:::rbd_info:head v 19'1, mlcod=0'0)", 2024-09-18T04:38:42.515 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:35.471260+0000", 2024-09-18T04:38:42.515 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 67.038945962, 2024-09-18T04:38:42.515 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021025950000000001, 2024-09-18T04:38:42.515 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:42.515 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:42.516 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:42.516 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.516 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:42.516 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471260+0000", 2024-09-18T04:38:42.516 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.516 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.516 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.516 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:42.516 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471260+0000", 2024-09-18T04:38:42.516 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.516 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.516 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.517 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:42.517 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471259+0000", 2024-09-18T04:38:42.517 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:42.517 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.517 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.517 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:42.517 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471275+0000", 2024-09-18T04:38:42.517 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6135999999999999e-05 2024-09-18T04:38:42.517 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.517 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.517 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:42.517 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471298+0000", 2024-09-18T04:38:42.518 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3632999999999999e-05 2024-09-18T04:38:42.518 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.518 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.518 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:42.518 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471320+0000", 2024-09-18T04:38:42.518 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1852999999999999e-05 2024-09-18T04:38:42.518 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.518 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.518 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:42.518 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471445+0000", 2024-09-18T04:38:42.518 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012461699999999999 2024-09-18T04:38:42.518 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.518 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.519 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:42.519 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.471529+0000", 2024-09-18T04:38:42.519 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3875000000000006e-05 2024-09-18T04:38:42.519 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.519 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.519 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:42.519 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.473317+0000", 2024-09-18T04:38:42.519 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017886899999999999 2024-09-18T04:38:42.519 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.519 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.519 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:42.519 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:35.473362+0000", 2024-09-18T04:38:42.519 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4724000000000001e-05 2024-09-18T04:38:42.519 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.520 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:42.520 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.520 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.520 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.520 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4697.0:1 2.4 2:377edd40:::rbd_namespace:head [call rbd.namespace_add in=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:42.520 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.120439+0000", 2024-09-18T04:38:42.520 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.38976650800000001, 2024-09-18T04:38:42.520 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0063421789999999999, 2024-09-18T04:38:42.520 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:42.520 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:42.520 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:42.520 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4697", 2024-09-18T04:38:42.520 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/1536363535", 2024-09-18T04:38:42.521 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 1 2024-09-18T04:38:42.521 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.521 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:42.521 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.521 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:42.521 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.120439+0000", 2024-09-18T04:38:42.521 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.521 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.521 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.521 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:42.521 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.120439+0000", 2024-09-18T04:38:42.521 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.521 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.522 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.522 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:42.522 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.120437+0000", 2024-09-18T04:38:42.522 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:42.522 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.522 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.522 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:42.522 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.120451+0000", 2024-09-18T04:38:42.522 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4630000000000001e-05 2024-09-18T04:38:42.522 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.522 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.522 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:42.522 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.120476+0000", 2024-09-18T04:38:42.523 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4522e-05 2024-09-18T04:38:42.523 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.523 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.523 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:42.523 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.120499+0000", 2024-09-18T04:38:42.523 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2606e-05 2024-09-18T04:38:42.523 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.523 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.523 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:42.523 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.120651+0000", 2024-09-18T04:38:42.523 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015254900000000001 2024-09-18T04:38:42.523 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.523 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.524 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-09-18T04:38:42.524 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.120668+0000", 2024-09-18T04:38:42.524 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6586999999999999e-05 2024-09-18T04:38:42.524 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.524 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.524 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:42.524 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.124016+0000", 2024-09-18T04:38:42.524 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0033482970000000001 2024-09-18T04:38:42.524 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.524 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.524 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:42.524 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.124235+0000", 2024-09-18T04:38:42.524 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00021938 2024-09-18T04:38:42.524 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.525 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.525 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:42.525 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.124512+0000", 2024-09-18T04:38:42.525 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00027653900000000002 2024-09-18T04:38:42.525 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.525 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.525 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:42.525 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.125695+0000", 2024-09-18T04:38:42.525 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001182633 2024-09-18T04:38:42.525 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.525 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.525 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:42.525 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.126640+0000", 2024-09-18T04:38:42.526 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00094525299999999996 2024-09-18T04:38:42.526 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.526 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.526 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:42.526 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.126726+0000", 2024-09-18T04:38:42.526 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.6559999999999998e-05 2024-09-18T04:38:42.526 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.526 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.526 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:42.526 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.126781+0000", 2024-09-18T04:38:42.526 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4809000000000003e-05 2024-09-18T04:38:42.526 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.526 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:42.526 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.527 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.527 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.527 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4697.0:3 2.6 2:774f7988:ns::rbd_directory:head [call rbd.dir_state_set in=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:42.527 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.131398+0000", 2024-09-18T04:38:42.527 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.37880724500000001, 2024-09-18T04:38:42.527 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027686239999999999, 2024-09-18T04:38:42.527 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:42.527 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:42.527 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:42.527 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4697", 2024-09-18T04:38:42.527 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/1536363535", 2024-09-18T04:38:42.527 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 3 2024-09-18T04:38:42.527 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.527 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:42.527 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.528 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:42.528 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.131398+0000", 2024-09-18T04:38:42.528 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.528 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.528 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.528 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:42.528 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.131398+0000", 2024-09-18T04:38:42.528 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.528 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.528 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.528 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:42.528 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.131396+0000", 2024-09-18T04:38:42.528 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:42.529 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.529 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.529 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:42.529 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.131424+0000", 2024-09-18T04:38:42.529 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7645999999999999e-05 2024-09-18T04:38:42.529 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.529 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.529 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:42.529 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.131455+0000", 2024-09-18T04:38:42.529 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.116e-05 2024-09-18T04:38:42.529 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.529 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.529 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:42.530 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.131470+0000", 2024-09-18T04:38:42.530 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5255e-05 2024-09-18T04:38:42.530 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.530 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.530 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:42.530 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.131570+0000", 2024-09-18T04:38:42.530 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9680999999999996e-05 2024-09-18T04:38:42.530 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.530 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.530 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:42.530 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.131803+0000", 2024-09-18T04:38:42.530 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000232797 2024-09-18T04:38:42.530 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.530 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.531 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:38:42.531 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.132073+0000", 2024-09-18T04:38:42.531 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00027003900000000002 2024-09-18T04:38:42.531 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.531 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.531 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:42.531 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.133258+0000", 2024-09-18T04:38:42.531 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0011854389999999999 2024-09-18T04:38:42.531 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.531 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.531 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:42.531 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.134013+0000", 2024-09-18T04:38:42.531 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00075469800000000004 2024-09-18T04:38:42.531 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.532 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.532 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:42.532 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.134106+0000", 2024-09-18T04:38:42.532 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.2508000000000003e-05 2024-09-18T04:38:42.532 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.532 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.532 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:42.532 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.134167+0000", 2024-09-18T04:38:42.532 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1381000000000001e-05 2024-09-18T04:38:42.532 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.532 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:42.532 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.532 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.533 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.533 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:5 2.4 2:3831950c:::rbd_directory:head [call rbd.dir_add_image in=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:42.533 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.196747+0000", 2024-09-18T04:38:42.533 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.31345814599999999, 2024-09-18T04:38:42.533 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0032450959999999998, 2024-09-18T04:38:42.533 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:42.533 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:42.533 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:42.533 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:38:42.533 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:38:42.533 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 5 2024-09-18T04:38:42.533 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.533 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:42.534 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.534 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:42.534 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.196747+0000", 2024-09-18T04:38:42.534 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.534 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.534 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.534 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:42.534 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.196747+0000", 2024-09-18T04:38:42.534 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.534 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.534 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.534 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:42.534 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.196745+0000", 2024-09-18T04:38:42.534 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:42.535 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.535 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.535 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:42.535 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.196762+0000", 2024-09-18T04:38:42.535 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6976000000000001e-05 2024-09-18T04:38:42.535 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.535 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.535 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:42.535 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.196800+0000", 2024-09-18T04:38:42.535 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7601999999999997e-05 2024-09-18T04:38:42.535 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.535 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.535 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:42.535 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.196844+0000", 2024-09-18T04:38:42.536 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4749000000000001e-05 2024-09-18T04:38:42.536 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.536 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.536 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:42.536 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.196972+0000", 2024-09-18T04:38:42.536 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012753800000000001 2024-09-18T04:38:42.536 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.536 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.536 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:42.536 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.197344+0000", 2024-09-18T04:38:42.536 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000372166 2024-09-18T04:38:42.536 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.536 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.537 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:42.537 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.197935+0000", 2024-09-18T04:38:42.537 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00059046599999999999 2024-09-18T04:38:42.537 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.537 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.537 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:42.537 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.199658+0000", 2024-09-18T04:38:42.537 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001723348 2024-09-18T04:38:42.537 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.537 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.537 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:42.537 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.199855+0000", 2024-09-18T04:38:42.537 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000197586 2024-09-18T04:38:42.538 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.538 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.538 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:42.538 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.199938+0000", 2024-09-18T04:38:42.538 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2362999999999997e-05 2024-09-18T04:38:42.538 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.538 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.538 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:42.538 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.199992+0000", 2024-09-18T04:38:42.538 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4679000000000003e-05 2024-09-18T04:38:42.538 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.538 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:42.538 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.539 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.539 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.539 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:29 2.0 e51/17 2:08ed9131:::rbd_object_map.125fc09df6d1:head v 51'6, mlcod=51'2)", 2024-09-18T04:38:42.539 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.259394+0000", 2024-09-18T04:38:42.539 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.25081183699999998, 2024-09-18T04:38:42.539 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026720540000000001, 2024-09-18T04:38:42.539 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:42.539 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:42.539 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:42.539 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.539 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:42.539 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.259394+0000", 2024-09-18T04:38:42.539 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.539 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.540 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.540 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:42.540 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.259394+0000", 2024-09-18T04:38:42.540 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.540 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.540 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.540 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:42.540 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.259392+0000", 2024-09-18T04:38:42.540 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:42.540 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.540 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.540 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:42.540 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.259404+0000", 2024-09-18T04:38:42.540 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2136e-05 2024-09-18T04:38:42.541 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.541 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.541 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:42.541 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.259429+0000", 2024-09-18T04:38:42.541 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4573e-05 2024-09-18T04:38:42.541 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.541 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.541 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:42.541 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.259441+0000", 2024-09-18T04:38:42.541 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1813e-05 2024-09-18T04:38:42.541 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.541 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.541 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:42.541 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.259551+0000", 2024-09-18T04:38:42.542 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000110567 2024-09-18T04:38:42.542 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.542 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.542 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:42.542 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.259611+0000", 2024-09-18T04:38:42.542 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9531000000000003e-05 2024-09-18T04:38:42.542 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.542 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.542 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:42.542 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.261787+0000", 2024-09-18T04:38:42.542 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021765880000000001 2024-09-18T04:38:42.542 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.542 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.542 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:42.543 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.262066+0000", 2024-09-18T04:38:42.543 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00027839199999999997 2024-09-18T04:38:42.543 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.543 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:42.543 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.543 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.543 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.543 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:32 2.0 e51/17 2:08ed9131:::rbd_object_map.125fc09df6d1:head v 51'8, mlcod=51'4)", 2024-09-18T04:38:42.543 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.262967+0000", 2024-09-18T04:38:42.543 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.247238492, 2024-09-18T04:38:42.543 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027341069999999999, 2024-09-18T04:38:42.543 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:42.543 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:42.543 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:42.544 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.544 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:42.544 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.262967+0000", 2024-09-18T04:38:42.544 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.544 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.544 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.544 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:42.544 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.262967+0000", 2024-09-18T04:38:42.544 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.544 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.544 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.544 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:42.544 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.262966+0000", 2024-09-18T04:38:42.545 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:42.545 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.545 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.545 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:42.545 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.262981+0000", 2024-09-18T04:38:42.545 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5206e-05 2024-09-18T04:38:42.545 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.545 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.545 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:42.545 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.263013+0000", 2024-09-18T04:38:42.545 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1760000000000001e-05 2024-09-18T04:38:42.545 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.545 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.546 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:42.546 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.263051+0000", 2024-09-18T04:38:42.546 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8222000000000001e-05 2024-09-18T04:38:42.546 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.546 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.546 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:42.546 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.263148+0000", 2024-09-18T04:38:42.546 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6813999999999998e-05 2024-09-18T04:38:42.546 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.546 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.546 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:42.546 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.263207+0000", 2024-09-18T04:38:42.546 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9067999999999998e-05 2024-09-18T04:38:42.547 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.547 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.547 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:42.547 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.265655+0000", 2024-09-18T04:38:42.547 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024478270000000001 2024-09-18T04:38:42.547 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.547 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.547 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:42.547 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.265701+0000", 2024-09-18T04:38:42.547 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6629999999999999e-05 2024-09-18T04:38:42.547 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.547 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:42.547 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.547 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.548 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.548 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:35 2.0 e51/17 2:08ed9131:::rbd_object_map.125fc09df6d1:head v 51'9, mlcod=51'4)", 2024-09-18T04:38:42.548 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.264540+0000", 2024-09-18T04:38:42.548 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.24566573999999999, 2024-09-18T04:38:42.548 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00186728, 2024-09-18T04:38:42.548 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:42.548 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:42.548 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:42.548 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.548 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:42.548 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.264540+0000", 2024-09-18T04:38:42.548 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.548 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.549 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.549 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:42.549 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.264540+0000", 2024-09-18T04:38:42.549 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.549 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.549 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.549 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:42.549 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.264538+0000", 2024-09-18T04:38:42.549 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:42.549 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.549 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.549 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:42.549 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.264554+0000", 2024-09-18T04:38:42.549 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6388999999999998e-05 2024-09-18T04:38:42.550 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.550 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.550 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:42.550 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.264602+0000", 2024-09-18T04:38:42.550 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8025999999999997e-05 2024-09-18T04:38:42.550 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.550 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.550 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:42.550 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.264615+0000", 2024-09-18T04:38:42.550 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.294e-05 2024-09-18T04:38:42.550 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.550 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.550 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:42.550 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.264702+0000", 2024-09-18T04:38:42.551 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.6572999999999999e-05 2024-09-18T04:38:42.551 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.551 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.551 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:42.551 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.264756+0000", 2024-09-18T04:38:42.551 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4159000000000001e-05 2024-09-18T04:38:42.551 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.551 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.551 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:42.551 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.266358+0000", 2024-09-18T04:38:42.551 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0016021410000000001 2024-09-18T04:38:42.551 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.551 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.551 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:42.552 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.266407+0000", 2024-09-18T04:38:42.552 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8779000000000002e-05 2024-09-18T04:38:42.552 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.552 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:42.552 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.552 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.552 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.552 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:33 2.1 e51/24 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head v 51'4, mlcod=51'2)", 2024-09-18T04:38:42.552 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:38:42.552 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.231007142, 2024-09-18T04:38:42.552 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018702106, 2024-09-18T04:38:42.552 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:42.552 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:42.553 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:42.553 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.553 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:42.553 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:38:42.553 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.553 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.553 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.553 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:42.553 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:38:42.553 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.553 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.553 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.553 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:42.553 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279197+0000", 2024-09-18T04:38:42.554 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:42.554 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.554 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.554 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:42.554 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282682+0000", 2024-09-18T04:38:42.554 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0034848689999999998 2024-09-18T04:38:42.554 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.554 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.554 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:42.554 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282709+0000", 2024-09-18T04:38:42.554 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7599000000000001e-05 2024-09-18T04:38:42.554 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.554 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.554 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:42.555 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282720+0000", 2024-09-18T04:38:42.555 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.092e-05 2024-09-18T04:38:42.555 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.555 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.555 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:42.555 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282791+0000", 2024-09-18T04:38:42.555 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1268999999999998e-05 2024-09-18T04:38:42.555 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.555 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.555 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:42.555 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282843+0000", 2024-09-18T04:38:42.555 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1771999999999998e-05 2024-09-18T04:38:42.555 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.555 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.556 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:42.556 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.297830+0000", 2024-09-18T04:38:42.556 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014987228999999999 2024-09-18T04:38:42.556 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.556 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.556 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:42.556 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.297901+0000", 2024-09-18T04:38:42.556 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0209999999999994e-05 2024-09-18T04:38:42.556 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.556 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:42.556 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.556 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.556 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.556 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:38 2.0 e51/17 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head v 51'10, mlcod=51'8)", 2024-09-18T04:38:42.557 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:38:42.557 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.229588712, 2024-09-18T04:38:42.557 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.023984472, 2024-09-18T04:38:42.557 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:42.557 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:42.557 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:42.557 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.557 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:42.557 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:38:42.557 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.557 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.557 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.557 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:42.558 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:38:42.558 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.558 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.558 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.558 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:42.558 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280614+0000", 2024-09-18T04:38:42.558 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:42.558 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.558 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.558 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:42.558 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293094+0000", 2024-09-18T04:38:42.558 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012479571 2024-09-18T04:38:42.560 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.560 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.560 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:42.560 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293111+0000", 2024-09-18T04:38:42.560 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7354999999999999e-05 2024-09-18T04:38:42.560 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.560 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.560 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:42.560 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293121+0000", 2024-09-18T04:38:42.560 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6979999999999993e-06 2024-09-18T04:38:42.560 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.560 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.561 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:42.561 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293180+0000", 2024-09-18T04:38:42.561 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9333000000000003e-05 2024-09-18T04:38:42.561 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.561 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.561 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:42.561 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293203+0000", 2024-09-18T04:38:42.561 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2633000000000002e-05 2024-09-18T04:38:42.561 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.561 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.561 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:42.561 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304534+0000", 2024-09-18T04:38:42.561 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011330761 2024-09-18T04:38:42.561 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.562 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.562 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:42.562 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304601+0000", 2024-09-18T04:38:42.562 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7657000000000006e-05 2024-09-18T04:38:42.562 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.562 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:42.562 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.562 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.562 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.563 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:40 2.0 e51/17 2:08ed9131:::rbd_object_map.125fc09df6d1:head v 51'11, mlcod=51'8)", 2024-09-18T04:38:42.563 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.299452+0000", 2024-09-18T04:38:42.563 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.210753991, 2024-09-18T04:38:42.563 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0053119539999999998, 2024-09-18T04:38:42.563 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:42.563 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:42.563 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:42.563 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.563 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:42.563 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.299452+0000", 2024-09-18T04:38:42.563 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.563 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.563 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.564 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:42.564 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.299452+0000", 2024-09-18T04:38:42.564 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.564 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.564 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.564 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:42.564 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.299451+0000", 2024-09-18T04:38:42.564 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:42.564 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.564 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.564 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:42.564 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.299466+0000", 2024-09-18T04:38:42.564 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5528000000000001e-05 2024-09-18T04:38:42.565 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.565 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.565 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:42.565 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.299481+0000", 2024-09-18T04:38:42.565 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5265e-05 2024-09-18T04:38:42.565 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.565 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.565 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:42.565 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.299489+0000", 2024-09-18T04:38:42.565 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9270000000000005e-06 2024-09-18T04:38:42.565 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.565 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.565 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:42.565 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.300123+0000", 2024-09-18T04:38:42.566 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00063408900000000001 2024-09-18T04:38:42.566 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.566 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.566 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:42.566 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.300145+0000", 2024-09-18T04:38:42.566 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1271999999999999e-05 2024-09-18T04:38:42.566 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.566 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.566 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:42.566 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304708+0000", 2024-09-18T04:38:42.566 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0045632609999999999 2024-09-18T04:38:42.566 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.566 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.567 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:42.567 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304764+0000", 2024-09-18T04:38:42.567 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5575999999999998e-05 2024-09-18T04:38:42.567 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.567 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:42.567 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.567 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.567 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.567 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:46 2.6 2:7ba25245:::rbd_data.125fc09df6d1.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:42.567 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:38:42.567 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.20204591199999999, 2024-09-18T04:38:42.567 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.035604595000000003, 2024-09-18T04:38:42.567 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:42.568 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:42.568 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:42.568 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:38:42.568 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:38:42.568 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 46 2024-09-18T04:38:42.568 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.568 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:42.568 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.568 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:42.568 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:38:42.568 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.568 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.568 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.568 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:42.569 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:38:42.569 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.569 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.569 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.569 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:42.569 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308158+0000", 2024-09-18T04:38:42.569 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:42.569 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.569 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.569 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:42.569 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313191+0000", 2024-09-18T04:38:42.569 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0050323929999999996 2024-09-18T04:38:42.569 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.569 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.570 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:42.570 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313210+0000", 2024-09-18T04:38:42.570 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9673e-05 2024-09-18T04:38:42.570 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.570 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.570 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:42.570 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313221+0000", 2024-09-18T04:38:42.570 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0444e-05 2024-09-18T04:38:42.570 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.570 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.570 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:42.570 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313294+0000", 2024-09-18T04:38:42.570 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2650000000000004e-05 2024-09-18T04:38:42.570 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.571 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.571 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:42.571 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313493+0000", 2024-09-18T04:38:42.571 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019968300000000001 2024-09-18T04:38:42.571 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.571 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.571 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:38:42.571 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313666+0000", 2024-09-18T04:38:42.571 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00017248599999999999 2024-09-18T04:38:42.571 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.571 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.571 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:42.571 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.323175+0000", 2024-09-18T04:38:42.571 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0095088819999999998 2024-09-18T04:38:42.572 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.572 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.572 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:42.572 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343651+0000", 2024-09-18T04:38:42.572 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020475892999999998 2024-09-18T04:38:42.572 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.572 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.572 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:42.572 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343715+0000", 2024-09-18T04:38:42.572 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4443999999999995e-05 2024-09-18T04:38:42.572 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.572 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.572 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:42.573 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343764+0000", 2024-09-18T04:38:42.573 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9280000000000003e-05 2024-09-18T04:38:42.573 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.573 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:42.573 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.573 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.573 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.573 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:47 2.4 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:42.573 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:38:42.573 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.19696387500000001, 2024-09-18T04:38:42.573 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.032570693999999997, 2024-09-18T04:38:42.573 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:42.573 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:42.574 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:42.574 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:38:42.574 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:38:42.574 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 47 2024-09-18T04:38:42.574 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.574 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:42.574 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.574 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:42.574 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:38:42.574 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.575 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.575 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.575 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:42.575 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:38:42.575 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.575 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.575 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.575 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:42.575 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313240+0000", 2024-09-18T04:38:42.575 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:42.575 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.575 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.575 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:42.575 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325825+0000", 2024-09-18T04:38:42.576 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012584425999999999 2024-09-18T04:38:42.576 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.576 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.576 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:42.576 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325848+0000", 2024-09-18T04:38:42.576 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3025999999999999e-05 2024-09-18T04:38:42.576 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.576 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.576 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:42.576 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325863+0000", 2024-09-18T04:38:42.576 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5152e-05 2024-09-18T04:38:42.576 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.576 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.577 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:42.577 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325938+0000", 2024-09-18T04:38:42.577 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4982999999999997e-05 2024-09-18T04:38:42.577 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.577 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.577 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:42.577 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326090+0000", 2024-09-18T04:38:42.577 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015207900000000001 2024-09-18T04:38:42.577 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.577 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.577 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:42.577 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326242+0000", 2024-09-18T04:38:42.577 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015163500000000001 2024-09-18T04:38:42.577 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.578 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.578 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:42.578 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343015+0000", 2024-09-18T04:38:42.578 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016773414 2024-09-18T04:38:42.578 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.578 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.578 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:42.578 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345664+0000", 2024-09-18T04:38:42.578 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026487870000000001 2024-09-18T04:38:42.578 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.578 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.578 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:42.578 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345739+0000", 2024-09-18T04:38:42.579 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5018999999999994e-05 2024-09-18T04:38:42.579 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.579 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.579 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:42.579 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345812+0000", 2024-09-18T04:38:42.579 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3547000000000006e-05 2024-09-18T04:38:42.579 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.579 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:42.579 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.579 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.579 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.579 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:48 2.6 2:61db7a3a:::rbd_data.125fc09df6d1.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:42.579 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:38:42.579 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.184307203, 2024-09-18T04:38:42.580 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.025064388, 2024-09-18T04:38:42.580 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:42.580 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:42.580 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:42.580 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:38:42.580 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:38:42.580 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 48 2024-09-18T04:38:42.580 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.580 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:42.580 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.580 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:42.580 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:38:42.580 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.581 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.581 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.581 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:42.581 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:38:42.581 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.581 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.581 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.581 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:42.581 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325897+0000", 2024-09-18T04:38:42.581 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:42.581 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.581 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.581 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:42.582 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333697+0000", 2024-09-18T04:38:42.582 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0078002649999999998 2024-09-18T04:38:42.582 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.582 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.582 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:42.582 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333716+0000", 2024-09-18T04:38:42.582 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8910999999999999e-05 2024-09-18T04:38:42.582 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.582 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.582 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:42.582 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333728+0000", 2024-09-18T04:38:42.582 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1936000000000001e-05 2024-09-18T04:38:42.582 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.583 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.583 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:42.583 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333792+0000", 2024-09-18T04:38:42.583 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4010999999999994e-05 2024-09-18T04:38:42.583 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.583 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.583 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:42.583 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.334082+0000", 2024-09-18T04:38:42.583 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00028955799999999998 2024-09-18T04:38:42.583 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.583 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.583 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:38:42.583 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.334326+0000", 2024-09-18T04:38:42.583 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00024399099999999999 2024-09-18T04:38:42.584 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.584 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.584 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:42.584 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350745+0000", 2024-09-18T04:38:42.584 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016419474999999999 2024-09-18T04:38:42.584 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.584 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.584 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:42.584 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350808+0000", 2024-09-18T04:38:42.584 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3244999999999995e-05 2024-09-18T04:38:42.584 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.584 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.585 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:42.585 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350892+0000", 2024-09-18T04:38:42.585 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4074999999999997e-05 2024-09-18T04:38:42.585 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.585 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.585 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:42.585 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350963+0000", 2024-09-18T04:38:42.585 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0412000000000003e-05 2024-09-18T04:38:42.585 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.585 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:42.585 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.585 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.585 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.585 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:51 2.6 2:775d7b11:::rbd_data.125fc09df6d1.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:42.586 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:38:42.586 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.17645506699999999, 2024-09-18T04:38:42.586 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.023824221, 2024-09-18T04:38:42.586 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:42.586 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:42.586 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:42.586 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:38:42.586 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:38:42.586 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 51 2024-09-18T04:38:42.586 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.586 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:42.586 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.586 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:42.587 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:38:42.587 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.587 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.587 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.587 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:42.587 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:38:42.587 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.587 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.587 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.587 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:42.587 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333749+0000", 2024-09-18T04:38:42.587 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:42.587 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.588 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.588 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:42.588 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340899+0000", 2024-09-18T04:38:42.588 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0071500340000000004 2024-09-18T04:38:42.588 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.588 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.588 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:42.588 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340929+0000", 2024-09-18T04:38:42.588 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0093000000000001e-05 2024-09-18T04:38:42.588 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.588 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.588 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:42.588 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340943+0000", 2024-09-18T04:38:42.588 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4019000000000001e-05 2024-09-18T04:38:42.589 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.589 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.589 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:42.589 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343834+0000", 2024-09-18T04:38:42.589 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028918120000000001 2024-09-18T04:38:42.589 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.589 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.589 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:42.589 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344020+0000", 2024-09-18T04:38:42.589 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000185892 2024-09-18T04:38:42.589 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.589 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.589 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:38:42.590 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344190+0000", 2024-09-18T04:38:42.590 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016946800000000001 2024-09-18T04:38:42.590 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.590 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.590 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:42.590 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.351291+0000", 2024-09-18T04:38:42.590 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.007100805 2024-09-18T04:38:42.590 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.590 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.590 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:42.590 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357470+0000", 2024-09-18T04:38:42.590 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0061797570000000001 2024-09-18T04:38:42.590 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.590 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.591 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:42.591 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357533+0000", 2024-09-18T04:38:42.591 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2906999999999999e-05 2024-09-18T04:38:42.591 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.591 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.591 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:42.591 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357575+0000", 2024-09-18T04:38:42.591 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1403999999999999e-05 2024-09-18T04:38:42.591 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.591 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:42.591 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.591 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.591 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.591 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop_reply(client.4703.0:46 2.6 e51/17 ondisk, result = 0)", 2024-09-18T04:38:42.592 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.333902+0000", 2024-09-18T04:38:42.592 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.176303875, 2024-09-18T04:38:42.592 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0098763370000000007, 2024-09-18T04:38:42.592 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:42.592 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:38:42.592 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:42.592 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.592 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:42.592 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333902+0000", 2024-09-18T04:38:42.592 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.592 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.592 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.592 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:42.592 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333902+0000", 2024-09-18T04:38:42.592 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.593 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.593 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.593 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:42.593 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333901+0000", 2024-09-18T04:38:42.593 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:42.593 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.593 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.593 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:42.593 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333907+0000", 2024-09-18T04:38:42.593 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1990000000000003e-06 2024-09-18T04:38:42.593 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.593 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.593 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:42.594 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333918+0000", 2024-09-18T04:38:42.594 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0482999999999999e-05 2024-09-18T04:38:42.594 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.594 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.594 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:42.594 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333923+0000", 2024-09-18T04:38:42.594 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5319999999999997e-06 2024-09-18T04:38:42.594 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.594 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.594 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:42.594 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343635+0000", 2024-09-18T04:38:42.594 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0097121450000000002 2024-09-18T04:38:42.594 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.595 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.595 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:42.595 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343645+0000", 2024-09-18T04:38:42.595 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4180000000000003e-06 2024-09-18T04:38:42.595 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.595 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.595 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:42.595 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343778+0000", 2024-09-18T04:38:42.595 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013329600000000001 2024-09-18T04:38:42.595 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.595 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:42.595 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.595 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.595 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.596 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop_reply(client.4703.0:48 2.6 e51/17 ondisk, result = 0)", 2024-09-18T04:38:42.596 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.347795+0000", 2024-09-18T04:38:42.596 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.162410899, 2024-09-18T04:38:42.596 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002957332, 2024-09-18T04:38:42.596 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:42.596 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:38:42.596 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:42.596 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.596 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:42.596 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.347795+0000", 2024-09-18T04:38:42.596 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.596 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.596 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.596 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:42.597 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.347795+0000", 2024-09-18T04:38:42.597 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.597 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.597 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.597 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:42.597 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.347794+0000", 2024-09-18T04:38:42.597 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:42.598 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.598 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.598 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:42.598 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.347799+0000", 2024-09-18T04:38:42.598 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2970000000000003e-06 2024-09-18T04:38:42.598 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.598 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.598 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:42.598 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.347811+0000", 2024-09-18T04:38:42.598 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1623999999999999e-05 2024-09-18T04:38:42.598 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.598 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.599 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:42.599 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.347818+0000", 2024-09-18T04:38:42.599 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2540000000000003e-06 2024-09-18T04:38:42.599 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.599 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.599 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:42.599 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350730+0000", 2024-09-18T04:38:42.599 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0029118619999999999 2024-09-18T04:38:42.599 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.599 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.599 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:42.599 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350738+0000", 2024-09-18T04:38:42.599 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0199999999999994e-06 2024-09-18T04:38:42.599 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.599 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.600 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:42.600 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350752+0000", 2024-09-18T04:38:42.600 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3932e-05 2024-09-18T04:38:42.600 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.600 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:42.600 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.600 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.600 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.600 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:56 2.4 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:42.600 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:38:42.600 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.16040997400000001, 2024-09-18T04:38:42.600 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.034295870999999999, 2024-09-18T04:38:42.600 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:42.601 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:42.601 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:42.601 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:38:42.601 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:38:42.601 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 56 2024-09-18T04:38:42.601 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.601 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:42.601 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.601 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:42.601 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:38:42.601 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.601 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.601 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.601 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:42.602 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:38:42.602 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.602 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.602 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.602 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:42.602 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349794+0000", 2024-09-18T04:38:42.602 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:42.602 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.602 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.602 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:42.602 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366632+0000", 2024-09-18T04:38:42.602 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01683755 2024-09-18T04:38:42.602 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.603 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.603 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:42.603 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366669+0000", 2024-09-18T04:38:42.603 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7351000000000002e-05 2024-09-18T04:38:42.603 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.603 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.603 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:42.603 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366686+0000", 2024-09-18T04:38:42.603 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6507999999999998e-05 2024-09-18T04:38:42.603 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.603 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.603 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:42.603 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366962+0000", 2024-09-18T04:38:42.603 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00027655900000000001 2024-09-18T04:38:42.604 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.604 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.604 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:42.604 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.367452+0000", 2024-09-18T04:38:42.604 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00049025299999999996 2024-09-18T04:38:42.604 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.604 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.604 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:42.604 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368113+0000", 2024-09-18T04:38:42.604 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00066093399999999996 2024-09-18T04:38:42.604 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.604 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.604 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:42.604 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383719+0000", 2024-09-18T04:38:42.605 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015606106999999999 2024-09-18T04:38:42.605 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.605 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.605 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:42.605 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384035+0000", 2024-09-18T04:38:42.605 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00031590399999999999 2024-09-18T04:38:42.605 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.605 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.605 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:42.605 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384067+0000", 2024-09-18T04:38:42.605 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.129e-05 2024-09-18T04:38:42.605 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.605 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.606 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:42.606 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384091+0000", 2024-09-18T04:38:42.606 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4788000000000001e-05 2024-09-18T04:38:42.606 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.606 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:42.606 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.606 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.606 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.606 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:57 2.0 e51/17 2:0cdd070e:::rbd_data.125fc09df6d1.000000000000000d:head v 51'21, mlcod=51'19)", 2024-09-18T04:38:42.606 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:38:42.606 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.15258248699999999, 2024-09-18T04:38:42.606 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.025216311000000002, 2024-09-18T04:38:42.606 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:42.606 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:42.607 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:42.607 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.607 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:42.607 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:38:42.607 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.607 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.607 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.607 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:42.607 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:38:42.607 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.607 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.607 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.607 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:42.608 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:38:42.608 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:42.608 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.608 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.608 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:42.608 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363844+0000", 2024-09-18T04:38:42.608 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0062218450000000002 2024-09-18T04:38:42.608 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.608 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.608 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:42.608 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363879+0000", 2024-09-18T04:38:42.608 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4382999999999999e-05 2024-09-18T04:38:42.608 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.609 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.609 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:42.609 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363893+0000", 2024-09-18T04:38:42.609 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4348999999999999e-05 2024-09-18T04:38:42.609 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.609 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.609 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:42.609 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363977+0000", 2024-09-18T04:38:42.609 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3499999999999997e-05 2024-09-18T04:38:42.609 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.609 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.609 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:42.609 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364009+0000", 2024-09-18T04:38:42.609 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2292000000000002e-05 2024-09-18T04:38:42.610 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.610 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.610 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:42.610 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.382781+0000", 2024-09-18T04:38:42.610 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018771626 2024-09-18T04:38:42.610 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.610 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.610 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:42.610 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.382839+0000", 2024-09-18T04:38:42.610 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8758000000000002e-05 2024-09-18T04:38:42.610 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.610 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:42.610 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.611 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.611 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.611 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:59 2.0 e51/17 2:19c52cde:::rbd_data.125fc09df6d1.000000000000000f:head v 51'22, mlcod=51'19)", 2024-09-18T04:38:42.611 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:38:42.611 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.14592256100000001, 2024-09-18T04:38:42.611 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.019497112, 2024-09-18T04:38:42.611 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:42.611 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:42.611 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:42.611 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.611 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:42.611 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:38:42.611 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.612 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.612 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.612 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:42.612 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:38:42.612 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.612 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.612 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.612 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:42.612 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364282+0000", 2024-09-18T04:38:42.612 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:42.612 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.612 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.612 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:42.613 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371823+0000", 2024-09-18T04:38:42.613 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0075411829999999999 2024-09-18T04:38:42.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.613 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.613 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:42.613 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371837+0000", 2024-09-18T04:38:42.613 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4336e-05 2024-09-18T04:38:42.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.613 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.613 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:42.613 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371846+0000", 2024-09-18T04:38:42.613 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9770000000000006e-06 2024-09-18T04:38:42.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.614 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.614 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:42.614 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.373566+0000", 2024-09-18T04:38:42.614 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017192539999999999 2024-09-18T04:38:42.614 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.614 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.614 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:42.614 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.373595+0000", 2024-09-18T04:38:42.614 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8975999999999998e-05 2024-09-18T04:38:42.614 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.614 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.614 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:42.614 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383740+0000", 2024-09-18T04:38:42.614 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01014525 2024-09-18T04:38:42.615 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.615 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.615 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:42.615 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383780+0000", 2024-09-18T04:38:42.615 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0275999999999999e-05 2024-09-18T04:38:42.615 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.615 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:42.615 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.615 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.615 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.615 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:6 2.6 2:774f7988:ns::rbd_directory:head [call rbd.dir_state_assert in=19b,call rbd.dir_add_image in=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:42.615 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.465732+0000", 2024-09-18T04:38:42.615 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.044473219000000001, 2024-09-18T04:38:42.616 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0029885369999999999, 2024-09-18T04:38:42.616 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:42.616 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:42.616 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:42.616 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:38:42.616 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:42.616 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 6 2024-09-18T04:38:42.616 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.616 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:42.616 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.616 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:42.616 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.465732+0000", 2024-09-18T04:38:42.616 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.617 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.617 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.617 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:42.617 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.465732+0000", 2024-09-18T04:38:42.617 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.617 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.617 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.617 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:42.617 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.465730+0000", 2024-09-18T04:38:42.617 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999971 2024-09-18T04:38:42.617 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.617 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.617 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:42.618 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.465747+0000", 2024-09-18T04:38:42.618 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6901999999999998e-05 2024-09-18T04:38:42.618 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.618 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.618 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:42.618 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.465785+0000", 2024-09-18T04:38:42.618 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8331999999999998e-05 2024-09-18T04:38:42.618 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.618 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.618 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:42.618 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.465804+0000", 2024-09-18T04:38:42.618 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9093000000000002e-05 2024-09-18T04:38:42.618 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.618 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.619 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:42.619 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.465916+0000", 2024-09-18T04:38:42.619 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000112347 2024-09-18T04:38:42.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.619 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.619 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:42.619 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.466068+0000", 2024-09-18T04:38:42.619 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015185200000000001 2024-09-18T04:38:42.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.619 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.619 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:38:42.619 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.466501+0000", 2024-09-18T04:38:42.619 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00043277099999999998 2024-09-18T04:38:42.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.620 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.620 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:42.620 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.467976+0000", 2024-09-18T04:38:42.620 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001474831 2024-09-18T04:38:42.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.620 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.620 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:42.620 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.468574+0000", 2024-09-18T04:38:42.620 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000598413 2024-09-18T04:38:42.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.620 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.620 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:42.620 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.468653+0000", 2024-09-18T04:38:42.621 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9085000000000006e-05 2024-09-18T04:38:42.621 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.621 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.621 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:42.621 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.468721+0000", 2024-09-18T04:38:42.621 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7603999999999999e-05 2024-09-18T04:38:42.621 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.621 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:42.621 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.621 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.621 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:42.621 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:42.623 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:38:42.623 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:42.618+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:42.646 INFO:tasks.workunit.client.0.smithi115.stderr:+ ceph osd pool create new_default_pool17342 8 2024-09-18T04:38:42.723 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:42.718+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:42.724 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:42.724 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:42.724 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:38:42.724 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:42.734 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:38:42.824 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:42.818+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:42.844 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:42.844 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:42.844 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:38:42.844 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:38:42.844 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:42.854 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:38:42.925 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:42.918+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:42.970 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:42.970 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:38:42.970 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:38:42.970 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:38:42.971 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.971 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:8 2.1 2:9370a598:::rbd_header.125fc09df6d1:head [create,call rbd.create in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:42.971 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.206256+0000", 2024-09-18T04:38:42.971 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.75930250200000005, 2024-09-18T04:38:42.971 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030429329999999998, 2024-09-18T04:38:42.971 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:42.971 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:42.971 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:42.971 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:38:42.971 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:38:42.971 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 8 2024-09-18T04:38:42.971 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.971 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:42.972 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.972 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:42.972 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.206256+0000", 2024-09-18T04:38:42.972 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.972 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.972 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.972 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:42.972 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.206256+0000", 2024-09-18T04:38:42.972 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.972 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.972 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.972 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:42.972 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.206254+0000", 2024-09-18T04:38:42.972 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:42.973 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.973 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.973 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:42.973 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.206266+0000", 2024-09-18T04:38:42.973 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2353e-05 2024-09-18T04:38:42.973 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.973 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.973 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:42.973 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.206287+0000", 2024-09-18T04:38:42.973 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0942999999999999e-05 2024-09-18T04:38:42.973 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.973 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.973 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:42.973 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.206304+0000", 2024-09-18T04:38:42.974 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6782999999999998e-05 2024-09-18T04:38:42.974 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.974 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.974 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:42.974 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.206418+0000", 2024-09-18T04:38:42.974 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011377700000000001 2024-09-18T04:38:42.974 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.974 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.974 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:42.974 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.206696+0000", 2024-09-18T04:38:42.974 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000278029 2024-09-18T04:38:42.974 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.974 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.975 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:42.975 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.207055+0000", 2024-09-18T04:38:42.975 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00035913200000000002 2024-09-18T04:38:42.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.975 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.975 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:42.975 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.208686+0000", 2024-09-18T04:38:42.975 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0016314039999999999 2024-09-18T04:38:42.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.975 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.975 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:42.975 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.209128+0000", 2024-09-18T04:38:42.975 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00044201400000000001 2024-09-18T04:38:42.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.976 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.976 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:42.976 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.209208+0000", 2024-09-18T04:38:42.976 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9714000000000003e-05 2024-09-18T04:38:42.976 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.976 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.976 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:42.976 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.209299+0000", 2024-09-18T04:38:42.976 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1263999999999999e-05 2024-09-18T04:38:42.976 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.976 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:42.976 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.976 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.977 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.977 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:21 2.1 2:9370a598:::rbd_header.125fc09df6d1:head [watch watch cookie 140314165667744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:42.977 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.233053+0000", 2024-09-18T04:38:42.977 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.73250553699999998, 2024-09-18T04:38:42.977 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027093289999999999, 2024-09-18T04:38:42.977 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:42.977 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:42.977 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:42.977 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:38:42.977 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:38:42.977 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 21 2024-09-18T04:38:42.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.977 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:42.978 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.978 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:42.978 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.233053+0000", 2024-09-18T04:38:42.978 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.978 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.978 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.978 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:42.978 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.233053+0000", 2024-09-18T04:38:42.978 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.978 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.978 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.978 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:42.978 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.233052+0000", 2024-09-18T04:38:42.978 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:42.978 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.979 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.979 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:42.979 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.233062+0000", 2024-09-18T04:38:42.979 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0336999999999999e-05 2024-09-18T04:38:42.979 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.979 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.979 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:42.979 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.233085+0000", 2024-09-18T04:38:42.979 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2807999999999999e-05 2024-09-18T04:38:42.979 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.979 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.979 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:42.979 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.233095+0000", 2024-09-18T04:38:42.979 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0037000000000001e-05 2024-09-18T04:38:42.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.980 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.980 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:42.980 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.233216+0000", 2024-09-18T04:38:42.980 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000121432 2024-09-18T04:38:42.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.980 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.980 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:42.980 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.233454+0000", 2024-09-18T04:38:42.980 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000237316 2024-09-18T04:38:42.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.980 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.980 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:42.980 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.233866+0000", 2024-09-18T04:38:42.981 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00041284900000000002 2024-09-18T04:38:42.981 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.981 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.981 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:42.981 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.235033+0000", 2024-09-18T04:38:42.981 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0011666300000000001 2024-09-18T04:38:42.981 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.981 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.981 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:42.981 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.235655+0000", 2024-09-18T04:38:42.981 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000621866 2024-09-18T04:38:42.981 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.981 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.981 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:42.982 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.235702+0000", 2024-09-18T04:38:42.982 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6785e-05 2024-09-18T04:38:42.982 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.982 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.982 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:42.982 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.235763+0000", 2024-09-18T04:38:42.982 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0809e-05 2024-09-18T04:38:42.982 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.982 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:42.982 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.982 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.982 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.982 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:23 2.1 2:9370a598:::rbd_header.125fc09df6d1:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:42.982 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.245592+0000", 2024-09-18T04:38:42.982 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.71996721100000005, 2024-09-18T04:38:42.983 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020934339999999999, 2024-09-18T04:38:42.983 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:42.983 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:42.983 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:42.983 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:38:42.983 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:38:42.983 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 23 2024-09-18T04:38:42.983 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.983 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:42.983 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.983 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:42.983 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.245592+0000", 2024-09-18T04:38:42.983 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.983 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.984 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.984 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:42.984 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.245592+0000", 2024-09-18T04:38:42.984 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.984 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.984 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.984 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:42.984 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.245590+0000", 2024-09-18T04:38:42.984 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:42.984 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.984 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.984 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:42.984 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.245601+0000", 2024-09-18T04:38:42.985 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0689e-05 2024-09-18T04:38:42.985 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.985 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.985 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:42.985 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.245643+0000", 2024-09-18T04:38:42.985 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2012999999999999e-05 2024-09-18T04:38:42.985 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.985 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.985 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:42.985 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.245654+0000", 2024-09-18T04:38:42.985 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1493e-05 2024-09-18T04:38:42.985 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.985 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.985 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:42.986 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.245712+0000", 2024-09-18T04:38:42.986 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.7454999999999998e-05 2024-09-18T04:38:42.986 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.986 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.986 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:42.986 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.245796+0000", 2024-09-18T04:38:42.986 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4011999999999995e-05 2024-09-18T04:38:42.986 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.986 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.986 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:42.986 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.245990+0000", 2024-09-18T04:38:42.986 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000194463 2024-09-18T04:38:42.986 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.986 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.986 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:42.987 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.247030+0000", 2024-09-18T04:38:42.987 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0010395339999999999 2024-09-18T04:38:42.987 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.987 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.987 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:42.987 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.247556+0000", 2024-09-18T04:38:42.987 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00052647800000000004 2024-09-18T04:38:42.987 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.987 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.987 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:42.987 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.247628+0000", 2024-09-18T04:38:42.987 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2237999999999995e-05 2024-09-18T04:38:42.987 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.987 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.987 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:42.988 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.247685+0000", 2024-09-18T04:38:42.988 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6651999999999999e-05 2024-09-18T04:38:42.988 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.988 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:42.988 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.988 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.988 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.988 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:33 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:42.988 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.261182+0000", 2024-09-18T04:38:42.988 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.70437627400000002, 2024-09-18T04:38:42.988 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.037082208999999998, 2024-09-18T04:38:42.988 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:42.988 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:42.989 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:42.989 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:38:42.989 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:38:42.989 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 33 2024-09-18T04:38:42.989 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.989 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:42.989 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.989 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:42.989 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.261182+0000", 2024-09-18T04:38:42.989 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.989 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.989 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.989 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:42.989 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.261182+0000", 2024-09-18T04:38:42.990 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.990 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.990 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.990 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:42.990 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.261180+0000", 2024-09-18T04:38:42.990 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:42.990 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.990 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.990 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:42.990 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278669+0000", 2024-09-18T04:38:42.990 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.017488877999999999 2024-09-18T04:38:42.990 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.990 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.990 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:42.991 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278689+0000", 2024-09-18T04:38:42.991 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0047999999999998e-05 2024-09-18T04:38:42.991 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.991 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.991 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:42.991 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278699+0000", 2024-09-18T04:38:42.991 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6800000000000005e-06 2024-09-18T04:38:42.991 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.991 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.991 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:42.991 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278775+0000", 2024-09-18T04:38:42.991 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6087999999999994e-05 2024-09-18T04:38:42.991 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.991 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.991 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:42.992 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278935+0000", 2024-09-18T04:38:42.992 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000159834 2024-09-18T04:38:42.992 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.992 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.992 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:42.992 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279064+0000", 2024-09-18T04:38:42.992 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012954199999999999 2024-09-18T04:38:42.992 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.992 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.992 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:42.992 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295562+0000", 2024-09-18T04:38:42.992 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016497583999999999 2024-09-18T04:38:42.992 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.993 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.993 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:42.993 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.298147+0000", 2024-09-18T04:38:42.993 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002584982 2024-09-18T04:38:42.993 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.993 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.993 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:42.993 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.298217+0000", 2024-09-18T04:38:42.993 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0400000000000004e-05 2024-09-18T04:38:42.993 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.993 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.993 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:42.993 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.298265+0000", 2024-09-18T04:38:42.993 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7580000000000002e-05 2024-09-18T04:38:42.993 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.994 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:42.994 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.994 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.994 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:34 2.3 e51/24 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head v 51'3, mlcod=18'1)", 2024-09-18T04:38:42.994 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.273067+0000", 2024-09-18T04:38:42.994 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.69249148400000005, 2024-09-18T04:38:42.994 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.021784103999999999, 2024-09-18T04:38:42.994 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:42.994 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:42.994 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:42.994 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.994 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:42.994 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.273067+0000", 2024-09-18T04:38:42.994 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.995 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.995 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:42.995 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.273067+0000", 2024-09-18T04:38:42.995 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:42.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.995 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.995 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:42.995 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.273060+0000", 2024-09-18T04:38:42.995 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999924 2024-09-18T04:38:42.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.995 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.995 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:42.995 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278596+0000", 2024-09-18T04:38:42.995 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055365839999999998 2024-09-18T04:38:42.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.996 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.996 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:42.996 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278637+0000", 2024-09-18T04:38:42.996 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0596000000000002e-05 2024-09-18T04:38:42.996 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.996 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.996 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:42.996 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278653+0000", 2024-09-18T04:38:42.996 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6626999999999999e-05 2024-09-18T04:38:42.996 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.996 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.996 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:42.996 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278748+0000", 2024-09-18T04:38:42.996 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4483999999999999e-05 2024-09-18T04:38:42.997 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.997 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.997 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:42.997 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278799+0000", 2024-09-18T04:38:42.997 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1484999999999999e-05 2024-09-18T04:38:42.997 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.997 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.997 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:42.997 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.294785+0000", 2024-09-18T04:38:42.997 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015985256999999999 2024-09-18T04:38:42.997 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.997 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.997 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:42.998 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.294851+0000", 2024-09-18T04:38:42.998 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6728000000000002e-05 2024-09-18T04:38:42.998 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.998 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:42.998 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:42.998 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.998 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:42.998 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:37 2.5 2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:42.998 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.278722+0000", 2024-09-18T04:38:42.998 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.68683643500000002, 2024-09-18T04:38:42.998 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.026323801000000001, 2024-09-18T04:38:42.998 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:42.998 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:42.999 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:42.999 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:38:42.999 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:38:42.999 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 37 2024-09-18T04:38:42.999 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:42.999 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:43.000 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.000 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:43.000 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278722+0000", 2024-09-18T04:38:43.000 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:43.000 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.000 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.000 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:43.000 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278722+0000", 2024-09-18T04:38:43.000 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:43.000 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.000 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.000 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:43.000 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278720+0000", 2024-09-18T04:38:43.001 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:43.001 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.001 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.001 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:43.001 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.291875+0000", 2024-09-18T04:38:43.001 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013154414 2024-09-18T04:38:43.001 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.001 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.001 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:43.001 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.291901+0000", 2024-09-18T04:38:43.001 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6339999999999999e-05 2024-09-18T04:38:43.001 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.001 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.001 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:43.001 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.291914+0000", 2024-09-18T04:38:43.002 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3043e-05 2024-09-18T04:38:43.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.002 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.002 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:43.002 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.291975+0000", 2024-09-18T04:38:43.002 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0949999999999998e-05 2024-09-18T04:38:43.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.002 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.002 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:43.002 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292173+0000", 2024-09-18T04:38:43.002 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019825499999999999 2024-09-18T04:38:43.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.002 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.002 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:43.002 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292300+0000", 2024-09-18T04:38:43.003 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000126616 2024-09-18T04:38:43.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.003 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.003 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:43.003 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304334+0000", 2024-09-18T04:38:43.003 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012033934 2024-09-18T04:38:43.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.003 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.003 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:43.003 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304901+0000", 2024-09-18T04:38:43.003 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00056694099999999995 2024-09-18T04:38:43.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.003 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.003 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:43.004 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304981+0000", 2024-09-18T04:38:43.004 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9974000000000004e-05 2024-09-18T04:38:43.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.004 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.004 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:43.004 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.305046+0000", 2024-09-18T04:38:43.004 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5184000000000005e-05 2024-09-18T04:38:43.004 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:43.004 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:43.004 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:43.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.004 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.004 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:49 2.5 2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:43.004 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.314271+0000", 2024-09-18T04:38:43.005 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.65128783300000004, 2024-09-18T04:38:43.005 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.029779953000000001, 2024-09-18T04:38:43.005 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:43.005 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:43.005 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:43.005 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:38:43.005 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:38:43.005 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 49 2024-09-18T04:38:43.005 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.005 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:43.005 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.005 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:43.005 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.314271+0000", 2024-09-18T04:38:43.006 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:43.006 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.006 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.006 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:43.006 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.314271+0000", 2024-09-18T04:38:43.006 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:43.006 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.006 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.006 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:43.006 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.314269+0000", 2024-09-18T04:38:43.006 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:43.006 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.006 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.006 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:43.007 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.324780+0000", 2024-09-18T04:38:43.007 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010511609999999999 2024-09-18T04:38:43.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.007 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.007 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:43.007 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.324821+0000", 2024-09-18T04:38:43.007 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0899000000000003e-05 2024-09-18T04:38:43.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.007 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.007 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:43.007 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.324840+0000", 2024-09-18T04:38:43.007 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8947e-05 2024-09-18T04:38:43.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.007 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.007 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:43.008 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.324936+0000", 2024-09-18T04:38:43.008 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6342999999999996e-05 2024-09-18T04:38:43.008 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.008 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.008 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:43.008 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325284+0000", 2024-09-18T04:38:43.008 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000347146 2024-09-18T04:38:43.008 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.008 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.008 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:43.008 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325586+0000", 2024-09-18T04:38:43.008 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00030199799999999999 2024-09-18T04:38:43.008 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.009 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.009 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:43.009 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343710+0000", 2024-09-18T04:38:43.009 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018124583999999999 2024-09-18T04:38:43.009 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.009 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.009 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:43.009 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343991+0000", 2024-09-18T04:38:43.009 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00028085600000000001 2024-09-18T04:38:43.009 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.009 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.009 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:43.009 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344028+0000", 2024-09-18T04:38:43.009 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6637000000000002e-05 2024-09-18T04:38:43.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.010 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.010 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:43.010 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344051+0000", 2024-09-18T04:38:43.010 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3260000000000001e-05 2024-09-18T04:38:43.010 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:43.010 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:43.010 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:43.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.010 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.010 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:50 2.3 e51/24 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head v 51'4, mlcod=51'2)", 2024-09-18T04:38:43.010 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.323066+0000", 2024-09-18T04:38:43.010 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.64249244000000005, 2024-09-18T04:38:43.010 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.022135774, 2024-09-18T04:38:43.010 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:43.011 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:43.011 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:43.011 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.011 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:43.011 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.323066+0000", 2024-09-18T04:38:43.011 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:43.011 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.011 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.011 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:43.011 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.323066+0000", 2024-09-18T04:38:43.011 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:43.011 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.011 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.011 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:43.012 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.323065+0000", 2024-09-18T04:38:43.012 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:43.012 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.012 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.012 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:43.012 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.328664+0000", 2024-09-18T04:38:43.012 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055993600000000003 2024-09-18T04:38:43.012 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.012 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.012 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:43.012 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.328692+0000", 2024-09-18T04:38:43.012 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7654e-05 2024-09-18T04:38:43.012 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.012 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.013 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:43.013 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.328706+0000", 2024-09-18T04:38:43.013 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4097000000000001e-05 2024-09-18T04:38:43.013 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.013 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.013 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:43.013 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.328816+0000", 2024-09-18T04:38:43.013 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000110281 2024-09-18T04:38:43.013 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.013 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.013 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:43.013 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.328848+0000", 2024-09-18T04:38:43.013 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1837e-05 2024-09-18T04:38:43.013 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.014 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.014 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:43.014 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345160+0000", 2024-09-18T04:38:43.014 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016312099 2024-09-18T04:38:43.014 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.014 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.014 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:43.014 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345202+0000", 2024-09-18T04:38:43.014 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1857999999999998e-05 2024-09-18T04:38:43.014 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:43.014 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:43.014 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:43.014 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.014 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.015 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:58 2.5 2:b227e6b1:::rbd_data.125fc09df6d1.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:43.015 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.351564+0000", 2024-09-18T04:38:43.015 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.61399498799999996, 2024-09-18T04:38:43.015 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.032125956999999997, 2024-09-18T04:38:43.015 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:43.015 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:43.015 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:43.015 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:38:43.015 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:38:43.015 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 58 2024-09-18T04:38:43.015 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.015 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:43.015 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.015 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:43.015 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.351564+0000", 2024-09-18T04:38:43.016 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:43.016 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.016 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.016 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:43.016 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.351564+0000", 2024-09-18T04:38:43.016 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:43.016 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.016 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.016 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:43.016 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.351561+0000", 2024-09-18T04:38:43.016 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:43.016 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.016 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.016 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:43.016 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364390+0000", 2024-09-18T04:38:43.017 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012829063999999999 2024-09-18T04:38:43.017 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.017 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.017 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:43.017 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364418+0000", 2024-09-18T04:38:43.017 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7307000000000001e-05 2024-09-18T04:38:43.017 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.017 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.017 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:43.017 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364428+0000", 2024-09-18T04:38:43.017 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0675e-05 2024-09-18T04:38:43.017 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.017 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.017 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:43.017 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364523+0000", 2024-09-18T04:38:43.018 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4698999999999996e-05 2024-09-18T04:38:43.018 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.018 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.018 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:43.018 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364785+0000", 2024-09-18T04:38:43.018 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00026170099999999998 2024-09-18T04:38:43.018 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.018 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.018 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:43.018 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.365039+0000", 2024-09-18T04:38:43.018 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00025462299999999999 2024-09-18T04:38:43.018 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.018 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.018 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:43.019 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383143+0000", 2024-09-18T04:38:43.019 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018103768999999999 2024-09-18T04:38:43.019 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.019 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.019 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:43.019 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383582+0000", 2024-09-18T04:38:43.019 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00043860600000000002 2024-09-18T04:38:43.019 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.019 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.019 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:43.019 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383644+0000", 2024-09-18T04:38:43.019 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2682999999999996e-05 2024-09-18T04:38:43.019 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.019 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.019 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:43.019 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383690+0000", 2024-09-18T04:38:43.020 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5247999999999998e-05 2024-09-18T04:38:43.020 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:43.020 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:43.020 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:43.020 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.020 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.020 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:33 2.1 2:8b40f1c5:ns::rbd_data.133e813ab5bc.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:43.020 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.523765+0000", 2024-09-18T04:38:43.020 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.44179394799999999, 2024-09-18T04:38:43.020 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027472973000000001, 2024-09-18T04:38:43.020 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:43.020 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:43.020 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:43.020 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:38:43.020 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:43.021 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 33 2024-09-18T04:38:43.021 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.021 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:43.021 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.021 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:43.021 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.523765+0000", 2024-09-18T04:38:43.021 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:43.021 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.021 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.021 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:43.021 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.523765+0000", 2024-09-18T04:38:43.021 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:43.021 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.021 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.022 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:43.022 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.523762+0000", 2024-09-18T04:38:43.022 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:43.022 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.022 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.022 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:43.022 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531127+0000", 2024-09-18T04:38:43.022 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0073644560000000001 2024-09-18T04:38:43.022 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.022 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.022 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:43.022 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531189+0000", 2024-09-18T04:38:43.022 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2315000000000002e-05 2024-09-18T04:38:43.022 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.022 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.023 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:43.023 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531211+0000", 2024-09-18T04:38:43.023 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1542000000000001e-05 2024-09-18T04:38:43.023 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.023 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.023 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:43.023 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531314+0000", 2024-09-18T04:38:43.023 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000103582 2024-09-18T04:38:43.023 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.023 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.023 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:43.023 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531493+0000", 2024-09-18T04:38:43.023 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000178433 2024-09-18T04:38:43.023 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.024 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.024 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:43.024 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531653+0000", 2024-09-18T04:38:43.024 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000160388 2024-09-18T04:38:43.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.024 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.024 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:43.024 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.547345+0000", 2024-09-18T04:38:43.024 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015691742000000002 2024-09-18T04:38:43.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.024 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.024 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:43.024 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551076+0000", 2024-09-18T04:38:43.024 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003730957 2024-09-18T04:38:43.025 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.025 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.025 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:43.025 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551176+0000", 2024-09-18T04:38:43.025 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010002599999999999 2024-09-18T04:38:43.025 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.025 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.025 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:43.025 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551238+0000", 2024-09-18T04:38:43.025 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1945000000000004e-05 2024-09-18T04:38:43.025 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:43.025 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:43.025 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:43.025 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.026 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.026 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:32 2.3 e51/24 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head v 51'10, mlcod=51'7)", 2024-09-18T04:38:43.026 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.524206+0000", 2024-09-18T04:38:43.026 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.44135273000000003, 2024-09-18T04:38:43.026 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0037857780000000001, 2024-09-18T04:38:43.026 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:43.026 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:43.026 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:43.026 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.026 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:43.026 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524206+0000", 2024-09-18T04:38:43.026 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:43.026 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.026 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.027 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:43.027 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524206+0000", 2024-09-18T04:38:43.027 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:43.027 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.027 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.027 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:43.027 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524205+0000", 2024-09-18T04:38:43.027 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:43.027 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.027 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.027 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:43.027 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524229+0000", 2024-09-18T04:38:43.027 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4182999999999999e-05 2024-09-18T04:38:43.027 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.027 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.028 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:43.028 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524265+0000", 2024-09-18T04:38:43.028 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6032000000000003e-05 2024-09-18T04:38:43.028 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.028 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.028 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:43.028 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524282+0000", 2024-09-18T04:38:43.028 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6773e-05 2024-09-18T04:38:43.028 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.028 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.028 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:43.028 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524439+0000", 2024-09-18T04:38:43.028 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015771900000000001 2024-09-18T04:38:43.028 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.029 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.029 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:43.029 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524484+0000", 2024-09-18T04:38:43.029 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4465000000000002e-05 2024-09-18T04:38:43.029 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.029 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.029 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:43.029 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.527910+0000", 2024-09-18T04:38:43.029 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0034260860000000001 2024-09-18T04:38:43.029 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.029 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.029 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:43.029 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.527992+0000", 2024-09-18T04:38:43.029 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1934000000000005e-05 2024-09-18T04:38:43.030 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:43.030 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:43.030 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:43.030 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.030 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.030 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:35 2.3 e51/24 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head v 51'11, mlcod=51'7)", 2024-09-18T04:38:43.030 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.525811+0000", 2024-09-18T04:38:43.030 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.43974820599999997, 2024-09-18T04:38:43.030 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028483979999999998, 2024-09-18T04:38:43.030 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:43.030 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:43.030 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:43.030 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.031 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:43.031 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.525811+0000", 2024-09-18T04:38:43.031 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:43.031 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.031 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.031 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:43.031 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.525811+0000", 2024-09-18T04:38:43.031 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:43.031 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.031 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.031 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:43.031 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.525802+0000", 2024-09-18T04:38:43.031 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999919 2024-09-18T04:38:43.031 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.031 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.031 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:43.032 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.525828+0000", 2024-09-18T04:38:43.032 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5602999999999999e-05 2024-09-18T04:38:43.032 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.032 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.032 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:43.032 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.525861+0000", 2024-09-18T04:38:43.032 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2916000000000001e-05 2024-09-18T04:38:43.032 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.032 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.032 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:43.032 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.525878+0000", 2024-09-18T04:38:43.032 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6616000000000002e-05 2024-09-18T04:38:43.032 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.033 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.033 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:43.033 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.526706+0000", 2024-09-18T04:38:43.033 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00082842300000000001 2024-09-18T04:38:43.033 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.033 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.033 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:43.033 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.526795+0000", 2024-09-18T04:38:43.033 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9271000000000006e-05 2024-09-18T04:38:43.033 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.033 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.033 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:43.033 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.528510+0000", 2024-09-18T04:38:43.033 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001715034 2024-09-18T04:38:43.034 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.034 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.034 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:43.034 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.528659+0000", 2024-09-18T04:38:43.034 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000148589 2024-09-18T04:38:43.034 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:43.034 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:43.034 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:43.034 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.034 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.034 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:36 2.5 2:bd5b8dd8:ns::rbd_data.133e813ab5bc.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:43.034 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.531253+0000", 2024-09-18T04:38:43.034 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.43430563900000002, 2024-09-18T04:38:43.034 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027295212999999999, 2024-09-18T04:38:43.034 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:43.035 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:43.035 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:43.035 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:38:43.035 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:43.035 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 36 2024-09-18T04:38:43.035 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.035 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:43.035 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.035 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:43.035 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531253+0000", 2024-09-18T04:38:43.035 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:43.035 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.035 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.035 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:43.036 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531253+0000", 2024-09-18T04:38:43.036 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:43.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.036 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.036 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:43.036 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531251+0000", 2024-09-18T04:38:43.036 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:43.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.036 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.036 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:43.036 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536540+0000", 2024-09-18T04:38:43.036 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0052885839999999998 2024-09-18T04:38:43.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.036 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.037 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:43.037 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536604+0000", 2024-09-18T04:38:43.037 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4824000000000001e-05 2024-09-18T04:38:43.037 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.037 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.037 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:43.037 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536621+0000", 2024-09-18T04:38:43.037 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6215999999999999e-05 2024-09-18T04:38:43.037 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.037 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.037 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:43.037 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536768+0000", 2024-09-18T04:38:43.037 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014767900000000001 2024-09-18T04:38:43.037 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.037 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.038 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:43.038 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537084+0000", 2024-09-18T04:38:43.038 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00031553499999999998 2024-09-18T04:38:43.038 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.038 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.038 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:43.038 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537300+0000", 2024-09-18T04:38:43.038 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000216145 2024-09-18T04:38:43.038 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.038 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.038 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:43.038 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555326+0000", 2024-09-18T04:38:43.038 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018025861000000001 2024-09-18T04:38:43.039 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.039 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.039 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:43.039 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558449+0000", 2024-09-18T04:38:43.039 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0031229750000000001 2024-09-18T04:38:43.039 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.039 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.039 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:43.039 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558502+0000", 2024-09-18T04:38:43.039 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3572000000000001e-05 2024-09-18T04:38:43.039 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.039 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.039 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:43.039 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558548+0000", 2024-09-18T04:38:43.040 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5921999999999998e-05 2024-09-18T04:38:43.040 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:43.040 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:43.040 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:43.040 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.040 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.040 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:37 2.5 2:bf39b8e4:ns::rbd_data.133e813ab5bc.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:43.040 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.536664+0000", 2024-09-18T04:38:43.040 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.42889422300000002, 2024-09-18T04:38:43.040 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.026331765, 2024-09-18T04:38:43.040 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:43.040 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:43.040 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:43.040 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:38:43.041 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:43.041 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 37 2024-09-18T04:38:43.041 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.041 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:43.041 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.041 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:43.041 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536664+0000", 2024-09-18T04:38:43.041 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:43.041 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.041 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.041 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:43.041 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536664+0000", 2024-09-18T04:38:43.041 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:43.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.042 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.042 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:43.042 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536662+0000", 2024-09-18T04:38:43.042 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:43.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.042 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.042 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:43.042 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.543699+0000", 2024-09-18T04:38:43.042 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0070367119999999997 2024-09-18T04:38:43.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.042 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.042 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:43.042 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.543749+0000", 2024-09-18T04:38:43.043 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0476999999999998e-05 2024-09-18T04:38:43.043 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.043 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.043 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:43.043 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.543800+0000", 2024-09-18T04:38:43.043 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0822999999999998e-05 2024-09-18T04:38:43.043 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.043 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.043 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:43.043 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.547868+0000", 2024-09-18T04:38:43.043 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0040679339999999996 2024-09-18T04:38:43.043 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.043 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.043 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:43.044 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548087+0000", 2024-09-18T04:38:43.044 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000218994 2024-09-18T04:38:43.044 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.044 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.044 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:43.044 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548383+0000", 2024-09-18T04:38:43.044 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000295588 2024-09-18T04:38:43.044 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.044 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.044 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:43.044 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562622+0000", 2024-09-18T04:38:43.044 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014239915000000001 2024-09-18T04:38:43.044 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.044 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.044 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:43.045 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562924+0000", 2024-09-18T04:38:43.045 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00030127600000000001 2024-09-18T04:38:43.045 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.045 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.045 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:43.045 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562957+0000", 2024-09-18T04:38:43.045 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3365999999999999e-05 2024-09-18T04:38:43.045 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.045 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.045 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:43.045 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562996+0000", 2024-09-18T04:38:43.045 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9153999999999998e-05 2024-09-18T04:38:43.045 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:43.045 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:43.046 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:43.046 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.046 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.046 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:42 2.3 e51/24 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head v 51'14, mlcod=51'11)", 2024-09-18T04:38:43.046 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.554081+0000", 2024-09-18T04:38:43.046 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.41147796199999997, 2024-09-18T04:38:43.046 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0080081739999999998, 2024-09-18T04:38:43.046 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:43.046 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:43.046 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:43.046 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.046 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:43.046 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.554081+0000", 2024-09-18T04:38:43.046 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:43.046 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.047 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.047 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:43.047 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.554081+0000", 2024-09-18T04:38:43.047 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:43.047 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.047 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.047 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:43.047 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.554079+0000", 2024-09-18T04:38:43.047 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:43.047 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.047 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.047 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:43.047 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.554092+0000", 2024-09-18T04:38:43.047 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3032e-05 2024-09-18T04:38:43.048 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.048 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.048 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:43.048 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.554105+0000", 2024-09-18T04:38:43.048 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3217000000000001e-05 2024-09-18T04:38:43.048 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.048 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.048 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:43.048 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.554115+0000", 2024-09-18T04:38:43.048 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9089999999999995e-06 2024-09-18T04:38:43.048 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.048 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.048 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:43.049 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.554165+0000", 2024-09-18T04:38:43.049 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9644000000000001e-05 2024-09-18T04:38:43.049 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.049 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.049 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:43.049 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.554188+0000", 2024-09-18T04:38:43.049 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3753000000000001e-05 2024-09-18T04:38:43.049 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.049 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.049 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:43.049 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562047+0000", 2024-09-18T04:38:43.049 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0078585519999999996 2024-09-18T04:38:43.049 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.050 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.050 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:43.050 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562089+0000", 2024-09-18T04:38:43.050 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1940000000000002e-05 2024-09-18T04:38:43.050 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:43.050 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:43.050 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:43.050 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.050 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.050 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:44 2.3 e51/24 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head v 51'15, mlcod=51'12)", 2024-09-18T04:38:43.050 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.559761+0000", 2024-09-18T04:38:43.050 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.40579771100000001, 2024-09-18T04:38:43.050 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028584270000000002, 2024-09-18T04:38:43.050 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:43.050 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:43.051 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:43.051 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.051 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:43.051 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.559761+0000", 2024-09-18T04:38:43.051 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:43.051 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.051 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.051 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:43.051 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.559761+0000", 2024-09-18T04:38:43.051 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:43.051 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.051 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.051 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:43.051 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.559760+0000", 2024-09-18T04:38:43.052 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:43.052 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.052 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.052 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:43.052 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.559769+0000", 2024-09-18T04:38:43.052 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0019999999999995e-06 2024-09-18T04:38:43.052 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.052 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.052 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:43.052 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.559782+0000", 2024-09-18T04:38:43.052 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2988e-05 2024-09-18T04:38:43.052 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.052 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.052 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:43.053 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.559789+0000", 2024-09-18T04:38:43.053 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0049999999999998e-06 2024-09-18T04:38:43.053 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.053 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.053 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:43.053 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.559837+0000", 2024-09-18T04:38:43.053 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8219e-05 2024-09-18T04:38:43.053 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.053 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.053 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:43.053 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.559854+0000", 2024-09-18T04:38:43.053 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6843000000000001e-05 2024-09-18T04:38:43.053 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.054 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.054 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:43.054 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562586+0000", 2024-09-18T04:38:43.054 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027315099999999999 2024-09-18T04:38:43.054 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.054 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.054 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:43.054 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562619+0000", 2024-09-18T04:38:43.054 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3612000000000003e-05 2024-09-18T04:38:43.054 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:43.054 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:43.054 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:43.054 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.054 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.055 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:50 2.1 2:8c706374:ns::rbd_data.133e813ab5bc.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:43.055 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.570513+0000", 2024-09-18T04:38:43.055 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.39504594100000001, 2024-09-18T04:38:43.055 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.024182995999999998, 2024-09-18T04:38:43.055 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:43.055 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:43.055 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:43.055 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:38:43.055 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:43.055 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 50 2024-09-18T04:38:43.055 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.055 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:43.055 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.055 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:43.055 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570513+0000", 2024-09-18T04:38:43.056 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:43.056 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.056 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.056 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:43.056 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570513+0000", 2024-09-18T04:38:43.056 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:43.056 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.056 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.056 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:43.056 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570511+0000", 2024-09-18T04:38:43.056 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:43.056 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.056 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.056 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:43.057 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.576327+0000", 2024-09-18T04:38:43.057 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0058162530000000004 2024-09-18T04:38:43.057 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.057 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.057 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:43.057 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.576355+0000", 2024-09-18T04:38:43.057 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8030000000000001e-05 2024-09-18T04:38:43.057 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.057 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.057 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:43.057 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.576366+0000", 2024-09-18T04:38:43.057 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1139e-05 2024-09-18T04:38:43.057 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.057 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.057 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:43.058 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.576458+0000", 2024-09-18T04:38:43.058 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1780999999999994e-05 2024-09-18T04:38:43.058 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.058 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.058 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:43.058 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.576701+0000", 2024-09-18T04:38:43.058 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00024303899999999999 2024-09-18T04:38:43.058 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.058 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.058 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:43.058 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.576876+0000", 2024-09-18T04:38:43.058 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000174637 2024-09-18T04:38:43.058 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.058 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.058 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:43.059 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588432+0000", 2024-09-18T04:38:43.059 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011555943000000001 2024-09-18T04:38:43.059 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.059 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.059 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:43.059 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594573+0000", 2024-09-18T04:38:43.059 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0061408670000000004 2024-09-18T04:38:43.059 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.059 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.059 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:43.059 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594640+0000", 2024-09-18T04:38:43.059 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7404999999999996e-05 2024-09-18T04:38:43.059 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.059 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.059 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:43.059 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594696+0000", 2024-09-18T04:38:43.060 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5612999999999998e-05 2024-09-18T04:38:43.060 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:43.060 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:43.060 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:43.060 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.060 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.060 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:52 2.3 e51/24 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head v 51'19, mlcod=51'16)", 2024-09-18T04:38:43.060 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.581089+0000", 2024-09-18T04:38:43.060 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.38447009700000001, 2024-09-18T04:38:43.060 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0036266890000000002, 2024-09-18T04:38:43.060 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:43.060 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:43.060 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:43.061 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.061 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:43.061 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581089+0000", 2024-09-18T04:38:43.061 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:43.061 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.061 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.061 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:43.061 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581089+0000", 2024-09-18T04:38:43.061 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:43.061 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.061 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.061 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:43.061 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581087+0000", 2024-09-18T04:38:43.062 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:43.062 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.062 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.062 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:43.062 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581101+0000", 2024-09-18T04:38:43.062 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.379e-05 2024-09-18T04:38:43.062 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.062 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.062 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:43.062 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581117+0000", 2024-09-18T04:38:43.062 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.609e-05 2024-09-18T04:38:43.062 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.062 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.063 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:43.063 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581126+0000", 2024-09-18T04:38:43.063 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2679999999999994e-06 2024-09-18T04:38:43.063 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.063 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.063 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:43.063 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.583238+0000", 2024-09-18T04:38:43.063 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021123380000000001 2024-09-18T04:38:43.063 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.063 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.063 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:43.063 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.583265+0000", 2024-09-18T04:38:43.063 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7554000000000001e-05 2024-09-18T04:38:43.063 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.063 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.064 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:43.064 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584647+0000", 2024-09-18T04:38:43.064 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0013813499999999999 2024-09-18T04:38:43.064 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.064 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.064 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:43.064 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584715+0000", 2024-09-18T04:38:43.064 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8551000000000001e-05 2024-09-18T04:38:43.064 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:43.064 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:43.064 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:43.064 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.064 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.065 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:55 2.3 e51/24 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head v 51'20, mlcod=51'18)", 2024-09-18T04:38:43.065 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.589520+0000", 2024-09-18T04:38:43.065 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.37603834600000002, 2024-09-18T04:38:43.065 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0040254549999999998, 2024-09-18T04:38:43.065 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:43.065 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:43.065 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:43.065 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.065 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:43.065 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.589520+0000", 2024-09-18T04:38:43.065 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:43.065 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.065 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.065 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:43.066 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.589520+0000", 2024-09-18T04:38:43.066 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:43.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.066 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.066 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:43.066 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.589519+0000", 2024-09-18T04:38:43.066 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:43.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.066 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.066 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:43.066 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.589532+0000", 2024-09-18T04:38:43.066 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3147e-05 2024-09-18T04:38:43.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.066 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.067 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:43.067 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.589549+0000", 2024-09-18T04:38:43.067 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6841999999999999e-05 2024-09-18T04:38:43.067 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.067 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.067 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:43.067 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.589559+0000", 2024-09-18T04:38:43.067 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6509999999999996e-06 2024-09-18T04:38:43.067 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.067 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.067 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:43.067 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.589644+0000", 2024-09-18T04:38:43.067 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4969000000000006e-05 2024-09-18T04:38:43.067 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.067 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.068 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:43.068 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.589672+0000", 2024-09-18T04:38:43.068 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7759999999999998e-05 2024-09-18T04:38:43.068 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.068 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.068 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:43.068 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593495+0000", 2024-09-18T04:38:43.068 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0038238730000000002 2024-09-18T04:38:43.068 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.068 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.068 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:43.068 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593546+0000", 2024-09-18T04:38:43.068 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0455999999999999e-05 2024-09-18T04:38:43.068 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:43.069 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:43.069 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:43.069 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.069 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.069 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:60 2.5 2:a06f51e4:ns::rbd_data.133e813ab5bc.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:43.069 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.604753+0000", 2024-09-18T04:38:43.069 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.360805822, 2024-09-18T04:38:43.069 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014400181, 2024-09-18T04:38:43.069 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:43.069 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:43.069 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:43.069 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:38:43.069 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:43.070 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 60 2024-09-18T04:38:43.070 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.070 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:43.070 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.070 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:43.070 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.604753+0000", 2024-09-18T04:38:43.070 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:43.070 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.070 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.070 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:43.070 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.604753+0000", 2024-09-18T04:38:43.070 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:43.070 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.070 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.070 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:43.071 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.604751+0000", 2024-09-18T04:38:43.071 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:43.071 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.071 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.071 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:43.071 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.608980+0000", 2024-09-18T04:38:43.071 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0042292420000000002 2024-09-18T04:38:43.071 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.071 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.071 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:43.071 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609014+0000", 2024-09-18T04:38:43.071 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3572999999999999e-05 2024-09-18T04:38:43.071 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.071 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.072 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:43.072 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609032+0000", 2024-09-18T04:38:43.072 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8150000000000001e-05 2024-09-18T04:38:43.072 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.072 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.072 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:43.072 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609090+0000", 2024-09-18T04:38:43.072 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8214999999999997e-05 2024-09-18T04:38:43.072 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.072 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.072 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:43.072 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609235+0000", 2024-09-18T04:38:43.072 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014437799999999999 2024-09-18T04:38:43.072 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.073 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.073 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:43.073 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609360+0000", 2024-09-18T04:38:43.073 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012526099999999999 2024-09-18T04:38:43.073 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.073 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.073 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:43.073 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.614952+0000", 2024-09-18T04:38:43.073 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055922300000000001 2024-09-18T04:38:43.073 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.073 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.073 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:43.073 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.619070+0000", 2024-09-18T04:38:43.073 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0041178250000000003 2024-09-18T04:38:43.073 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.074 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.074 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:43.074 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.619117+0000", 2024-09-18T04:38:43.074 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7275999999999999e-05 2024-09-18T04:38:43.074 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:43.074 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:43.074 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:43.074 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.619153+0000", 2024-09-18T04:38:43.074 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5970000000000003e-05 2024-09-18T04:38:43.074 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:43.074 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:43.074 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:43.074 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:43.074 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:43.074 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:43.076 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:38:43.077 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:43.074+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:43.177 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:43.170+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:43.190 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:43.190 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:43.190 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:38:43.190 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:43.197 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T04:38:43.200 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:38:43.278 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:43.270+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:43.308 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:43.309 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:43.309 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:38:43.309 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:38:43.309 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:43.318 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:38:43.349 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:38:43.372 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:38:43.378 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:43.374+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:43.393 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:38:43.416 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:38:43.421 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:43.421 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:43.431 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:38:43.440 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:38:43.468 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:38:43.470 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:38:43.470 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:38:43.470 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:38:43.470 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:38:43.470 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:38:43.470 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:38:43.479 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:43.474+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:43.537 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:43.537 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:43.547 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:38:43.581 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:43.574+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:43.682 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:43.678+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:43.701 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:43.702 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:43.711 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:38:43.782 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:43.778+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:43.816 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:43.816 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:43.826 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:38:43.884 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:43.878+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:43.934 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:43.935 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:43.946 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:38:43.953 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-18T04:38:43.985 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:43.978+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:44.045 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:44.045 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:44.056 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:38:44.057 INFO:tasks.workunit.client.0.smithi115.stderr:pool 'new_default_pool17342' already exists 2024-09-18T04:38:44.066 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd pool init new_default_pool17342 2024-09-18T04:38:44.102 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:44.098+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:44.187 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:44.182+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:44.215 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:44.215 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:44.225 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:38:44.287 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:44.282+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:44.334 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:44.334 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:44.344 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:38:44.389 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:44.382+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:44.453 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:44.453 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:44.463 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:38:44.490 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:44.486+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:44.567 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:44.568 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:44.577 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:38:44.618 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:44.614+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:44.692 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:44.686+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:44.728 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:44.728 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:44.738 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:38:44.793 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:44.790+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:44.846 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:44.846 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:44.856 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:38:44.893 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:44.890+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:44.925 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-18T04:38:44.925 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T04:38:44.925 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-18T04:38:44.925 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 1.000000 2024-09-18T04:38:44.925 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-09-18T04:38:44.952 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-09-18T04:38:44.972 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:44.973 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:44.983 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:38:44.994 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:44.990+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:45.096 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:45.090+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:45.132 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:45.132 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:45.142 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:38:45.196 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:45.190+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:45.248 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:45.248 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:45.258 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:38:45.302 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:45.298+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:45.361 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:45.361 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:45.372 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:38:45.399 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:45.394+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:45.483 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T04:38:45.489 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:45.489 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:45.499 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:38:45.530 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:45.526+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:45.600 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:45.594+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:45.607 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:45.608 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:45.618 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:38:45.633 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:38:45.656 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:38:45.674 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:38:45.695 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:38:45.700 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:45.694+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:45.717 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:38:45.729 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:45.729 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:38:45.729 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:38:45.729 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:38:45.729 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.729 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:45.729 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:38:45.729 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 77.937853922000002, 2024-09-18T04:38:45.729 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017399218000000001, 2024-09-18T04:38:45.730 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:45.730 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:45.730 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:45.730 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:45.730 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:45.730 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 114 2024-09-18T04:38:45.730 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.730 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:45.730 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.730 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:45.730 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:38:45.730 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:45.730 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.731 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.731 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:45.731 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:38:45.731 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:45.731 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.731 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.731 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:45.731 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786900+0000", 2024-09-18T04:38:45.731 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:45.731 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.731 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.731 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:45.731 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786908+0000", 2024-09-18T04:38:45.731 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8199999999999997e-06 2024-09-18T04:38:45.732 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.732 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.732 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:45.732 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786925+0000", 2024-09-18T04:38:45.732 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6849000000000001e-05 2024-09-18T04:38:45.732 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.732 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.732 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:45.732 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786934+0000", 2024-09-18T04:38:45.732 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6299999999999993e-06 2024-09-18T04:38:45.732 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.732 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.732 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:45.733 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800926+0000", 2024-09-18T04:38:45.733 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013991632 2024-09-18T04:38:45.733 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.733 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.733 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:45.733 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801060+0000", 2024-09-18T04:38:45.733 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013378199999999999 2024-09-18T04:38:45.733 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.733 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.733 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:45.733 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801166+0000", 2024-09-18T04:38:45.733 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010607099999999999 2024-09-18T04:38:45.733 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.734 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.734 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:45.734 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803026+0000", 2024-09-18T04:38:45.734 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001860469 2024-09-18T04:38:45.734 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.734 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.734 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:45.734 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804245+0000", 2024-09-18T04:38:45.734 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0012185900000000001 2024-09-18T04:38:45.734 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.734 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.734 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:45.734 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804278+0000", 2024-09-18T04:38:45.734 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3272000000000003e-05 2024-09-18T04:38:45.735 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.735 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.735 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:45.735 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804300+0000", 2024-09-18T04:38:45.735 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2266e-05 2024-09-18T04:38:45.735 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:45.735 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:45.735 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:45.735 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.735 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.735 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:45.735 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787566+0000", 2024-09-18T04:38:45.735 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 77.937188962999997, 2024-09-18T04:38:45.735 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017124120999999999, 2024-09-18T04:38:45.736 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:45.736 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:45.736 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:45.736 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:45.736 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:45.736 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 115 2024-09-18T04:38:45.736 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.736 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:45.736 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.736 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:45.736 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787566+0000", 2024-09-18T04:38:45.736 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:45.736 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.737 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.737 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:45.737 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787566+0000", 2024-09-18T04:38:45.737 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:45.737 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.737 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.737 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:45.737 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787564+0000", 2024-09-18T04:38:45.737 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:45.737 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.737 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.737 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:45.737 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787751+0000", 2024-09-18T04:38:45.737 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000186873 2024-09-18T04:38:45.738 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.738 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.738 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:45.738 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787772+0000", 2024-09-18T04:38:45.738 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.126e-05 2024-09-18T04:38:45.738 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.738 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.738 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:45.738 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787794+0000", 2024-09-18T04:38:45.738 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1418999999999999e-05 2024-09-18T04:38:45.738 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.738 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.738 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:45.738 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801659+0000", 2024-09-18T04:38:45.738 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013865125000000001 2024-09-18T04:38:45.739 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.739 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.739 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:45.739 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801750+0000", 2024-09-18T04:38:45.739 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.1427000000000004e-05 2024-09-18T04:38:45.739 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.739 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.739 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:45.739 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801894+0000", 2024-09-18T04:38:45.739 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000143514 2024-09-18T04:38:45.739 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.739 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.739 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:45.739 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804313+0000", 2024-09-18T04:38:45.740 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0024190879999999998 2024-09-18T04:38:45.740 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.740 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.740 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:45.740 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804644+0000", 2024-09-18T04:38:45.740 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00033100799999999999 2024-09-18T04:38:45.740 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.740 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.740 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:45.740 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804672+0000", 2024-09-18T04:38:45.740 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7988e-05 2024-09-18T04:38:45.740 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.740 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.741 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:45.741 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804690+0000", 2024-09-18T04:38:45.741 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8298999999999999e-05 2024-09-18T04:38:45.741 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:45.741 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:45.741 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:45.741 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.741 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.741 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:45.741 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:38:45.741 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 77.936941024000006, 2024-09-18T04:38:45.741 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017521662, 2024-09-18T04:38:45.741 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:45.741 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:45.742 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:45.742 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:45.742 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:45.742 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 116 2024-09-18T04:38:45.742 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.742 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:45.742 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.742 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:45.742 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:38:45.742 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:45.742 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.743 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.743 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:45.743 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:38:45.743 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:45.743 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.743 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.743 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:45.743 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787813+0000", 2024-09-18T04:38:45.743 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:45.743 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.743 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.743 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:45.743 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787824+0000", 2024-09-18T04:38:45.744 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1157e-05 2024-09-18T04:38:45.744 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.744 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.744 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:45.744 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787840+0000", 2024-09-18T04:38:45.744 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6269e-05 2024-09-18T04:38:45.744 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.744 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.744 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:45.744 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787853+0000", 2024-09-18T04:38:45.744 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3156000000000001e-05 2024-09-18T04:38:45.744 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.744 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.745 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:45.745 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803110+0000", 2024-09-18T04:38:45.745 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015256736 2024-09-18T04:38:45.745 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.745 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.745 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:45.745 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803197+0000", 2024-09-18T04:38:45.745 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7195999999999997e-05 2024-09-18T04:38:45.745 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.745 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.745 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:45.745 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803308+0000", 2024-09-18T04:38:45.745 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111005 2024-09-18T04:38:45.746 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.746 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.746 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:45.746 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804316+0000", 2024-09-18T04:38:45.746 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010073949999999999 2024-09-18T04:38:45.746 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.746 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.746 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:45.746 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805232+0000", 2024-09-18T04:38:45.746 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091636300000000001 2024-09-18T04:38:45.746 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.747 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.747 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:45.747 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805277+0000", 2024-09-18T04:38:45.747 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4864000000000003e-05 2024-09-18T04:38:45.747 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.747 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.748 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:45.748 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805336+0000", 2024-09-18T04:38:45.748 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8968000000000002e-05 2024-09-18T04:38:45.748 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:45.748 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:45.749 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:45.749 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.749 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.749 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:45.749 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:38:45.749 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 77.936888968999995, 2024-09-18T04:38:45.749 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017983453, 2024-09-18T04:38:45.749 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:45.750 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:45.750 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:45.750 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:45.750 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:45.750 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 117 2024-09-18T04:38:45.750 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.750 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:45.750 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.750 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:45.750 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:38:45.750 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:45.750 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.750 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.750 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:45.751 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:38:45.751 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:45.751 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.751 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.751 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:45.751 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787865+0000", 2024-09-18T04:38:45.751 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:45.751 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.751 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.751 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:45.751 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787889+0000", 2024-09-18T04:38:45.752 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4244999999999999e-05 2024-09-18T04:38:45.752 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.752 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.752 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:45.752 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787905+0000", 2024-09-18T04:38:45.752 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5739999999999998e-05 2024-09-18T04:38:45.752 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.752 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.752 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:45.753 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787918+0000", 2024-09-18T04:38:45.753 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.252e-05 2024-09-18T04:38:45.753 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.753 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.753 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:45.753 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803693+0000", 2024-09-18T04:38:45.753 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015774872999999998 2024-09-18T04:38:45.753 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.753 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.753 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:45.753 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803775+0000", 2024-09-18T04:38:45.753 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2761000000000003e-05 2024-09-18T04:38:45.754 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.754 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.754 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:45.754 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803887+0000", 2024-09-18T04:38:45.754 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111708 2024-09-18T04:38:45.754 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.754 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.754 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:45.754 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804823+0000", 2024-09-18T04:38:45.754 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00093625100000000003 2024-09-18T04:38:45.754 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.754 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.754 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:45.754 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805804+0000", 2024-09-18T04:38:45.755 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00098114699999999992 2024-09-18T04:38:45.755 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.755 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.755 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:45.755 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805831+0000", 2024-09-18T04:38:45.755 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6401999999999999e-05 2024-09-18T04:38:45.755 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.755 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.755 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:45.755 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805850+0000", 2024-09-18T04:38:45.755 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8862e-05 2024-09-18T04:38:45.755 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:45.755 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:45.756 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:45.756 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.756 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.756 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:45.756 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:38:45.756 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 77.493711837999996, 2024-09-18T04:38:45.756 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016528505999999998, 2024-09-18T04:38:45.756 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:45.756 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:45.756 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:45.756 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:45.757 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:45.757 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 138 2024-09-18T04:38:45.757 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.757 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:45.757 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.757 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:45.757 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:38:45.757 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:45.757 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.757 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.757 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:45.757 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:38:45.757 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:45.758 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.758 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.758 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:45.758 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:38:45.758 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:45.758 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.758 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.758 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:45.758 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231047+0000", 2024-09-18T04:38:45.758 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5399999999999997e-06 2024-09-18T04:38:45.758 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.758 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.759 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:45.759 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231068+0000", 2024-09-18T04:38:45.759 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0769000000000001e-05 2024-09-18T04:38:45.759 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.759 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.759 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:45.759 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231074+0000", 2024-09-18T04:38:45.759 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6180000000000003e-06 2024-09-18T04:38:45.759 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.759 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.759 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:45.759 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242556+0000", 2024-09-18T04:38:45.759 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011482654 2024-09-18T04:38:45.759 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.760 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.760 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:45.760 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242666+0000", 2024-09-18T04:38:45.760 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011003200000000001 2024-09-18T04:38:45.760 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.760 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.760 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:45.760 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242924+0000", 2024-09-18T04:38:45.760 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00025785899999999999 2024-09-18T04:38:45.761 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.761 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.761 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:45.761 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247263+0000", 2024-09-18T04:38:45.761 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0043387649999999996 2024-09-18T04:38:45.762 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.762 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.762 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:45.762 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247412+0000", 2024-09-18T04:38:45.762 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000149276 2024-09-18T04:38:45.762 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.762 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.762 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:45.763 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247522+0000", 2024-09-18T04:38:45.763 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000109788 2024-09-18T04:38:45.763 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.763 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.763 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:45.763 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247572+0000", 2024-09-18T04:38:45.763 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9808999999999997e-05 2024-09-18T04:38:45.763 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:45.763 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:45.763 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:45.763 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.764 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.764 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:45.764 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231083+0000", 2024-09-18T04:38:45.764 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 77.493672375000003, 2024-09-18T04:38:45.765 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0166731, 2024-09-18T04:38:45.765 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:45.765 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:45.765 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:45.765 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:45.765 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:45.765 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 139 2024-09-18T04:38:45.765 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.765 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:45.765 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.765 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:45.765 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231083+0000", 2024-09-18T04:38:45.766 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:45.766 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.766 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.766 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:45.766 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231083+0000", 2024-09-18T04:38:45.766 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:45.766 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.766 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.766 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:45.766 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231082+0000", 2024-09-18T04:38:45.766 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:45.766 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.766 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.766 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:45.767 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231090+0000", 2024-09-18T04:38:45.767 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1280000000000008e-06 2024-09-18T04:38:45.767 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.767 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.767 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:45.767 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231097+0000", 2024-09-18T04:38:45.767 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5119999999999997e-06 2024-09-18T04:38:45.767 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.767 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.767 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:45.767 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231106+0000", 2024-09-18T04:38:45.767 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9230000000000008e-06 2024-09-18T04:38:45.768 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.768 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.768 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:45.768 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.243977+0000", 2024-09-18T04:38:45.768 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012870754 2024-09-18T04:38:45.768 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.768 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.768 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:45.768 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244112+0000", 2024-09-18T04:38:45.768 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000135575 2024-09-18T04:38:45.768 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.768 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.768 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:45.769 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244265+0000", 2024-09-18T04:38:45.769 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015279900000000001 2024-09-18T04:38:45.769 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.769 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.769 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:45.769 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247649+0000", 2024-09-18T04:38:45.769 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003383896 2024-09-18T04:38:45.769 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.769 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.770 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:45.770 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247656+0000", 2024-09-18T04:38:45.770 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8920000000000003e-06 2024-09-18T04:38:45.770 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.770 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.770 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:45.770 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247692+0000", 2024-09-18T04:38:45.770 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6470000000000001e-05 2024-09-18T04:38:45.770 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.770 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.771 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:45.771 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247756+0000", 2024-09-18T04:38:45.771 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3676999999999994e-05 2024-09-18T04:38:45.771 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:45.771 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:45.771 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:45.771 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.771 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.772 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:45.772 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231190+0000", 2024-09-18T04:38:45.772 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 77.493565449000002, 2024-09-18T04:38:45.772 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016793556000000001, 2024-09-18T04:38:45.772 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:45.772 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:45.772 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:45.772 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:45.772 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:45.772 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 140 2024-09-18T04:38:45.772 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.772 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:45.773 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.773 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:45.773 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231190+0000", 2024-09-18T04:38:45.773 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:45.773 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.773 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.773 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:45.773 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231190+0000", 2024-09-18T04:38:45.773 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:45.773 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.773 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.773 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:45.773 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231189+0000", 2024-09-18T04:38:45.774 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:45.774 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.774 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.774 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:45.774 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231273+0000", 2024-09-18T04:38:45.774 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4445999999999998e-05 2024-09-18T04:38:45.774 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.774 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.774 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:45.774 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231282+0000", 2024-09-18T04:38:45.774 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8859999999999993e-06 2024-09-18T04:38:45.774 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.774 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.775 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:45.775 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231288+0000", 2024-09-18T04:38:45.775 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9819999999999996e-06 2024-09-18T04:38:45.775 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.775 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.775 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:45.775 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.245119+0000", 2024-09-18T04:38:45.775 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.01383121 2024-09-18T04:38:45.775 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.775 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.775 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:45.775 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.245247+0000", 2024-09-18T04:38:45.775 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012763299999999999 2024-09-18T04:38:45.775 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.775 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.776 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:45.776 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.245401+0000", 2024-09-18T04:38:45.776 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015399200000000001 2024-09-18T04:38:45.776 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.776 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.776 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:45.776 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247766+0000", 2024-09-18T04:38:45.776 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0023649510000000001 2024-09-18T04:38:45.776 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.776 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.776 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:45.776 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247904+0000", 2024-09-18T04:38:45.776 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013836499999999999 2024-09-18T04:38:45.776 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.777 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.777 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:45.777 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247950+0000", 2024-09-18T04:38:45.777 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5201000000000003e-05 2024-09-18T04:38:45.777 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.777 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.777 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:45.777 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247983+0000", 2024-09-18T04:38:45.777 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3686000000000002e-05 2024-09-18T04:38:45.777 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:45.777 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:45.777 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:45.778 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.778 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.778 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:45.778 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:38:45.778 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 77.493454705999994, 2024-09-18T04:38:45.778 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017344024, 2024-09-18T04:38:45.778 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:45.778 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:45.778 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:45.778 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:45.778 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:45.778 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 141 2024-09-18T04:38:45.779 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.779 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:45.779 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.779 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:45.779 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:38:45.779 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:45.779 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.779 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.779 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:45.779 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:38:45.779 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:45.779 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.779 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.779 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:45.780 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:38:45.780 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:45.780 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.780 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.780 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:45.780 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231309+0000", 2024-09-18T04:38:45.780 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5419999999999994e-06 2024-09-18T04:38:45.780 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.780 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.780 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:45.780 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231323+0000", 2024-09-18T04:38:45.780 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3108000000000001e-05 2024-09-18T04:38:45.780 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.781 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.781 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:45.781 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231340+0000", 2024-09-18T04:38:45.781 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.696e-05 2024-09-18T04:38:45.781 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.781 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.781 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:45.781 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.246143+0000", 2024-09-18T04:38:45.781 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014803320999999999 2024-09-18T04:38:45.782 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.782 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.782 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:45.782 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.246323+0000", 2024-09-18T04:38:45.782 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00018061600000000001 2024-09-18T04:38:45.782 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.782 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.782 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:45.782 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.246530+0000", 2024-09-18T04:38:45.782 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020625 2024-09-18T04:38:45.782 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.782 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.782 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:45.782 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248002+0000", 2024-09-18T04:38:45.783 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0014719379999999999 2024-09-18T04:38:45.783 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.783 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.783 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:45.783 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248582+0000", 2024-09-18T04:38:45.783 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00058020599999999999 2024-09-18T04:38:45.783 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.783 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.783 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:45.784 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248618+0000", 2024-09-18T04:38:45.784 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6559e-05 2024-09-18T04:38:45.784 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.784 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.784 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:45.784 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248645+0000", 2024-09-18T04:38:45.784 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6080000000000001e-05 2024-09-18T04:38:45.784 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:45.784 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:45.784 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:45.784 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.784 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.784 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:36 2.7 2:fe1b02ad:::rbd_data.125fc09df6d1.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:45.785 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:38:45.785 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 3.4590328590000001, 2024-09-18T04:38:45.785 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.040387804999999999, 2024-09-18T04:38:45.785 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:45.785 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:45.785 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:45.785 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:38:45.785 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:38:45.785 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 36 2024-09-18T04:38:45.785 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.785 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:45.785 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.786 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:45.786 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:38:45.786 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:45.786 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.786 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.786 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:45.786 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:38:45.786 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:45.786 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.786 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.787 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:45.787 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265720+0000", 2024-09-18T04:38:45.787 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:45.787 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.787 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.787 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:45.787 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272590+0000", 2024-09-18T04:38:45.787 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.006870508 2024-09-18T04:38:45.787 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.787 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.787 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:45.787 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272623+0000", 2024-09-18T04:38:45.787 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2716000000000003e-05 2024-09-18T04:38:45.788 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.788 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.788 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:45.788 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272637+0000", 2024-09-18T04:38:45.788 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4175999999999999e-05 2024-09-18T04:38:45.788 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.788 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.788 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:45.788 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272779+0000", 2024-09-18T04:38:45.788 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000141392 2024-09-18T04:38:45.788 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.788 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.788 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:45.789 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273031+0000", 2024-09-18T04:38:45.789 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00025249099999999998 2024-09-18T04:38:45.789 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.789 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.789 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:38:45.789 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273219+0000", 2024-09-18T04:38:45.789 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000187531 2024-09-18T04:38:45.789 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.789 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.789 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:45.789 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.302806+0000", 2024-09-18T04:38:45.789 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.029587553999999999 2024-09-18T04:38:45.789 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.789 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.790 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:45.790 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306004+0000", 2024-09-18T04:38:45.790 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0031982629999999998 2024-09-18T04:38:45.790 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.790 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.790 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:45.790 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306071+0000", 2024-09-18T04:38:45.790 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6829999999999995e-05 2024-09-18T04:38:45.790 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.790 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.791 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:45.791 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306110+0000", 2024-09-18T04:38:45.791 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8874000000000001e-05 2024-09-18T04:38:45.791 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:45.791 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:45.791 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:45.791 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.791 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.791 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:38 2.0 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:45.791 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:38:45.791 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 3.4520793219999999, 2024-09-18T04:38:45.791 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.033754348000000003, 2024-09-18T04:38:45.791 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:45.792 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:45.792 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:45.792 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:38:45.792 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:38:45.792 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 38 2024-09-18T04:38:45.792 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.792 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:45.792 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.792 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:45.792 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:38:45.792 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:45.792 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.792 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.792 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:45.792 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:38:45.793 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:45.793 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.793 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.793 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:45.793 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272673+0000", 2024-09-18T04:38:45.793 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:45.793 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.793 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.793 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:45.793 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277166+0000", 2024-09-18T04:38:45.793 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044922249999999999 2024-09-18T04:38:45.793 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.793 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.794 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:45.794 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277189+0000", 2024-09-18T04:38:45.794 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3283000000000001e-05 2024-09-18T04:38:45.794 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.794 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.794 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:45.794 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277205+0000", 2024-09-18T04:38:45.794 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6305000000000001e-05 2024-09-18T04:38:45.794 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.794 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.794 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:45.795 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277290+0000", 2024-09-18T04:38:45.795 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5135000000000004e-05 2024-09-18T04:38:45.795 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.795 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.795 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:45.796 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277531+0000", 2024-09-18T04:38:45.796 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00024014699999999999 2024-09-18T04:38:45.796 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.796 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.796 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:45.797 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277793+0000", 2024-09-18T04:38:45.797 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00026224499999999998 2024-09-18T04:38:45.797 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.797 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.797 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:45.797 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303990+0000", 2024-09-18T04:38:45.797 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.026197295999999998 2024-09-18T04:38:45.797 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.797 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.797 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:45.798 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306340+0000", 2024-09-18T04:38:45.798 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.002349524 2024-09-18T04:38:45.798 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.798 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.798 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:45.798 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306394+0000", 2024-09-18T04:38:45.798 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3934999999999998e-05 2024-09-18T04:38:45.798 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.798 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.798 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:45.799 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306430+0000", 2024-09-18T04:38:45.799 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6690000000000003e-05 2024-09-18T04:38:45.799 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:45.799 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:45.799 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:45.799 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.799 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.799 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:39 2.7 2:e9c909c5:::rbd_data.125fc09df6d1.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:45.799 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:38:45.799 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 3.447519717, 2024-09-18T04:38:45.799 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.029110673, 2024-09-18T04:38:45.799 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:45.799 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:45.800 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:45.800 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:38:45.800 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:38:45.800 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 39 2024-09-18T04:38:45.800 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.800 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:45.800 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.800 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:45.800 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:38:45.800 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:45.800 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.800 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.800 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:45.800 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:38:45.801 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:45.801 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.801 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.801 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:45.801 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277234+0000", 2024-09-18T04:38:45.801 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:45.801 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.801 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.801 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:45.801 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281351+0000", 2024-09-18T04:38:45.802 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0041170920000000001 2024-09-18T04:38:45.802 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.802 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.802 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:45.802 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281375+0000", 2024-09-18T04:38:45.802 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.446e-05 2024-09-18T04:38:45.802 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.802 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.802 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:45.802 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281390+0000", 2024-09-18T04:38:45.802 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4309e-05 2024-09-18T04:38:45.802 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.802 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.803 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:45.803 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.282306+0000", 2024-09-18T04:38:45.803 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091662000000000004 2024-09-18T04:38:45.803 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.803 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.803 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:45.803 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.282613+0000", 2024-09-18T04:38:45.803 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00030663799999999997 2024-09-18T04:38:45.803 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.803 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.803 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:38:45.803 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283001+0000", 2024-09-18T04:38:45.803 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00038801599999999998 2024-09-18T04:38:45.803 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.804 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.804 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:45.804 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306239+0000", 2024-09-18T04:38:45.804 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023238466999999999 2024-09-18T04:38:45.804 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.804 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.804 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:45.804 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306259+0000", 2024-09-18T04:38:45.804 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9837999999999999e-05 2024-09-18T04:38:45.804 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.804 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.804 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:45.804 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306299+0000", 2024-09-18T04:38:45.804 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0194000000000002e-05 2024-09-18T04:38:45.805 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.805 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.805 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:45.805 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306346+0000", 2024-09-18T04:38:45.805 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6845000000000003e-05 2024-09-18T04:38:45.805 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:45.805 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:45.805 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:45.805 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.805 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.805 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:46 2.6 e51/17 2:7ba25245:::rbd_data.125fc09df6d1.0000000000000006:head v 51'2, mlcod=0'0)", 2024-09-18T04:38:45.805 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:38:45.805 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 3.4115985850000001, 2024-09-18T04:38:45.806 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019843098, 2024-09-18T04:38:45.806 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:45.806 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:45.806 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:45.806 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.806 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:45.806 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:38:45.806 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:45.806 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.806 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.806 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:45.806 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:38:45.806 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:45.806 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.807 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.807 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:45.807 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313156+0000", 2024-09-18T04:38:45.807 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:45.807 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.807 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.807 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:45.807 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320144+0000", 2024-09-18T04:38:45.807 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.006988345 2024-09-18T04:38:45.807 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.807 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.808 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:45.808 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320175+0000", 2024-09-18T04:38:45.808 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.0861999999999997e-05 2024-09-18T04:38:45.808 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.808 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.808 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:45.808 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320188+0000", 2024-09-18T04:38:45.808 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2714999999999999e-05 2024-09-18T04:38:45.808 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.808 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.808 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:45.809 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320267+0000", 2024-09-18T04:38:45.809 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9586999999999995e-05 2024-09-18T04:38:45.809 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.809 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.809 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:45.809 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320311+0000", 2024-09-18T04:38:45.809 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3529000000000003e-05 2024-09-18T04:38:45.809 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.809 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.809 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:45.809 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.332949+0000", 2024-09-18T04:38:45.810 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012638267999999999 2024-09-18T04:38:45.810 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.810 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.810 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:45.810 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.333000+0000", 2024-09-18T04:38:45.810 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0736999999999999e-05 2024-09-18T04:38:45.810 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:45.810 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:45.810 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:45.810 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.810 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.811 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:57 2.0 2:0cdd070e:::rbd_data.125fc09df6d1.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:45.811 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:38:45.811 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 3.37574101, 2024-09-18T04:38:45.811 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.033329974999999998, 2024-09-18T04:38:45.811 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:45.811 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:45.811 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:45.811 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:38:45.811 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:38:45.811 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 57 2024-09-18T04:38:45.811 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.811 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:45.811 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.811 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:45.812 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:38:45.812 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:45.812 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.812 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.812 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:45.812 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:38:45.812 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:45.812 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.812 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.812 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:45.812 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349013+0000", 2024-09-18T04:38:45.812 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:45.812 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.812 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.813 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:45.813 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356138+0000", 2024-09-18T04:38:45.813 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071258010000000002 2024-09-18T04:38:45.813 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.813 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.813 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:45.813 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356168+0000", 2024-09-18T04:38:45.813 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9679000000000002e-05 2024-09-18T04:38:45.813 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.813 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.813 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:45.813 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356181+0000", 2024-09-18T04:38:45.813 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3164e-05 2024-09-18T04:38:45.813 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.814 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.814 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:45.814 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356254+0000", 2024-09-18T04:38:45.814 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3182999999999994e-05 2024-09-18T04:38:45.814 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.814 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.814 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:45.814 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356453+0000", 2024-09-18T04:38:45.814 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019876599999999999 2024-09-18T04:38:45.814 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.814 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.814 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:45.814 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356627+0000", 2024-09-18T04:38:45.815 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017340199999999999 2024-09-18T04:38:45.815 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.815 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.815 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:45.815 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.369200+0000", 2024-09-18T04:38:45.815 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012573302 2024-09-18T04:38:45.815 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.815 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.815 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:45.815 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382262+0000", 2024-09-18T04:38:45.816 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013062549 2024-09-18T04:38:45.816 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.816 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.816 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:45.816 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382310+0000", 2024-09-18T04:38:45.816 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7366e-05 2024-09-18T04:38:45.817 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.817 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.817 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:45.817 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382344+0000", 2024-09-18T04:38:45.817 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4319999999999997e-05 2024-09-18T04:38:45.818 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:45.818 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:45.818 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:45.818 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.818 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.818 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:59 2.0 2:19c52cde:::rbd_data.125fc09df6d1.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:45.818 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:38:45.819 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 3.368549845, 2024-09-18T04:38:45.819 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.027137247999999999, 2024-09-18T04:38:45.819 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:45.819 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:45.819 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:45.819 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:38:45.819 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:38:45.819 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 59 2024-09-18T04:38:45.819 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.819 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:45.819 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.819 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:45.820 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:38:45.820 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:45.820 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.820 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.820 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:45.820 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:38:45.820 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:45.820 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.820 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.820 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:45.820 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356204+0000", 2024-09-18T04:38:45.820 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:45.820 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.820 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.821 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:45.821 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362661+0000", 2024-09-18T04:38:45.821 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0064565129999999997 2024-09-18T04:38:45.821 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.821 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.821 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:45.821 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362684+0000", 2024-09-18T04:38:45.821 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3688000000000001e-05 2024-09-18T04:38:45.821 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.821 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.821 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:45.821 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362699+0000", 2024-09-18T04:38:45.821 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4798e-05 2024-09-18T04:38:45.822 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.822 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.822 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:45.822 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363016+0000", 2024-09-18T04:38:45.822 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031737300000000003 2024-09-18T04:38:45.822 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.822 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.822 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:45.822 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363201+0000", 2024-09-18T04:38:45.822 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000184131 2024-09-18T04:38:45.822 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.822 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.822 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:45.823 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363343+0000", 2024-09-18T04:38:45.823 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014236699999999999 2024-09-18T04:38:45.823 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.823 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.823 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:45.823 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.372189+0000", 2024-09-18T04:38:45.823 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0088464819999999993 2024-09-18T04:38:45.823 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.823 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.823 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:45.823 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383269+0000", 2024-09-18T04:38:45.823 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011079594 2024-09-18T04:38:45.823 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.823 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.824 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:45.824 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383314+0000", 2024-09-18T04:38:45.824 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4932000000000003e-05 2024-09-18T04:38:45.824 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.824 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.824 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:45.824 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383343+0000", 2024-09-18T04:38:45.824 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8600999999999999e-05 2024-09-18T04:38:45.824 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:45.824 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:45.824 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:45.824 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.824 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.824 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:40 2.6 e51/17 2:784d6cd1:ns::rbd_data.133e813ab5bc.0000000000000005:head v 51'7, mlcod=51'5)", 2024-09-18T04:38:45.825 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:38:45.825 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 3.1723001110000002, 2024-09-18T04:38:45.825 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020497496, 2024-09-18T04:38:45.825 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:45.825 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:45.825 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:45.826 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.826 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:45.826 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:38:45.826 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:45.826 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.826 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.826 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:45.826 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:38:45.826 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:45.826 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.827 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.827 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:45.827 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552454+0000", 2024-09-18T04:38:45.827 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:45.827 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.827 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.827 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:45.827 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556448+0000", 2024-09-18T04:38:45.827 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0039940990000000001 2024-09-18T04:38:45.827 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.827 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.827 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:45.827 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556481+0000", 2024-09-18T04:38:45.828 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3778000000000001e-05 2024-09-18T04:38:45.828 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.828 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.828 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:45.828 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556493+0000", 2024-09-18T04:38:45.828 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1622e-05 2024-09-18T04:38:45.828 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.828 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.828 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:45.828 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556556+0000", 2024-09-18T04:38:45.828 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2582999999999994e-05 2024-09-18T04:38:45.828 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.828 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.829 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:45.829 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556592+0000", 2024-09-18T04:38:45.829 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5880999999999997e-05 2024-09-18T04:38:45.829 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.829 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.829 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:45.829 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.572786+0000", 2024-09-18T04:38:45.829 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016194571000000001 2024-09-18T04:38:45.831 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.832 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.832 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:45.832 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.572953+0000", 2024-09-18T04:38:45.832 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000166436 2024-09-18T04:38:45.832 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:45.832 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:45.832 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:45.832 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.832 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.832 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:45 2.0 2:0fbfa3dc:ns::rbd_data.133e813ab5bc.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:45.832 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:38:45.832 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 3.1610858120000001, 2024-09-18T04:38:45.833 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.036558168000000002, 2024-09-18T04:38:45.833 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:45.833 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:45.833 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:45.833 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:38:45.833 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:45.833 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 45 2024-09-18T04:38:45.833 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.833 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:45.833 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.833 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:45.833 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:38:45.833 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:45.834 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.834 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.834 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:45.834 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:38:45.834 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:45.834 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.834 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.834 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:45.834 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:38:45.834 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:45.834 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.834 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.834 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:45.834 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578594+0000", 2024-09-18T04:38:45.835 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014927147999999999 2024-09-18T04:38:45.835 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.835 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.835 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:45.835 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578612+0000", 2024-09-18T04:38:45.835 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7623999999999999e-05 2024-09-18T04:38:45.835 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.835 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.835 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:45.835 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578620+0000", 2024-09-18T04:38:45.835 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9070000000000007e-06 2024-09-18T04:38:45.835 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.835 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.835 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:45.836 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578702+0000", 2024-09-18T04:38:45.836 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2349999999999996e-05 2024-09-18T04:38:45.836 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.836 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.836 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:45.836 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.579028+0000", 2024-09-18T04:38:45.836 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032622899999999998 2024-09-18T04:38:45.836 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.836 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.836 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:45.836 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.579343+0000", 2024-09-18T04:38:45.836 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031448899999999999 2024-09-18T04:38:45.836 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.836 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.837 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:45.837 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.590078+0000", 2024-09-18T04:38:45.837 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010734946 2024-09-18T04:38:45.837 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.837 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.837 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:45.837 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600089+0000", 2024-09-18T04:38:45.837 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010011195000000001 2024-09-18T04:38:45.837 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.837 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.837 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:45.837 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600177+0000", 2024-09-18T04:38:45.838 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8425999999999997e-05 2024-09-18T04:38:45.838 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.838 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.838 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:45.838 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600228+0000", 2024-09-18T04:38:45.838 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0216999999999997e-05 2024-09-18T04:38:45.838 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:45.838 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:45.839 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:45.839 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.839 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.839 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:48 2.6 e51/17 2:7c225365:ns::rbd_data.133e813ab5bc.0000000000000008:head v 51'9, mlcod=51'5)", 2024-09-18T04:38:45.839 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.570076+0000", 2024-09-18T04:38:45.839 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 3.1546787099999998, 2024-09-18T04:38:45.839 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016939773000000002, 2024-09-18T04:38:45.839 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:45.839 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:45.839 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:45.839 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.839 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:45.839 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.570076+0000", 2024-09-18T04:38:45.839 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:45.840 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.840 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.840 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:45.840 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.570076+0000", 2024-09-18T04:38:45.840 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:45.840 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.840 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.840 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:45.840 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.570074+0000", 2024-09-18T04:38:45.840 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:45.840 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.840 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.840 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:45.840 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.574672+0000", 2024-09-18T04:38:45.841 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045979819999999996 2024-09-18T04:38:45.841 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.841 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.841 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:45.841 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.574695+0000", 2024-09-18T04:38:45.841 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2529999999999999e-05 2024-09-18T04:38:45.841 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.841 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.841 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:45.841 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.574706+0000", 2024-09-18T04:38:45.841 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1357e-05 2024-09-18T04:38:45.841 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.841 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.842 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:45.842 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.574746+0000", 2024-09-18T04:38:45.842 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9586999999999999e-05 2024-09-18T04:38:45.842 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.842 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.842 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:45.842 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.574764+0000", 2024-09-18T04:38:45.842 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8000999999999999e-05 2024-09-18T04:38:45.842 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.842 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.842 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:45.842 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.586847+0000", 2024-09-18T04:38:45.842 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.01208359 2024-09-18T04:38:45.843 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.843 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.843 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:45.843 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.587016+0000", 2024-09-18T04:38:45.843 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000168796 2024-09-18T04:38:45.843 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:45.843 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:45.843 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:45.843 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.843 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.843 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:54 2.0 2:04f17ef6:ns::rbd_data.133e813ab5bc.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:45.843 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:38:45.843 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 3.1395139749999998, 2024-09-18T04:38:45.843 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.024612999, 2024-09-18T04:38:45.844 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:45.844 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:45.844 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:45.844 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:38:45.844 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:45.844 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 54 2024-09-18T04:38:45.844 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.844 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:45.844 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.844 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:45.844 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:38:45.844 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:45.845 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.845 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.845 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:45.845 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:38:45.845 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:45.845 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.845 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.845 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:45.845 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585239+0000", 2024-09-18T04:38:45.845 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:45.845 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.845 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.846 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:45.846 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593702+0000", 2024-09-18T04:38:45.846 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0084632990000000005 2024-09-18T04:38:45.846 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.846 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.846 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:45.846 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593719+0000", 2024-09-18T04:38:45.846 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.664e-05 2024-09-18T04:38:45.846 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.846 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.846 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:45.846 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593727+0000", 2024-09-18T04:38:45.846 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6839999999999996e-06 2024-09-18T04:38:45.846 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.847 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.847 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:45.847 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593767+0000", 2024-09-18T04:38:45.847 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0330999999999997e-05 2024-09-18T04:38:45.847 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.847 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.847 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:45.847 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593905+0000", 2024-09-18T04:38:45.847 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013754800000000001 2024-09-18T04:38:45.847 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.847 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.847 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:45.847 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.594013+0000", 2024-09-18T04:38:45.847 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000108805 2024-09-18T04:38:45.848 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.848 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.848 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:45.848 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604678+0000", 2024-09-18T04:38:45.848 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010664391 2024-09-18T04:38:45.848 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.848 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.848 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:45.848 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609662+0000", 2024-09-18T04:38:45.848 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0049846100000000004 2024-09-18T04:38:45.848 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.848 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.848 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:45.849 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609788+0000", 2024-09-18T04:38:45.849 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012518800000000001 2024-09-18T04:38:45.849 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.849 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.849 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:45.849 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609854+0000", 2024-09-18T04:38:45.849 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6556000000000004e-05 2024-09-18T04:38:45.849 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:45.849 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:45.849 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:45.849 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.849 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.849 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:58 2.7 2:fc7c6cd3:ns::rbd_data.133e813ab5bc.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:45.850 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:38:45.850 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 3.1266715459999999, 2024-09-18T04:38:45.850 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.027690566999999999, 2024-09-18T04:38:45.850 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:45.850 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:45.850 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:45.851 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:38:45.851 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:45.851 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 58 2024-09-18T04:38:45.851 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.851 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:45.852 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.852 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:45.852 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:38:45.852 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:45.852 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.852 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.852 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:45.852 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:38:45.852 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:45.853 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.853 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.853 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:45.853 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598082+0000", 2024-09-18T04:38:45.853 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:45.853 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.853 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.853 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:45.853 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604832+0000", 2024-09-18T04:38:45.854 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0067504510000000002 2024-09-18T04:38:45.854 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.854 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.854 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:45.854 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604852+0000", 2024-09-18T04:38:45.854 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0228999999999999e-05 2024-09-18T04:38:45.854 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.854 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.854 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:45.854 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604870+0000", 2024-09-18T04:38:45.854 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7952e-05 2024-09-18T04:38:45.854 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.854 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.855 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:45.855 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604936+0000", 2024-09-18T04:38:45.855 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5431000000000004e-05 2024-09-18T04:38:45.855 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.855 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.855 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:45.855 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605085+0000", 2024-09-18T04:38:45.855 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014910300000000001 2024-09-18T04:38:45.855 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.855 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.855 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:38:45.855 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605205+0000", 2024-09-18T04:38:45.855 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000119887 2024-09-18T04:38:45.855 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.856 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.856 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:45.856 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625586+0000", 2024-09-18T04:38:45.856 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020381399000000001 2024-09-18T04:38:45.856 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.856 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.856 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:45.856 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625628+0000", 2024-09-18T04:38:45.856 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1873999999999999e-05 2024-09-18T04:38:45.856 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.856 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.857 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:45.857 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625722+0000", 2024-09-18T04:38:45.857 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3787000000000002e-05 2024-09-18T04:38:45.857 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.857 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.857 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:45.857 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625774+0000", 2024-09-18T04:38:45.857 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2503999999999997e-05 2024-09-18T04:38:45.857 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:45.857 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:45.857 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:45.857 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.857 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.858 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:56 2.6 e51/17 2:72453c3e:ns::rbd_data.133e813ab5bc.000000000000000d:head v 51'10, mlcod=51'8)", 2024-09-18T04:38:45.858 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:38:45.858 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 3.1194744380000001, 2024-09-18T04:38:45.858 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019766552999999999, 2024-09-18T04:38:45.858 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:45.858 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:45.858 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:45.858 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.858 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:45.858 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:38:45.858 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:45.858 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.858 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.858 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:45.859 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:38:45.859 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:45.859 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.859 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.859 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:45.859 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605279+0000", 2024-09-18T04:38:45.859 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:45.859 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.859 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.859 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:45.859 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609417+0000", 2024-09-18T04:38:45.859 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0041378059999999999 2024-09-18T04:38:45.859 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.860 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.860 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:45.860 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609443+0000", 2024-09-18T04:38:45.860 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6098e-05 2024-09-18T04:38:45.860 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.860 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.860 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:45.860 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609458+0000", 2024-09-18T04:38:45.860 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4671000000000001e-05 2024-09-18T04:38:45.860 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.860 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.861 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:45.861 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609553+0000", 2024-09-18T04:38:45.861 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5265000000000004e-05 2024-09-18T04:38:45.861 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.861 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.861 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:45.861 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609583+0000", 2024-09-18T04:38:45.861 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9839e-05 2024-09-18T04:38:45.861 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.861 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.862 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:45.862 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625004+0000", 2024-09-18T04:38:45.862 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015420639999999999 2024-09-18T04:38:45.862 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:45.862 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:45.862 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:45.862 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625047+0000", 2024-09-18T04:38:45.862 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3665999999999998e-05 2024-09-18T04:38:45.862 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:45.862 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:45.862 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:45.862 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:45.862 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:45.863 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:45.864 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:38:45.865 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:38:45.865 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:45.862+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:45.866 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:38:45.866 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:38:45.866 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:38:45.866 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:38:45.866 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:38:45.866 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:38:45.965 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:45.958+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:45.973 INFO:teuthology.orchestra.run.smithi028.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-18T04:38:46.020 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:46.020 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:46.020 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:38:46.020 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:46.030 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:38:46.074 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:46.069+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:46.133 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:46.133 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:46.133 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:38:46.134 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:38:46.134 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:46.143 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:38:46.166 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:46.162+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:46.253 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:46.254 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:38:46.254 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:38:46.254 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [] 2024-09-18T04:38:46.254 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:46.263 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:38:46.267 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:46.262+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:46.367 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:46.366+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:46.371 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:46.372 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:46.372 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:38:46.372 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:46.381 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:38:46.468 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:46.461+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:46.491 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:46.491 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:46.491 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:38:46.491 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:38:46.491 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:46.500 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:38:46.569 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:46.561+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:46.657 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:46.657 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:38:46.657 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:38:46.657 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:38:46.657 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.657 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-18T04:38:46.657 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:38:46.658 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 72.181081032999998, 2024-09-18T04:38:46.658 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0095781189999999995, 2024-09-18T04:38:46.658 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:46.658 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:46.658 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:46.658 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:38:46.658 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:38:46.658 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 1 2024-09-18T04:38:46.658 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.658 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:46.658 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.658 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:46.658 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:38:46.658 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:46.659 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.659 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.659 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:46.659 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:38:46.659 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:46.659 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.659 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.659 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:46.659 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469862+0000", 2024-09-18T04:38:46.660 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:46.660 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.660 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.660 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:46.660 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469879+0000", 2024-09-18T04:38:46.660 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7431999999999998e-05 2024-09-18T04:38:46.660 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.660 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.660 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:46.660 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469896+0000", 2024-09-18T04:38:46.660 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6742999999999999e-05 2024-09-18T04:38:46.660 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.660 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.660 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:46.661 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469915+0000", 2024-09-18T04:38:46.661 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8991000000000002e-05 2024-09-18T04:38:46.661 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.661 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.661 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:46.661 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470397+0000", 2024-09-18T04:38:46.661 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048167799999999998 2024-09-18T04:38:46.661 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.661 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.661 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-09-18T04:38:46.661 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470412+0000", 2024-09-18T04:38:46.661 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5526e-05 2024-09-18T04:38:46.661 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.662 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.662 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:46.662 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473279+0000", 2024-09-18T04:38:46.662 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0028667469999999998 2024-09-18T04:38:46.662 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.662 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.662 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:46.662 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473441+0000", 2024-09-18T04:38:46.662 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000162082 2024-09-18T04:38:46.662 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.662 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.662 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:38:46.662 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473573+0000", 2024-09-18T04:38:46.662 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000132032 2024-09-18T04:38:46.663 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.663 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.663 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:46.663 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474826+0000", 2024-09-18T04:38:46.663 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001252717 2024-09-18T04:38:46.663 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.663 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.663 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:46.663 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479307+0000", 2024-09-18T04:38:46.663 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044813220000000003 2024-09-18T04:38:46.663 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.663 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.663 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:46.664 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479366+0000", 2024-09-18T04:38:46.664 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9024999999999998e-05 2024-09-18T04:38:46.664 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.664 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.664 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:46.664 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479443+0000", 2024-09-18T04:38:46.664 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6339000000000002e-05 2024-09-18T04:38:46.664 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:46.664 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:46.664 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:46.664 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.664 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.664 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-18T04:38:46.665 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:35.468698+0000", 2024-09-18T04:38:46.665 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 71.182247782000005, 2024-09-18T04:38:46.665 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0039954129999999997, 2024-09-18T04:38:46.665 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:46.665 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:46.665 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:46.665 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:38:46.665 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:38:46.665 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 5 2024-09-18T04:38:46.665 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.665 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:46.665 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.665 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:46.665 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468698+0000", 2024-09-18T04:38:46.666 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:46.666 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.666 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.666 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:46.666 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468698+0000", 2024-09-18T04:38:46.666 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:46.666 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.666 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.666 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:46.666 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468696+0000", 2024-09-18T04:38:46.666 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:46.666 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.666 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.667 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:46.667 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468714+0000", 2024-09-18T04:38:46.667 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8743e-05 2024-09-18T04:38:46.667 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.667 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.667 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:46.667 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468748+0000", 2024-09-18T04:38:46.667 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3339e-05 2024-09-18T04:38:46.667 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.667 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.667 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:46.667 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468771+0000", 2024-09-18T04:38:46.667 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3119999999999999e-05 2024-09-18T04:38:46.667 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.668 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.668 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:46.668 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.468922+0000", 2024-09-18T04:38:46.668 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000151383 2024-09-18T04:38:46.668 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.668 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.668 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:46.668 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.469278+0000", 2024-09-18T04:38:46.668 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00035628599999999998 2024-09-18T04:38:46.668 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.668 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.668 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:46.668 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.469629+0000", 2024-09-18T04:38:46.669 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00035014300000000003 2024-09-18T04:38:46.669 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.669 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.669 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:46.669 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.471620+0000", 2024-09-18T04:38:46.669 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001991278 2024-09-18T04:38:46.669 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.669 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.669 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:46.669 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472440+0000", 2024-09-18T04:38:46.669 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00081998599999999998 2024-09-18T04:38:46.669 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.669 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.670 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:46.670 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472592+0000", 2024-09-18T04:38:46.670 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015227400000000001 2024-09-18T04:38:46.670 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.670 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.670 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:46.670 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:35.472693+0000", 2024-09-18T04:38:46.670 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000101074 2024-09-18T04:38:46.670 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:46.670 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:46.670 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:46.670 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.670 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.671 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-18T04:38:46.671 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:38:46.671 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 70.168072613000007, 2024-09-18T04:38:46.671 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082699560000000002, 2024-09-18T04:38:46.671 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:46.671 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:46.671 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:46.671 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:38:46.671 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:38:46.671 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 7 2024-09-18T04:38:46.671 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.671 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:46.671 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.671 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:46.672 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:38:46.672 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:46.672 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.672 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.672 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:46.672 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:38:46.672 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:46.672 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.672 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.672 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:46.672 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482870+0000", 2024-09-18T04:38:46.672 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999967 2024-09-18T04:38:46.672 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.673 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.673 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:46.673 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482889+0000", 2024-09-18T04:38:46.673 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.965e-05 2024-09-18T04:38:46.673 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.673 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.673 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:46.673 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482938+0000", 2024-09-18T04:38:46.673 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8548999999999999e-05 2024-09-18T04:38:46.673 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.673 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.673 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:46.673 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482963+0000", 2024-09-18T04:38:46.673 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5075999999999998e-05 2024-09-18T04:38:46.674 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.674 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.674 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:46.674 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483121+0000", 2024-09-18T04:38:46.674 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015819400000000001 2024-09-18T04:38:46.674 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.674 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.674 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op must wait for map", 2024-09-18T04:38:46.674 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483138+0000", 2024-09-18T04:38:46.674 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7068999999999999e-05 2024-09-18T04:38:46.674 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.675 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.675 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:46.675 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488666+0000", 2024-09-18T04:38:46.675 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0055282049999999996 2024-09-18T04:38:46.675 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.675 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.675 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:46.675 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488783+0000", 2024-09-18T04:38:46.675 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000116737 2024-09-18T04:38:46.675 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.675 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.675 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:46.676 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488991+0000", 2024-09-18T04:38:46.676 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020781400000000001 2024-09-18T04:38:46.676 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.676 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.676 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:46.676 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490554+0000", 2024-09-18T04:38:46.676 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0015628370000000001 2024-09-18T04:38:46.676 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.676 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.676 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:46.676 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491019+0000", 2024-09-18T04:38:46.676 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000465608 2024-09-18T04:38:46.676 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.677 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.677 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:46.677 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491087+0000", 2024-09-18T04:38:46.677 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7988e-05 2024-09-18T04:38:46.677 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.677 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.677 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:46.677 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491143+0000", 2024-09-18T04:38:46.677 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5538999999999998e-05 2024-09-18T04:38:46.677 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:46.677 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:46.677 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:46.677 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.678 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.678 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:2 2.3 2:d80af3cb:::rbd_id.testrbdnbd17342:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:46.678 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.186139+0000", 2024-09-18T04:38:46.678 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 4.4648063630000001, 2024-09-18T04:38:46.678 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003747532, 2024-09-18T04:38:46.678 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:46.678 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-09-18T04:38:46.678 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:46.678 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:38:46.678 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:38:46.678 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 2 2024-09-18T04:38:46.678 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.678 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:46.678 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.678 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:46.679 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.186139+0000", 2024-09-18T04:38:46.679 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:46.679 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.679 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.679 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:46.679 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.186139+0000", 2024-09-18T04:38:46.679 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:46.679 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.679 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.679 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:46.679 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.186137+0000", 2024-09-18T04:38:46.679 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:46.679 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.679 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.680 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:46.680 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.186150+0000", 2024-09-18T04:38:46.680 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2913e-05 2024-09-18T04:38:46.680 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.680 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.680 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:46.680 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.186172+0000", 2024-09-18T04:38:46.680 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2653000000000001e-05 2024-09-18T04:38:46.680 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.680 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.680 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:46.680 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.186189+0000", 2024-09-18T04:38:46.680 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6840000000000001e-05 2024-09-18T04:38:46.681 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.681 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.681 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:46.681 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.186320+0000", 2024-09-18T04:38:46.681 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013040099999999999 2024-09-18T04:38:46.681 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.681 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.681 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op must wait for map", 2024-09-18T04:38:46.681 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.186330+0000", 2024-09-18T04:38:46.681 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0033e-05 2024-09-18T04:38:46.681 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.681 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.681 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:46.681 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.189708+0000", 2024-09-18T04:38:46.682 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033780609999999999 2024-09-18T04:38:46.682 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.682 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.682 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:46.682 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.189887+0000", 2024-09-18T04:38:46.682 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017895399999999999 2024-09-18T04:38:46.682 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:46.682 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:46.682 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:46.682 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.682 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.682 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:34 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:46.682 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.261758+0000", 2024-09-18T04:38:46.682 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 4.3891874619999998, 2024-09-18T04:38:46.683 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.042024191000000002, 2024-09-18T04:38:46.683 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:46.683 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:46.683 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:46.683 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:38:46.683 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:38:46.683 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 34 2024-09-18T04:38:46.683 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.683 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:46.683 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.683 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:46.683 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.261758+0000", 2024-09-18T04:38:46.683 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:46.683 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.683 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.684 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:46.684 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.261758+0000", 2024-09-18T04:38:46.684 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:46.684 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.684 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.684 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:46.684 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.261756+0000", 2024-09-18T04:38:46.684 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:46.684 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.684 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.684 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:46.684 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.270785+0000", 2024-09-18T04:38:46.684 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0090292340000000006 2024-09-18T04:38:46.685 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.685 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.685 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:46.685 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.270890+0000", 2024-09-18T04:38:46.685 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000104541 2024-09-18T04:38:46.685 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.685 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.685 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:46.685 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.270991+0000", 2024-09-18T04:38:46.685 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000100778 2024-09-18T04:38:46.685 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.685 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.685 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:46.685 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.271157+0000", 2024-09-18T04:38:46.685 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016653999999999999 2024-09-18T04:38:46.686 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.686 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.686 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:46.686 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.271591+0000", 2024-09-18T04:38:46.686 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00043409500000000001 2024-09-18T04:38:46.686 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.686 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.686 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:38:46.686 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272017+0000", 2024-09-18T04:38:46.686 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000425769 2024-09-18T04:38:46.686 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.686 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.686 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:46.687 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.294246+0000", 2024-09-18T04:38:46.687 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.022229237999999998 2024-09-18T04:38:46.687 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.687 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.687 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:46.687 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303571+0000", 2024-09-18T04:38:46.687 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0093250239999999995 2024-09-18T04:38:46.687 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.687 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.687 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:46.687 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303705+0000", 2024-09-18T04:38:46.687 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000133831 2024-09-18T04:38:46.687 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.687 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.688 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:46.688 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303782+0000", 2024-09-18T04:38:46.688 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7156000000000005e-05 2024-09-18T04:38:46.688 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:46.688 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:46.688 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:46.688 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.688 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.688 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:36 2.7 e51/32 2:fe1b02ad:::rbd_data.125fc09df6d1.0000000000000002:head v 51'1, mlcod=0'0)", 2024-09-18T04:38:46.688 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.273441+0000", 2024-09-18T04:38:46.688 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 4.3775044809999999, 2024-09-18T04:38:46.688 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.032400768000000003, 2024-09-18T04:38:46.688 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:46.689 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:46.689 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:46.689 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.689 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:46.689 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273441+0000", 2024-09-18T04:38:46.689 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:46.689 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.689 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.689 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:46.689 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273441+0000", 2024-09-18T04:38:46.690 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:46.690 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.690 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.690 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:46.690 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273440+0000", 2024-09-18T04:38:46.690 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:46.690 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.690 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.690 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:46.690 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278723+0000", 2024-09-18T04:38:46.690 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0052829909999999999 2024-09-18T04:38:46.690 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.691 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.691 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:46.691 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278752+0000", 2024-09-18T04:38:46.691 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9178000000000001e-05 2024-09-18T04:38:46.691 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.691 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.691 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:46.691 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278761+0000", 2024-09-18T04:38:46.691 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7900000000000005e-06 2024-09-18T04:38:46.691 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.691 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.691 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:46.691 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278864+0000", 2024-09-18T04:38:46.691 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000102699 2024-09-18T04:38:46.692 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.692 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.692 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:46.692 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278900+0000", 2024-09-18T04:38:46.692 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6001999999999999e-05 2024-09-18T04:38:46.692 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.692 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.692 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:46.692 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.305793+0000", 2024-09-18T04:38:46.692 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.026893344 2024-09-18T04:38:46.692 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.692 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.692 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:46.692 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.305842+0000", 2024-09-18T04:38:46.693 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8720000000000001e-05 2024-09-18T04:38:46.693 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:46.693 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:46.693 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:46.693 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.693 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.693 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:39 2.7 e51/32 2:e9c909c5:::rbd_data.125fc09df6d1.0000000000000005:head v 51'2, mlcod=0'0)", 2024-09-18T04:38:46.693 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.283175+0000", 2024-09-18T04:38:46.693 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 4.367770792, 2024-09-18T04:38:46.693 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.022864102000000001, 2024-09-18T04:38:46.693 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:46.693 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:46.693 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:46.694 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.694 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:46.694 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283175+0000", 2024-09-18T04:38:46.694 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:46.694 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.694 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.694 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:46.694 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283175+0000", 2024-09-18T04:38:46.694 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:46.694 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.694 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.694 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:46.694 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283173+0000", 2024-09-18T04:38:46.694 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:46.695 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.695 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.695 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:46.695 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.285951+0000", 2024-09-18T04:38:46.695 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0027784540000000001 2024-09-18T04:38:46.695 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.695 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.695 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:46.695 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.285970+0000", 2024-09-18T04:38:46.695 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8855000000000002e-05 2024-09-18T04:38:46.695 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.695 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.695 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:46.696 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.285984+0000", 2024-09-18T04:38:46.696 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4252e-05 2024-09-18T04:38:46.696 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.696 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.696 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:46.696 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.286041+0000", 2024-09-18T04:38:46.696 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6431999999999998e-05 2024-09-18T04:38:46.696 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.696 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.696 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:46.696 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.286086+0000", 2024-09-18T04:38:46.696 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5423000000000002e-05 2024-09-18T04:38:46.696 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.697 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.697 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:46.697 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.305998+0000", 2024-09-18T04:38:46.697 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019911697999999999 2024-09-18T04:38:46.697 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.697 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.697 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:46.697 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306039+0000", 2024-09-18T04:38:46.697 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0973999999999998e-05 2024-09-18T04:38:46.697 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:46.697 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:46.697 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:46.697 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.697 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.698 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:50 2.3 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:46.698 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.312563+0000", 2024-09-18T04:38:46.698 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 4.3383827650000004, 2024-09-18T04:38:46.698 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.032636001999999997, 2024-09-18T04:38:46.698 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:46.698 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:46.698 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:46.698 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:38:46.698 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:38:46.698 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 50 2024-09-18T04:38:46.698 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.698 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:46.698 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.698 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:46.699 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.312563+0000", 2024-09-18T04:38:46.699 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:46.699 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.699 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.699 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:46.699 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.312563+0000", 2024-09-18T04:38:46.699 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:46.699 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.699 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.699 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:46.699 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.312561+0000", 2024-09-18T04:38:46.699 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:46.699 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.700 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.700 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:46.700 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321709+0000", 2024-09-18T04:38:46.700 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0091486650000000003 2024-09-18T04:38:46.700 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.700 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.700 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:46.700 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321739+0000", 2024-09-18T04:38:46.700 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9666e-05 2024-09-18T04:38:46.700 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.700 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.700 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:46.700 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321751+0000", 2024-09-18T04:38:46.701 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2086000000000001e-05 2024-09-18T04:38:46.701 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.701 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.701 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:46.701 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321833+0000", 2024-09-18T04:38:46.701 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1842999999999996e-05 2024-09-18T04:38:46.701 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.701 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.701 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:46.701 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321981+0000", 2024-09-18T04:38:46.701 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000147441 2024-09-18T04:38:46.701 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.701 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.702 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:38:46.702 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.322109+0000", 2024-09-18T04:38:46.702 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012851200000000001 2024-09-18T04:38:46.702 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.702 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.702 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:46.702 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.332829+0000", 2024-09-18T04:38:46.702 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010720225999999999 2024-09-18T04:38:46.702 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.702 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.702 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:46.702 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.345039+0000", 2024-09-18T04:38:46.702 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012209831000000001 2024-09-18T04:38:46.703 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.703 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.703 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:46.703 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.345120+0000", 2024-09-18T04:38:46.703 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1145999999999999e-05 2024-09-18T04:38:46.703 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.703 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.703 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:46.703 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.345199+0000", 2024-09-18T04:38:46.703 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8546000000000003e-05 2024-09-18T04:38:46.703 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:46.703 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:46.703 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:46.703 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.704 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.704 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:28 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:46.704 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.515881+0000", 2024-09-18T04:38:46.704 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 4.1350642889999998, 2024-09-18T04:38:46.704 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0059728519999999998, 2024-09-18T04:38:46.704 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:46.704 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:46.704 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:46.704 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:38:46.704 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:46.704 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 28 2024-09-18T04:38:46.704 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.704 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:46.705 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.705 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:46.705 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.515881+0000", 2024-09-18T04:38:46.705 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:46.705 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.705 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.705 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:46.705 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.515881+0000", 2024-09-18T04:38:46.705 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:46.705 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.705 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.705 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:46.705 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.515879+0000", 2024-09-18T04:38:46.706 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:46.706 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.706 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.706 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:46.706 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.515896+0000", 2024-09-18T04:38:46.706 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7425999999999999e-05 2024-09-18T04:38:46.706 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.706 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.706 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:46.706 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.515925+0000", 2024-09-18T04:38:46.706 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8277e-05 2024-09-18T04:38:46.706 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.706 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.707 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:46.707 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.515949+0000", 2024-09-18T04:38:46.707 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3759000000000001e-05 2024-09-18T04:38:46.707 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.707 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.707 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:46.707 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516091+0000", 2024-09-18T04:38:46.707 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000142967 2024-09-18T04:38:46.707 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.707 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.707 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:46.707 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516392+0000", 2024-09-18T04:38:46.708 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00030068100000000002 2024-09-18T04:38:46.708 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.708 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.708 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:38:46.708 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.517345+0000", 2024-09-18T04:38:46.708 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00095256500000000005 2024-09-18T04:38:46.708 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.708 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.708 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:46.708 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.521517+0000", 2024-09-18T04:38:46.708 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0041722770000000003 2024-09-18T04:38:46.708 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.709 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.709 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:46.709 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.521621+0000", 2024-09-18T04:38:46.709 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000104374 2024-09-18T04:38:46.709 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.709 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.709 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:46.709 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.521788+0000", 2024-09-18T04:38:46.709 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016654699999999999 2024-09-18T04:38:46.709 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.709 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.709 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:46.709 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.521854+0000", 2024-09-18T04:38:46.710 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6168999999999996e-05 2024-09-18T04:38:46.710 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:46.710 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:46.710 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:46.710 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.710 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.710 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:29 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:46.710 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.515979+0000", 2024-09-18T04:38:46.710 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 4.1349662409999999, 2024-09-18T04:38:46.710 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0062405730000000001, 2024-09-18T04:38:46.710 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:46.710 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:46.710 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:46.710 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:38:46.711 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:46.711 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 29 2024-09-18T04:38:46.711 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.711 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:46.711 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.711 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:46.711 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.515979+0000", 2024-09-18T04:38:46.711 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:46.711 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.711 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.711 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:46.711 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.515979+0000", 2024-09-18T04:38:46.711 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:46.711 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.712 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.712 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:46.712 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.515978+0000", 2024-09-18T04:38:46.712 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:46.712 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.712 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.712 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:46.712 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.515987+0000", 2024-09-18T04:38:46.712 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6000000000000007e-06 2024-09-18T04:38:46.712 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.712 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.712 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:46.712 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516004+0000", 2024-09-18T04:38:46.713 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6826000000000001e-05 2024-09-18T04:38:46.713 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.713 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.713 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:46.713 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516044+0000", 2024-09-18T04:38:46.713 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0416000000000001e-05 2024-09-18T04:38:46.713 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.713 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.713 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:46.713 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.518153+0000", 2024-09-18T04:38:46.713 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0021086630000000002 2024-09-18T04:38:46.713 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.713 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.713 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:46.714 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.518338+0000", 2024-09-18T04:38:46.714 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00018579299999999999 2024-09-18T04:38:46.714 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.714 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.714 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:38:46.714 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.518970+0000", 2024-09-18T04:38:46.714 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00063170800000000003 2024-09-18T04:38:46.714 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.714 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.714 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:46.714 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.521885+0000", 2024-09-18T04:38:46.714 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0029147940000000001 2024-09-18T04:38:46.714 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.714 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.715 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:46.715 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.522007+0000", 2024-09-18T04:38:46.715 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012216399999999999 2024-09-18T04:38:46.715 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.715 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.715 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:46.715 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.522141+0000", 2024-09-18T04:38:46.715 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013367399999999999 2024-09-18T04:38:46.715 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.715 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.715 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:46.715 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.522220+0000", 2024-09-18T04:38:46.715 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9138e-05 2024-09-18T04:38:46.715 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:46.716 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:46.716 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:46.716 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.716 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.716 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:30 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:46.716 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.516122+0000", 2024-09-18T04:38:46.716 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 4.1348236030000001, 2024-09-18T04:38:46.716 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0091865030000000004, 2024-09-18T04:38:46.716 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:46.716 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:46.716 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:46.716 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:38:46.716 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:46.716 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 30 2024-09-18T04:38:46.717 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.717 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:46.717 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.717 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:46.717 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516122+0000", 2024-09-18T04:38:46.717 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:46.717 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.717 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.717 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:46.717 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516122+0000", 2024-09-18T04:38:46.717 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:46.717 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.717 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.717 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:46.718 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516120+0000", 2024-09-18T04:38:46.718 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:46.718 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.718 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.718 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:46.718 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516129+0000", 2024-09-18T04:38:46.718 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.85e-06 2024-09-18T04:38:46.718 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.718 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.718 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:46.718 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516155+0000", 2024-09-18T04:38:46.718 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5882000000000001e-05 2024-09-18T04:38:46.718 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.719 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.719 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:46.719 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516175+0000", 2024-09-18T04:38:46.719 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.022e-05 2024-09-18T04:38:46.719 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.719 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.719 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:46.719 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519780+0000", 2024-09-18T04:38:46.719 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003604187 2024-09-18T04:38:46.719 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.719 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.719 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:46.719 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519964+0000", 2024-09-18T04:38:46.719 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00018391999999999999 2024-09-18T04:38:46.720 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.720 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.720 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:38:46.720 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.520794+0000", 2024-09-18T04:38:46.720 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00083078100000000005 2024-09-18T04:38:46.720 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.720 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.720 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:46.720 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.524129+0000", 2024-09-18T04:38:46.720 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033342419999999998 2024-09-18T04:38:46.720 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.720 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.720 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:46.721 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525239+0000", 2024-09-18T04:38:46.721 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001110778 2024-09-18T04:38:46.721 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.721 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.721 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:46.721 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525277+0000", 2024-09-18T04:38:46.721 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7596999999999999e-05 2024-09-18T04:38:46.721 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.721 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.721 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:46.721 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525308+0000", 2024-09-18T04:38:46.721 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.1535999999999998e-05 2024-09-18T04:38:46.721 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:46.721 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:46.722 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:46.722 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.722 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.722 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:32 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:46.722 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.519217+0000", 2024-09-18T04:38:46.722 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 4.1317282830000002, 2024-09-18T04:38:46.722 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082266370000000002, 2024-09-18T04:38:46.722 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:46.722 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:46.722 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:46.722 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:38:46.722 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:46.722 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 32 2024-09-18T04:38:46.722 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.723 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:46.723 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.723 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:46.723 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519217+0000", 2024-09-18T04:38:46.723 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:46.723 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.723 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.723 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:46.723 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519217+0000", 2024-09-18T04:38:46.723 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:46.723 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.723 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.723 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:46.723 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519215+0000", 2024-09-18T04:38:46.724 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:46.724 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.724 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.724 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:46.724 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519229+0000", 2024-09-18T04:38:46.724 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3329999999999999e-05 2024-09-18T04:38:46.724 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.724 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.724 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:46.724 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519268+0000", 2024-09-18T04:38:46.724 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9533000000000002e-05 2024-09-18T04:38:46.724 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.724 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.725 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:46.725 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519285+0000", 2024-09-18T04:38:46.725 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6446000000000001e-05 2024-09-18T04:38:46.725 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.725 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.725 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:46.725 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.522358+0000", 2024-09-18T04:38:46.725 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030734310000000002 2024-09-18T04:38:46.725 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.725 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.725 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:46.725 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.522551+0000", 2024-09-18T04:38:46.725 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019317600000000001 2024-09-18T04:38:46.726 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.726 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.726 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:38:46.726 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.523262+0000", 2024-09-18T04:38:46.726 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00071069999999999998 2024-09-18T04:38:46.726 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.726 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.726 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:46.726 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525318+0000", 2024-09-18T04:38:46.726 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0020561469999999999 2024-09-18T04:38:46.726 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.726 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.726 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:46.726 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.527376+0000", 2024-09-18T04:38:46.726 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0020584969999999998 2024-09-18T04:38:46.727 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.727 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.727 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:46.727 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.527416+0000", 2024-09-18T04:38:46.727 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9054000000000003e-05 2024-09-18T04:38:46.727 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.727 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.727 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:46.727 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.527444+0000", 2024-09-18T04:38:46.727 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8347000000000002e-05 2024-09-18T04:38:46.727 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:46.727 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:46.727 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:46.728 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.728 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.728 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:35 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:46.728 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.523233+0000", 2024-09-18T04:38:46.728 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 4.1277122540000004, 2024-09-18T04:38:46.728 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0048600689999999998, 2024-09-18T04:38:46.728 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:46.728 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:46.728 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:46.728 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:38:46.728 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:46.728 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 35 2024-09-18T04:38:46.728 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.729 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:46.729 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.729 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:46.729 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.523233+0000", 2024-09-18T04:38:46.729 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:46.729 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.729 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.729 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:46.729 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.523233+0000", 2024-09-18T04:38:46.729 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:46.729 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.729 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.729 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:46.729 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.523231+0000", 2024-09-18T04:38:46.730 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:46.730 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.730 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.730 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:46.730 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.523245+0000", 2024-09-18T04:38:46.730 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4027e-05 2024-09-18T04:38:46.730 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.730 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.730 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:46.730 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.523271+0000", 2024-09-18T04:38:46.730 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6029000000000001e-05 2024-09-18T04:38:46.730 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.730 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.730 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:46.731 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.523290+0000", 2024-09-18T04:38:46.731 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9015999999999999e-05 2024-09-18T04:38:46.731 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.731 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.731 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:46.731 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.524183+0000", 2024-09-18T04:38:46.731 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00089286300000000003 2024-09-18T04:38:46.731 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.731 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.731 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:46.731 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.524343+0000", 2024-09-18T04:38:46.731 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000160524 2024-09-18T04:38:46.731 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.731 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.732 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:38:46.732 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.524869+0000", 2024-09-18T04:38:46.732 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00052553099999999998 2024-09-18T04:38:46.732 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.732 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.732 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:46.732 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525894+0000", 2024-09-18T04:38:46.732 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001025296 2024-09-18T04:38:46.732 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.732 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.732 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:46.732 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.528018+0000", 2024-09-18T04:38:46.732 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0021235889999999999 2024-09-18T04:38:46.733 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.733 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.733 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:46.733 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.528064+0000", 2024-09-18T04:38:46.733 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5849000000000001e-05 2024-09-18T04:38:46.733 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.733 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.733 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:46.733 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.528093+0000", 2024-09-18T04:38:46.733 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9598e-05 2024-09-18T04:38:46.733 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:46.733 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:46.733 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:46.734 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.734 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.734 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:42 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:46.734 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.551160+0000", 2024-09-18T04:38:46.734 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 4.0997858909999998, 2024-09-18T04:38:46.734 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010692757000000001, 2024-09-18T04:38:46.734 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:46.734 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:46.734 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:46.734 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:38:46.734 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:46.734 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 42 2024-09-18T04:38:46.734 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.734 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:46.735 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.735 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:46.735 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551160+0000", 2024-09-18T04:38:46.735 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:46.735 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.735 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.735 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:46.735 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551160+0000", 2024-09-18T04:38:46.735 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:46.735 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.735 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.735 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:46.735 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551159+0000", 2024-09-18T04:38:46.735 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:46.736 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.736 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.736 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:46.736 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551166+0000", 2024-09-18T04:38:46.736 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1029999999999996e-06 2024-09-18T04:38:46.736 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.736 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.736 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:46.736 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551177+0000", 2024-09-18T04:38:46.736 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1384e-05 2024-09-18T04:38:46.736 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.736 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.736 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:46.736 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551185+0000", 2024-09-18T04:38:46.737 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1990000000000006e-06 2024-09-18T04:38:46.737 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.737 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.737 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:46.737 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552021+0000", 2024-09-18T04:38:46.737 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00083574400000000003 2024-09-18T04:38:46.737 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.737 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.737 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:46.737 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552339+0000", 2024-09-18T04:38:46.737 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031771399999999997 2024-09-18T04:38:46.737 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.737 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.737 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:38:46.738 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.553140+0000", 2024-09-18T04:38:46.738 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00080088100000000003 2024-09-18T04:38:46.738 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.738 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.738 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:46.738 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.554585+0000", 2024-09-18T04:38:46.738 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001445801 2024-09-18T04:38:46.738 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.738 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.738 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:46.738 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.561711+0000", 2024-09-18T04:38:46.738 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071260109999999998 2024-09-18T04:38:46.738 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.738 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.739 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:46.739 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.561790+0000", 2024-09-18T04:38:46.739 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8967000000000005e-05 2024-09-18T04:38:46.739 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.739 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.739 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:46.739 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.561852+0000", 2024-09-18T04:38:46.739 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2045999999999995e-05 2024-09-18T04:38:46.739 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:46.739 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:46.739 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:46.739 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.739 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.739 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:44 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:46.739 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.558099+0000", 2024-09-18T04:38:46.740 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 4.0928467179999997, 2024-09-18T04:38:46.740 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0042364710000000003, 2024-09-18T04:38:46.740 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:46.740 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:46.740 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:46.740 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:38:46.740 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:46.740 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 44 2024-09-18T04:38:46.740 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.740 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:46.740 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.740 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:46.740 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.558099+0000", 2024-09-18T04:38:46.741 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:46.741 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.741 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.741 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:46.741 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.558099+0000", 2024-09-18T04:38:46.741 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:46.741 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.741 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.741 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:46.741 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.558096+0000", 2024-09-18T04:38:46.741 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:46.741 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.741 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.741 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:46.742 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.558111+0000", 2024-09-18T04:38:46.742 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4369000000000001e-05 2024-09-18T04:38:46.742 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.742 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.742 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:46.742 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.558140+0000", 2024-09-18T04:38:46.742 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9600000000000001e-05 2024-09-18T04:38:46.742 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.742 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.742 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:46.742 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.558155+0000", 2024-09-18T04:38:46.742 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5013e-05 2024-09-18T04:38:46.742 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.742 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.743 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:46.743 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.558230+0000", 2024-09-18T04:38:46.743 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4843000000000002e-05 2024-09-18T04:38:46.743 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.743 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.743 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:46.743 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.558358+0000", 2024-09-18T04:38:46.743 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000127787 2024-09-18T04:38:46.743 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.743 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.743 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:38:46.743 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.558831+0000", 2024-09-18T04:38:46.743 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00047311400000000001 2024-09-18T04:38:46.744 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.744 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.744 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:46.744 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.560221+0000", 2024-09-18T04:38:46.744 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001389569 2024-09-18T04:38:46.744 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.744 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.744 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:46.744 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.562107+0000", 2024-09-18T04:38:46.744 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0018861920000000001 2024-09-18T04:38:46.744 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.744 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.744 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:46.744 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.562234+0000", 2024-09-18T04:38:46.745 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000127131 2024-09-18T04:38:46.745 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.745 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.745 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:46.745 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.562335+0000", 2024-09-18T04:38:46.745 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000101302 2024-09-18T04:38:46.745 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:46.745 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:46.745 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:46.745 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.745 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.745 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:46 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:46.745 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.562373+0000", 2024-09-18T04:38:46.745 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 4.0885727169999999, 2024-09-18T04:38:46.746 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0055256250000000002, 2024-09-18T04:38:46.746 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:46.746 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:46.746 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:46.746 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:38:46.746 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:46.746 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 46 2024-09-18T04:38:46.746 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.746 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:46.746 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.746 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:46.746 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.562373+0000", 2024-09-18T04:38:46.746 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:46.746 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.747 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.747 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:46.747 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.562373+0000", 2024-09-18T04:38:46.747 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:46.747 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.747 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.747 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:46.747 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.562370+0000", 2024-09-18T04:38:46.747 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999967 2024-09-18T04:38:46.747 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.747 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.747 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:46.747 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.562389+0000", 2024-09-18T04:38:46.747 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8994000000000001e-05 2024-09-18T04:38:46.748 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.748 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.748 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:46.748 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.562431+0000", 2024-09-18T04:38:46.748 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.2035e-05 2024-09-18T04:38:46.748 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.748 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.748 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:46.748 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.562452+0000", 2024-09-18T04:38:46.748 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1214000000000001e-05 2024-09-18T04:38:46.748 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.748 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.748 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:46.748 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.562522+0000", 2024-09-18T04:38:46.749 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0463000000000006e-05 2024-09-18T04:38:46.749 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.749 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.749 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:46.749 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.562617+0000", 2024-09-18T04:38:46.749 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4179999999999996e-05 2024-09-18T04:38:46.749 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.749 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.749 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:38:46.749 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.562906+0000", 2024-09-18T04:38:46.749 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000289525 2024-09-18T04:38:46.749 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.749 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.749 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:46.750 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.565351+0000", 2024-09-18T04:38:46.750 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0024444340000000001 2024-09-18T04:38:46.750 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.750 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.750 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:46.750 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.567762+0000", 2024-09-18T04:38:46.750 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0024110009999999999 2024-09-18T04:38:46.750 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.750 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.750 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:46.750 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.567832+0000", 2024-09-18T04:38:46.750 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0073999999999993e-05 2024-09-18T04:38:46.750 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.750 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.751 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:46.751 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.567898+0000", 2024-09-18T04:38:46.751 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6824999999999998e-05 2024-09-18T04:38:46.751 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:46.751 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:46.751 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:46.751 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.751 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.751 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:47 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:46.751 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.562481+0000", 2024-09-18T04:38:46.751 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 4.0884646150000004, 2024-09-18T04:38:46.751 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.005709586, 2024-09-18T04:38:46.751 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:46.751 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:46.752 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:46.752 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:38:46.752 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:46.752 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 47 2024-09-18T04:38:46.752 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.752 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:46.752 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.752 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:46.752 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.562481+0000", 2024-09-18T04:38:46.752 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:46.752 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.752 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.752 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:46.752 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.562481+0000", 2024-09-18T04:38:46.753 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:46.753 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.753 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.753 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:46.753 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.562480+0000", 2024-09-18T04:38:46.753 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:46.753 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.753 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.753 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:46.753 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.562488+0000", 2024-09-18T04:38:46.753 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6599999999999995e-06 2024-09-18T04:38:46.753 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.753 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.753 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:46.754 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.562498+0000", 2024-09-18T04:38:46.754 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.065e-05 2024-09-18T04:38:46.754 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.754 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.754 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:46.754 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.562506+0000", 2024-09-18T04:38:46.754 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.396e-06 2024-09-18T04:38:46.754 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.754 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.754 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:46.754 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563378+0000", 2024-09-18T04:38:46.754 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00087218199999999999 2024-09-18T04:38:46.754 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.754 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.755 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:46.755 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563589+0000", 2024-09-18T04:38:46.755 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000210919 2024-09-18T04:38:46.755 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.755 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.755 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:38:46.755 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.564485+0000", 2024-09-18T04:38:46.755 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00089623299999999997 2024-09-18T04:38:46.755 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.755 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.755 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:46.755 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.566778+0000", 2024-09-18T04:38:46.755 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.002293444 2024-09-18T04:38:46.755 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.755 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.756 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:46.756 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.568109+0000", 2024-09-18T04:38:46.756 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001330251 2024-09-18T04:38:46.756 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.756 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.756 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:46.756 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.568163+0000", 2024-09-18T04:38:46.756 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4240000000000002e-05 2024-09-18T04:38:46.756 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.756 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.756 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:46.756 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.568191+0000", 2024-09-18T04:38:46.756 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7591000000000001e-05 2024-09-18T04:38:46.757 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:46.757 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:46.757 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:46.757 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.757 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.757 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:52 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:46.757 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.577756+0000", 2024-09-18T04:38:46.757 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 4.0731900059999999, 2024-09-18T04:38:46.757 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0065803340000000002, 2024-09-18T04:38:46.757 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:46.757 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:46.757 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:46.757 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:38:46.757 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:46.757 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 52 2024-09-18T04:38:46.758 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.758 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:46.758 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.758 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:46.758 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.577756+0000", 2024-09-18T04:38:46.758 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:46.758 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.758 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.758 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:46.758 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.577756+0000", 2024-09-18T04:38:46.758 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:46.758 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.759 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.759 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:46.759 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.577754+0000", 2024-09-18T04:38:46.759 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:46.759 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.759 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.759 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:46.759 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.577762+0000", 2024-09-18T04:38:46.759 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4179999999999997e-06 2024-09-18T04:38:46.759 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.759 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.759 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:46.759 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.577783+0000", 2024-09-18T04:38:46.759 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0105000000000001e-05 2024-09-18T04:38:46.760 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.760 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.760 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:46.760 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.577791+0000", 2024-09-18T04:38:46.760 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2430000000000005e-06 2024-09-18T04:38:46.760 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.760 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.760 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:46.760 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.579272+0000", 2024-09-18T04:38:46.760 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0014808759999999999 2024-09-18T04:38:46.760 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.760 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.760 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:46.761 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.579515+0000", 2024-09-18T04:38:46.761 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000243098 2024-09-18T04:38:46.761 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.761 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.761 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:38:46.761 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.579992+0000", 2024-09-18T04:38:46.761 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00047668899999999998 2024-09-18T04:38:46.761 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.761 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.761 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:46.761 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.581350+0000", 2024-09-18T04:38:46.761 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0013580160000000001 2024-09-18T04:38:46.761 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.761 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.762 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:46.762 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.584240+0000", 2024-09-18T04:38:46.762 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0028906520000000001 2024-09-18T04:38:46.762 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.762 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.762 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:46.762 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.584298+0000", 2024-09-18T04:38:46.762 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7958000000000003e-05 2024-09-18T04:38:46.762 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.762 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.762 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:46.762 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.584336+0000", 2024-09-18T04:38:46.762 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7741000000000003e-05 2024-09-18T04:38:46.762 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:46.762 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:46.763 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:46.763 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.763 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.763 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:55 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:46.763 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.587954+0000", 2024-09-18T04:38:46.763 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 4.0629916699999997, 2024-09-18T04:38:46.763 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0051545690000000003, 2024-09-18T04:38:46.763 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:46.763 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:46.763 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:46.763 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:38:46.763 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:46.763 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 55 2024-09-18T04:38:46.763 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.764 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:46.764 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.764 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:46.764 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.587954+0000", 2024-09-18T04:38:46.764 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:46.764 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.764 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.764 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:46.764 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.587954+0000", 2024-09-18T04:38:46.764 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:46.764 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.764 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.764 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:46.764 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.587952+0000", 2024-09-18T04:38:46.765 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:46.765 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.765 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.765 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:46.765 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.587964+0000", 2024-09-18T04:38:46.765 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.15e-05 2024-09-18T04:38:46.765 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.765 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.765 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:46.765 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.587984+0000", 2024-09-18T04:38:46.765 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0409999999999999e-05 2024-09-18T04:38:46.765 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.765 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.765 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:46.766 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.587999+0000", 2024-09-18T04:38:46.766 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4448e-05 2024-09-18T04:38:46.766 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.766 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.766 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:46.766 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.588060+0000", 2024-09-18T04:38:46.766 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.1569000000000006e-05 2024-09-18T04:38:46.766 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.766 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.766 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:46.766 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.588176+0000", 2024-09-18T04:38:46.766 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000115605 2024-09-18T04:38:46.766 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.766 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.767 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:38:46.767 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.588538+0000", 2024-09-18T04:38:46.767 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00036184299999999998 2024-09-18T04:38:46.767 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.767 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.767 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:46.767 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.590375+0000", 2024-09-18T04:38:46.767 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0018372760000000001 2024-09-18T04:38:46.767 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.767 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.767 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:46.767 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.592985+0000", 2024-09-18T04:38:46.767 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.002609793 2024-09-18T04:38:46.767 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.768 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.768 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:46.768 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593033+0000", 2024-09-18T04:38:46.768 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7942999999999998e-05 2024-09-18T04:38:46.768 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.768 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.768 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:46.768 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593108+0000", 2024-09-18T04:38:46.768 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.5821000000000005e-05 2024-09-18T04:38:46.768 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:46.768 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:46.768 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:46.768 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.768 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.769 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:58 2.7 e51/32 2:fc7c6cd3:ns::rbd_data.133e813ab5bc.000000000000000e:head v 51'7, mlcod=51'5)", 2024-09-18T04:38:46.769 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.605333+0000", 2024-09-18T04:38:46.769 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 4.0456126860000001, 2024-09-18T04:38:46.769 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020085902999999999, 2024-09-18T04:38:46.769 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:46.769 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:46.769 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:46.769 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.769 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:46.769 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605333+0000", 2024-09-18T04:38:46.769 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:46.769 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.769 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.769 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:46.770 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605333+0000", 2024-09-18T04:38:46.770 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:46.770 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.770 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.770 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:46.770 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605332+0000", 2024-09-18T04:38:46.770 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:46.770 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.770 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.770 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:46.770 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608667+0000", 2024-09-18T04:38:46.770 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033350300000000001 2024-09-18T04:38:46.770 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.770 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.770 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:46.771 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608706+0000", 2024-09-18T04:38:46.771 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8951e-05 2024-09-18T04:38:46.771 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.771 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.771 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:46.771 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608725+0000", 2024-09-18T04:38:46.771 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9675999999999999e-05 2024-09-18T04:38:46.771 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.771 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.771 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:46.771 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608798+0000", 2024-09-18T04:38:46.771 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3135999999999999e-05 2024-09-18T04:38:46.771 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.771 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.772 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:46.772 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608824+0000", 2024-09-18T04:38:46.772 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5284000000000001e-05 2024-09-18T04:38:46.772 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.772 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.772 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:46.772 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625370+0000", 2024-09-18T04:38:46.772 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016545975000000001 2024-09-18T04:38:46.772 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:46.772 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:46.772 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:46.772 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625419+0000", 2024-09-18T04:38:46.772 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9076999999999998e-05 2024-09-18T04:38:46.772 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:46.772 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:46.773 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:46.773 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:46.773 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:46.773 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:46.774 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:38:46.775 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:46.770+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:46.876 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:46.870+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:46.892 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:46.893 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:46.893 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:38:46.893 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:46.903 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:38:46.966 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-18T04:38:46.976 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:46.970+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:46.982 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false rbd-nbd 2024-09-18T04:38:46.982 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd-nbd 2024-09-18T04:38:47.016 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:47.016 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:47.016 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:38:47.016 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:38:47.016 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:47.026 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:38:47.030 INFO:tasks.workunit.client.0.smithi115.stderr:rbd-nbd: must specify command 2024-09-18T04:38:47.031 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:38:47.032 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false rbd-nbd INVALIDCMD 2024-09-18T04:38:47.032 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd-nbd INVALIDCMD 2024-09-18T04:38:47.062 INFO:tasks.workunit.client.0.smithi115.stderr:rbd-nbd: unknown command: INVALIDCMD 2024-09-18T04:38:47.064 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:38:47.065 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:38:47.067 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -ne 0 ']' 2024-09-18T04:38:47.067 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false rbd device --device-type nbd map testrbdnbd17342 2024-09-18T04:38:47.067 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd device --device-type nbd map testrbdnbd17342 2024-09-18T04:38:47.078 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:47.069+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:47.179 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:47.174+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:47.185 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:47.185 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:38:47.185 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:38:47.185 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:38:47.185 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.185 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:30 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'16, mlcod=15'14)", 2024-09-18T04:38:47.186 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.486330+0000", 2024-09-18T04:38:47.186 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 79.694197996, 2024-09-18T04:38:47.186 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026383700000000001, 2024-09-18T04:38:47.186 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:47.186 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:47.186 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:47.186 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.186 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:47.186 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486330+0000", 2024-09-18T04:38:47.186 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.186 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.186 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:47.187 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486330+0000", 2024-09-18T04:38:47.187 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.187 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.187 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:47.187 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486328+0000", 2024-09-18T04:38:47.187 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:47.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.187 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.187 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:47.187 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486342+0000", 2024-09-18T04:38:47.187 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3783e-05 2024-09-18T04:38:47.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.187 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.188 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:47.188 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486373+0000", 2024-09-18T04:38:47.188 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1362999999999998e-05 2024-09-18T04:38:47.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.188 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.188 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:47.188 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486387+0000", 2024-09-18T04:38:47.188 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3219e-05 2024-09-18T04:38:47.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.188 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.188 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:47.188 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486489+0000", 2024-09-18T04:38:47.188 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010232499999999999 2024-09-18T04:38:47.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.189 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.189 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:47.189 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.486524+0000", 2024-09-18T04:38:47.189 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4972000000000003e-05 2024-09-18T04:38:47.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.189 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.189 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:47.189 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.488902+0000", 2024-09-18T04:38:47.189 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023776240000000001 2024-09-18T04:38:47.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.189 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.189 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:47.189 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.488968+0000", 2024-09-18T04:38:47.190 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6761000000000007e-05 2024-09-18T04:38:47.190 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.190 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:47.190 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.190 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.190 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:57 1.0 e15/14 1:30306672:devicehealth::main.db.0000000000000000:head v 15'35, mlcod=15'33)", 2024-09-18T04:38:47.190 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.529997+0000", 2024-09-18T04:38:47.190 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 79.650531224000005, 2024-09-18T04:38:47.190 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002623869, 2024-09-18T04:38:47.190 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:47.190 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:47.190 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:47.190 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.191 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:47.191 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.529997+0000", 2024-09-18T04:38:47.191 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.191 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.191 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:47.191 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.529997+0000", 2024-09-18T04:38:47.191 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.191 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.191 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:47.191 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.529995+0000", 2024-09-18T04:38:47.192 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:47.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.192 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.192 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:47.192 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530116+0000", 2024-09-18T04:38:47.192 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000121141 2024-09-18T04:38:47.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.192 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.192 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:47.192 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530146+0000", 2024-09-18T04:38:47.192 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0423e-05 2024-09-18T04:38:47.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.192 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.193 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:47.193 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530159+0000", 2024-09-18T04:38:47.193 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2802999999999999e-05 2024-09-18T04:38:47.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.193 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.193 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:47.193 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530218+0000", 2024-09-18T04:38:47.193 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8461000000000002e-05 2024-09-18T04:38:47.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.193 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.193 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:47.193 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.530253+0000", 2024-09-18T04:38:47.193 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4796999999999999e-05 2024-09-18T04:38:47.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.194 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.194 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:47.194 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.532559+0000", 2024-09-18T04:38:47.194 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002306939 2024-09-18T04:38:47.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.194 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.194 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:47.194 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.532621+0000", 2024-09-18T04:38:47.194 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1131999999999997e-05 2024-09-18T04:38:47.194 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.194 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:47.194 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.194 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.195 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:68 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'42, mlcod=15'39)", 2024-09-18T04:38:47.195 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.546404+0000", 2024-09-18T04:38:47.195 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 79.634123813000002, 2024-09-18T04:38:47.195 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024499769999999999, 2024-09-18T04:38:47.195 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:47.195 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:47.195 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:47.195 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.195 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:47.195 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546404+0000", 2024-09-18T04:38:47.195 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.195 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.196 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:47.196 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546404+0000", 2024-09-18T04:38:47.196 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.196 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.196 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:47.196 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546403+0000", 2024-09-18T04:38:47.196 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:47.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.196 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.196 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:47.196 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546416+0000", 2024-09-18T04:38:47.196 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.306e-05 2024-09-18T04:38:47.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.197 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.197 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:47.197 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546465+0000", 2024-09-18T04:38:47.197 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9441999999999999e-05 2024-09-18T04:38:47.197 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.197 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.197 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:47.197 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546480+0000", 2024-09-18T04:38:47.197 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4919999999999999e-05 2024-09-18T04:38:47.197 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.197 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.197 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:47.197 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546541+0000", 2024-09-18T04:38:47.198 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1085000000000002e-05 2024-09-18T04:38:47.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.198 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.198 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:47.198 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.546569+0000", 2024-09-18T04:38:47.198 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7803000000000001e-05 2024-09-18T04:38:47.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.198 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.198 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:47.198 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.548725+0000", 2024-09-18T04:38:47.198 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021556629999999999 2024-09-18T04:38:47.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.198 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.199 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:47.199 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.548854+0000", 2024-09-18T04:38:47.199 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000129524 2024-09-18T04:38:47.199 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.199 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:47.199 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.199 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.199 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:105 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'67, mlcod=15'64)", 2024-09-18T04:38:47.199 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.789089+0000", 2024-09-18T04:38:47.199 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 79.391438983, 2024-09-18T04:38:47.199 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024384229999999999, 2024-09-18T04:38:47.199 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:47.200 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:47.200 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:47.200 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.200 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:47.200 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789089+0000", 2024-09-18T04:38:47.200 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.200 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.200 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.200 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:47.200 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789089+0000", 2024-09-18T04:38:47.200 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.200 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.200 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.200 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:47.201 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789088+0000", 2024-09-18T04:38:47.201 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:47.201 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.201 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.201 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:47.201 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789107+0000", 2024-09-18T04:38:47.201 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9017000000000001e-05 2024-09-18T04:38:47.201 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.201 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.201 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:47.201 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789139+0000", 2024-09-18T04:38:47.201 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2261999999999997e-05 2024-09-18T04:38:47.201 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.202 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.202 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:47.202 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789153+0000", 2024-09-18T04:38:47.202 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4025999999999999e-05 2024-09-18T04:38:47.202 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.202 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.202 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:47.202 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789255+0000", 2024-09-18T04:38:47.202 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000101969 2024-09-18T04:38:47.202 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.202 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.202 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:47.202 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.789314+0000", 2024-09-18T04:38:47.202 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8867999999999999e-05 2024-09-18T04:38:47.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.203 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.203 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:47.203 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.791453+0000", 2024-09-18T04:38:47.203 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002138938 2024-09-18T04:38:47.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.203 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.203 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:47.203 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.791527+0000", 2024-09-18T04:38:47.203 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4464999999999999e-05 2024-09-18T04:38:47.203 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.203 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:47.203 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.204 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.204 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:118 1.0 e15/14 1:30306672:devicehealth::main.db.0000000000000000:head v 15'80, mlcod=15'78)", 2024-09-18T04:38:47.204 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:27.807930+0000", 2024-09-18T04:38:47.204 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 79.372597897000006, 2024-09-18T04:38:47.204 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028002259999999998, 2024-09-18T04:38:47.204 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:47.204 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:47.204 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:47.204 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.204 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:47.204 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.807930+0000", 2024-09-18T04:38:47.204 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.205 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.205 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.205 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:47.205 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.807930+0000", 2024-09-18T04:38:47.205 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.205 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.205 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.205 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:47.205 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.807929+0000", 2024-09-18T04:38:47.205 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:47.205 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.205 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.205 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:47.206 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808024+0000", 2024-09-18T04:38:47.206 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5258000000000002e-05 2024-09-18T04:38:47.206 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.206 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.206 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:47.206 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808044+0000", 2024-09-18T04:38:47.206 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0203e-05 2024-09-18T04:38:47.206 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.206 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.206 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:47.206 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808053+0000", 2024-09-18T04:38:47.206 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2689999999999999e-06 2024-09-18T04:38:47.206 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.206 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.207 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:47.207 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808137+0000", 2024-09-18T04:38:47.207 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4441e-05 2024-09-18T04:38:47.207 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.207 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.207 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:47.207 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.808172+0000", 2024-09-18T04:38:47.207 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4453999999999999e-05 2024-09-18T04:38:47.207 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.207 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.207 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:47.207 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.810663+0000", 2024-09-18T04:38:47.207 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024910269999999998 2024-09-18T04:38:47.208 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.208 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.208 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:47.208 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:27.810730+0000", 2024-09-18T04:38:47.208 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7658999999999997e-05 2024-09-18T04:38:47.208 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.208 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:47.208 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.208 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.208 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.208 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:130 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'88, mlcod=15'85)", 2024-09-18T04:38:47.208 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.232853+0000", 2024-09-18T04:38:47.208 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 78.947675269000001, 2024-09-18T04:38:47.208 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027351739999999999, 2024-09-18T04:38:47.209 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:47.209 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:47.209 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:47.209 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.209 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:47.209 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232853+0000", 2024-09-18T04:38:47.209 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.209 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.209 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.209 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:47.209 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232853+0000", 2024-09-18T04:38:47.209 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.209 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.209 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.210 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:47.210 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232851+0000", 2024-09-18T04:38:47.210 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:47.210 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.210 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.210 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:47.210 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232866+0000", 2024-09-18T04:38:47.210 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4946000000000001e-05 2024-09-18T04:38:47.210 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.210 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.210 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:47.210 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232897+0000", 2024-09-18T04:38:47.210 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0982999999999998e-05 2024-09-18T04:38:47.211 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.211 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.211 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:47.211 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.232915+0000", 2024-09-18T04:38:47.211 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8476000000000001e-05 2024-09-18T04:38:47.211 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.211 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.211 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:47.211 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.233029+0000", 2024-09-18T04:38:47.211 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011305400000000001 2024-09-18T04:38:47.211 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.211 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.211 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:47.211 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.233079+0000", 2024-09-18T04:38:47.212 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0668999999999999e-05 2024-09-18T04:38:47.212 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.212 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.212 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:47.212 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235534+0000", 2024-09-18T04:38:47.212 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024546030000000001 2024-09-18T04:38:47.212 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.212 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.212 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:47.212 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235588+0000", 2024-09-18T04:38:47.212 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4098999999999998e-05 2024-09-18T04:38:47.212 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.212 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:47.212 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.213 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.213 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.213 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:131 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'89, mlcod=15'85)", 2024-09-18T04:38:47.213 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.234089+0000", 2024-09-18T04:38:47.213 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 78.946439415, 2024-09-18T04:38:47.213 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026975609999999998, 2024-09-18T04:38:47.213 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:47.213 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:47.213 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:47.213 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.213 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:47.213 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234089+0000", 2024-09-18T04:38:47.213 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.213 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.214 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.214 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:47.214 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234089+0000", 2024-09-18T04:38:47.214 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.214 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.214 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.214 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:47.214 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234087+0000", 2024-09-18T04:38:47.214 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:47.214 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.214 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.214 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:47.214 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234145+0000", 2024-09-18T04:38:47.214 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8687999999999998e-05 2024-09-18T04:38:47.215 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.215 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.215 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:47.215 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234177+0000", 2024-09-18T04:38:47.215 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1526999999999998e-05 2024-09-18T04:38:47.215 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.215 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.215 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:47.215 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234190+0000", 2024-09-18T04:38:47.216 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2965999999999999e-05 2024-09-18T04:38:47.216 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.216 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.216 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:47.216 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234309+0000", 2024-09-18T04:38:47.216 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011875399999999999 2024-09-18T04:38:47.216 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.216 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.216 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:47.216 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.234365+0000", 2024-09-18T04:38:47.216 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6436e-05 2024-09-18T04:38:47.216 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.216 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.216 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:47.217 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236675+0000", 2024-09-18T04:38:47.217 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023099520000000001 2024-09-18T04:38:47.217 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.217 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.217 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:47.217 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.236786+0000", 2024-09-18T04:38:47.217 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000111025 2024-09-18T04:38:47.217 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.217 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:47.217 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.217 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.217 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.217 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:132 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'90, mlcod=15'85)", 2024-09-18T04:38:47.217 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.235300+0000", 2024-09-18T04:38:47.218 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 78.945227965000001, 2024-09-18T04:38:47.218 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025982599999999998, 2024-09-18T04:38:47.218 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:47.218 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:47.218 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:47.218 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.218 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:47.218 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235300+0000", 2024-09-18T04:38:47.218 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.218 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.218 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.218 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:47.218 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235300+0000", 2024-09-18T04:38:47.219 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.219 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.219 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.219 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:47.219 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235298+0000", 2024-09-18T04:38:47.219 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:47.219 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.219 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.219 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:47.219 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235318+0000", 2024-09-18T04:38:47.219 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9714000000000001e-05 2024-09-18T04:38:47.219 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.219 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.219 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:47.220 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235337+0000", 2024-09-18T04:38:47.220 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8892000000000001e-05 2024-09-18T04:38:47.220 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.220 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.220 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:47.220 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235348+0000", 2024-09-18T04:38:47.220 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.129e-05 2024-09-18T04:38:47.220 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.220 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.220 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:47.220 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235681+0000", 2024-09-18T04:38:47.220 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00033323000000000001 2024-09-18T04:38:47.220 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.220 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.221 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:47.221 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.235733+0000", 2024-09-18T04:38:47.221 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1391999999999999e-05 2024-09-18T04:38:47.221 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.221 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.221 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:47.221 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237839+0000", 2024-09-18T04:38:47.221 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002105647 2024-09-18T04:38:47.221 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.221 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.221 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:47.221 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237898+0000", 2024-09-18T04:38:47.221 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9759000000000002e-05 2024-09-18T04:38:47.221 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.222 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:47.222 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.222 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.222 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.222 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4186.0:134 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'92, mlcod=15'86)", 2024-09-18T04:38:47.222 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:37:28.237894+0000", 2024-09-18T04:38:47.222 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 78.942634054999999, 2024-09-18T04:38:47.222 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025299519999999998, 2024-09-18T04:38:47.222 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:47.222 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:47.222 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:47.222 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.222 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:47.222 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237894+0000", 2024-09-18T04:38:47.223 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.223 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.223 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.223 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:47.223 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237894+0000", 2024-09-18T04:38:47.223 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.223 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.223 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.223 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:47.223 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237892+0000", 2024-09-18T04:38:47.223 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:47.223 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.223 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.223 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:47.224 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.237986+0000", 2024-09-18T04:38:47.224 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3341e-05 2024-09-18T04:38:47.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.224 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.224 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:47.224 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.238012+0000", 2024-09-18T04:38:47.224 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5843e-05 2024-09-18T04:38:47.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.224 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.224 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:47.224 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.238027+0000", 2024-09-18T04:38:47.224 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5469e-05 2024-09-18T04:38:47.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.224 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.225 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:47.225 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.238114+0000", 2024-09-18T04:38:47.225 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7483000000000002e-05 2024-09-18T04:38:47.225 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.225 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.225 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:47.225 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.238152+0000", 2024-09-18T04:38:47.225 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7565999999999999e-05 2024-09-18T04:38:47.225 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.225 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.226 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:47.226 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.240297+0000", 2024-09-18T04:38:47.226 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021447620000000001 2024-09-18T04:38:47.226 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.226 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.226 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:47.226 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:37:28.240424+0000", 2024-09-18T04:38:47.226 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012705400000000001 2024-09-18T04:38:47.226 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.226 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:47.226 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.226 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.226 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.226 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:37 2.5 e51/28 2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:head v 51'1, mlcod=0'0)", 2024-09-18T04:38:47.227 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:38:47.227 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.8880593470000004, 2024-09-18T04:38:47.227 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012279973, 2024-09-18T04:38:47.227 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:47.227 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:47.227 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:47.227 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.227 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:47.227 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:38:47.227 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.227 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.227 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.227 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:47.228 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:38:47.228 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.228 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.228 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.228 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:47.228 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292468+0000", 2024-09-18T04:38:47.228 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:47.228 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.228 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.228 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:47.228 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295837+0000", 2024-09-18T04:38:47.228 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0033691699999999999 2024-09-18T04:38:47.228 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.229 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.229 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:47.229 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295860+0000", 2024-09-18T04:38:47.229 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2756999999999999e-05 2024-09-18T04:38:47.229 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.229 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.229 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:47.229 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295872+0000", 2024-09-18T04:38:47.229 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2442e-05 2024-09-18T04:38:47.229 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.229 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.229 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:47.229 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295934+0000", 2024-09-18T04:38:47.230 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1187999999999998e-05 2024-09-18T04:38:47.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.230 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.230 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:47.230 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295967+0000", 2024-09-18T04:38:47.230 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3185999999999997e-05 2024-09-18T04:38:47.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.230 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.230 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:47.230 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304697+0000", 2024-09-18T04:38:47.230 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0087298059999999997 2024-09-18T04:38:47.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.230 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.230 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:47.231 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304749+0000", 2024-09-18T04:38:47.231 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1922000000000002e-05 2024-09-18T04:38:47.231 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.231 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:47.231 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.231 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.231 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.231 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:49 2.5 e51/28 2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head v 51'2, mlcod=0'0)", 2024-09-18T04:38:47.231 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:38:47.231 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.8547325089999998, 2024-09-18T04:38:47.231 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018033041999999999, 2024-09-18T04:38:47.231 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:47.231 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:47.231 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:47.232 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.232 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:47.232 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:38:47.232 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.232 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.232 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.232 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:47.232 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:38:47.232 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.232 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.232 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.232 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:47.232 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325793+0000", 2024-09-18T04:38:47.233 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:47.233 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.233 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.233 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:47.233 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329038+0000", 2024-09-18T04:38:47.233 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0032444010000000001 2024-09-18T04:38:47.233 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.233 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.233 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:47.233 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329065+0000", 2024-09-18T04:38:47.233 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7500000000000001e-05 2024-09-18T04:38:47.233 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.233 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.234 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:47.234 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329077+0000", 2024-09-18T04:38:47.234 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1919999999999999e-05 2024-09-18T04:38:47.234 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.234 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.234 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:47.234 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329149+0000", 2024-09-18T04:38:47.234 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2349999999999997e-05 2024-09-18T04:38:47.234 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.234 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.234 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:47.234 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329192+0000", 2024-09-18T04:38:47.234 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2744000000000003e-05 2024-09-18T04:38:47.234 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.235 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.235 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:47.235 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343773+0000", 2024-09-18T04:38:47.235 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014581246000000001 2024-09-18T04:38:47.235 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.235 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.235 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:47.235 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343829+0000", 2024-09-18T04:38:47.235 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5143999999999999e-05 2024-09-18T04:38:47.235 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.235 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:47.235 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.235 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.235 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.236 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:47 2.4 e51/17 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 51'3, mlcod=51'1)", 2024-09-18T04:38:47.236 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:38:47.236 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.8541522439999998, 2024-09-18T04:38:47.236 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018960186, 2024-09-18T04:38:47.236 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:47.236 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:47.236 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:47.236 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.236 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:47.236 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:38:47.236 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.236 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.236 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.236 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:47.237 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:38:47.237 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.237 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.237 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.237 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:47.237 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326374+0000", 2024-09-18T04:38:47.237 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:47.237 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.237 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.237 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:47.237 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330883+0000", 2024-09-18T04:38:47.237 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0045084740000000002 2024-09-18T04:38:47.237 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.237 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.238 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:47.238 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330901+0000", 2024-09-18T04:38:47.238 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7890999999999999e-05 2024-09-18T04:38:47.238 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.238 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.238 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:47.238 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330912+0000", 2024-09-18T04:38:47.238 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1041e-05 2024-09-18T04:38:47.238 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.238 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.238 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:47.238 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.331003+0000", 2024-09-18T04:38:47.238 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1323999999999995e-05 2024-09-18T04:38:47.238 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.239 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.239 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:47.239 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.331030+0000", 2024-09-18T04:38:47.239 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7351999999999998e-05 2024-09-18T04:38:47.239 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.239 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.239 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:47.239 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345287+0000", 2024-09-18T04:38:47.239 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014256332 2024-09-18T04:38:47.239 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.239 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.239 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:47.239 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345336+0000", 2024-09-18T04:38:47.239 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9356e-05 2024-09-18T04:38:47.240 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.240 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:47.240 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.240 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.240 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.240 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:58 2.5 e51/28 2:b227e6b1:::rbd_data.125fc09df6d1.000000000000000e:head v 51'3, mlcod=51'1)", 2024-09-18T04:38:47.240 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:38:47.240 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.8137256170000002, 2024-09-18T04:38:47.240 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016513678, 2024-09-18T04:38:47.240 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:47.241 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:47.241 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:47.241 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.241 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:47.241 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:38:47.241 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.241 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.241 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.241 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:47.241 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:38:47.241 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.241 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.241 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.241 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:47.242 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366800+0000", 2024-09-18T04:38:47.242 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:47.242 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.242 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.242 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:47.242 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368644+0000", 2024-09-18T04:38:47.242 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001844399 2024-09-18T04:38:47.242 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.242 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.242 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:47.242 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368681+0000", 2024-09-18T04:38:47.242 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6526999999999998e-05 2024-09-18T04:38:47.242 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.243 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.243 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:47.243 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368698+0000", 2024-09-18T04:38:47.243 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7272999999999999e-05 2024-09-18T04:38:47.243 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.243 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.243 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:47.243 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368773+0000", 2024-09-18T04:38:47.243 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4862000000000003e-05 2024-09-18T04:38:47.243 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.243 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.243 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:47.243 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368801+0000", 2024-09-18T04:38:47.243 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7925000000000001e-05 2024-09-18T04:38:47.244 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.244 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.244 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:47.244 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383261+0000", 2024-09-18T04:38:47.244 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014460377999999999 2024-09-18T04:38:47.244 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.244 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.244 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:47.244 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383316+0000", 2024-09-18T04:38:47.244 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4623999999999997e-05 2024-09-18T04:38:47.244 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.244 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:47.244 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.245 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.245 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.245 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:56 2.4 e51/17 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 51'4, mlcod=51'2)", 2024-09-18T04:38:47.245 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:38:47.245 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.8117215909999995, 2024-09-18T04:38:47.245 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015105221, 2024-09-18T04:38:47.245 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:47.245 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:47.245 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:47.245 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.245 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:47.245 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:38:47.245 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.246 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.246 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.246 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:47.246 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:38:47.246 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.246 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.246 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.246 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:47.246 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368805+0000", 2024-09-18T04:38:47.246 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:47.246 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.246 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.246 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:47.246 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371603+0000", 2024-09-18T04:38:47.247 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027982660000000002 2024-09-18T04:38:47.247 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.247 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.247 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:47.247 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371633+0000", 2024-09-18T04:38:47.247 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0073000000000002e-05 2024-09-18T04:38:47.247 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.247 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.247 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:47.247 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371652+0000", 2024-09-18T04:38:47.247 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8597000000000002e-05 2024-09-18T04:38:47.247 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.247 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.247 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:47.248 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371752+0000", 2024-09-18T04:38:47.248 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9772999999999995e-05 2024-09-18T04:38:47.248 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.248 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.248 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:47.248 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371793+0000", 2024-09-18T04:38:47.248 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1173e-05 2024-09-18T04:38:47.248 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.248 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.248 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:47.248 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383875+0000", 2024-09-18T04:38:47.248 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012081852000000001 2024-09-18T04:38:47.248 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.248 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.249 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:47.249 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383912+0000", 2024-09-18T04:38:47.249 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7045000000000002e-05 2024-09-18T04:38:47.249 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.249 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:47.249 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.249 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.249 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.249 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:36 2.5 e51/28 2:bd5b8dd8:ns::rbd_data.133e813ab5bc.0000000000000002:head v 51'4, mlcod=51'2)", 2024-09-18T04:38:47.249 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:38:47.249 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.6430026, 2024-09-18T04:38:47.249 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020758161000000001, 2024-09-18T04:38:47.249 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:47.250 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:47.250 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:47.250 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.250 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:47.250 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:38:47.250 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.250 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.250 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.250 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:47.250 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:38:47.250 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.250 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.250 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.251 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:47.251 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537523+0000", 2024-09-18T04:38:47.251 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:47.251 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.251 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.251 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:47.251 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541578+0000", 2024-09-18T04:38:47.251 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0040549710000000001 2024-09-18T04:38:47.251 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.251 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.251 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:47.251 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541608+0000", 2024-09-18T04:38:47.251 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9349e-05 2024-09-18T04:38:47.251 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.252 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.252 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:47.252 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541621+0000", 2024-09-18T04:38:47.252 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2736e-05 2024-09-18T04:38:47.252 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.252 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.252 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:47.252 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541715+0000", 2024-09-18T04:38:47.252 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4197999999999995e-05 2024-09-18T04:38:47.252 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.252 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.252 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:47.252 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541755+0000", 2024-09-18T04:38:47.253 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9818999999999999e-05 2024-09-18T04:38:47.253 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.253 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.253 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:47.253 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558217+0000", 2024-09-18T04:38:47.253 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016462234999999999 2024-09-18T04:38:47.253 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.253 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.253 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:47.253 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558284+0000", 2024-09-18T04:38:47.253 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6774999999999997e-05 2024-09-18T04:38:47.253 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.253 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:47.253 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.254 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.254 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.254 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:39 2.4 e51/17 2:304364d1:ns::rbd_data.133e813ab5bc.0000000000000004:head v 51'5, mlcod=51'3)", 2024-09-18T04:38:47.254 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:38:47.254 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.6348634730000002, 2024-09-18T04:38:47.254 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016777558000000001, 2024-09-18T04:38:47.254 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:47.254 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:47.254 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:47.254 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.254 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:47.254 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:38:47.254 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.254 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.255 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.255 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:47.255 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:38:47.255 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.255 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.255 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.255 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:47.255 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545663+0000", 2024-09-18T04:38:47.255 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:47.255 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.256 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.256 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:47.256 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548286+0000", 2024-09-18T04:38:47.256 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026233390000000001 2024-09-18T04:38:47.256 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.256 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.256 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:47.256 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548305+0000", 2024-09-18T04:38:47.256 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8425000000000001e-05 2024-09-18T04:38:47.256 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.256 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.256 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:47.256 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548316+0000", 2024-09-18T04:38:47.257 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1163e-05 2024-09-18T04:38:47.257 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.257 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.257 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:47.257 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548379+0000", 2024-09-18T04:38:47.257 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2780000000000005e-05 2024-09-18T04:38:47.257 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.257 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.257 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:47.257 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548399+0000", 2024-09-18T04:38:47.257 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0080000000000001e-05 2024-09-18T04:38:47.257 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.257 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.257 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:47.258 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562383+0000", 2024-09-18T04:38:47.258 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013984671000000001 2024-09-18T04:38:47.258 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.258 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.258 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:47.258 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562442+0000", 2024-09-18T04:38:47.258 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8700999999999999e-05 2024-09-18T04:38:47.258 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.258 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:47.258 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.258 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.258 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.258 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:37 2.5 e51/28 2:bf39b8e4:ns::rbd_data.133e813ab5bc.0000000000000003:head v 51'5, mlcod=51'2)", 2024-09-18T04:38:47.258 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:38:47.259 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.6319952039999999, 2024-09-18T04:38:47.259 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014264492, 2024-09-18T04:38:47.259 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:47.259 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:47.259 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:47.259 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.259 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:47.259 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:38:47.259 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.259 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.259 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.259 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:47.259 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:38:47.260 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.260 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.260 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.260 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:47.260 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548531+0000", 2024-09-18T04:38:47.260 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:47.260 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.260 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.260 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:47.260 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551500+0000", 2024-09-18T04:38:47.260 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0029688129999999998 2024-09-18T04:38:47.260 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.260 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.261 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:47.261 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551516+0000", 2024-09-18T04:38:47.261 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5994e-05 2024-09-18T04:38:47.261 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.261 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.261 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:47.261 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551526+0000", 2024-09-18T04:38:47.261 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9299999999999998e-06 2024-09-18T04:38:47.261 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.261 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.261 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:47.261 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551577+0000", 2024-09-18T04:38:47.262 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0871000000000001e-05 2024-09-18T04:38:47.262 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.262 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.262 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:47.262 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551602+0000", 2024-09-18T04:38:47.262 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4813000000000001e-05 2024-09-18T04:38:47.262 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.262 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.262 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:47.262 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562741+0000", 2024-09-18T04:38:47.262 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011139428999999999 2024-09-18T04:38:47.262 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.262 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.262 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:47.263 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562797+0000", 2024-09-18T04:38:47.263 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6329000000000002e-05 2024-09-18T04:38:47.263 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.263 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:47.263 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.263 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.263 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.263 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:49 2.4 e51/17 2:20f0b0c9:ns::rbd_data.133e813ab5bc.0000000000000009:head v 51'6, mlcod=51'4)", 2024-09-18T04:38:47.263 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:38:47.263 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.6005606459999999, 2024-09-18T04:38:47.263 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013849046, 2024-09-18T04:38:47.263 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:47.263 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:47.263 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:47.264 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.264 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:47.264 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:38:47.264 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.264 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.264 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.264 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:47.264 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:38:47.264 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.264 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.264 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.264 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:47.264 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579966+0000", 2024-09-18T04:38:47.264 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:47.265 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.265 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.265 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:47.265 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582324+0000", 2024-09-18T04:38:47.265 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023578890000000002 2024-09-18T04:38:47.265 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.265 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.265 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:47.265 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582355+0000", 2024-09-18T04:38:47.265 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1485000000000001e-05 2024-09-18T04:38:47.265 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.265 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.265 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:47.265 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582371+0000", 2024-09-18T04:38:47.266 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6147999999999999e-05 2024-09-18T04:38:47.266 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.266 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.266 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:47.266 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582467+0000", 2024-09-18T04:38:47.266 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6059999999999998e-05 2024-09-18T04:38:47.266 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.266 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.266 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:47.266 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582505+0000", 2024-09-18T04:38:47.266 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7679000000000003e-05 2024-09-18T04:38:47.266 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.266 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.266 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:47.267 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593743+0000", 2024-09-18T04:38:47.267 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011238356 2024-09-18T04:38:47.267 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.267 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.267 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:47.267 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593816+0000", 2024-09-18T04:38:47.267 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2935000000000006e-05 2024-09-18T04:38:47.267 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.267 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:47.267 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.267 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.267 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.267 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:53 2.4 e51/17 2:39f7f174:ns::rbd_data.133e813ab5bc.000000000000000b:head v 51'7, mlcod=51'4)", 2024-09-18T04:38:47.267 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:38:47.268 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.5858594909999999, 2024-09-18T04:38:47.268 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0079914159999999995, 2024-09-18T04:38:47.268 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:47.268 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:47.268 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:47.268 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.268 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:47.268 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:38:47.268 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.268 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.268 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.268 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:47.268 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:38:47.268 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.269 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.269 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.269 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:47.269 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594667+0000", 2024-09-18T04:38:47.269 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:47.269 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.269 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.269 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:47.269 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596446+0000", 2024-09-18T04:38:47.269 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001779033 2024-09-18T04:38:47.269 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.269 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.269 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:47.269 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596472+0000", 2024-09-18T04:38:47.270 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6120000000000001e-05 2024-09-18T04:38:47.270 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.270 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.270 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:47.270 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596500+0000", 2024-09-18T04:38:47.270 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7518e-05 2024-09-18T04:38:47.270 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.270 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.270 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:47.270 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596564+0000", 2024-09-18T04:38:47.270 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3880999999999994e-05 2024-09-18T04:38:47.270 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.270 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.270 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:47.271 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596596+0000", 2024-09-18T04:38:47.271 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2638999999999997e-05 2024-09-18T04:38:47.271 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.271 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.271 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:47.271 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602605+0000", 2024-09-18T04:38:47.271 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0060080699999999999 2024-09-18T04:38:47.271 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.271 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.271 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:47.271 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602660+0000", 2024-09-18T04:38:47.271 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5396000000000003e-05 2024-09-18T04:38:47.271 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.271 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:47.272 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.272 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.272 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.272 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:60 2.5 e51/28 2:a06f51e4:ns::rbd_data.133e813ab5bc.000000000000000f:head v 51'6, mlcod=51'4)", 2024-09-18T04:38:47.272 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:38:47.272 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.5710425059999995, 2024-09-18T04:38:47.272 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0093954989999999999, 2024-09-18T04:38:47.272 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:47.272 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:47.272 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:47.272 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.272 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:47.272 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:38:47.273 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.273 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.273 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.273 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:47.273 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:38:47.273 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.273 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.273 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.273 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:47.273 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609483+0000", 2024-09-18T04:38:47.273 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:47.273 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.273 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.273 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:47.274 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611767+0000", 2024-09-18T04:38:47.274 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002283724 2024-09-18T04:38:47.274 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.274 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.274 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:47.274 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611791+0000", 2024-09-18T04:38:47.274 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3547e-05 2024-09-18T04:38:47.274 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.274 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.274 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:47.274 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611800+0000", 2024-09-18T04:38:47.274 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4979999999999995e-06 2024-09-18T04:38:47.274 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.274 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.275 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:47.275 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611864+0000", 2024-09-18T04:38:47.275 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3974999999999996e-05 2024-09-18T04:38:47.275 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.275 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.275 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:47.275 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611887+0000", 2024-09-18T04:38:47.275 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2633000000000002e-05 2024-09-18T04:38:47.275 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.275 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.275 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:47.275 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.618777+0000", 2024-09-18T04:38:47.275 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0068897979999999999 2024-09-18T04:38:47.275 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.276 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.276 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:47.276 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.618881+0000", 2024-09-18T04:38:47.276 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010437699999999999 2024-09-18T04:38:47.276 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.276 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:47.276 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.276 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.276 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:47.276 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:47.278 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:38:47.279 INFO:tasks.workunit.client.0.smithi115.stderr:modprobe: ERROR: could not insert 'nbd': Operation not permitted 2024-09-18T04:38:47.279 INFO:tasks.workunit.client.0.smithi115.stderr:rbd-nbd: failed to find unused device 2024-09-18T04:38:47.279 INFO:tasks.workunit.client.0.smithi115.stderr:rbd: rbd-nbd failed with error: /usr/bin/rbd-nbd: exit status: 1 2024-09-18T04:38:47.279 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:38:47.279 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false _sudo rbd device --device-type nbd map INVALIDIMAGE 2024-09-18T04:38:47.280 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo rbd device --device-type nbd map INVALIDIMAGE 2024-09-18T04:38:47.280 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:38:47.280 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:38:47.280 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:38:47.280 INFO:tasks.workunit.client.0.smithi115.stderr:++ which rbd 2024-09-18T04:38:47.280 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/bin/rbd 2024-09-18T04:38:47.280 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:38:47.280 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd map INVALIDIMAGE 2024-09-18T04:38:47.281 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:47.274+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:47.383 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:47.377+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:47.398 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:47.398 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:47.398 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:38:47.398 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:47.408 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:38:47.483 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:47.477+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:47.513 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:47.514 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:47.514 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:38:47.514 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:38:47.514 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:47.523 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:38:47.584 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:47.578+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:47.603 INFO:tasks.workunit.client.0.smithi115.stderr:rbd: rbd-nbd failed with error: /usr/bin/rbd-nbd: exit status: 1 2024-09-18T04:38:47.611 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:38:47.611 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false _sudo rbd-nbd --device INVALIDDEV map testrbdnbd17342 2024-09-18T04:38:47.611 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo rbd-nbd --device INVALIDDEV map testrbdnbd17342 2024-09-18T04:38:47.611 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:38:47.612 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:38:47.613 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:38:47.614 INFO:tasks.workunit.client.0.smithi115.stderr:++ which rbd-nbd 2024-09-18T04:38:47.616 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/bin/rbd-nbd 2024-09-18T04:38:47.616 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:38:47.616 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/bin/rbd-nbd --device INVALIDDEV map testrbdnbd17342 2024-09-18T04:38:47.638 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:47.638 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:38:47.638 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:38:47.638 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:38:47.638 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.639 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:33 2.1 e51/24 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head v 51'4, mlcod=51'2)", 2024-09-18T04:38:47.639 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:38:47.639 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 5.3543653940000002, 2024-09-18T04:38:47.639 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018702106, 2024-09-18T04:38:47.639 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:47.639 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:47.639 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:47.639 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.639 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:47.639 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:38:47.639 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.639 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.640 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:47.640 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:38:47.640 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.640 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.640 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:47.640 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279197+0000", 2024-09-18T04:38:47.640 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:47.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.640 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.640 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:47.640 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282682+0000", 2024-09-18T04:38:47.640 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0034848689999999998 2024-09-18T04:38:47.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.641 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.641 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:47.641 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282709+0000", 2024-09-18T04:38:47.641 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7599000000000001e-05 2024-09-18T04:38:47.641 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.641 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.641 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:47.641 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282720+0000", 2024-09-18T04:38:47.641 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.092e-05 2024-09-18T04:38:47.641 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.641 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.641 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:47.641 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282791+0000", 2024-09-18T04:38:47.641 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1268999999999998e-05 2024-09-18T04:38:47.642 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.642 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.642 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:47.642 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282843+0000", 2024-09-18T04:38:47.642 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1771999999999998e-05 2024-09-18T04:38:47.642 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.642 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.642 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:47.642 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.297830+0000", 2024-09-18T04:38:47.642 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014987228999999999 2024-09-18T04:38:47.642 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.642 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.642 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:47.643 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.297901+0000", 2024-09-18T04:38:47.643 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0209999999999994e-05 2024-09-18T04:38:47.643 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.643 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:47.643 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.643 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.643 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.643 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:38 2.0 e51/17 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head v 51'10, mlcod=51'8)", 2024-09-18T04:38:47.643 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:38:47.643 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 5.352946964, 2024-09-18T04:38:47.643 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.023984472, 2024-09-18T04:38:47.643 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:47.643 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:47.643 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:47.644 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.644 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:47.644 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:38:47.644 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.644 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.644 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.644 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:47.644 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:38:47.644 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.644 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.644 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.644 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:47.644 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280614+0000", 2024-09-18T04:38:47.645 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:47.645 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.645 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.645 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:47.645 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293094+0000", 2024-09-18T04:38:47.645 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012479571 2024-09-18T04:38:47.645 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.645 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.645 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:47.645 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293111+0000", 2024-09-18T04:38:47.645 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7354999999999999e-05 2024-09-18T04:38:47.645 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.645 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.645 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:47.646 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293121+0000", 2024-09-18T04:38:47.646 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6979999999999993e-06 2024-09-18T04:38:47.646 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.646 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.646 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:47.646 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293180+0000", 2024-09-18T04:38:47.646 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9333000000000003e-05 2024-09-18T04:38:47.646 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.646 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.646 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:47.646 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293203+0000", 2024-09-18T04:38:47.646 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2633000000000002e-05 2024-09-18T04:38:47.646 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.646 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.647 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:47.647 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304534+0000", 2024-09-18T04:38:47.647 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011330761 2024-09-18T04:38:47.647 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.647 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.647 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:47.647 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304601+0000", 2024-09-18T04:38:47.647 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7657000000000006e-05 2024-09-18T04:38:47.647 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.647 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:47.647 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.647 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.647 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.647 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:46 2.6 2:7ba25245:::rbd_data.125fc09df6d1.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:47.648 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:38:47.648 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 5.3254041640000001, 2024-09-18T04:38:47.648 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.035604595000000003, 2024-09-18T04:38:47.648 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:47.648 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:47.648 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:47.648 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:38:47.648 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:38:47.648 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 46 2024-09-18T04:38:47.648 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.648 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:47.648 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.648 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:47.649 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:38:47.649 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.649 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.649 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.649 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:47.649 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:38:47.649 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.649 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.649 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.649 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:47.649 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308158+0000", 2024-09-18T04:38:47.649 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:47.649 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.649 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.650 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:47.650 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313191+0000", 2024-09-18T04:38:47.650 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0050323929999999996 2024-09-18T04:38:47.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.650 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.650 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:47.650 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313210+0000", 2024-09-18T04:38:47.650 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9673e-05 2024-09-18T04:38:47.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.650 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.650 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:47.650 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313221+0000", 2024-09-18T04:38:47.650 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0444e-05 2024-09-18T04:38:47.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.651 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.651 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:47.651 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313294+0000", 2024-09-18T04:38:47.651 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2650000000000004e-05 2024-09-18T04:38:47.651 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.651 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.651 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:47.651 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313493+0000", 2024-09-18T04:38:47.651 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019968300000000001 2024-09-18T04:38:47.651 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.651 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.651 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:38:47.651 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313666+0000", 2024-09-18T04:38:47.651 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00017248599999999999 2024-09-18T04:38:47.652 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.652 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.652 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:47.652 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.323175+0000", 2024-09-18T04:38:47.652 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0095088819999999998 2024-09-18T04:38:47.652 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.652 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.652 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:47.652 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343651+0000", 2024-09-18T04:38:47.652 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020475892999999998 2024-09-18T04:38:47.652 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.652 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.652 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:47.653 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343715+0000", 2024-09-18T04:38:47.653 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4443999999999995e-05 2024-09-18T04:38:47.653 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.653 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.653 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:47.653 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343764+0000", 2024-09-18T04:38:47.653 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9280000000000003e-05 2024-09-18T04:38:47.653 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.653 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:47.653 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.653 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.653 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.653 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:47 2.4 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:47.653 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:38:47.654 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 5.3203221269999998, 2024-09-18T04:38:47.654 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.032570693999999997, 2024-09-18T04:38:47.654 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:47.654 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:47.654 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:47.654 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:38:47.654 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:38:47.654 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 47 2024-09-18T04:38:47.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.654 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:47.654 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.654 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:47.654 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:38:47.654 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.655 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.655 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:47.655 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:38:47.655 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.655 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.655 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:47.655 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313240+0000", 2024-09-18T04:38:47.655 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:47.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.655 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.655 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:47.655 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325825+0000", 2024-09-18T04:38:47.656 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012584425999999999 2024-09-18T04:38:47.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.656 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.656 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:47.656 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325848+0000", 2024-09-18T04:38:47.656 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3025999999999999e-05 2024-09-18T04:38:47.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.656 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.656 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:47.656 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325863+0000", 2024-09-18T04:38:47.657 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5152e-05 2024-09-18T04:38:47.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.657 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.657 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:47.657 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325938+0000", 2024-09-18T04:38:47.657 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4982999999999997e-05 2024-09-18T04:38:47.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.657 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.657 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:47.657 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326090+0000", 2024-09-18T04:38:47.657 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015207900000000001 2024-09-18T04:38:47.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.658 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.658 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:47.658 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326242+0000", 2024-09-18T04:38:47.658 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015163500000000001 2024-09-18T04:38:47.658 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.658 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.658 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:47.658 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343015+0000", 2024-09-18T04:38:47.658 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016773414 2024-09-18T04:38:47.658 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.658 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.658 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:47.658 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345664+0000", 2024-09-18T04:38:47.658 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026487870000000001 2024-09-18T04:38:47.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.659 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.659 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:47.659 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345739+0000", 2024-09-18T04:38:47.659 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5018999999999994e-05 2024-09-18T04:38:47.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.659 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.659 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:47.659 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345812+0000", 2024-09-18T04:38:47.659 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3547000000000006e-05 2024-09-18T04:38:47.659 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.659 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:47.659 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.660 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.660 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:48 2.6 2:61db7a3a:::rbd_data.125fc09df6d1.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:47.660 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:38:47.660 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 5.3076654550000004, 2024-09-18T04:38:47.660 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.025064388, 2024-09-18T04:38:47.660 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:47.660 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:47.660 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:47.660 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:38:47.660 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:38:47.660 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 48 2024-09-18T04:38:47.660 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.660 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:47.661 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.661 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:47.661 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:38:47.661 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.661 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.661 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.661 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:47.661 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:38:47.661 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.661 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.661 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.661 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:47.661 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325897+0000", 2024-09-18T04:38:47.662 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:47.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.662 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.662 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:47.662 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333697+0000", 2024-09-18T04:38:47.662 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0078002649999999998 2024-09-18T04:38:47.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.662 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.662 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:47.662 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333716+0000", 2024-09-18T04:38:47.662 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8910999999999999e-05 2024-09-18T04:38:47.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.662 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.662 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:47.663 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333728+0000", 2024-09-18T04:38:47.663 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1936000000000001e-05 2024-09-18T04:38:47.663 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.663 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.663 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:47.663 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333792+0000", 2024-09-18T04:38:47.663 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4010999999999994e-05 2024-09-18T04:38:47.663 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.663 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.663 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:47.663 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.334082+0000", 2024-09-18T04:38:47.663 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00028955799999999998 2024-09-18T04:38:47.663 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.663 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.663 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:38:47.664 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.334326+0000", 2024-09-18T04:38:47.664 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00024399099999999999 2024-09-18T04:38:47.664 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.664 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.664 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:47.664 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350745+0000", 2024-09-18T04:38:47.664 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016419474999999999 2024-09-18T04:38:47.664 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.664 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.664 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:47.664 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350808+0000", 2024-09-18T04:38:47.664 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3244999999999995e-05 2024-09-18T04:38:47.664 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.664 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.665 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:47.665 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350892+0000", 2024-09-18T04:38:47.665 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4074999999999997e-05 2024-09-18T04:38:47.665 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.665 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.665 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:47.665 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350963+0000", 2024-09-18T04:38:47.665 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0412000000000003e-05 2024-09-18T04:38:47.665 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.665 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:47.665 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.665 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.665 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.665 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:51 2.6 2:775d7b11:::rbd_data.125fc09df6d1.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:47.666 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:38:47.666 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 5.2998133190000001, 2024-09-18T04:38:47.666 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.023824221, 2024-09-18T04:38:47.666 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:47.666 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:47.666 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:47.666 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:38:47.666 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:38:47.666 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 51 2024-09-18T04:38:47.666 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.666 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:47.666 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.666 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:47.667 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:38:47.667 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.667 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.667 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.667 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:47.667 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:38:47.667 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.667 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.667 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.667 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:47.667 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333749+0000", 2024-09-18T04:38:47.667 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:47.667 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.668 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.668 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:47.668 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340899+0000", 2024-09-18T04:38:47.668 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0071500340000000004 2024-09-18T04:38:47.668 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.668 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.668 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:47.668 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340929+0000", 2024-09-18T04:38:47.668 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0093000000000001e-05 2024-09-18T04:38:47.668 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.668 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.668 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:47.668 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340943+0000", 2024-09-18T04:38:47.669 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4019000000000001e-05 2024-09-18T04:38:47.669 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.669 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.669 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:47.669 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343834+0000", 2024-09-18T04:38:47.669 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028918120000000001 2024-09-18T04:38:47.669 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.669 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.669 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:47.669 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344020+0000", 2024-09-18T04:38:47.669 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000185892 2024-09-18T04:38:47.669 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.669 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.670 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:38:47.670 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344190+0000", 2024-09-18T04:38:47.670 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016946800000000001 2024-09-18T04:38:47.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.670 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.670 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:47.670 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.351291+0000", 2024-09-18T04:38:47.670 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.007100805 2024-09-18T04:38:47.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.670 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.670 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:47.670 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357470+0000", 2024-09-18T04:38:47.670 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0061797570000000001 2024-09-18T04:38:47.671 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.671 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.671 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:47.671 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357533+0000", 2024-09-18T04:38:47.671 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2906999999999999e-05 2024-09-18T04:38:47.671 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.671 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.671 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:47.671 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357575+0000", 2024-09-18T04:38:47.671 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1403999999999999e-05 2024-09-18T04:38:47.671 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.671 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:47.671 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.671 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.672 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.672 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:56 2.4 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:47.672 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:38:47.672 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 5.2837682260000003, 2024-09-18T04:38:47.672 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.034295870999999999, 2024-09-18T04:38:47.672 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:47.672 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:47.672 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:47.672 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:38:47.672 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:38:47.672 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 56 2024-09-18T04:38:47.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.672 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:47.673 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.673 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:47.673 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:38:47.673 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.673 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.673 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.673 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:47.673 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:38:47.673 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.673 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.673 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.673 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:47.673 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349794+0000", 2024-09-18T04:38:47.673 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:47.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.674 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.674 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:47.674 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366632+0000", 2024-09-18T04:38:47.674 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01683755 2024-09-18T04:38:47.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.674 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.674 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:47.674 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366669+0000", 2024-09-18T04:38:47.674 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7351000000000002e-05 2024-09-18T04:38:47.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.674 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.674 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:47.674 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366686+0000", 2024-09-18T04:38:47.675 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6507999999999998e-05 2024-09-18T04:38:47.675 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.675 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.675 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:47.675 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366962+0000", 2024-09-18T04:38:47.675 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00027655900000000001 2024-09-18T04:38:47.675 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.675 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.675 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:47.675 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.367452+0000", 2024-09-18T04:38:47.675 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00049025299999999996 2024-09-18T04:38:47.675 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.675 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.675 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:47.676 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368113+0000", 2024-09-18T04:38:47.676 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00066093399999999996 2024-09-18T04:38:47.676 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.676 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.676 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:47.676 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383719+0000", 2024-09-18T04:38:47.676 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015606106999999999 2024-09-18T04:38:47.676 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.676 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.676 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:47.676 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384035+0000", 2024-09-18T04:38:47.676 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00031590399999999999 2024-09-18T04:38:47.676 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.676 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.677 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:47.677 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384067+0000", 2024-09-18T04:38:47.677 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.129e-05 2024-09-18T04:38:47.677 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.677 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.677 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:47.677 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384091+0000", 2024-09-18T04:38:47.677 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4788000000000001e-05 2024-09-18T04:38:47.677 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.677 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:47.677 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.677 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.677 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.677 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:57 2.0 e51/17 2:0cdd070e:::rbd_data.125fc09df6d1.000000000000000d:head v 51'21, mlcod=51'19)", 2024-09-18T04:38:47.678 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:38:47.678 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 5.2759407390000002, 2024-09-18T04:38:47.678 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.025216311000000002, 2024-09-18T04:38:47.678 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:47.678 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:47.678 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:47.678 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.678 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:47.678 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:38:47.678 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.678 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.678 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.678 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:47.678 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:38:47.679 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.679 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.679 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:47.679 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:38:47.679 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:47.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.679 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.679 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:47.679 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363844+0000", 2024-09-18T04:38:47.679 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0062218450000000002 2024-09-18T04:38:47.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.679 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.679 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:47.680 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363879+0000", 2024-09-18T04:38:47.680 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4382999999999999e-05 2024-09-18T04:38:47.680 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.680 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.680 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:47.680 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363893+0000", 2024-09-18T04:38:47.680 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4348999999999999e-05 2024-09-18T04:38:47.680 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.680 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.680 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:47.680 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363977+0000", 2024-09-18T04:38:47.680 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3499999999999997e-05 2024-09-18T04:38:47.680 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.681 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.681 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:47.681 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364009+0000", 2024-09-18T04:38:47.681 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2292000000000002e-05 2024-09-18T04:38:47.681 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.681 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.681 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:47.681 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.382781+0000", 2024-09-18T04:38:47.681 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018771626 2024-09-18T04:38:47.681 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.681 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.681 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:47.681 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.382839+0000", 2024-09-18T04:38:47.681 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8758000000000002e-05 2024-09-18T04:38:47.682 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.682 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:47.682 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.682 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.682 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.682 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:59 2.0 e51/17 2:19c52cde:::rbd_data.125fc09df6d1.000000000000000f:head v 51'22, mlcod=51'19)", 2024-09-18T04:38:47.682 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:38:47.682 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 5.269280813, 2024-09-18T04:38:47.682 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.019497112, 2024-09-18T04:38:47.682 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:47.682 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:47.682 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:47.682 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.682 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:47.683 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:38:47.683 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.683 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.683 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:47.683 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:38:47.683 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.683 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.683 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:47.683 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364282+0000", 2024-09-18T04:38:47.683 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:47.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.684 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.684 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:47.684 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371823+0000", 2024-09-18T04:38:47.684 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0075411829999999999 2024-09-18T04:38:47.684 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.684 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.684 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:47.684 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371837+0000", 2024-09-18T04:38:47.684 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4336e-05 2024-09-18T04:38:47.684 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.684 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.684 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:47.684 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371846+0000", 2024-09-18T04:38:47.684 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9770000000000006e-06 2024-09-18T04:38:47.685 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.685 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.685 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:47.685 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.373566+0000", 2024-09-18T04:38:47.685 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017192539999999999 2024-09-18T04:38:47.685 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.685 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.685 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:47.685 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.373595+0000", 2024-09-18T04:38:47.685 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8975999999999998e-05 2024-09-18T04:38:47.685 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.685 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.685 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:47.686 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383740+0000", 2024-09-18T04:38:47.686 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01014525 2024-09-18T04:38:47.686 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.686 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.686 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:47.686 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383780+0000", 2024-09-18T04:38:47.686 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0275999999999999e-05 2024-09-18T04:38:47.686 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.686 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:47.686 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.686 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.686 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.686 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:34 2.6 2:66289171:ns::rbd_data.133e813ab5bc.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:47.686 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.524015+0000", 2024-09-18T04:38:47.687 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 5.1095492040000003, 2024-09-18T04:38:47.687 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027582111999999999, 2024-09-18T04:38:47.687 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:47.687 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:47.687 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:47.687 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:38:47.687 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:47.687 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 34 2024-09-18T04:38:47.687 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.687 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:47.687 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.687 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:47.687 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524015+0000", 2024-09-18T04:38:47.687 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.688 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.688 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.688 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:47.688 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524015+0000", 2024-09-18T04:38:47.688 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.688 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.688 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.688 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:47.688 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524012+0000", 2024-09-18T04:38:47.688 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:47.688 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.688 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.688 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:47.689 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534956+0000", 2024-09-18T04:38:47.689 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010943403000000001 2024-09-18T04:38:47.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.689 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.689 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:47.689 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534978+0000", 2024-09-18T04:38:47.689 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1965999999999999e-05 2024-09-18T04:38:47.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.689 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.689 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:47.689 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534990+0000", 2024-09-18T04:38:47.689 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2065999999999999e-05 2024-09-18T04:38:47.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.689 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.690 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:47.690 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535065+0000", 2024-09-18T04:38:47.690 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5523000000000002e-05 2024-09-18T04:38:47.690 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.690 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.690 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:47.690 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535208+0000", 2024-09-18T04:38:47.690 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014309500000000001 2024-09-18T04:38:47.690 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.690 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.690 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:38:47.690 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535343+0000", 2024-09-18T04:38:47.690 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013437400000000001 2024-09-18T04:38:47.690 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.691 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.691 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:47.691 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.546576+0000", 2024-09-18T04:38:47.691 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011233425999999999 2024-09-18T04:38:47.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.691 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.691 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:47.691 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551498+0000", 2024-09-18T04:38:47.691 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0049219349999999997 2024-09-18T04:38:47.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.691 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.691 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:47.691 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551554+0000", 2024-09-18T04:38:47.691 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6400999999999998e-05 2024-09-18T04:38:47.692 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.692 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.692 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:47.692 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551597+0000", 2024-09-18T04:38:47.692 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2283000000000002e-05 2024-09-18T04:38:47.692 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.692 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:47.692 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.692 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.692 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.692 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:33 2.1 e51/24 2:8b40f1c5:ns::rbd_data.133e813ab5bc.0000000000000000:head v 51'7, mlcod=51'5)", 2024-09-18T04:38:47.692 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.531767+0000", 2024-09-18T04:38:47.692 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 5.1017972870000001, 2024-09-18T04:38:47.692 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015370295000000001, 2024-09-18T04:38:47.693 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:47.693 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:47.693 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:47.693 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.693 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:47.693 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531767+0000", 2024-09-18T04:38:47.693 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.693 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.693 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.693 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:47.693 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531767+0000", 2024-09-18T04:38:47.693 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.693 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.693 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.694 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:47.694 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531765+0000", 2024-09-18T04:38:47.694 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:47.694 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.694 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.694 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:47.694 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534258+0000", 2024-09-18T04:38:47.694 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002493561 2024-09-18T04:38:47.694 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.694 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.694 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:47.694 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534288+0000", 2024-09-18T04:38:47.694 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.976e-05 2024-09-18T04:38:47.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.695 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.695 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:47.695 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534300+0000", 2024-09-18T04:38:47.695 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1928e-05 2024-09-18T04:38:47.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.695 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.695 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:47.695 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534367+0000", 2024-09-18T04:38:47.695 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7045999999999995e-05 2024-09-18T04:38:47.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.695 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.695 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:47.695 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534394+0000", 2024-09-18T04:38:47.696 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6903999999999999e-05 2024-09-18T04:38:47.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.696 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.696 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:47.696 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.547073+0000", 2024-09-18T04:38:47.696 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012679365 2024-09-18T04:38:47.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.696 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.696 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:47.696 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.547137+0000", 2024-09-18T04:38:47.696 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3691999999999999e-05 2024-09-18T04:38:47.696 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.696 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:47.696 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.697 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.697 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:39 2.4 2:304364d1:ns::rbd_data.133e813ab5bc.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:47.697 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.535013+0000", 2024-09-18T04:38:47.697 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 5.0985503520000002, 2024-09-18T04:38:47.697 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027764487000000001, 2024-09-18T04:38:47.697 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:47.697 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:47.697 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:47.697 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:38:47.697 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:47.697 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 39 2024-09-18T04:38:47.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.698 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:47.698 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.698 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:47.698 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535013+0000", 2024-09-18T04:38:47.698 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.698 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.698 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:47.698 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535013+0000", 2024-09-18T04:38:47.698 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.698 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.698 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:47.698 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535012+0000", 2024-09-18T04:38:47.699 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:47.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.699 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.699 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:47.699 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545016+0000", 2024-09-18T04:38:47.699 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01000322 2024-09-18T04:38:47.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.699 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.699 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:47.699 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545035+0000", 2024-09-18T04:38:47.699 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9845999999999999e-05 2024-09-18T04:38:47.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.699 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.699 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:47.700 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545045+0000", 2024-09-18T04:38:47.700 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3640000000000005e-06 2024-09-18T04:38:47.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.700 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.700 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:47.700 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545152+0000", 2024-09-18T04:38:47.700 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000107025 2024-09-18T04:38:47.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.700 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.700 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:47.700 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545338+0000", 2024-09-18T04:38:47.700 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018643500000000001 2024-09-18T04:38:47.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.701 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.701 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:47.701 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545492+0000", 2024-09-18T04:38:47.701 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015393199999999999 2024-09-18T04:38:47.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.701 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.701 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:47.701 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.553549+0000", 2024-09-18T04:38:47.701 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0080569170000000002 2024-09-18T04:38:47.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.701 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.701 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:47.701 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562693+0000", 2024-09-18T04:38:47.701 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0091441609999999996 2024-09-18T04:38:47.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.702 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.702 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:47.702 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562748+0000", 2024-09-18T04:38:47.702 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5047000000000003e-05 2024-09-18T04:38:47.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.702 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.702 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:47.702 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562778+0000", 2024-09-18T04:38:47.702 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9652e-05 2024-09-18T04:38:47.702 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.702 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:47.702 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.703 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.703 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:40 2.6 2:784d6cd1:ns::rbd_data.133e813ab5bc.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:47.703 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.545062+0000", 2024-09-18T04:38:47.703 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 5.088501301, 2024-09-18T04:38:47.703 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.032771623, 2024-09-18T04:38:47.703 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:47.703 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:47.703 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:47.703 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:38:47.703 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:47.703 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 40 2024-09-18T04:38:47.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.703 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:47.703 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.704 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:47.704 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545062+0000", 2024-09-18T04:38:47.704 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.704 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.704 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.704 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:47.704 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545062+0000", 2024-09-18T04:38:47.704 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.704 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.704 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.704 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:47.704 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545061+0000", 2024-09-18T04:38:47.704 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:47.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.705 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.705 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:47.705 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552477+0000", 2024-09-18T04:38:47.705 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.007415914 2024-09-18T04:38:47.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.705 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.705 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:47.705 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552500+0000", 2024-09-18T04:38:47.705 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3546000000000001e-05 2024-09-18T04:38:47.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.705 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.705 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:47.705 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552515+0000", 2024-09-18T04:38:47.706 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4606999999999999e-05 2024-09-18T04:38:47.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.706 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.706 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:47.706 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552661+0000", 2024-09-18T04:38:47.706 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000146007 2024-09-18T04:38:47.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.706 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.706 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:47.706 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552876+0000", 2024-09-18T04:38:47.706 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000214618 2024-09-18T04:38:47.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.706 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.706 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:38:47.706 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.553061+0000", 2024-09-18T04:38:47.707 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018564100000000001 2024-09-18T04:38:47.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.707 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.707 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:47.707 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.565612+0000", 2024-09-18T04:38:47.707 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012550301999999999 2024-09-18T04:38:47.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.707 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.707 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:47.707 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577695+0000", 2024-09-18T04:38:47.707 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012083493000000001 2024-09-18T04:38:47.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.707 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.708 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:47.708 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577772+0000", 2024-09-18T04:38:47.708 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7158999999999998e-05 2024-09-18T04:38:47.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.708 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.708 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:47.709 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577834+0000", 2024-09-18T04:38:47.709 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1851000000000001e-05 2024-09-18T04:38:47.709 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.709 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:47.709 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.709 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.709 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.709 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:43 2.6 2:759f27b6:ns::rbd_data.133e813ab5bc.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:47.709 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.555868+0000", 2024-09-18T04:38:47.709 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 5.0776958069999996, 2024-09-18T04:38:47.709 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.022332497999999999, 2024-09-18T04:38:47.709 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:47.709 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:47.710 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:47.710 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:38:47.710 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:47.710 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 43 2024-09-18T04:38:47.710 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.710 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:47.710 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.710 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:47.710 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555868+0000", 2024-09-18T04:38:47.710 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.710 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.710 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.710 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:47.710 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555868+0000", 2024-09-18T04:38:47.711 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.711 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.711 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:47.711 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555865+0000", 2024-09-18T04:38:47.711 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999971 2024-09-18T04:38:47.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.711 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.711 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:47.711 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561031+0000", 2024-09-18T04:38:47.711 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0051658770000000001 2024-09-18T04:38:47.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.711 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.712 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:47.712 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561048+0000", 2024-09-18T04:38:47.712 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6569e-05 2024-09-18T04:38:47.712 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.712 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.712 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:47.712 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561055+0000", 2024-09-18T04:38:47.714 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.328e-06 2024-09-18T04:38:47.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.715 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.715 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:47.715 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561114+0000", 2024-09-18T04:38:47.715 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9540000000000003e-05 2024-09-18T04:38:47.715 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.715 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.715 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:47.715 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561314+0000", 2024-09-18T04:38:47.715 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019965799999999999 2024-09-18T04:38:47.715 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.715 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.715 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:38:47.716 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561473+0000", 2024-09-18T04:38:47.716 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000158904 2024-09-18T04:38:47.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.716 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.716 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:47.716 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.567015+0000", 2024-09-18T04:38:47.716 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055420859999999999 2024-09-18T04:38:47.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.716 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.716 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:47.716 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577982+0000", 2024-09-18T04:38:47.716 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010966872000000001 2024-09-18T04:38:47.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.717 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.717 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:47.717 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578162+0000", 2024-09-18T04:38:47.717 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018050999999999999 2024-09-18T04:38:47.717 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.717 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.717 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:47.717 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578200+0000", 2024-09-18T04:38:47.717 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8000999999999998e-05 2024-09-18T04:38:47.717 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.717 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:47.717 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.717 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.718 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.718 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:48 2.6 2:7c225365:ns::rbd_data.133e813ab5bc.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:47.718 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:38:47.718 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 5.0698965840000003, 2024-09-18T04:38:47.718 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.024576377, 2024-09-18T04:38:47.718 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:47.718 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:47.718 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:47.718 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:38:47.718 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:47.718 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 48 2024-09-18T04:38:47.718 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.718 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:47.718 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.719 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:47.719 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:38:47.719 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.719 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.719 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.719 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:47.720 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:38:47.720 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.720 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.720 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.720 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:47.720 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.563666+0000", 2024-09-18T04:38:47.720 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:47.720 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.720 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.720 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:47.720 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570331+0000", 2024-09-18T04:38:47.720 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0066653329999999998 2024-09-18T04:38:47.720 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.721 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.721 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:47.721 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570348+0000", 2024-09-18T04:38:47.721 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6507e-05 2024-09-18T04:38:47.721 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.721 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.721 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:47.721 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570355+0000", 2024-09-18T04:38:47.721 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2749999999999998e-06 2024-09-18T04:38:47.721 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.721 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.721 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:47.721 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570396+0000", 2024-09-18T04:38:47.722 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0809000000000002e-05 2024-09-18T04:38:47.722 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.722 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.722 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:47.722 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570571+0000", 2024-09-18T04:38:47.722 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000174823 2024-09-18T04:38:47.722 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.722 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.722 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:38:47.722 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570722+0000", 2024-09-18T04:38:47.722 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015154599999999999 2024-09-18T04:38:47.722 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.722 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.722 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:47.723 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579756+0000", 2024-09-18T04:38:47.723 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0090337000000000004 2024-09-18T04:38:47.723 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.723 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.723 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:47.723 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588093+0000", 2024-09-18T04:38:47.724 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.008336553 2024-09-18T04:38:47.724 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.724 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.724 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:47.724 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588173+0000", 2024-09-18T04:38:47.724 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0333999999999994e-05 2024-09-18T04:38:47.724 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.724 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.724 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:47.724 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588244+0000", 2024-09-18T04:38:47.724 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0573000000000003e-05 2024-09-18T04:38:47.725 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.725 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:47.725 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.725 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.725 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.725 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:49 2.4 2:20f0b0c9:ns::rbd_data.133e813ab5bc.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:47.725 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.570370+0000", 2024-09-18T04:38:47.725 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 5.0631942570000001, 2024-09-18T04:38:47.725 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.029633981, 2024-09-18T04:38:47.725 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:47.725 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:47.725 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:47.725 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:38:47.726 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:47.726 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 49 2024-09-18T04:38:47.726 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.726 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:47.726 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.726 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:47.726 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570370+0000", 2024-09-18T04:38:47.726 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.726 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.726 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.726 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:47.726 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570370+0000", 2024-09-18T04:38:47.727 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.727 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.727 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.727 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:47.727 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570369+0000", 2024-09-18T04:38:47.727 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:47.727 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.727 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.727 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:47.727 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579090+0000", 2024-09-18T04:38:47.727 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0087212339999999996 2024-09-18T04:38:47.727 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.727 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.728 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:47.728 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579109+0000", 2024-09-18T04:38:47.728 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9120999999999999e-05 2024-09-18T04:38:47.728 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.728 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.728 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:47.728 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579123+0000", 2024-09-18T04:38:47.728 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4382000000000001e-05 2024-09-18T04:38:47.728 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.728 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.728 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:47.728 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579221+0000", 2024-09-18T04:38:47.728 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.7503999999999993e-05 2024-09-18T04:38:47.728 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.729 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.729 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:47.729 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579553+0000", 2024-09-18T04:38:47.729 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00033254200000000002 2024-09-18T04:38:47.729 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.729 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.729 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:47.729 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579817+0000", 2024-09-18T04:38:47.729 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00026317099999999998 2024-09-18T04:38:47.729 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.729 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.729 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:47.729 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.599905+0000", 2024-09-18T04:38:47.730 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020088575000000001 2024-09-18T04:38:47.730 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.730 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.730 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:47.730 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.599923+0000", 2024-09-18T04:38:47.730 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7425999999999999e-05 2024-09-18T04:38:47.730 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.730 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.730 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:47.730 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.599973+0000", 2024-09-18T04:38:47.730 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0649999999999998e-05 2024-09-18T04:38:47.730 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.730 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.730 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:47.731 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.600004+0000", 2024-09-18T04:38:47.731 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0361999999999998e-05 2024-09-18T04:38:47.731 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.731 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:47.731 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.731 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.731 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.731 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:50 2.1 e51/24 2:8c706374:ns::rbd_data.133e813ab5bc.000000000000000a:head v 51'8, mlcod=51'6)", 2024-09-18T04:38:47.731 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.577029+0000", 2024-09-18T04:38:47.731 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 5.0565348859999997, 2024-09-18T04:38:47.731 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.017362855, 2024-09-18T04:38:47.731 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:47.731 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:47.731 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:47.732 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.732 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:47.732 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577029+0000", 2024-09-18T04:38:47.732 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.732 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.732 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.732 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:47.732 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577029+0000", 2024-09-18T04:38:47.732 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.732 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.732 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.732 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:47.732 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577027+0000", 2024-09-18T04:38:47.732 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:47.733 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.733 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.733 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:47.733 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578880+0000", 2024-09-18T04:38:47.733 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0018532080000000001 2024-09-18T04:38:47.733 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.733 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.733 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:47.733 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578906+0000", 2024-09-18T04:38:47.733 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5573999999999999e-05 2024-09-18T04:38:47.733 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.733 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.733 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:47.733 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578920+0000", 2024-09-18T04:38:47.734 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4483e-05 2024-09-18T04:38:47.734 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.734 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.734 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:47.734 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578983+0000", 2024-09-18T04:38:47.734 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2248000000000004e-05 2024-09-18T04:38:47.734 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.734 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.734 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:47.734 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579016+0000", 2024-09-18T04:38:47.734 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3364000000000001e-05 2024-09-18T04:38:47.734 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.734 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.735 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:47.735 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594342+0000", 2024-09-18T04:38:47.735 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01532564 2024-09-18T04:38:47.735 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.735 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.735 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:47.735 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594392+0000", 2024-09-18T04:38:47.735 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0068999999999998e-05 2024-09-18T04:38:47.735 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.735 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:47.735 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.735 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.735 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.735 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:45 2.0 e51/17 2:0fbfa3dc:ns::rbd_data.133e813ab5bc.0000000000000007:head v 51'24, mlcod=51'22)", 2024-09-18T04:38:47.736 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.581892+0000", 2024-09-18T04:38:47.736 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 5.0516722139999999, 2024-09-18T04:38:47.736 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015646006, 2024-09-18T04:38:47.736 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:47.736 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:47.736 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:47.736 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.736 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:47.736 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581892+0000", 2024-09-18T04:38:47.736 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.736 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.736 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.736 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:47.737 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581892+0000", 2024-09-18T04:38:47.737 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.737 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.737 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.737 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:47.737 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581890+0000", 2024-09-18T04:38:47.737 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:47.737 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.737 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.737 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:47.737 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586188+0000", 2024-09-18T04:38:47.737 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0042985769999999996 2024-09-18T04:38:47.737 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.737 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.738 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:47.738 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586210+0000", 2024-09-18T04:38:47.738 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1069000000000001e-05 2024-09-18T04:38:47.738 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.738 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.738 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:47.738 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586221+0000", 2024-09-18T04:38:47.738 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1287000000000001e-05 2024-09-18T04:38:47.738 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.738 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.738 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:47.738 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586313+0000", 2024-09-18T04:38:47.738 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1935999999999995e-05 2024-09-18T04:38:47.739 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.739 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.739 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:47.739 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586368+0000", 2024-09-18T04:38:47.739 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5578000000000002e-05 2024-09-18T04:38:47.739 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.739 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.739 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:47.739 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.597473+0000", 2024-09-18T04:38:47.739 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011105139999999999 2024-09-18T04:38:47.739 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.739 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.739 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:47.739 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.597538+0000", 2024-09-18T04:38:47.739 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4133000000000004e-05 2024-09-18T04:38:47.740 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.740 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:47.740 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.740 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.740 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.740 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:53 2.4 2:39f7f174:ns::rbd_data.133e813ab5bc.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:47.740 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.584263+0000", 2024-09-18T04:38:47.740 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 5.0493003400000003, 2024-09-18T04:38:47.740 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018701180000000001, 2024-09-18T04:38:47.740 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:47.740 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:47.740 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:47.740 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:38:47.741 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:47.741 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 53 2024-09-18T04:38:47.741 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.741 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:47.741 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.741 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:47.741 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584263+0000", 2024-09-18T04:38:47.741 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.741 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.741 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.741 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:47.741 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584263+0000", 2024-09-18T04:38:47.741 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.742 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.742 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.742 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:47.742 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584262+0000", 2024-09-18T04:38:47.742 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:47.742 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.742 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.742 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:47.742 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.591617+0000", 2024-09-18T04:38:47.742 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0073550600000000001 2024-09-18T04:38:47.742 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.742 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.742 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:47.742 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.591637+0000", 2024-09-18T04:38:47.743 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0432999999999999e-05 2024-09-18T04:38:47.743 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.743 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.743 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:47.743 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.591646+0000", 2024-09-18T04:38:47.743 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7849999999999997e-06 2024-09-18T04:38:47.743 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.743 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.743 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:47.743 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593478+0000", 2024-09-18T04:38:47.743 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001831495 2024-09-18T04:38:47.743 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.743 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.744 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:47.744 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593661+0000", 2024-09-18T04:38:47.744 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018364999999999999 2024-09-18T04:38:47.744 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.744 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.744 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:47.744 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593834+0000", 2024-09-18T04:38:47.744 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000173065 2024-09-18T04:38:47.744 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.744 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.744 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:47.744 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.600670+0000", 2024-09-18T04:38:47.744 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0068354970000000003 2024-09-18T04:38:47.745 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.745 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.745 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:47.745 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602851+0000", 2024-09-18T04:38:47.745 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002181653 2024-09-18T04:38:47.745 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.745 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.745 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:47.745 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602930+0000", 2024-09-18T04:38:47.745 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8045000000000002e-05 2024-09-18T04:38:47.745 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.745 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.745 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:47.746 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602965+0000", 2024-09-18T04:38:47.746 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5102999999999999e-05 2024-09-18T04:38:47.746 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.746 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:47.746 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.746 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.746 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.746 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:56 2.6 2:72453c3e:ns::rbd_data.133e813ab5bc.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:47.746 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.595988+0000", 2024-09-18T04:38:47.746 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 5.0375759379999998, 2024-09-18T04:38:47.746 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.030296172, 2024-09-18T04:38:47.746 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:47.746 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:47.746 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:47.747 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:38:47.747 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:47.747 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 56 2024-09-18T04:38:47.747 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.747 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:47.747 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.747 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:47.747 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.595988+0000", 2024-09-18T04:38:47.747 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.747 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.747 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.747 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:47.747 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.595988+0000", 2024-09-18T04:38:47.747 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:47.748 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.748 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.748 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:47.748 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.595986+0000", 2024-09-18T04:38:47.748 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:47.748 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.748 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.748 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:47.748 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605262+0000", 2024-09-18T04:38:47.748 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0092767240000000001 2024-09-18T04:38:47.748 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.748 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.748 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:47.749 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605282+0000", 2024-09-18T04:38:47.749 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9307e-05 2024-09-18T04:38:47.749 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.749 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.749 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:47.749 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605292+0000", 2024-09-18T04:38:47.749 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9510000000000001e-06 2024-09-18T04:38:47.749 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.749 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.749 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:47.749 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605369+0000", 2024-09-18T04:38:47.749 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7846e-05 2024-09-18T04:38:47.749 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.749 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.750 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:47.750 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605668+0000", 2024-09-18T04:38:47.750 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00029896099999999999 2024-09-18T04:38:47.750 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.750 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.750 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:38:47.750 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605906+0000", 2024-09-18T04:38:47.750 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000237758 2024-09-18T04:38:47.750 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.750 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.750 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:47.750 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.613532+0000", 2024-09-18T04:38:47.750 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0076260590000000001 2024-09-18T04:38:47.750 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.751 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.751 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:47.751 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.626116+0000", 2024-09-18T04:38:47.751 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012584092999999999 2024-09-18T04:38:47.751 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.751 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.751 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:47.751 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.626216+0000", 2024-09-18T04:38:47.751 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9630999999999995e-05 2024-09-18T04:38:47.751 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:47.751 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:47.751 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:47.751 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.626284+0000", 2024-09-18T04:38:47.752 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8034999999999995e-05 2024-09-18T04:38:47.752 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.752 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:47.752 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.752 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:47.752 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:47.752 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:47.754 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:38:47.754 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:47.746+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:47.854 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:47.849+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:47.867 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T04:38:47.916 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:47.917 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:47.917 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:38:47.917 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:47.927 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:38:47.955 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:47.950+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:47.987 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-09-18T04:38:48.014 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:38:48.031 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:48.031 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:48.031 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:38:48.031 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:38:48.031 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:48.041 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:38:48.042 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:38:48.064 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:38:48.065 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:48.058+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:48.090 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:38:48.112 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:38:48.142 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:38:48.144 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:38:48.145 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:38:48.145 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:38:48.145 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:38:48.145 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:38:48.145 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:38:48.157 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:48.149+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:48.167 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:48.167 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:48.177 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:38:48.258 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:48.254+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:48.276 INFO:tasks.workunit.client.0.smithi115.stderr:rbd-nbd: invalid device path: INVALIDDEV (expected /dev/nbd{num}) 2024-09-18T04:38:48.290 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:38:48.290 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false rbd device --device-type nbd --format INVALID list 2024-09-18T04:38:48.290 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd device --device-type nbd --format INVALID list 2024-09-18T04:38:48.293 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:48.293 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:48.303 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:38:48.325 INFO:tasks.workunit.client.0.smithi115.stderr:rbd: the argument for option '--format' is invalid 2024-09-18T04:38:48.326 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:38:48.327 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd device --device-type nbd --format json --pretty-format list 2024-09-18T04:38:48.359 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:48.354+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:48.395 INFO:tasks.workunit.client.0.smithi115.stdout:[] 2024-09-18T04:38:48.399 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd device --device-type nbd --format xml list 2024-09-18T04:38:48.408 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:48.408 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:48.418 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:38:48.460 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:48.453+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:48.475 INFO:tasks.workunit.client.0.smithi115.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd17342 2024-09-18T04:38:48.475 INFO:tasks.workunit.client.0.smithi115.stderr:++ local cmd 2024-09-18T04:38:48.475 INFO:tasks.workunit.client.0.smithi115.stderr:+++ id -u 2024-09-18T04:38:48.477 INFO:tasks.workunit.client.0.smithi115.stderr:++ '[' 1000 -eq 0 ']' 2024-09-18T04:38:48.477 INFO:tasks.workunit.client.0.smithi115.stderr:+++ which rbd 2024-09-18T04:38:48.479 INFO:tasks.workunit.client.0.smithi115.stderr:++ cmd=/usr/bin/rbd 2024-09-18T04:38:48.479 INFO:tasks.workunit.client.0.smithi115.stderr:++ shift 2024-09-18T04:38:48.479 INFO:tasks.workunit.client.0.smithi115.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd17342 2024-09-18T04:38:48.526 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:48.526 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:48.535 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:38:48.561 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:48.554+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:48.637 INFO:tasks.workunit.client.0.smithi115.stderr:+ DEV=/dev/nbd0 2024-09-18T04:38:48.637 INFO:tasks.workunit.client.0.smithi115.stderr:+ get_pid rbd 2024-09-18T04:38:48.637 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pool=rbd 2024-09-18T04:38:48.637 INFO:tasks.workunit.client.0.smithi115.stderr:+ local ns= 2024-09-18T04:38:48.638 INFO:tasks.workunit.client.0.smithi115.stderr:++ rbd device --device-type nbd --format xml list 2024-09-18T04:38:48.638 INFO:tasks.workunit.client.0.smithi115.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd17342'\''][device='\''/dev/nbd0'\'']/id' 2024-09-18T04:38:48.644 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:48.644 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:48.654 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:38:48.661 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:48.658+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:48.734 INFO:tasks.workunit.client.0.smithi115.stderr:+ PID=17640 2024-09-18T04:38:48.735 INFO:tasks.workunit.client.0.smithi115.stderr:+ test -n 17640 2024-09-18T04:38:48.735 INFO:tasks.workunit.client.0.smithi115.stderr:+ ps -p 17640 -C rbd-nbd 2024-09-18T04:38:48.748 INFO:tasks.workunit.client.0.smithi115.stdout: PID TTY TIME CMD 2024-09-18T04:38:48.748 INFO:tasks.workunit.client.0.smithi115.stdout: 17640 ? 00:00:00 rbd-nbd 2024-09-18T04:38:48.749 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false _sudo rbd-nbd --device /dev/nbd0 map rbd/testrbdnbd17342 2024-09-18T04:38:48.749 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo rbd-nbd --device /dev/nbd0 map rbd/testrbdnbd17342 2024-09-18T04:38:48.749 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:38:48.749 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:38:48.750 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:38:48.751 INFO:tasks.workunit.client.0.smithi115.stderr:++ which rbd-nbd 2024-09-18T04:38:48.752 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/bin/rbd-nbd 2024-09-18T04:38:48.753 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:38:48.753 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/bin/rbd-nbd --device /dev/nbd0 map rbd/testrbdnbd17342 2024-09-18T04:38:48.756 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:48.757 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:48.762 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:48.758+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:48.767 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:38:48.863 INFO:tasks.workunit.client.0.smithi115.stderr:rbd-nbd: the device /dev/nbd0 is busy 2024-09-18T04:38:48.863 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:48.858+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:48.873 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:48.874 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:48.877 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:38:48.877 INFO:tasks.workunit.client.0.smithi115.stderr:+ dev1=/dev/nbd0 2024-09-18T04:38:48.877 INFO:tasks.workunit.client.0.smithi115.stderr:+ unmap_device /dev/nbd0 17640 2024-09-18T04:38:48.878 INFO:tasks.workunit.client.0.smithi115.stderr:+ local args=/dev/nbd0 2024-09-18T04:38:48.878 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pid=17640 2024-09-18T04:38:48.878 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-18T04:38:48.878 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:38:48.878 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:38:48.879 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:38:48.879 INFO:tasks.workunit.client.0.smithi115.stderr:++ which rbd 2024-09-18T04:38:48.881 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/bin/rbd 2024-09-18T04:38:48.881 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:38:48.881 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-18T04:38:48.884 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:38:48.965 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:48.957+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:48.982 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false grep '^17640\b' 2024-09-18T04:38:48.982 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd device --device-type nbd list 2024-09-18T04:38:48.983 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep '^17640\b' 2024-09-18T04:38:48.988 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:48.988 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:48.998 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:38:49.042 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:38:49.043 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false grep '^ *17640\b' 2024-09-18T04:38:49.043 INFO:tasks.workunit.client.0.smithi115.stderr:+ ps -C rbd-nbd 2024-09-18T04:38:49.043 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep '^ *17640\b' 2024-09-18T04:38:49.056 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:38:49.056 INFO:tasks.workunit.client.0.smithi115.stderr:+ sleep 0.5 2024-09-18T04:38:49.066 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:49.062+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:49.150 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:49.150 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:49.160 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:38:49.167 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:49.161+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:49.266 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:49.267 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:49.268 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:49.261+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:49.276 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:38:49.370 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:49.361+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:49.397 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:49.397 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:49.409 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:38:49.471 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:49.465+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:49.517 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:49.518 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:49.528 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:38:49.558 INFO:tasks.workunit.client.0.smithi115.stderr:+ DEV= 2024-09-18T04:38:49.558 INFO:tasks.workunit.client.0.smithi115.stderr:++ _sudo rbd-nbd --device /dev/nbd0 map rbd/testrbdnbd17342 2024-09-18T04:38:49.559 INFO:tasks.workunit.client.0.smithi115.stderr:++ local cmd 2024-09-18T04:38:49.559 INFO:tasks.workunit.client.0.smithi115.stderr:+++ id -u 2024-09-18T04:38:49.561 INFO:tasks.workunit.client.0.smithi115.stderr:++ '[' 1000 -eq 0 ']' 2024-09-18T04:38:49.562 INFO:tasks.workunit.client.0.smithi115.stderr:+++ which rbd-nbd 2024-09-18T04:38:49.564 INFO:tasks.workunit.client.0.smithi115.stderr:++ cmd=/usr/bin/rbd-nbd 2024-09-18T04:38:49.564 INFO:tasks.workunit.client.0.smithi115.stderr:++ shift 2024-09-18T04:38:49.565 INFO:tasks.workunit.client.0.smithi115.stderr:++ sudo -nE /usr/bin/rbd-nbd --device /dev/nbd0 map rbd/testrbdnbd17342 2024-09-18T04:38:49.572 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:49.565+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:49.635 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:49.635 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:49.645 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:38:49.673 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:49.666+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:49.683 INFO:tasks.workunit.client.0.smithi115.stderr:+ DEV=/dev/nbd0 2024-09-18T04:38:49.683 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' /dev/nbd0 = /dev/nbd0 ']' 2024-09-18T04:38:49.683 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd device --device-type nbd list 2024-09-18T04:38:49.683 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep testrbdnbd17342 2024-09-18T04:38:49.756 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:49.756 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:49.763 INFO:tasks.workunit.client.0.smithi115.stdout:17738 rbd testrbdnbd17342 - /dev/nbd0 2024-09-18T04:38:49.763 INFO:tasks.workunit.client.0.smithi115.stderr:+ get_pid rbd 2024-09-18T04:38:49.763 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pool=rbd 2024-09-18T04:38:49.764 INFO:tasks.workunit.client.0.smithi115.stderr:+ local ns= 2024-09-18T04:38:49.764 INFO:tasks.workunit.client.0.smithi115.stderr:++ rbd device --device-type nbd --format xml list 2024-09-18T04:38:49.765 INFO:tasks.workunit.client.0.smithi115.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd17342'\''][device='\''/dev/nbd0'\'']/id' 2024-09-18T04:38:49.766 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:38:49.774 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:49.769+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:49.834 INFO:tasks.workunit.client.0.smithi115.stderr:+ PID=17738 2024-09-18T04:38:49.834 INFO:tasks.workunit.client.0.smithi115.stderr:+ test -n 17738 2024-09-18T04:38:49.834 INFO:tasks.workunit.client.0.smithi115.stderr:+ ps -p 17738 -C rbd-nbd 2024-09-18T04:38:49.847 INFO:tasks.workunit.client.0.smithi115.stdout: PID TTY TIME CMD 2024-09-18T04:38:49.847 INFO:tasks.workunit.client.0.smithi115.stdout: 17738 ? 00:00:00 rbd-nbd 2024-09-18T04:38:49.848 INFO:tasks.workunit.client.0.smithi115.stderr:++ md5sum 2024-09-18T04:38:49.848 INFO:tasks.workunit.client.0.smithi115.stderr:++ dd if=/tmp/tmp.WJMNJxIdAN/data bs=1M 2024-09-18T04:38:49.875 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:49.870+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:49.922 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:49.922 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:49.932 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:38:49.976 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:49.970+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:49.978 INFO:tasks.workunit.client.0.smithi115.stderr:64+0 records in 2024-09-18T04:38:49.979 INFO:tasks.workunit.client.0.smithi115.stderr:64+0 records out 2024-09-18T04:38:49.979 INFO:tasks.workunit.client.0.smithi115.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.129284 s, 519 MB/s 2024-09-18T04:38:49.980 INFO:tasks.workunit.client.0.smithi115.stderr:++ _sudo dd if=/dev/nbd0 bs=1M 2024-09-18T04:38:49.980 INFO:tasks.workunit.client.0.smithi115.stderr:++ local cmd 2024-09-18T04:38:49.980 INFO:tasks.workunit.client.0.smithi115.stderr:++ md5sum 2024-09-18T04:38:49.980 INFO:tasks.workunit.client.0.smithi115.stderr:+++ id -u 2024-09-18T04:38:49.982 INFO:tasks.workunit.client.0.smithi115.stderr:++ '[' 1000 -eq 0 ']' 2024-09-18T04:38:49.982 INFO:tasks.workunit.client.0.smithi115.stderr:+++ which dd 2024-09-18T04:38:49.984 INFO:tasks.workunit.client.0.smithi115.stderr:++ cmd=/usr/bin/dd 2024-09-18T04:38:49.984 INFO:tasks.workunit.client.0.smithi115.stderr:++ shift 2024-09-18T04:38:49.984 INFO:tasks.workunit.client.0.smithi115.stderr:++ sudo -nE /usr/bin/dd if=/dev/nbd0 bs=1M 2024-09-18T04:38:50.002 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-18T04:38:50.039 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:50.039 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:50.050 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:38:50.098 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:50.094+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:50.155 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T04:38:50.177 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:50.170+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:50.226 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:50.226 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:50.237 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:38:50.282 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:50.278+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:50.294 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:38:50.321 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:38:50.345 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:38:50.369 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:38:50.378 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:50.373+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:50.393 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:38:50.404 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:50.405 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:50.405 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:38:50.405 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:38:50.405 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:50.414 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:38:50.417 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:38:50.419 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:38:50.419 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:38:50.419 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:38:50.419 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:38:50.419 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:38:50.420 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:38:50.431 INFO:tasks.workunit.client.0.smithi115.stderr:64+0 records in 2024-09-18T04:38:50.432 INFO:tasks.workunit.client.0.smithi115.stderr:64+0 records out 2024-09-18T04:38:50.432 INFO:tasks.workunit.client.0.smithi115.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.438136 s, 153 MB/s 2024-09-18T04:38:50.434 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' '5e881cd3ef5042c5a2c883b42df3b026 -' = '5e881cd3ef5042c5a2c883b42df3b026 -' ']' 2024-09-18T04:38:50.434 INFO:tasks.workunit.client.0.smithi115.stderr:+ dd if=/dev/urandom of=/tmp/tmp.WJMNJxIdAN/data bs=1M count=64 2024-09-18T04:38:50.479 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:50.474+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:50.516 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:50.517 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:38:50.517 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:38:50.517 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:38:50.517 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.517 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:8 2.1 2:9370a598:::rbd_header.125fc09df6d1:head [create,call rbd.create in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:50.517 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.206256+0000", 2024-09-18T04:38:50.517 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 8.3059085679999995, 2024-09-18T04:38:50.517 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030429329999999998, 2024-09-18T04:38:50.517 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:50.517 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:50.517 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:50.518 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:38:50.518 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:38:50.518 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 8 2024-09-18T04:38:50.518 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.518 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:50.518 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.518 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:50.518 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.206256+0000", 2024-09-18T04:38:50.518 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:50.518 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.518 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.518 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:50.518 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.206256+0000", 2024-09-18T04:38:50.518 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:50.519 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.519 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.519 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:50.519 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.206254+0000", 2024-09-18T04:38:50.519 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:50.519 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.519 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.519 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:50.519 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.206266+0000", 2024-09-18T04:38:50.519 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2353e-05 2024-09-18T04:38:50.519 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.519 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.519 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:50.520 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.206287+0000", 2024-09-18T04:38:50.520 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0942999999999999e-05 2024-09-18T04:38:50.520 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.520 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.520 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:50.520 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.206304+0000", 2024-09-18T04:38:50.520 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6782999999999998e-05 2024-09-18T04:38:50.520 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.520 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.520 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:50.520 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.206418+0000", 2024-09-18T04:38:50.520 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011377700000000001 2024-09-18T04:38:50.520 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.520 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.521 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:50.521 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.206696+0000", 2024-09-18T04:38:50.521 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000278029 2024-09-18T04:38:50.521 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.521 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.521 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:50.521 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.207055+0000", 2024-09-18T04:38:50.521 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00035913200000000002 2024-09-18T04:38:50.521 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.521 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.521 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:50.521 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.208686+0000", 2024-09-18T04:38:50.521 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0016314039999999999 2024-09-18T04:38:50.522 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.522 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.522 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:50.522 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.209128+0000", 2024-09-18T04:38:50.522 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00044201400000000001 2024-09-18T04:38:50.522 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.522 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.522 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:50.522 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.209208+0000", 2024-09-18T04:38:50.522 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9714000000000003e-05 2024-09-18T04:38:50.522 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.524 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.524 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:50.524 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.209299+0000", 2024-09-18T04:38:50.524 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1263999999999999e-05 2024-09-18T04:38:50.524 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:50.525 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:50.525 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:50.525 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.525 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.525 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:21 2.1 2:9370a598:::rbd_header.125fc09df6d1:head [watch watch cookie 140314165667744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:50.525 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.233053+0000", 2024-09-18T04:38:50.525 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 8.2791116030000005, 2024-09-18T04:38:50.525 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027093289999999999, 2024-09-18T04:38:50.525 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:50.525 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:50.525 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:50.525 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:38:50.525 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:38:50.525 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 21 2024-09-18T04:38:50.526 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.526 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:50.526 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.526 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:50.526 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.233053+0000", 2024-09-18T04:38:50.526 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:50.526 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.526 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.526 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:50.526 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.233053+0000", 2024-09-18T04:38:50.526 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:50.526 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.526 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.526 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:50.527 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.233052+0000", 2024-09-18T04:38:50.527 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:50.527 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.527 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.527 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:50.527 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.233062+0000", 2024-09-18T04:38:50.527 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0336999999999999e-05 2024-09-18T04:38:50.527 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.527 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.527 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:50.527 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.233085+0000", 2024-09-18T04:38:50.527 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2807999999999999e-05 2024-09-18T04:38:50.527 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.528 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.528 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:50.528 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.233095+0000", 2024-09-18T04:38:50.528 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0037000000000001e-05 2024-09-18T04:38:50.528 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.528 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.528 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:50.528 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.233216+0000", 2024-09-18T04:38:50.528 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000121432 2024-09-18T04:38:50.529 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.529 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.529 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:50.529 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.233454+0000", 2024-09-18T04:38:50.529 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000237316 2024-09-18T04:38:50.529 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.529 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.529 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:50.529 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.233866+0000", 2024-09-18T04:38:50.529 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00041284900000000002 2024-09-18T04:38:50.529 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.529 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.529 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:50.529 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.235033+0000", 2024-09-18T04:38:50.530 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0011666300000000001 2024-09-18T04:38:50.530 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.530 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.530 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:50.530 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.235655+0000", 2024-09-18T04:38:50.530 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000621866 2024-09-18T04:38:50.530 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.530 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.530 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:50.530 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.235702+0000", 2024-09-18T04:38:50.530 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6785e-05 2024-09-18T04:38:50.530 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.530 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.531 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:50.531 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.235763+0000", 2024-09-18T04:38:50.531 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0809e-05 2024-09-18T04:38:50.531 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:50.531 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:50.531 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:50.531 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.531 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.531 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:33 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:50.531 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.261182+0000", 2024-09-18T04:38:50.531 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 8.2509823400000002, 2024-09-18T04:38:50.531 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.037082208999999998, 2024-09-18T04:38:50.531 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:50.531 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:50.532 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:50.532 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:38:50.532 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:38:50.532 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 33 2024-09-18T04:38:50.532 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.532 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:50.532 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.532 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:50.532 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.261182+0000", 2024-09-18T04:38:50.532 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:50.532 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.532 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.532 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:50.532 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.261182+0000", 2024-09-18T04:38:50.533 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:50.533 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.533 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.533 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:50.533 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.261180+0000", 2024-09-18T04:38:50.533 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:50.533 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.533 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.533 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:50.533 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278669+0000", 2024-09-18T04:38:50.533 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.017488877999999999 2024-09-18T04:38:50.533 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.533 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.534 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:50.534 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278689+0000", 2024-09-18T04:38:50.534 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0047999999999998e-05 2024-09-18T04:38:50.534 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.534 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.534 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:50.534 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278699+0000", 2024-09-18T04:38:50.534 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6800000000000005e-06 2024-09-18T04:38:50.534 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.534 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.534 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:50.534 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278775+0000", 2024-09-18T04:38:50.534 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6087999999999994e-05 2024-09-18T04:38:50.534 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.535 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.535 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:50.535 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278935+0000", 2024-09-18T04:38:50.535 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000159834 2024-09-18T04:38:50.535 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.535 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.535 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:50.535 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279064+0000", 2024-09-18T04:38:50.535 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012954199999999999 2024-09-18T04:38:50.535 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.535 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.535 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:50.535 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295562+0000", 2024-09-18T04:38:50.535 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016497583999999999 2024-09-18T04:38:50.536 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.536 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.536 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:50.536 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.298147+0000", 2024-09-18T04:38:50.536 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002584982 2024-09-18T04:38:50.536 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.536 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.536 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:50.536 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.298217+0000", 2024-09-18T04:38:50.536 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0400000000000004e-05 2024-09-18T04:38:50.536 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.536 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.536 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:50.537 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.298265+0000", 2024-09-18T04:38:50.537 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7580000000000002e-05 2024-09-18T04:38:50.537 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:50.537 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:50.537 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:50.537 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.537 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.537 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:34 2.3 e51/24 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head v 51'3, mlcod=18'1)", 2024-09-18T04:38:50.537 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.273067+0000", 2024-09-18T04:38:50.537 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 8.2390975500000003, 2024-09-18T04:38:50.537 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.021784103999999999, 2024-09-18T04:38:50.537 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:50.537 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:50.538 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:50.538 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.538 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:50.538 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.273067+0000", 2024-09-18T04:38:50.538 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:50.538 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.538 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.538 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:50.538 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.273067+0000", 2024-09-18T04:38:50.538 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:50.538 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.538 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.538 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:50.538 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.273060+0000", 2024-09-18T04:38:50.538 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999924 2024-09-18T04:38:50.539 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.539 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.539 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:50.539 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278596+0000", 2024-09-18T04:38:50.539 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055365839999999998 2024-09-18T04:38:50.539 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.539 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.539 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:50.539 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278637+0000", 2024-09-18T04:38:50.539 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0596000000000002e-05 2024-09-18T04:38:50.539 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.539 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.539 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:50.539 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278653+0000", 2024-09-18T04:38:50.540 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6626999999999999e-05 2024-09-18T04:38:50.540 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.540 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.540 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:50.540 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278748+0000", 2024-09-18T04:38:50.540 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4483999999999999e-05 2024-09-18T04:38:50.540 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.540 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.540 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:50.540 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278799+0000", 2024-09-18T04:38:50.540 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1484999999999999e-05 2024-09-18T04:38:50.540 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.540 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.540 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:50.541 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.294785+0000", 2024-09-18T04:38:50.541 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015985256999999999 2024-09-18T04:38:50.541 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.541 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.541 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:50.541 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.294851+0000", 2024-09-18T04:38:50.541 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6728000000000002e-05 2024-09-18T04:38:50.541 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:50.541 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:50.541 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:50.541 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.541 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.541 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:37 2.5 2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:50.542 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.278722+0000", 2024-09-18T04:38:50.542 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 8.2334425010000007, 2024-09-18T04:38:50.542 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.026323801000000001, 2024-09-18T04:38:50.542 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:50.542 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:50.542 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:50.542 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:38:50.542 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:38:50.542 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 37 2024-09-18T04:38:50.542 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.542 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:50.542 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.542 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:50.542 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278722+0000", 2024-09-18T04:38:50.542 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:50.543 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.543 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.543 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:50.543 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278722+0000", 2024-09-18T04:38:50.543 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:50.543 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.543 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.543 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:50.543 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278720+0000", 2024-09-18T04:38:50.543 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:50.543 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.543 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.543 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:50.543 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.291875+0000", 2024-09-18T04:38:50.544 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013154414 2024-09-18T04:38:50.544 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.544 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.544 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:50.544 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.291901+0000", 2024-09-18T04:38:50.544 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6339999999999999e-05 2024-09-18T04:38:50.544 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.544 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.544 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:50.544 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.291914+0000", 2024-09-18T04:38:50.544 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3043e-05 2024-09-18T04:38:50.544 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.544 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.544 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:50.544 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.291975+0000", 2024-09-18T04:38:50.545 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0949999999999998e-05 2024-09-18T04:38:50.545 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.545 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.545 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:50.545 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292173+0000", 2024-09-18T04:38:50.545 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019825499999999999 2024-09-18T04:38:50.545 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.545 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.545 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:50.545 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292300+0000", 2024-09-18T04:38:50.545 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000126616 2024-09-18T04:38:50.545 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.545 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.545 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:50.546 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304334+0000", 2024-09-18T04:38:50.546 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012033934 2024-09-18T04:38:50.546 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.546 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.546 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:50.546 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304901+0000", 2024-09-18T04:38:50.546 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00056694099999999995 2024-09-18T04:38:50.546 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.546 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.546 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:50.546 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304981+0000", 2024-09-18T04:38:50.546 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9974000000000004e-05 2024-09-18T04:38:50.546 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.546 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.547 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:50.547 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.305046+0000", 2024-09-18T04:38:50.547 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5184000000000005e-05 2024-09-18T04:38:50.547 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:50.547 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:50.547 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:50.547 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.547 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.547 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:49 2.5 2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:50.547 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.314271+0000", 2024-09-18T04:38:50.547 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 8.1978938990000003, 2024-09-18T04:38:50.547 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.029779953000000001, 2024-09-18T04:38:50.547 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:50.547 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:50.548 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:50.548 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:38:50.548 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:38:50.548 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 49 2024-09-18T04:38:50.548 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.548 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:50.548 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.548 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:50.548 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.314271+0000", 2024-09-18T04:38:50.548 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:50.548 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.548 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.548 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:50.548 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.314271+0000", 2024-09-18T04:38:50.549 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:50.549 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.549 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.549 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:50.549 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.314269+0000", 2024-09-18T04:38:50.549 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:50.549 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.549 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.549 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:50.549 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.324780+0000", 2024-09-18T04:38:50.549 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010511609999999999 2024-09-18T04:38:50.549 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.549 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.549 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:50.549 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.324821+0000", 2024-09-18T04:38:50.550 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0899000000000003e-05 2024-09-18T04:38:50.550 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.550 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.550 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:50.550 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.324840+0000", 2024-09-18T04:38:50.550 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8947e-05 2024-09-18T04:38:50.550 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.550 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.550 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:50.550 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.324936+0000", 2024-09-18T04:38:50.550 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6342999999999996e-05 2024-09-18T04:38:50.550 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.550 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.550 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:50.550 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325284+0000", 2024-09-18T04:38:50.551 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000347146 2024-09-18T04:38:50.551 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.551 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.551 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:50.551 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325586+0000", 2024-09-18T04:38:50.551 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00030199799999999999 2024-09-18T04:38:50.551 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.551 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.551 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:50.551 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343710+0000", 2024-09-18T04:38:50.551 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018124583999999999 2024-09-18T04:38:50.551 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.551 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.551 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:50.552 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343991+0000", 2024-09-18T04:38:50.552 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00028085600000000001 2024-09-18T04:38:50.552 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.552 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.552 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:50.552 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344028+0000", 2024-09-18T04:38:50.552 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6637000000000002e-05 2024-09-18T04:38:50.552 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.552 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.552 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:50.553 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344051+0000", 2024-09-18T04:38:50.553 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3260000000000001e-05 2024-09-18T04:38:50.553 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:50.553 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:50.553 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:50.553 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.553 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.553 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:50 2.3 e51/24 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head v 51'4, mlcod=51'2)", 2024-09-18T04:38:50.553 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.323066+0000", 2024-09-18T04:38:50.553 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 8.1890985060000006, 2024-09-18T04:38:50.553 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.022135774, 2024-09-18T04:38:50.553 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:50.553 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:50.554 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:50.554 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.554 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:50.554 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.323066+0000", 2024-09-18T04:38:50.554 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:50.554 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.554 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.554 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:50.554 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.323066+0000", 2024-09-18T04:38:50.554 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:50.554 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.554 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.554 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:50.554 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.323065+0000", 2024-09-18T04:38:50.554 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:50.555 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.555 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.555 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:50.555 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.328664+0000", 2024-09-18T04:38:50.555 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055993600000000003 2024-09-18T04:38:50.555 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.555 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.555 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:50.555 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.328692+0000", 2024-09-18T04:38:50.555 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7654e-05 2024-09-18T04:38:50.555 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.555 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.555 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:50.555 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.328706+0000", 2024-09-18T04:38:50.555 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4097000000000001e-05 2024-09-18T04:38:50.556 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.556 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.556 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:50.556 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.328816+0000", 2024-09-18T04:38:50.556 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000110281 2024-09-18T04:38:50.556 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.556 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.556 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:50.556 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.328848+0000", 2024-09-18T04:38:50.556 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1837e-05 2024-09-18T04:38:50.556 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.556 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.556 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:50.556 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345160+0000", 2024-09-18T04:38:50.556 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016312099 2024-09-18T04:38:50.557 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.557 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.557 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:50.557 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345202+0000", 2024-09-18T04:38:50.557 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1857999999999998e-05 2024-09-18T04:38:50.557 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:50.557 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:50.557 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:50.557 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.557 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.557 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:58 2.5 2:b227e6b1:::rbd_data.125fc09df6d1.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:50.557 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.351564+0000", 2024-09-18T04:38:50.557 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 8.1606010540000007, 2024-09-18T04:38:50.557 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.032125956999999997, 2024-09-18T04:38:50.557 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:50.558 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:50.558 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:50.558 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:38:50.558 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:38:50.558 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 58 2024-09-18T04:38:50.558 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.558 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:50.558 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.558 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:50.558 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.351564+0000", 2024-09-18T04:38:50.558 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:50.558 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.558 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.558 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:50.558 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.351564+0000", 2024-09-18T04:38:50.559 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:50.559 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.559 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.559 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:50.559 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.351561+0000", 2024-09-18T04:38:50.559 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:50.559 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.559 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.559 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:50.559 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364390+0000", 2024-09-18T04:38:50.559 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012829063999999999 2024-09-18T04:38:50.559 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.559 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.559 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:50.560 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364418+0000", 2024-09-18T04:38:50.560 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7307000000000001e-05 2024-09-18T04:38:50.560 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.560 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.560 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:50.560 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364428+0000", 2024-09-18T04:38:50.560 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0675e-05 2024-09-18T04:38:50.560 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.560 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.560 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:50.560 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364523+0000", 2024-09-18T04:38:50.560 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4698999999999996e-05 2024-09-18T04:38:50.560 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.561 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.561 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:50.561 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364785+0000", 2024-09-18T04:38:50.561 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00026170099999999998 2024-09-18T04:38:50.561 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.561 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.561 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:50.561 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.365039+0000", 2024-09-18T04:38:50.561 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00025462299999999999 2024-09-18T04:38:50.561 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.561 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.561 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:50.561 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383143+0000", 2024-09-18T04:38:50.561 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018103768999999999 2024-09-18T04:38:50.561 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.562 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.562 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:50.562 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383582+0000", 2024-09-18T04:38:50.562 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00043860600000000002 2024-09-18T04:38:50.562 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.562 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.562 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:50.562 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383644+0000", 2024-09-18T04:38:50.562 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2682999999999996e-05 2024-09-18T04:38:50.562 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.562 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.562 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:50.562 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383690+0000", 2024-09-18T04:38:50.563 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5247999999999998e-05 2024-09-18T04:38:50.563 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:50.563 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:50.563 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:50.563 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.563 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.563 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:33 2.1 2:8b40f1c5:ns::rbd_data.133e813ab5bc.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:50.563 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.523765+0000", 2024-09-18T04:38:50.563 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 7.9884000139999998, 2024-09-18T04:38:50.563 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027472973000000001, 2024-09-18T04:38:50.563 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:50.563 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:50.563 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:50.563 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:38:50.563 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:50.564 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 33 2024-09-18T04:38:50.564 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.565 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:50.565 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.565 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:50.565 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.523765+0000", 2024-09-18T04:38:50.565 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:50.565 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.565 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.565 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:50.565 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.523765+0000", 2024-09-18T04:38:50.565 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:50.565 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.566 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.566 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:50.566 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.523762+0000", 2024-09-18T04:38:50.566 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:50.566 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.566 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.566 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:50.566 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531127+0000", 2024-09-18T04:38:50.566 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0073644560000000001 2024-09-18T04:38:50.567 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.567 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.567 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:50.567 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531189+0000", 2024-09-18T04:38:50.567 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2315000000000002e-05 2024-09-18T04:38:50.567 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.567 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.567 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:50.567 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531211+0000", 2024-09-18T04:38:50.567 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1542000000000001e-05 2024-09-18T04:38:50.567 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.567 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.567 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:50.568 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531314+0000", 2024-09-18T04:38:50.568 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000103582 2024-09-18T04:38:50.568 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.568 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.568 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:50.568 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531493+0000", 2024-09-18T04:38:50.568 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000178433 2024-09-18T04:38:50.568 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.568 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.568 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:50.568 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531653+0000", 2024-09-18T04:38:50.568 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000160388 2024-09-18T04:38:50.569 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.569 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.570 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:50.570 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.547345+0000", 2024-09-18T04:38:50.570 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015691742000000002 2024-09-18T04:38:50.570 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.570 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.570 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:50.570 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551076+0000", 2024-09-18T04:38:50.570 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003730957 2024-09-18T04:38:50.570 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.570 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.570 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:50.571 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551176+0000", 2024-09-18T04:38:50.571 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010002599999999999 2024-09-18T04:38:50.571 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.571 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.571 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:50.571 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551238+0000", 2024-09-18T04:38:50.571 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1945000000000004e-05 2024-09-18T04:38:50.571 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:50.571 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:50.571 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:50.571 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.572 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.572 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:32 2.3 e51/24 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head v 51'10, mlcod=51'7)", 2024-09-18T04:38:50.572 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.524206+0000", 2024-09-18T04:38:50.572 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 7.987958796, 2024-09-18T04:38:50.572 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0037857780000000001, 2024-09-18T04:38:50.572 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:50.572 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:50.572 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:50.572 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.572 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:50.572 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524206+0000", 2024-09-18T04:38:50.573 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:50.573 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.573 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.573 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:50.573 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524206+0000", 2024-09-18T04:38:50.573 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:50.573 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.573 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.573 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:50.574 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524205+0000", 2024-09-18T04:38:50.574 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:50.574 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.574 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.574 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:50.574 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524229+0000", 2024-09-18T04:38:50.574 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4182999999999999e-05 2024-09-18T04:38:50.574 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.574 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.574 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:50.575 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524265+0000", 2024-09-18T04:38:50.575 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6032000000000003e-05 2024-09-18T04:38:50.575 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.575 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.575 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:50.575 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524282+0000", 2024-09-18T04:38:50.575 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6773e-05 2024-09-18T04:38:50.575 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.575 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.575 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:50.575 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524439+0000", 2024-09-18T04:38:50.575 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015771900000000001 2024-09-18T04:38:50.576 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.576 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.576 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:50.576 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524484+0000", 2024-09-18T04:38:50.576 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4465000000000002e-05 2024-09-18T04:38:50.576 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.576 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.576 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:50.576 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.527910+0000", 2024-09-18T04:38:50.576 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0034260860000000001 2024-09-18T04:38:50.576 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.576 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.577 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:50.577 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.527992+0000", 2024-09-18T04:38:50.577 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1934000000000005e-05 2024-09-18T04:38:50.577 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:50.577 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:50.577 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:50.577 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.577 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.577 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:35 2.3 e51/24 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head v 51'11, mlcod=51'7)", 2024-09-18T04:38:50.577 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.525811+0000", 2024-09-18T04:38:50.577 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 7.9863542719999998, 2024-09-18T04:38:50.578 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028483979999999998, 2024-09-18T04:38:50.578 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:50.578 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:50.578 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:50.578 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.578 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:50.578 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.525811+0000", 2024-09-18T04:38:50.578 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:50.578 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.578 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.578 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:50.578 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.525811+0000", 2024-09-18T04:38:50.579 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:50.579 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.579 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.579 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:50.579 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.525802+0000", 2024-09-18T04:38:50.579 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999919 2024-09-18T04:38:50.579 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.579 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.579 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:50.579 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.525828+0000", 2024-09-18T04:38:50.579 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5602999999999999e-05 2024-09-18T04:38:50.580 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.580 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.580 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:50.580 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.525861+0000", 2024-09-18T04:38:50.580 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2916000000000001e-05 2024-09-18T04:38:50.580 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.581 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.581 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:50.581 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.525878+0000", 2024-09-18T04:38:50.581 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6616000000000002e-05 2024-09-18T04:38:50.581 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.581 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.581 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:50.581 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.526706+0000", 2024-09-18T04:38:50.581 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00082842300000000001 2024-09-18T04:38:50.581 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.581 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.581 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:50.582 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.526795+0000", 2024-09-18T04:38:50.582 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9271000000000006e-05 2024-09-18T04:38:50.582 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.582 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.582 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:50.582 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.528510+0000", 2024-09-18T04:38:50.582 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001715034 2024-09-18T04:38:50.582 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.582 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.582 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:50.582 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.528659+0000", 2024-09-18T04:38:50.582 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000148589 2024-09-18T04:38:50.583 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:50.583 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:50.583 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:50.583 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.583 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.583 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:36 2.5 2:bd5b8dd8:ns::rbd_data.133e813ab5bc.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:50.583 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.531253+0000", 2024-09-18T04:38:50.583 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 7.9809117050000005, 2024-09-18T04:38:50.583 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027295212999999999, 2024-09-18T04:38:50.583 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:50.583 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:50.583 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:50.584 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:38:50.584 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:50.584 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 36 2024-09-18T04:38:50.584 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.584 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:50.584 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.584 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:50.584 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531253+0000", 2024-09-18T04:38:50.584 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:50.584 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.585 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.585 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:50.585 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531253+0000", 2024-09-18T04:38:50.585 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:50.585 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.585 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.586 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:50.586 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531251+0000", 2024-09-18T04:38:50.586 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:50.586 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.586 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.586 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:50.587 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536540+0000", 2024-09-18T04:38:50.587 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0052885839999999998 2024-09-18T04:38:50.587 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.587 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.587 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:50.587 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536604+0000", 2024-09-18T04:38:50.587 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4824000000000001e-05 2024-09-18T04:38:50.587 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.587 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.587 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:50.587 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536621+0000", 2024-09-18T04:38:50.587 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6215999999999999e-05 2024-09-18T04:38:50.588 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.588 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.588 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:50.588 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536768+0000", 2024-09-18T04:38:50.588 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014767900000000001 2024-09-18T04:38:50.588 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.588 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.588 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:50.588 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537084+0000", 2024-09-18T04:38:50.588 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00031553499999999998 2024-09-18T04:38:50.588 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.588 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.589 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:50.589 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537300+0000", 2024-09-18T04:38:50.589 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000216145 2024-09-18T04:38:50.589 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.589 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.589 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:50.589 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555326+0000", 2024-09-18T04:38:50.589 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018025861000000001 2024-09-18T04:38:50.589 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.589 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.590 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:50.590 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558449+0000", 2024-09-18T04:38:50.590 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0031229750000000001 2024-09-18T04:38:50.590 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.590 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.590 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:50.590 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558502+0000", 2024-09-18T04:38:50.590 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3572000000000001e-05 2024-09-18T04:38:50.590 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.590 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.590 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:50.590 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558548+0000", 2024-09-18T04:38:50.590 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5921999999999998e-05 2024-09-18T04:38:50.591 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:50.591 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:50.591 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:50.591 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.591 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.591 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:37 2.5 2:bf39b8e4:ns::rbd_data.133e813ab5bc.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:50.591 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.536664+0000", 2024-09-18T04:38:50.591 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 7.9755002890000002, 2024-09-18T04:38:50.591 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.026331765, 2024-09-18T04:38:50.591 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:50.591 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:50.591 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:50.592 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:38:50.592 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:50.592 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 37 2024-09-18T04:38:50.592 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.592 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:50.592 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.592 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:50.592 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536664+0000", 2024-09-18T04:38:50.592 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:50.592 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.592 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.592 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:50.592 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536664+0000", 2024-09-18T04:38:50.593 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:50.593 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.593 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.593 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:50.593 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536662+0000", 2024-09-18T04:38:50.593 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:50.593 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.593 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.593 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:50.593 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.543699+0000", 2024-09-18T04:38:50.593 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0070367119999999997 2024-09-18T04:38:50.594 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.594 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.594 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:50.594 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.543749+0000", 2024-09-18T04:38:50.594 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0476999999999998e-05 2024-09-18T04:38:50.594 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.594 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.594 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:50.594 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.543800+0000", 2024-09-18T04:38:50.595 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0822999999999998e-05 2024-09-18T04:38:50.595 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.595 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.595 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:50.595 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.547868+0000", 2024-09-18T04:38:50.595 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0040679339999999996 2024-09-18T04:38:50.595 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.595 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.595 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:50.595 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548087+0000", 2024-09-18T04:38:50.595 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000218994 2024-09-18T04:38:50.596 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.596 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.596 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:50.596 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548383+0000", 2024-09-18T04:38:50.596 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000295588 2024-09-18T04:38:50.596 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.596 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.596 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:50.596 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562622+0000", 2024-09-18T04:38:50.596 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014239915000000001 2024-09-18T04:38:50.597 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.597 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.597 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:50.597 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562924+0000", 2024-09-18T04:38:50.597 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00030127600000000001 2024-09-18T04:38:50.597 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.597 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.597 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:50.597 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562957+0000", 2024-09-18T04:38:50.597 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3365999999999999e-05 2024-09-18T04:38:50.597 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.597 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.597 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:50.598 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562996+0000", 2024-09-18T04:38:50.598 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9153999999999998e-05 2024-09-18T04:38:50.598 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:50.598 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:50.598 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:50.598 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.598 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.598 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:42 2.3 e51/24 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head v 51'14, mlcod=51'11)", 2024-09-18T04:38:50.598 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.554081+0000", 2024-09-18T04:38:50.598 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 7.958084028, 2024-09-18T04:38:50.598 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0080081739999999998, 2024-09-18T04:38:50.599 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:50.599 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:50.599 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:50.599 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.599 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:50.599 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.554081+0000", 2024-09-18T04:38:50.599 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:50.599 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.599 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.599 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:50.599 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.554081+0000", 2024-09-18T04:38:50.599 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:50.599 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.599 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.600 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:50.600 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.554079+0000", 2024-09-18T04:38:50.600 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:50.600 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.600 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.600 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:50.600 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.554092+0000", 2024-09-18T04:38:50.600 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3032e-05 2024-09-18T04:38:50.600 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.600 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.600 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:50.600 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.554105+0000", 2024-09-18T04:38:50.601 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3217000000000001e-05 2024-09-18T04:38:50.601 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.601 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.601 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:50.601 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.554115+0000", 2024-09-18T04:38:50.601 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9089999999999995e-06 2024-09-18T04:38:50.601 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.601 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.601 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:50.601 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.554165+0000", 2024-09-18T04:38:50.601 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9644000000000001e-05 2024-09-18T04:38:50.602 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.602 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.602 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:50.602 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.554188+0000", 2024-09-18T04:38:50.602 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3753000000000001e-05 2024-09-18T04:38:50.602 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.602 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.602 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:50.602 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562047+0000", 2024-09-18T04:38:50.602 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0078585519999999996 2024-09-18T04:38:50.602 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.603 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.603 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:50.603 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562089+0000", 2024-09-18T04:38:50.603 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1940000000000002e-05 2024-09-18T04:38:50.603 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:50.604 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:50.604 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:50.604 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.604 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.604 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:44 2.3 e51/24 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head v 51'15, mlcod=51'12)", 2024-09-18T04:38:50.605 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.559761+0000", 2024-09-18T04:38:50.605 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 7.9524037769999998, 2024-09-18T04:38:50.605 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028584270000000002, 2024-09-18T04:38:50.605 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:50.605 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:50.605 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:50.605 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.606 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:50.606 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.559761+0000", 2024-09-18T04:38:50.606 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:50.606 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.606 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.606 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:50.607 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.559761+0000", 2024-09-18T04:38:50.607 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:50.607 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.607 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.607 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:50.607 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.559760+0000", 2024-09-18T04:38:50.607 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:50.607 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.607 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.607 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:50.608 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.559769+0000", 2024-09-18T04:38:50.608 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0019999999999995e-06 2024-09-18T04:38:50.608 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.608 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.608 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:50.608 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.559782+0000", 2024-09-18T04:38:50.608 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2988e-05 2024-09-18T04:38:50.608 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.608 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.608 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:50.609 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.559789+0000", 2024-09-18T04:38:50.609 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0049999999999998e-06 2024-09-18T04:38:50.609 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.609 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.609 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:50.609 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.559837+0000", 2024-09-18T04:38:50.609 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8219e-05 2024-09-18T04:38:50.609 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.610 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.610 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:50.610 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.559854+0000", 2024-09-18T04:38:50.610 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6843000000000001e-05 2024-09-18T04:38:50.610 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.610 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.610 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:50.610 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562586+0000", 2024-09-18T04:38:50.610 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027315099999999999 2024-09-18T04:38:50.610 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.610 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.610 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:50.611 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562619+0000", 2024-09-18T04:38:50.611 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3612000000000003e-05 2024-09-18T04:38:50.611 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:50.611 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:50.611 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:50.611 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.611 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.611 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:50 2.1 2:8c706374:ns::rbd_data.133e813ab5bc.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:50.611 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.570513+0000", 2024-09-18T04:38:50.611 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 7.9416520070000001, 2024-09-18T04:38:50.612 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.024182995999999998, 2024-09-18T04:38:50.612 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:50.612 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:50.612 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:50.612 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:38:50.612 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:50.612 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 50 2024-09-18T04:38:50.612 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.612 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:50.612 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.612 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:50.612 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570513+0000", 2024-09-18T04:38:50.613 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:50.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.613 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.613 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:50.613 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570513+0000", 2024-09-18T04:38:50.613 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:50.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.613 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.613 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:50.613 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570511+0000", 2024-09-18T04:38:50.613 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:50.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.613 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.613 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:50.614 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.576327+0000", 2024-09-18T04:38:50.614 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0058162530000000004 2024-09-18T04:38:50.614 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.614 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.614 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:50.614 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.576355+0000", 2024-09-18T04:38:50.614 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8030000000000001e-05 2024-09-18T04:38:50.614 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.614 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.614 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:50.614 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.576366+0000", 2024-09-18T04:38:50.614 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1139e-05 2024-09-18T04:38:50.614 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.614 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.615 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:50.615 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.576458+0000", 2024-09-18T04:38:50.615 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1780999999999994e-05 2024-09-18T04:38:50.615 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.615 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.615 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:50.615 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.576701+0000", 2024-09-18T04:38:50.615 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00024303899999999999 2024-09-18T04:38:50.615 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.615 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.615 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:50.615 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.576876+0000", 2024-09-18T04:38:50.615 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000174637 2024-09-18T04:38:50.615 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.616 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.616 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:50.616 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588432+0000", 2024-09-18T04:38:50.616 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011555943000000001 2024-09-18T04:38:50.616 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.616 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.616 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:50.616 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594573+0000", 2024-09-18T04:38:50.616 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0061408670000000004 2024-09-18T04:38:50.616 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.616 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.616 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:50.616 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594640+0000", 2024-09-18T04:38:50.617 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7404999999999996e-05 2024-09-18T04:38:50.617 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.617 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.617 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:50.617 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594696+0000", 2024-09-18T04:38:50.617 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5612999999999998e-05 2024-09-18T04:38:50.617 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:50.617 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:50.617 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:50.617 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.617 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.617 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:52 2.3 e51/24 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head v 51'19, mlcod=51'16)", 2024-09-18T04:38:50.617 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.581089+0000", 2024-09-18T04:38:50.617 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 7.9310761630000002, 2024-09-18T04:38:50.618 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0036266890000000002, 2024-09-18T04:38:50.618 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:50.618 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:50.618 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:50.618 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.618 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:50.618 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581089+0000", 2024-09-18T04:38:50.618 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:50.618 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.618 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.618 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:50.618 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581089+0000", 2024-09-18T04:38:50.618 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:50.618 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.618 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.619 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:50.619 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581087+0000", 2024-09-18T04:38:50.619 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:50.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.619 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.619 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:50.619 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581101+0000", 2024-09-18T04:38:50.619 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.379e-05 2024-09-18T04:38:50.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.619 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.619 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:50.619 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581117+0000", 2024-09-18T04:38:50.619 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.609e-05 2024-09-18T04:38:50.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.619 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.620 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:50.620 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581126+0000", 2024-09-18T04:38:50.620 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2679999999999994e-06 2024-09-18T04:38:50.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.620 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.620 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:50.620 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.583238+0000", 2024-09-18T04:38:50.620 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021123380000000001 2024-09-18T04:38:50.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.620 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.620 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:50.620 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.583265+0000", 2024-09-18T04:38:50.620 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7554000000000001e-05 2024-09-18T04:38:50.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.621 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.621 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:50.621 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584647+0000", 2024-09-18T04:38:50.621 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0013813499999999999 2024-09-18T04:38:50.621 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.621 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.621 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:50.621 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584715+0000", 2024-09-18T04:38:50.621 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8551000000000001e-05 2024-09-18T04:38:50.621 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:50.621 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:50.621 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:50.621 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.622 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.622 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:55 2.3 e51/24 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head v 51'20, mlcod=51'18)", 2024-09-18T04:38:50.622 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.589520+0000", 2024-09-18T04:38:50.622 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 7.9226444120000004, 2024-09-18T04:38:50.622 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0040254549999999998, 2024-09-18T04:38:50.622 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:50.622 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:50.622 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:50.622 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.622 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:50.622 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.589520+0000", 2024-09-18T04:38:50.622 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:50.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.622 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.623 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:50.623 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.589520+0000", 2024-09-18T04:38:50.623 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:50.623 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.623 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.623 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:50.623 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.589519+0000", 2024-09-18T04:38:50.623 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:50.623 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.623 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.623 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:50.623 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.589532+0000", 2024-09-18T04:38:50.624 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3147e-05 2024-09-18T04:38:50.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.624 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.624 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:50.624 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.589549+0000", 2024-09-18T04:38:50.624 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6841999999999999e-05 2024-09-18T04:38:50.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.624 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.625 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:50.625 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.589559+0000", 2024-09-18T04:38:50.625 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6509999999999996e-06 2024-09-18T04:38:50.625 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.626 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.626 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:50.626 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.589644+0000", 2024-09-18T04:38:50.626 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4969000000000006e-05 2024-09-18T04:38:50.626 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.627 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.627 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:50.627 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.589672+0000", 2024-09-18T04:38:50.627 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7759999999999998e-05 2024-09-18T04:38:50.627 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.627 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.627 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:50.627 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593495+0000", 2024-09-18T04:38:50.627 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0038238730000000002 2024-09-18T04:38:50.627 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.628 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.628 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:50.628 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593546+0000", 2024-09-18T04:38:50.628 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0455999999999999e-05 2024-09-18T04:38:50.628 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:50.628 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:50.628 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:50.629 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.629 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.629 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:60 2.5 2:a06f51e4:ns::rbd_data.133e813ab5bc.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:50.629 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.604753+0000", 2024-09-18T04:38:50.629 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 7.9074118880000004, 2024-09-18T04:38:50.629 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014400181, 2024-09-18T04:38:50.630 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:50.630 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:50.630 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:50.630 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:38:50.630 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:50.630 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 60 2024-09-18T04:38:50.630 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.630 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:50.630 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.630 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:50.630 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.604753+0000", 2024-09-18T04:38:50.630 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:50.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.631 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.631 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:50.631 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.604753+0000", 2024-09-18T04:38:50.631 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:50.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.631 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.631 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:50.631 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.604751+0000", 2024-09-18T04:38:50.631 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:50.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.632 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.632 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:50.632 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.608980+0000", 2024-09-18T04:38:50.632 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0042292420000000002 2024-09-18T04:38:50.632 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.632 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.632 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:50.632 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609014+0000", 2024-09-18T04:38:50.632 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3572999999999999e-05 2024-09-18T04:38:50.632 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.633 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.633 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:50.633 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609032+0000", 2024-09-18T04:38:50.633 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8150000000000001e-05 2024-09-18T04:38:50.633 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.633 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.633 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:50.633 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609090+0000", 2024-09-18T04:38:50.633 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8214999999999997e-05 2024-09-18T04:38:50.633 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.633 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.633 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:50.634 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609235+0000", 2024-09-18T04:38:50.634 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014437799999999999 2024-09-18T04:38:50.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.634 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.634 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:50.634 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609360+0000", 2024-09-18T04:38:50.634 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012526099999999999 2024-09-18T04:38:50.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.634 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.634 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:50.634 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.614952+0000", 2024-09-18T04:38:50.635 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055922300000000001 2024-09-18T04:38:50.635 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.635 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.635 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:50.635 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.619070+0000", 2024-09-18T04:38:50.635 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0041178250000000003 2024-09-18T04:38:50.635 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.635 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.635 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:50.635 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.619117+0000", 2024-09-18T04:38:50.635 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7275999999999999e-05 2024-09-18T04:38:50.636 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.636 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.636 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:50.636 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.619153+0000", 2024-09-18T04:38:50.636 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5970000000000003e-05 2024-09-18T04:38:50.636 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:50.636 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:50.636 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:50.636 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.636 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.636 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4778.0:12 2.1 2:9370a598:::rbd_header.125fc09df6d1:head [watch watch cookie 140494621380016] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-18T04:38:50.637 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:47.699107+0000", 2024-09-18T04:38:50.637 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 2.813058195, 2024-09-18T04:38:50.637 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026706960000000002, 2024-09-18T04:38:50.637 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:50.637 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:50.637 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:50.637 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4778", 2024-09-18T04:38:50.637 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/235380645", 2024-09-18T04:38:50.637 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 12 2024-09-18T04:38:50.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.638 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:50.638 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.638 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:50.638 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:47.699107+0000", 2024-09-18T04:38:50.638 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:50.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.638 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.638 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:50.638 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:47.699107+0000", 2024-09-18T04:38:50.638 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:50.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.639 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.639 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:50.639 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:47.699105+0000", 2024-09-18T04:38:50.639 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:50.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.639 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.639 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:50.639 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:47.699113+0000", 2024-09-18T04:38:50.639 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8259999999999993e-06 2024-09-18T04:38:50.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.640 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.640 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:50.640 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:47.699135+0000", 2024-09-18T04:38:50.640 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2503000000000001e-05 2024-09-18T04:38:50.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.640 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.640 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:50.640 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:47.699152+0000", 2024-09-18T04:38:50.640 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6042999999999999e-05 2024-09-18T04:38:50.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.640 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.641 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:50.641 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:47.699261+0000", 2024-09-18T04:38:50.641 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000109343 2024-09-18T04:38:50.641 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.641 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.641 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:50.641 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:47.699383+0000", 2024-09-18T04:38:50.641 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012220699999999999 2024-09-18T04:38:50.641 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.641 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.641 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:50.642 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:47.699605+0000", 2024-09-18T04:38:50.642 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00022173700000000001 2024-09-18T04:38:50.642 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.642 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.642 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:50.642 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:47.700939+0000", 2024-09-18T04:38:50.642 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0013338029999999999 2024-09-18T04:38:50.642 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.642 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.642 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:50.642 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:47.701614+0000", 2024-09-18T04:38:50.642 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00067528100000000001 2024-09-18T04:38:50.642 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.642 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.643 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:50.643 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:47.701678+0000", 2024-09-18T04:38:50.643 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3881999999999996e-05 2024-09-18T04:38:50.643 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:50.643 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:50.643 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:50.643 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:47.701777+0000", 2024-09-18T04:38:50.643 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9520999999999998e-05 2024-09-18T04:38:50.643 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:50.643 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:50.643 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:50.643 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:50.643 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:50.643 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:50.645 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:38:50.645 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:50.642+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:50.745 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:50.738+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:50.761 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:50.761 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:50.761 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:38:50.761 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:50.771 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:38:50.849 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:50.841+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:50.873 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:50.873 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:50.873 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:38:50.873 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:38:50.873 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:50.883 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:38:50.947 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:50.946+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:50.986 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-18T04:38:50.987 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T04:38:50.987 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-18T04:38:50.987 INFO:tasks.thrashosds.thrasher:Setting osd 3 primary_affinity to 1.000000 2024-09-18T04:38:50.987 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-09-18T04:38:50.994 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:50.995 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:38:50.995 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:38:50.995 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:38:50.995 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:50.995 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:50.995 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:38:50.995 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 83.203737473999993, 2024-09-18T04:38:50.995 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017399218000000001, 2024-09-18T04:38:50.995 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:50.995 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:50.995 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:50.995 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:50.996 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:50.996 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 114 2024-09-18T04:38:50.996 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:50.996 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:50.996 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:50.996 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:50.996 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:38:50.996 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:50.996 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:50.996 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:50.996 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:50.996 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:38:50.996 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:50.997 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:50.997 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:50.997 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:50.997 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786900+0000", 2024-09-18T04:38:50.997 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:50.997 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:50.997 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:50.997 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:50.997 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786908+0000", 2024-09-18T04:38:50.997 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8199999999999997e-06 2024-09-18T04:38:50.997 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:50.997 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:50.998 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:50.998 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786925+0000", 2024-09-18T04:38:50.998 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6849000000000001e-05 2024-09-18T04:38:50.998 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:50.998 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:50.998 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:50.998 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786934+0000", 2024-09-18T04:38:50.998 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6299999999999993e-06 2024-09-18T04:38:50.998 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:50.998 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:50.998 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:50.998 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800926+0000", 2024-09-18T04:38:50.999 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013991632 2024-09-18T04:38:50.999 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:50.999 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:50.999 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:50.999 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801060+0000", 2024-09-18T04:38:50.999 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013378199999999999 2024-09-18T04:38:50.999 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:50.999 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:50.999 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:51.000 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801166+0000", 2024-09-18T04:38:51.000 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010607099999999999 2024-09-18T04:38:51.000 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.000 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.000 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:51.000 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803026+0000", 2024-09-18T04:38:51.000 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001860469 2024-09-18T04:38:51.000 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.000 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.000 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:51.000 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804245+0000", 2024-09-18T04:38:51.000 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0012185900000000001 2024-09-18T04:38:51.000 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.001 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.001 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:51.001 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804278+0000", 2024-09-18T04:38:51.001 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3272000000000003e-05 2024-09-18T04:38:51.001 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.001 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.001 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:51.001 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804300+0000", 2024-09-18T04:38:51.001 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2266e-05 2024-09-18T04:38:51.001 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.001 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:51.001 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.001 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.001 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.002 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:51.002 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787566+0000", 2024-09-18T04:38:51.002 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 83.203072515000002, 2024-09-18T04:38:51.002 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017124120999999999, 2024-09-18T04:38:51.002 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:51.002 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:51.002 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:51.002 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:51.002 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:51.002 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 115 2024-09-18T04:38:51.002 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.002 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:51.002 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.002 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:51.003 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787566+0000", 2024-09-18T04:38:51.003 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.003 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.003 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.003 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:51.003 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787566+0000", 2024-09-18T04:38:51.003 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.003 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.003 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.003 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:51.003 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787564+0000", 2024-09-18T04:38:51.003 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:51.003 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.003 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.004 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:51.004 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787751+0000", 2024-09-18T04:38:51.004 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000186873 2024-09-18T04:38:51.004 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.004 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.004 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:51.004 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787772+0000", 2024-09-18T04:38:51.004 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.126e-05 2024-09-18T04:38:51.004 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.004 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.004 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:51.004 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787794+0000", 2024-09-18T04:38:51.004 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1418999999999999e-05 2024-09-18T04:38:51.004 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.005 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.005 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:51.005 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801659+0000", 2024-09-18T04:38:51.005 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013865125000000001 2024-09-18T04:38:51.005 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.005 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.005 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:51.005 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801750+0000", 2024-09-18T04:38:51.005 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.1427000000000004e-05 2024-09-18T04:38:51.005 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.005 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.005 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:51.005 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801894+0000", 2024-09-18T04:38:51.005 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000143514 2024-09-18T04:38:51.005 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.006 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.006 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:51.006 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804313+0000", 2024-09-18T04:38:51.006 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0024190879999999998 2024-09-18T04:38:51.006 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.006 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.006 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:51.006 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804644+0000", 2024-09-18T04:38:51.006 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00033100799999999999 2024-09-18T04:38:51.006 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.006 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.006 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:51.006 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804672+0000", 2024-09-18T04:38:51.006 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7988e-05 2024-09-18T04:38:51.007 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.007 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.007 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:51.007 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804690+0000", 2024-09-18T04:38:51.007 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8298999999999999e-05 2024-09-18T04:38:51.007 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.007 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:51.007 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.007 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.007 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.007 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:51.007 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:38:51.007 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 83.202824575999998, 2024-09-18T04:38:51.007 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017521662, 2024-09-18T04:38:51.007 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:51.008 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:51.008 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:51.008 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:51.008 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:51.008 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 116 2024-09-18T04:38:51.008 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.008 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:51.008 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.008 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:51.008 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:38:51.008 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.008 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.008 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.008 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:51.009 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:38:51.009 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.009 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.009 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.009 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:51.009 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787813+0000", 2024-09-18T04:38:51.009 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:51.009 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.009 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.009 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:51.009 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787824+0000", 2024-09-18T04:38:51.009 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1157e-05 2024-09-18T04:38:51.009 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.010 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.010 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:51.010 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787840+0000", 2024-09-18T04:38:51.010 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6269e-05 2024-09-18T04:38:51.010 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.010 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.010 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:51.010 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787853+0000", 2024-09-18T04:38:51.010 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3156000000000001e-05 2024-09-18T04:38:51.010 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.010 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.010 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:51.010 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803110+0000", 2024-09-18T04:38:51.010 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015256736 2024-09-18T04:38:51.011 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.011 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.011 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:51.011 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803197+0000", 2024-09-18T04:38:51.011 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7195999999999997e-05 2024-09-18T04:38:51.011 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.011 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.011 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:51.011 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803308+0000", 2024-09-18T04:38:51.011 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111005 2024-09-18T04:38:51.011 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.011 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.011 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:51.011 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804316+0000", 2024-09-18T04:38:51.012 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010073949999999999 2024-09-18T04:38:51.012 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.012 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.012 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:51.012 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805232+0000", 2024-09-18T04:38:51.012 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091636300000000001 2024-09-18T04:38:51.012 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.012 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.012 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:51.012 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805277+0000", 2024-09-18T04:38:51.012 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4864000000000003e-05 2024-09-18T04:38:51.012 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.012 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.012 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:51.013 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805336+0000", 2024-09-18T04:38:51.013 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8968000000000002e-05 2024-09-18T04:38:51.013 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.013 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:51.013 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.013 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.013 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.013 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:51.013 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:38:51.013 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 83.202772521, 2024-09-18T04:38:51.013 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017983453, 2024-09-18T04:38:51.013 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:51.013 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:51.013 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:51.014 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:51.014 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:51.014 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 117 2024-09-18T04:38:51.014 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.014 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:51.014 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.014 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:51.014 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:38:51.014 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.014 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.014 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.014 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:51.014 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:38:51.014 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.015 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.015 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.015 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:51.015 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787865+0000", 2024-09-18T04:38:51.015 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:51.015 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.015 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.015 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:51.015 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787889+0000", 2024-09-18T04:38:51.015 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4244999999999999e-05 2024-09-18T04:38:51.015 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.015 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.015 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:51.015 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787905+0000", 2024-09-18T04:38:51.016 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5739999999999998e-05 2024-09-18T04:38:51.016 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.016 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.016 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:51.016 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787918+0000", 2024-09-18T04:38:51.016 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.252e-05 2024-09-18T04:38:51.016 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.016 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.016 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:51.016 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803693+0000", 2024-09-18T04:38:51.016 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015774872999999998 2024-09-18T04:38:51.016 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.016 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.016 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:51.016 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803775+0000", 2024-09-18T04:38:51.017 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2761000000000003e-05 2024-09-18T04:38:51.017 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.017 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.017 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:51.017 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803887+0000", 2024-09-18T04:38:51.017 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111708 2024-09-18T04:38:51.017 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.017 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.017 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:51.017 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804823+0000", 2024-09-18T04:38:51.017 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00093625100000000003 2024-09-18T04:38:51.017 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.017 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.018 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:51.018 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805804+0000", 2024-09-18T04:38:51.018 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00098114699999999992 2024-09-18T04:38:51.018 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.018 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.018 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:51.018 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805831+0000", 2024-09-18T04:38:51.018 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6401999999999999e-05 2024-09-18T04:38:51.018 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.018 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.018 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:51.018 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805850+0000", 2024-09-18T04:38:51.018 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8862e-05 2024-09-18T04:38:51.018 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.019 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:51.019 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.019 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.019 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.019 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:51.019 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:38:51.019 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 82.759595390000001, 2024-09-18T04:38:51.019 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016528505999999998, 2024-09-18T04:38:51.019 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:51.019 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:51.019 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:51.019 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:51.019 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:51.020 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 138 2024-09-18T04:38:51.020 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.020 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:51.020 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.020 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:51.020 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:38:51.020 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.020 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.020 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.020 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:51.020 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:38:51.020 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.020 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.021 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.021 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:51.021 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:38:51.021 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:51.021 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.021 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.021 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:51.021 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231047+0000", 2024-09-18T04:38:51.021 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5399999999999997e-06 2024-09-18T04:38:51.021 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.021 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.021 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:51.021 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231068+0000", 2024-09-18T04:38:51.021 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0769000000000001e-05 2024-09-18T04:38:51.022 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.022 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.022 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:51.022 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231074+0000", 2024-09-18T04:38:51.022 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6180000000000003e-06 2024-09-18T04:38:51.022 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.022 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.022 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:51.022 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242556+0000", 2024-09-18T04:38:51.022 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011482654 2024-09-18T04:38:51.022 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.023 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.023 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:51.023 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242666+0000", 2024-09-18T04:38:51.023 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011003200000000001 2024-09-18T04:38:51.023 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.023 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.023 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:51.023 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242924+0000", 2024-09-18T04:38:51.023 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00025785899999999999 2024-09-18T04:38:51.023 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.024 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.024 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:51.024 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247263+0000", 2024-09-18T04:38:51.024 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0043387649999999996 2024-09-18T04:38:51.024 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.024 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.024 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:51.024 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247412+0000", 2024-09-18T04:38:51.024 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000149276 2024-09-18T04:38:51.024 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.024 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.024 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:51.024 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247522+0000", 2024-09-18T04:38:51.024 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000109788 2024-09-18T04:38:51.025 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.025 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.025 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:51.025 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247572+0000", 2024-09-18T04:38:51.025 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9808999999999997e-05 2024-09-18T04:38:51.025 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.025 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:51.025 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.025 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.025 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.026 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:51.026 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231083+0000", 2024-09-18T04:38:51.026 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 82.759555926999994, 2024-09-18T04:38:51.026 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0166731, 2024-09-18T04:38:51.026 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:51.026 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:51.026 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:51.026 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:51.026 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:51.026 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 139 2024-09-18T04:38:51.026 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.026 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:51.026 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.027 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:51.027 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231083+0000", 2024-09-18T04:38:51.027 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.027 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.027 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.027 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:51.027 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231083+0000", 2024-09-18T04:38:51.027 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.027 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.027 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.027 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:51.028 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231082+0000", 2024-09-18T04:38:51.028 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:51.028 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.028 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.028 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:51.028 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231090+0000", 2024-09-18T04:38:51.028 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1280000000000008e-06 2024-09-18T04:38:51.028 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.028 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.028 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:51.029 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231097+0000", 2024-09-18T04:38:51.029 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5119999999999997e-06 2024-09-18T04:38:51.029 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.029 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.029 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:51.029 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231106+0000", 2024-09-18T04:38:51.029 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9230000000000008e-06 2024-09-18T04:38:51.029 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.029 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.029 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:51.029 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.243977+0000", 2024-09-18T04:38:51.029 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012870754 2024-09-18T04:38:51.029 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.030 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.030 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:51.030 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244112+0000", 2024-09-18T04:38:51.030 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000135575 2024-09-18T04:38:51.030 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.030 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.030 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:51.030 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244265+0000", 2024-09-18T04:38:51.030 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015279900000000001 2024-09-18T04:38:51.031 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.031 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.031 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:51.031 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247649+0000", 2024-09-18T04:38:51.031 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003383896 2024-09-18T04:38:51.031 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.031 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.031 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:51.031 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247656+0000", 2024-09-18T04:38:51.031 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8920000000000003e-06 2024-09-18T04:38:51.032 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.032 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.032 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:51.032 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247692+0000", 2024-09-18T04:38:51.032 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6470000000000001e-05 2024-09-18T04:38:51.032 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.032 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.032 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:51.032 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247756+0000", 2024-09-18T04:38:51.033 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3676999999999994e-05 2024-09-18T04:38:51.033 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.033 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:51.033 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.033 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.033 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.033 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:51.033 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231190+0000", 2024-09-18T04:38:51.033 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 82.759449000999993, 2024-09-18T04:38:51.033 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016793556000000001, 2024-09-18T04:38:51.033 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:51.033 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:51.033 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:51.034 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:51.034 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:51.034 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 140 2024-09-18T04:38:51.034 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.034 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:51.034 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.034 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:51.034 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231190+0000", 2024-09-18T04:38:51.034 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.034 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.034 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.034 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:51.034 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231190+0000", 2024-09-18T04:38:51.034 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.035 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.035 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.035 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:51.035 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231189+0000", 2024-09-18T04:38:51.035 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:51.035 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.035 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.035 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:51.035 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231273+0000", 2024-09-18T04:38:51.035 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4445999999999998e-05 2024-09-18T04:38:51.035 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.035 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.035 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:51.035 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231282+0000", 2024-09-18T04:38:51.036 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8859999999999993e-06 2024-09-18T04:38:51.036 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.036 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.036 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:51.036 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231288+0000", 2024-09-18T04:38:51.036 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9819999999999996e-06 2024-09-18T04:38:51.036 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.036 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.036 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:51.036 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.245119+0000", 2024-09-18T04:38:51.036 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.01383121 2024-09-18T04:38:51.036 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.036 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.036 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:51.037 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.245247+0000", 2024-09-18T04:38:51.037 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012763299999999999 2024-09-18T04:38:51.037 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.037 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.037 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:51.037 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.245401+0000", 2024-09-18T04:38:51.037 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015399200000000001 2024-09-18T04:38:51.037 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.037 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.037 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:51.037 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247766+0000", 2024-09-18T04:38:51.037 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0023649510000000001 2024-09-18T04:38:51.037 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.038 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.038 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:51.038 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247904+0000", 2024-09-18T04:38:51.038 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013836499999999999 2024-09-18T04:38:51.038 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.038 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.038 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:51.038 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247950+0000", 2024-09-18T04:38:51.038 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5201000000000003e-05 2024-09-18T04:38:51.038 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.038 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.038 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:51.038 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247983+0000", 2024-09-18T04:38:51.038 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3686000000000002e-05 2024-09-18T04:38:51.039 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.039 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:51.039 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.039 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.039 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.039 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:51.039 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:38:51.039 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 82.759338258, 2024-09-18T04:38:51.039 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017344024, 2024-09-18T04:38:51.039 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:51.039 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:51.039 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:51.039 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:51.040 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:51.040 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 141 2024-09-18T04:38:51.040 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.040 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:51.040 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.040 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:51.040 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:38:51.040 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.040 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.040 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.040 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:51.040 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:38:51.040 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.040 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.041 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.041 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:51.041 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:38:51.041 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:51.041 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.041 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.041 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:51.041 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231309+0000", 2024-09-18T04:38:51.041 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5419999999999994e-06 2024-09-18T04:38:51.041 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.041 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.041 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:51.041 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231323+0000", 2024-09-18T04:38:51.042 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3108000000000001e-05 2024-09-18T04:38:51.042 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.042 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.042 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:51.042 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231340+0000", 2024-09-18T04:38:51.042 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.696e-05 2024-09-18T04:38:51.042 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.042 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.042 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:51.042 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.246143+0000", 2024-09-18T04:38:51.042 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014803320999999999 2024-09-18T04:38:51.042 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.042 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.043 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:51.043 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.246323+0000", 2024-09-18T04:38:51.043 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00018061600000000001 2024-09-18T04:38:51.043 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.043 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.043 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:51.043 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.246530+0000", 2024-09-18T04:38:51.043 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020625 2024-09-18T04:38:51.043 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.043 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.043 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:51.043 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248002+0000", 2024-09-18T04:38:51.043 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0014719379999999999 2024-09-18T04:38:51.044 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.044 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.044 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:51.044 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248582+0000", 2024-09-18T04:38:51.044 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00058020599999999999 2024-09-18T04:38:51.044 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.044 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.044 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:51.044 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248618+0000", 2024-09-18T04:38:51.044 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6559e-05 2024-09-18T04:38:51.044 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.044 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.045 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:51.045 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248645+0000", 2024-09-18T04:38:51.045 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6080000000000001e-05 2024-09-18T04:38:51.045 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.045 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:51.045 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.045 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.045 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.045 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:36 2.7 2:fe1b02ad:::rbd_data.125fc09df6d1.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:51.045 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:38:51.045 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 8.7249164110000006, 2024-09-18T04:38:51.045 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.040387804999999999, 2024-09-18T04:38:51.045 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:51.046 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:51.046 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:51.046 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:38:51.046 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:38:51.046 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 36 2024-09-18T04:38:51.046 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.046 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:51.046 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.046 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:51.046 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:38:51.046 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.046 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.046 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.046 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:51.046 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:38:51.047 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.047 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.047 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.047 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:51.047 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265720+0000", 2024-09-18T04:38:51.047 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:51.047 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.047 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.047 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:51.047 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272590+0000", 2024-09-18T04:38:51.047 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.006870508 2024-09-18T04:38:51.047 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.047 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.048 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:51.048 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272623+0000", 2024-09-18T04:38:51.048 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2716000000000003e-05 2024-09-18T04:38:51.048 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.048 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.048 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:51.048 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272637+0000", 2024-09-18T04:38:51.048 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4175999999999999e-05 2024-09-18T04:38:51.048 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.048 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.048 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:51.048 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272779+0000", 2024-09-18T04:38:51.048 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000141392 2024-09-18T04:38:51.048 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.049 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.049 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:51.049 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273031+0000", 2024-09-18T04:38:51.049 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00025249099999999998 2024-09-18T04:38:51.049 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.049 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.049 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:38:51.049 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273219+0000", 2024-09-18T04:38:51.049 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000187531 2024-09-18T04:38:51.049 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.049 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.049 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:51.049 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.302806+0000", 2024-09-18T04:38:51.050 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.029587553999999999 2024-09-18T04:38:51.050 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.050 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.050 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:51.050 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306004+0000", 2024-09-18T04:38:51.050 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0031982629999999998 2024-09-18T04:38:51.050 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.050 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.050 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:51.050 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306071+0000", 2024-09-18T04:38:51.050 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6829999999999995e-05 2024-09-18T04:38:51.050 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.050 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.050 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:51.050 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306110+0000", 2024-09-18T04:38:51.051 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8874000000000001e-05 2024-09-18T04:38:51.051 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.051 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:51.051 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.051 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.051 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.051 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:38 2.0 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:51.051 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:38:51.051 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 8.7179628739999995, 2024-09-18T04:38:51.051 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.033754348000000003, 2024-09-18T04:38:51.051 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:51.051 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:51.051 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:51.051 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:38:51.052 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:38:51.052 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 38 2024-09-18T04:38:51.052 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.052 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:51.052 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.052 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:51.052 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:38:51.052 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.052 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.052 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.052 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:51.052 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:38:51.052 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.052 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.053 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.053 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:51.053 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272673+0000", 2024-09-18T04:38:51.053 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:51.053 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.053 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.053 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:51.053 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277166+0000", 2024-09-18T04:38:51.053 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044922249999999999 2024-09-18T04:38:51.053 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.054 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.054 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:51.054 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277189+0000", 2024-09-18T04:38:51.054 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3283000000000001e-05 2024-09-18T04:38:51.054 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.054 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.054 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:51.054 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277205+0000", 2024-09-18T04:38:51.054 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6305000000000001e-05 2024-09-18T04:38:51.054 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.054 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.054 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:51.055 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277290+0000", 2024-09-18T04:38:51.055 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5135000000000004e-05 2024-09-18T04:38:51.055 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.055 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.055 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:51.055 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277531+0000", 2024-09-18T04:38:51.055 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00024014699999999999 2024-09-18T04:38:51.055 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.055 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.055 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:51.055 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277793+0000", 2024-09-18T04:38:51.055 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00026224499999999998 2024-09-18T04:38:51.056 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.056 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.056 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:51.056 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303990+0000", 2024-09-18T04:38:51.056 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.026197295999999998 2024-09-18T04:38:51.056 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.056 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.056 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:51.056 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306340+0000", 2024-09-18T04:38:51.056 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.002349524 2024-09-18T04:38:51.057 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.057 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.057 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:51.057 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306394+0000", 2024-09-18T04:38:51.057 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3934999999999998e-05 2024-09-18T04:38:51.057 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.057 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.057 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:51.057 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306430+0000", 2024-09-18T04:38:51.057 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6690000000000003e-05 2024-09-18T04:38:51.057 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.057 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:51.058 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.058 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.058 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.058 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:39 2.7 2:e9c909c5:::rbd_data.125fc09df6d1.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:51.058 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:38:51.058 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 8.7134032690000005, 2024-09-18T04:38:51.058 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.029110673, 2024-09-18T04:38:51.058 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:51.058 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:51.058 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:51.058 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:38:51.059 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:38:51.059 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 39 2024-09-18T04:38:51.059 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.059 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:51.059 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.059 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:51.059 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:38:51.059 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.059 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.059 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.060 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:51.060 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:38:51.060 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.060 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.060 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.060 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:51.060 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277234+0000", 2024-09-18T04:38:51.060 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:51.060 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.060 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.060 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:51.060 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281351+0000", 2024-09-18T04:38:51.061 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0041170920000000001 2024-09-18T04:38:51.061 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.061 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.061 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:51.061 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281375+0000", 2024-09-18T04:38:51.061 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.446e-05 2024-09-18T04:38:51.061 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.061 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.061 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:51.061 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281390+0000", 2024-09-18T04:38:51.061 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4309e-05 2024-09-18T04:38:51.062 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.062 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.062 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:51.062 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.282306+0000", 2024-09-18T04:38:51.062 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091662000000000004 2024-09-18T04:38:51.062 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.062 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.062 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:51.062 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.282613+0000", 2024-09-18T04:38:51.062 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00030663799999999997 2024-09-18T04:38:51.062 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.063 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.063 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:38:51.063 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283001+0000", 2024-09-18T04:38:51.063 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00038801599999999998 2024-09-18T04:38:51.063 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.063 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.063 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:51.063 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306239+0000", 2024-09-18T04:38:51.063 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023238466999999999 2024-09-18T04:38:51.064 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.064 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.064 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:51.064 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306259+0000", 2024-09-18T04:38:51.064 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9837999999999999e-05 2024-09-18T04:38:51.064 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.064 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.064 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:51.064 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306299+0000", 2024-09-18T04:38:51.065 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0194000000000002e-05 2024-09-18T04:38:51.065 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.065 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.065 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:51.065 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306346+0000", 2024-09-18T04:38:51.065 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6845000000000003e-05 2024-09-18T04:38:51.065 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.065 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:51.065 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.066 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.066 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.066 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:46 2.6 e51/17 2:7ba25245:::rbd_data.125fc09df6d1.0000000000000006:head v 51'2, mlcod=0'0)", 2024-09-18T04:38:51.066 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:38:51.066 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 8.6774821370000002, 2024-09-18T04:38:51.066 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019843098, 2024-09-18T04:38:51.066 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:51.066 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:51.066 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:51.066 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.066 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:51.066 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:38:51.066 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.067 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.067 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.067 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:51.067 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:38:51.067 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.067 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.067 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.067 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:51.067 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313156+0000", 2024-09-18T04:38:51.067 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:51.067 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.067 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.067 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:51.067 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320144+0000", 2024-09-18T04:38:51.068 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.006988345 2024-09-18T04:38:51.068 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.068 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.068 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:51.068 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320175+0000", 2024-09-18T04:38:51.068 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.0861999999999997e-05 2024-09-18T04:38:51.068 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.068 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.068 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:51.068 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320188+0000", 2024-09-18T04:38:51.068 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2714999999999999e-05 2024-09-18T04:38:51.068 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.068 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.068 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:51.069 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320267+0000", 2024-09-18T04:38:51.069 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9586999999999995e-05 2024-09-18T04:38:51.069 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.069 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.069 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:51.069 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320311+0000", 2024-09-18T04:38:51.069 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3529000000000003e-05 2024-09-18T04:38:51.069 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.069 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.069 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:51.069 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.332949+0000", 2024-09-18T04:38:51.069 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012638267999999999 2024-09-18T04:38:51.069 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.069 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.069 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:51.070 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.333000+0000", 2024-09-18T04:38:51.070 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0736999999999999e-05 2024-09-18T04:38:51.070 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.070 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:51.070 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.070 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.070 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.070 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:57 2.0 2:0cdd070e:::rbd_data.125fc09df6d1.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:51.070 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:38:51.070 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 8.6416245620000005, 2024-09-18T04:38:51.070 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.033329974999999998, 2024-09-18T04:38:51.070 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:51.070 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:51.071 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:51.071 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:38:51.071 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:38:51.071 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 57 2024-09-18T04:38:51.071 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.071 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:51.071 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.071 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:51.071 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:38:51.071 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.071 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.071 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.071 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:51.071 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:38:51.072 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.072 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.072 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.072 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:51.072 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349013+0000", 2024-09-18T04:38:51.072 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:51.072 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.072 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.072 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:51.072 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356138+0000", 2024-09-18T04:38:51.072 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071258010000000002 2024-09-18T04:38:51.072 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.072 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.072 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:51.073 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356168+0000", 2024-09-18T04:38:51.073 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9679000000000002e-05 2024-09-18T04:38:51.073 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.073 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.073 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:51.073 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356181+0000", 2024-09-18T04:38:51.073 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3164e-05 2024-09-18T04:38:51.073 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.073 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.073 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:51.073 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356254+0000", 2024-09-18T04:38:51.073 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3182999999999994e-05 2024-09-18T04:38:51.073 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.073 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.074 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:51.074 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356453+0000", 2024-09-18T04:38:51.074 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019876599999999999 2024-09-18T04:38:51.074 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.074 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.074 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:51.074 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356627+0000", 2024-09-18T04:38:51.074 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017340199999999999 2024-09-18T04:38:51.074 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.074 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.074 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:51.074 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.369200+0000", 2024-09-18T04:38:51.074 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012573302 2024-09-18T04:38:51.074 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.075 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.075 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:51.075 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382262+0000", 2024-09-18T04:38:51.075 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013062549 2024-09-18T04:38:51.075 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.075 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.075 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:51.075 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382310+0000", 2024-09-18T04:38:51.075 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7366e-05 2024-09-18T04:38:51.075 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.075 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.075 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:51.075 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382344+0000", 2024-09-18T04:38:51.075 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4319999999999997e-05 2024-09-18T04:38:51.076 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.076 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:51.076 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.076 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.076 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.076 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:59 2.0 2:19c52cde:::rbd_data.125fc09df6d1.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:51.076 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:38:51.076 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 8.6344333970000005, 2024-09-18T04:38:51.076 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.027137247999999999, 2024-09-18T04:38:51.076 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:51.076 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:51.076 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:51.076 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:38:51.076 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:38:51.076 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 59 2024-09-18T04:38:51.077 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.077 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:51.077 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.077 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:51.077 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:38:51.077 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.077 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.077 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.077 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:51.077 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:38:51.077 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.077 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.077 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.078 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:51.078 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356204+0000", 2024-09-18T04:38:51.078 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:51.078 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.078 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.078 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:51.078 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362661+0000", 2024-09-18T04:38:51.078 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0064565129999999997 2024-09-18T04:38:51.078 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.078 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.078 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:51.078 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362684+0000", 2024-09-18T04:38:51.078 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3688000000000001e-05 2024-09-18T04:38:51.079 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.079 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.079 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:51.079 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362699+0000", 2024-09-18T04:38:51.079 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4798e-05 2024-09-18T04:38:51.079 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.079 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.079 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:51.079 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363016+0000", 2024-09-18T04:38:51.079 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031737300000000003 2024-09-18T04:38:51.079 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.079 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.079 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:51.079 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363201+0000", 2024-09-18T04:38:51.079 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000184131 2024-09-18T04:38:51.080 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.080 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.080 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:51.080 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363343+0000", 2024-09-18T04:38:51.080 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014236699999999999 2024-09-18T04:38:51.080 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.080 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.080 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:51.080 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.372189+0000", 2024-09-18T04:38:51.080 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0088464819999999993 2024-09-18T04:38:51.080 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.080 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.080 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:51.080 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383269+0000", 2024-09-18T04:38:51.081 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011079594 2024-09-18T04:38:51.081 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.081 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.081 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:51.081 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383314+0000", 2024-09-18T04:38:51.081 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4932000000000003e-05 2024-09-18T04:38:51.081 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.081 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.081 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:51.081 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383343+0000", 2024-09-18T04:38:51.081 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8600999999999999e-05 2024-09-18T04:38:51.081 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.081 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:51.082 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.082 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.082 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.082 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:40 2.6 e51/17 2:784d6cd1:ns::rbd_data.133e813ab5bc.0000000000000005:head v 51'7, mlcod=51'5)", 2024-09-18T04:38:51.082 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:38:51.082 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 8.4381836630000002, 2024-09-18T04:38:51.082 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020497496, 2024-09-18T04:38:51.082 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:51.082 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:51.082 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:51.082 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.082 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:51.082 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:38:51.082 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.082 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.083 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.083 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:51.083 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:38:51.083 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.083 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.083 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.083 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:51.083 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552454+0000", 2024-09-18T04:38:51.083 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:51.083 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.083 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.083 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:51.083 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556448+0000", 2024-09-18T04:38:51.084 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0039940990000000001 2024-09-18T04:38:51.084 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.084 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.084 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:51.084 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556481+0000", 2024-09-18T04:38:51.084 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3778000000000001e-05 2024-09-18T04:38:51.084 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.084 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.084 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:51.084 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556493+0000", 2024-09-18T04:38:51.084 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1622e-05 2024-09-18T04:38:51.084 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.084 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.085 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:51.085 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556556+0000", 2024-09-18T04:38:51.085 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2582999999999994e-05 2024-09-18T04:38:51.085 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.085 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.085 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:51.085 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556592+0000", 2024-09-18T04:38:51.085 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5880999999999997e-05 2024-09-18T04:38:51.085 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.085 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.085 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:51.085 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.572786+0000", 2024-09-18T04:38:51.085 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016194571000000001 2024-09-18T04:38:51.086 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.086 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.086 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:51.086 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.572953+0000", 2024-09-18T04:38:51.086 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000166436 2024-09-18T04:38:51.086 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.086 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:51.086 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.086 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.086 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.086 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:45 2.0 2:0fbfa3dc:ns::rbd_data.133e813ab5bc.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:51.086 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:38:51.086 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 8.4269693639999996, 2024-09-18T04:38:51.086 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.036558168000000002, 2024-09-18T04:38:51.087 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:51.087 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:51.087 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:51.087 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:38:51.087 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:51.087 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 45 2024-09-18T04:38:51.087 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.087 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:51.087 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.087 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:51.087 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:38:51.087 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.087 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.087 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.087 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:51.088 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:38:51.088 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.088 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.088 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.088 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:51.088 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:38:51.088 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:51.088 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.088 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.088 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:51.088 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578594+0000", 2024-09-18T04:38:51.088 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014927147999999999 2024-09-18T04:38:51.088 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.088 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.089 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:51.089 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578612+0000", 2024-09-18T04:38:51.089 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7623999999999999e-05 2024-09-18T04:38:51.089 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.089 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.089 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:51.089 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578620+0000", 2024-09-18T04:38:51.089 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9070000000000007e-06 2024-09-18T04:38:51.089 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.089 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.089 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:51.089 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578702+0000", 2024-09-18T04:38:51.089 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2349999999999996e-05 2024-09-18T04:38:51.089 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.090 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.090 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:51.090 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.579028+0000", 2024-09-18T04:38:51.090 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032622899999999998 2024-09-18T04:38:51.090 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.090 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.090 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:51.090 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.579343+0000", 2024-09-18T04:38:51.090 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031448899999999999 2024-09-18T04:38:51.090 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.090 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.090 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:51.090 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.590078+0000", 2024-09-18T04:38:51.091 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010734946 2024-09-18T04:38:51.091 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.091 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.091 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:51.091 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600089+0000", 2024-09-18T04:38:51.091 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010011195000000001 2024-09-18T04:38:51.091 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.091 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.091 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:51.091 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600177+0000", 2024-09-18T04:38:51.091 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8425999999999997e-05 2024-09-18T04:38:51.091 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.091 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.091 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:51.091 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600228+0000", 2024-09-18T04:38:51.092 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0216999999999997e-05 2024-09-18T04:38:51.092 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.092 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:51.092 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.092 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.092 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.092 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:48 2.6 e51/17 2:7c225365:ns::rbd_data.133e813ab5bc.0000000000000008:head v 51'9, mlcod=51'5)", 2024-09-18T04:38:51.092 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.570076+0000", 2024-09-18T04:38:51.092 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 8.4205622620000007, 2024-09-18T04:38:51.092 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016939773000000002, 2024-09-18T04:38:51.092 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:51.092 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:51.092 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:51.093 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.093 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:51.093 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.570076+0000", 2024-09-18T04:38:51.093 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.093 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.093 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.093 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:51.093 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.570076+0000", 2024-09-18T04:38:51.093 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.093 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.094 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.094 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:51.094 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.570074+0000", 2024-09-18T04:38:51.094 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:51.094 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.094 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.094 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:51.094 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.574672+0000", 2024-09-18T04:38:51.094 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045979819999999996 2024-09-18T04:38:51.094 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.094 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.094 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:51.094 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.574695+0000", 2024-09-18T04:38:51.094 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2529999999999999e-05 2024-09-18T04:38:51.094 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.095 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.095 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:51.095 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.574706+0000", 2024-09-18T04:38:51.095 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1357e-05 2024-09-18T04:38:51.095 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.095 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.095 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:51.095 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.574746+0000", 2024-09-18T04:38:51.095 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9586999999999999e-05 2024-09-18T04:38:51.095 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.096 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.096 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:51.096 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.574764+0000", 2024-09-18T04:38:51.096 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8000999999999999e-05 2024-09-18T04:38:51.096 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.096 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.096 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:51.096 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.586847+0000", 2024-09-18T04:38:51.096 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.01208359 2024-09-18T04:38:51.096 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.096 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.096 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:51.096 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.587016+0000", 2024-09-18T04:38:51.097 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000168796 2024-09-18T04:38:51.097 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.097 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:51.097 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.097 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.097 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.097 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:54 2.0 2:04f17ef6:ns::rbd_data.133e813ab5bc.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:51.097 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:38:51.097 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 8.4053975269999999, 2024-09-18T04:38:51.098 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.024612999, 2024-09-18T04:38:51.098 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:51.098 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:51.098 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:51.098 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:38:51.098 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:51.098 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 54 2024-09-18T04:38:51.098 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.098 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:51.098 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.098 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:51.099 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:38:51.099 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.099 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.099 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.099 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:51.099 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:38:51.099 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.099 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.099 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.099 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:51.099 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585239+0000", 2024-09-18T04:38:51.099 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:51.100 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.100 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.100 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:51.100 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593702+0000", 2024-09-18T04:38:51.100 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0084632990000000005 2024-09-18T04:38:51.100 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.100 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.100 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:51.100 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593719+0000", 2024-09-18T04:38:51.100 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.664e-05 2024-09-18T04:38:51.101 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.101 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.101 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:51.101 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593727+0000", 2024-09-18T04:38:51.101 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6839999999999996e-06 2024-09-18T04:38:51.101 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.101 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.101 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:51.101 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593767+0000", 2024-09-18T04:38:51.102 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0330999999999997e-05 2024-09-18T04:38:51.102 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.102 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.102 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:51.102 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593905+0000", 2024-09-18T04:38:51.102 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013754800000000001 2024-09-18T04:38:51.102 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.102 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.102 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:51.103 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.594013+0000", 2024-09-18T04:38:51.103 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000108805 2024-09-18T04:38:51.103 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.103 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.103 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:51.103 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604678+0000", 2024-09-18T04:38:51.103 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010664391 2024-09-18T04:38:51.103 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.103 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.103 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:51.104 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609662+0000", 2024-09-18T04:38:51.104 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0049846100000000004 2024-09-18T04:38:51.104 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.104 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.104 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:51.104 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609788+0000", 2024-09-18T04:38:51.104 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012518800000000001 2024-09-18T04:38:51.104 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.104 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.104 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:51.104 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609854+0000", 2024-09-18T04:38:51.104 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6556000000000004e-05 2024-09-18T04:38:51.104 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.105 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:51.105 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.105 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.105 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.105 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:58 2.7 2:fc7c6cd3:ns::rbd_data.133e813ab5bc.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:51.105 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:38:51.105 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 8.3925550980000008, 2024-09-18T04:38:51.105 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.027690566999999999, 2024-09-18T04:38:51.105 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:51.105 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:51.105 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:51.105 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:38:51.106 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:51.106 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 58 2024-09-18T04:38:51.106 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.106 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:51.106 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.106 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:51.106 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:38:51.106 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.106 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.106 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.106 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:51.106 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:38:51.106 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.106 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.107 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.107 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:51.107 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598082+0000", 2024-09-18T04:38:51.107 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:51.107 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.107 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.107 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:51.107 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604832+0000", 2024-09-18T04:38:51.107 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0067504510000000002 2024-09-18T04:38:51.107 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.107 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.107 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:51.107 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604852+0000", 2024-09-18T04:38:51.108 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0228999999999999e-05 2024-09-18T04:38:51.108 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.108 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.108 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:51.108 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604870+0000", 2024-09-18T04:38:51.108 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7952e-05 2024-09-18T04:38:51.108 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.108 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.108 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:51.108 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604936+0000", 2024-09-18T04:38:51.108 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5431000000000004e-05 2024-09-18T04:38:51.108 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.108 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.108 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:51.109 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605085+0000", 2024-09-18T04:38:51.109 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014910300000000001 2024-09-18T04:38:51.109 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.109 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.109 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:38:51.109 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605205+0000", 2024-09-18T04:38:51.109 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000119887 2024-09-18T04:38:51.109 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.109 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.109 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:51.109 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625586+0000", 2024-09-18T04:38:51.109 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020381399000000001 2024-09-18T04:38:51.109 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.109 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.110 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:51.110 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625628+0000", 2024-09-18T04:38:51.110 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1873999999999999e-05 2024-09-18T04:38:51.110 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.110 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.110 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:51.110 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625722+0000", 2024-09-18T04:38:51.110 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3787000000000002e-05 2024-09-18T04:38:51.110 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.110 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.110 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:51.110 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625774+0000", 2024-09-18T04:38:51.110 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2503999999999997e-05 2024-09-18T04:38:51.110 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.110 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:51.111 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.111 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.111 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.111 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:56 2.6 e51/17 2:72453c3e:ns::rbd_data.133e813ab5bc.000000000000000d:head v 51'10, mlcod=51'8)", 2024-09-18T04:38:51.111 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:38:51.111 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 8.3853579899999993, 2024-09-18T04:38:51.111 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019766552999999999, 2024-09-18T04:38:51.111 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:51.111 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:51.111 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:51.111 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.111 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:51.111 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:38:51.111 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.112 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.112 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.112 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:51.112 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:38:51.112 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.112 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.112 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.112 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:51.112 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605279+0000", 2024-09-18T04:38:51.112 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:51.112 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.112 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.112 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:51.112 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609417+0000", 2024-09-18T04:38:51.113 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0041378059999999999 2024-09-18T04:38:51.113 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.113 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.113 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:51.113 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609443+0000", 2024-09-18T04:38:51.113 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6098e-05 2024-09-18T04:38:51.113 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.113 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.113 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:51.113 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609458+0000", 2024-09-18T04:38:51.113 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4671000000000001e-05 2024-09-18T04:38:51.113 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.113 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.113 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:51.114 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609553+0000", 2024-09-18T04:38:51.114 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5265000000000004e-05 2024-09-18T04:38:51.114 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.114 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.114 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:51.114 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609583+0000", 2024-09-18T04:38:51.114 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9839e-05 2024-09-18T04:38:51.114 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.114 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.114 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:51.114 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625004+0000", 2024-09-18T04:38:51.114 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015420639999999999 2024-09-18T04:38:51.114 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.115 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.115 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:51.115 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625047+0000", 2024-09-18T04:38:51.115 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3665999999999998e-05 2024-09-18T04:38:51.115 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.115 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:51.115 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.115 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.115 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:51.115 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:51.118 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-09-18T04:38:51.118 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:38:51.119 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:51.109+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:51.119 INFO:tasks.workunit.client.0.smithi115.stderr:64+0 records in 2024-09-18T04:38:51.119 INFO:tasks.workunit.client.0.smithi115.stderr:64+0 records out 2024-09-18T04:38:51.119 INFO:tasks.workunit.client.0.smithi115.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.549851 s, 122 MB/s 2024-09-18T04:38:51.119 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo dd if=/tmp/tmp.WJMNJxIdAN/data of=/dev/nbd0 bs=1M oflag=direct 2024-09-18T04:38:51.119 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:38:51.119 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:38:51.119 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:38:51.120 INFO:tasks.workunit.client.0.smithi115.stderr:++ which dd 2024-09-18T04:38:51.120 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/bin/dd 2024-09-18T04:38:51.120 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:38:51.120 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.WJMNJxIdAN/data of=/dev/nbd0 bs=1M oflag=direct 2024-09-18T04:38:51.217 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:51.210+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:51.229 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:51.229 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:51.229 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:38:51.229 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:51.239 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:38:51.318 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:51.314+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:51.344 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:51.345 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:51.345 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:38:51.345 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:38:51.345 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:51.354 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:38:51.419 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:51.414+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:51.461 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:51.461 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:38:51.461 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:38:51.462 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [] 2024-09-18T04:38:51.462 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:51.472 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:38:51.520 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:51.514+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:51.576 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:51.577 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:51.577 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:38:51.577 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:51.587 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:38:51.622 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:51.613+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:51.696 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:51.696 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:51.696 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:38:51.696 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:38:51.696 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:51.707 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:38:51.748 INFO:tasks.workunit.client.0.smithi115.stderr:64+0 records in 2024-09-18T04:38:51.748 INFO:tasks.workunit.client.0.smithi115.stderr:64+0 records out 2024-09-18T04:38:51.748 INFO:tasks.workunit.client.0.smithi115.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.7337 s, 91.5 MB/s 2024-09-18T04:38:51.750 INFO:tasks.workunit.client.0.smithi115.stderr:++ dd if=/tmp/tmp.WJMNJxIdAN/data bs=1M 2024-09-18T04:38:51.751 INFO:tasks.workunit.client.0.smithi115.stderr:++ md5sum 2024-09-18T04:38:51.754 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:51.749+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:51.823 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:51.817+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:51.870 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:51.871 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:38:51.871 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:38:51.871 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:38:51.871 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.871 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-18T04:38:51.871 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:38:51.871 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 77.394234581000006, 2024-09-18T04:38:51.871 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0095781189999999995, 2024-09-18T04:38:51.871 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:51.872 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:51.872 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:51.872 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:38:51.872 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:38:51.872 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 1 2024-09-18T04:38:51.872 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.872 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:51.872 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.872 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:51.872 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:38:51.872 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.872 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.872 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.873 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:51.873 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:38:51.873 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.873 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.873 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.873 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:51.873 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469862+0000", 2024-09-18T04:38:51.873 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:51.873 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.873 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.873 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:51.873 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469879+0000", 2024-09-18T04:38:51.873 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7431999999999998e-05 2024-09-18T04:38:51.874 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.874 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.874 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:51.874 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469896+0000", 2024-09-18T04:38:51.874 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6742999999999999e-05 2024-09-18T04:38:51.874 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.874 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.874 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:51.874 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469915+0000", 2024-09-18T04:38:51.875 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8991000000000002e-05 2024-09-18T04:38:51.875 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.875 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.875 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:51.875 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470397+0000", 2024-09-18T04:38:51.875 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048167799999999998 2024-09-18T04:38:51.875 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.875 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.875 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-09-18T04:38:51.875 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470412+0000", 2024-09-18T04:38:51.875 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5526e-05 2024-09-18T04:38:51.875 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.875 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.876 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:51.876 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473279+0000", 2024-09-18T04:38:51.876 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0028667469999999998 2024-09-18T04:38:51.876 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.876 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.876 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:51.876 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473441+0000", 2024-09-18T04:38:51.876 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000162082 2024-09-18T04:38:51.876 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.876 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.876 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:38:51.876 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473573+0000", 2024-09-18T04:38:51.876 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000132032 2024-09-18T04:38:51.877 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.877 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.877 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:51.877 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474826+0000", 2024-09-18T04:38:51.877 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001252717 2024-09-18T04:38:51.877 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.877 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.877 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:51.877 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479307+0000", 2024-09-18T04:38:51.877 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044813220000000003 2024-09-18T04:38:51.877 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.877 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.877 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:51.878 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479366+0000", 2024-09-18T04:38:51.878 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9024999999999998e-05 2024-09-18T04:38:51.878 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.878 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.878 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:51.878 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479443+0000", 2024-09-18T04:38:51.878 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6339000000000002e-05 2024-09-18T04:38:51.878 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.878 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:51.878 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.878 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.878 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.878 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-18T04:38:51.879 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:38:51.879 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 75.381226161000001, 2024-09-18T04:38:51.879 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082699560000000002, 2024-09-18T04:38:51.879 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:51.879 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:51.879 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:51.879 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:38:51.879 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:38:51.879 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 7 2024-09-18T04:38:51.879 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.879 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:51.879 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.879 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:51.880 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:38:51.880 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.880 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.880 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.880 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:51.880 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:38:51.880 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.880 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.880 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.880 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:51.880 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482870+0000", 2024-09-18T04:38:51.880 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999967 2024-09-18T04:38:51.880 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.881 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.881 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:51.881 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482889+0000", 2024-09-18T04:38:51.881 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.965e-05 2024-09-18T04:38:51.881 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.881 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.881 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:51.881 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482938+0000", 2024-09-18T04:38:51.881 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8548999999999999e-05 2024-09-18T04:38:51.881 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.881 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.881 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:51.882 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482963+0000", 2024-09-18T04:38:51.882 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5075999999999998e-05 2024-09-18T04:38:51.882 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.882 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.882 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:51.882 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483121+0000", 2024-09-18T04:38:51.882 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015819400000000001 2024-09-18T04:38:51.882 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.882 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.882 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op must wait for map", 2024-09-18T04:38:51.882 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483138+0000", 2024-09-18T04:38:51.882 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7068999999999999e-05 2024-09-18T04:38:51.882 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.882 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.883 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:51.883 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488666+0000", 2024-09-18T04:38:51.883 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0055282049999999996 2024-09-18T04:38:51.883 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.883 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.883 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:51.883 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488783+0000", 2024-09-18T04:38:51.883 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000116737 2024-09-18T04:38:51.883 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.883 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.883 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:51.883 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488991+0000", 2024-09-18T04:38:51.883 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020781400000000001 2024-09-18T04:38:51.884 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.884 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.884 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:51.884 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490554+0000", 2024-09-18T04:38:51.884 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0015628370000000001 2024-09-18T04:38:51.884 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.884 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.884 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:51.884 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491019+0000", 2024-09-18T04:38:51.884 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000465608 2024-09-18T04:38:51.884 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.884 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.884 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:51.885 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491087+0000", 2024-09-18T04:38:51.885 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7988e-05 2024-09-18T04:38:51.885 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.885 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.885 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:51.885 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491143+0000", 2024-09-18T04:38:51.885 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5538999999999998e-05 2024-09-18T04:38:51.885 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.885 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:51.885 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.885 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.885 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.885 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:34 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:51.885 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.261758+0000", 2024-09-18T04:38:51.886 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 9.60234101, 2024-09-18T04:38:51.886 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.042024191000000002, 2024-09-18T04:38:51.886 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:51.886 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:51.886 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:51.886 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:38:51.886 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:38:51.886 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 34 2024-09-18T04:38:51.886 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.886 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:51.886 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.886 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:51.886 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.261758+0000", 2024-09-18T04:38:51.887 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.887 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.887 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.887 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:51.887 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.261758+0000", 2024-09-18T04:38:51.887 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.887 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.887 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.887 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:51.887 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.261756+0000", 2024-09-18T04:38:51.887 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:51.887 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.888 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.888 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:51.888 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.270785+0000", 2024-09-18T04:38:51.888 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0090292340000000006 2024-09-18T04:38:51.888 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.888 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.888 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:51.888 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.270890+0000", 2024-09-18T04:38:51.888 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000104541 2024-09-18T04:38:51.888 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.888 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.888 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:51.888 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.270991+0000", 2024-09-18T04:38:51.889 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000100778 2024-09-18T04:38:51.889 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.889 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.889 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:51.889 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.271157+0000", 2024-09-18T04:38:51.889 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016653999999999999 2024-09-18T04:38:51.889 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.889 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.889 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:51.889 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.271591+0000", 2024-09-18T04:38:51.889 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00043409500000000001 2024-09-18T04:38:51.889 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.889 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.890 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:38:51.890 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272017+0000", 2024-09-18T04:38:51.890 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000425769 2024-09-18T04:38:51.890 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.890 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.890 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:51.890 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.294246+0000", 2024-09-18T04:38:51.890 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.022229237999999998 2024-09-18T04:38:51.890 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.890 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.890 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:51.890 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303571+0000", 2024-09-18T04:38:51.890 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0093250239999999995 2024-09-18T04:38:51.890 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.891 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.891 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:51.891 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303705+0000", 2024-09-18T04:38:51.891 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000133831 2024-09-18T04:38:51.891 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.891 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.891 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:51.891 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303782+0000", 2024-09-18T04:38:51.892 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7156000000000005e-05 2024-09-18T04:38:51.892 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.892 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:51.892 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.892 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.892 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.892 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:36 2.7 e51/32 2:fe1b02ad:::rbd_data.125fc09df6d1.0000000000000002:head v 51'1, mlcod=0'0)", 2024-09-18T04:38:51.892 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.273441+0000", 2024-09-18T04:38:51.892 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 9.5906580290000001, 2024-09-18T04:38:51.892 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.032400768000000003, 2024-09-18T04:38:51.892 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:51.893 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:51.893 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:51.893 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.893 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:51.893 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273441+0000", 2024-09-18T04:38:51.893 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.893 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.893 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.893 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:51.894 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273441+0000", 2024-09-18T04:38:51.894 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.894 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.894 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.894 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:51.894 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273440+0000", 2024-09-18T04:38:51.894 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:51.894 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.894 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.894 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:51.894 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278723+0000", 2024-09-18T04:38:51.895 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0052829909999999999 2024-09-18T04:38:51.895 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.895 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.895 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:51.895 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278752+0000", 2024-09-18T04:38:51.895 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9178000000000001e-05 2024-09-18T04:38:51.895 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.895 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.895 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:51.896 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278761+0000", 2024-09-18T04:38:51.896 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7900000000000005e-06 2024-09-18T04:38:51.896 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.896 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.896 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:51.896 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278864+0000", 2024-09-18T04:38:51.896 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000102699 2024-09-18T04:38:51.896 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.896 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.896 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:51.896 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278900+0000", 2024-09-18T04:38:51.896 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6001999999999999e-05 2024-09-18T04:38:51.897 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.897 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.897 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:51.897 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.305793+0000", 2024-09-18T04:38:51.897 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.026893344 2024-09-18T04:38:51.897 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.897 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.897 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:51.897 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.305842+0000", 2024-09-18T04:38:51.898 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8720000000000001e-05 2024-09-18T04:38:51.898 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.898 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:51.898 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.898 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.898 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.898 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:39 2.7 e51/32 2:e9c909c5:::rbd_data.125fc09df6d1.0000000000000005:head v 51'2, mlcod=0'0)", 2024-09-18T04:38:51.898 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.283175+0000", 2024-09-18T04:38:51.898 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 9.5809243399999993, 2024-09-18T04:38:51.898 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.022864102000000001, 2024-09-18T04:38:51.898 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:51.899 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:51.899 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:51.899 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.899 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:51.899 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283175+0000", 2024-09-18T04:38:51.899 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.899 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.899 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.899 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:51.899 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283175+0000", 2024-09-18T04:38:51.899 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.900 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.900 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.900 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:51.900 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283173+0000", 2024-09-18T04:38:51.900 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:51.900 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.900 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.900 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:51.900 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.285951+0000", 2024-09-18T04:38:51.901 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0027784540000000001 2024-09-18T04:38:51.901 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.901 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.901 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:51.901 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.285970+0000", 2024-09-18T04:38:51.901 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8855000000000002e-05 2024-09-18T04:38:51.901 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.901 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.901 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:51.902 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.285984+0000", 2024-09-18T04:38:51.902 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4252e-05 2024-09-18T04:38:51.902 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.902 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.902 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:51.902 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.286041+0000", 2024-09-18T04:38:51.902 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6431999999999998e-05 2024-09-18T04:38:51.902 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.902 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.903 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:51.903 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.286086+0000", 2024-09-18T04:38:51.903 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5423000000000002e-05 2024-09-18T04:38:51.903 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.903 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.903 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:51.903 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.305998+0000", 2024-09-18T04:38:51.903 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019911697999999999 2024-09-18T04:38:51.903 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.903 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.903 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:51.903 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306039+0000", 2024-09-18T04:38:51.903 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0973999999999998e-05 2024-09-18T04:38:51.904 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.904 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:51.904 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.904 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.904 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.904 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:50 2.3 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:51.904 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.312563+0000", 2024-09-18T04:38:51.904 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 9.5515363129999997, 2024-09-18T04:38:51.904 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.032636001999999997, 2024-09-18T04:38:51.904 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:51.904 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:51.904 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:51.904 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:38:51.905 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:38:51.905 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 50 2024-09-18T04:38:51.905 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.905 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:51.905 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.905 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:51.905 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.312563+0000", 2024-09-18T04:38:51.905 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.905 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.905 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.905 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:51.905 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.312563+0000", 2024-09-18T04:38:51.905 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.906 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.906 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.906 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:51.906 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.312561+0000", 2024-09-18T04:38:51.906 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:51.906 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.906 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.906 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:51.907 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321709+0000", 2024-09-18T04:38:51.907 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0091486650000000003 2024-09-18T04:38:51.907 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.907 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.907 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:51.907 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321739+0000", 2024-09-18T04:38:51.907 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9666e-05 2024-09-18T04:38:51.907 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.907 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.907 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:51.907 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321751+0000", 2024-09-18T04:38:51.908 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2086000000000001e-05 2024-09-18T04:38:51.908 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.908 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.908 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:51.908 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321833+0000", 2024-09-18T04:38:51.908 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1842999999999996e-05 2024-09-18T04:38:51.908 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.908 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.908 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:51.908 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321981+0000", 2024-09-18T04:38:51.908 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000147441 2024-09-18T04:38:51.908 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.908 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.908 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:38:51.909 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.322109+0000", 2024-09-18T04:38:51.909 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012851200000000001 2024-09-18T04:38:51.909 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.909 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.909 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:51.909 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.332829+0000", 2024-09-18T04:38:51.909 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010720225999999999 2024-09-18T04:38:51.909 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.909 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.909 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:51.909 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.345039+0000", 2024-09-18T04:38:51.909 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012209831000000001 2024-09-18T04:38:51.909 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.910 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.910 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:51.910 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.345120+0000", 2024-09-18T04:38:51.910 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1145999999999999e-05 2024-09-18T04:38:51.910 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.910 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.910 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:51.910 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.345199+0000", 2024-09-18T04:38:51.910 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8546000000000003e-05 2024-09-18T04:38:51.910 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.910 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:51.910 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.910 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.911 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.911 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:28 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:51.911 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.515881+0000", 2024-09-18T04:38:51.911 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 9.348217837, 2024-09-18T04:38:51.911 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0059728519999999998, 2024-09-18T04:38:51.911 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:51.911 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:51.911 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:51.911 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:38:51.911 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:51.911 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 28 2024-09-18T04:38:51.911 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.911 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:51.912 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.912 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:51.912 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.515881+0000", 2024-09-18T04:38:51.912 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.912 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.912 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.912 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:51.912 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.515881+0000", 2024-09-18T04:38:51.912 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.912 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.912 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.912 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:51.912 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.515879+0000", 2024-09-18T04:38:51.913 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:51.913 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.913 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.913 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:51.913 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.515896+0000", 2024-09-18T04:38:51.913 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7425999999999999e-05 2024-09-18T04:38:51.913 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.913 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.913 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:51.913 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.515925+0000", 2024-09-18T04:38:51.913 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8277e-05 2024-09-18T04:38:51.913 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.913 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.914 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:51.914 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.515949+0000", 2024-09-18T04:38:51.914 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3759000000000001e-05 2024-09-18T04:38:51.914 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.914 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.914 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:51.914 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516091+0000", 2024-09-18T04:38:51.914 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000142967 2024-09-18T04:38:51.914 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.914 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.914 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:51.914 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516392+0000", 2024-09-18T04:38:51.914 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00030068100000000002 2024-09-18T04:38:51.914 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.915 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.915 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:38:51.915 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.517345+0000", 2024-09-18T04:38:51.915 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00095256500000000005 2024-09-18T04:38:51.915 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.915 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.915 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:51.915 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.521517+0000", 2024-09-18T04:38:51.915 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0041722770000000003 2024-09-18T04:38:51.915 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.915 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.915 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:51.915 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.521621+0000", 2024-09-18T04:38:51.916 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000104374 2024-09-18T04:38:51.916 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.916 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.916 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:51.916 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.521788+0000", 2024-09-18T04:38:51.916 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016654699999999999 2024-09-18T04:38:51.916 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.916 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.916 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:51.917 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.521854+0000", 2024-09-18T04:38:51.917 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6168999999999996e-05 2024-09-18T04:38:51.917 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.917 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:51.917 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.917 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.917 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.917 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:29 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:51.917 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.515979+0000", 2024-09-18T04:38:51.917 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 9.3481197890000001, 2024-09-18T04:38:51.917 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0062405730000000001, 2024-09-18T04:38:51.917 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:51.917 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:51.917 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:51.918 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:38:51.918 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:51.918 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 29 2024-09-18T04:38:51.918 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.918 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:51.918 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.918 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:51.918 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.515979+0000", 2024-09-18T04:38:51.918 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.918 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.918 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.918 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:51.918 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.515979+0000", 2024-09-18T04:38:51.918 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.919 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.919 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.919 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:51.919 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.515978+0000", 2024-09-18T04:38:51.919 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:51.919 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.919 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.919 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:51.919 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.515987+0000", 2024-09-18T04:38:51.919 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6000000000000007e-06 2024-09-18T04:38:51.919 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.919 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.919 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:51.919 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516004+0000", 2024-09-18T04:38:51.920 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6826000000000001e-05 2024-09-18T04:38:51.920 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.920 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.920 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:51.920 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516044+0000", 2024-09-18T04:38:51.920 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0416000000000001e-05 2024-09-18T04:38:51.920 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.920 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.920 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:51.920 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.518153+0000", 2024-09-18T04:38:51.920 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0021086630000000002 2024-09-18T04:38:51.920 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.920 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.921 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:51.921 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.518338+0000", 2024-09-18T04:38:51.921 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00018579299999999999 2024-09-18T04:38:51.921 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.921 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.921 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:38:51.921 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.518970+0000", 2024-09-18T04:38:51.921 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00063170800000000003 2024-09-18T04:38:51.921 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.921 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.921 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:51.921 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.521885+0000", 2024-09-18T04:38:51.921 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0029147940000000001 2024-09-18T04:38:51.921 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.922 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.922 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:51.922 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.522007+0000", 2024-09-18T04:38:51.922 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012216399999999999 2024-09-18T04:38:51.922 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.922 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.922 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:51.922 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.522141+0000", 2024-09-18T04:38:51.922 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013367399999999999 2024-09-18T04:38:51.922 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.922 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.922 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:51.922 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.522220+0000", 2024-09-18T04:38:51.923 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9138e-05 2024-09-18T04:38:51.923 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.923 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:51.923 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.923 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.923 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.923 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:30 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:51.923 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.516122+0000", 2024-09-18T04:38:51.923 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 9.3479771510000003, 2024-09-18T04:38:51.923 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0091865030000000004, 2024-09-18T04:38:51.923 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:51.923 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:51.923 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:51.923 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:38:51.924 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:51.924 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 30 2024-09-18T04:38:51.924 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.924 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:51.924 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.924 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:51.924 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516122+0000", 2024-09-18T04:38:51.924 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.924 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.924 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.924 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:51.924 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516122+0000", 2024-09-18T04:38:51.924 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.924 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.925 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.925 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:51.925 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516120+0000", 2024-09-18T04:38:51.925 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:51.925 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.925 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.925 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:51.925 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516129+0000", 2024-09-18T04:38:51.925 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.85e-06 2024-09-18T04:38:51.925 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.925 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.925 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:51.925 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516155+0000", 2024-09-18T04:38:51.926 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5882000000000001e-05 2024-09-18T04:38:51.926 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.926 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.926 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:51.926 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516175+0000", 2024-09-18T04:38:51.926 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.022e-05 2024-09-18T04:38:51.926 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.926 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.926 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:51.926 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519780+0000", 2024-09-18T04:38:51.926 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003604187 2024-09-18T04:38:51.926 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.927 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.927 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:51.927 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519964+0000", 2024-09-18T04:38:51.927 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00018391999999999999 2024-09-18T04:38:51.927 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.927 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.927 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:38:51.927 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.520794+0000", 2024-09-18T04:38:51.927 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00083078100000000005 2024-09-18T04:38:51.928 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.928 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.928 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:51.928 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.524129+0000", 2024-09-18T04:38:51.928 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033342419999999998 2024-09-18T04:38:51.928 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.928 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.928 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:51.928 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525239+0000", 2024-09-18T04:38:51.928 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001110778 2024-09-18T04:38:51.928 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.928 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.928 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:51.929 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525277+0000", 2024-09-18T04:38:51.929 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7596999999999999e-05 2024-09-18T04:38:51.929 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.929 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.929 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:51.930 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525308+0000", 2024-09-18T04:38:51.930 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.1535999999999998e-05 2024-09-18T04:38:51.930 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.930 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:51.930 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.930 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.930 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.930 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:32 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:51.930 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.519217+0000", 2024-09-18T04:38:51.930 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 9.3448818310000004, 2024-09-18T04:38:51.930 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082266370000000002, 2024-09-18T04:38:51.930 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:51.930 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:51.931 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:51.931 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:38:51.931 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:51.931 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 32 2024-09-18T04:38:51.931 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.931 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:51.931 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.932 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:51.932 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519217+0000", 2024-09-18T04:38:51.932 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.932 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.932 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.932 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:51.932 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519217+0000", 2024-09-18T04:38:51.932 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.932 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.932 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.932 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:51.933 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519215+0000", 2024-09-18T04:38:51.933 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:51.933 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.933 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.933 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:51.933 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519229+0000", 2024-09-18T04:38:51.933 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3329999999999999e-05 2024-09-18T04:38:51.933 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.933 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.934 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:51.934 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519268+0000", 2024-09-18T04:38:51.934 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9533000000000002e-05 2024-09-18T04:38:51.934 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.934 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.934 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:51.934 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519285+0000", 2024-09-18T04:38:51.934 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6446000000000001e-05 2024-09-18T04:38:51.934 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.934 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.935 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:51.935 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.522358+0000", 2024-09-18T04:38:51.935 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030734310000000002 2024-09-18T04:38:51.935 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.935 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.935 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:51.935 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.522551+0000", 2024-09-18T04:38:51.936 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019317600000000001 2024-09-18T04:38:51.936 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.936 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.936 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:38:51.936 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.523262+0000", 2024-09-18T04:38:51.936 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00071069999999999998 2024-09-18T04:38:51.936 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.936 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.937 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:51.937 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525318+0000", 2024-09-18T04:38:51.937 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0020561469999999999 2024-09-18T04:38:51.937 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.937 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.937 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:51.938 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.527376+0000", 2024-09-18T04:38:51.938 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0020584969999999998 2024-09-18T04:38:51.938 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.938 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.938 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:51.938 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.527416+0000", 2024-09-18T04:38:51.939 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9054000000000003e-05 2024-09-18T04:38:51.939 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.939 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.939 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:51.939 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.527444+0000", 2024-09-18T04:38:51.939 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8347000000000002e-05 2024-09-18T04:38:51.939 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.939 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:51.939 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.939 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.939 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.939 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:42 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:51.939 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.551160+0000", 2024-09-18T04:38:51.940 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 9.3129394390000009, 2024-09-18T04:38:51.940 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010692757000000001, 2024-09-18T04:38:51.940 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:51.940 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:51.940 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:51.940 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:38:51.940 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:51.940 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 42 2024-09-18T04:38:51.940 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.940 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:51.940 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.940 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:51.940 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551160+0000", 2024-09-18T04:38:51.941 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.941 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.941 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.941 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:51.941 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551160+0000", 2024-09-18T04:38:51.941 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.941 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.941 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.941 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:51.941 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551159+0000", 2024-09-18T04:38:51.941 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:51.941 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.942 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.942 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:51.942 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551166+0000", 2024-09-18T04:38:51.942 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1029999999999996e-06 2024-09-18T04:38:51.942 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.942 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.942 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:51.942 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551177+0000", 2024-09-18T04:38:51.942 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1384e-05 2024-09-18T04:38:51.942 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.942 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.942 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:51.942 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551185+0000", 2024-09-18T04:38:51.942 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1990000000000006e-06 2024-09-18T04:38:51.943 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.943 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.943 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:51.943 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552021+0000", 2024-09-18T04:38:51.943 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00083574400000000003 2024-09-18T04:38:51.943 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.943 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.943 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:51.943 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552339+0000", 2024-09-18T04:38:51.943 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031771399999999997 2024-09-18T04:38:51.943 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.943 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.943 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:38:51.943 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.553140+0000", 2024-09-18T04:38:51.944 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00080088100000000003 2024-09-18T04:38:51.944 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.944 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.944 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:51.944 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.554585+0000", 2024-09-18T04:38:51.944 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001445801 2024-09-18T04:38:51.944 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.944 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.944 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:51.944 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.561711+0000", 2024-09-18T04:38:51.944 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071260109999999998 2024-09-18T04:38:51.944 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.944 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.944 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:51.945 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.561790+0000", 2024-09-18T04:38:51.945 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8967000000000005e-05 2024-09-18T04:38:51.945 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.945 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.945 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:51.945 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.561852+0000", 2024-09-18T04:38:51.945 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2045999999999995e-05 2024-09-18T04:38:51.945 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.945 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:51.945 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.945 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.945 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.945 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:52 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:51.946 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.577756+0000", 2024-09-18T04:38:51.946 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 9.2863435540000001, 2024-09-18T04:38:51.946 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0065803340000000002, 2024-09-18T04:38:51.946 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:51.946 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:51.946 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:51.946 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:38:51.946 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:51.946 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 52 2024-09-18T04:38:51.946 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.946 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:51.946 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.946 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:51.946 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.577756+0000", 2024-09-18T04:38:51.947 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.947 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.947 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.947 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:51.947 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.577756+0000", 2024-09-18T04:38:51.947 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.947 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.947 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.947 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:51.947 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.577754+0000", 2024-09-18T04:38:51.947 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:51.947 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.947 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.947 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:51.947 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.577762+0000", 2024-09-18T04:38:51.948 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4179999999999997e-06 2024-09-18T04:38:51.948 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.948 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.948 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:51.948 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.577783+0000", 2024-09-18T04:38:51.948 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0105000000000001e-05 2024-09-18T04:38:51.948 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.948 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.948 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:51.948 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.577791+0000", 2024-09-18T04:38:51.948 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2430000000000005e-06 2024-09-18T04:38:51.948 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.948 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.949 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:51.949 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.579272+0000", 2024-09-18T04:38:51.949 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0014808759999999999 2024-09-18T04:38:51.949 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.949 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.949 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:51.949 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.579515+0000", 2024-09-18T04:38:51.949 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000243098 2024-09-18T04:38:51.949 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.949 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.949 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:38:51.949 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.579992+0000", 2024-09-18T04:38:51.949 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00047668899999999998 2024-09-18T04:38:51.949 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.949 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.950 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:51.950 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.581350+0000", 2024-09-18T04:38:51.950 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0013580160000000001 2024-09-18T04:38:51.950 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.950 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.950 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:51.950 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.584240+0000", 2024-09-18T04:38:51.950 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0028906520000000001 2024-09-18T04:38:51.950 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.950 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.950 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:51.950 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.584298+0000", 2024-09-18T04:38:51.950 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7958000000000003e-05 2024-09-18T04:38:51.950 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.951 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.951 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:51.951 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.584336+0000", 2024-09-18T04:38:51.951 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7741000000000003e-05 2024-09-18T04:38:51.951 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.951 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:51.951 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.951 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.951 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.951 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:58 2.7 e51/32 2:fc7c6cd3:ns::rbd_data.133e813ab5bc.000000000000000e:head v 51'7, mlcod=51'5)", 2024-09-18T04:38:51.951 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.605333+0000", 2024-09-18T04:38:51.951 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 9.2587662339999994, 2024-09-18T04:38:51.951 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020085902999999999, 2024-09-18T04:38:51.951 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:51.952 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:51.952 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:51.952 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.952 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:51.952 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605333+0000", 2024-09-18T04:38:51.952 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.952 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.952 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.952 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:51.952 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605333+0000", 2024-09-18T04:38:51.952 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.953 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.953 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.953 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:51.953 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605332+0000", 2024-09-18T04:38:51.953 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:51.953 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.953 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.953 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:51.953 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608667+0000", 2024-09-18T04:38:51.953 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033350300000000001 2024-09-18T04:38:51.953 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.953 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.954 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:51.954 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608706+0000", 2024-09-18T04:38:51.954 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8951e-05 2024-09-18T04:38:51.954 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.954 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.954 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:51.954 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608725+0000", 2024-09-18T04:38:51.954 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9675999999999999e-05 2024-09-18T04:38:51.954 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.954 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.954 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:51.954 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608798+0000", 2024-09-18T04:38:51.954 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3135999999999999e-05 2024-09-18T04:38:51.954 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.955 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.955 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:51.955 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608824+0000", 2024-09-18T04:38:51.955 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5284000000000001e-05 2024-09-18T04:38:51.955 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.955 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.955 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:51.955 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625370+0000", 2024-09-18T04:38:51.955 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016545975000000001 2024-09-18T04:38:51.955 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.955 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.955 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:51.955 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625419+0000", 2024-09-18T04:38:51.956 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9076999999999998e-05 2024-09-18T04:38:51.956 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.956 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:51.956 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.956 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.956 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.956 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:610 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:38:51.956 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.062333+0000", 2024-09-18T04:38:51.956 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.80176626399999995, 2024-09-18T04:38:51.956 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0067149669999999996, 2024-09-18T04:38:51.956 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:51.956 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:51.956 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:51.956 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:38:51.957 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:38:51.957 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 610 2024-09-18T04:38:51.957 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.957 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:51.957 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.957 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:51.957 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.062333+0000", 2024-09-18T04:38:51.957 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.957 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.957 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.957 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:51.957 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.062333+0000", 2024-09-18T04:38:51.958 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.958 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.958 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.958 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:51.958 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.062330+0000", 2024-09-18T04:38:51.958 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:51.958 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.958 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.958 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:51.958 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.062628+0000", 2024-09-18T04:38:51.958 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00029734800000000001 2024-09-18T04:38:51.958 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.958 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.958 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:51.959 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.062662+0000", 2024-09-18T04:38:51.959 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4853e-05 2024-09-18T04:38:51.959 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.959 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.959 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:51.959 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.062679+0000", 2024-09-18T04:38:51.959 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6646e-05 2024-09-18T04:38:51.959 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.959 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.959 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:51.959 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.062776+0000", 2024-09-18T04:38:51.959 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6865000000000002e-05 2024-09-18T04:38:51.960 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.960 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.960 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op must wait for map", 2024-09-18T04:38:51.960 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.062784+0000", 2024-09-18T04:38:51.960 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.3769999999999996e-06 2024-09-18T04:38:51.960 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.960 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.960 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:51.960 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.066331+0000", 2024-09-18T04:38:51.960 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003546418 2024-09-18T04:38:51.960 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.960 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.960 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:51.961 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.066446+0000", 2024-09-18T04:38:51.961 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000115074 2024-09-18T04:38:51.961 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.961 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.961 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:38:51.961 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.066638+0000", 2024-09-18T04:38:51.961 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000192597 2024-09-18T04:38:51.961 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.961 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.961 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:51.962 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.068643+0000", 2024-09-18T04:38:51.962 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.002004235 2024-09-18T04:38:51.962 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.962 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.962 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:51.962 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.068919+0000", 2024-09-18T04:38:51.962 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000276111 2024-09-18T04:38:51.962 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.962 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.962 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:51.962 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.068996+0000", 2024-09-18T04:38:51.962 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7474999999999999e-05 2024-09-18T04:38:51.962 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.963 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.963 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:51.963 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.069048+0000", 2024-09-18T04:38:51.963 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1502000000000003e-05 2024-09-18T04:38:51.963 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.963 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:51.963 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.963 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.963 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.963 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:611 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:38:51.964 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.069561+0000", 2024-09-18T04:38:51.964 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.79453768000000002, 2024-09-18T04:38:51.964 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0072330299999999997, 2024-09-18T04:38:51.964 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:51.964 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:51.964 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:51.964 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:38:51.964 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:38:51.964 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 611 2024-09-18T04:38:51.964 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.964 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:51.965 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.965 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:51.965 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.069561+0000", 2024-09-18T04:38:51.965 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.965 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.965 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.965 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:51.965 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.069561+0000", 2024-09-18T04:38:51.965 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.965 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.966 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.966 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:51.966 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.069559+0000", 2024-09-18T04:38:51.966 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:51.966 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.966 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.966 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:51.966 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070801+0000", 2024-09-18T04:38:51.966 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0012423359999999999 2024-09-18T04:38:51.966 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.966 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.966 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:51.967 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070828+0000", 2024-09-18T04:38:51.967 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6387e-05 2024-09-18T04:38:51.967 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.967 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.967 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:51.967 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070840+0000", 2024-09-18T04:38:51.967 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2689e-05 2024-09-18T04:38:51.967 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.967 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.967 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:51.967 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070990+0000", 2024-09-18T04:38:51.968 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014946800000000001 2024-09-18T04:38:51.968 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.968 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.968 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:51.968 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.071128+0000", 2024-09-18T04:38:51.968 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013796699999999999 2024-09-18T04:38:51.968 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.968 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.968 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:38:51.969 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.071318+0000", 2024-09-18T04:38:51.969 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000190106 2024-09-18T04:38:51.969 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.969 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.969 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:51.969 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.075711+0000", 2024-09-18T04:38:51.970 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.004393002 2024-09-18T04:38:51.970 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.970 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.970 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:51.970 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.076665+0000", 2024-09-18T04:38:51.970 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00095390399999999999 2024-09-18T04:38:51.970 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.970 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.970 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:51.971 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.076742+0000", 2024-09-18T04:38:51.971 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7112000000000003e-05 2024-09-18T04:38:51.971 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.971 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.971 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:51.971 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.076794+0000", 2024-09-18T04:38:51.971 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2475999999999997e-05 2024-09-18T04:38:51.971 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.971 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:51.971 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.971 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.971 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.972 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4805.0:621 2.7 e57/32 2:fe1b02ad:::rbd_data.125fc09df6d1.0000000000000002:head v 57'13, mlcod=57'11)", 2024-09-18T04:38:51.972 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.127765+0000", 2024-09-18T04:38:51.972 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.73633395000000001, 2024-09-18T04:38:51.972 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0063402759999999997, 2024-09-18T04:38:51.972 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:51.972 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:51.972 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:51.972 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.972 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:51.972 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.127765+0000", 2024-09-18T04:38:51.972 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.972 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.972 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.973 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:51.973 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.127765+0000", 2024-09-18T04:38:51.973 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.973 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.973 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.973 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:51.973 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.127764+0000", 2024-09-18T04:38:51.973 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:51.973 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.973 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.973 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:51.973 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.128306+0000", 2024-09-18T04:38:51.974 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00054202000000000005 2024-09-18T04:38:51.974 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.974 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.974 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:51.974 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.128325+0000", 2024-09-18T04:38:51.974 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9045000000000001e-05 2024-09-18T04:38:51.974 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.974 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.974 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:51.974 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.128333+0000", 2024-09-18T04:38:51.974 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7819999999999994e-06 2024-09-18T04:38:51.974 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.974 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.975 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:51.975 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.128421+0000", 2024-09-18T04:38:51.975 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8546000000000002e-05 2024-09-18T04:38:51.975 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.975 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.975 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:51.975 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.128452+0000", 2024-09-18T04:38:51.975 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.0395e-05 2024-09-18T04:38:51.975 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.975 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.975 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:51.975 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.134055+0000", 2024-09-18T04:38:51.976 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0056030239999999999 2024-09-18T04:38:51.976 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.976 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.976 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:51.976 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.134105+0000", 2024-09-18T04:38:51.976 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0599000000000001e-05 2024-09-18T04:38:51.976 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.976 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:51.976 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.976 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.976 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.976 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:683 2.3 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:38:51.977 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.465333+0000", 2024-09-18T04:38:51.977 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.39876599299999999, 2024-09-18T04:38:51.977 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0067688740000000002, 2024-09-18T04:38:51.977 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:51.977 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:51.977 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:51.977 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:38:51.977 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:38:51.977 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 683 2024-09-18T04:38:51.977 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.977 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:51.977 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.977 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:51.977 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.465333+0000", 2024-09-18T04:38:51.978 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.978 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.978 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.978 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:51.978 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.465333+0000", 2024-09-18T04:38:51.978 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.978 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.978 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.978 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:51.978 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.465331+0000", 2024-09-18T04:38:51.978 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:51.978 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.979 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.979 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:51.979 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.466305+0000", 2024-09-18T04:38:51.979 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000973894 2024-09-18T04:38:51.979 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.979 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.979 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:51.979 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.466337+0000", 2024-09-18T04:38:51.979 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2140000000000001e-05 2024-09-18T04:38:51.979 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.979 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.979 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:51.979 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.466352+0000", 2024-09-18T04:38:51.979 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4816e-05 2024-09-18T04:38:51.980 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.980 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.980 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:51.980 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.466421+0000", 2024-09-18T04:38:51.980 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.9518999999999996e-05 2024-09-18T04:38:51.980 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.980 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.980 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:51.980 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.466570+0000", 2024-09-18T04:38:51.980 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014881000000000001 2024-09-18T04:38:51.980 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.980 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.980 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:38:51.981 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.466798+0000", 2024-09-18T04:38:51.981 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00022834600000000001 2024-09-18T04:38:51.981 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.981 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.981 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:51.981 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.470760+0000", 2024-09-18T04:38:51.981 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0039617469999999998 2024-09-18T04:38:51.981 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.981 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.981 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:51.981 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.472007+0000", 2024-09-18T04:38:51.981 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0012472010000000001 2024-09-18T04:38:51.982 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.982 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.982 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:51.982 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.472063+0000", 2024-09-18T04:38:51.982 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6119999999999998e-05 2024-09-18T04:38:51.982 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.982 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.982 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:51.982 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.472102+0000", 2024-09-18T04:38:51.982 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8494000000000001e-05 2024-09-18T04:38:51.982 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.982 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:51.982 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.982 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.983 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.983 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:685 2.3 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:38:51.983 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.476620+0000", 2024-09-18T04:38:51.983 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.38747933299999998, 2024-09-18T04:38:51.983 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0067145440000000002, 2024-09-18T04:38:51.983 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:51.983 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:51.983 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:51.983 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:38:51.983 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:38:51.983 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 685 2024-09-18T04:38:51.983 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.983 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:51.984 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.984 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:51.984 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.476620+0000", 2024-09-18T04:38:51.984 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.984 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.984 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.984 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:51.984 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.476620+0000", 2024-09-18T04:38:51.984 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.984 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.984 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.984 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:51.984 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.476618+0000", 2024-09-18T04:38:51.984 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:51.985 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.985 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.985 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:51.985 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.477606+0000", 2024-09-18T04:38:51.985 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00098743500000000005 2024-09-18T04:38:51.985 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.985 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.985 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:51.985 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.477624+0000", 2024-09-18T04:38:51.985 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8181000000000001e-05 2024-09-18T04:38:51.985 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.985 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.985 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:51.986 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.477633+0000", 2024-09-18T04:38:51.986 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3780000000000007e-06 2024-09-18T04:38:51.986 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.986 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.986 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:51.986 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.477709+0000", 2024-09-18T04:38:51.986 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.5807000000000001e-05 2024-09-18T04:38:51.986 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.986 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.986 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:51.986 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.477804+0000", 2024-09-18T04:38:51.986 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4741999999999995e-05 2024-09-18T04:38:51.986 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.987 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.987 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:38:51.987 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.477929+0000", 2024-09-18T04:38:51.987 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012492100000000001 2024-09-18T04:38:51.987 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.987 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.987 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:51.987 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.481167+0000", 2024-09-18T04:38:51.987 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0032382639999999998 2024-09-18T04:38:51.987 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.987 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.987 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:51.987 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.483250+0000", 2024-09-18T04:38:51.987 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0020829139999999999 2024-09-18T04:38:51.988 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.988 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.988 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:51.988 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.483296+0000", 2024-09-18T04:38:51.988 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6205999999999998e-05 2024-09-18T04:38:51.988 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.988 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.988 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:51.988 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.483334+0000", 2024-09-18T04:38:51.988 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8235999999999998e-05 2024-09-18T04:38:51.988 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.988 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:51.988 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.988 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.989 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.989 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:687 2.3 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:38:51.989 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.487994+0000", 2024-09-18T04:38:51.989 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.37610469699999999, 2024-09-18T04:38:51.989 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0074167690000000001, 2024-09-18T04:38:51.989 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:51.989 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:51.989 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:51.989 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:38:51.989 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:38:51.989 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 687 2024-09-18T04:38:51.989 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.989 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:51.990 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.990 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:51.990 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.487994+0000", 2024-09-18T04:38:51.990 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.990 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.990 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.990 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:51.990 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.487994+0000", 2024-09-18T04:38:51.990 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.990 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.990 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.990 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:51.990 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.487993+0000", 2024-09-18T04:38:51.990 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:51.991 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.991 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.991 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:51.991 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.488979+0000", 2024-09-18T04:38:51.991 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00098620700000000001 2024-09-18T04:38:51.991 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.991 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.991 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:51.991 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489002+0000", 2024-09-18T04:38:51.991 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2500000000000001e-05 2024-09-18T04:38:51.991 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.991 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.991 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:51.991 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489020+0000", 2024-09-18T04:38:51.992 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7691000000000001e-05 2024-09-18T04:38:51.992 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.992 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.992 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:51.992 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489091+0000", 2024-09-18T04:38:51.992 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1307e-05 2024-09-18T04:38:51.992 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.992 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.992 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:51.992 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489219+0000", 2024-09-18T04:38:51.992 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012766000000000001 2024-09-18T04:38:51.992 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.992 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.992 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:38:51.993 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489387+0000", 2024-09-18T04:38:51.993 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016855400000000001 2024-09-18T04:38:51.993 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.993 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.993 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:51.993 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495243+0000", 2024-09-18T04:38:51.993 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058563189999999996 2024-09-18T04:38:51.993 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.993 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.993 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:51.993 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495266+0000", 2024-09-18T04:38:51.993 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2810999999999999e-05 2024-09-18T04:38:51.993 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.994 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.994 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:51.994 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495332+0000", 2024-09-18T04:38:51.994 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5469999999999995e-05 2024-09-18T04:38:51.994 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.994 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.994 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:51.994 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495411+0000", 2024-09-18T04:38:51.994 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9395999999999997e-05 2024-09-18T04:38:51.994 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.994 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:51.994 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:51.995 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.995 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.995 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:689 2.3 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:38:51.995 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.500070+0000", 2024-09-18T04:38:51.995 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.36402888100000003, 2024-09-18T04:38:51.995 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071096650000000003, 2024-09-18T04:38:51.995 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:51.995 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:51.995 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:51.995 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:38:51.995 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:38:51.995 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 689 2024-09-18T04:38:51.996 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.996 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:51.996 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.996 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:51.996 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.500070+0000", 2024-09-18T04:38:51.996 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.996 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.996 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.996 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:51.996 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.500070+0000", 2024-09-18T04:38:51.996 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:51.996 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.997 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.997 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:51.997 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.500069+0000", 2024-09-18T04:38:51.997 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:51.997 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.997 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.997 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:51.997 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501082+0000", 2024-09-18T04:38:51.997 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010136570000000001 2024-09-18T04:38:51.998 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.998 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.998 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:51.998 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501104+0000", 2024-09-18T04:38:51.998 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1883e-05 2024-09-18T04:38:51.998 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.998 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.998 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:51.998 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501115+0000", 2024-09-18T04:38:51.998 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0414000000000001e-05 2024-09-18T04:38:51.998 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.998 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.998 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:51.998 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501178+0000", 2024-09-18T04:38:51.999 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3128999999999998e-05 2024-09-18T04:38:51.999 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.999 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.999 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:51.999 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501257+0000", 2024-09-18T04:38:51.999 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9592999999999995e-05 2024-09-18T04:38:51.999 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:51.999 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:51.999 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:38:51.999 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501366+0000", 2024-09-18T04:38:51.999 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000108643 2024-09-18T04:38:52.000 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:52.000 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:52.000 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:52.000 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.504041+0000", 2024-09-18T04:38:52.000 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0026750889999999999 2024-09-18T04:38:52.000 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:52.000 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:52.000 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:52.000 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.507097+0000", 2024-09-18T04:38:52.000 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030556070000000001 2024-09-18T04:38:52.000 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:52.000 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:52.001 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:52.001 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.507140+0000", 2024-09-18T04:38:52.001 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3745000000000002e-05 2024-09-18T04:38:52.001 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:52.001 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:52.001 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:52.001 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.507180+0000", 2024-09-18T04:38:52.001 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9475999999999999e-05 2024-09-18T04:38:52.001 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:52.001 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:52.002 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:52.002 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:52.002 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:52.002 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:52.004 INFO:tasks.workunit.client.0.smithi115.stderr:64+0 records in 2024-09-18T04:38:52.004 INFO:tasks.workunit.client.0.smithi115.stderr:64+0 records out 2024-09-18T04:38:52.005 INFO:tasks.workunit.client.0.smithi115.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.136378 s, 492 MB/s 2024-09-18T04:38:52.005 INFO:tasks.workunit.client.0.smithi115.stderr:++ rbd -p rbd --no-progress export testrbdnbd17342 - 2024-09-18T04:38:52.005 INFO:tasks.workunit.client.0.smithi115.stderr:++ md5sum 2024-09-18T04:38:52.005 INFO:tasks.workunit.client.0.smithi115.stderr:2024-09-18T04:38:51.979+0000 7f6b99cbe700 0 -- 172.21.15.115:0/2280251472 >> [v2:172.21.15.28:6816/2623426735,v1:172.21.15.28:6817/2623426735] conn(0x7f6b78008d50 msgr2=0x7f6b7800b1e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-18T04:38:52.005 INFO:tasks.workunit.client.0.smithi115.stderr:2024-09-18T04:38:51.987+0000 7f6b9af20700 0 -- 172.21.15.115:0/2280251472 >> [v2:172.21.15.28:6800/511141401,v1:172.21.15.28:6802/511141401] conn(0x7f6b78014c70 msgr2=0x7f6b78015120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-18T04:38:52.005 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:51.997+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:52.006 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:38:52.008 INFO:teuthology.orchestra.run.smithi028.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-18T04:38:52.105 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:52.101+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:52.114 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:52.114 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:52.114 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:38:52.114 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:52.121 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' '5ae868888c159bc31bc5ab12f9b8764b -' = '5ae868888c159bc31bc5ab12f9b8764b -' ']' 2024-09-18T04:38:52.121 INFO:tasks.workunit.client.0.smithi115.stderr:+ unmap_device /dev/nbd0 17738 2024-09-18T04:38:52.121 INFO:tasks.workunit.client.0.smithi115.stderr:+ local args=/dev/nbd0 2024-09-18T04:38:52.122 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pid=17738 2024-09-18T04:38:52.122 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-18T04:38:52.122 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:38:52.122 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:38:52.123 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:38:52.124 INFO:tasks.workunit.client.0.smithi115.stderr:++ which rbd 2024-09-18T04:38:52.124 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:38:52.126 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/bin/rbd 2024-09-18T04:38:52.126 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:38:52.126 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-18T04:38:52.206 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:52.197+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:52.228 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd device --device-type nbd list 2024-09-18T04:38:52.228 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false grep '^17738\b' 2024-09-18T04:38:52.228 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep '^17738\b' 2024-09-18T04:38:52.237 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:52.237 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:52.237 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:38:52.237 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:38:52.237 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:52.247 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:38:52.292 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:38:52.293 INFO:tasks.workunit.client.0.smithi115.stderr:+ ps -C rbd-nbd 2024-09-18T04:38:52.293 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false grep '^ *17738\b' 2024-09-18T04:38:52.293 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep '^ *17738\b' 2024-09-18T04:38:52.305 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:38:52.305 INFO:tasks.workunit.client.0.smithi115.stderr:+ sleep 0.5 2024-09-18T04:38:52.306 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:52.301+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:52.361 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:52.362 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:38:52.362 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:38:52.362 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:38:52.362 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.362 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:37 2.5 e51/28 2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:head v 51'1, mlcod=0'0)", 2024-09-18T04:38:52.362 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:38:52.362 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 10.063348236, 2024-09-18T04:38:52.362 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012279973, 2024-09-18T04:38:52.362 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:52.362 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:52.363 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:52.363 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.363 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:52.363 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:38:52.363 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:52.363 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.363 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.363 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:52.364 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:38:52.364 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:52.364 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.364 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.364 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:52.364 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292468+0000", 2024-09-18T04:38:52.364 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:52.364 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.364 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.364 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:52.364 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295837+0000", 2024-09-18T04:38:52.365 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0033691699999999999 2024-09-18T04:38:52.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.365 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.365 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:52.365 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295860+0000", 2024-09-18T04:38:52.365 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2756999999999999e-05 2024-09-18T04:38:52.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.365 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.365 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:52.366 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295872+0000", 2024-09-18T04:38:52.366 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2442e-05 2024-09-18T04:38:52.366 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.366 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.366 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:52.366 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295934+0000", 2024-09-18T04:38:52.366 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1187999999999998e-05 2024-09-18T04:38:52.366 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.366 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.366 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:52.366 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295967+0000", 2024-09-18T04:38:52.367 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3185999999999997e-05 2024-09-18T04:38:52.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.367 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.367 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:52.367 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304697+0000", 2024-09-18T04:38:52.367 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0087298059999999997 2024-09-18T04:38:52.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.367 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.368 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:52.368 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304749+0000", 2024-09-18T04:38:52.368 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1922000000000002e-05 2024-09-18T04:38:52.368 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:52.368 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:52.368 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:52.368 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.368 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.368 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:49 2.5 e51/28 2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head v 51'2, mlcod=0'0)", 2024-09-18T04:38:52.368 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:38:52.369 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 10.030021398000001, 2024-09-18T04:38:52.369 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018033041999999999, 2024-09-18T04:38:52.369 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:52.369 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:52.369 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:52.369 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.369 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:52.369 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:38:52.369 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:52.369 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.370 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.370 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:52.370 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:38:52.370 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:52.370 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.370 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.370 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:52.370 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325793+0000", 2024-09-18T04:38:52.371 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:52.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.371 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.371 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:52.371 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329038+0000", 2024-09-18T04:38:52.371 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0032444010000000001 2024-09-18T04:38:52.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.371 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.371 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:52.371 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329065+0000", 2024-09-18T04:38:52.372 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7500000000000001e-05 2024-09-18T04:38:52.372 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.372 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.372 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:52.372 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329077+0000", 2024-09-18T04:38:52.372 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1919999999999999e-05 2024-09-18T04:38:52.373 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.373 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.373 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:52.373 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329149+0000", 2024-09-18T04:38:52.373 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2349999999999997e-05 2024-09-18T04:38:52.373 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.373 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.373 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:52.374 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329192+0000", 2024-09-18T04:38:52.374 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2744000000000003e-05 2024-09-18T04:38:52.374 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.374 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.374 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:52.374 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343773+0000", 2024-09-18T04:38:52.374 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014581246000000001 2024-09-18T04:38:52.374 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.375 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.375 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:52.375 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343829+0000", 2024-09-18T04:38:52.375 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5143999999999999e-05 2024-09-18T04:38:52.375 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:52.375 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:52.375 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:52.375 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.375 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.375 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:47 2.4 e51/17 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 51'3, mlcod=51'1)", 2024-09-18T04:38:52.376 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:38:52.376 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 10.029441133000001, 2024-09-18T04:38:52.376 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018960186, 2024-09-18T04:38:52.376 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:52.376 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:52.376 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:52.376 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.376 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:52.376 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:38:52.376 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:52.376 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.376 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.376 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:52.376 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:38:52.377 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:52.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.377 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.377 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:52.377 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326374+0000", 2024-09-18T04:38:52.377 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:52.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.377 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.377 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:52.377 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330883+0000", 2024-09-18T04:38:52.377 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0045084740000000002 2024-09-18T04:38:52.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.377 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.377 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:52.377 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330901+0000", 2024-09-18T04:38:52.378 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7890999999999999e-05 2024-09-18T04:38:52.378 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.378 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.378 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:52.378 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330912+0000", 2024-09-18T04:38:52.378 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1041e-05 2024-09-18T04:38:52.378 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.378 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.378 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:52.378 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.331003+0000", 2024-09-18T04:38:52.378 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1323999999999995e-05 2024-09-18T04:38:52.378 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.378 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.378 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:52.379 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.331030+0000", 2024-09-18T04:38:52.379 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7351999999999998e-05 2024-09-18T04:38:52.379 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.379 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.379 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:52.379 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345287+0000", 2024-09-18T04:38:52.379 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014256332 2024-09-18T04:38:52.379 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.379 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.379 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:52.379 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345336+0000", 2024-09-18T04:38:52.379 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9356e-05 2024-09-18T04:38:52.379 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:52.379 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:52.380 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:52.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.380 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.380 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:58 2.5 e51/28 2:b227e6b1:::rbd_data.125fc09df6d1.000000000000000e:head v 51'3, mlcod=51'1)", 2024-09-18T04:38:52.380 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:38:52.380 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 9.9890145060000002, 2024-09-18T04:38:52.380 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016513678, 2024-09-18T04:38:52.380 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:52.380 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:52.380 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:52.380 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.380 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:52.380 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:38:52.380 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:52.381 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.381 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.381 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:52.381 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:38:52.381 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:52.381 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.381 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.381 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:52.381 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366800+0000", 2024-09-18T04:38:52.381 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:52.381 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.381 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.381 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:52.381 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368644+0000", 2024-09-18T04:38:52.382 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001844399 2024-09-18T04:38:52.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.382 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.382 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:52.382 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368681+0000", 2024-09-18T04:38:52.382 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6526999999999998e-05 2024-09-18T04:38:52.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.382 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.382 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:52.382 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368698+0000", 2024-09-18T04:38:52.382 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7272999999999999e-05 2024-09-18T04:38:52.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.382 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.382 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:52.382 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368773+0000", 2024-09-18T04:38:52.383 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4862000000000003e-05 2024-09-18T04:38:52.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.383 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.383 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:52.383 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368801+0000", 2024-09-18T04:38:52.383 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7925000000000001e-05 2024-09-18T04:38:52.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.383 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.383 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:52.383 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383261+0000", 2024-09-18T04:38:52.383 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014460377999999999 2024-09-18T04:38:52.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.383 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.383 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:52.384 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383316+0000", 2024-09-18T04:38:52.384 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4623999999999997e-05 2024-09-18T04:38:52.384 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:52.384 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:52.384 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:52.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.384 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.384 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:56 2.4 e51/17 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 51'4, mlcod=51'2)", 2024-09-18T04:38:52.384 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:38:52.384 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 9.9870104800000004, 2024-09-18T04:38:52.384 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015105221, 2024-09-18T04:38:52.384 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:52.384 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:52.384 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:52.384 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.385 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:52.385 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:38:52.385 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:52.385 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.385 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.385 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:52.385 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:38:52.385 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:52.385 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.385 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.385 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:52.385 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368805+0000", 2024-09-18T04:38:52.385 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:52.385 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.386 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.386 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:52.386 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371603+0000", 2024-09-18T04:38:52.386 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027982660000000002 2024-09-18T04:38:52.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.386 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.386 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:52.386 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371633+0000", 2024-09-18T04:38:52.386 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0073000000000002e-05 2024-09-18T04:38:52.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.386 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.386 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:52.386 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371652+0000", 2024-09-18T04:38:52.387 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8597000000000002e-05 2024-09-18T04:38:52.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.387 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:52.387 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371752+0000", 2024-09-18T04:38:52.387 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9772999999999995e-05 2024-09-18T04:38:52.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.387 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:52.387 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371793+0000", 2024-09-18T04:38:52.387 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1173e-05 2024-09-18T04:38:52.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.387 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:52.388 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383875+0000", 2024-09-18T04:38:52.388 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012081852000000001 2024-09-18T04:38:52.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.388 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.388 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:52.388 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383912+0000", 2024-09-18T04:38:52.388 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7045000000000002e-05 2024-09-18T04:38:52.388 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:52.388 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:52.388 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:52.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.388 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.388 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:36 2.5 e51/28 2:bd5b8dd8:ns::rbd_data.133e813ab5bc.0000000000000002:head v 51'4, mlcod=51'2)", 2024-09-18T04:38:52.389 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:38:52.389 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 9.8182914889999999, 2024-09-18T04:38:52.389 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020758161000000001, 2024-09-18T04:38:52.389 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:52.389 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:52.389 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:52.389 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.389 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:52.389 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:38:52.389 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:52.389 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.389 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.389 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:52.389 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:38:52.390 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:52.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.390 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.390 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:52.390 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537523+0000", 2024-09-18T04:38:52.390 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:52.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.390 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.390 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:52.390 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541578+0000", 2024-09-18T04:38:52.390 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0040549710000000001 2024-09-18T04:38:52.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.390 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.390 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:52.391 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541608+0000", 2024-09-18T04:38:52.391 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9349e-05 2024-09-18T04:38:52.391 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.391 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.391 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:52.391 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541621+0000", 2024-09-18T04:38:52.391 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2736e-05 2024-09-18T04:38:52.391 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.391 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.391 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:52.391 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541715+0000", 2024-09-18T04:38:52.391 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4197999999999995e-05 2024-09-18T04:38:52.391 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.391 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.392 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:52.392 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541755+0000", 2024-09-18T04:38:52.392 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9818999999999999e-05 2024-09-18T04:38:52.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.392 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.392 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:52.392 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558217+0000", 2024-09-18T04:38:52.392 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016462234999999999 2024-09-18T04:38:52.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.392 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.392 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:52.392 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558284+0000", 2024-09-18T04:38:52.392 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6774999999999997e-05 2024-09-18T04:38:52.392 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:52.393 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:52.393 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:52.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.393 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.393 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:39 2.4 e51/17 2:304364d1:ns::rbd_data.133e813ab5bc.0000000000000004:head v 51'5, mlcod=51'3)", 2024-09-18T04:38:52.393 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:38:52.393 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 9.8101523620000002, 2024-09-18T04:38:52.393 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016777558000000001, 2024-09-18T04:38:52.393 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:52.393 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:52.393 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:52.393 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.393 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:52.393 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:38:52.394 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:52.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.394 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.394 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:52.394 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:38:52.394 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:52.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.394 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.394 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:52.394 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545663+0000", 2024-09-18T04:38:52.394 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:52.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.394 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.394 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:52.395 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548286+0000", 2024-09-18T04:38:52.395 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026233390000000001 2024-09-18T04:38:52.395 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.395 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.395 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:52.395 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548305+0000", 2024-09-18T04:38:52.395 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8425000000000001e-05 2024-09-18T04:38:52.395 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.395 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.395 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:52.395 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548316+0000", 2024-09-18T04:38:52.396 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1163e-05 2024-09-18T04:38:52.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.396 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.396 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:52.396 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548379+0000", 2024-09-18T04:38:52.396 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2780000000000005e-05 2024-09-18T04:38:52.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.396 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.396 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:52.396 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548399+0000", 2024-09-18T04:38:52.396 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0080000000000001e-05 2024-09-18T04:38:52.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.397 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.397 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:52.397 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562383+0000", 2024-09-18T04:38:52.397 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013984671000000001 2024-09-18T04:38:52.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.397 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.397 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:52.397 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562442+0000", 2024-09-18T04:38:52.397 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8700999999999999e-05 2024-09-18T04:38:52.397 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:52.397 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:52.397 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:52.398 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.398 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.398 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:37 2.5 e51/28 2:bf39b8e4:ns::rbd_data.133e813ab5bc.0000000000000003:head v 51'5, mlcod=51'2)", 2024-09-18T04:38:52.398 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:38:52.398 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 9.8072840929999998, 2024-09-18T04:38:52.398 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014264492, 2024-09-18T04:38:52.398 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:52.398 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:52.398 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:52.399 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.399 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:52.399 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:38:52.399 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:52.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.399 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.399 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:52.399 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:38:52.399 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:52.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.399 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.399 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:52.400 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548531+0000", 2024-09-18T04:38:52.400 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:52.400 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.400 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.400 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:52.400 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551500+0000", 2024-09-18T04:38:52.400 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0029688129999999998 2024-09-18T04:38:52.400 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.400 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.400 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:52.400 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551516+0000", 2024-09-18T04:38:52.401 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5994e-05 2024-09-18T04:38:52.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.401 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.401 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:52.401 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551526+0000", 2024-09-18T04:38:52.401 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9299999999999998e-06 2024-09-18T04:38:52.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.401 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.401 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:52.401 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551577+0000", 2024-09-18T04:38:52.402 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0871000000000001e-05 2024-09-18T04:38:52.402 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.402 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.402 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:52.402 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551602+0000", 2024-09-18T04:38:52.402 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4813000000000001e-05 2024-09-18T04:38:52.402 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.402 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.402 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:52.402 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562741+0000", 2024-09-18T04:38:52.402 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011139428999999999 2024-09-18T04:38:52.402 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.402 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.403 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:52.403 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562797+0000", 2024-09-18T04:38:52.403 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6329000000000002e-05 2024-09-18T04:38:52.403 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:52.403 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:52.403 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:52.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.403 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.403 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:49 2.4 e51/17 2:20f0b0c9:ns::rbd_data.133e813ab5bc.0000000000000009:head v 51'6, mlcod=51'4)", 2024-09-18T04:38:52.404 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:38:52.404 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 9.7758495350000008, 2024-09-18T04:38:52.404 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013849046, 2024-09-18T04:38:52.404 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:52.404 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:52.404 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:52.404 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.404 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:52.404 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:38:52.404 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:52.404 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.405 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.405 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:52.405 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:38:52.405 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:52.405 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.405 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.405 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:52.406 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579966+0000", 2024-09-18T04:38:52.406 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:52.406 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.406 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.406 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:52.406 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582324+0000", 2024-09-18T04:38:52.406 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023578890000000002 2024-09-18T04:38:52.406 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.407 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.407 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:52.407 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582355+0000", 2024-09-18T04:38:52.407 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1485000000000001e-05 2024-09-18T04:38:52.407 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.407 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.407 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:52.407 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582371+0000", 2024-09-18T04:38:52.408 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6147999999999999e-05 2024-09-18T04:38:52.408 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.408 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.408 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:52.408 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582467+0000", 2024-09-18T04:38:52.408 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6059999999999998e-05 2024-09-18T04:38:52.408 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.408 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.408 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:52.408 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582505+0000", 2024-09-18T04:38:52.408 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7679000000000003e-05 2024-09-18T04:38:52.409 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.409 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.409 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:52.409 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593743+0000", 2024-09-18T04:38:52.409 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011238356 2024-09-18T04:38:52.409 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.409 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.409 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:52.409 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593816+0000", 2024-09-18T04:38:52.409 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2935000000000006e-05 2024-09-18T04:38:52.409 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:52.409 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:52.409 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:52.409 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.410 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.410 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:53 2.4 e51/17 2:39f7f174:ns::rbd_data.133e813ab5bc.000000000000000b:head v 51'7, mlcod=51'4)", 2024-09-18T04:38:52.410 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:38:52.410 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 9.7611483799999998, 2024-09-18T04:38:52.410 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0079914159999999995, 2024-09-18T04:38:52.410 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:52.410 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:52.410 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:52.410 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.410 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:52.410 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:38:52.410 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:52.410 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.410 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.410 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:52.411 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:38:52.411 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:52.411 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.411 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.411 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:52.411 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594667+0000", 2024-09-18T04:38:52.411 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:52.411 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.411 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.411 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:52.411 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596446+0000", 2024-09-18T04:38:52.411 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001779033 2024-09-18T04:38:52.411 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.411 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.412 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:52.412 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596472+0000", 2024-09-18T04:38:52.412 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6120000000000001e-05 2024-09-18T04:38:52.412 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.412 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.412 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:52.412 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596500+0000", 2024-09-18T04:38:52.412 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7518e-05 2024-09-18T04:38:52.412 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.412 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.412 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:52.412 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596564+0000", 2024-09-18T04:38:52.412 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3880999999999994e-05 2024-09-18T04:38:52.412 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.413 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.413 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:52.413 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596596+0000", 2024-09-18T04:38:52.413 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2638999999999997e-05 2024-09-18T04:38:52.413 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.413 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.413 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:52.413 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602605+0000", 2024-09-18T04:38:52.413 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0060080699999999999 2024-09-18T04:38:52.413 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.413 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.413 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:52.413 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602660+0000", 2024-09-18T04:38:52.413 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5396000000000003e-05 2024-09-18T04:38:52.414 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:52.414 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:52.414 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:52.414 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.414 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.414 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:60 2.5 e51/28 2:a06f51e4:ns::rbd_data.133e813ab5bc.000000000000000f:head v 51'6, mlcod=51'4)", 2024-09-18T04:38:52.414 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:38:52.414 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 9.7463313950000003, 2024-09-18T04:38:52.414 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0093954989999999999, 2024-09-18T04:38:52.414 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:52.414 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:52.414 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:52.414 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.414 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:52.415 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:38:52.415 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:52.415 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.415 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.415 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:52.415 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:38:52.415 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:52.415 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.415 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.415 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:52.415 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609483+0000", 2024-09-18T04:38:52.415 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:52.415 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.415 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.415 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:52.416 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611767+0000", 2024-09-18T04:38:52.416 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002283724 2024-09-18T04:38:52.416 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.416 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.416 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:52.416 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611791+0000", 2024-09-18T04:38:52.416 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3547e-05 2024-09-18T04:38:52.416 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.416 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.416 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:52.416 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611800+0000", 2024-09-18T04:38:52.416 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4979999999999995e-06 2024-09-18T04:38:52.416 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.417 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.417 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:52.417 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611864+0000", 2024-09-18T04:38:52.417 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3974999999999996e-05 2024-09-18T04:38:52.417 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.417 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.417 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:52.417 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611887+0000", 2024-09-18T04:38:52.417 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2633000000000002e-05 2024-09-18T04:38:52.417 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.417 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.417 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:52.418 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.618777+0000", 2024-09-18T04:38:52.418 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0068897979999999999 2024-09-18T04:38:52.418 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.418 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.418 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:52.418 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.618881+0000", 2024-09-18T04:38:52.418 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010437699999999999 2024-09-18T04:38:52.418 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:52.418 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:52.418 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:52.419 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.419 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.419 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:627 2.5 e57/28 2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:head v 57'8, mlcod=51'6)", 2024-09-18T04:38:52.419 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.163545+0000", 2024-09-18T04:38:52.419 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 1.1922719150000001, 2024-09-18T04:38:52.419 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0043737209999999997, 2024-09-18T04:38:52.419 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:52.419 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:52.419 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:52.419 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.419 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:52.419 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.163545+0000", 2024-09-18T04:38:52.420 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:52.420 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.420 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.420 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:52.420 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.163545+0000", 2024-09-18T04:38:52.420 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:52.420 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.420 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.420 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:52.420 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.163544+0000", 2024-09-18T04:38:52.420 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:52.420 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.420 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.420 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:52.421 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.164269+0000", 2024-09-18T04:38:52.421 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00072451899999999997 2024-09-18T04:38:52.421 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.421 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.421 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:52.421 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.164288+0000", 2024-09-18T04:38:52.421 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8824000000000001e-05 2024-09-18T04:38:52.421 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.421 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.421 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:52.421 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.164297+0000", 2024-09-18T04:38:52.421 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4229999999999994e-06 2024-09-18T04:38:52.421 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.422 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.422 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:52.422 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.164377+0000", 2024-09-18T04:38:52.422 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0007999999999997e-05 2024-09-18T04:38:52.422 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.422 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.422 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:52.422 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.164422+0000", 2024-09-18T04:38:52.422 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5222000000000002e-05 2024-09-18T04:38:52.422 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.422 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.422 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:52.422 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.167863+0000", 2024-09-18T04:38:52.423 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0034412050000000001 2024-09-18T04:38:52.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.423 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.423 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:52.423 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.167919+0000", 2024-09-18T04:38:52.423 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5296e-05 2024-09-18T04:38:52.423 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:52.423 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:52.423 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:52.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.423 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.423 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:629 2.5 e57/28 2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:head v 57'10, mlcod=57'8)", 2024-09-18T04:38:52.423 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.173967+0000", 2024-09-18T04:38:52.424 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 1.181849937, 2024-09-18T04:38:52.424 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0043585999999999998, 2024-09-18T04:38:52.424 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:52.424 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:52.424 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:52.424 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.424 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:52.424 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.173967+0000", 2024-09-18T04:38:52.424 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:52.424 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.424 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.424 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:52.425 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.173967+0000", 2024-09-18T04:38:52.425 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:52.425 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.425 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.425 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:52.425 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.173965+0000", 2024-09-18T04:38:52.425 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:52.425 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.425 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.425 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:52.425 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.174507+0000", 2024-09-18T04:38:52.425 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00054151999999999998 2024-09-18T04:38:52.425 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.425 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.425 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:52.426 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.174526+0000", 2024-09-18T04:38:52.426 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8683999999999999e-05 2024-09-18T04:38:52.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.426 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.426 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:52.426 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.174538+0000", 2024-09-18T04:38:52.426 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2561000000000001e-05 2024-09-18T04:38:52.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.426 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.426 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:52.426 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.174679+0000", 2024-09-18T04:38:52.426 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000140841 2024-09-18T04:38:52.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.426 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.427 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:52.427 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.174742+0000", 2024-09-18T04:38:52.427 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2800999999999997e-05 2024-09-18T04:38:52.427 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.427 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.427 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:52.427 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.178265+0000", 2024-09-18T04:38:52.427 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0035236849999999999 2024-09-18T04:38:52.427 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.427 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.427 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:52.427 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.178326+0000", 2024-09-18T04:38:52.427 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0078000000000003e-05 2024-09-18T04:38:52.427 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:52.428 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:52.428 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:52.428 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.428 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.428 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:663 2.4 e57/17 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 57'13, mlcod=57'11)", 2024-09-18T04:38:52.428 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.355300+0000", 2024-09-18T04:38:52.428 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 1.0005164600000001, 2024-09-18T04:38:52.428 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0049192760000000002, 2024-09-18T04:38:52.428 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:52.428 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:52.428 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:52.428 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.428 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:52.429 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355300+0000", 2024-09-18T04:38:52.429 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:52.429 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.429 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.429 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:52.429 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355300+0000", 2024-09-18T04:38:52.429 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:52.429 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.429 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.429 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:52.429 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355299+0000", 2024-09-18T04:38:52.429 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:52.429 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.429 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.430 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:52.430 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355876+0000", 2024-09-18T04:38:52.430 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00057730900000000003 2024-09-18T04:38:52.430 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.430 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.430 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:52.430 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355897+0000", 2024-09-18T04:38:52.430 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0477000000000001e-05 2024-09-18T04:38:52.430 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.430 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.430 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:52.430 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355905+0000", 2024-09-18T04:38:52.431 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5739999999999996e-06 2024-09-18T04:38:52.431 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.431 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.431 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:52.431 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355989+0000", 2024-09-18T04:38:52.431 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4002e-05 2024-09-18T04:38:52.431 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.431 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.431 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:52.431 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.356041+0000", 2024-09-18T04:38:52.432 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1347000000000002e-05 2024-09-18T04:38:52.432 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.432 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.432 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:52.432 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.360172+0000", 2024-09-18T04:38:52.432 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0041315090000000002 2024-09-18T04:38:52.432 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.432 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.432 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:52.432 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.360220+0000", 2024-09-18T04:38:52.432 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7584999999999999e-05 2024-09-18T04:38:52.432 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:52.432 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:52.432 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:52.433 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.433 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.433 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:677 2.5 e57/28 2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head v 57'18, mlcod=57'16)", 2024-09-18T04:38:52.433 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.434699+0000", 2024-09-18T04:38:52.433 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.92111754099999998, 2024-09-18T04:38:52.433 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0043950960000000002, 2024-09-18T04:38:52.433 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:52.433 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:52.433 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:52.433 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.434 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:52.434 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.434699+0000", 2024-09-18T04:38:52.434 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:52.434 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.434 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.434 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:52.434 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.434699+0000", 2024-09-18T04:38:52.434 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:52.434 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.434 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.434 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:52.434 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.434697+0000", 2024-09-18T04:38:52.434 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:52.434 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.435 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.435 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:52.435 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.435430+0000", 2024-09-18T04:38:52.435 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00073206900000000001 2024-09-18T04:38:52.435 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.435 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.435 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:52.435 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.435457+0000", 2024-09-18T04:38:52.435 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7039e-05 2024-09-18T04:38:52.436 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.436 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.436 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:52.436 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.435467+0000", 2024-09-18T04:38:52.436 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0482999999999999e-05 2024-09-18T04:38:52.436 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.436 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.436 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:52.436 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.435535+0000", 2024-09-18T04:38:52.436 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8308999999999999e-05 2024-09-18T04:38:52.436 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.436 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.436 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:52.436 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.435575+0000", 2024-09-18T04:38:52.437 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0018999999999997e-05 2024-09-18T04:38:52.437 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.437 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.437 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:52.437 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.439029+0000", 2024-09-18T04:38:52.437 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0034535019999999998 2024-09-18T04:38:52.437 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.437 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.437 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:52.437 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.439094+0000", 2024-09-18T04:38:52.438 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5517000000000003e-05 2024-09-18T04:38:52.438 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:52.438 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:52.438 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:52.438 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.438 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.438 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:679 2.5 e57/28 2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head v 57'20, mlcod=57'18)", 2024-09-18T04:38:52.438 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.445316+0000", 2024-09-18T04:38:52.438 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.91050098300000004, 2024-09-18T04:38:52.438 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0047016760000000001, 2024-09-18T04:38:52.438 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:52.438 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:52.439 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:52.439 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.439 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:52.439 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445316+0000", 2024-09-18T04:38:52.439 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:52.439 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.439 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.439 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:52.439 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445316+0000", 2024-09-18T04:38:52.439 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:52.439 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.440 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.440 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:52.440 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445314+0000", 2024-09-18T04:38:52.440 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:52.440 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.440 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.440 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:52.440 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445947+0000", 2024-09-18T04:38:52.440 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00063298900000000003 2024-09-18T04:38:52.440 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.440 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.440 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:52.441 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445968+0000", 2024-09-18T04:38:52.441 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.054e-05 2024-09-18T04:38:52.441 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.441 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.441 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:52.441 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445981+0000", 2024-09-18T04:38:52.441 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2697000000000001e-05 2024-09-18T04:38:52.441 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.441 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.441 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:52.441 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.446102+0000", 2024-09-18T04:38:52.442 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012189300000000001 2024-09-18T04:38:52.442 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.442 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.442 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:52.442 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.446151+0000", 2024-09-18T04:38:52.442 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8279000000000003e-05 2024-09-18T04:38:52.442 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.442 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.442 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:52.442 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.449953+0000", 2024-09-18T04:38:52.443 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003801979 2024-09-18T04:38:52.443 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.443 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.443 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:52.443 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.450018+0000", 2024-09-18T04:38:52.443 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4873e-05 2024-09-18T04:38:52.443 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:52.443 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:52.443 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:52.443 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.443 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.443 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:699 2.4 e57/17 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 57'17, mlcod=57'15)", 2024-09-18T04:38:52.443 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.566073+0000", 2024-09-18T04:38:52.444 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.78974347, 2024-09-18T04:38:52.444 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055398380000000001, 2024-09-18T04:38:52.444 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:52.444 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:52.444 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:52.444 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.444 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:52.444 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566073+0000", 2024-09-18T04:38:52.444 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:52.444 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.444 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.444 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:52.445 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566073+0000", 2024-09-18T04:38:52.445 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:52.445 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.445 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.445 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:52.445 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566071+0000", 2024-09-18T04:38:52.445 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:52.445 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.445 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.445 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:52.445 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566637+0000", 2024-09-18T04:38:52.445 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00056516699999999999 2024-09-18T04:38:52.445 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.446 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.446 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:52.446 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566668+0000", 2024-09-18T04:38:52.446 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1544999999999997e-05 2024-09-18T04:38:52.446 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.446 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.446 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:52.446 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566685+0000", 2024-09-18T04:38:52.446 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6857000000000001e-05 2024-09-18T04:38:52.446 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.446 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.446 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:52.446 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566816+0000", 2024-09-18T04:38:52.447 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013092799999999999 2024-09-18T04:38:52.447 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.447 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.447 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:52.447 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566875+0000", 2024-09-18T04:38:52.447 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9583999999999997e-05 2024-09-18T04:38:52.447 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.447 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.447 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:52.447 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.571567+0000", 2024-09-18T04:38:52.447 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.004691682 2024-09-18T04:38:52.447 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.447 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.447 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:52.448 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.571613+0000", 2024-09-18T04:38:52.448 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6125000000000003e-05 2024-09-18T04:38:52.448 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:52.448 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:52.448 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:52.448 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.448 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.448 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:703 2.4 e57/17 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 57'21, mlcod=57'19)", 2024-09-18T04:38:52.448 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.588442+0000", 2024-09-18T04:38:52.448 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.76737493400000001, 2024-09-18T04:38:52.448 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0044132429999999999, 2024-09-18T04:38:52.448 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:52.448 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:52.448 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:52.449 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.449 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:52.449 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.588442+0000", 2024-09-18T04:38:52.449 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:52.449 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.449 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.449 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:52.449 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.588442+0000", 2024-09-18T04:38:52.449 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:52.449 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.449 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.449 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:52.449 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.588439+0000", 2024-09-18T04:38:52.450 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999971 2024-09-18T04:38:52.450 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.450 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.450 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:52.450 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.589194+0000", 2024-09-18T04:38:52.450 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000754447 2024-09-18T04:38:52.450 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.450 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.450 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:52.450 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.589231+0000", 2024-09-18T04:38:52.450 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6912000000000002e-05 2024-09-18T04:38:52.450 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.450 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.450 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:52.450 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.589249+0000", 2024-09-18T04:38:52.451 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8712999999999999e-05 2024-09-18T04:38:52.451 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.451 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.451 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:52.451 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.589336+0000", 2024-09-18T04:38:52.451 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.6991999999999997e-05 2024-09-18T04:38:52.451 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.451 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.451 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:52.451 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.589387+0000", 2024-09-18T04:38:52.451 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0698000000000001e-05 2024-09-18T04:38:52.451 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.451 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.452 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:52.452 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.592782+0000", 2024-09-18T04:38:52.452 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003394925 2024-09-18T04:38:52.452 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.452 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.452 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:52.452 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.592855+0000", 2024-09-18T04:38:52.452 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3245999999999996e-05 2024-09-18T04:38:52.452 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:52.452 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:52.452 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:52.452 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.452 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.453 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:717 2.5 e57/28 2:b227e6b1:::rbd_data.125fc09df6d1.000000000000000e:head v 57'26, mlcod=57'24)", 2024-09-18T04:38:52.453 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.667326+0000", 2024-09-18T04:38:52.453 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.68849084800000004, 2024-09-18T04:38:52.453 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0042492520000000002, 2024-09-18T04:38:52.453 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:52.453 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:52.453 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:52.453 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.453 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:52.453 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.667326+0000", 2024-09-18T04:38:52.453 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:52.453 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.453 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.453 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:52.454 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.667326+0000", 2024-09-18T04:38:52.454 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:52.454 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.454 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.454 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:52.454 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.667324+0000", 2024-09-18T04:38:52.454 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:52.454 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.454 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.454 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:52.454 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.667849+0000", 2024-09-18T04:38:52.454 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00052490600000000001 2024-09-18T04:38:52.454 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.454 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.454 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:52.455 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.667889+0000", 2024-09-18T04:38:52.455 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9632999999999998e-05 2024-09-18T04:38:52.455 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.455 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.455 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:52.455 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.667907+0000", 2024-09-18T04:38:52.455 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7564999999999999e-05 2024-09-18T04:38:52.455 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.455 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.455 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:52.455 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.668059+0000", 2024-09-18T04:38:52.455 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000152263 2024-09-18T04:38:52.455 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.456 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.456 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:52.456 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.668099+0000", 2024-09-18T04:38:52.456 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0413000000000001e-05 2024-09-18T04:38:52.456 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.456 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.456 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:52.456 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.671512+0000", 2024-09-18T04:38:52.456 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0034132339999999998 2024-09-18T04:38:52.456 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.456 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.456 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:52.456 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.671575+0000", 2024-09-18T04:38:52.457 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2848999999999994e-05 2024-09-18T04:38:52.457 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:52.457 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:52.457 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:52.457 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.457 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.457 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:721 2.5 e57/28 2:b227e6b1:::rbd_data.125fc09df6d1.000000000000000e:head v 57'30, mlcod=57'28)", 2024-09-18T04:38:52.457 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.686880+0000", 2024-09-18T04:38:52.457 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.66893702899999996, 2024-09-18T04:38:52.457 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0044157340000000002, 2024-09-18T04:38:52.457 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:52.457 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:52.457 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:52.457 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.458 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:52.458 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.686880+0000", 2024-09-18T04:38:52.458 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:52.458 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.458 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.458 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:52.458 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.686880+0000", 2024-09-18T04:38:52.458 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:52.458 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.458 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.458 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:52.458 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.686878+0000", 2024-09-18T04:38:52.458 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:52.458 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.459 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.459 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:52.459 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.687586+0000", 2024-09-18T04:38:52.459 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00070759499999999999 2024-09-18T04:38:52.459 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.459 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.459 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:52.459 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.687610+0000", 2024-09-18T04:38:52.459 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4304e-05 2024-09-18T04:38:52.459 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.459 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.459 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:52.459 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.687619+0000", 2024-09-18T04:38:52.459 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3710000000000006e-06 2024-09-18T04:38:52.460 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.460 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.460 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:52.460 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.687700+0000", 2024-09-18T04:38:52.460 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0916999999999998e-05 2024-09-18T04:38:52.460 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.460 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.460 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:52.460 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.687735+0000", 2024-09-18T04:38:52.460 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5138999999999997e-05 2024-09-18T04:38:52.460 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.460 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.460 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:52.460 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.691244+0000", 2024-09-18T04:38:52.461 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0035090189999999999 2024-09-18T04:38:52.461 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:52.461 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:52.461 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:52.461 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.691296+0000", 2024-09-18T04:38:52.461 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1267000000000002e-05 2024-09-18T04:38:52.461 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:52.461 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:52.461 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:52.461 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:52.461 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:52.461 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:52.462 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T04:38:52.463 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:38:52.464 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:52.458+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:52.563 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:52.558+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:52.577 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:52.577 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:52.577 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:38:52.577 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:52.588 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:38:52.606 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:38:52.626 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:38:52.646 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:38:52.664 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:52.657+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:52.669 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:38:52.693 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:38:52.709 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:52.710 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:52.718 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:38:52.720 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:38:52.720 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:38:52.720 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:38:52.721 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:38:52.721 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:38:52.721 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:38:52.721 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:38:52.774 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:52.770+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:52.807 INFO:tasks.workunit.client.0.smithi115.stderr:++ _sudo rbd device --device-type nbd --options notrim map rbd/testrbdnbd17342 2024-09-18T04:38:52.808 INFO:tasks.workunit.client.0.smithi115.stderr:++ local cmd 2024-09-18T04:38:52.808 INFO:tasks.workunit.client.0.smithi115.stderr:+++ id -u 2024-09-18T04:38:52.811 INFO:tasks.workunit.client.0.smithi115.stderr:++ '[' 1000 -eq 0 ']' 2024-09-18T04:38:52.811 INFO:tasks.workunit.client.0.smithi115.stderr:+++ which rbd 2024-09-18T04:38:52.813 INFO:tasks.workunit.client.0.smithi115.stderr:++ cmd=/usr/bin/rbd 2024-09-18T04:38:52.813 INFO:tasks.workunit.client.0.smithi115.stderr:++ shift 2024-09-18T04:38:52.813 INFO:tasks.workunit.client.0.smithi115.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd --options notrim map rbd/testrbdnbd17342 2024-09-18T04:38:52.821 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:52.821 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:52.831 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:38:52.866 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:52.861+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:52.949 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:52.950 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:52.960 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:38:52.967 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:52.962+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:52.977 INFO:tasks.workunit.client.0.smithi115.stderr:+ DEV=/dev/nbd0 2024-09-18T04:38:52.977 INFO:tasks.workunit.client.0.smithi115.stderr:+ get_pid rbd 2024-09-18T04:38:52.977 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pool=rbd 2024-09-18T04:38:52.978 INFO:tasks.workunit.client.0.smithi115.stderr:+ local ns= 2024-09-18T04:38:52.978 INFO:tasks.workunit.client.0.smithi115.stderr:++ rbd device --device-type nbd --format xml list 2024-09-18T04:38:52.978 INFO:tasks.workunit.client.0.smithi115.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd17342'\''][device='\''/dev/nbd0'\'']/id' 2024-09-18T04:38:53.057 INFO:tasks.workunit.client.0.smithi115.stderr:+ PID=17851 2024-09-18T04:38:53.057 INFO:tasks.workunit.client.0.smithi115.stderr:+ test -n 17851 2024-09-18T04:38:53.057 INFO:tasks.workunit.client.0.smithi115.stderr:+ ps -p 17851 -C rbd-nbd 2024-09-18T04:38:53.069 INFO:tasks.workunit.client.0.smithi115.stdout: PID TTY TIME CMD 2024-09-18T04:38:53.069 INFO:tasks.workunit.client.0.smithi115.stdout: 17851 ? 00:00:00 rbd-nbd 2024-09-18T04:38:53.069 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:53.061+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:53.070 INFO:tasks.workunit.client.0.smithi115.stderr:++ rbd -p rbd --format xml du testrbdnbd17342 2024-09-18T04:38:53.070 INFO:tasks.workunit.client.0.smithi115.stderr:++ xmlstarlet sel -t -m //stats/images/image/provisioned_size -v . 2024-09-18T04:38:53.072 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:53.072 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:53.082 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:38:53.120 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-18T04:38:53.145 INFO:tasks.workunit.client.0.smithi115.stderr:+ provisioned=67108864 2024-09-18T04:38:53.146 INFO:tasks.workunit.client.0.smithi115.stderr:++ rbd -p rbd --format xml du testrbdnbd17342 2024-09-18T04:38:53.146 INFO:tasks.workunit.client.0.smithi115.stderr:++ xmlstarlet sel -t -m //stats/images/image/used_size -v . 2024-09-18T04:38:53.170 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:53.161+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:53.190 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:53.190 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:53.202 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:38:53.226 INFO:tasks.workunit.client.0.smithi115.stderr:+ used=67108864 2024-09-18T04:38:53.226 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 67108864 -eq 67108864 ']' 2024-09-18T04:38:53.227 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false _sudo blkdiscard /dev/nbd0 2024-09-18T04:38:53.227 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo blkdiscard /dev/nbd0 2024-09-18T04:38:53.227 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:38:53.227 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:38:53.229 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:38:53.229 INFO:tasks.workunit.client.0.smithi115.stderr:++ which blkdiscard 2024-09-18T04:38:53.231 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/sbin/blkdiscard 2024-09-18T04:38:53.231 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:38:53.231 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/sbin/blkdiscard /dev/nbd0 2024-09-18T04:38:53.254 INFO:tasks.workunit.client.0.smithi115.stderr:blkdiscard: /dev/nbd0: BLKDISCARD ioctl failed: Operation not supported 2024-09-18T04:38:53.255 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:38:53.255 INFO:tasks.workunit.client.0.smithi115.stderr:+ sync 2024-09-18T04:38:53.270 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:53.266+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:53.318 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:53.318 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:53.328 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:38:53.372 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:53.366+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:53.430 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:53.431 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:53.441 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:38:53.472 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:53.466+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:53.547 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:53.547 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:53.557 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:38:53.574 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:53.570+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:53.667 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:53.667 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:53.674 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:53.669+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:53.677 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:38:53.775 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:53.769+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:53.783 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:53.783 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:53.793 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:38:53.876 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:53.869+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:53.901 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:53.901 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:53.911 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:38:53.977 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:53.970+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:54.024 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-09-18T04:38:54.063 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:54.063 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:54.073 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:38:54.082 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:54.078+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:54.177 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:54.178 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:54.178 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:54.174+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:54.187 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:38:54.279 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:54.274+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:54.297 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:54.297 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:54.307 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:38:54.380 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:54.377+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:54.426 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:54.426 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:54.436 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:38:54.481 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:54.474+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:54.546 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:54.546 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:54.556 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:38:54.602 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:54.598+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:54.665 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:54.665 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:54.675 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:38:54.682 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:54.678+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:54.733 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T04:38:54.780 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:54.780 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:54.783 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:54.777+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:54.790 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:38:54.883 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:54.878+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:54.885 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:38:54.903 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:54.903 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:54.905 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:38:54.913 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:38:54.923 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:38:54.945 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:38:54.969 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:38:54.984 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:54.978+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:54.991 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:38:54.994 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:38:54.994 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:38:54.994 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:38:54.994 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:38:54.994 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:38:54.995 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:38:55.029 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:55.029 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:38:55.029 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:38:55.029 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:38:55.029 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.029 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:33 2.1 e51/24 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head v 51'4, mlcod=51'2)", 2024-09-18T04:38:55.029 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:38:55.029 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 12.746039979000001, 2024-09-18T04:38:55.029 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018702106, 2024-09-18T04:38:55.030 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:55.030 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:55.030 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:55.030 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.030 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:55.030 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:38:55.030 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.030 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.030 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.030 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:55.030 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:38:55.031 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.031 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.031 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.031 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:55.031 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279197+0000", 2024-09-18T04:38:55.031 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:55.031 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.031 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.031 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:55.031 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282682+0000", 2024-09-18T04:38:55.031 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0034848689999999998 2024-09-18T04:38:55.031 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.032 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.032 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:55.032 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282709+0000", 2024-09-18T04:38:55.032 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7599000000000001e-05 2024-09-18T04:38:55.032 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.032 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.032 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:55.032 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282720+0000", 2024-09-18T04:38:55.032 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.092e-05 2024-09-18T04:38:55.032 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.032 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.032 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:55.033 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282791+0000", 2024-09-18T04:38:55.033 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1268999999999998e-05 2024-09-18T04:38:55.033 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.033 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.033 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:55.033 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282843+0000", 2024-09-18T04:38:55.033 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1771999999999998e-05 2024-09-18T04:38:55.033 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.033 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.033 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:55.033 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.297830+0000", 2024-09-18T04:38:55.034 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014987228999999999 2024-09-18T04:38:55.034 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.034 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.034 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:55.034 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.297901+0000", 2024-09-18T04:38:55.034 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0209999999999994e-05 2024-09-18T04:38:55.034 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.034 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:55.034 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.034 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.034 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.034 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:38 2.0 e51/17 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head v 51'10, mlcod=51'8)", 2024-09-18T04:38:55.034 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:38:55.035 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 12.744621549, 2024-09-18T04:38:55.035 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.023984472, 2024-09-18T04:38:55.035 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:55.035 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:55.035 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:55.035 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.035 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:55.035 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:38:55.035 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.035 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.035 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.035 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:55.035 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:38:55.036 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.036 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.036 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:55.036 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280614+0000", 2024-09-18T04:38:55.036 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:55.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.036 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.036 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:55.036 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293094+0000", 2024-09-18T04:38:55.036 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012479571 2024-09-18T04:38:55.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.036 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.037 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:55.037 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293111+0000", 2024-09-18T04:38:55.037 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7354999999999999e-05 2024-09-18T04:38:55.037 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.037 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.037 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:55.037 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293121+0000", 2024-09-18T04:38:55.037 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6979999999999993e-06 2024-09-18T04:38:55.037 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.037 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.037 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:55.037 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293180+0000", 2024-09-18T04:38:55.037 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9333000000000003e-05 2024-09-18T04:38:55.038 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.038 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.038 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:55.038 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293203+0000", 2024-09-18T04:38:55.038 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2633000000000002e-05 2024-09-18T04:38:55.038 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.038 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.038 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:55.038 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304534+0000", 2024-09-18T04:38:55.038 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011330761 2024-09-18T04:38:55.038 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.038 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.038 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:55.038 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304601+0000", 2024-09-18T04:38:55.039 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7657000000000006e-05 2024-09-18T04:38:55.039 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.039 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:55.039 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.039 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.039 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.039 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:46 2.6 2:7ba25245:::rbd_data.125fc09df6d1.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:55.039 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:38:55.039 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 12.717078749000001, 2024-09-18T04:38:55.039 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.035604595000000003, 2024-09-18T04:38:55.039 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:55.039 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:55.039 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:55.039 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:38:55.040 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:38:55.040 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 46 2024-09-18T04:38:55.040 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.040 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:55.040 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.040 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:55.040 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:38:55.040 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.040 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.040 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.040 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:55.040 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:38:55.040 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.041 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.041 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.041 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:55.041 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308158+0000", 2024-09-18T04:38:55.041 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:55.041 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.041 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.041 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:55.041 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313191+0000", 2024-09-18T04:38:55.041 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0050323929999999996 2024-09-18T04:38:55.041 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.041 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.041 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:55.041 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313210+0000", 2024-09-18T04:38:55.042 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9673e-05 2024-09-18T04:38:55.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.042 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.042 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:55.042 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313221+0000", 2024-09-18T04:38:55.042 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0444e-05 2024-09-18T04:38:55.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.042 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.042 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:55.042 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313294+0000", 2024-09-18T04:38:55.042 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2650000000000004e-05 2024-09-18T04:38:55.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.042 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.042 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:55.043 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313493+0000", 2024-09-18T04:38:55.043 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019968300000000001 2024-09-18T04:38:55.043 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.043 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.043 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:38:55.043 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313666+0000", 2024-09-18T04:38:55.043 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00017248599999999999 2024-09-18T04:38:55.043 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.043 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.043 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:55.043 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.323175+0000", 2024-09-18T04:38:55.043 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0095088819999999998 2024-09-18T04:38:55.043 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.044 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.044 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:55.044 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343651+0000", 2024-09-18T04:38:55.044 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020475892999999998 2024-09-18T04:38:55.044 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.044 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.044 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:55.044 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343715+0000", 2024-09-18T04:38:55.044 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4443999999999995e-05 2024-09-18T04:38:55.044 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.045 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.045 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:55.045 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343764+0000", 2024-09-18T04:38:55.045 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9280000000000003e-05 2024-09-18T04:38:55.045 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.045 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:55.045 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.045 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.045 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.046 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:47 2.4 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:55.046 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:38:55.046 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 12.711996711999999, 2024-09-18T04:38:55.046 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.032570693999999997, 2024-09-18T04:38:55.046 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:55.046 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:55.046 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:55.046 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:38:55.046 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:38:55.046 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 47 2024-09-18T04:38:55.046 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.047 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:55.047 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.047 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:55.047 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:38:55.047 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.047 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.047 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.047 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:55.047 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:38:55.047 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.047 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.047 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.048 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:55.048 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313240+0000", 2024-09-18T04:38:55.048 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:55.048 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.048 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.048 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:55.048 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325825+0000", 2024-09-18T04:38:55.048 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012584425999999999 2024-09-18T04:38:55.048 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.048 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.048 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:55.049 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325848+0000", 2024-09-18T04:38:55.049 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3025999999999999e-05 2024-09-18T04:38:55.050 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.050 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.050 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:55.050 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325863+0000", 2024-09-18T04:38:55.050 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5152e-05 2024-09-18T04:38:55.050 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.050 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.050 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:55.050 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325938+0000", 2024-09-18T04:38:55.050 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4982999999999997e-05 2024-09-18T04:38:55.051 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.051 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.051 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:55.051 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326090+0000", 2024-09-18T04:38:55.051 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015207900000000001 2024-09-18T04:38:55.051 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.051 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.051 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:55.051 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326242+0000", 2024-09-18T04:38:55.051 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015163500000000001 2024-09-18T04:38:55.051 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.051 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.051 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:55.051 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343015+0000", 2024-09-18T04:38:55.051 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016773414 2024-09-18T04:38:55.052 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.052 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.052 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:55.052 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345664+0000", 2024-09-18T04:38:55.052 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026487870000000001 2024-09-18T04:38:55.052 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.052 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.052 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:55.052 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345739+0000", 2024-09-18T04:38:55.052 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5018999999999994e-05 2024-09-18T04:38:55.052 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.052 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.052 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:55.052 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345812+0000", 2024-09-18T04:38:55.053 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3547000000000006e-05 2024-09-18T04:38:55.053 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.053 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:55.053 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.053 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.053 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.053 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:48 2.6 2:61db7a3a:::rbd_data.125fc09df6d1.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:55.053 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:38:55.053 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 12.699340039999999, 2024-09-18T04:38:55.053 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.025064388, 2024-09-18T04:38:55.053 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:55.053 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:55.053 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:55.053 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:38:55.053 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:38:55.054 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 48 2024-09-18T04:38:55.054 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.054 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:55.054 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.054 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:55.054 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:38:55.054 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.054 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.054 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.054 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:55.054 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:38:55.054 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.054 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.054 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.054 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:55.055 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325897+0000", 2024-09-18T04:38:55.055 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:55.055 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.055 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.055 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:55.055 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333697+0000", 2024-09-18T04:38:55.055 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0078002649999999998 2024-09-18T04:38:55.055 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.055 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.055 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:55.055 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333716+0000", 2024-09-18T04:38:55.055 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8910999999999999e-05 2024-09-18T04:38:55.055 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.055 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.055 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:55.056 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333728+0000", 2024-09-18T04:38:55.056 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1936000000000001e-05 2024-09-18T04:38:55.056 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.056 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.056 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:55.056 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333792+0000", 2024-09-18T04:38:55.056 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4010999999999994e-05 2024-09-18T04:38:55.056 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.056 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.056 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:55.056 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.334082+0000", 2024-09-18T04:38:55.056 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00028955799999999998 2024-09-18T04:38:55.056 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.056 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.057 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:38:55.057 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.334326+0000", 2024-09-18T04:38:55.057 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00024399099999999999 2024-09-18T04:38:55.057 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.057 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.057 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:55.057 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350745+0000", 2024-09-18T04:38:55.057 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016419474999999999 2024-09-18T04:38:55.057 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.057 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.057 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:55.057 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350808+0000", 2024-09-18T04:38:55.057 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3244999999999995e-05 2024-09-18T04:38:55.057 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.058 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.058 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:55.058 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350892+0000", 2024-09-18T04:38:55.058 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4074999999999997e-05 2024-09-18T04:38:55.058 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.058 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.058 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:55.058 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350963+0000", 2024-09-18T04:38:55.058 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0412000000000003e-05 2024-09-18T04:38:55.058 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.058 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:55.058 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.058 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.058 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.058 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:51 2.6 2:775d7b11:::rbd_data.125fc09df6d1.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:55.058 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:38:55.059 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 12.691487904000001, 2024-09-18T04:38:55.059 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.023824221, 2024-09-18T04:38:55.059 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:55.059 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:55.059 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:55.059 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:38:55.059 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:38:55.059 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 51 2024-09-18T04:38:55.059 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.059 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:55.059 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.059 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:55.059 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:38:55.059 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.060 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.060 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.060 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:55.060 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:38:55.060 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.060 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.060 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.060 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:55.060 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333749+0000", 2024-09-18T04:38:55.060 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:55.060 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.060 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.060 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:55.061 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340899+0000", 2024-09-18T04:38:55.061 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0071500340000000004 2024-09-18T04:38:55.061 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.061 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.061 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:55.061 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340929+0000", 2024-09-18T04:38:55.061 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0093000000000001e-05 2024-09-18T04:38:55.061 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.061 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.061 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:55.061 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340943+0000", 2024-09-18T04:38:55.061 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4019000000000001e-05 2024-09-18T04:38:55.061 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.061 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.061 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:55.062 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343834+0000", 2024-09-18T04:38:55.062 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028918120000000001 2024-09-18T04:38:55.062 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.062 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.062 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:55.062 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344020+0000", 2024-09-18T04:38:55.062 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000185892 2024-09-18T04:38:55.062 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.062 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.062 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:38:55.062 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344190+0000", 2024-09-18T04:38:55.062 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016946800000000001 2024-09-18T04:38:55.062 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.062 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.063 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:55.063 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.351291+0000", 2024-09-18T04:38:55.063 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.007100805 2024-09-18T04:38:55.063 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.063 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.063 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:55.063 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357470+0000", 2024-09-18T04:38:55.063 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0061797570000000001 2024-09-18T04:38:55.063 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.063 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.063 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:55.063 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357533+0000", 2024-09-18T04:38:55.063 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2906999999999999e-05 2024-09-18T04:38:55.063 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.063 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.064 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:55.064 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357575+0000", 2024-09-18T04:38:55.064 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1403999999999999e-05 2024-09-18T04:38:55.064 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.064 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:55.064 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.064 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.064 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.064 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:56 2.4 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:55.064 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:38:55.064 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 12.675442811, 2024-09-18T04:38:55.064 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.034295870999999999, 2024-09-18T04:38:55.064 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:55.064 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:55.065 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:55.065 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:38:55.065 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:38:55.065 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 56 2024-09-18T04:38:55.065 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.065 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:55.065 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.065 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:55.065 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:38:55.065 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.065 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.065 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.065 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:55.065 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:38:55.066 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.066 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.066 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:55.066 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349794+0000", 2024-09-18T04:38:55.066 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:55.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.066 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.066 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:55.066 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366632+0000", 2024-09-18T04:38:55.066 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01683755 2024-09-18T04:38:55.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.066 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.066 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:55.066 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366669+0000", 2024-09-18T04:38:55.066 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7351000000000002e-05 2024-09-18T04:38:55.067 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.067 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.067 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:55.067 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366686+0000", 2024-09-18T04:38:55.067 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6507999999999998e-05 2024-09-18T04:38:55.067 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.067 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.067 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:55.067 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366962+0000", 2024-09-18T04:38:55.067 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00027655900000000001 2024-09-18T04:38:55.067 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.067 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.067 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:55.067 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.367452+0000", 2024-09-18T04:38:55.067 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00049025299999999996 2024-09-18T04:38:55.067 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.068 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.068 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:55.068 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368113+0000", 2024-09-18T04:38:55.068 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00066093399999999996 2024-09-18T04:38:55.068 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.068 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.068 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:55.068 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383719+0000", 2024-09-18T04:38:55.068 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015606106999999999 2024-09-18T04:38:55.068 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.068 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.068 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:55.068 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384035+0000", 2024-09-18T04:38:55.068 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00031590399999999999 2024-09-18T04:38:55.068 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.069 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.069 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:55.069 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384067+0000", 2024-09-18T04:38:55.069 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.129e-05 2024-09-18T04:38:55.070 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.070 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.070 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:55.070 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384091+0000", 2024-09-18T04:38:55.070 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4788000000000001e-05 2024-09-18T04:38:55.070 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.070 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:55.070 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.070 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.070 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.070 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:57 2.0 e51/17 2:0cdd070e:::rbd_data.125fc09df6d1.000000000000000d:head v 51'21, mlcod=51'19)", 2024-09-18T04:38:55.070 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:38:55.070 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 12.667615324, 2024-09-18T04:38:55.070 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.025216311000000002, 2024-09-18T04:38:55.071 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:55.071 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:55.071 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:55.071 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.071 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:55.071 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:38:55.071 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.071 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.071 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.071 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:55.071 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:38:55.071 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.071 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.072 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.072 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:55.072 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:38:55.072 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:55.072 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.072 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.072 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:55.072 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363844+0000", 2024-09-18T04:38:55.072 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0062218450000000002 2024-09-18T04:38:55.072 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.072 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.072 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:55.072 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363879+0000", 2024-09-18T04:38:55.073 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4382999999999999e-05 2024-09-18T04:38:55.073 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.073 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.073 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:55.073 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363893+0000", 2024-09-18T04:38:55.073 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4348999999999999e-05 2024-09-18T04:38:55.073 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.073 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.073 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:55.073 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363977+0000", 2024-09-18T04:38:55.073 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3499999999999997e-05 2024-09-18T04:38:55.073 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.074 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.074 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:55.074 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364009+0000", 2024-09-18T04:38:55.074 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2292000000000002e-05 2024-09-18T04:38:55.074 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.074 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.074 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:55.074 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.382781+0000", 2024-09-18T04:38:55.074 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018771626 2024-09-18T04:38:55.074 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.074 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.074 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:55.074 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.382839+0000", 2024-09-18T04:38:55.074 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8758000000000002e-05 2024-09-18T04:38:55.075 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.075 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:55.075 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.075 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.075 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.075 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:59 2.0 e51/17 2:19c52cde:::rbd_data.125fc09df6d1.000000000000000f:head v 51'22, mlcod=51'19)", 2024-09-18T04:38:55.075 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:38:55.075 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 12.660955398, 2024-09-18T04:38:55.075 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.019497112, 2024-09-18T04:38:55.075 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:55.075 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:55.075 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:55.075 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.075 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:55.076 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:38:55.076 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.076 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.076 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.076 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:55.076 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:38:55.076 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.076 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.076 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.076 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:55.076 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364282+0000", 2024-09-18T04:38:55.076 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:55.077 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.077 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.077 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:55.077 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371823+0000", 2024-09-18T04:38:55.077 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0075411829999999999 2024-09-18T04:38:55.077 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.077 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.077 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:55.077 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371837+0000", 2024-09-18T04:38:55.077 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4336e-05 2024-09-18T04:38:55.077 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.077 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.077 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:55.077 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371846+0000", 2024-09-18T04:38:55.078 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9770000000000006e-06 2024-09-18T04:38:55.078 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.078 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.078 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:55.078 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.373566+0000", 2024-09-18T04:38:55.078 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017192539999999999 2024-09-18T04:38:55.078 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.078 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.078 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:55.078 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.373595+0000", 2024-09-18T04:38:55.078 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8975999999999998e-05 2024-09-18T04:38:55.078 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.078 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.078 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:55.079 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383740+0000", 2024-09-18T04:38:55.079 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01014525 2024-09-18T04:38:55.079 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.079 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.079 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:55.079 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383780+0000", 2024-09-18T04:38:55.079 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0275999999999999e-05 2024-09-18T04:38:55.079 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.079 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:55.079 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.079 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.079 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.079 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:34 2.6 2:66289171:ns::rbd_data.133e813ab5bc.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:55.079 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.524015+0000", 2024-09-18T04:38:55.080 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 12.501223789000001, 2024-09-18T04:38:55.080 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027582111999999999, 2024-09-18T04:38:55.080 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:55.080 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:55.080 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:55.080 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:38:55.080 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:55.080 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 34 2024-09-18T04:38:55.080 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.080 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:55.080 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.080 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:55.080 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524015+0000", 2024-09-18T04:38:55.081 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.081 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.081 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.081 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:55.081 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524015+0000", 2024-09-18T04:38:55.081 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.081 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.081 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.081 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:55.081 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524012+0000", 2024-09-18T04:38:55.081 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:55.081 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.081 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.082 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:55.082 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534956+0000", 2024-09-18T04:38:55.082 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010943403000000001 2024-09-18T04:38:55.082 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.082 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.082 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:55.082 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534978+0000", 2024-09-18T04:38:55.082 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1965999999999999e-05 2024-09-18T04:38:55.082 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.082 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.082 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:55.082 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534990+0000", 2024-09-18T04:38:55.082 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2065999999999999e-05 2024-09-18T04:38:55.082 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.083 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.083 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:55.083 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535065+0000", 2024-09-18T04:38:55.083 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5523000000000002e-05 2024-09-18T04:38:55.083 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.083 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.083 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:55.083 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535208+0000", 2024-09-18T04:38:55.083 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014309500000000001 2024-09-18T04:38:55.083 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.084 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.084 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:38:55.084 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535343+0000", 2024-09-18T04:38:55.084 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013437400000000001 2024-09-18T04:38:55.084 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.084 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.084 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:55.084 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.546576+0000", 2024-09-18T04:38:55.084 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011233425999999999 2024-09-18T04:38:55.084 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.084 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.084 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:55.085 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551498+0000", 2024-09-18T04:38:55.085 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0049219349999999997 2024-09-18T04:38:55.085 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.085 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.085 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:55.085 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551554+0000", 2024-09-18T04:38:55.085 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6400999999999998e-05 2024-09-18T04:38:55.085 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.085 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.085 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:55.085 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551597+0000", 2024-09-18T04:38:55.085 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2283000000000002e-05 2024-09-18T04:38:55.086 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.086 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:55.086 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.086 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.086 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.086 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:33 2.1 e51/24 2:8b40f1c5:ns::rbd_data.133e813ab5bc.0000000000000000:head v 51'7, mlcod=51'5)", 2024-09-18T04:38:55.086 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.531767+0000", 2024-09-18T04:38:55.086 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 12.493471872000001, 2024-09-18T04:38:55.086 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015370295000000001, 2024-09-18T04:38:55.086 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:55.086 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:55.086 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:55.086 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.086 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:55.087 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531767+0000", 2024-09-18T04:38:55.087 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.087 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.087 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:55.087 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531767+0000", 2024-09-18T04:38:55.087 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.087 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.087 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:55.087 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531765+0000", 2024-09-18T04:38:55.087 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:55.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.087 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.088 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:55.088 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534258+0000", 2024-09-18T04:38:55.088 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002493561 2024-09-18T04:38:55.088 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.088 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.088 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:55.088 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534288+0000", 2024-09-18T04:38:55.088 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.976e-05 2024-09-18T04:38:55.088 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.088 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.088 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:55.088 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534300+0000", 2024-09-18T04:38:55.088 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1928e-05 2024-09-18T04:38:55.089 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.089 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.089 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:55.089 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534367+0000", 2024-09-18T04:38:55.089 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7045999999999995e-05 2024-09-18T04:38:55.089 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.089 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.089 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:55.089 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534394+0000", 2024-09-18T04:38:55.089 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6903999999999999e-05 2024-09-18T04:38:55.089 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.089 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.089 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:55.089 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.547073+0000", 2024-09-18T04:38:55.090 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012679365 2024-09-18T04:38:55.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.090 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.090 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:55.090 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.547137+0000", 2024-09-18T04:38:55.090 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3691999999999999e-05 2024-09-18T04:38:55.090 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.090 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:55.090 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.090 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.090 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:39 2.4 2:304364d1:ns::rbd_data.133e813ab5bc.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:55.090 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.535013+0000", 2024-09-18T04:38:55.090 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 12.490224937000001, 2024-09-18T04:38:55.090 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027764487000000001, 2024-09-18T04:38:55.091 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:55.091 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:55.091 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:55.091 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:38:55.091 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:55.091 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 39 2024-09-18T04:38:55.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.091 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:55.091 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.091 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:55.091 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535013+0000", 2024-09-18T04:38:55.091 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.092 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.092 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:55.092 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535013+0000", 2024-09-18T04:38:55.092 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.092 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.092 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.092 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:55.092 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535012+0000", 2024-09-18T04:38:55.092 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:55.092 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.092 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.092 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:55.092 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545016+0000", 2024-09-18T04:38:55.093 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01000322 2024-09-18T04:38:55.093 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.093 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.093 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:55.093 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545035+0000", 2024-09-18T04:38:55.093 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9845999999999999e-05 2024-09-18T04:38:55.093 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.093 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.093 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:55.093 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545045+0000", 2024-09-18T04:38:55.093 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3640000000000005e-06 2024-09-18T04:38:55.093 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.094 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.094 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:55.094 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545152+0000", 2024-09-18T04:38:55.094 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000107025 2024-09-18T04:38:55.094 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.094 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.094 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:55.094 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545338+0000", 2024-09-18T04:38:55.094 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018643500000000001 2024-09-18T04:38:55.094 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.094 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.094 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:55.094 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545492+0000", 2024-09-18T04:38:55.094 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015393199999999999 2024-09-18T04:38:55.095 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.095 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.095 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:55.095 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.553549+0000", 2024-09-18T04:38:55.095 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0080569170000000002 2024-09-18T04:38:55.095 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.095 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.095 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:55.095 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562693+0000", 2024-09-18T04:38:55.095 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0091441609999999996 2024-09-18T04:38:55.095 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.095 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.095 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:55.095 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562748+0000", 2024-09-18T04:38:55.095 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5047000000000003e-05 2024-09-18T04:38:55.096 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.096 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.096 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:55.096 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562778+0000", 2024-09-18T04:38:55.096 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9652e-05 2024-09-18T04:38:55.096 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.096 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:55.096 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.096 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.096 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.096 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:40 2.6 2:784d6cd1:ns::rbd_data.133e813ab5bc.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:55.096 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.545062+0000", 2024-09-18T04:38:55.096 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 12.480175886, 2024-09-18T04:38:55.096 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.032771623, 2024-09-18T04:38:55.097 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:55.097 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:55.097 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:55.097 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:38:55.097 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:55.097 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 40 2024-09-18T04:38:55.097 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.097 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:55.097 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.097 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:55.097 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545062+0000", 2024-09-18T04:38:55.097 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.097 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.097 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.098 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:55.098 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545062+0000", 2024-09-18T04:38:55.098 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.098 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.098 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.098 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:55.098 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545061+0000", 2024-09-18T04:38:55.098 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:55.098 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.098 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.098 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:55.098 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552477+0000", 2024-09-18T04:38:55.098 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.007415914 2024-09-18T04:38:55.098 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.098 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.099 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:55.099 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552500+0000", 2024-09-18T04:38:55.099 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3546000000000001e-05 2024-09-18T04:38:55.099 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.099 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.099 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:55.099 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552515+0000", 2024-09-18T04:38:55.099 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4606999999999999e-05 2024-09-18T04:38:55.099 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.099 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.099 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:55.099 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552661+0000", 2024-09-18T04:38:55.099 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000146007 2024-09-18T04:38:55.099 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.100 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.100 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:55.100 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552876+0000", 2024-09-18T04:38:55.100 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000214618 2024-09-18T04:38:55.100 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.100 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.100 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:38:55.100 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.553061+0000", 2024-09-18T04:38:55.100 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018564100000000001 2024-09-18T04:38:55.100 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.100 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.100 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:55.100 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.565612+0000", 2024-09-18T04:38:55.101 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012550301999999999 2024-09-18T04:38:55.101 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.101 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.101 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:55.101 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577695+0000", 2024-09-18T04:38:55.101 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012083493000000001 2024-09-18T04:38:55.101 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.101 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.101 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:55.101 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577772+0000", 2024-09-18T04:38:55.101 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7158999999999998e-05 2024-09-18T04:38:55.101 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.101 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.101 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:55.102 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577834+0000", 2024-09-18T04:38:55.102 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1851000000000001e-05 2024-09-18T04:38:55.102 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.102 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:55.102 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.102 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.102 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.102 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:43 2.6 2:759f27b6:ns::rbd_data.133e813ab5bc.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:55.102 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.555868+0000", 2024-09-18T04:38:55.102 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 12.469370392, 2024-09-18T04:38:55.102 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.022332497999999999, 2024-09-18T04:38:55.102 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:55.102 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:55.102 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:55.102 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:38:55.103 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:55.103 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 43 2024-09-18T04:38:55.103 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.103 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:55.103 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.103 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:55.103 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555868+0000", 2024-09-18T04:38:55.103 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.103 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.103 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.103 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:55.103 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555868+0000", 2024-09-18T04:38:55.103 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.103 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.104 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.104 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:55.104 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555865+0000", 2024-09-18T04:38:55.104 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999971 2024-09-18T04:38:55.104 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.104 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.104 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:55.104 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561031+0000", 2024-09-18T04:38:55.104 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0051658770000000001 2024-09-18T04:38:55.104 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.104 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.104 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:55.104 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561048+0000", 2024-09-18T04:38:55.105 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6569e-05 2024-09-18T04:38:55.105 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.105 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.105 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:55.105 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561055+0000", 2024-09-18T04:38:55.105 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.328e-06 2024-09-18T04:38:55.105 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.105 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.105 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:55.105 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561114+0000", 2024-09-18T04:38:55.105 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9540000000000003e-05 2024-09-18T04:38:55.105 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.105 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.105 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:55.105 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561314+0000", 2024-09-18T04:38:55.106 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019965799999999999 2024-09-18T04:38:55.106 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.106 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.106 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:38:55.106 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561473+0000", 2024-09-18T04:38:55.106 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000158904 2024-09-18T04:38:55.106 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.106 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.106 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:55.106 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.567015+0000", 2024-09-18T04:38:55.106 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055420859999999999 2024-09-18T04:38:55.106 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.106 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.107 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:55.107 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577982+0000", 2024-09-18T04:38:55.107 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010966872000000001 2024-09-18T04:38:55.107 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.107 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.107 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:55.107 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578162+0000", 2024-09-18T04:38:55.107 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018050999999999999 2024-09-18T04:38:55.107 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.107 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.107 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:55.107 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578200+0000", 2024-09-18T04:38:55.107 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8000999999999998e-05 2024-09-18T04:38:55.107 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.108 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:55.108 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.108 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.108 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.108 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:48 2.6 2:7c225365:ns::rbd_data.133e813ab5bc.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:55.108 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:38:55.108 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 12.461571169000001, 2024-09-18T04:38:55.108 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.024576377, 2024-09-18T04:38:55.108 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:55.108 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:55.108 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:55.108 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:38:55.108 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:55.108 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 48 2024-09-18T04:38:55.109 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.109 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:55.109 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.109 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:55.109 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:38:55.109 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.109 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.109 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.109 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:55.109 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:38:55.109 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.110 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.110 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.110 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:55.110 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.563666+0000", 2024-09-18T04:38:55.110 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:55.110 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.110 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.110 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:55.110 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570331+0000", 2024-09-18T04:38:55.110 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0066653329999999998 2024-09-18T04:38:55.110 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.110 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.110 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:55.110 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570348+0000", 2024-09-18T04:38:55.110 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6507e-05 2024-09-18T04:38:55.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.111 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.111 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:55.111 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570355+0000", 2024-09-18T04:38:55.111 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2749999999999998e-06 2024-09-18T04:38:55.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.111 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.111 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:55.111 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570396+0000", 2024-09-18T04:38:55.111 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0809000000000002e-05 2024-09-18T04:38:55.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.111 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.111 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:55.111 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570571+0000", 2024-09-18T04:38:55.112 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000174823 2024-09-18T04:38:55.112 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.112 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.112 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:38:55.112 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570722+0000", 2024-09-18T04:38:55.112 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015154599999999999 2024-09-18T04:38:55.112 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.112 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.112 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:55.112 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579756+0000", 2024-09-18T04:38:55.112 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0090337000000000004 2024-09-18T04:38:55.112 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.112 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.113 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:55.113 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588093+0000", 2024-09-18T04:38:55.113 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.008336553 2024-09-18T04:38:55.113 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.113 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.113 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:55.113 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588173+0000", 2024-09-18T04:38:55.113 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0333999999999994e-05 2024-09-18T04:38:55.113 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.113 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.113 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:55.113 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588244+0000", 2024-09-18T04:38:55.113 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0573000000000003e-05 2024-09-18T04:38:55.113 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.114 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:55.114 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.114 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.114 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.114 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:49 2.4 2:20f0b0c9:ns::rbd_data.133e813ab5bc.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:55.114 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.570370+0000", 2024-09-18T04:38:55.114 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 12.454868842, 2024-09-18T04:38:55.114 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.029633981, 2024-09-18T04:38:55.114 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:55.114 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:55.114 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:55.114 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:38:55.114 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:55.114 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 49 2024-09-18T04:38:55.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.115 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:55.115 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.115 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:55.115 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570370+0000", 2024-09-18T04:38:55.115 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.115 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.115 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:55.115 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570370+0000", 2024-09-18T04:38:55.115 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.115 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.115 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:55.116 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570369+0000", 2024-09-18T04:38:55.116 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:55.116 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.116 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.116 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:55.116 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579090+0000", 2024-09-18T04:38:55.116 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0087212339999999996 2024-09-18T04:38:55.116 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.116 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.116 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:55.116 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579109+0000", 2024-09-18T04:38:55.116 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9120999999999999e-05 2024-09-18T04:38:55.116 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.116 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.117 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:55.117 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579123+0000", 2024-09-18T04:38:55.117 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4382000000000001e-05 2024-09-18T04:38:55.117 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.117 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.117 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:55.117 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579221+0000", 2024-09-18T04:38:55.117 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.7503999999999993e-05 2024-09-18T04:38:55.117 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.117 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.117 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:55.117 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579553+0000", 2024-09-18T04:38:55.117 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00033254200000000002 2024-09-18T04:38:55.117 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.118 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.118 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:55.118 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579817+0000", 2024-09-18T04:38:55.118 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00026317099999999998 2024-09-18T04:38:55.118 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.118 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.118 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:55.118 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.599905+0000", 2024-09-18T04:38:55.118 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020088575000000001 2024-09-18T04:38:55.118 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.118 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.118 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:55.118 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.599923+0000", 2024-09-18T04:38:55.118 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7425999999999999e-05 2024-09-18T04:38:55.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.119 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.119 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:55.119 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.599973+0000", 2024-09-18T04:38:55.119 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0649999999999998e-05 2024-09-18T04:38:55.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.119 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.119 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:55.119 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.600004+0000", 2024-09-18T04:38:55.119 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0361999999999998e-05 2024-09-18T04:38:55.119 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.119 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:55.119 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.120 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.120 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:50 2.1 e51/24 2:8c706374:ns::rbd_data.133e813ab5bc.000000000000000a:head v 51'8, mlcod=51'6)", 2024-09-18T04:38:55.120 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.577029+0000", 2024-09-18T04:38:55.120 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 12.448209471, 2024-09-18T04:38:55.120 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.017362855, 2024-09-18T04:38:55.120 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:55.120 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:55.120 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:55.120 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.120 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:55.120 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577029+0000", 2024-09-18T04:38:55.120 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.120 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.120 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.121 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:55.121 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577029+0000", 2024-09-18T04:38:55.121 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.121 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.121 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.121 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:55.121 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577027+0000", 2024-09-18T04:38:55.121 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:55.121 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.121 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.121 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:55.121 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578880+0000", 2024-09-18T04:38:55.121 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0018532080000000001 2024-09-18T04:38:55.121 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.122 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.122 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:55.122 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578906+0000", 2024-09-18T04:38:55.122 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5573999999999999e-05 2024-09-18T04:38:55.122 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.122 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.122 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:55.122 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578920+0000", 2024-09-18T04:38:55.122 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4483e-05 2024-09-18T04:38:55.122 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.122 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.122 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:55.122 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578983+0000", 2024-09-18T04:38:55.122 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2248000000000004e-05 2024-09-18T04:38:55.123 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.123 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.123 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:55.123 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579016+0000", 2024-09-18T04:38:55.123 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3364000000000001e-05 2024-09-18T04:38:55.123 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.123 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.123 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:55.123 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594342+0000", 2024-09-18T04:38:55.123 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01532564 2024-09-18T04:38:55.123 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.123 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.123 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:55.123 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594392+0000", 2024-09-18T04:38:55.123 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0068999999999998e-05 2024-09-18T04:38:55.124 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.124 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:55.124 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.124 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.124 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.124 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:45 2.0 e51/17 2:0fbfa3dc:ns::rbd_data.133e813ab5bc.0000000000000007:head v 51'24, mlcod=51'22)", 2024-09-18T04:38:55.124 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.581892+0000", 2024-09-18T04:38:55.124 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 12.443346799, 2024-09-18T04:38:55.124 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015646006, 2024-09-18T04:38:55.124 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:55.124 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:55.124 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:55.124 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.124 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:55.125 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581892+0000", 2024-09-18T04:38:55.125 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.125 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.125 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.125 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:55.125 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581892+0000", 2024-09-18T04:38:55.125 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.125 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.125 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.125 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:55.125 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581890+0000", 2024-09-18T04:38:55.125 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:55.125 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.125 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.125 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:55.126 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586188+0000", 2024-09-18T04:38:55.126 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0042985769999999996 2024-09-18T04:38:55.126 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.126 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.126 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:55.126 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586210+0000", 2024-09-18T04:38:55.126 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1069000000000001e-05 2024-09-18T04:38:55.126 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.126 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.126 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:55.126 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586221+0000", 2024-09-18T04:38:55.126 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1287000000000001e-05 2024-09-18T04:38:55.126 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.126 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.127 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:55.127 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586313+0000", 2024-09-18T04:38:55.127 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1935999999999995e-05 2024-09-18T04:38:55.127 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.127 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.127 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:55.127 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586368+0000", 2024-09-18T04:38:55.127 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5578000000000002e-05 2024-09-18T04:38:55.127 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.127 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.127 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:55.127 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.597473+0000", 2024-09-18T04:38:55.127 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011105139999999999 2024-09-18T04:38:55.127 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.128 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.128 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:55.128 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.597538+0000", 2024-09-18T04:38:55.128 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4133000000000004e-05 2024-09-18T04:38:55.128 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.128 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:55.128 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.128 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.128 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.128 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:53 2.4 2:39f7f174:ns::rbd_data.133e813ab5bc.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:55.128 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.584263+0000", 2024-09-18T04:38:55.128 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 12.440974925000001, 2024-09-18T04:38:55.128 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018701180000000001, 2024-09-18T04:38:55.129 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:55.129 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:55.129 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:55.129 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:38:55.129 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:55.129 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 53 2024-09-18T04:38:55.129 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.129 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:55.129 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.129 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:55.129 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584263+0000", 2024-09-18T04:38:55.129 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.129 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.129 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.130 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:55.130 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584263+0000", 2024-09-18T04:38:55.130 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.130 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.130 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.130 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:55.130 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584262+0000", 2024-09-18T04:38:55.130 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:55.130 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.130 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.130 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:55.130 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.591617+0000", 2024-09-18T04:38:55.130 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0073550600000000001 2024-09-18T04:38:55.130 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.131 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.131 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:55.131 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.591637+0000", 2024-09-18T04:38:55.131 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0432999999999999e-05 2024-09-18T04:38:55.131 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.131 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.131 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:55.131 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.591646+0000", 2024-09-18T04:38:55.131 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7849999999999997e-06 2024-09-18T04:38:55.131 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.131 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.131 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:55.131 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593478+0000", 2024-09-18T04:38:55.131 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001831495 2024-09-18T04:38:55.132 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.132 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.132 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:55.132 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593661+0000", 2024-09-18T04:38:55.132 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018364999999999999 2024-09-18T04:38:55.132 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.132 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.132 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:55.132 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593834+0000", 2024-09-18T04:38:55.132 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000173065 2024-09-18T04:38:55.132 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.132 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.132 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:55.132 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.600670+0000", 2024-09-18T04:38:55.133 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0068354970000000003 2024-09-18T04:38:55.133 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.133 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.133 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:55.133 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602851+0000", 2024-09-18T04:38:55.133 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002181653 2024-09-18T04:38:55.133 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.133 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.133 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:55.133 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602930+0000", 2024-09-18T04:38:55.133 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8045000000000002e-05 2024-09-18T04:38:55.133 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.133 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.133 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:55.134 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602965+0000", 2024-09-18T04:38:55.134 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5102999999999999e-05 2024-09-18T04:38:55.134 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.134 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:55.134 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.134 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.134 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.134 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:56 2.6 2:72453c3e:ns::rbd_data.133e813ab5bc.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:55.134 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.595988+0000", 2024-09-18T04:38:55.134 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 12.429250523, 2024-09-18T04:38:55.134 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.030296172, 2024-09-18T04:38:55.134 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:55.134 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:55.134 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:55.134 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:38:55.135 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:55.135 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 56 2024-09-18T04:38:55.135 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.135 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:55.135 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.135 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:55.135 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.595988+0000", 2024-09-18T04:38:55.135 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.135 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.135 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.135 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:55.135 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.595988+0000", 2024-09-18T04:38:55.135 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.135 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.135 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.136 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:55.136 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.595986+0000", 2024-09-18T04:38:55.136 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:55.136 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.136 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.136 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:55.136 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605262+0000", 2024-09-18T04:38:55.136 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0092767240000000001 2024-09-18T04:38:55.136 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.136 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.136 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:55.136 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605282+0000", 2024-09-18T04:38:55.136 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9307e-05 2024-09-18T04:38:55.136 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.137 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.137 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:55.137 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605292+0000", 2024-09-18T04:38:55.137 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9510000000000001e-06 2024-09-18T04:38:55.137 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.137 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.137 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:55.137 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605369+0000", 2024-09-18T04:38:55.137 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7846e-05 2024-09-18T04:38:55.137 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.137 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.137 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:55.137 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605668+0000", 2024-09-18T04:38:55.137 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00029896099999999999 2024-09-18T04:38:55.137 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.138 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.138 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:38:55.138 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605906+0000", 2024-09-18T04:38:55.138 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000237758 2024-09-18T04:38:55.138 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.138 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.138 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:55.138 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.613532+0000", 2024-09-18T04:38:55.138 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0076260590000000001 2024-09-18T04:38:55.138 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.138 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.138 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:55.138 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.626116+0000", 2024-09-18T04:38:55.138 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012584092999999999 2024-09-18T04:38:55.138 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.139 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.139 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:55.139 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.626216+0000", 2024-09-18T04:38:55.139 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9630999999999995e-05 2024-09-18T04:38:55.139 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.139 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.139 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:55.139 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.626284+0000", 2024-09-18T04:38:55.139 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8034999999999995e-05 2024-09-18T04:38:55.139 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.139 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:55.139 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.139 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.139 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:55.139 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:55.141 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:38:55.142 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:55.137+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:55.242 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:55.238+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:55.267 INFO:tasks.workunit.client.0.smithi115.stderr:++ rbd -p rbd --format xml du testrbdnbd17342 2024-09-18T04:38:55.267 INFO:tasks.workunit.client.0.smithi115.stderr:++ xmlstarlet sel -t -m //stats/images/image/provisioned_size -v . 2024-09-18T04:38:55.298 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:55.298 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:55.299 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:38:55.299 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:55.309 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:38:55.343 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:55.337+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:55.346 INFO:tasks.workunit.client.0.smithi115.stderr:+ provisioned=67108864 2024-09-18T04:38:55.347 INFO:tasks.workunit.client.0.smithi115.stderr:++ rbd -p rbd --format xml du testrbdnbd17342 2024-09-18T04:38:55.347 INFO:tasks.workunit.client.0.smithi115.stderr:++ xmlstarlet sel -t -m //stats/images/image/used_size -v . 2024-09-18T04:38:55.424 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:55.425 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:55.425 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:38:55.425 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:38:55.425 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:55.426 INFO:tasks.workunit.client.0.smithi115.stderr:+ used=67108864 2024-09-18T04:38:55.426 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 67108864 -eq 67108864 ']' 2024-09-18T04:38:55.426 INFO:tasks.workunit.client.0.smithi115.stderr:+ unmap_device /dev/nbd0 17851 2024-09-18T04:38:55.426 INFO:tasks.workunit.client.0.smithi115.stderr:+ local args=/dev/nbd0 2024-09-18T04:38:55.426 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pid=17851 2024-09-18T04:38:55.426 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-18T04:38:55.426 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:38:55.427 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:38:55.428 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:38:55.429 INFO:tasks.workunit.client.0.smithi115.stderr:++ which rbd 2024-09-18T04:38:55.430 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/bin/rbd 2024-09-18T04:38:55.430 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:38:55.430 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-18T04:38:55.436 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:38:55.444 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:55.438+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:55.535 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd device --device-type nbd list 2024-09-18T04:38:55.536 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false grep '^17851\b' 2024-09-18T04:38:55.536 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep '^17851\b' 2024-09-18T04:38:55.545 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:55.538+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:55.548 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:55.548 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:38:55.548 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:38:55.548 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:38:55.548 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.548 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:33 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:55.548 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.261182+0000", 2024-09-18T04:38:55.548 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 13.281891286, 2024-09-18T04:38:55.549 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.037082208999999998, 2024-09-18T04:38:55.549 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:55.549 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:55.549 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:55.549 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:38:55.549 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:38:55.549 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 33 2024-09-18T04:38:55.549 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.549 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:55.549 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.549 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:55.549 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.261182+0000", 2024-09-18T04:38:55.549 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.549 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.550 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.550 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:55.550 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.261182+0000", 2024-09-18T04:38:55.550 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.550 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.550 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.550 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:55.550 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.261180+0000", 2024-09-18T04:38:55.550 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:55.550 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.550 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.550 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:55.550 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278669+0000", 2024-09-18T04:38:55.550 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.017488877999999999 2024-09-18T04:38:55.551 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.551 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.551 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:55.551 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278689+0000", 2024-09-18T04:38:55.551 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0047999999999998e-05 2024-09-18T04:38:55.551 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.551 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.551 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:55.551 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278699+0000", 2024-09-18T04:38:55.551 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6800000000000005e-06 2024-09-18T04:38:55.551 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.551 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.551 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:55.551 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278775+0000", 2024-09-18T04:38:55.552 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6087999999999994e-05 2024-09-18T04:38:55.552 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.552 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.552 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:55.552 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278935+0000", 2024-09-18T04:38:55.552 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000159834 2024-09-18T04:38:55.552 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.552 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.552 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:55.552 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279064+0000", 2024-09-18T04:38:55.552 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012954199999999999 2024-09-18T04:38:55.552 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.553 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.553 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:55.553 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295562+0000", 2024-09-18T04:38:55.553 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016497583999999999 2024-09-18T04:38:55.553 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.553 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.553 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:55.553 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.298147+0000", 2024-09-18T04:38:55.553 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002584982 2024-09-18T04:38:55.553 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.553 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.553 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:55.553 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.298217+0000", 2024-09-18T04:38:55.553 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0400000000000004e-05 2024-09-18T04:38:55.554 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.554 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.554 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:55.554 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.298265+0000", 2024-09-18T04:38:55.554 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7580000000000002e-05 2024-09-18T04:38:55.554 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.554 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:55.554 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.554 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.554 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.554 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:34 2.3 e51/24 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head v 51'3, mlcod=18'1)", 2024-09-18T04:38:55.554 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.273067+0000", 2024-09-18T04:38:55.554 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 13.270006496000001, 2024-09-18T04:38:55.555 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.021784103999999999, 2024-09-18T04:38:55.555 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:55.555 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:55.555 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:55.555 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.555 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:55.555 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.273067+0000", 2024-09-18T04:38:55.555 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.555 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.555 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.555 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:55.555 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.273067+0000", 2024-09-18T04:38:55.555 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.555 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.555 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.556 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:55.556 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.273060+0000", 2024-09-18T04:38:55.556 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999924 2024-09-18T04:38:55.556 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.556 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.556 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:55.556 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278596+0000", 2024-09-18T04:38:55.556 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055365839999999998 2024-09-18T04:38:55.556 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.556 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.556 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:55.556 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278637+0000", 2024-09-18T04:38:55.556 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0596000000000002e-05 2024-09-18T04:38:55.556 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.557 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.557 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:55.557 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278653+0000", 2024-09-18T04:38:55.557 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6626999999999999e-05 2024-09-18T04:38:55.557 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.557 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.557 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:55.557 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278748+0000", 2024-09-18T04:38:55.557 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4483999999999999e-05 2024-09-18T04:38:55.557 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.557 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.557 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:55.558 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278799+0000", 2024-09-18T04:38:55.558 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1484999999999999e-05 2024-09-18T04:38:55.558 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.558 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.558 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:55.558 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.294785+0000", 2024-09-18T04:38:55.558 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015985256999999999 2024-09-18T04:38:55.558 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.558 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.558 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:55.558 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.294851+0000", 2024-09-18T04:38:55.558 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6728000000000002e-05 2024-09-18T04:38:55.558 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.558 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:55.559 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.559 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.559 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.559 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:37 2.5 2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:55.559 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.278722+0000", 2024-09-18T04:38:55.559 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 13.264351446999999, 2024-09-18T04:38:55.559 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.026323801000000001, 2024-09-18T04:38:55.559 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:55.559 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:55.559 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:55.559 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:38:55.559 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:38:55.559 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 37 2024-09-18T04:38:55.560 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.560 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:55.560 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.560 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:55.560 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278722+0000", 2024-09-18T04:38:55.560 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.560 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.560 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.560 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:55.560 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278722+0000", 2024-09-18T04:38:55.560 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.560 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.560 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.560 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:55.561 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278720+0000", 2024-09-18T04:38:55.561 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:55.561 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.561 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.561 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:55.561 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.291875+0000", 2024-09-18T04:38:55.561 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013154414 2024-09-18T04:38:55.561 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.561 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.561 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:55.561 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.291901+0000", 2024-09-18T04:38:55.561 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6339999999999999e-05 2024-09-18T04:38:55.561 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.562 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.562 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:55.562 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.291914+0000", 2024-09-18T04:38:55.562 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3043e-05 2024-09-18T04:38:55.562 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.562 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.562 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:55.562 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.291975+0000", 2024-09-18T04:38:55.562 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0949999999999998e-05 2024-09-18T04:38:55.562 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.562 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.562 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:55.562 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292173+0000", 2024-09-18T04:38:55.562 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019825499999999999 2024-09-18T04:38:55.563 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.563 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.563 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:55.563 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292300+0000", 2024-09-18T04:38:55.563 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000126616 2024-09-18T04:38:55.563 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.563 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.563 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:55.563 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304334+0000", 2024-09-18T04:38:55.563 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012033934 2024-09-18T04:38:55.563 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.563 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.563 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:55.564 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304901+0000", 2024-09-18T04:38:55.564 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00056694099999999995 2024-09-18T04:38:55.564 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.564 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.564 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:55.564 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304981+0000", 2024-09-18T04:38:55.564 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9974000000000004e-05 2024-09-18T04:38:55.564 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.564 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.564 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:55.564 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.305046+0000", 2024-09-18T04:38:55.564 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5184000000000005e-05 2024-09-18T04:38:55.564 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.564 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:55.565 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.565 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.565 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.565 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:49 2.5 2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:55.565 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.314271+0000", 2024-09-18T04:38:55.565 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 13.228802845000001, 2024-09-18T04:38:55.565 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.029779953000000001, 2024-09-18T04:38:55.565 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:55.565 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:55.565 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:55.566 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:38:55.566 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:38:55.566 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 49 2024-09-18T04:38:55.566 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.566 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:55.566 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.566 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:55.566 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.314271+0000", 2024-09-18T04:38:55.566 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.566 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.566 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.566 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:55.566 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.314271+0000", 2024-09-18T04:38:55.566 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.567 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.567 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.567 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:55.567 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.314269+0000", 2024-09-18T04:38:55.567 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:55.567 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.567 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.567 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:55.567 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.324780+0000", 2024-09-18T04:38:55.567 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010511609999999999 2024-09-18T04:38:55.567 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.567 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.567 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:55.567 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.324821+0000", 2024-09-18T04:38:55.568 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0899000000000003e-05 2024-09-18T04:38:55.568 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.568 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.568 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:55.568 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.324840+0000", 2024-09-18T04:38:55.568 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8947e-05 2024-09-18T04:38:55.568 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.568 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.568 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:55.568 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.324936+0000", 2024-09-18T04:38:55.568 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6342999999999996e-05 2024-09-18T04:38:55.568 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.569 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.569 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:55.569 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325284+0000", 2024-09-18T04:38:55.569 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000347146 2024-09-18T04:38:55.569 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.569 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.569 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:55.569 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325586+0000", 2024-09-18T04:38:55.569 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00030199799999999999 2024-09-18T04:38:55.569 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.569 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.569 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:55.569 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343710+0000", 2024-09-18T04:38:55.569 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018124583999999999 2024-09-18T04:38:55.570 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.570 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.570 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:55.570 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343991+0000", 2024-09-18T04:38:55.570 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00028085600000000001 2024-09-18T04:38:55.570 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.570 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.570 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:55.570 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344028+0000", 2024-09-18T04:38:55.570 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6637000000000002e-05 2024-09-18T04:38:55.570 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.570 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.570 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:55.570 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344051+0000", 2024-09-18T04:38:55.571 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3260000000000001e-05 2024-09-18T04:38:55.571 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.571 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:55.571 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.571 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.571 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.571 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:50 2.3 e51/24 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head v 51'4, mlcod=51'2)", 2024-09-18T04:38:55.571 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.323066+0000", 2024-09-18T04:38:55.571 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 13.220007452000001, 2024-09-18T04:38:55.571 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.022135774, 2024-09-18T04:38:55.571 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:55.571 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:55.571 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:55.572 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.572 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:55.572 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.323066+0000", 2024-09-18T04:38:55.572 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.572 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.572 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.572 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:55.572 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.323066+0000", 2024-09-18T04:38:55.572 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.572 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.572 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.572 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:55.572 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.323065+0000", 2024-09-18T04:38:55.572 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:55.573 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.573 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.573 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:55.573 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.328664+0000", 2024-09-18T04:38:55.573 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055993600000000003 2024-09-18T04:38:55.573 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.573 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.573 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:55.573 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.328692+0000", 2024-09-18T04:38:55.573 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7654e-05 2024-09-18T04:38:55.573 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.573 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.574 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:55.574 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.328706+0000", 2024-09-18T04:38:55.574 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4097000000000001e-05 2024-09-18T04:38:55.574 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.574 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.574 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:55.574 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.328816+0000", 2024-09-18T04:38:55.574 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000110281 2024-09-18T04:38:55.574 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.574 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.574 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:55.574 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.328848+0000", 2024-09-18T04:38:55.574 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1837e-05 2024-09-18T04:38:55.575 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.575 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.575 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:55.575 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345160+0000", 2024-09-18T04:38:55.575 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016312099 2024-09-18T04:38:55.575 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.575 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.575 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:55.575 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345202+0000", 2024-09-18T04:38:55.575 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1857999999999998e-05 2024-09-18T04:38:55.575 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.575 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:55.575 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.575 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.576 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.576 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:58 2.5 2:b227e6b1:::rbd_data.125fc09df6d1.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:55.576 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.351564+0000", 2024-09-18T04:38:55.576 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 13.191509999999999, 2024-09-18T04:38:55.576 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.032125956999999997, 2024-09-18T04:38:55.576 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:55.576 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:55.576 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:55.576 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:38:55.576 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:38:55.576 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 58 2024-09-18T04:38:55.576 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.576 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:55.577 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.577 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:55.577 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.351564+0000", 2024-09-18T04:38:55.577 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.577 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.577 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.577 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:55.577 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.351564+0000", 2024-09-18T04:38:55.577 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.577 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.577 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.577 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:55.578 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.351561+0000", 2024-09-18T04:38:55.578 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:55.578 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.578 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.578 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:55.578 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364390+0000", 2024-09-18T04:38:55.578 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012829063999999999 2024-09-18T04:38:55.578 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.578 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.578 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:55.578 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364418+0000", 2024-09-18T04:38:55.578 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7307000000000001e-05 2024-09-18T04:38:55.578 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.578 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.579 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:55.579 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364428+0000", 2024-09-18T04:38:55.579 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0675e-05 2024-09-18T04:38:55.579 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.579 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.579 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:55.579 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364523+0000", 2024-09-18T04:38:55.579 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4698999999999996e-05 2024-09-18T04:38:55.579 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.579 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.579 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:55.579 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364785+0000", 2024-09-18T04:38:55.579 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00026170099999999998 2024-09-18T04:38:55.580 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.580 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.580 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:55.580 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.365039+0000", 2024-09-18T04:38:55.580 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00025462299999999999 2024-09-18T04:38:55.580 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.580 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.580 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:55.580 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383143+0000", 2024-09-18T04:38:55.580 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018103768999999999 2024-09-18T04:38:55.580 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.580 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.580 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:55.581 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383582+0000", 2024-09-18T04:38:55.581 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00043860600000000002 2024-09-18T04:38:55.581 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.581 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.581 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:55.581 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383644+0000", 2024-09-18T04:38:55.581 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2682999999999996e-05 2024-09-18T04:38:55.581 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.581 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.581 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:55.581 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383690+0000", 2024-09-18T04:38:55.581 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5247999999999998e-05 2024-09-18T04:38:55.581 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.582 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:55.582 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.582 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.582 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.582 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:33 2.1 2:8b40f1c5:ns::rbd_data.133e813ab5bc.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:55.582 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.523765+0000", 2024-09-18T04:38:55.582 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 13.01930896, 2024-09-18T04:38:55.582 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027472973000000001, 2024-09-18T04:38:55.582 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:55.582 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:55.582 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:55.582 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:38:55.582 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:55.582 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 33 2024-09-18T04:38:55.583 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.583 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:55.583 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.583 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:55.583 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.523765+0000", 2024-09-18T04:38:55.583 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.583 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.583 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.583 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:55.583 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.523765+0000", 2024-09-18T04:38:55.583 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.583 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.583 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.584 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:55.584 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.523762+0000", 2024-09-18T04:38:55.584 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:55.584 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.584 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.584 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:55.584 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531127+0000", 2024-09-18T04:38:55.584 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0073644560000000001 2024-09-18T04:38:55.584 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.584 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.584 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:55.584 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531189+0000", 2024-09-18T04:38:55.585 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2315000000000002e-05 2024-09-18T04:38:55.585 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.585 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.585 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:55.585 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531211+0000", 2024-09-18T04:38:55.585 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1542000000000001e-05 2024-09-18T04:38:55.585 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.585 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.585 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:55.585 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531314+0000", 2024-09-18T04:38:55.585 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000103582 2024-09-18T04:38:55.585 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.585 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.585 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:55.586 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531493+0000", 2024-09-18T04:38:55.586 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000178433 2024-09-18T04:38:55.586 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.586 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.586 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:55.586 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531653+0000", 2024-09-18T04:38:55.586 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000160388 2024-09-18T04:38:55.586 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.586 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.586 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:55.586 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.547345+0000", 2024-09-18T04:38:55.586 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015691742000000002 2024-09-18T04:38:55.586 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.586 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.587 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:55.587 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551076+0000", 2024-09-18T04:38:55.587 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003730957 2024-09-18T04:38:55.587 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.587 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.587 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:55.587 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551176+0000", 2024-09-18T04:38:55.587 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010002599999999999 2024-09-18T04:38:55.587 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.587 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.587 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:55.587 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551238+0000", 2024-09-18T04:38:55.587 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1945000000000004e-05 2024-09-18T04:38:55.588 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.588 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:55.588 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.588 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.588 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.588 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:36 2.5 2:bd5b8dd8:ns::rbd_data.133e813ab5bc.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:55.588 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.531253+0000", 2024-09-18T04:38:55.588 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 13.011820651000001, 2024-09-18T04:38:55.588 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027295212999999999, 2024-09-18T04:38:55.588 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:55.588 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:55.588 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:55.588 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:38:55.589 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:55.589 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 36 2024-09-18T04:38:55.589 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.589 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:55.589 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.589 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:55.589 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531253+0000", 2024-09-18T04:38:55.589 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.589 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.589 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.589 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:55.589 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531253+0000", 2024-09-18T04:38:55.589 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.590 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.590 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.590 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:55.590 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531251+0000", 2024-09-18T04:38:55.590 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:55.590 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.590 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.590 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:55.590 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536540+0000", 2024-09-18T04:38:55.590 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0052885839999999998 2024-09-18T04:38:55.590 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.590 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.590 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:55.591 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536604+0000", 2024-09-18T04:38:55.591 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4824000000000001e-05 2024-09-18T04:38:55.591 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.591 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.591 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:55.591 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536621+0000", 2024-09-18T04:38:55.591 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6215999999999999e-05 2024-09-18T04:38:55.591 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.591 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.591 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:55.591 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536768+0000", 2024-09-18T04:38:55.591 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014767900000000001 2024-09-18T04:38:55.591 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.591 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.592 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:55.592 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537084+0000", 2024-09-18T04:38:55.592 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00031553499999999998 2024-09-18T04:38:55.592 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.592 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.592 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:55.592 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537300+0000", 2024-09-18T04:38:55.592 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000216145 2024-09-18T04:38:55.592 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.592 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.592 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:55.592 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555326+0000", 2024-09-18T04:38:55.592 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018025861000000001 2024-09-18T04:38:55.593 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.593 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.593 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:55.593 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558449+0000", 2024-09-18T04:38:55.593 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0031229750000000001 2024-09-18T04:38:55.593 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.593 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.593 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:55.593 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558502+0000", 2024-09-18T04:38:55.593 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3572000000000001e-05 2024-09-18T04:38:55.593 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.593 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.593 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:55.594 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558548+0000", 2024-09-18T04:38:55.594 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5921999999999998e-05 2024-09-18T04:38:55.594 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.594 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:55.594 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.594 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.594 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.594 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:37 2.5 2:bf39b8e4:ns::rbd_data.133e813ab5bc.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:55.594 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.536664+0000", 2024-09-18T04:38:55.594 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 13.006409235, 2024-09-18T04:38:55.594 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.026331765, 2024-09-18T04:38:55.594 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:55.594 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:55.594 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:55.595 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:38:55.595 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:55.595 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 37 2024-09-18T04:38:55.595 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.595 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:55.595 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.595 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:55.595 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536664+0000", 2024-09-18T04:38:55.595 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.595 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.595 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.595 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:55.595 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536664+0000", 2024-09-18T04:38:55.595 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.596 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.596 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.596 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:55.596 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536662+0000", 2024-09-18T04:38:55.596 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:55.596 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.596 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.596 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:55.596 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.543699+0000", 2024-09-18T04:38:55.596 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0070367119999999997 2024-09-18T04:38:55.596 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.596 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.597 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:55.597 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.543749+0000", 2024-09-18T04:38:55.597 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0476999999999998e-05 2024-09-18T04:38:55.597 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.597 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.597 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:55.597 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.543800+0000", 2024-09-18T04:38:55.597 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0822999999999998e-05 2024-09-18T04:38:55.597 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.597 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.597 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:55.597 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.547868+0000", 2024-09-18T04:38:55.597 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0040679339999999996 2024-09-18T04:38:55.598 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.598 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.598 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:55.598 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548087+0000", 2024-09-18T04:38:55.598 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000218994 2024-09-18T04:38:55.598 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.598 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.598 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:55.598 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548383+0000", 2024-09-18T04:38:55.598 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000295588 2024-09-18T04:38:55.598 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.598 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.598 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:55.598 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562622+0000", 2024-09-18T04:38:55.598 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014239915000000001 2024-09-18T04:38:55.599 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.599 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.599 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:55.599 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562924+0000", 2024-09-18T04:38:55.599 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00030127600000000001 2024-09-18T04:38:55.599 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.599 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.599 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:55.599 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562957+0000", 2024-09-18T04:38:55.599 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3365999999999999e-05 2024-09-18T04:38:55.599 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.599 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.599 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:55.600 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562996+0000", 2024-09-18T04:38:55.600 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9153999999999998e-05 2024-09-18T04:38:55.600 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.600 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:55.600 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.600 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.600 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.600 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:42 2.3 e51/24 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head v 51'14, mlcod=51'11)", 2024-09-18T04:38:55.600 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.554081+0000", 2024-09-18T04:38:55.600 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 12.988992974, 2024-09-18T04:38:55.600 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0080081739999999998, 2024-09-18T04:38:55.600 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:55.600 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:55.601 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:55.601 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.601 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:55.601 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.554081+0000", 2024-09-18T04:38:55.601 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.601 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.601 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.601 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:55.601 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.554081+0000", 2024-09-18T04:38:55.601 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.601 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.601 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.601 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:55.602 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.554079+0000", 2024-09-18T04:38:55.602 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:55.602 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.602 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.602 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:55.602 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.554092+0000", 2024-09-18T04:38:55.602 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3032e-05 2024-09-18T04:38:55.602 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.602 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.602 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:55.602 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.554105+0000", 2024-09-18T04:38:55.602 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3217000000000001e-05 2024-09-18T04:38:55.602 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.603 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.603 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:55.603 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.554115+0000", 2024-09-18T04:38:55.603 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9089999999999995e-06 2024-09-18T04:38:55.603 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.603 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.603 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:55.603 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.554165+0000", 2024-09-18T04:38:55.603 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9644000000000001e-05 2024-09-18T04:38:55.603 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.603 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.603 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:55.603 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.554188+0000", 2024-09-18T04:38:55.603 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3753000000000001e-05 2024-09-18T04:38:55.604 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.604 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.604 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:55.604 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562047+0000", 2024-09-18T04:38:55.604 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0078585519999999996 2024-09-18T04:38:55.604 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.604 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.604 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:55.604 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562089+0000", 2024-09-18T04:38:55.604 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1940000000000002e-05 2024-09-18T04:38:55.604 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.604 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:55.604 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.605 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.605 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.605 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:50 2.1 2:8c706374:ns::rbd_data.133e813ab5bc.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:55.605 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.570513+0000", 2024-09-18T04:38:55.605 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 12.972560953, 2024-09-18T04:38:55.605 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.024182995999999998, 2024-09-18T04:38:55.605 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:55.605 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:55.605 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:55.605 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:38:55.605 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:55.605 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 50 2024-09-18T04:38:55.605 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.605 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:55.606 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.606 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:55.606 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570513+0000", 2024-09-18T04:38:55.606 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.606 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.606 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.606 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:55.606 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570513+0000", 2024-09-18T04:38:55.606 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.606 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.606 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.606 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:55.606 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570511+0000", 2024-09-18T04:38:55.607 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:55.607 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.607 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.607 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:55.607 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.576327+0000", 2024-09-18T04:38:55.607 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0058162530000000004 2024-09-18T04:38:55.607 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.607 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.607 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:55.607 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.576355+0000", 2024-09-18T04:38:55.607 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8030000000000001e-05 2024-09-18T04:38:55.607 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.607 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.607 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:55.608 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.576366+0000", 2024-09-18T04:38:55.608 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1139e-05 2024-09-18T04:38:55.608 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.608 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.608 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:55.608 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.576458+0000", 2024-09-18T04:38:55.608 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1780999999999994e-05 2024-09-18T04:38:55.608 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.608 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.608 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:55.608 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.576701+0000", 2024-09-18T04:38:55.608 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00024303899999999999 2024-09-18T04:38:55.608 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.609 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.609 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:55.609 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.576876+0000", 2024-09-18T04:38:55.609 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000174637 2024-09-18T04:38:55.609 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.609 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.609 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:55.609 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588432+0000", 2024-09-18T04:38:55.609 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011555943000000001 2024-09-18T04:38:55.609 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.609 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.609 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:55.609 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594573+0000", 2024-09-18T04:38:55.610 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0061408670000000004 2024-09-18T04:38:55.610 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.610 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.610 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:55.610 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594640+0000", 2024-09-18T04:38:55.610 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7404999999999996e-05 2024-09-18T04:38:55.610 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.610 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.610 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:55.610 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594696+0000", 2024-09-18T04:38:55.610 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5612999999999998e-05 2024-09-18T04:38:55.610 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.610 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:55.611 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.611 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.611 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.611 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:60 2.5 2:a06f51e4:ns::rbd_data.133e813ab5bc.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:55.611 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.604753+0000", 2024-09-18T04:38:55.611 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 12.938320834000001, 2024-09-18T04:38:55.611 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014400181, 2024-09-18T04:38:55.611 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:55.611 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:55.611 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:55.611 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:38:55.611 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:55.611 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 60 2024-09-18T04:38:55.612 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.612 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:55.612 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.612 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:55.612 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.604753+0000", 2024-09-18T04:38:55.612 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.612 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.612 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.612 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:55.612 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.604753+0000", 2024-09-18T04:38:55.612 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.612 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.612 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.613 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:55.613 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.604751+0000", 2024-09-18T04:38:55.613 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:55.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.613 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.613 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:55.613 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.608980+0000", 2024-09-18T04:38:55.613 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0042292420000000002 2024-09-18T04:38:55.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.613 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.613 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:55.613 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609014+0000", 2024-09-18T04:38:55.613 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3572999999999999e-05 2024-09-18T04:38:55.614 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.614 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.614 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:55.614 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609032+0000", 2024-09-18T04:38:55.614 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8150000000000001e-05 2024-09-18T04:38:55.614 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.614 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.614 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:55.614 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609090+0000", 2024-09-18T04:38:55.614 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8214999999999997e-05 2024-09-18T04:38:55.614 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.614 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.614 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:55.614 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609235+0000", 2024-09-18T04:38:55.615 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014437799999999999 2024-09-18T04:38:55.615 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.615 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.615 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:55.615 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609360+0000", 2024-09-18T04:38:55.615 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012526099999999999 2024-09-18T04:38:55.615 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.615 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.615 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:55.615 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.614952+0000", 2024-09-18T04:38:55.615 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055922300000000001 2024-09-18T04:38:55.615 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.615 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.616 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:55.616 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.619070+0000", 2024-09-18T04:38:55.616 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0041178250000000003 2024-09-18T04:38:55.616 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.616 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.616 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:55.616 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.619117+0000", 2024-09-18T04:38:55.616 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7275999999999999e-05 2024-09-18T04:38:55.616 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.616 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.616 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:55.616 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.619153+0000", 2024-09-18T04:38:55.616 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5970000000000003e-05 2024-09-18T04:38:55.616 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.617 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:55.617 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.617 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.617 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.617 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4805.0:592 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [write 393216~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:38:55.617 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.023386+0000", 2024-09-18T04:38:55.617 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.5196881290000004, 2024-09-18T04:38:55.617 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0086813870000000005, 2024-09-18T04:38:55.617 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:55.617 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:55.617 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:55.617 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4805", 2024-09-18T04:38:55.617 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:38:55.618 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 592 2024-09-18T04:38:55.618 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.618 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:55.618 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.618 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:55.618 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023386+0000", 2024-09-18T04:38:55.618 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.618 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.618 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.618 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:55.618 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023386+0000", 2024-09-18T04:38:55.618 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.618 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.618 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.619 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:55.619 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023384+0000", 2024-09-18T04:38:55.619 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:55.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.619 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.619 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:55.619 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023616+0000", 2024-09-18T04:38:55.619 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00023165600000000001 2024-09-18T04:38:55.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.619 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.619 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:55.619 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023637+0000", 2024-09-18T04:38:55.619 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1163e-05 2024-09-18T04:38:55.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.620 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.620 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:55.620 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023652+0000", 2024-09-18T04:38:55.620 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4562000000000001e-05 2024-09-18T04:38:55.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.620 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.620 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:55.620 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.027732+0000", 2024-09-18T04:38:55.620 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0040803929999999999 2024-09-18T04:38:55.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.620 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.620 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:55.621 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.027841+0000", 2024-09-18T04:38:55.621 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000108978 2024-09-18T04:38:55.621 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.621 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.621 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:55.621 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.027990+0000", 2024-09-18T04:38:55.621 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014852999999999999 2024-09-18T04:38:55.621 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.621 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.621 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:55.621 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.031930+0000", 2024-09-18T04:38:55.621 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0039406299999999997 2024-09-18T04:38:55.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.622 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.622 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:55.622 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.031978+0000", 2024-09-18T04:38:55.622 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7817e-05 2024-09-18T04:38:55.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.622 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.622 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:55.622 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.032019+0000", 2024-09-18T04:38:55.622 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0531999999999998e-05 2024-09-18T04:38:55.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.622 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.622 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:55.624 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.032067+0000", 2024-09-18T04:38:55.624 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8371000000000002e-05 2024-09-18T04:38:55.624 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.624 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:55.624 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.624 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.624 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4805.0:593 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [write 524288~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:38:55.625 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.023665+0000", 2024-09-18T04:38:55.625 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.5194082819999997, 2024-09-18T04:38:55.625 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0085518090000000005, 2024-09-18T04:38:55.625 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:55.625 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:55.625 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:55.625 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4805", 2024-09-18T04:38:55.625 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:38:55.625 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 593 2024-09-18T04:38:55.625 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.625 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:55.625 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.625 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:55.625 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023665+0000", 2024-09-18T04:38:55.626 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.626 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.626 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.626 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:55.626 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023665+0000", 2024-09-18T04:38:55.626 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.626 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.626 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.626 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:55.626 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023664+0000", 2024-09-18T04:38:55.626 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:55.626 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.626 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.627 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:55.627 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023875+0000", 2024-09-18T04:38:55.627 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000210862 2024-09-18T04:38:55.627 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.627 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.627 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:55.627 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023906+0000", 2024-09-18T04:38:55.627 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0660000000000001e-05 2024-09-18T04:38:55.627 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.627 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.627 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:55.627 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023922+0000", 2024-09-18T04:38:55.627 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5831000000000001e-05 2024-09-18T04:38:55.627 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.628 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.628 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:55.628 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.029603+0000", 2024-09-18T04:38:55.628 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0056816719999999996 2024-09-18T04:38:55.628 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.628 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.628 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:55.628 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.029766+0000", 2024-09-18T04:38:55.628 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000162903 2024-09-18T04:38:55.628 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.628 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.628 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:55.628 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.030042+0000", 2024-09-18T04:38:55.629 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00027561399999999998 2024-09-18T04:38:55.629 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.629 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.629 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:55.629 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.032127+0000", 2024-09-18T04:38:55.629 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020850920000000002 2024-09-18T04:38:55.629 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.629 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.629 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:55.630 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.032137+0000", 2024-09-18T04:38:55.630 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0312000000000001e-05 2024-09-18T04:38:55.630 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.630 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.630 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:55.630 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.032183+0000", 2024-09-18T04:38:55.630 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6020999999999999e-05 2024-09-18T04:38:55.630 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.630 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.630 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:55.630 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.032217+0000", 2024-09-18T04:38:55.630 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4109999999999997e-05 2024-09-18T04:38:55.630 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.630 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:55.631 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.631 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.631 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4805.0:594 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [write 655360~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:38:55.631 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.023941+0000", 2024-09-18T04:38:55.631 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.5191327440000002, 2024-09-18T04:38:55.631 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010767616000000001, 2024-09-18T04:38:55.631 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:55.631 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:55.631 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:55.631 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4805", 2024-09-18T04:38:55.631 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:38:55.631 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 594 2024-09-18T04:38:55.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.632 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:55.632 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.632 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:55.632 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023941+0000", 2024-09-18T04:38:55.632 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.632 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.632 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.632 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:55.632 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023941+0000", 2024-09-18T04:38:55.632 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.632 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.632 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.632 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:55.633 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023940+0000", 2024-09-18T04:38:55.633 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:55.633 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.633 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.633 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:55.633 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024163+0000", 2024-09-18T04:38:55.633 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000223017 2024-09-18T04:38:55.633 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.633 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.633 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:55.633 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024188+0000", 2024-09-18T04:38:55.633 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5174e-05 2024-09-18T04:38:55.633 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.633 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.634 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:55.634 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024214+0000", 2024-09-18T04:38:55.634 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5782000000000002e-05 2024-09-18T04:38:55.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.634 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.634 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:55.634 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.031116+0000", 2024-09-18T04:38:55.634 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.006902282 2024-09-18T04:38:55.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.634 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.634 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:55.634 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.031199+0000", 2024-09-18T04:38:55.634 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3203999999999998e-05 2024-09-18T04:38:55.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.634 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.635 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:55.635 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.031315+0000", 2024-09-18T04:38:55.635 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000115742 2024-09-18T04:38:55.635 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.635 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.635 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:55.635 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.033780+0000", 2024-09-18T04:38:55.635 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002465058 2024-09-18T04:38:55.635 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.635 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.635 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:55.635 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.034630+0000", 2024-09-18T04:38:55.635 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00084966800000000004 2024-09-18T04:38:55.636 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.636 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.636 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:55.636 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.034682+0000", 2024-09-18T04:38:55.636 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2562999999999998e-05 2024-09-18T04:38:55.636 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.636 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.636 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:55.636 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.034709+0000", 2024-09-18T04:38:55.636 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.62e-05 2024-09-18T04:38:55.636 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.636 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:55.636 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.636 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.637 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.637 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4805.0:595 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [write 786432~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:38:55.637 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.024233+0000", 2024-09-18T04:38:55.637 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.5188403690000003, 2024-09-18T04:38:55.637 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010599558, 2024-09-18T04:38:55.637 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:55.637 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:55.637 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:55.637 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4805", 2024-09-18T04:38:55.637 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:38:55.637 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 595 2024-09-18T04:38:55.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.637 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:55.638 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.638 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:55.638 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024233+0000", 2024-09-18T04:38:55.638 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.638 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.638 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:55.638 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024233+0000", 2024-09-18T04:38:55.638 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.638 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.638 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:55.638 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024232+0000", 2024-09-18T04:38:55.638 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:55.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.639 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.639 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:55.639 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024449+0000", 2024-09-18T04:38:55.639 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000216868 2024-09-18T04:38:55.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.639 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.639 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:55.639 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024474+0000", 2024-09-18T04:38:55.639 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4598999999999999e-05 2024-09-18T04:38:55.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.639 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.639 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:55.639 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024487+0000", 2024-09-18T04:38:55.640 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3451999999999999e-05 2024-09-18T04:38:55.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.640 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.640 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:55.640 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.032272+0000", 2024-09-18T04:38:55.640 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0077845709999999997 2024-09-18T04:38:55.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.640 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.640 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:55.640 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.032332+0000", 2024-09-18T04:38:55.640 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0736999999999998e-05 2024-09-18T04:38:55.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.640 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.641 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:55.641 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.032421+0000", 2024-09-18T04:38:55.641 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8993000000000007e-05 2024-09-18T04:38:55.641 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.641 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.641 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:55.641 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.033786+0000", 2024-09-18T04:38:55.641 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001364451 2024-09-18T04:38:55.641 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.641 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.641 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:55.641 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.034752+0000", 2024-09-18T04:38:55.641 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00096642400000000002 2024-09-18T04:38:55.642 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.642 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.642 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:55.642 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.034789+0000", 2024-09-18T04:38:55.642 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6899e-05 2024-09-18T04:38:55.642 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.642 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.642 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:55.642 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.034833+0000", 2024-09-18T04:38:55.642 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3846e-05 2024-09-18T04:38:55.642 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.642 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:55.642 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.642 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.643 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.643 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4805.0:596 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [write 917504~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:38:55.643 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.024500+0000", 2024-09-18T04:38:55.643 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.5185735840000003, 2024-09-18T04:38:55.643 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010770534, 2024-09-18T04:38:55.643 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:55.643 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:55.643 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:55.643 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4805", 2024-09-18T04:38:55.643 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:38:55.643 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 596 2024-09-18T04:38:55.643 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.643 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:55.643 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.643 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:55.644 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024500+0000", 2024-09-18T04:38:55.644 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.644 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.644 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.644 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:55.644 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024500+0000", 2024-09-18T04:38:55.644 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.644 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.644 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.644 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:55.644 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024499+0000", 2024-09-18T04:38:55.644 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:55.644 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.644 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.645 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:55.645 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024700+0000", 2024-09-18T04:38:55.645 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00020038599999999999 2024-09-18T04:38:55.645 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.645 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.645 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:55.645 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024729+0000", 2024-09-18T04:38:55.645 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9224999999999999e-05 2024-09-18T04:38:55.645 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.645 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.645 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:55.645 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024776+0000", 2024-09-18T04:38:55.645 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7385000000000001e-05 2024-09-18T04:38:55.645 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.646 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.646 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:55.646 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.032998+0000", 2024-09-18T04:38:55.646 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0082216189999999995 2024-09-18T04:38:55.646 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.646 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.646 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:55.646 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.033076+0000", 2024-09-18T04:38:55.646 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8668e-05 2024-09-18T04:38:55.646 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.646 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.646 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:55.646 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.033167+0000", 2024-09-18T04:38:55.646 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0728000000000003e-05 2024-09-18T04:38:55.647 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.647 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.647 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:38:55.647 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.034399+0000", 2024-09-18T04:38:55.647 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00123234 2024-09-18T04:38:55.647 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.647 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.647 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:55.647 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.035206+0000", 2024-09-18T04:38:55.647 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00080701799999999995 2024-09-18T04:38:55.647 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.647 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.647 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:55.647 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.035244+0000", 2024-09-18T04:38:55.648 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7487999999999998e-05 2024-09-18T04:38:55.648 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.648 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.648 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:55.648 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.035271+0000", 2024-09-18T04:38:55.648 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6709000000000001e-05 2024-09-18T04:38:55.648 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.648 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:55.648 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.648 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.648 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.648 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4805.0:597 2.1 2:9370a598:::rbd_header.125fc09df6d1:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:38:55.648 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.025206+0000", 2024-09-18T04:38:55.649 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.5178674829999999, 2024-09-18T04:38:55.649 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0091536900000000008, 2024-09-18T04:38:55.649 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:55.649 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:38:55.649 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:55.649 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4805", 2024-09-18T04:38:55.649 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:38:55.649 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 597 2024-09-18T04:38:55.649 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.649 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:55.649 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.649 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:55.649 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.025206+0000", 2024-09-18T04:38:55.650 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.650 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.650 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:55.650 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.025206+0000", 2024-09-18T04:38:55.650 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.650 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.650 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:55.650 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.025204+0000", 2024-09-18T04:38:55.650 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:55.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.650 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.650 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:55.651 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.025216+0000", 2024-09-18T04:38:55.651 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2150999999999999e-05 2024-09-18T04:38:55.651 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.651 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.651 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:55.651 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.025237+0000", 2024-09-18T04:38:55.651 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0740000000000001e-05 2024-09-18T04:38:55.651 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.651 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.651 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:55.651 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.025254+0000", 2024-09-18T04:38:55.651 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6731e-05 2024-09-18T04:38:55.651 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.651 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.652 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:55.652 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.034049+0000", 2024-09-18T04:38:55.652 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0087952810000000003 2024-09-18T04:38:55.652 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.652 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.652 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:55.652 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.034202+0000", 2024-09-18T04:38:55.652 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0001526 2024-09-18T04:38:55.652 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.652 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.652 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:55.652 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.034360+0000", 2024-09-18T04:38:55.652 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015809299999999999 2024-09-18T04:38:55.652 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.653 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:55.653 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.653 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.653 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.653 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4972.0:14 2.5 2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:head [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:38:55.654 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.949251+0000", 2024-09-18T04:38:55.654 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 3.5938225369999999, 2024-09-18T04:38:55.654 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0098324480000000006, 2024-09-18T04:38:55.654 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:55.654 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:38:55.654 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:55.654 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4972", 2024-09-18T04:38:55.654 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2280251472", 2024-09-18T04:38:55.654 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 14 2024-09-18T04:38:55.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.655 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:55.655 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.655 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:55.655 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949251+0000", 2024-09-18T04:38:55.655 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.655 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.655 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:55.655 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949251+0000", 2024-09-18T04:38:55.655 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.655 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.655 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:55.655 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949250+0000", 2024-09-18T04:38:55.656 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:55.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.656 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.656 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:55.656 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949258+0000", 2024-09-18T04:38:55.656 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0590000000000003e-06 2024-09-18T04:38:55.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.656 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.656 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:55.656 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949269+0000", 2024-09-18T04:38:55.656 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0607e-05 2024-09-18T04:38:55.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.656 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.656 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:55.657 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949277+0000", 2024-09-18T04:38:55.657 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8269999999999998e-06 2024-09-18T04:38:55.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.657 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.657 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:55.657 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949394+0000", 2024-09-18T04:38:55.657 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000117113 2024-09-18T04:38:55.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.657 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.657 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:55.657 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949527+0000", 2024-09-18T04:38:55.657 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013288300000000001 2024-09-18T04:38:55.658 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.658 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.658 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:55.658 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.959084+0000", 2024-09-18T04:38:55.658 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0095569550000000007 2024-09-18T04:38:55.658 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.658 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:55.658 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.658 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.658 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.658 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4972.0:20 2.5 2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:38:55.658 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.949296+0000", 2024-09-18T04:38:55.658 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 3.5937782309999999, 2024-09-18T04:38:55.658 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014709732, 2024-09-18T04:38:55.659 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:55.659 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:38:55.659 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:38:55.659 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4972", 2024-09-18T04:38:55.659 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2280251472", 2024-09-18T04:38:55.659 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 20 2024-09-18T04:38:55.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.659 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:55.659 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.659 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:55.659 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949296+0000", 2024-09-18T04:38:55.659 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.659 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.660 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:55.660 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949296+0000", 2024-09-18T04:38:55.660 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:55.660 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.660 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.660 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:55.660 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949295+0000", 2024-09-18T04:38:55.660 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:55.660 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.660 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.660 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:55.660 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949301+0000", 2024-09-18T04:38:55.660 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4829999999999996e-06 2024-09-18T04:38:55.661 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.661 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.661 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:55.661 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949317+0000", 2024-09-18T04:38:55.661 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5866999999999999e-05 2024-09-18T04:38:55.661 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.661 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.661 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:55.661 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949331+0000", 2024-09-18T04:38:55.661 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3485999999999999e-05 2024-09-18T04:38:55.661 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.661 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.661 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:55.662 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.959137+0000", 2024-09-18T04:38:55.662 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0098066520000000008 2024-09-18T04:38:55.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.662 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.662 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:55.662 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.959251+0000", 2024-09-18T04:38:55.662 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000113783 2024-09-18T04:38:55.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:55.662 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:55.662 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:55.662 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.964005+0000", 2024-09-18T04:38:55.662 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0047542080000000002 2024-09-18T04:38:55.662 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.662 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:55.663 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.663 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:55.663 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:55.663 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:55.665 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:38:55.665 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false grep '^ *17851\b' 2024-09-18T04:38:55.665 INFO:tasks.workunit.client.0.smithi115.stderr:+ ps -C rbd-nbd 2024-09-18T04:38:55.665 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep '^ *17851\b' 2024-09-18T04:38:55.665 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:38:55.665 INFO:tasks.workunit.client.0.smithi115.stderr:+ sleep 0.5 2024-09-18T04:38:55.665 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:38:55.666 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:55.661+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:55.765 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:55.758+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:55.783 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:55.783 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:55.783 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:38:55.783 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:55.793 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:38:55.866 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:55.861+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:55.898 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:55.898 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:55.898 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:38:55.898 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:38:55.898 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:55.907 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:38:55.967 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:55.962+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:56.017 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:56.017 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:38:56.017 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:38:56.017 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:38:56.017 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.017 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:56.017 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:38:56.017 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 88.224432246999996, 2024-09-18T04:38:56.017 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017399218000000001, 2024-09-18T04:38:56.017 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:56.017 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:56.017 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:56.017 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:56.018 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:56.018 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 114 2024-09-18T04:38:56.018 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.018 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:56.018 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.018 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:56.018 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:38:56.018 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.018 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.018 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.018 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:56.018 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:38:56.018 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.018 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.019 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.019 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:56.019 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786900+0000", 2024-09-18T04:38:56.019 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:56.019 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.019 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.019 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:56.019 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786908+0000", 2024-09-18T04:38:56.019 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8199999999999997e-06 2024-09-18T04:38:56.019 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.019 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.019 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:56.019 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786925+0000", 2024-09-18T04:38:56.019 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6849000000000001e-05 2024-09-18T04:38:56.020 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.020 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.020 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:56.020 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786934+0000", 2024-09-18T04:38:56.020 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6299999999999993e-06 2024-09-18T04:38:56.020 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.020 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.020 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:56.020 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800926+0000", 2024-09-18T04:38:56.020 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013991632 2024-09-18T04:38:56.020 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.020 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.020 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:56.020 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801060+0000", 2024-09-18T04:38:56.021 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013378199999999999 2024-09-18T04:38:56.021 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.021 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.021 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:56.021 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801166+0000", 2024-09-18T04:38:56.021 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010607099999999999 2024-09-18T04:38:56.021 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.021 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.021 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:56.021 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803026+0000", 2024-09-18T04:38:56.021 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001860469 2024-09-18T04:38:56.021 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.021 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.021 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:56.022 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804245+0000", 2024-09-18T04:38:56.022 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0012185900000000001 2024-09-18T04:38:56.022 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.022 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.022 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:56.022 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804278+0000", 2024-09-18T04:38:56.022 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3272000000000003e-05 2024-09-18T04:38:56.022 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.022 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.022 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:56.022 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804300+0000", 2024-09-18T04:38:56.022 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2266e-05 2024-09-18T04:38:56.022 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.022 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:56.022 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.023 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.023 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.023 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:56.023 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787566+0000", 2024-09-18T04:38:56.023 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 88.223767288000005, 2024-09-18T04:38:56.023 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017124120999999999, 2024-09-18T04:38:56.023 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:56.023 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:56.023 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:56.023 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:56.023 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:56.023 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 115 2024-09-18T04:38:56.023 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.023 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:56.023 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.023 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:56.024 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787566+0000", 2024-09-18T04:38:56.024 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.024 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.024 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.024 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:56.024 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787566+0000", 2024-09-18T04:38:56.024 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.024 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.024 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.024 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:56.024 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787564+0000", 2024-09-18T04:38:56.024 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:56.024 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.024 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.024 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:56.025 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787751+0000", 2024-09-18T04:38:56.025 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000186873 2024-09-18T04:38:56.025 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.025 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.025 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:56.025 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787772+0000", 2024-09-18T04:38:56.025 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.126e-05 2024-09-18T04:38:56.025 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.025 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.025 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:56.025 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787794+0000", 2024-09-18T04:38:56.025 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1418999999999999e-05 2024-09-18T04:38:56.025 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.026 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.026 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:56.026 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801659+0000", 2024-09-18T04:38:56.026 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013865125000000001 2024-09-18T04:38:56.026 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.026 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.026 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:56.026 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801750+0000", 2024-09-18T04:38:56.026 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.1427000000000004e-05 2024-09-18T04:38:56.026 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.026 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.026 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:56.026 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801894+0000", 2024-09-18T04:38:56.026 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000143514 2024-09-18T04:38:56.026 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.027 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.027 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:56.027 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804313+0000", 2024-09-18T04:38:56.027 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0024190879999999998 2024-09-18T04:38:56.027 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.027 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.027 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:56.027 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804644+0000", 2024-09-18T04:38:56.027 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00033100799999999999 2024-09-18T04:38:56.027 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.027 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.027 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:56.027 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804672+0000", 2024-09-18T04:38:56.027 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7988e-05 2024-09-18T04:38:56.027 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.028 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.028 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:56.028 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804690+0000", 2024-09-18T04:38:56.028 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8298999999999999e-05 2024-09-18T04:38:56.028 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.028 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:56.028 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.028 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.028 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.028 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:56.028 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:38:56.028 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 88.223519349, 2024-09-18T04:38:56.028 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017521662, 2024-09-18T04:38:56.028 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:56.028 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:56.028 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:56.029 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:56.029 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:56.029 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 116 2024-09-18T04:38:56.029 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.029 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:56.029 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.029 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:56.029 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:38:56.029 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.029 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.029 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.029 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:56.029 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:38:56.029 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.030 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.030 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.030 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:56.030 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787813+0000", 2024-09-18T04:38:56.030 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:56.030 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.030 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.030 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:56.030 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787824+0000", 2024-09-18T04:38:56.030 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1157e-05 2024-09-18T04:38:56.030 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.030 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.030 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:56.031 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787840+0000", 2024-09-18T04:38:56.031 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6269e-05 2024-09-18T04:38:56.031 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.031 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.031 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:56.031 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787853+0000", 2024-09-18T04:38:56.031 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3156000000000001e-05 2024-09-18T04:38:56.031 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.031 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.031 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:56.031 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803110+0000", 2024-09-18T04:38:56.031 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015256736 2024-09-18T04:38:56.031 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.031 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.031 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:56.032 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803197+0000", 2024-09-18T04:38:56.032 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7195999999999997e-05 2024-09-18T04:38:56.032 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.032 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.032 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:56.032 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803308+0000", 2024-09-18T04:38:56.032 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111005 2024-09-18T04:38:56.032 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.032 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.032 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:56.032 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804316+0000", 2024-09-18T04:38:56.032 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010073949999999999 2024-09-18T04:38:56.032 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.032 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.032 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:56.033 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805232+0000", 2024-09-18T04:38:56.033 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091636300000000001 2024-09-18T04:38:56.033 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.033 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.033 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:56.033 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805277+0000", 2024-09-18T04:38:56.033 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4864000000000003e-05 2024-09-18T04:38:56.033 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.033 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.033 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:56.033 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805336+0000", 2024-09-18T04:38:56.033 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8968000000000002e-05 2024-09-18T04:38:56.033 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.033 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:56.034 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.034 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.034 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.034 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:56.034 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:38:56.034 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 88.223467294000002, 2024-09-18T04:38:56.034 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017983453, 2024-09-18T04:38:56.034 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:56.034 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:56.034 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:56.034 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:56.034 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:56.034 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 117 2024-09-18T04:38:56.034 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.035 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:56.035 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.035 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:56.035 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:38:56.035 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.035 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.035 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.035 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:56.035 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:38:56.035 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.035 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.035 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.035 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:56.035 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787865+0000", 2024-09-18T04:38:56.035 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:56.036 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.036 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.036 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:56.036 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787889+0000", 2024-09-18T04:38:56.036 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4244999999999999e-05 2024-09-18T04:38:56.036 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.036 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.036 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:56.036 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787905+0000", 2024-09-18T04:38:56.036 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5739999999999998e-05 2024-09-18T04:38:56.036 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.036 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.036 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:56.036 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787918+0000", 2024-09-18T04:38:56.037 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.252e-05 2024-09-18T04:38:56.037 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.037 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.037 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:56.037 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803693+0000", 2024-09-18T04:38:56.037 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015774872999999998 2024-09-18T04:38:56.037 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.037 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.037 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:56.037 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803775+0000", 2024-09-18T04:38:56.037 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2761000000000003e-05 2024-09-18T04:38:56.037 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.037 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.038 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:56.038 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803887+0000", 2024-09-18T04:38:56.038 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111708 2024-09-18T04:38:56.038 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.038 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.038 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:56.038 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804823+0000", 2024-09-18T04:38:56.038 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00093625100000000003 2024-09-18T04:38:56.038 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.038 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.038 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:56.038 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805804+0000", 2024-09-18T04:38:56.038 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00098114699999999992 2024-09-18T04:38:56.038 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.038 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.039 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:56.039 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805831+0000", 2024-09-18T04:38:56.039 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6401999999999999e-05 2024-09-18T04:38:56.039 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.039 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.039 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:56.039 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805850+0000", 2024-09-18T04:38:56.039 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8862e-05 2024-09-18T04:38:56.039 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.039 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:56.039 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.039 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.039 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.039 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:56.040 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:38:56.040 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 87.780290163000004, 2024-09-18T04:38:56.040 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016528505999999998, 2024-09-18T04:38:56.040 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:56.040 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:56.040 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:56.040 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:56.040 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:56.040 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 138 2024-09-18T04:38:56.040 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.040 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:56.040 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.040 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:56.040 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:38:56.040 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.040 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.041 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.041 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:56.041 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:38:56.041 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.041 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.041 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.041 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:56.041 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231043+0000", 2024-09-18T04:38:56.041 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:56.041 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.041 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.041 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:56.041 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231047+0000", 2024-09-18T04:38:56.041 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5399999999999997e-06 2024-09-18T04:38:56.042 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.042 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.042 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:56.042 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231068+0000", 2024-09-18T04:38:56.042 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0769000000000001e-05 2024-09-18T04:38:56.042 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.042 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.042 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:56.042 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231074+0000", 2024-09-18T04:38:56.042 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6180000000000003e-06 2024-09-18T04:38:56.042 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.042 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.042 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:56.042 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242556+0000", 2024-09-18T04:38:56.043 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011482654 2024-09-18T04:38:56.043 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.043 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.043 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:56.043 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242666+0000", 2024-09-18T04:38:56.043 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011003200000000001 2024-09-18T04:38:56.043 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.043 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.043 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:56.043 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.242924+0000", 2024-09-18T04:38:56.043 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00025785899999999999 2024-09-18T04:38:56.043 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.043 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.043 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:56.044 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247263+0000", 2024-09-18T04:38:56.044 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0043387649999999996 2024-09-18T04:38:56.044 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.044 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.044 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:56.044 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247412+0000", 2024-09-18T04:38:56.044 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000149276 2024-09-18T04:38:56.044 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.044 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.044 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:56.044 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247522+0000", 2024-09-18T04:38:56.044 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000109788 2024-09-18T04:38:56.044 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.044 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.044 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:56.045 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247572+0000", 2024-09-18T04:38:56.045 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9808999999999997e-05 2024-09-18T04:38:56.045 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.045 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:56.045 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.045 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.045 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.045 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:56.045 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231083+0000", 2024-09-18T04:38:56.045 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 87.780250699999996, 2024-09-18T04:38:56.045 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0166731, 2024-09-18T04:38:56.045 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:56.045 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:56.045 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:56.046 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:56.046 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:56.046 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 139 2024-09-18T04:38:56.046 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.046 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:56.046 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.046 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:56.046 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231083+0000", 2024-09-18T04:38:56.046 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.046 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.046 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.046 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:56.046 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231083+0000", 2024-09-18T04:38:56.046 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.046 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.047 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.047 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:56.047 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231082+0000", 2024-09-18T04:38:56.047 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:56.047 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.047 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.047 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:56.047 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231090+0000", 2024-09-18T04:38:56.047 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1280000000000008e-06 2024-09-18T04:38:56.047 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.047 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.047 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:56.047 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231097+0000", 2024-09-18T04:38:56.047 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5119999999999997e-06 2024-09-18T04:38:56.048 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.048 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.048 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:56.048 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231106+0000", 2024-09-18T04:38:56.048 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9230000000000008e-06 2024-09-18T04:38:56.048 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.048 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.048 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:56.048 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.243977+0000", 2024-09-18T04:38:56.048 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012870754 2024-09-18T04:38:56.048 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.048 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.048 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:56.048 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244112+0000", 2024-09-18T04:38:56.049 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000135575 2024-09-18T04:38:56.049 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.049 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.049 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:56.049 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.244265+0000", 2024-09-18T04:38:56.049 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015279900000000001 2024-09-18T04:38:56.049 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.050 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.050 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:56.050 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247649+0000", 2024-09-18T04:38:56.050 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003383896 2024-09-18T04:38:56.050 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.050 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.050 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:56.050 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247656+0000", 2024-09-18T04:38:56.050 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8920000000000003e-06 2024-09-18T04:38:56.050 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.050 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.050 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:56.050 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247692+0000", 2024-09-18T04:38:56.050 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6470000000000001e-05 2024-09-18T04:38:56.050 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.051 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.051 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:56.051 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247756+0000", 2024-09-18T04:38:56.051 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3676999999999994e-05 2024-09-18T04:38:56.051 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.051 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:56.051 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.051 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.051 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.051 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:56.051 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231190+0000", 2024-09-18T04:38:56.051 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 87.780143773999995, 2024-09-18T04:38:56.051 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016793556000000001, 2024-09-18T04:38:56.051 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:56.051 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:56.052 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:56.052 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:56.052 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:56.052 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 140 2024-09-18T04:38:56.052 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.052 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:56.052 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.052 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:56.052 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231190+0000", 2024-09-18T04:38:56.052 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.052 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.052 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.052 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:56.052 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231190+0000", 2024-09-18T04:38:56.053 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.053 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.053 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.053 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:56.053 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231189+0000", 2024-09-18T04:38:56.053 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:56.053 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.053 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.053 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:56.053 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231273+0000", 2024-09-18T04:38:56.053 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4445999999999998e-05 2024-09-18T04:38:56.053 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.053 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.053 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:56.053 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231282+0000", 2024-09-18T04:38:56.054 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8859999999999993e-06 2024-09-18T04:38:56.054 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.054 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.054 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:56.054 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231288+0000", 2024-09-18T04:38:56.054 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9819999999999996e-06 2024-09-18T04:38:56.054 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.054 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.054 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:56.054 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.245119+0000", 2024-09-18T04:38:56.054 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.01383121 2024-09-18T04:38:56.054 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.054 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.054 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:56.055 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.245247+0000", 2024-09-18T04:38:56.055 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012763299999999999 2024-09-18T04:38:56.055 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.055 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.055 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:56.055 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.245401+0000", 2024-09-18T04:38:56.055 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015399200000000001 2024-09-18T04:38:56.055 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.055 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.055 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:56.055 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247766+0000", 2024-09-18T04:38:56.055 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0023649510000000001 2024-09-18T04:38:56.055 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.055 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.056 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:56.056 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247904+0000", 2024-09-18T04:38:56.056 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013836499999999999 2024-09-18T04:38:56.056 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.056 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.056 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:56.056 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247950+0000", 2024-09-18T04:38:56.056 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5201000000000003e-05 2024-09-18T04:38:56.056 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.056 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.056 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:56.056 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.247983+0000", 2024-09-18T04:38:56.056 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3686000000000002e-05 2024-09-18T04:38:56.056 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.057 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:56.057 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.057 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.057 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.057 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:38:56.057 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:38:56.057 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 87.780033031000002, 2024-09-18T04:38:56.057 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017344024, 2024-09-18T04:38:56.057 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:56.057 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:56.057 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:56.057 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:38:56.057 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:38:56.058 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 141 2024-09-18T04:38:56.058 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.058 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:56.058 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.058 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:56.058 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:38:56.058 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.058 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.058 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.058 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:56.058 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:38:56.058 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.058 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.058 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.059 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:56.059 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231300+0000", 2024-09-18T04:38:56.059 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:56.059 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.059 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.059 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:56.059 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231309+0000", 2024-09-18T04:38:56.059 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5419999999999994e-06 2024-09-18T04:38:56.059 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.059 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.059 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:56.059 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231323+0000", 2024-09-18T04:38:56.059 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3108000000000001e-05 2024-09-18T04:38:56.059 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.059 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.060 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:56.060 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.231340+0000", 2024-09-18T04:38:56.060 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.696e-05 2024-09-18T04:38:56.060 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.060 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.060 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:56.060 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.246143+0000", 2024-09-18T04:38:56.060 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014803320999999999 2024-09-18T04:38:56.060 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.060 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.060 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:56.060 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.246323+0000", 2024-09-18T04:38:56.060 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00018061600000000001 2024-09-18T04:38:56.061 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.061 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.061 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:38:56.061 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.246530+0000", 2024-09-18T04:38:56.061 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020625 2024-09-18T04:38:56.061 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.061 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.061 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:56.061 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248002+0000", 2024-09-18T04:38:56.061 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0014719379999999999 2024-09-18T04:38:56.061 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.061 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.061 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:56.062 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248582+0000", 2024-09-18T04:38:56.062 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00058020599999999999 2024-09-18T04:38:56.062 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.062 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.062 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:56.062 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248618+0000", 2024-09-18T04:38:56.062 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6559e-05 2024-09-18T04:38:56.062 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.062 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.062 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:56.062 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:28.248645+0000", 2024-09-18T04:38:56.062 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6080000000000001e-05 2024-09-18T04:38:56.062 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.062 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:56.063 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.063 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.063 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.063 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:36 2.7 2:fe1b02ad:::rbd_data.125fc09df6d1.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:56.063 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:38:56.063 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 13.745611183999999, 2024-09-18T04:38:56.063 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.040387804999999999, 2024-09-18T04:38:56.063 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:56.063 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:56.063 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:56.063 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:38:56.063 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:38:56.063 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 36 2024-09-18T04:38:56.063 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.064 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:56.064 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.064 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:56.064 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:38:56.064 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.064 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.064 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.064 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:56.064 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:38:56.064 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.064 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.064 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.064 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:56.064 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265720+0000", 2024-09-18T04:38:56.064 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:56.065 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.065 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.065 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:56.065 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272590+0000", 2024-09-18T04:38:56.065 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.006870508 2024-09-18T04:38:56.065 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.065 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.065 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:56.065 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272623+0000", 2024-09-18T04:38:56.065 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2716000000000003e-05 2024-09-18T04:38:56.065 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.065 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.065 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:56.065 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272637+0000", 2024-09-18T04:38:56.065 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4175999999999999e-05 2024-09-18T04:38:56.066 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.066 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.066 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:56.066 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272779+0000", 2024-09-18T04:38:56.066 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000141392 2024-09-18T04:38:56.066 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.066 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.066 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:56.066 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273031+0000", 2024-09-18T04:38:56.066 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00025249099999999998 2024-09-18T04:38:56.066 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.066 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.066 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:38:56.066 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273219+0000", 2024-09-18T04:38:56.066 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000187531 2024-09-18T04:38:56.066 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.067 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.067 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:56.067 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.302806+0000", 2024-09-18T04:38:56.067 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.029587553999999999 2024-09-18T04:38:56.067 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.067 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.067 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:56.067 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306004+0000", 2024-09-18T04:38:56.067 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0031982629999999998 2024-09-18T04:38:56.067 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.067 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.067 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:56.067 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306071+0000", 2024-09-18T04:38:56.067 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6829999999999995e-05 2024-09-18T04:38:56.067 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.067 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.068 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:56.068 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306110+0000", 2024-09-18T04:38:56.068 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8874000000000001e-05 2024-09-18T04:38:56.068 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.068 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:56.068 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.068 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.068 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.068 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:38 2.0 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:56.068 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:38:56.068 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 13.738657647, 2024-09-18T04:38:56.068 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.033754348000000003, 2024-09-18T04:38:56.068 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:56.068 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:56.068 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:56.069 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:38:56.069 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:38:56.069 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 38 2024-09-18T04:38:56.069 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.069 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:56.069 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.069 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:56.069 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:38:56.069 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.069 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.069 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.069 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:56.069 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:38:56.069 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.069 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.070 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.070 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:56.070 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272673+0000", 2024-09-18T04:38:56.070 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:56.070 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.070 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.070 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:56.070 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277166+0000", 2024-09-18T04:38:56.070 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044922249999999999 2024-09-18T04:38:56.070 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.070 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.070 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:56.070 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277189+0000", 2024-09-18T04:38:56.071 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3283000000000001e-05 2024-09-18T04:38:56.071 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.071 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.071 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:56.071 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277205+0000", 2024-09-18T04:38:56.071 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6305000000000001e-05 2024-09-18T04:38:56.071 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.071 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.071 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:56.071 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277290+0000", 2024-09-18T04:38:56.071 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5135000000000004e-05 2024-09-18T04:38:56.071 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.071 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.071 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:56.071 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277531+0000", 2024-09-18T04:38:56.071 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00024014699999999999 2024-09-18T04:38:56.072 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.072 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.072 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:56.072 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277793+0000", 2024-09-18T04:38:56.072 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00026224499999999998 2024-09-18T04:38:56.072 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.072 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.072 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:56.072 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303990+0000", 2024-09-18T04:38:56.072 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.026197295999999998 2024-09-18T04:38:56.072 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.072 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.072 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:56.072 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306340+0000", 2024-09-18T04:38:56.072 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.002349524 2024-09-18T04:38:56.073 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.073 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.073 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:56.073 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306394+0000", 2024-09-18T04:38:56.073 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3934999999999998e-05 2024-09-18T04:38:56.073 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.073 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.073 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:56.073 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306430+0000", 2024-09-18T04:38:56.073 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6690000000000003e-05 2024-09-18T04:38:56.073 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.073 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:56.073 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.073 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.074 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.074 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:39 2.7 2:e9c909c5:::rbd_data.125fc09df6d1.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:56.074 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:38:56.074 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 13.734098041999999, 2024-09-18T04:38:56.074 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.029110673, 2024-09-18T04:38:56.074 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:56.074 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:56.074 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:56.074 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:38:56.074 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:38:56.074 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 39 2024-09-18T04:38:56.074 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.074 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:56.074 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.075 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:56.075 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:38:56.075 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.075 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.075 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.075 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:56.075 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:38:56.075 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.075 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.075 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.075 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:56.075 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277234+0000", 2024-09-18T04:38:56.075 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:56.075 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.075 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.076 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:56.076 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281351+0000", 2024-09-18T04:38:56.076 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0041170920000000001 2024-09-18T04:38:56.076 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.076 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.076 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:56.076 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281375+0000", 2024-09-18T04:38:56.076 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.446e-05 2024-09-18T04:38:56.076 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.076 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.076 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:56.076 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281390+0000", 2024-09-18T04:38:56.076 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4309e-05 2024-09-18T04:38:56.076 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.076 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.077 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:56.077 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.282306+0000", 2024-09-18T04:38:56.077 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091662000000000004 2024-09-18T04:38:56.077 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.077 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.077 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:56.077 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.282613+0000", 2024-09-18T04:38:56.077 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00030663799999999997 2024-09-18T04:38:56.077 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.077 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.077 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:38:56.077 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283001+0000", 2024-09-18T04:38:56.077 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00038801599999999998 2024-09-18T04:38:56.077 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.078 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.078 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:56.078 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306239+0000", 2024-09-18T04:38:56.078 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023238466999999999 2024-09-18T04:38:56.078 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.078 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.078 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:56.078 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306259+0000", 2024-09-18T04:38:56.078 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9837999999999999e-05 2024-09-18T04:38:56.078 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.078 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.078 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:56.078 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306299+0000", 2024-09-18T04:38:56.078 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0194000000000002e-05 2024-09-18T04:38:56.078 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.079 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.079 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:56.079 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306346+0000", 2024-09-18T04:38:56.079 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6845000000000003e-05 2024-09-18T04:38:56.079 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.079 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:56.079 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.079 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.079 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.079 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:46 2.6 e51/17 2:7ba25245:::rbd_data.125fc09df6d1.0000000000000006:head v 51'2, mlcod=0'0)", 2024-09-18T04:38:56.079 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:38:56.079 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 13.698176910000001, 2024-09-18T04:38:56.079 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019843098, 2024-09-18T04:38:56.079 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:56.080 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:56.080 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:56.080 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.080 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:56.080 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:38:56.080 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.080 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.080 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.080 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:56.080 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:38:56.080 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.080 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.080 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.080 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:56.081 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313156+0000", 2024-09-18T04:38:56.081 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:56.081 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.081 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.081 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:56.081 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320144+0000", 2024-09-18T04:38:56.081 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.006988345 2024-09-18T04:38:56.081 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.081 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.081 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:56.081 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320175+0000", 2024-09-18T04:38:56.081 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.0861999999999997e-05 2024-09-18T04:38:56.081 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.081 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.082 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:56.082 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320188+0000", 2024-09-18T04:38:56.082 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2714999999999999e-05 2024-09-18T04:38:56.082 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.082 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.082 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:56.082 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320267+0000", 2024-09-18T04:38:56.082 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9586999999999995e-05 2024-09-18T04:38:56.082 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.082 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.082 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:56.082 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320311+0000", 2024-09-18T04:38:56.082 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3529000000000003e-05 2024-09-18T04:38:56.083 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.083 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.083 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:56.083 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.332949+0000", 2024-09-18T04:38:56.083 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012638267999999999 2024-09-18T04:38:56.083 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.083 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.083 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:56.083 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.333000+0000", 2024-09-18T04:38:56.083 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0736999999999999e-05 2024-09-18T04:38:56.083 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.083 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:56.083 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.084 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.084 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.084 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:57 2.0 2:0cdd070e:::rbd_data.125fc09df6d1.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:56.084 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:38:56.084 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 13.662319334999999, 2024-09-18T04:38:56.084 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.033329974999999998, 2024-09-18T04:38:56.084 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:56.084 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:56.084 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:56.084 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:38:56.084 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:38:56.084 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 57 2024-09-18T04:38:56.084 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.084 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:56.085 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.085 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:56.085 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:38:56.085 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.085 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.085 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.085 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:56.085 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:38:56.085 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.085 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.085 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.085 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:56.085 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349013+0000", 2024-09-18T04:38:56.085 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:56.085 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.086 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.086 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:56.086 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356138+0000", 2024-09-18T04:38:56.086 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071258010000000002 2024-09-18T04:38:56.086 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.086 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.086 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:56.086 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356168+0000", 2024-09-18T04:38:56.086 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9679000000000002e-05 2024-09-18T04:38:56.086 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.086 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.086 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:56.086 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356181+0000", 2024-09-18T04:38:56.086 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3164e-05 2024-09-18T04:38:56.087 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.087 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.087 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:56.087 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356254+0000", 2024-09-18T04:38:56.087 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3182999999999994e-05 2024-09-18T04:38:56.087 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.087 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.087 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:56.087 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356453+0000", 2024-09-18T04:38:56.087 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019876599999999999 2024-09-18T04:38:56.087 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.087 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.088 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:56.088 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356627+0000", 2024-09-18T04:38:56.088 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017340199999999999 2024-09-18T04:38:56.088 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.088 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.088 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:56.088 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.369200+0000", 2024-09-18T04:38:56.088 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012573302 2024-09-18T04:38:56.088 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.088 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.088 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:56.089 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382262+0000", 2024-09-18T04:38:56.089 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013062549 2024-09-18T04:38:56.089 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.089 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.089 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:56.089 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382310+0000", 2024-09-18T04:38:56.089 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7366e-05 2024-09-18T04:38:56.089 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.089 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.089 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:56.089 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382344+0000", 2024-09-18T04:38:56.089 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4319999999999997e-05 2024-09-18T04:38:56.089 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.090 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:56.090 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.090 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.090 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.090 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:59 2.0 2:19c52cde:::rbd_data.125fc09df6d1.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:56.090 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:38:56.090 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 13.655128169999999, 2024-09-18T04:38:56.090 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.027137247999999999, 2024-09-18T04:38:56.090 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:56.090 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:56.090 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:56.090 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:38:56.090 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:38:56.091 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 59 2024-09-18T04:38:56.091 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.091 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:56.091 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.091 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:56.091 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:38:56.091 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.091 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.091 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.091 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:56.091 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:38:56.091 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.091 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.091 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.091 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:56.091 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356204+0000", 2024-09-18T04:38:56.092 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:56.092 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.092 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.092 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:56.092 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362661+0000", 2024-09-18T04:38:56.092 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0064565129999999997 2024-09-18T04:38:56.092 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.092 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.092 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:56.092 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362684+0000", 2024-09-18T04:38:56.092 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3688000000000001e-05 2024-09-18T04:38:56.092 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.092 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.092 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:56.092 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362699+0000", 2024-09-18T04:38:56.093 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4798e-05 2024-09-18T04:38:56.093 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.093 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.093 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:56.093 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363016+0000", 2024-09-18T04:38:56.093 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031737300000000003 2024-09-18T04:38:56.093 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.093 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.093 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:56.093 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363201+0000", 2024-09-18T04:38:56.093 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000184131 2024-09-18T04:38:56.093 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.093 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.093 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:56.094 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363343+0000", 2024-09-18T04:38:56.094 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014236699999999999 2024-09-18T04:38:56.094 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.094 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.094 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:56.094 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.372189+0000", 2024-09-18T04:38:56.094 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0088464819999999993 2024-09-18T04:38:56.094 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.094 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.094 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:56.094 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383269+0000", 2024-09-18T04:38:56.094 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011079594 2024-09-18T04:38:56.094 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.094 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.094 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:56.094 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383314+0000", 2024-09-18T04:38:56.095 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4932000000000003e-05 2024-09-18T04:38:56.095 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.095 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.095 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:56.095 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383343+0000", 2024-09-18T04:38:56.095 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8600999999999999e-05 2024-09-18T04:38:56.095 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.095 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:56.095 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.095 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.095 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.095 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:40 2.6 e51/17 2:784d6cd1:ns::rbd_data.133e813ab5bc.0000000000000005:head v 51'7, mlcod=51'5)", 2024-09-18T04:38:56.095 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:38:56.095 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 13.458878435999999, 2024-09-18T04:38:56.096 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020497496, 2024-09-18T04:38:56.096 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:56.096 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:56.096 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:56.096 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.096 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:56.096 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:38:56.096 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.096 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.096 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.096 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:56.096 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:38:56.096 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.096 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.097 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.097 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:56.097 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552454+0000", 2024-09-18T04:38:56.097 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:56.097 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.097 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.097 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:56.097 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556448+0000", 2024-09-18T04:38:56.097 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0039940990000000001 2024-09-18T04:38:56.097 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.097 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.097 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:56.097 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556481+0000", 2024-09-18T04:38:56.097 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3778000000000001e-05 2024-09-18T04:38:56.098 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.098 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.098 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:56.098 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556493+0000", 2024-09-18T04:38:56.098 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1622e-05 2024-09-18T04:38:56.098 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.098 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.098 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:56.098 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556556+0000", 2024-09-18T04:38:56.098 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2582999999999994e-05 2024-09-18T04:38:56.098 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.098 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.098 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:56.098 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556592+0000", 2024-09-18T04:38:56.098 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5880999999999997e-05 2024-09-18T04:38:56.099 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.099 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.099 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:56.099 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.572786+0000", 2024-09-18T04:38:56.099 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016194571000000001 2024-09-18T04:38:56.099 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.099 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.099 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:56.099 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.572953+0000", 2024-09-18T04:38:56.099 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000166436 2024-09-18T04:38:56.099 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.099 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:56.099 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.099 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.099 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.099 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:45 2.0 2:0fbfa3dc:ns::rbd_data.133e813ab5bc.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:56.100 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:38:56.100 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 13.447664137, 2024-09-18T04:38:56.100 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.036558168000000002, 2024-09-18T04:38:56.100 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:56.100 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:56.100 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:56.100 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:38:56.100 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:56.100 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 45 2024-09-18T04:38:56.100 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.100 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:56.100 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.100 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:56.100 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:38:56.101 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.101 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.101 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.101 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:56.101 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:38:56.101 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.101 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.101 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.101 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:56.101 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:38:56.101 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:56.101 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.101 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.101 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:56.102 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578594+0000", 2024-09-18T04:38:56.102 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014927147999999999 2024-09-18T04:38:56.102 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.102 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.102 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:56.102 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578612+0000", 2024-09-18T04:38:56.102 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7623999999999999e-05 2024-09-18T04:38:56.102 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.102 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.102 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:56.102 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578620+0000", 2024-09-18T04:38:56.102 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9070000000000007e-06 2024-09-18T04:38:56.102 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.102 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.103 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:56.103 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578702+0000", 2024-09-18T04:38:56.103 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2349999999999996e-05 2024-09-18T04:38:56.103 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.103 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.103 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:56.103 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.579028+0000", 2024-09-18T04:38:56.103 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032622899999999998 2024-09-18T04:38:56.103 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.103 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.103 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:56.103 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.579343+0000", 2024-09-18T04:38:56.103 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031448899999999999 2024-09-18T04:38:56.103 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.104 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.104 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:56.104 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.590078+0000", 2024-09-18T04:38:56.104 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010734946 2024-09-18T04:38:56.104 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.104 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.104 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:56.104 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600089+0000", 2024-09-18T04:38:56.104 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010011195000000001 2024-09-18T04:38:56.104 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.104 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.104 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:56.104 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600177+0000", 2024-09-18T04:38:56.104 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8425999999999997e-05 2024-09-18T04:38:56.104 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.104 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.105 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:56.105 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600228+0000", 2024-09-18T04:38:56.105 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0216999999999997e-05 2024-09-18T04:38:56.105 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.105 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:56.105 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.105 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.105 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.105 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:48 2.6 e51/17 2:7c225365:ns::rbd_data.133e813ab5bc.0000000000000008:head v 51'9, mlcod=51'5)", 2024-09-18T04:38:56.105 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.570076+0000", 2024-09-18T04:38:56.105 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 13.441257035, 2024-09-18T04:38:56.105 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016939773000000002, 2024-09-18T04:38:56.105 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:56.105 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:56.105 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:56.106 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.106 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:56.106 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.570076+0000", 2024-09-18T04:38:56.106 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.106 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.106 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.106 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:56.106 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.570076+0000", 2024-09-18T04:38:56.106 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.106 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.106 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.106 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:56.106 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.570074+0000", 2024-09-18T04:38:56.106 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:56.106 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.107 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.107 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:56.107 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.574672+0000", 2024-09-18T04:38:56.107 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045979819999999996 2024-09-18T04:38:56.107 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.107 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.107 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:56.107 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.574695+0000", 2024-09-18T04:38:56.107 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2529999999999999e-05 2024-09-18T04:38:56.107 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.107 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.107 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:56.107 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.574706+0000", 2024-09-18T04:38:56.107 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1357e-05 2024-09-18T04:38:56.107 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.108 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.108 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:56.108 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.574746+0000", 2024-09-18T04:38:56.108 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9586999999999999e-05 2024-09-18T04:38:56.108 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.108 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.108 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:56.108 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.574764+0000", 2024-09-18T04:38:56.108 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8000999999999999e-05 2024-09-18T04:38:56.108 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.108 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.108 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:56.108 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.586847+0000", 2024-09-18T04:38:56.108 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.01208359 2024-09-18T04:38:56.108 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.109 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.109 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:56.109 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.587016+0000", 2024-09-18T04:38:56.109 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000168796 2024-09-18T04:38:56.109 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.109 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:56.109 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.109 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.109 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.109 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:54 2.0 2:04f17ef6:ns::rbd_data.133e813ab5bc.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:56.109 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:38:56.109 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 13.426092300000001, 2024-09-18T04:38:56.109 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.024612999, 2024-09-18T04:38:56.110 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:56.110 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:56.110 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:56.110 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:38:56.110 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:56.110 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 54 2024-09-18T04:38:56.110 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.110 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:56.110 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.110 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:56.110 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:38:56.110 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.110 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.110 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.110 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:56.111 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:38:56.111 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.111 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.111 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.111 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:56.111 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585239+0000", 2024-09-18T04:38:56.111 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:56.111 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.111 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.111 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:56.111 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593702+0000", 2024-09-18T04:38:56.111 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0084632990000000005 2024-09-18T04:38:56.111 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.111 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.111 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:56.112 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593719+0000", 2024-09-18T04:38:56.112 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.664e-05 2024-09-18T04:38:56.112 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.112 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.112 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:56.112 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593727+0000", 2024-09-18T04:38:56.112 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6839999999999996e-06 2024-09-18T04:38:56.112 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.112 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.112 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:56.112 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593767+0000", 2024-09-18T04:38:56.112 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0330999999999997e-05 2024-09-18T04:38:56.112 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.112 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.112 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:56.113 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593905+0000", 2024-09-18T04:38:56.113 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013754800000000001 2024-09-18T04:38:56.113 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.113 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.113 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:56.113 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.594013+0000", 2024-09-18T04:38:56.113 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000108805 2024-09-18T04:38:56.113 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.113 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.113 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:56.113 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604678+0000", 2024-09-18T04:38:56.113 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010664391 2024-09-18T04:38:56.113 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.113 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.113 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:56.114 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609662+0000", 2024-09-18T04:38:56.114 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0049846100000000004 2024-09-18T04:38:56.114 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.114 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.114 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:56.114 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609788+0000", 2024-09-18T04:38:56.114 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012518800000000001 2024-09-18T04:38:56.114 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.114 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.114 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:56.114 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609854+0000", 2024-09-18T04:38:56.114 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6556000000000004e-05 2024-09-18T04:38:56.114 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.114 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:56.114 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.115 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.115 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.115 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:58 2.7 2:fc7c6cd3:ns::rbd_data.133e813ab5bc.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:56.115 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:38:56.115 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 13.413249871, 2024-09-18T04:38:56.115 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.027690566999999999, 2024-09-18T04:38:56.115 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:56.115 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:56.115 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:56.115 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:38:56.115 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:56.115 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 58 2024-09-18T04:38:56.115 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.115 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:56.115 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.116 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:56.116 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:38:56.116 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.116 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.116 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.116 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:56.116 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:38:56.116 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.116 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.116 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.116 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:56.116 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598082+0000", 2024-09-18T04:38:56.116 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:56.116 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.117 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.117 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:56.117 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604832+0000", 2024-09-18T04:38:56.117 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0067504510000000002 2024-09-18T04:38:56.117 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.117 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.117 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:56.117 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604852+0000", 2024-09-18T04:38:56.117 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0228999999999999e-05 2024-09-18T04:38:56.117 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.117 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.117 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:56.117 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604870+0000", 2024-09-18T04:38:56.117 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7952e-05 2024-09-18T04:38:56.118 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.118 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.118 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:56.118 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604936+0000", 2024-09-18T04:38:56.118 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5431000000000004e-05 2024-09-18T04:38:56.118 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.118 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.118 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:56.118 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605085+0000", 2024-09-18T04:38:56.118 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014910300000000001 2024-09-18T04:38:56.118 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.118 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.118 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:38:56.118 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605205+0000", 2024-09-18T04:38:56.118 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000119887 2024-09-18T04:38:56.119 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.119 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.119 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:56.119 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625586+0000", 2024-09-18T04:38:56.119 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020381399000000001 2024-09-18T04:38:56.119 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.119 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.119 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:56.119 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625628+0000", 2024-09-18T04:38:56.119 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1873999999999999e-05 2024-09-18T04:38:56.119 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.119 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.119 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:56.119 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625722+0000", 2024-09-18T04:38:56.119 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3787000000000002e-05 2024-09-18T04:38:56.120 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.120 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.120 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:56.120 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625774+0000", 2024-09-18T04:38:56.120 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2503999999999997e-05 2024-09-18T04:38:56.120 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.120 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:56.120 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.120 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.120 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.120 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:56 2.6 e51/17 2:72453c3e:ns::rbd_data.133e813ab5bc.000000000000000d:head v 51'10, mlcod=51'8)", 2024-09-18T04:38:56.120 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:38:56.120 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 13.406052763, 2024-09-18T04:38:56.120 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019766552999999999, 2024-09-18T04:38:56.120 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:56.121 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:56.121 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:56.121 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.121 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:56.121 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:38:56.121 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.121 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.121 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.121 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:56.121 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:38:56.121 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.121 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.121 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.121 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:56.121 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605279+0000", 2024-09-18T04:38:56.121 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:56.122 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.122 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.122 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:56.122 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609417+0000", 2024-09-18T04:38:56.122 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0041378059999999999 2024-09-18T04:38:56.122 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.122 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.122 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:56.122 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609443+0000", 2024-09-18T04:38:56.122 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6098e-05 2024-09-18T04:38:56.122 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.122 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.122 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:56.122 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609458+0000", 2024-09-18T04:38:56.122 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4671000000000001e-05 2024-09-18T04:38:56.123 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.123 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.123 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:56.123 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609553+0000", 2024-09-18T04:38:56.123 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5265000000000004e-05 2024-09-18T04:38:56.123 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.123 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.123 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:56.123 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609583+0000", 2024-09-18T04:38:56.123 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9839e-05 2024-09-18T04:38:56.123 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.123 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.123 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:56.123 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625004+0000", 2024-09-18T04:38:56.123 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015420639999999999 2024-09-18T04:38:56.123 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.124 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.124 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:56.124 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625047+0000", 2024-09-18T04:38:56.124 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3665999999999998e-05 2024-09-18T04:38:56.124 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.124 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:56.124 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.124 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.124 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:56.124 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:56.124 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-18T04:38:56.126 INFO:tasks.workunit.client.0.smithi115.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd17342 2024-09-18T04:38:56.126 INFO:tasks.workunit.client.0.smithi115.stderr:++ local cmd 2024-09-18T04:38:56.126 INFO:tasks.workunit.client.0.smithi115.stderr:+++ id -u 2024-09-18T04:38:56.127 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:38:56.127 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:56.122+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:56.128 INFO:tasks.workunit.client.0.smithi115.stderr:++ '[' 1000 -eq 0 ']' 2024-09-18T04:38:56.129 INFO:tasks.workunit.client.0.smithi115.stderr:+++ which rbd 2024-09-18T04:38:56.131 INFO:tasks.workunit.client.0.smithi115.stderr:++ cmd=/usr/bin/rbd 2024-09-18T04:38:56.131 INFO:tasks.workunit.client.0.smithi115.stderr:++ shift 2024-09-18T04:38:56.131 INFO:tasks.workunit.client.0.smithi115.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd17342 2024-09-18T04:38:56.227 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:56.222+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:56.237 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:56.240 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:56.240 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:38:56.240 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:56.248 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:38:56.291 INFO:tasks.workunit.client.0.smithi115.stderr:+ DEV=/dev/nbd0 2024-09-18T04:38:56.291 INFO:tasks.workunit.client.0.smithi115.stderr:+ get_pid rbd 2024-09-18T04:38:56.291 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pool=rbd 2024-09-18T04:38:56.291 INFO:tasks.workunit.client.0.smithi115.stderr:+ local ns= 2024-09-18T04:38:56.292 INFO:tasks.workunit.client.0.smithi115.stderr:++ rbd device --device-type nbd --format xml list 2024-09-18T04:38:56.292 INFO:tasks.workunit.client.0.smithi115.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd17342'\''][device='\''/dev/nbd0'\'']/id' 2024-09-18T04:38:56.327 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:56.321+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:56.352 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:56.352 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:56.352 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:38:56.352 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:38:56.352 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:56.361 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:38:56.378 INFO:tasks.workunit.client.0.smithi115.stderr:+ PID=18009 2024-09-18T04:38:56.378 INFO:tasks.workunit.client.0.smithi115.stderr:+ test -n 18009 2024-09-18T04:38:56.378 INFO:tasks.workunit.client.0.smithi115.stderr:+ ps -p 18009 -C rbd-nbd 2024-09-18T04:38:56.384 INFO:tasks.workunit.client.0.smithi115.stdout: PID TTY TIME CMD 2024-09-18T04:38:56.384 INFO:tasks.workunit.client.0.smithi115.stdout: 18009 ? 00:00:00 rbd-nbd 2024-09-18T04:38:56.386 INFO:tasks.workunit.client.0.smithi115.stderr:++ rbd -p rbd --format xml du testrbdnbd17342 2024-09-18T04:38:56.386 INFO:tasks.workunit.client.0.smithi115.stderr:++ xmlstarlet sel -t -m //stats/images/image/provisioned_size -v . 2024-09-18T04:38:56.428 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:56.421+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:56.463 INFO:tasks.workunit.client.0.smithi115.stderr:+ provisioned=67108864 2024-09-18T04:38:56.464 INFO:tasks.workunit.client.0.smithi115.stderr:++ rbd -p rbd --format xml du testrbdnbd17342 2024-09-18T04:38:56.464 INFO:tasks.workunit.client.0.smithi115.stderr:++ xmlstarlet sel -t -m //stats/images/image/used_size -v . 2024-09-18T04:38:56.467 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:56.467 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:38:56.467 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:38:56.467 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [] 2024-09-18T04:38:56.467 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:56.477 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:38:56.528 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:56.521+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:56.549 INFO:tasks.workunit.client.0.smithi115.stderr:+ used=67108864 2024-09-18T04:38:56.549 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 67108864 -eq 67108864 ']' 2024-09-18T04:38:56.549 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo blkdiscard /dev/nbd0 2024-09-18T04:38:56.549 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:38:56.549 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:38:56.551 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:38:56.551 INFO:tasks.workunit.client.0.smithi115.stderr:++ which blkdiscard 2024-09-18T04:38:56.554 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/sbin/blkdiscard 2024-09-18T04:38:56.554 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:38:56.554 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/sbin/blkdiscard /dev/nbd0 2024-09-18T04:38:56.628 INFO:tasks.workunit.client.0.smithi115.stderr:+ sync 2024-09-18T04:38:56.629 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:56.621+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:56.635 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:56.635 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:56.635 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:38:56.635 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:56.645 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:38:56.729 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:56.725+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:56.746 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:56.746 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:38:56.746 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:38:56.746 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:38:56.746 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:56.756 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:38:56.763 INFO:tasks.workunit.client.0.smithi115.stderr:++ rbd -p rbd --format xml du testrbdnbd17342 2024-09-18T04:38:56.763 INFO:tasks.workunit.client.0.smithi115.stderr:++ xmlstarlet sel -t -m //stats/images/image/provisioned_size -v . 2024-09-18T04:38:56.830 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:56.825+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:56.842 INFO:tasks.workunit.client.0.smithi115.stderr:+ provisioned=67108864 2024-09-18T04:38:56.844 INFO:tasks.workunit.client.0.smithi115.stderr:++ rbd -p rbd --format xml du testrbdnbd17342 2024-09-18T04:38:56.844 INFO:tasks.workunit.client.0.smithi115.stderr:++ xmlstarlet sel -t -m //stats/images/image/used_size -v . 2024-09-18T04:38:56.869 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:38:56.869 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:38:56.869 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:38:56.869 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:38:56.870 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.870 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-18T04:38:56.870 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:38:56.870 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 82.393796038999994, 2024-09-18T04:38:56.870 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0095781189999999995, 2024-09-18T04:38:56.870 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:56.870 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:56.870 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:56.870 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:38:56.870 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:38:56.871 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 1 2024-09-18T04:38:56.871 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.871 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:56.871 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.871 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:56.871 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:38:56.871 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.871 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.871 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.871 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:56.871 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:38:56.871 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.871 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.872 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.872 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:56.872 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469862+0000", 2024-09-18T04:38:56.872 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:56.872 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.872 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.872 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:56.872 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469879+0000", 2024-09-18T04:38:56.872 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7431999999999998e-05 2024-09-18T04:38:56.872 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.872 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.872 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:56.872 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469896+0000", 2024-09-18T04:38:56.872 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6742999999999999e-05 2024-09-18T04:38:56.873 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.873 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.873 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:56.873 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469915+0000", 2024-09-18T04:38:56.873 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8991000000000002e-05 2024-09-18T04:38:56.873 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.873 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.873 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:56.873 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470397+0000", 2024-09-18T04:38:56.873 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048167799999999998 2024-09-18T04:38:56.873 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.873 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.873 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-09-18T04:38:56.873 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470412+0000", 2024-09-18T04:38:56.874 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5526e-05 2024-09-18T04:38:56.874 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.874 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.874 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:56.874 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473279+0000", 2024-09-18T04:38:56.874 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0028667469999999998 2024-09-18T04:38:56.874 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.874 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.874 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:56.874 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473441+0000", 2024-09-18T04:38:56.874 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000162082 2024-09-18T04:38:56.874 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.874 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.874 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:38:56.875 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473573+0000", 2024-09-18T04:38:56.875 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000132032 2024-09-18T04:38:56.875 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.875 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.875 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:56.875 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474826+0000", 2024-09-18T04:38:56.875 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001252717 2024-09-18T04:38:56.875 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.875 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.875 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:56.875 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479307+0000", 2024-09-18T04:38:56.875 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044813220000000003 2024-09-18T04:38:56.875 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.876 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.876 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:56.876 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479366+0000", 2024-09-18T04:38:56.876 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9024999999999998e-05 2024-09-18T04:38:56.876 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.876 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.876 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:56.876 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479443+0000", 2024-09-18T04:38:56.876 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6339000000000002e-05 2024-09-18T04:38:56.876 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.876 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:56.876 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.876 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.876 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.877 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-18T04:38:56.877 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:38:56.877 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 80.380787619000003, 2024-09-18T04:38:56.877 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082699560000000002, 2024-09-18T04:38:56.877 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:56.877 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:56.877 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:56.877 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:38:56.877 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:38:56.877 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 7 2024-09-18T04:38:56.877 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.877 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:56.877 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.878 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:56.878 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:38:56.878 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.878 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.878 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.878 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:56.878 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:38:56.878 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.878 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.878 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.878 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:56.878 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482870+0000", 2024-09-18T04:38:56.878 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999967 2024-09-18T04:38:56.878 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.879 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.879 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:56.879 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482889+0000", 2024-09-18T04:38:56.879 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.965e-05 2024-09-18T04:38:56.879 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.879 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.879 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:56.879 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482938+0000", 2024-09-18T04:38:56.879 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8548999999999999e-05 2024-09-18T04:38:56.879 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.879 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.879 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:56.879 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482963+0000", 2024-09-18T04:38:56.879 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5075999999999998e-05 2024-09-18T04:38:56.880 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.880 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.880 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:56.880 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483121+0000", 2024-09-18T04:38:56.880 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015819400000000001 2024-09-18T04:38:56.880 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.880 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.880 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op must wait for map", 2024-09-18T04:38:56.880 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483138+0000", 2024-09-18T04:38:56.880 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7068999999999999e-05 2024-09-18T04:38:56.880 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.880 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.880 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:56.880 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488666+0000", 2024-09-18T04:38:56.881 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0055282049999999996 2024-09-18T04:38:56.881 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.881 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.881 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:56.881 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488783+0000", 2024-09-18T04:38:56.881 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000116737 2024-09-18T04:38:56.881 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.881 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.881 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:38:56.881 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488991+0000", 2024-09-18T04:38:56.881 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020781400000000001 2024-09-18T04:38:56.881 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.881 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.881 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:56.882 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490554+0000", 2024-09-18T04:38:56.882 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0015628370000000001 2024-09-18T04:38:56.882 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.882 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.882 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:56.882 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491019+0000", 2024-09-18T04:38:56.882 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000465608 2024-09-18T04:38:56.882 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.882 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.882 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:56.882 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491087+0000", 2024-09-18T04:38:56.882 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7988e-05 2024-09-18T04:38:56.883 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.883 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.883 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:56.883 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491143+0000", 2024-09-18T04:38:56.883 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5538999999999998e-05 2024-09-18T04:38:56.883 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.883 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:56.883 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.883 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.883 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.883 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:34 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:56.883 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.261758+0000", 2024-09-18T04:38:56.883 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 14.601902468, 2024-09-18T04:38:56.884 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.042024191000000002, 2024-09-18T04:38:56.884 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:56.884 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:56.884 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:56.884 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:38:56.884 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:38:56.884 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 34 2024-09-18T04:38:56.884 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.884 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:56.884 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.884 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:56.885 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.261758+0000", 2024-09-18T04:38:56.885 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.885 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.885 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.885 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:56.885 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.261758+0000", 2024-09-18T04:38:56.885 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.885 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.885 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.885 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:56.885 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.261756+0000", 2024-09-18T04:38:56.885 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:56.885 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.886 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.886 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:56.886 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.270785+0000", 2024-09-18T04:38:56.886 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0090292340000000006 2024-09-18T04:38:56.886 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.886 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.886 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:56.886 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.270890+0000", 2024-09-18T04:38:56.887 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000104541 2024-09-18T04:38:56.887 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.887 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.887 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:56.887 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.270991+0000", 2024-09-18T04:38:56.887 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000100778 2024-09-18T04:38:56.887 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.887 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.887 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:56.887 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.271157+0000", 2024-09-18T04:38:56.887 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016653999999999999 2024-09-18T04:38:56.887 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.888 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.888 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:56.888 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.271591+0000", 2024-09-18T04:38:56.888 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00043409500000000001 2024-09-18T04:38:56.888 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.888 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.888 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:38:56.888 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272017+0000", 2024-09-18T04:38:56.888 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000425769 2024-09-18T04:38:56.888 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.888 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.888 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:56.888 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.294246+0000", 2024-09-18T04:38:56.888 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.022229237999999998 2024-09-18T04:38:56.889 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.889 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.889 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:56.889 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303571+0000", 2024-09-18T04:38:56.889 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0093250239999999995 2024-09-18T04:38:56.889 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.889 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.889 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:56.889 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303705+0000", 2024-09-18T04:38:56.889 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000133831 2024-09-18T04:38:56.889 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.889 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.889 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:56.890 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303782+0000", 2024-09-18T04:38:56.890 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7156000000000005e-05 2024-09-18T04:38:56.890 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.890 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:56.890 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.890 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.890 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.890 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:36 2.7 e51/32 2:fe1b02ad:::rbd_data.125fc09df6d1.0000000000000002:head v 51'1, mlcod=0'0)", 2024-09-18T04:38:56.890 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.273441+0000", 2024-09-18T04:38:56.890 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 14.590219487000001, 2024-09-18T04:38:56.891 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.032400768000000003, 2024-09-18T04:38:56.891 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:56.891 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:56.891 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:56.891 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.891 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:56.891 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273441+0000", 2024-09-18T04:38:56.891 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.891 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.891 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.891 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:56.891 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273441+0000", 2024-09-18T04:38:56.891 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.892 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.892 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.892 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:56.892 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273440+0000", 2024-09-18T04:38:56.892 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:56.892 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.892 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.892 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:56.892 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278723+0000", 2024-09-18T04:38:56.892 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0052829909999999999 2024-09-18T04:38:56.892 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.892 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.892 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:56.893 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278752+0000", 2024-09-18T04:38:56.893 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9178000000000001e-05 2024-09-18T04:38:56.893 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.893 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.893 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:56.893 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278761+0000", 2024-09-18T04:38:56.893 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7900000000000005e-06 2024-09-18T04:38:56.893 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.893 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.893 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:56.893 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278864+0000", 2024-09-18T04:38:56.893 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000102699 2024-09-18T04:38:56.894 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.894 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.894 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:56.894 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278900+0000", 2024-09-18T04:38:56.894 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6001999999999999e-05 2024-09-18T04:38:56.894 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.894 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.894 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:56.894 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.305793+0000", 2024-09-18T04:38:56.894 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.026893344 2024-09-18T04:38:56.894 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.894 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.895 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:56.895 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.305842+0000", 2024-09-18T04:38:56.895 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8720000000000001e-05 2024-09-18T04:38:56.895 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.895 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:56.895 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.895 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.895 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.895 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:39 2.7 e51/32 2:e9c909c5:::rbd_data.125fc09df6d1.0000000000000005:head v 51'2, mlcod=0'0)", 2024-09-18T04:38:56.895 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.283175+0000", 2024-09-18T04:38:56.895 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 14.580485798, 2024-09-18T04:38:56.895 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.022864102000000001, 2024-09-18T04:38:56.895 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:56.896 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:56.896 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:56.896 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.896 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:56.896 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283175+0000", 2024-09-18T04:38:56.896 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.896 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.896 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.896 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:56.896 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283175+0000", 2024-09-18T04:38:56.896 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.896 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.896 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.897 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:56.897 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283173+0000", 2024-09-18T04:38:56.897 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:56.897 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.897 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.897 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:56.897 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.285951+0000", 2024-09-18T04:38:56.897 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0027784540000000001 2024-09-18T04:38:56.897 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.897 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.897 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:56.897 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.285970+0000", 2024-09-18T04:38:56.898 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8855000000000002e-05 2024-09-18T04:38:56.898 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.898 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.898 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:56.898 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.285984+0000", 2024-09-18T04:38:56.898 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4252e-05 2024-09-18T04:38:56.898 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.898 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.898 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:56.898 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.286041+0000", 2024-09-18T04:38:56.898 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6431999999999998e-05 2024-09-18T04:38:56.898 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.899 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.899 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:56.899 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.286086+0000", 2024-09-18T04:38:56.899 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5423000000000002e-05 2024-09-18T04:38:56.899 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.899 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.899 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:56.899 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.305998+0000", 2024-09-18T04:38:56.899 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019911697999999999 2024-09-18T04:38:56.899 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.899 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.899 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:56.900 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306039+0000", 2024-09-18T04:38:56.900 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0973999999999998e-05 2024-09-18T04:38:56.900 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.900 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:56.900 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.900 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.900 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.900 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:50 2.3 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:56.901 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.312563+0000", 2024-09-18T04:38:56.901 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 14.551097771, 2024-09-18T04:38:56.901 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.032636001999999997, 2024-09-18T04:38:56.901 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:56.901 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:56.901 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:56.901 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:38:56.901 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:38:56.901 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 50 2024-09-18T04:38:56.901 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.901 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:56.901 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.902 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:56.902 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.312563+0000", 2024-09-18T04:38:56.902 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.902 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.902 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.902 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:56.902 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.312563+0000", 2024-09-18T04:38:56.902 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.902 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.902 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.902 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:56.902 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.312561+0000", 2024-09-18T04:38:56.902 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:56.902 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.903 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.903 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:56.903 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321709+0000", 2024-09-18T04:38:56.903 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0091486650000000003 2024-09-18T04:38:56.903 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.903 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.903 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:56.903 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321739+0000", 2024-09-18T04:38:56.903 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9666e-05 2024-09-18T04:38:56.903 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.903 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.903 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:56.903 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321751+0000", 2024-09-18T04:38:56.903 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2086000000000001e-05 2024-09-18T04:38:56.904 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.904 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.904 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:56.904 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321833+0000", 2024-09-18T04:38:56.904 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1842999999999996e-05 2024-09-18T04:38:56.904 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.904 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.904 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:56.904 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321981+0000", 2024-09-18T04:38:56.904 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000147441 2024-09-18T04:38:56.904 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.904 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.904 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:38:56.905 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.322109+0000", 2024-09-18T04:38:56.905 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012851200000000001 2024-09-18T04:38:56.905 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.905 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.905 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:56.905 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.332829+0000", 2024-09-18T04:38:56.905 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010720225999999999 2024-09-18T04:38:56.905 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.905 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.905 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:56.905 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.345039+0000", 2024-09-18T04:38:56.905 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012209831000000001 2024-09-18T04:38:56.906 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.906 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.906 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:56.906 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.345120+0000", 2024-09-18T04:38:56.906 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1145999999999999e-05 2024-09-18T04:38:56.906 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.906 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.906 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:56.906 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.345199+0000", 2024-09-18T04:38:56.906 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8546000000000003e-05 2024-09-18T04:38:56.906 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.906 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:56.906 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.906 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.907 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.907 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:30 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:56.907 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.516122+0000", 2024-09-18T04:38:56.907 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 14.347538609000001, 2024-09-18T04:38:56.907 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0091865030000000004, 2024-09-18T04:38:56.907 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:56.907 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:56.907 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:56.907 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:38:56.907 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:56.907 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 30 2024-09-18T04:38:56.907 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.907 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:56.907 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.908 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:56.908 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516122+0000", 2024-09-18T04:38:56.908 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.908 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.908 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.908 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:56.908 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516122+0000", 2024-09-18T04:38:56.908 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.908 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.908 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.908 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:56.908 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516120+0000", 2024-09-18T04:38:56.908 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:56.909 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.909 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.909 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:56.909 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516129+0000", 2024-09-18T04:38:56.909 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.85e-06 2024-09-18T04:38:56.909 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.909 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.909 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:56.909 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516155+0000", 2024-09-18T04:38:56.909 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5882000000000001e-05 2024-09-18T04:38:56.909 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.909 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.909 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:56.910 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516175+0000", 2024-09-18T04:38:56.910 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.022e-05 2024-09-18T04:38:56.910 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.910 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.910 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:56.910 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519780+0000", 2024-09-18T04:38:56.910 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003604187 2024-09-18T04:38:56.910 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.910 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.910 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:56.910 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519964+0000", 2024-09-18T04:38:56.910 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00018391999999999999 2024-09-18T04:38:56.911 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.911 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.911 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:38:56.911 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.520794+0000", 2024-09-18T04:38:56.911 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00083078100000000005 2024-09-18T04:38:56.911 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.911 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.911 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:56.911 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.524129+0000", 2024-09-18T04:38:56.911 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033342419999999998 2024-09-18T04:38:56.911 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.911 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.912 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:56.912 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525239+0000", 2024-09-18T04:38:56.912 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001110778 2024-09-18T04:38:56.912 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.912 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.912 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:56.912 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525277+0000", 2024-09-18T04:38:56.912 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7596999999999999e-05 2024-09-18T04:38:56.912 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.912 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.912 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:56.912 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525308+0000", 2024-09-18T04:38:56.913 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.1535999999999998e-05 2024-09-18T04:38:56.913 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.913 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:56.913 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.913 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.913 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.913 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:32 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:56.913 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.519217+0000", 2024-09-18T04:38:56.913 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 14.344443289000001, 2024-09-18T04:38:56.913 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082266370000000002, 2024-09-18T04:38:56.913 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:56.913 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:56.913 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:56.913 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:38:56.914 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:56.914 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 32 2024-09-18T04:38:56.914 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.914 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:56.914 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.914 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:56.914 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519217+0000", 2024-09-18T04:38:56.914 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.914 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.914 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.914 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:56.914 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519217+0000", 2024-09-18T04:38:56.914 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.915 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.915 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.915 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:56.915 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519215+0000", 2024-09-18T04:38:56.915 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:56.915 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.915 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.915 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:56.915 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519229+0000", 2024-09-18T04:38:56.915 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3329999999999999e-05 2024-09-18T04:38:56.915 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.915 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.915 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:56.915 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519268+0000", 2024-09-18T04:38:56.916 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9533000000000002e-05 2024-09-18T04:38:56.916 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.916 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.916 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:56.916 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519285+0000", 2024-09-18T04:38:56.916 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6446000000000001e-05 2024-09-18T04:38:56.916 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.916 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.916 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:56.916 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.522358+0000", 2024-09-18T04:38:56.916 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030734310000000002 2024-09-18T04:38:56.916 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.916 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.916 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:56.917 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.522551+0000", 2024-09-18T04:38:56.917 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019317600000000001 2024-09-18T04:38:56.917 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.917 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.917 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:38:56.917 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.523262+0000", 2024-09-18T04:38:56.917 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00071069999999999998 2024-09-18T04:38:56.917 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.917 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.917 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:56.917 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525318+0000", 2024-09-18T04:38:56.917 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0020561469999999999 2024-09-18T04:38:56.917 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.917 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.918 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:56.918 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.527376+0000", 2024-09-18T04:38:56.918 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0020584969999999998 2024-09-18T04:38:56.918 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.918 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.918 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:56.918 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.527416+0000", 2024-09-18T04:38:56.918 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9054000000000003e-05 2024-09-18T04:38:56.918 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.918 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.918 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:56.918 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.527444+0000", 2024-09-18T04:38:56.919 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8347000000000002e-05 2024-09-18T04:38:56.919 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.919 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:56.919 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.919 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.919 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.919 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:42 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:56.919 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.551160+0000", 2024-09-18T04:38:56.919 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 14.312500897, 2024-09-18T04:38:56.919 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010692757000000001, 2024-09-18T04:38:56.919 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:56.919 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:56.919 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:56.919 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:38:56.920 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:56.920 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 42 2024-09-18T04:38:56.920 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.920 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:56.920 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.920 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:56.920 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551160+0000", 2024-09-18T04:38:56.920 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.920 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.920 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.920 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:56.920 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551160+0000", 2024-09-18T04:38:56.920 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.921 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.921 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.921 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:56.921 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551159+0000", 2024-09-18T04:38:56.921 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:56.921 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.921 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.921 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:56.921 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551166+0000", 2024-09-18T04:38:56.921 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1029999999999996e-06 2024-09-18T04:38:56.921 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.921 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.921 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:56.921 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551177+0000", 2024-09-18T04:38:56.921 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1384e-05 2024-09-18T04:38:56.922 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.922 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.922 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:56.922 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551185+0000", 2024-09-18T04:38:56.922 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1990000000000006e-06 2024-09-18T04:38:56.922 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.922 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.922 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:56.922 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552021+0000", 2024-09-18T04:38:56.922 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00083574400000000003 2024-09-18T04:38:56.922 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.922 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.922 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:56.922 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552339+0000", 2024-09-18T04:38:56.923 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031771399999999997 2024-09-18T04:38:56.923 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.923 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.923 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:38:56.923 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.553140+0000", 2024-09-18T04:38:56.923 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00080088100000000003 2024-09-18T04:38:56.923 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.923 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.923 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:56.923 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.554585+0000", 2024-09-18T04:38:56.923 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001445801 2024-09-18T04:38:56.923 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.923 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.924 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:56.924 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.561711+0000", 2024-09-18T04:38:56.924 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071260109999999998 2024-09-18T04:38:56.924 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.924 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.924 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:56.924 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.561790+0000", 2024-09-18T04:38:56.924 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8967000000000005e-05 2024-09-18T04:38:56.924 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.924 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.924 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:56.924 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.561852+0000", 2024-09-18T04:38:56.924 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2045999999999995e-05 2024-09-18T04:38:56.925 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.925 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:56.925 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.925 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.925 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.925 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:52 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:38:56.925 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.577756+0000", 2024-09-18T04:38:56.925 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 14.285905012000001, 2024-09-18T04:38:56.925 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0065803340000000002, 2024-09-18T04:38:56.925 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:56.925 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:56.925 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:56.926 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:38:56.926 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:38:56.926 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 52 2024-09-18T04:38:56.926 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.926 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:56.926 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.926 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:56.926 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.577756+0000", 2024-09-18T04:38:56.926 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.926 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.926 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.927 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:56.927 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.577756+0000", 2024-09-18T04:38:56.927 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.927 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.927 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.927 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:56.927 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.577754+0000", 2024-09-18T04:38:56.927 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:56.927 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.927 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.927 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:56.928 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.577762+0000", 2024-09-18T04:38:56.928 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4179999999999997e-06 2024-09-18T04:38:56.928 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.928 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.928 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:56.928 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.577783+0000", 2024-09-18T04:38:56.928 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0105000000000001e-05 2024-09-18T04:38:56.928 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.928 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.928 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:56.928 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.577791+0000", 2024-09-18T04:38:56.928 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2430000000000005e-06 2024-09-18T04:38:56.928 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.929 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.929 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:56.929 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.579272+0000", 2024-09-18T04:38:56.929 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0014808759999999999 2024-09-18T04:38:56.929 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.929 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.929 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:56.929 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.579515+0000", 2024-09-18T04:38:56.929 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000243098 2024-09-18T04:38:56.929 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.929 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.930 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:38:56.930 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.579992+0000", 2024-09-18T04:38:56.930 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00047668899999999998 2024-09-18T04:38:56.930 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.930 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.930 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:56.930 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.581350+0000", 2024-09-18T04:38:56.930 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0013580160000000001 2024-09-18T04:38:56.930 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.930 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.930 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:56.930 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.584240+0000", 2024-09-18T04:38:56.930 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0028906520000000001 2024-09-18T04:38:56.930 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.931 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.931 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:56.931 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.584298+0000", 2024-09-18T04:38:56.931 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7958000000000003e-05 2024-09-18T04:38:56.931 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.931 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.931 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:56.931 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.584336+0000", 2024-09-18T04:38:56.931 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7741000000000003e-05 2024-09-18T04:38:56.931 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.931 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:56.931 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.931 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.931 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.932 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:58 2.7 e51/32 2:fc7c6cd3:ns::rbd_data.133e813ab5bc.000000000000000e:head v 51'7, mlcod=51'5)", 2024-09-18T04:38:56.932 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.605333+0000", 2024-09-18T04:38:56.932 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 14.258327692, 2024-09-18T04:38:56.932 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020085902999999999, 2024-09-18T04:38:56.932 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:56.932 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:56.932 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:56.932 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.932 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:56.932 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605333+0000", 2024-09-18T04:38:56.932 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.932 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.932 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.933 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:56.933 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605333+0000", 2024-09-18T04:38:56.933 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.933 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.933 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.933 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:56.933 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605332+0000", 2024-09-18T04:38:56.933 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:56.933 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.933 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.933 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:56.933 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608667+0000", 2024-09-18T04:38:56.933 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033350300000000001 2024-09-18T04:38:56.934 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.934 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.934 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:56.934 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608706+0000", 2024-09-18T04:38:56.934 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8951e-05 2024-09-18T04:38:56.934 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.934 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.934 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:56.934 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608725+0000", 2024-09-18T04:38:56.934 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9675999999999999e-05 2024-09-18T04:38:56.934 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.934 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.934 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:56.935 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608798+0000", 2024-09-18T04:38:56.935 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3135999999999999e-05 2024-09-18T04:38:56.935 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.935 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.935 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:56.935 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608824+0000", 2024-09-18T04:38:56.935 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5284000000000001e-05 2024-09-18T04:38:56.935 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.935 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.935 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:56.935 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625370+0000", 2024-09-18T04:38:56.935 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016545975000000001 2024-09-18T04:38:56.936 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.936 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.936 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:56.936 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625419+0000", 2024-09-18T04:38:56.936 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9076999999999998e-05 2024-09-18T04:38:56.936 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.936 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:56.936 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.936 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.936 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.936 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:610 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:38:56.936 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.062333+0000", 2024-09-18T04:38:56.937 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 5.8013277219999999, 2024-09-18T04:38:56.937 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0067149669999999996, 2024-09-18T04:38:56.937 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:56.937 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:56.937 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:56.937 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:38:56.937 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:38:56.937 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 610 2024-09-18T04:38:56.937 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.937 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:56.937 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.937 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:56.937 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.062333+0000", 2024-09-18T04:38:56.938 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.938 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.938 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.938 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:56.938 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.062333+0000", 2024-09-18T04:38:56.938 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.938 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.938 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.938 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:56.938 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.062330+0000", 2024-09-18T04:38:56.938 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:56.938 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.938 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.938 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:56.939 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.062628+0000", 2024-09-18T04:38:56.939 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00029734800000000001 2024-09-18T04:38:56.939 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.939 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.939 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:56.939 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.062662+0000", 2024-09-18T04:38:56.939 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4853e-05 2024-09-18T04:38:56.939 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.939 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.939 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:56.939 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.062679+0000", 2024-09-18T04:38:56.939 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6646e-05 2024-09-18T04:38:56.939 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.940 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.940 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:56.940 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.062776+0000", 2024-09-18T04:38:56.940 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6865000000000002e-05 2024-09-18T04:38:56.940 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.940 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.940 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op must wait for map", 2024-09-18T04:38:56.940 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.062784+0000", 2024-09-18T04:38:56.940 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.3769999999999996e-06 2024-09-18T04:38:56.940 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.940 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.940 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:56.940 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.066331+0000", 2024-09-18T04:38:56.941 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003546418 2024-09-18T04:38:56.941 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.941 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.941 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:56.941 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.066446+0000", 2024-09-18T04:38:56.941 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000115074 2024-09-18T04:38:56.941 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.941 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.941 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:38:56.941 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.066638+0000", 2024-09-18T04:38:56.941 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000192597 2024-09-18T04:38:56.941 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.942 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.942 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:56.942 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.068643+0000", 2024-09-18T04:38:56.942 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.002004235 2024-09-18T04:38:56.942 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.942 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.942 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:56.942 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.068919+0000", 2024-09-18T04:38:56.942 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000276111 2024-09-18T04:38:56.943 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.943 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.943 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:56.943 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.068996+0000", 2024-09-18T04:38:56.943 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7474999999999999e-05 2024-09-18T04:38:56.943 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.943 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.943 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:56.943 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.069048+0000", 2024-09-18T04:38:56.943 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1502000000000003e-05 2024-09-18T04:38:56.943 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.943 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:56.943 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.943 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.944 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.944 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:611 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:38:56.944 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.069561+0000", 2024-09-18T04:38:56.944 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 5.794099138, 2024-09-18T04:38:56.944 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0072330299999999997, 2024-09-18T04:38:56.944 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:56.944 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:56.944 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:56.944 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:38:56.944 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:38:56.944 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 611 2024-09-18T04:38:56.944 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.944 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:56.945 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.945 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:56.945 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.069561+0000", 2024-09-18T04:38:56.945 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.945 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.945 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.945 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:56.945 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.069561+0000", 2024-09-18T04:38:56.945 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.945 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.945 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.945 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:56.945 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.069559+0000", 2024-09-18T04:38:56.945 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:56.946 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.946 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.946 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:56.946 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070801+0000", 2024-09-18T04:38:56.946 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0012423359999999999 2024-09-18T04:38:56.946 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.946 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.946 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:56.946 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070828+0000", 2024-09-18T04:38:56.946 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6387e-05 2024-09-18T04:38:56.946 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.946 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.946 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:56.947 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070840+0000", 2024-09-18T04:38:56.947 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2689e-05 2024-09-18T04:38:56.947 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.947 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.947 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:56.947 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070990+0000", 2024-09-18T04:38:56.947 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014946800000000001 2024-09-18T04:38:56.947 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.947 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.947 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:56.947 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.071128+0000", 2024-09-18T04:38:56.947 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013796699999999999 2024-09-18T04:38:56.947 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.948 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.948 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:38:56.948 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.071318+0000", 2024-09-18T04:38:56.948 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000190106 2024-09-18T04:38:56.948 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.948 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.948 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:56.948 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.075711+0000", 2024-09-18T04:38:56.948 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.004393002 2024-09-18T04:38:56.948 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.948 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.948 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:56.948 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.076665+0000", 2024-09-18T04:38:56.949 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00095390399999999999 2024-09-18T04:38:56.949 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.949 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.949 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:56.949 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.076742+0000", 2024-09-18T04:38:56.949 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7112000000000003e-05 2024-09-18T04:38:56.949 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.949 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.949 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:56.949 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.076794+0000", 2024-09-18T04:38:56.949 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2475999999999997e-05 2024-09-18T04:38:56.949 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.949 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:56.949 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.950 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.950 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.950 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4805.0:621 2.7 e57/32 2:fe1b02ad:::rbd_data.125fc09df6d1.0000000000000002:head v 57'13, mlcod=57'11)", 2024-09-18T04:38:56.950 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.127765+0000", 2024-09-18T04:38:56.950 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 5.7358954080000002, 2024-09-18T04:38:56.950 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0063402759999999997, 2024-09-18T04:38:56.950 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:56.950 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:56.950 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:56.950 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.950 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:56.950 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.127765+0000", 2024-09-18T04:38:56.950 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.950 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.951 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.951 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:56.951 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.127765+0000", 2024-09-18T04:38:56.951 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.951 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.951 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.951 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:56.951 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.127764+0000", 2024-09-18T04:38:56.951 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:56.951 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.951 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.951 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:56.951 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.128306+0000", 2024-09-18T04:38:56.951 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00054202000000000005 2024-09-18T04:38:56.952 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.952 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.952 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:56.952 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.128325+0000", 2024-09-18T04:38:56.952 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9045000000000001e-05 2024-09-18T04:38:56.952 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.952 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.952 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:56.953 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.128333+0000", 2024-09-18T04:38:56.953 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7819999999999994e-06 2024-09-18T04:38:56.953 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.953 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.953 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:56.953 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.128421+0000", 2024-09-18T04:38:56.953 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8546000000000002e-05 2024-09-18T04:38:56.953 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.953 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.953 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:56.953 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.128452+0000", 2024-09-18T04:38:56.954 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.0395e-05 2024-09-18T04:38:56.954 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.954 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.954 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:56.954 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.134055+0000", 2024-09-18T04:38:56.954 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0056030239999999999 2024-09-18T04:38:56.954 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.954 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.954 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:56.954 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.134105+0000", 2024-09-18T04:38:56.954 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0599000000000001e-05 2024-09-18T04:38:56.954 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.954 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:56.954 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.955 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.955 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.955 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:683 2.3 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:38:56.955 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.465333+0000", 2024-09-18T04:38:56.955 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 5.3983274510000001, 2024-09-18T04:38:56.955 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0067688740000000002, 2024-09-18T04:38:56.955 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:56.955 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:56.955 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:56.955 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:38:56.955 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:38:56.955 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 683 2024-09-18T04:38:56.956 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.956 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:56.956 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.956 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:56.956 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.465333+0000", 2024-09-18T04:38:56.956 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.956 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.956 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.956 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:56.956 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.465333+0000", 2024-09-18T04:38:56.956 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.956 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.956 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.957 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:56.957 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.465331+0000", 2024-09-18T04:38:56.957 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:56.957 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.957 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.957 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:56.957 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.466305+0000", 2024-09-18T04:38:56.957 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000973894 2024-09-18T04:38:56.957 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.957 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.957 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:56.957 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.466337+0000", 2024-09-18T04:38:56.957 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2140000000000001e-05 2024-09-18T04:38:56.958 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.958 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.958 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:56.958 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.466352+0000", 2024-09-18T04:38:56.958 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4816e-05 2024-09-18T04:38:56.958 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.958 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.958 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:56.958 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.466421+0000", 2024-09-18T04:38:56.958 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.9518999999999996e-05 2024-09-18T04:38:56.958 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.958 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.958 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:56.959 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.466570+0000", 2024-09-18T04:38:56.959 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014881000000000001 2024-09-18T04:38:56.959 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.959 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.959 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:38:56.959 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.466798+0000", 2024-09-18T04:38:56.959 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00022834600000000001 2024-09-18T04:38:56.959 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.959 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.959 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:56.959 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.470760+0000", 2024-09-18T04:38:56.959 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0039617469999999998 2024-09-18T04:38:56.959 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.960 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.960 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:56.960 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.472007+0000", 2024-09-18T04:38:56.960 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0012472010000000001 2024-09-18T04:38:56.960 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.960 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.960 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:56.960 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.472063+0000", 2024-09-18T04:38:56.960 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6119999999999998e-05 2024-09-18T04:38:56.960 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.960 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.960 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:56.960 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.472102+0000", 2024-09-18T04:38:56.961 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8494000000000001e-05 2024-09-18T04:38:56.961 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.961 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:56.961 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.961 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.961 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.961 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:685 2.3 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:38:56.961 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.476620+0000", 2024-09-18T04:38:56.961 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 5.3870407910000004, 2024-09-18T04:38:56.961 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0067145440000000002, 2024-09-18T04:38:56.961 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:56.962 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:56.962 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:56.962 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:38:56.962 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:38:56.962 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 685 2024-09-18T04:38:56.962 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.962 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:56.962 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.962 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:56.962 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.476620+0000", 2024-09-18T04:38:56.962 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.962 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.962 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.962 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:56.962 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.476620+0000", 2024-09-18T04:38:56.963 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.963 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.963 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.963 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:56.963 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.476618+0000", 2024-09-18T04:38:56.963 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:56.963 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.963 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.963 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:56.963 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.477606+0000", 2024-09-18T04:38:56.963 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00098743500000000005 2024-09-18T04:38:56.963 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.963 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.964 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:56.964 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.477624+0000", 2024-09-18T04:38:56.964 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8181000000000001e-05 2024-09-18T04:38:56.964 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.964 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.964 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:56.964 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.477633+0000", 2024-09-18T04:38:56.964 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3780000000000007e-06 2024-09-18T04:38:56.964 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.964 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.964 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:56.964 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.477709+0000", 2024-09-18T04:38:56.964 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.5807000000000001e-05 2024-09-18T04:38:56.965 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.965 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.965 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:56.965 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.477804+0000", 2024-09-18T04:38:56.965 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4741999999999995e-05 2024-09-18T04:38:56.965 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.965 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.965 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:38:56.965 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.477929+0000", 2024-09-18T04:38:56.965 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012492100000000001 2024-09-18T04:38:56.965 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.965 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.965 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:56.965 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.481167+0000", 2024-09-18T04:38:56.966 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0032382639999999998 2024-09-18T04:38:56.966 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.966 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.966 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:56.966 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.483250+0000", 2024-09-18T04:38:56.966 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0020829139999999999 2024-09-18T04:38:56.966 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.966 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.966 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:56.966 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.483296+0000", 2024-09-18T04:38:56.966 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6205999999999998e-05 2024-09-18T04:38:56.966 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.966 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.967 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:56.967 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.483334+0000", 2024-09-18T04:38:56.967 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8235999999999998e-05 2024-09-18T04:38:56.967 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.967 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:56.967 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.967 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.967 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.967 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:687 2.3 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:38:56.967 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.487994+0000", 2024-09-18T04:38:56.967 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 5.3756661550000002, 2024-09-18T04:38:56.967 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0074167690000000001, 2024-09-18T04:38:56.967 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:56.968 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:56.968 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:56.968 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:38:56.968 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:38:56.968 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 687 2024-09-18T04:38:56.968 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.968 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:56.968 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.968 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:56.968 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.487994+0000", 2024-09-18T04:38:56.968 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.968 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.968 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.969 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:56.969 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.487994+0000", 2024-09-18T04:38:56.969 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.969 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.969 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.969 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:56.969 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.487993+0000", 2024-09-18T04:38:56.969 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:38:56.969 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.969 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.969 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:56.969 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.488979+0000", 2024-09-18T04:38:56.969 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00098620700000000001 2024-09-18T04:38:56.970 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.970 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.970 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:56.970 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489002+0000", 2024-09-18T04:38:56.970 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2500000000000001e-05 2024-09-18T04:38:56.970 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.970 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.970 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:56.970 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489020+0000", 2024-09-18T04:38:56.970 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7691000000000001e-05 2024-09-18T04:38:56.970 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.970 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.970 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:56.971 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489091+0000", 2024-09-18T04:38:56.971 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1307e-05 2024-09-18T04:38:56.971 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.971 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.971 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:56.971 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489219+0000", 2024-09-18T04:38:56.971 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012766000000000001 2024-09-18T04:38:56.971 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.971 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.971 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:38:56.971 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489387+0000", 2024-09-18T04:38:56.971 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016855400000000001 2024-09-18T04:38:56.971 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.971 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.972 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:56.972 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495243+0000", 2024-09-18T04:38:56.972 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058563189999999996 2024-09-18T04:38:56.972 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.972 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.972 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:56.972 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495266+0000", 2024-09-18T04:38:56.972 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2810999999999999e-05 2024-09-18T04:38:56.972 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.972 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.972 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:56.972 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495332+0000", 2024-09-18T04:38:56.972 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5469999999999995e-05 2024-09-18T04:38:56.972 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.973 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.973 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:56.973 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495411+0000", 2024-09-18T04:38:56.973 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9395999999999997e-05 2024-09-18T04:38:56.973 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.973 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:56.973 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.973 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.973 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.973 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:689 2.3 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:38:56.973 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.500070+0000", 2024-09-18T04:38:56.973 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 5.3635903389999999, 2024-09-18T04:38:56.973 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071096650000000003, 2024-09-18T04:38:56.974 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:56.974 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:56.974 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:56.974 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:38:56.974 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:38:56.974 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 689 2024-09-18T04:38:56.974 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.974 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:56.974 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.974 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:56.974 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.500070+0000", 2024-09-18T04:38:56.974 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.974 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.974 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.975 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:56.975 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.500070+0000", 2024-09-18T04:38:56.975 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.975 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.975 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.975 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:56.975 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.500069+0000", 2024-09-18T04:38:56.975 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:56.975 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.975 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.975 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:56.975 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501082+0000", 2024-09-18T04:38:56.975 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010136570000000001 2024-09-18T04:38:56.975 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.976 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.976 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:56.976 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501104+0000", 2024-09-18T04:38:56.976 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1883e-05 2024-09-18T04:38:56.976 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.976 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.976 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:56.976 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501115+0000", 2024-09-18T04:38:56.976 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0414000000000001e-05 2024-09-18T04:38:56.976 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.976 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.976 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:56.976 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501178+0000", 2024-09-18T04:38:56.976 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3128999999999998e-05 2024-09-18T04:38:56.976 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.977 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.977 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:56.977 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501257+0000", 2024-09-18T04:38:56.977 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9592999999999995e-05 2024-09-18T04:38:56.977 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.977 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.977 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:38:56.977 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501366+0000", 2024-09-18T04:38:56.977 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000108643 2024-09-18T04:38:56.977 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.977 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.977 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:56.977 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.504041+0000", 2024-09-18T04:38:56.978 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0026750889999999999 2024-09-18T04:38:56.978 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.978 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.978 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:56.978 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.507097+0000", 2024-09-18T04:38:56.978 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030556070000000001 2024-09-18T04:38:56.978 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.978 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.978 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:56.978 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.507140+0000", 2024-09-18T04:38:56.978 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3745000000000002e-05 2024-09-18T04:38:56.978 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.978 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.979 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:56.979 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.507180+0000", 2024-09-18T04:38:56.979 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9475999999999999e-05 2024-09-18T04:38:56.979 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.979 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:56.979 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.979 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.979 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.979 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4972.0:13 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:38:56.979 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.948263+0000", 2024-09-18T04:38:56.979 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 4.9153975479999996, 2024-09-18T04:38:56.979 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0091358970000000005, 2024-09-18T04:38:56.979 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:56.980 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:38:56.980 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:56.980 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4972", 2024-09-18T04:38:56.980 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2280251472", 2024-09-18T04:38:56.980 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 13 2024-09-18T04:38:56.980 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.980 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:56.980 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.980 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:56.980 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948263+0000", 2024-09-18T04:38:56.980 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.980 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.980 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.981 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:56.981 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948263+0000", 2024-09-18T04:38:56.981 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.981 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.981 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.981 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:56.981 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948261+0000", 2024-09-18T04:38:56.981 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:56.981 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.981 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.981 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:56.981 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948273+0000", 2024-09-18T04:38:56.981 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1576999999999999e-05 2024-09-18T04:38:56.981 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.982 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.982 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:56.982 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948305+0000", 2024-09-18T04:38:56.982 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2209000000000003e-05 2024-09-18T04:38:56.982 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.982 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.982 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:56.982 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948325+0000", 2024-09-18T04:38:56.982 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.022e-05 2024-09-18T04:38:56.982 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.982 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.982 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:56.982 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948455+0000", 2024-09-18T04:38:56.983 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000129624 2024-09-18T04:38:56.983 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.983 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.983 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:56.983 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948598+0000", 2024-09-18T04:38:56.983 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014376699999999999 2024-09-18T04:38:56.983 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.983 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.983 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:56.983 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.957399+0000", 2024-09-18T04:38:56.983 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0088004299999999997 2024-09-18T04:38:56.983 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.983 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:56.984 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.984 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.984 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.984 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:101 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:38:56.984 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.576367+0000", 2024-09-18T04:38:56.984 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.28729384000000002, 2024-09-18T04:38:56.984 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0078671020000000008, 2024-09-18T04:38:56.984 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:38:56.984 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:56.984 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:38:56.984 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:38:56.984 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:38:56.984 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 101 2024-09-18T04:38:56.985 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.985 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:38:56.985 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.985 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:38:56.985 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576367+0000", 2024-09-18T04:38:56.985 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.985 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.985 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.985 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:38:56.985 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576367+0000", 2024-09-18T04:38:56.985 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:38:56.985 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.985 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.986 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:38:56.986 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576364+0000", 2024-09-18T04:38:56.986 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999971 2024-09-18T04:38:56.986 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.986 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.986 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:38:56.986 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576378+0000", 2024-09-18T04:38:56.986 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3879999999999999e-05 2024-09-18T04:38:56.986 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.986 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.986 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:38:56.986 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576410+0000", 2024-09-18T04:38:56.986 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2286000000000002e-05 2024-09-18T04:38:56.987 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.987 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.987 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:38:56.987 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576425+0000", 2024-09-18T04:38:56.987 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5290000000000001e-05 2024-09-18T04:38:56.987 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.987 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.987 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:38:56.988 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576551+0000", 2024-09-18T04:38:56.988 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012555199999999999 2024-09-18T04:38:56.988 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.988 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.988 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:38:56.988 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576719+0000", 2024-09-18T04:38:56.988 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016787400000000001 2024-09-18T04:38:56.988 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.988 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.988 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:38:56.988 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576951+0000", 2024-09-18T04:38:56.988 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00023252400000000001 2024-09-18T04:38:56.988 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.989 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.989 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:38:56.989 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.581459+0000", 2024-09-18T04:38:56.989 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045074410000000001 2024-09-18T04:38:56.989 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.989 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.989 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:38:56.989 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584076+0000", 2024-09-18T04:38:56.989 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0026173860000000002 2024-09-18T04:38:56.989 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.989 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.989 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:38:56.989 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584160+0000", 2024-09-18T04:38:56.989 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4191999999999996e-05 2024-09-18T04:38:56.990 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:38:56.990 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:38:56.990 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:38:56.990 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584234+0000", 2024-09-18T04:38:56.990 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.339e-05 2024-09-18T04:38:56.990 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.990 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:56.990 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.990 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:38:56.990 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:38:56.990 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:38:56.993 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:56.985+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:56.993 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:38:56.995 INFO:tasks.workunit.client.0.smithi115.stderr:+ used=0 2024-09-18T04:38:56.995 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 0 -lt 67108864 ']' 2024-09-18T04:38:56.995 INFO:tasks.workunit.client.0.smithi115.stderr:+ unmap_device /dev/nbd0 18009 2024-09-18T04:38:56.995 INFO:tasks.workunit.client.0.smithi115.stderr:+ local args=/dev/nbd0 2024-09-18T04:38:56.995 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pid=18009 2024-09-18T04:38:56.995 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-18T04:38:56.995 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:38:56.995 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:38:56.995 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:38:56.995 INFO:tasks.workunit.client.0.smithi115.stderr:++ which rbd 2024-09-18T04:38:56.995 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/bin/rbd 2024-09-18T04:38:56.995 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:38:56.995 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-18T04:38:57.006 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T04:38:57.022 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-18T04:38:57.022 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T04:38:57.022 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-18T04:38:57.023 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-18T04:38:57.038 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd device --device-type nbd list 2024-09-18T04:38:57.038 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false grep '^18009\b' 2024-09-18T04:38:57.038 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep '^18009\b' 2024-09-18T04:38:57.092 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:57.085+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:57.106 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:38:57.107 INFO:tasks.workunit.client.0.smithi115.stderr:+ ps -C rbd-nbd 2024-09-18T04:38:57.107 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false grep '^ *18009\b' 2024-09-18T04:38:57.107 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep '^ *18009\b' 2024-09-18T04:38:57.115 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:57.115 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:57.115 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:38:57.115 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:57.118 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:38:57.118 INFO:tasks.workunit.client.0.smithi115.stderr:+ sleep 0.5 2024-09-18T04:38:57.126 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:38:57.189 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:38:57.193 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:57.185+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:57.210 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:38:57.230 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:38:57.248 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:57.248 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:57.252 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:38:57.258 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:38:57.278 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:38:57.294 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:57.289+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:57.315 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:38:57.317 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:38:57.318 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:38:57.318 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:38:57.318 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:38:57.318 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:38:57.318 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:38:57.332 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:38:57.332 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:57.332 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 60, 2024-09-18T04:38:57.332 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "be6146ba-a991-4f6e-b359-99cbedc9f9b1", 2024-09-18T04:38:57.332 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-09-18T04:37:15.676634+0000", 2024-09-18T04:38:57.332 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-09-18T04:38:55.424447+0000", 2024-09-18T04:38:57.332 INFO:teuthology.orchestra.run.smithi028.stdout: "last_up_change": "2024-09-18T04:37:25.403175+0000", 2024-09-18T04:38:57.332 INFO:teuthology.orchestra.run.smithi028.stdout: "last_in_change": "2024-09-18T04:37:50.652169+0000", 2024-09-18T04:38:57.332 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-18T04:38:57.332 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_num": 5799936, 2024-09-18T04:38:57.332 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_set": [ 2024-09-18T04:38:57.332 INFO:teuthology.orchestra.run.smithi028.stdout: "pglog_hardlimit", 2024-09-18T04:38:57.333 INFO:teuthology.orchestra.run.smithi028.stdout: "purged_snapdirs", 2024-09-18T04:38:57.333 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_deletes", 2024-09-18T04:38:57.333 INFO:teuthology.orchestra.run.smithi028.stdout: "sortbitwise" 2024-09-18T04:38:57.333 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-18T04:38:57.333 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_version": 5, 2024-09-18T04:38:57.333 INFO:teuthology.orchestra.run.smithi028.stdout: "full_ratio": 0.94999998807907104, 2024-09-18T04:38:57.333 INFO:teuthology.orchestra.run.smithi028.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-18T04:38:57.333 INFO:teuthology.orchestra.run.smithi028.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-18T04:38:57.333 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_snapshot": "", 2024-09-18T04:38:57.333 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_max": 3, 2024-09-18T04:38:57.333 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd": 6, 2024-09-18T04:38:57.333 INFO:teuthology.orchestra.run.smithi028.stdout: "require_min_compat_client": "luminous", 2024-09-18T04:38:57.333 INFO:teuthology.orchestra.run.smithi028.stdout: "min_compat_client": "luminous", 2024-09-18T04:38:57.334 INFO:teuthology.orchestra.run.smithi028.stdout: "require_osd_release": "reef", 2024-09-18T04:38:57.334 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_crimson": false, 2024-09-18T04:38:57.334 INFO:teuthology.orchestra.run.smithi028.stdout: "pools": [ 2024-09-18T04:38:57.334 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.334 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 1, 2024-09-18T04:38:57.334 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": ".mgr", 2024-09-18T04:38:57.334 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-18T04:37:25.654950+0000", 2024-09-18T04:38:57.334 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-18T04:38:57.334 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-09-18T04:38:57.334 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-18T04:38:57.334 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 2, 2024-09-18T04:38:57.334 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 1, 2024-09-18T04:38:57.334 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-18T04:38:57.334 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-18T04:38:57.335 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-18T04:38:57.335 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T04:38:57.335 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T04:38:57.335 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-18T04:38:57.335 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-18T04:38:57.335 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 1, 2024-09-18T04:38:57.335 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 1, 2024-09-18T04:38:57.335 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 1, 2024-09-18T04:38:57.335 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 1, 2024-09-18T04:38:57.335 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 1, 2024-09-18T04:38:57.335 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-18T04:38:57.335 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-18T04:38:57.335 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-18T04:38:57.336 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-18T04:38:57.336 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-18T04:38:57.336 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-18T04:38:57.336 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-18T04:38:57.336 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.336 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "16", 2024-09-18T04:38:57.336 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-18T04:38:57.336 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T04:38:57.336 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T04:38:57.336 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-18T04:38:57.336 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-18T04:38:57.336 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-09-18T04:38:57.336 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-09-18T04:38:57.337 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-18T04:38:57.337 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-18T04:38:57.337 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-18T04:38:57.337 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-18T04:38:57.337 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-18T04:38:57.337 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-18T04:38:57.337 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-18T04:38:57.337 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-18T04:38:57.337 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-18T04:38:57.337 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-18T04:38:57.337 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-18T04:38:57.337 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T04:38:57.337 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T04:38:57.337 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T04:38:57.338 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-18T04:38:57.338 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-18T04:38:57.338 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-18T04:38:57.338 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-18T04:38:57.338 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-18T04:38:57.338 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.338 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-18T04:38:57.338 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-18T04:38:57.338 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-18T04:38:57.338 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-18T04:38:57.338 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-18T04:38:57.338 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T04:38:57.338 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-18T04:38:57.338 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-18T04:38:57.338 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-18T04:38:57.339 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-18T04:38:57.339 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-18T04:38:57.339 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-09-18T04:38:57.339 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_max": 32, 2024-09-18T04:38:57.339 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_min": 1 2024-09-18T04:38:57.339 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.339 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-18T04:38:57.339 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": {} 2024-09-18T04:38:57.339 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.339 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-09-18T04:38:57.339 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 6.059999942779541, 2024-09-18T04:38:57.339 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 6.059999942779541, 2024-09-18T04:38:57.339 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.33000001311302185, 2024-09-18T04:38:57.339 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2, 2024-09-18T04:38:57.340 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2, 2024-09-18T04:38:57.340 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 1, 2024-09-18T04:38:57.340 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 1, 2024-09-18T04:38:57.340 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 1 2024-09-18T04:38:57.340 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:57.340 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.340 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.340 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 2, 2024-09-18T04:38:57.340 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "rbd", 2024-09-18T04:38:57.340 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-18T04:37:32.672462+0000", 2024-09-18T04:38:57.340 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 8193, 2024-09-18T04:38:57.340 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-18T04:38:57.340 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-18T04:38:57.340 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 2, 2024-09-18T04:38:57.341 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 1, 2024-09-18T04:38:57.341 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-18T04:38:57.341 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-18T04:38:57.341 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-18T04:38:57.341 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T04:38:57.341 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T04:38:57.341 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-18T04:38:57.341 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-18T04:38:57.341 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 8, 2024-09-18T04:38:57.341 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 8, 2024-09-18T04:38:57.341 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 8, 2024-09-18T04:38:57.341 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 8, 2024-09-18T04:38:57.341 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 8, 2024-09-18T04:38:57.341 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-18T04:38:57.342 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-18T04:38:57.342 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-18T04:38:57.342 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-18T04:38:57.342 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-18T04:38:57.342 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-18T04:38:57.342 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-18T04:38:57.342 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.342 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "20", 2024-09-18T04:38:57.342 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-18T04:38:57.342 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T04:38:57.342 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T04:38:57.342 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-18T04:38:57.342 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-18T04:38:57.342 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 3, 2024-09-18T04:38:57.342 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 20, 2024-09-18T04:38:57.343 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-18T04:38:57.343 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-18T04:38:57.343 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-18T04:38:57.343 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-18T04:38:57.343 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-18T04:38:57.343 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-18T04:38:57.343 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-18T04:38:57.343 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-18T04:38:57.343 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-18T04:38:57.343 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-18T04:38:57.343 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-18T04:38:57.343 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T04:38:57.343 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T04:38:57.343 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T04:38:57.344 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-18T04:38:57.344 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-18T04:38:57.344 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-18T04:38:57.344 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-18T04:38:57.344 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-18T04:38:57.344 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.344 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-18T04:38:57.344 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-18T04:38:57.344 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-18T04:38:57.344 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-18T04:38:57.344 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-18T04:38:57.344 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T04:38:57.344 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-18T04:38:57.344 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-18T04:38:57.345 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-18T04:38:57.345 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-18T04:38:57.345 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-18T04:38:57.345 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-09-18T04:38:57.345 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-18T04:38:57.345 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd": {} 2024-09-18T04:38:57.345 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.345 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-09-18T04:38:57.345 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.5099999904632568, 2024-09-18T04:38:57.345 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.5099999904632568, 2024-09-18T04:38:57.345 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.82999998331069946, 2024-09-18T04:38:57.345 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.25, 2024-09-18T04:38:57.345 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.25, 2024-09-18T04:38:57.345 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 1, 2024-09-18T04:38:57.346 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 1, 2024-09-18T04:38:57.346 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 1 2024-09-18T04:38:57.346 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:57.346 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.346 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.346 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 3, 2024-09-18T04:38:57.346 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "new_default_pool17342", 2024-09-18T04:38:57.346 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-18T04:38:42.986132+0000", 2024-09-18T04:38:57.346 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 8193, 2024-09-18T04:38:57.346 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-18T04:38:57.346 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-18T04:38:57.346 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 2, 2024-09-18T04:38:57.346 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 1, 2024-09-18T04:38:57.346 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-18T04:38:57.347 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-18T04:38:57.347 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-18T04:38:57.347 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T04:38:57.347 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T04:38:57.347 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-18T04:38:57.347 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-18T04:38:57.347 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 8, 2024-09-18T04:38:57.347 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 8, 2024-09-18T04:38:57.347 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 8, 2024-09-18T04:38:57.347 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 8, 2024-09-18T04:38:57.347 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 8, 2024-09-18T04:38:57.347 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-18T04:38:57.347 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-18T04:38:57.347 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-18T04:38:57.347 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-18T04:38:57.348 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-18T04:38:57.348 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-18T04:38:57.348 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-18T04:38:57.348 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.348 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "55", 2024-09-18T04:38:57.348 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-18T04:38:57.348 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T04:38:57.348 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T04:38:57.348 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-18T04:38:57.348 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-18T04:38:57.348 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 3, 2024-09-18T04:38:57.348 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 55, 2024-09-18T04:38:57.348 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-18T04:38:57.349 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-18T04:38:57.349 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-18T04:38:57.349 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-18T04:38:57.349 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-18T04:38:57.349 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-18T04:38:57.349 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-18T04:38:57.349 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-18T04:38:57.349 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-18T04:38:57.349 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-18T04:38:57.349 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-18T04:38:57.349 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T04:38:57.349 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T04:38:57.349 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T04:38:57.349 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-18T04:38:57.349 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-18T04:38:57.350 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-18T04:38:57.350 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-18T04:38:57.350 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-18T04:38:57.350 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.350 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-18T04:38:57.350 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-18T04:38:57.350 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-18T04:38:57.350 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-18T04:38:57.350 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-18T04:38:57.350 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T04:38:57.350 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-18T04:38:57.350 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-18T04:38:57.350 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-18T04:38:57.350 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-18T04:38:57.351 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-18T04:38:57.351 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-09-18T04:38:57.351 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-18T04:38:57.351 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd": {} 2024-09-18T04:38:57.351 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.351 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-09-18T04:38:57.351 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2.2699999809265137, 2024-09-18T04:38:57.351 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2.2699999809265137, 2024-09-18T04:38:57.351 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.82999998331069946, 2024-09-18T04:38:57.351 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.8799999952316284, 2024-09-18T04:38:57.351 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.8799999952316284, 2024-09-18T04:38:57.351 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 1, 2024-09-18T04:38:57.351 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 1, 2024-09-18T04:38:57.351 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 1 2024-09-18T04:38:57.352 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:57.352 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:57.352 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-18T04:38:57.352 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-09-18T04:38:57.352 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.352 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-09-18T04:38:57.352 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "6177002d-72e6-4888-bcde-29b6a92d9dc4", 2024-09-18T04:38:57.352 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-18T04:38:57.352 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-18T04:38:57.352 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-18T04:38:57.352 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-18T04:38:57.352 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-18T04:38:57.352 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-18T04:38:57.352 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 13, 2024-09-18T04:38:57.353 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 52, 2024-09-18T04:38:57.353 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-18T04:38:57.353 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-18T04:38:57.353 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-18T04:38:57.353 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:57.353 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.353 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:57.353 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6801", 2024-09-18T04:38:57.353 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1554407851 2024-09-18T04:38:57.353 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.353 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.353 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:57.353 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6803", 2024-09-18T04:38:57.354 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1554407851 2024-09-18T04:38:57.354 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:57.354 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:57.354 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.354 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-18T04:38:57.354 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:57.354 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.355 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:57.355 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6805", 2024-09-18T04:38:57.355 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1554407851 2024-09-18T04:38:57.355 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.355 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.355 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:57.355 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6807", 2024-09-18T04:38:57.355 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1554407851 2024-09-18T04:38:57.355 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:57.355 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:57.355 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.355 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-18T04:38:57.355 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:57.355 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.356 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:57.356 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6813", 2024-09-18T04:38:57.356 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1554407851 2024-09-18T04:38:57.356 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.356 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.356 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:57.356 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6815", 2024-09-18T04:38:57.356 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1554407851 2024-09-18T04:38:57.356 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:57.356 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:57.356 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.356 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-18T04:38:57.356 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:57.357 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.357 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:57.357 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6809", 2024-09-18T04:38:57.357 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1554407851 2024-09-18T04:38:57.357 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.357 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.357 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:57.357 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6811", 2024-09-18T04:38:57.357 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1554407851 2024-09-18T04:38:57.357 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:57.357 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:57.357 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.357 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6803/1554407851", 2024-09-18T04:38:57.357 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6807/1554407851", 2024-09-18T04:38:57.358 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6815/1554407851", 2024-09-18T04:38:57.358 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6811/1554407851", 2024-09-18T04:38:57.358 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-18T04:38:57.358 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-18T04:38:57.358 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-18T04:38:57.358 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:57.358 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.358 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.358 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-09-18T04:38:57.358 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "4f62cb09-3457-418c-bb69-491d9b70914c", 2024-09-18T04:38:57.358 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-18T04:38:57.358 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-18T04:38:57.358 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-18T04:38:57.358 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-18T04:38:57.359 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-18T04:38:57.359 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-18T04:38:57.359 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 12, 2024-09-18T04:38:57.359 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 52, 2024-09-18T04:38:57.359 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-18T04:38:57.359 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-18T04:38:57.359 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-18T04:38:57.359 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:57.359 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.359 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:57.359 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6800", 2024-09-18T04:38:57.359 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 511141401 2024-09-18T04:38:57.359 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.359 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.360 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:57.360 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6802", 2024-09-18T04:38:57.360 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 511141401 2024-09-18T04:38:57.360 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:57.360 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:57.360 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.360 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-18T04:38:57.360 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:57.360 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.360 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:57.360 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6804", 2024-09-18T04:38:57.360 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 511141401 2024-09-18T04:38:57.360 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.360 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.361 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:57.361 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6806", 2024-09-18T04:38:57.361 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 511141401 2024-09-18T04:38:57.361 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:57.361 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:57.361 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.361 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-18T04:38:57.361 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:57.361 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.361 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:57.361 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6812", 2024-09-18T04:38:57.361 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 511141401 2024-09-18T04:38:57.361 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.361 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.362 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:57.362 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6814", 2024-09-18T04:38:57.362 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 511141401 2024-09-18T04:38:57.362 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:57.362 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:57.362 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.362 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-18T04:38:57.362 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:57.362 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.362 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:57.362 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6808", 2024-09-18T04:38:57.362 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 511141401 2024-09-18T04:38:57.362 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.362 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.363 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:57.363 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6810", 2024-09-18T04:38:57.363 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 511141401 2024-09-18T04:38:57.363 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:57.363 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:57.363 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.363 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6802/511141401", 2024-09-18T04:38:57.363 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6806/511141401", 2024-09-18T04:38:57.363 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6814/511141401", 2024-09-18T04:38:57.363 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6810/511141401", 2024-09-18T04:38:57.363 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-18T04:38:57.363 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-18T04:38:57.363 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-18T04:38:57.363 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:57.364 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.364 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.364 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-09-18T04:38:57.364 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "854fe031-3b87-4290-9c2f-3fb196fb806c", 2024-09-18T04:38:57.364 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-18T04:38:57.364 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-18T04:38:57.364 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-18T04:38:57.364 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-18T04:38:57.364 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-18T04:38:57.364 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-18T04:38:57.364 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 12, 2024-09-18T04:38:57.364 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 52, 2024-09-18T04:38:57.364 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-18T04:38:57.364 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-18T04:38:57.365 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-18T04:38:57.365 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:57.365 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.365 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:57.365 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6816", 2024-09-18T04:38:57.365 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2623426735 2024-09-18T04:38:57.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.365 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.365 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:57.365 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6817", 2024-09-18T04:38:57.365 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2623426735 2024-09-18T04:38:57.365 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:57.365 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:57.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.366 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-18T04:38:57.366 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:57.366 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.366 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:57.366 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6818", 2024-09-18T04:38:57.366 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2623426735 2024-09-18T04:38:57.366 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.366 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.366 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:57.366 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6819", 2024-09-18T04:38:57.366 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2623426735 2024-09-18T04:38:57.366 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:57.366 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:57.366 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.367 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-18T04:38:57.367 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:57.367 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.367 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:57.367 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6822", 2024-09-18T04:38:57.367 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2623426735 2024-09-18T04:38:57.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.367 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.367 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:57.367 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6823", 2024-09-18T04:38:57.367 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2623426735 2024-09-18T04:38:57.367 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:57.367 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:57.368 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.368 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-18T04:38:57.368 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:57.368 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.368 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:57.368 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6820", 2024-09-18T04:38:57.368 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2623426735 2024-09-18T04:38:57.368 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.368 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.368 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:57.368 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6821", 2024-09-18T04:38:57.368 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2623426735 2024-09-18T04:38:57.369 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:57.369 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:57.369 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.369 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6817/2623426735", 2024-09-18T04:38:57.369 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6819/2623426735", 2024-09-18T04:38:57.369 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6823/2623426735", 2024-09-18T04:38:57.369 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6821/2623426735", 2024-09-18T04:38:57.369 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-18T04:38:57.369 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-18T04:38:57.369 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-18T04:38:57.369 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:57.369 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.369 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.370 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-09-18T04:38:57.370 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "c2625f98-780b-4f86-bf36-a48d2a227756", 2024-09-18T04:38:57.370 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-18T04:38:57.370 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-18T04:38:57.370 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-18T04:38:57.370 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-18T04:38:57.370 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-18T04:38:57.370 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-18T04:38:57.370 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 12, 2024-09-18T04:38:57.370 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 32, 2024-09-18T04:38:57.370 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-18T04:38:57.370 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-18T04:38:57.370 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-18T04:38:57.370 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:57.371 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.371 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:57.371 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6808", 2024-09-18T04:38:57.371 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2746051132 2024-09-18T04:38:57.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.371 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.371 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:57.371 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6809", 2024-09-18T04:38:57.371 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2746051132 2024-09-18T04:38:57.371 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:57.371 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:57.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.371 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-18T04:38:57.371 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:57.372 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.372 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:57.372 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6810", 2024-09-18T04:38:57.372 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2746051132 2024-09-18T04:38:57.372 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.372 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.372 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:57.372 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6811", 2024-09-18T04:38:57.372 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2746051132 2024-09-18T04:38:57.372 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:57.372 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:57.372 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.372 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-18T04:38:57.372 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:57.373 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.373 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:57.373 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6814", 2024-09-18T04:38:57.373 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2746051132 2024-09-18T04:38:57.373 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.373 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.373 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:57.373 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6815", 2024-09-18T04:38:57.373 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2746051132 2024-09-18T04:38:57.373 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:57.373 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:57.373 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.373 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-18T04:38:57.374 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:57.374 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.374 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:57.374 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6812", 2024-09-18T04:38:57.374 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2746051132 2024-09-18T04:38:57.374 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.374 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.374 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:57.374 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6813", 2024-09-18T04:38:57.374 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2746051132 2024-09-18T04:38:57.374 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:57.374 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:57.374 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.374 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.82:6809/2746051132", 2024-09-18T04:38:57.375 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.82:6811/2746051132", 2024-09-18T04:38:57.375 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.82:6815/2746051132", 2024-09-18T04:38:57.375 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.82:6813/2746051132", 2024-09-18T04:38:57.375 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-18T04:38:57.375 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-18T04:38:57.375 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-18T04:38:57.375 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:57.375 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.375 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.375 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-09-18T04:38:57.375 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "deb22d86-4081-484d-96eb-9cb9d96b60bc", 2024-09-18T04:38:57.375 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-18T04:38:57.375 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-18T04:38:57.375 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-18T04:38:57.376 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-18T04:38:57.376 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-18T04:38:57.376 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-18T04:38:57.376 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 13, 2024-09-18T04:38:57.376 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 52, 2024-09-18T04:38:57.376 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-18T04:38:57.376 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-18T04:38:57.376 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-18T04:38:57.376 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:57.376 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.376 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:57.376 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6800", 2024-09-18T04:38:57.376 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2005712460 2024-09-18T04:38:57.376 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.377 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.377 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:57.377 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6801", 2024-09-18T04:38:57.377 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2005712460 2024-09-18T04:38:57.377 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:57.377 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:57.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.377 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-18T04:38:57.377 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:57.377 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.377 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:57.377 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6802", 2024-09-18T04:38:57.377 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2005712460 2024-09-18T04:38:57.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.378 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.378 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:57.378 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6803", 2024-09-18T04:38:57.378 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2005712460 2024-09-18T04:38:57.378 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:57.378 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:57.378 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.378 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-18T04:38:57.378 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:57.378 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.378 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:57.378 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6806", 2024-09-18T04:38:57.378 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2005712460 2024-09-18T04:38:57.378 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.379 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.379 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:57.379 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6807", 2024-09-18T04:38:57.379 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2005712460 2024-09-18T04:38:57.379 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:57.379 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:57.379 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.379 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-18T04:38:57.379 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:57.379 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.379 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:57.379 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6804", 2024-09-18T04:38:57.379 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2005712460 2024-09-18T04:38:57.379 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.380 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.380 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:57.380 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6805", 2024-09-18T04:38:57.380 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2005712460 2024-09-18T04:38:57.380 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:57.380 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:57.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.380 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.82:6801/2005712460", 2024-09-18T04:38:57.380 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.82:6803/2005712460", 2024-09-18T04:38:57.380 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.82:6807/2005712460", 2024-09-18T04:38:57.380 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.82:6805/2005712460", 2024-09-18T04:38:57.380 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-18T04:38:57.380 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-18T04:38:57.380 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-18T04:38:57.381 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:57.381 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.381 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.381 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-09-18T04:38:57.381 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "061d332c-f51b-403d-bb18-eba415c97e7f", 2024-09-18T04:38:57.381 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-18T04:38:57.381 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-18T04:38:57.381 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-18T04:38:57.381 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-18T04:38:57.381 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-18T04:38:57.381 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-18T04:38:57.381 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 13, 2024-09-18T04:38:57.381 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 52, 2024-09-18T04:38:57.381 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-18T04:38:57.382 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-18T04:38:57.382 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-18T04:38:57.382 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:57.382 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.382 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:57.382 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6816", 2024-09-18T04:38:57.382 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1267644232 2024-09-18T04:38:57.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.382 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.382 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:57.382 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6817", 2024-09-18T04:38:57.382 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1267644232 2024-09-18T04:38:57.382 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:57.382 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:57.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.383 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-18T04:38:57.383 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:57.383 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.383 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:57.383 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6818", 2024-09-18T04:38:57.383 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1267644232 2024-09-18T04:38:57.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.383 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.383 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:57.383 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6819", 2024-09-18T04:38:57.383 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1267644232 2024-09-18T04:38:57.383 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:57.383 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:57.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.384 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-18T04:38:57.384 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:57.384 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.384 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:57.384 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6822", 2024-09-18T04:38:57.384 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1267644232 2024-09-18T04:38:57.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.384 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.384 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:57.384 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6823", 2024-09-18T04:38:57.384 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1267644232 2024-09-18T04:38:57.384 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:57.385 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:57.385 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.385 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-18T04:38:57.385 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:38:57.385 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.385 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:38:57.385 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6820", 2024-09-18T04:38:57.385 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1267644232 2024-09-18T04:38:57.385 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.385 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.385 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:38:57.385 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6821", 2024-09-18T04:38:57.385 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1267644232 2024-09-18T04:38:57.386 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:57.386 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:57.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.386 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.82:6817/1267644232", 2024-09-18T04:38:57.386 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.82:6819/1267644232", 2024-09-18T04:38:57.386 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.82:6823/1267644232", 2024-09-18T04:38:57.386 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.82:6821/1267644232", 2024-09-18T04:38:57.386 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-18T04:38:57.386 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-18T04:38:57.386 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-18T04:38:57.386 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:57.386 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:57.386 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-18T04:38:57.386 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_xinfo": [ 2024-09-18T04:38:57.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.387 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-09-18T04:38:57.387 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-18T04:38:57.387 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-18T04:38:57.387 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-18T04:38:57.387 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138322906710015, 2024-09-18T04:38:57.387 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-18T04:38:57.387 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-18T04:37:22.192087+0000", 2024-09-18T04:38:57.387 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-18T04:38:57.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.387 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-09-18T04:38:57.387 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-18T04:38:57.387 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-18T04:38:57.388 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-18T04:38:57.388 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138322906710015, 2024-09-18T04:38:57.388 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-18T04:38:57.388 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-18T04:37:22.017833+0000", 2024-09-18T04:38:57.388 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-18T04:38:57.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.388 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.388 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-09-18T04:38:57.388 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-18T04:38:57.388 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-18T04:38:57.388 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-18T04:38:57.388 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138322906710015, 2024-09-18T04:38:57.388 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-18T04:38:57.388 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-18T04:37:22.755774+0000", 2024-09-18T04:38:57.389 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-18T04:38:57.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.390 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.390 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-09-18T04:38:57.390 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-18T04:38:57.391 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-18T04:38:57.391 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-18T04:38:57.391 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138322906710015, 2024-09-18T04:38:57.391 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-18T04:38:57.391 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-18T04:37:22.663785+0000", 2024-09-18T04:38:57.392 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-18T04:38:57.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.392 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.392 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-09-18T04:38:57.392 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-18T04:38:57.392 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-18T04:38:57.392 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-18T04:38:57.392 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138322906710015, 2024-09-18T04:38:57.392 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-18T04:38:57.392 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-18T04:37:22.531758+0000", 2024-09-18T04:38:57.393 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-18T04:38:57.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.393 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.393 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-09-18T04:38:57.393 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-18T04:38:57.393 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-18T04:38:57.393 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-18T04:38:57.393 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138322906710015, 2024-09-18T04:38:57.393 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-18T04:38:57.393 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-18T04:37:22.400774+0000", 2024-09-18T04:38:57.393 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-18T04:38:57.393 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:57.393 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-18T04:38:57.394 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap": [ 2024-09-18T04:38:57.394 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.394 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "2.5", 2024-09-18T04:38:57.394 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-09-18T04:38:57.394 INFO:teuthology.orchestra.run.smithi028.stdout: 2, 2024-09-18T04:38:57.394 INFO:teuthology.orchestra.run.smithi028.stdout: 0 2024-09-18T04:38:57.394 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:57.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.394 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.394 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "2.7", 2024-09-18T04:38:57.394 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-09-18T04:38:57.394 INFO:teuthology.orchestra.run.smithi028.stdout: 3, 2024-09-18T04:38:57.394 INFO:teuthology.orchestra.run.smithi028.stdout: 5 2024-09-18T04:38:57.395 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:57.395 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:57.395 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-18T04:38:57.395 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap_items": [ 2024-09-18T04:38:57.395 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.395 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "2.1", 2024-09-18T04:38:57.395 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-09-18T04:38:57.395 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.395 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 2, 2024-09-18T04:38:57.395 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 1 2024-09-18T04:38:57.395 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:57.395 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:57.395 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:57.396 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-18T04:38:57.396 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap_primaries": [], 2024-09-18T04:38:57.396 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_temp": [], 2024-09-18T04:38:57.396 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_temp": [], 2024-09-18T04:38:57.396 INFO:teuthology.orchestra.run.smithi028.stdout: "blocklist": {}, 2024-09-18T04:38:57.396 INFO:teuthology.orchestra.run.smithi028.stdout: "range_blocklist": {}, 2024-09-18T04:38:57.396 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profiles": { 2024-09-18T04:38:57.396 INFO:teuthology.orchestra.run.smithi028.stdout: "default": { 2024-09-18T04:38:57.396 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-failure-domain": "osd", 2024-09-18T04:38:57.396 INFO:teuthology.orchestra.run.smithi028.stdout: "k": "2", 2024-09-18T04:38:57.396 INFO:teuthology.orchestra.run.smithi028.stdout: "m": "1", 2024-09-18T04:38:57.396 INFO:teuthology.orchestra.run.smithi028.stdout: "plugin": "jerasure", 2024-09-18T04:38:57.396 INFO:teuthology.orchestra.run.smithi028.stdout: "technique": "reed_sol_van" 2024-09-18T04:38:57.397 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:57.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:57.397 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps_queue": [], 2024-09-18T04:38:57.397 INFO:teuthology.orchestra.run.smithi028.stdout: "new_removed_snaps": [], 2024-09-18T04:38:57.397 INFO:teuthology.orchestra.run.smithi028.stdout: "new_purged_snaps": [ 2024-09-18T04:38:57.397 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.397 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 3, 2024-09-18T04:38:57.397 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-09-18T04:38:57.397 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:57.397 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-09-18T04:38:57.397 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-09-18T04:38:57.398 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:57.398 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:57.398 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:57.398 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-18T04:38:57.398 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_node_flags": {}, 2024-09-18T04:38:57.398 INFO:teuthology.orchestra.run.smithi028.stdout: "device_class_flags": {}, 2024-09-18T04:38:57.398 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": { 2024-09-18T04:38:57.398 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_enabled": false, 2024-09-18T04:38:57.398 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_bucket_count": 0, 2024-09-18T04:38:57.398 INFO:teuthology.orchestra.run.smithi028.stdout: "degraded_stretch_mode": 0, 2024-09-18T04:38:57.398 INFO:teuthology.orchestra.run.smithi028.stdout: "recovering_stretch_mode": 0, 2024-09-18T04:38:57.398 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_bucket": 0 2024-09-18T04:38:57.399 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:57.399 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:57.401 INFO:tasks.thrashosds.thrasher:j is {'epoch': 60, 'fsid': 'be6146ba-a991-4f6e-b359-99cbedc9f9b1', 'created': '2024-09-18T04:37:15.676634+0000', 'modified': '2024-09-18T04:38:55.424447+0000', 'last_up_change': '2024-09-18T04:37:25.403175+0000', 'last_in_change': '2024-09-18T04:37:50.652169+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-18T04:37:25.654950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-18T04:37:32.672462+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 20, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.5099999904632568, 'score_stable': 1.5099999904632568, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'new_default_pool17342', 'create_time': '2024-09-18T04:38:42.986132+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 55, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2.2699999809265137, 'score_stable': 2.2699999809265137, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '6177002d-72e6-4888-bcde-29b6a92d9dc4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 52, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6801', 'nonce': 1554407851}, {'type': 'v1', 'addr': '172.21.15.28:6803', 'nonce': 1554407851}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6805', 'nonce': 1554407851}, {'type': 'v1', 'addr': '172.21.15.28:6807', 'nonce': 1554407851}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6813', 'nonce': 1554407851}, {'type': 'v1', 'addr': '172.21.15.28:6815', 'nonce': 1554407851}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6809', 'nonce': 1554407851}, {'type': 'v1', 'addr': '172.21.15.28:6811', 'nonce': 1554407851}]}, 'public_addr': '172.21.15.28:6803/1554407851', 'cluster_addr': '172.21.15.28:6807/1554407851', 'heartbeat_back_addr': '172.21.15.28:6815/1554407851', 'heartbeat_front_addr': '172.21.15.28:6811/1554407851', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '4f62cb09-3457-418c-bb69-491d9b70914c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 52, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6800', 'nonce': 511141401}, {'type': 'v1', 'addr': '172.21.15.28:6802', 'nonce': 511141401}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6804', 'nonce': 511141401}, {'type': 'v1', 'addr': '172.21.15.28:6806', 'nonce': 511141401}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6812', 'nonce': 511141401}, {'type': 'v1', 'addr': '172.21.15.28:6814', 'nonce': 511141401}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6808', 'nonce': 511141401}, {'type': 'v1', 'addr': '172.21.15.28:6810', 'nonce': 511141401}]}, 'public_addr': '172.21.15.28:6802/511141401', 'cluster_addr': '172.21.15.28:6806/511141401', 'heartbeat_back_addr': '172.21.15.28:6814/511141401', 'heartbeat_front_addr': '172.21.15.28:6810/511141401', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '854fe031-3b87-4290-9c2f-3fb196fb806c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 52, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6816', 'nonce': 2623426735}, {'type': 'v1', 'addr': '172.21.15.28:6817', 'nonce': 2623426735}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6818', 'nonce': 2623426735}, {'type': 'v1', 'addr': '172.21.15.28:6819', 'nonce': 2623426735}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6822', 'nonce': 2623426735}, {'type': 'v1', 'addr': '172.21.15.28:6823', 'nonce': 2623426735}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6820', 'nonce': 2623426735}, {'type': 'v1', 'addr': '172.21.15.28:6821', 'nonce': 2623426735}]}, 'public_addr': '172.21.15.28:6817/2623426735', 'cluster_addr': '172.21.15.28:6819/2623426735', 'heartbeat_back_addr': '172.21.15.28:6823/2623426735', 'heartbeat_front_addr': '172.21.15.28:6821/2623426735', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'c2625f98-780b-4f86-bf36-a48d2a227756', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 32, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6808', 'nonce': 2746051132}, {'type': 'v1', 'addr': '172.21.15.82:6809', 'nonce': 2746051132}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6810', 'nonce': 2746051132}, {'type': 'v1', 'addr': '172.21.15.82:6811', 'nonce': 2746051132}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6814', 'nonce': 2746051132}, {'type': 'v1', 'addr': '172.21.15.82:6815', 'nonce': 2746051132}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6812', 'nonce': 2746051132}, {'type': 'v1', 'addr': '172.21.15.82:6813', 'nonce': 2746051132}]}, 'public_addr': '172.21.15.82:6809/2746051132', 'cluster_addr': '172.21.15.82:6811/2746051132', 'heartbeat_back_addr': '172.21.15.82:6815/2746051132', 'heartbeat_front_addr': '172.21.15.82:6813/2746051132', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'deb22d86-4081-484d-96eb-9cb9d96b60bc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 52, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6800', 'nonce': 2005712460}, {'type': 'v1', 'addr': '172.21.15.82:6801', 'nonce': 2005712460}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6802', 'nonce': 2005712460}, {'type': 'v1', 'addr': '172.21.15.82:6803', 'nonce': 2005712460}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6806', 'nonce': 2005712460}, {'type': 'v1', 'addr': '172.21.15.82:6807', 'nonce': 2005712460}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6804', 'nonce': 2005712460}, {'type': 'v1', 'addr': '172.21.15.82:6805', 'nonce': 2005712460}]}, 'public_addr': '172.21.15.82:6801/2005712460', 'cluster_addr': '172.21.15.82:6803/2005712460', 'heartbeat_back_addr': '172.21.15.82:6807/2005712460', 'heartbeat_front_addr': '172.21.15.82:6805/2005712460', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '061d332c-f51b-403d-bb18-eba415c97e7f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 52, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6816', 'nonce': 1267644232}, {'type': 'v1', 'addr': '172.21.15.82:6817', 'nonce': 1267644232}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6818', 'nonce': 1267644232}, {'type': 'v1', 'addr': '172.21.15.82:6819', 'nonce': 1267644232}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6822', 'nonce': 1267644232}, {'type': 'v1', 'addr': '172.21.15.82:6823', 'nonce': 1267644232}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6820', 'nonce': 1267644232}, {'type': 'v1', 'addr': '172.21.15.82:6821', 'nonce': 1267644232}]}, 'public_addr': '172.21.15.82:6817/1267644232', 'cluster_addr': '172.21.15.82:6819/1267644232', 'heartbeat_back_addr': '172.21.15.82:6823/1267644232', 'heartbeat_front_addr': '172.21.15.82:6821/1267644232', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T04:37:22.192087+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T04:37:22.017833+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T04:37:22.755774+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T04:37:22.663785+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T04:37:22.531758+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T04:37:22.400774+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.5', 'osds': [2, 0]}, {'pgid': '2.7', 'osds': [3, 5]}], 'pg_upmap_items': [{'pgid': '2.1', 'mappings': [{'from': 2, 'to': 1}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [{'pool': 3, 'snaps': [{'begin': 2, 'length': 1}]}], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-18T04:38:57.402 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T04:38:57.404 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:57.404 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:57.404 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:38:57.405 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:57.393+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:57.453 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-09-18T04:38:57.501 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:57.497+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:57.548 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:57.548 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:57.559 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:38:57.602 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:57.597+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:57.621 INFO:tasks.workunit.client.0.smithi115.stderr:++ _sudo rbd device -t nbd -o try-netlink map rbd/testrbdnbd17342 2024-09-18T04:38:57.621 INFO:tasks.workunit.client.0.smithi115.stderr:++ local cmd 2024-09-18T04:38:57.621 INFO:tasks.workunit.client.0.smithi115.stderr:+++ id -u 2024-09-18T04:38:57.623 INFO:tasks.workunit.client.0.smithi115.stderr:++ '[' 1000 -eq 0 ']' 2024-09-18T04:38:57.624 INFO:tasks.workunit.client.0.smithi115.stderr:+++ which rbd 2024-09-18T04:38:57.626 INFO:tasks.workunit.client.0.smithi115.stderr:++ cmd=/usr/bin/rbd 2024-09-18T04:38:57.626 INFO:tasks.workunit.client.0.smithi115.stderr:++ shift 2024-09-18T04:38:57.627 INFO:tasks.workunit.client.0.smithi115.stderr:++ sudo -nE /usr/bin/rbd device -t nbd -o try-netlink map rbd/testrbdnbd17342 2024-09-18T04:38:57.659 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:57.660 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:57.669 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:38:57.702 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:57.697+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:57.745 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:38:57.748 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-18T04:38:57.761 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":102,"stamp":"2024-09-18T04:38:57.615406+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":134808242,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":1622,"num_read_kb":145841,"num_write":421,"num_write_kb":199102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":368,"ondisk_log_size":368,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":457028,"kb_used_data":266232,"kb_used_omap":9,"kb_used_meta":190710,"kb_avail":565774012,"statfs":{"total":579820584960,"available":579352588288,"internally_reserved":0,"allocated":272621568,"data_stored":271098254,"data_compressed":66040,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":9561,"internal_metadata":195287719},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":739,"num_read_kb":124769,"num_write":139,"num_write_kb":63360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.618291"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":202,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T04:38:52.054428+0000","last_change":"2024-09-18T04:37:27.423069+0000","last_active":"2024-09-18T04:38:52.054428+0000","last_peered":"2024-09-18T04:38:52.054428+0000","last_clean":"2024-09-18T04:38:52.054428+0000","last_became_active":"2024-09-18T04:37:27.422350+0000","last_became_peered":"2024-09-18T04:37:27.422350+0000","last_unstale":"2024-09-18T04:38:52.054428+0000","last_undegraded":"2024-09-18T04:38:52.054428+0000","last_fullsized":"2024-09-18T04:38:52.054428+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:26.406292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:26.406292+0000","last_clean_scrub_stamp":"2024-09-18T04:37:26.406292+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:31:45.852149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"57'39","reported_seq":219,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T04:38:55.400376+0000","last_change":"2024-09-18T04:37:51.681357+0000","last_active":"2024-09-18T04:38:55.400376+0000","last_peered":"2024-09-18T04:38:55.400376+0000","last_clean":"2024-09-18T04:38:55.400376+0000","last_became_active":"2024-09-18T04:37:51.676131+0000","last_became_peered":"2024-09-18T04:37:51.676131+0000","last_unstale":"2024-09-18T04:38:55.400376+0000","last_undegraded":"2024-09-18T04:38:55.400376+0000","last_fullsized":"2024-09-18T04:38:55.400376+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:45:22.605009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018293420000000001,"stat_sum":{"num_bytes":8388662,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":217,"num_read_kb":16758,"num_write":54,"num_write_kb":16418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"55'2","reported_seq":21,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T04:38:52.000634+0000","last_change":"2024-09-18T04:38:46.966799+0000","last_active":"2024-09-18T04:38:52.000634+0000","last_peered":"2024-09-18T04:38:52.000634+0000","last_clean":"2024-09-18T04:38:52.000634+0000","last_became_active":"2024-09-18T04:38:44.953092+0000","last_became_peered":"2024-09-18T04:38:44.953092+0000","last_unstale":"2024-09-18T04:38:52.000634+0000","last_undegraded":"2024-09-18T04:38:52.000634+0000","last_fullsized":"2024-09-18T04:38:52.000634+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_clean_scrub_stamp":"2024-09-18T04:38:43.928423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:04:46.888906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010424099999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"58'51","reported_seq":377,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T04:38:53.303554+0000","last_change":"2024-09-18T04:37:36.484950+0000","last_active":"2024-09-18T04:38:53.303554+0000","last_peered":"2024-09-18T04:38:53.303554+0000","last_clean":"2024-09-18T04:38:53.303554+0000","last_became_active":"2024-09-18T04:37:34.470841+0000","last_became_peered":"2024-09-18T04:37:34.470841+0000","last_unstale":"2024-09-18T04:38:53.303554+0000","last_undegraded":"2024-09-18T04:38:53.303554+0000","last_fullsized":"2024-09-18T04:38:53.303554+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:16:34.528783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090513900000000001,"stat_sum":{"num_bytes":20971558,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":377,"num_read_kb":27327,"num_write":66,"num_write_kb":32801,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T04:38:55.431970+0000","last_change":"2024-09-18T04:38:47.337598+0000","last_active":"2024-09-18T04:38:55.431970+0000","last_peered":"2024-09-18T04:38:55.431970+0000","last_clean":"2024-09-18T04:38:55.431970+0000","last_became_active":"2024-09-18T04:38:44.952820+0000","last_became_peered":"2024-09-18T04:38:44.952820+0000","last_unstale":"2024-09-18T04:38:55.431970+0000","last_undegraded":"2024-09-18T04:38:55.431970+0000","last_fullsized":"2024-09-18T04:38:55.431970+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_clean_scrub_stamp":"2024-09-18T04:38:43.928423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:10:04.992053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075989799999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"58'44","reported_seq":441,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T04:38:53.295438+0000","last_change":"2024-09-18T04:37:51.679120+0000","last_active":"2024-09-18T04:38:53.295438+0000","last_peered":"2024-09-18T04:38:53.295438+0000","last_clean":"2024-09-18T04:38:53.295438+0000","last_became_active":"2024-09-18T04:37:51.678711+0000","last_became_peered":"2024-09-18T04:37:51.678711+0000","last_unstale":"2024-09-18T04:38:53.295438+0000","last_undegraded":"2024-09-18T04:38:53.295438+0000","last_fullsized":"2024-09-18T04:38:53.295438+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T04:46:09.459905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":510,"num_read_kb":11232,"num_write":28,"num_write_kb":16385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":19,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T04:38:52.000353+0000","last_change":"2024-09-18T04:38:46.967459+0000","last_active":"2024-09-18T04:38:52.000353+0000","last_peered":"2024-09-18T04:38:52.000353+0000","last_clean":"2024-09-18T04:38:52.000353+0000","last_became_active":"2024-09-18T04:38:44.950578+0000","last_became_peered":"2024-09-18T04:38:44.950578+0000","last_unstale":"2024-09-18T04:38:52.000353+0000","last_undegraded":"2024-09-18T04:38:52.000353+0000","last_fullsized":"2024-09-18T04:38:52.000353+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_clean_scrub_stamp":"2024-09-18T04:38:43.928423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:45:13.410610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011574549999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"20'2","reported_seq":79,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T04:38:55.411091+0000","last_change":"2024-09-18T04:37:36.490461+0000","last_active":"2024-09-18T04:38:55.411091+0000","last_peered":"2024-09-18T04:38:55.411091+0000","last_clean":"2024-09-18T04:38:55.411091+0000","last_became_active":"2024-09-18T04:37:34.473088+0000","last_became_peered":"2024-09-18T04:37:34.473088+0000","last_unstale":"2024-09-18T04:38:55.411091+0000","last_undegraded":"2024-09-18T04:38:55.411091+0000","last_fullsized":"2024-09-18T04:38:55.411091+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:17:57.352397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001879932,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"53'1","reported_seq":22,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T04:38:55.573213+0000","last_change":"2024-09-18T04:38:47.545918+0000","last_active":"2024-09-18T04:38:55.573213+0000","last_peered":"2024-09-18T04:38:55.573213+0000","last_clean":"2024-09-18T04:38:55.573213+0000","last_became_active":"2024-09-18T04:38:44.950045+0000","last_became_peered":"2024-09-18T04:38:44.950045+0000","last_unstale":"2024-09-18T04:38:55.573213+0000","last_undegraded":"2024-09-18T04:38:55.573213+0000","last_fullsized":"2024-09-18T04:38:55.573213+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_clean_scrub_stamp":"2024-09-18T04:38:43.928423+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:25:54.288210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010091900000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T04:38:55.430928+0000","last_change":"2024-09-18T04:38:46.974127+0000","last_active":"2024-09-18T04:38:55.430928+0000","last_peered":"2024-09-18T04:38:55.430928+0000","last_clean":"2024-09-18T04:38:55.430928+0000","last_became_active":"2024-09-18T04:38:44.949428+0000","last_became_peered":"2024-09-18T04:38:44.949428+0000","last_unstale":"2024-09-18T04:38:55.430928+0000","last_undegraded":"2024-09-18T04:38:55.430928+0000","last_fullsized":"2024-09-18T04:38:55.430928+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_clean_scrub_stamp":"2024-09-18T04:38:43.928423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:24:58.364051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065509499999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"57'23","reported_seq":150,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T04:38:53.196236+0000","last_change":"2024-09-18T04:37:36.494228+0000","last_active":"2024-09-18T04:38:53.196236+0000","last_peered":"2024-09-18T04:38:53.196236+0000","last_clean":"2024-09-18T04:38:53.196236+0000","last_became_active":"2024-09-18T04:37:34.471265+0000","last_became_peered":"2024-09-18T04:37:34.471265+0000","last_unstale":"2024-09-18T04:38:53.196236+0000","last_undegraded":"2024-09-18T04:38:53.196236+0000","last_fullsized":"2024-09-18T04:38:53.196236+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:23:05.748076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081514499999999998,"stat_sum":{"num_bytes":20971520,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":75,"num_read_kb":16645,"num_write":23,"num_write_kb":28674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":19,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T04:38:52.000842+0000","last_change":"2024-09-18T04:38:46.967504+0000","last_active":"2024-09-18T04:38:52.000842+0000","last_peered":"2024-09-18T04:38:52.000842+0000","last_clean":"2024-09-18T04:38:52.000842+0000","last_became_active":"2024-09-18T04:38:44.950268+0000","last_became_peered":"2024-09-18T04:38:44.950268+0000","last_unstale":"2024-09-18T04:38:52.000842+0000","last_undegraded":"2024-09-18T04:38:52.000842+0000","last_fullsized":"2024-09-18T04:38:52.000842+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_clean_scrub_stamp":"2024-09-18T04:38:43.928423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:07:16.193764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016757619999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"57'30","reported_seq":170,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T04:38:52.057542+0000","last_change":"2024-09-18T04:37:59.284861+0000","last_active":"2024-09-18T04:38:52.057542+0000","last_peered":"2024-09-18T04:38:52.057542+0000","last_clean":"2024-09-18T04:38:52.057542+0000","last_became_active":"2024-09-18T04:37:59.283691+0000","last_became_peered":"2024-09-18T04:37:59.283691+0000","last_unstale":"2024-09-18T04:38:52.057542+0000","last_undegraded":"2024-09-18T04:38:52.057542+0000","last_fullsized":"2024-09-18T04:38:52.057542+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:19:36.536686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":100,"num_read_kb":28672,"num_write":30,"num_write_kb":36864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":19,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T04:38:52.003373+0000","last_change":"2024-09-18T04:38:46.969605+0000","last_active":"2024-09-18T04:38:52.003373+0000","last_peered":"2024-09-18T04:38:52.003373+0000","last_clean":"2024-09-18T04:38:52.003373+0000","last_became_active":"2024-09-18T04:38:44.948788+0000","last_became_peered":"2024-09-18T04:38:44.948788+0000","last_unstale":"2024-09-18T04:38:52.003373+0000","last_undegraded":"2024-09-18T04:38:52.003373+0000","last_fullsized":"2024-09-18T04:38:52.003373+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_clean_scrub_stamp":"2024-09-18T04:38:43.928423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:18:24.302150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014497589999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"57'34","reported_seq":190,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T04:38:52.060709+0000","last_change":"2024-09-18T04:37:36.494193+0000","last_active":"2024-09-18T04:38:52.060709+0000","last_peered":"2024-09-18T04:38:52.060709+0000","last_clean":"2024-09-18T04:38:52.060709+0000","last_became_active":"2024-09-18T04:37:34.472940+0000","last_became_peered":"2024-09-18T04:37:34.472940+0000","last_unstale":"2024-09-18T04:38:52.060709+0000","last_undegraded":"2024-09-18T04:38:52.060709+0000","last_fullsized":"2024-09-18T04:38:52.060709+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:18:29.266815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070553900000000004,"stat_sum":{"num_bytes":33554432,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":105,"num_read_kb":28676,"num_write":34,"num_write_kb":45058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T04:38:55.432487+0000","last_change":"2024-09-18T04:38:47.337380+0000","last_active":"2024-09-18T04:38:55.432487+0000","last_peered":"2024-09-18T04:38:55.432487+0000","last_clean":"2024-09-18T04:38:55.432487+0000","last_became_active":"2024-09-18T04:38:44.948732+0000","last_became_peered":"2024-09-18T04:38:44.948732+0000","last_unstale":"2024-09-18T04:38:55.432487+0000","last_undegraded":"2024-09-18T04:38:55.432487+0000","last_fullsized":"2024-09-18T04:38:55.432487+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_clean_scrub_stamp":"2024-09-18T04:38:43.928423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:07:02.430621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058611500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"57'25","reported_seq":168,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T04:38:52.054388+0000","last_change":"2024-09-18T04:38:05.754820+0000","last_active":"2024-09-18T04:38:52.054388+0000","last_peered":"2024-09-18T04:38:52.054388+0000","last_clean":"2024-09-18T04:38:52.054388+0000","last_became_active":"2024-09-18T04:38:05.753962+0000","last_became_peered":"2024-09-18T04:38:05.753962+0000","last_unstale":"2024-09-18T04:38:52.054388+0000","last_undegraded":"2024-09-18T04:38:52.054388+0000","last_fullsized":"2024-09-18T04:38:52.054388+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:53:52.224098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047343899999999998,"stat_sum":{"num_bytes":12582928,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":99,"num_read_kb":16408,"num_write":23,"num_write_kb":20482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":5},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":134217855,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":1506,"num_read_kb":145741,"num_write":260,"num_write_kb":196684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":268476416,"data_stored":268435710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":248,"ondisk_log_size":248,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":66040,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":13,"seq":55834574870,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48184,"kb_used_data":21164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94323656,"statfs":{"total":96636764160,"available":96587423744,"internally_reserved":0,"allocated":21671936,"data_stored":21223669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:38:26 2024","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.873,"15min":0.873},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.5249999999999999,"5min":1.5249999999999999,"15min":1.5249999999999999},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.369,"5min":1.369,"15min":1.369},"last":0.59099999999999997}]},{"osd":1,"last update":"Wed Sep 18 04:38:26 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.68100000000000005}]},{"osd":2,"last update":"Wed Sep 18 04:38:26 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.4930000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.83799999999999997}]},{"osd":3,"last update":"Wed Sep 18 04:38:26 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.22,"5min":1.22,"15min":1.22},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.109,"5min":1.109,"15min":1.109},"last":0.60999999999999999}]},{"osd":4,"last update":"Wed Sep 18 04:38:26 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.121,"5min":1.121,"15min":1.121},"last":0.88500000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574870,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27672,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344168,"statfs":{"total":96636764160,"available":96608428032,"internally_reserved":0,"allocated":667648,"data_stored":246957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.175,"5min":1.175,"15min":1.175},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.258,"5min":1.258,"15min":1.258},"last":0.93000000000000005}]},{"osd":1,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.89800000000000002}]},{"osd":2,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.619,"5min":1.619,"15min":1.619},"last":0.86499999999999999}]},{"osd":3,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.94799999999999995},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.67400000000000004}]},{"osd":5,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.99399999999999999},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.214,"5min":1.214,"15min":1.214},"last":0.77700000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574871,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80984,"kb_used_data":45772,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94290856,"statfs":{"total":96636764160,"available":96553836544,"internally_reserved":0,"allocated":46870528,"data_stored":46974669,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 04:38:28 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.113,"5min":1.113,"15min":1.113},"last":0.626},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.45000000000000001}]},{"osd":2,"last update":"Wed Sep 18 04:38:28 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.8180000000000001,"5min":1.8180000000000001,"15min":1.8180000000000001},"last":0.71399999999999997}]},{"osd":3,"last update":"Wed Sep 18 04:38:28 2024","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":1.169,"5min":1.169,"15min":1.169},"last":0.753},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.83999999999999997}]},{"osd":4,"last update":"Wed Sep 18 04:38:28 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.91700000000000004},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.85999999999999999}]},{"osd":5,"last update":"Wed Sep 18 04:38:28 2024","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.8919999999999999,"5min":1.8919999999999999,"15min":1.8919999999999999},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":1.224,"5min":1.224,"15min":1.224},"last":0.96099999999999997}]}]},{"osd":3,"up_from":12,"seq":51539607574,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":97380,"kb_used_data":66260,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94274460,"statfs":{"total":96636764160,"available":96537047040,"internally_reserved":0,"allocated":67850240,"data_stored":67946243,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":2.665,"5min":2.665,"15min":2.665},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.89000000000000001}]},{"osd":1,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":2.4510000000000001,"5min":2.4510000000000001,"15min":2.4510000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.63400000000000001}]},{"osd":2,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":2.577,"5min":2.577,"15min":2.577},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":1.181}]},{"osd":4,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":1.0209999999999999}]},{"osd":5,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":1.069},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.93899999999999995}]}]},{"osd":2,"up_from":12,"seq":51539607572,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80932,"kb_used_data":45716,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94290908,"statfs":{"total":96636764160,"available":96553889792,"internally_reserved":0,"allocated":46813184,"data_stored":46384355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.82399999999999995}]},{"osd":1,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.78500000000000003}]},{"osd":3,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.753},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.68000000000000005}]},{"osd":4,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.91500000000000004},{"interface":"front","average":{"1min":0.879,"5min":0.879,"15min":0.879},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.64200000000000002}]},{"osd":5,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.88300000000000001},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.70599999999999996}]}]},{"osd":1,"up_from":12,"seq":51539607574,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":121876,"kb_used_data":86668,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94249964,"statfs":{"total":96636764160,"available":96511963136,"internally_reserved":0,"allocated":88748032,"data_stored":88322361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.67000000000000004}]},{"osd":2,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.327,"5min":1.327,"15min":1.327},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.56200000000000006}]},{"osd":3,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.121,"5min":1.121,"15min":1.121},"last":0.92200000000000004},{"interface":"front","average":{"1min":0.873,"5min":0.873,"15min":0.873},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":1.0800000000000001}]},{"osd":4,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.117,"5min":1.117,"15min":1.117},"last":1.0289999999999999}]},{"osd":5,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":1.3859999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.63700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":46137344,"data_stored":46137344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":88088576,"data_stored":88080441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":46145536,"data_stored":46137398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":67117056,"data_stored":67108918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20987904,"data_stored":20971609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T04:38:57.763 INFO:tasks.thrashosds.thrasher:Setting 2.4 to [2, 4, 1, 5] 2024-09-18T04:38:57.763 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.4', '2', '4', '1', '5'] 2024-09-18T04:38:57.763 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 2.4 2 4 1 5 2024-09-18T04:38:57.775 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:57.775 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:57.785 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:38:57.796 INFO:tasks.workunit.client.0.smithi115.stderr:+ DEV=/dev/nbd0 2024-09-18T04:38:57.796 INFO:tasks.workunit.client.0.smithi115.stderr:+ get_pid rbd 2024-09-18T04:38:57.796 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pool=rbd 2024-09-18T04:38:57.796 INFO:tasks.workunit.client.0.smithi115.stderr:+ local ns= 2024-09-18T04:38:57.797 INFO:tasks.workunit.client.0.smithi115.stderr:++ rbd device --device-type nbd --format xml list 2024-09-18T04:38:57.797 INFO:tasks.workunit.client.0.smithi115.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd17342'\''][device='\''/dev/nbd0'\'']/id' 2024-09-18T04:38:57.803 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:57.797+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:57.879 INFO:tasks.workunit.client.0.smithi115.stderr:+ PID=18167 2024-09-18T04:38:57.880 INFO:tasks.workunit.client.0.smithi115.stderr:+ test -n 18167 2024-09-18T04:38:57.880 INFO:tasks.workunit.client.0.smithi115.stderr:+ ps -p 18167 -C rbd-nbd 2024-09-18T04:38:57.886 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:57.886 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:57.890 INFO:tasks.workunit.client.0.smithi115.stdout: PID TTY TIME CMD 2024-09-18T04:38:57.890 INFO:tasks.workunit.client.0.smithi115.stdout: 18167 ? 00:00:00 rbd-nbd 2024-09-18T04:38:57.891 INFO:tasks.workunit.client.0.smithi115.stderr:++ basename /dev/nbd0 2024-09-18T04:38:57.892 INFO:tasks.workunit.client.0.smithi115.stderr:+ devname=nbd0 2024-09-18T04:38:57.893 INFO:tasks.workunit.client.0.smithi115.stderr:++ awk -v dev=nbd0 '$4 == dev {print $3}' /proc/partitions 2024-09-18T04:38:57.896 INFO:tasks.workunit.client.0.smithi115.stderr:+ blocks=65536 2024-09-18T04:38:57.896 INFO:tasks.workunit.client.0.smithi115.stderr:+ test -n 65536 2024-09-18T04:38:57.896 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd resize rbd/testrbdnbd17342 --size 128M 2024-09-18T04:38:57.897 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:38:57.904 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:57.897+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:57.974 INFO:tasks.workunit.client.0.smithi115.stderr: Resizing image: 100% complete...done. 2024-09-18T04:38:57.989 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd info rbd/testrbdnbd17342 2024-09-18T04:38:58.005 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:58.001+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:58.009 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:58.009 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:58.019 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:38:58.044 INFO:tasks.workunit.client.0.smithi115.stdout:rbd image 'testrbdnbd17342': 2024-09-18T04:38:58.044 INFO:tasks.workunit.client.0.smithi115.stdout: size 128 MiB in 32 objects 2024-09-18T04:38:58.045 INFO:tasks.workunit.client.0.smithi115.stdout: order 22 (4 MiB objects) 2024-09-18T04:38:58.045 INFO:tasks.workunit.client.0.smithi115.stdout: snapshot_count: 0 2024-09-18T04:38:58.045 INFO:tasks.workunit.client.0.smithi115.stdout: id: 125fc09df6d1 2024-09-18T04:38:58.045 INFO:tasks.workunit.client.0.smithi115.stdout: block_name_prefix: rbd_data.125fc09df6d1 2024-09-18T04:38:58.045 INFO:tasks.workunit.client.0.smithi115.stdout: format: 2 2024-09-18T04:38:58.045 INFO:tasks.workunit.client.0.smithi115.stdout: features: layering, exclusive-lock, object-map, fast-diff, deep-flatten 2024-09-18T04:38:58.045 INFO:tasks.workunit.client.0.smithi115.stdout: op_features: 2024-09-18T04:38:58.045 INFO:tasks.workunit.client.0.smithi115.stdout: flags: 2024-09-18T04:38:58.045 INFO:tasks.workunit.client.0.smithi115.stdout: create_timestamp: Wed Sep 18 04:38:42 2024 2024-09-18T04:38:58.045 INFO:tasks.workunit.client.0.smithi115.stdout: access_timestamp: Wed Sep 18 04:38:42 2024 2024-09-18T04:38:58.045 INFO:tasks.workunit.client.0.smithi115.stdout: modify_timestamp: Wed Sep 18 04:38:42 2024 2024-09-18T04:38:58.051 INFO:tasks.workunit.client.0.smithi115.stderr:++ awk -v dev=nbd0 '$4 == dev {print $3}' /proc/partitions 2024-09-18T04:38:58.054 INFO:tasks.workunit.client.0.smithi115.stderr:+ blocks2=131072 2024-09-18T04:38:58.054 INFO:tasks.workunit.client.0.smithi115.stderr:+ test -n 131072 2024-09-18T04:38:58.054 INFO:tasks.workunit.client.0.smithi115.stderr:+ test 131072 -eq 131072 2024-09-18T04:38:58.054 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd resize rbd/testrbdnbd17342 --allow-shrink --size 64M 2024-09-18T04:38:58.107 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:58.101+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:58.124 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:58.124 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:58.133 INFO:tasks.workunit.client.0.smithi115.stderr: Resizing image: 50% complete... Resizing image: 53% complete... Resizing image: 56% complete... Resizing image: 59% complete... Resizing image: 62% complete... Resizing image: 65% complete... Resizing image: 68% complete... Resizing image: 71% complete... Resizing image: 75% complete... Resizing image: 78% complete... Resizing image: 81% complete... Resizing image: 84% complete... Resizing image: 87% complete... Resizing image: 90% complete... Resizing image: 93% complete... Resizing image: 96% complete... Resizing image: 100% complete...done. 2024-09-18T04:38:58.135 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:38:58.150 INFO:tasks.workunit.client.0.smithi115.stderr:++ awk -v dev=nbd0 '$4 == dev {print $3}' /proc/partitions 2024-09-18T04:38:58.154 INFO:tasks.workunit.client.0.smithi115.stderr:+ blocks2=65536 2024-09-18T04:38:58.154 INFO:tasks.workunit.client.0.smithi115.stderr:+ test -n 65536 2024-09-18T04:38:58.154 INFO:tasks.workunit.client.0.smithi115.stderr:+ test 65536 -eq 65536 2024-09-18T04:38:58.155 INFO:tasks.workunit.client.0.smithi115.stderr:+ unmap_device /dev/nbd0 18167 2024-09-18T04:38:58.155 INFO:tasks.workunit.client.0.smithi115.stderr:+ local args=/dev/nbd0 2024-09-18T04:38:58.155 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pid=18167 2024-09-18T04:38:58.155 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-18T04:38:58.155 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:38:58.155 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:38:58.157 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:38:58.157 INFO:tasks.workunit.client.0.smithi115.stderr:++ which rbd 2024-09-18T04:38:58.159 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/bin/rbd 2024-09-18T04:38:58.159 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:38:58.159 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-18T04:38:58.207 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:58.201+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:58.248 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:58.248 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:58.258 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:38:58.260 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd device --device-type nbd list 2024-09-18T04:38:58.260 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false grep '^18167\b' 2024-09-18T04:38:58.260 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep '^18167\b' 2024-09-18T04:38:58.308 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:58.301+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:58.323 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:38:58.324 INFO:tasks.workunit.client.0.smithi115.stderr:+ ps -C rbd-nbd 2024-09-18T04:38:58.324 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false grep '^ *18167\b' 2024-09-18T04:38:58.324 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep '^ *18167\b' 2024-09-18T04:38:58.334 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:38:58.334 INFO:tasks.workunit.client.0.smithi115.stderr:+ sleep 0.5 2024-09-18T04:38:58.363 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:58.363 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:58.373 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:38:58.409 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:58.401+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:58.459 INFO:teuthology.orchestra.run.smithi028.stderr:set 2.4 pg_upmap_items mapping to [2->4,1->5] 2024-09-18T04:38:58.510 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:58.505+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:58.532 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:58.532 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:58.542 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:38:58.612 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:58.605+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:38:58.645 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:58.645 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:58.654 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:38:58.713 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:58.705+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:58.762 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:58.762 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:58.771 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:38:58.813 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:58.805+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:58.836 INFO:tasks.workunit.client.0.smithi115.stderr:++ _sudo rbd --device-type nbd map --read-only rbd/testrbdnbd17342 2024-09-18T04:38:58.837 INFO:tasks.workunit.client.0.smithi115.stderr:++ local cmd 2024-09-18T04:38:58.837 INFO:tasks.workunit.client.0.smithi115.stderr:+++ id -u 2024-09-18T04:38:58.839 INFO:tasks.workunit.client.0.smithi115.stderr:++ '[' 1000 -eq 0 ']' 2024-09-18T04:38:58.839 INFO:tasks.workunit.client.0.smithi115.stderr:+++ which rbd 2024-09-18T04:38:58.842 INFO:tasks.workunit.client.0.smithi115.stderr:++ cmd=/usr/bin/rbd 2024-09-18T04:38:58.842 INFO:tasks.workunit.client.0.smithi115.stderr:++ shift 2024-09-18T04:38:58.842 INFO:tasks.workunit.client.0.smithi115.stderr:++ sudo -nE /usr/bin/rbd --device-type nbd map --read-only rbd/testrbdnbd17342 2024-09-18T04:38:58.876 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:58.876 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:58.886 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:38:58.914 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:38:58.909+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:38:58.993 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:58.993 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:59.001 INFO:tasks.workunit.client.0.smithi115.stderr:+ DEV=/dev/nbd0 2024-09-18T04:38:59.002 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:38:59.003 INFO:tasks.workunit.client.0.smithi115.stderr:++ rbd device --device-type nbd list 2024-09-18T04:38:59.003 INFO:tasks.workunit.client.0.smithi115.stderr:++ awk -v pool=rbd -v img=testrbdnbd17342 -v dev=/dev/nbd0 '$2 == pool && $3 == img && $5 == dev {print $1}' 2024-09-18T04:38:59.015 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:59.009+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:59.080 INFO:tasks.workunit.client.0.smithi115.stderr:+ PID=18295 2024-09-18T04:38:59.080 INFO:tasks.workunit.client.0.smithi115.stderr:+ test -n 18295 2024-09-18T04:38:59.080 INFO:tasks.workunit.client.0.smithi115.stderr:+ ps -p 18295 -C rbd-nbd 2024-09-18T04:38:59.089 INFO:tasks.workunit.client.0.smithi115.stdout: PID TTY TIME CMD 2024-09-18T04:38:59.089 INFO:tasks.workunit.client.0.smithi115.stdout: 18295 ? 00:00:00 rbd-nbd 2024-09-18T04:38:59.090 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo dd if=/dev/nbd0 of=/dev/null bs=1M 2024-09-18T04:38:59.090 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:38:59.090 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:38:59.091 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:38:59.091 INFO:tasks.workunit.client.0.smithi115.stderr:++ which dd 2024-09-18T04:38:59.093 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/bin/dd 2024-09-18T04:38:59.093 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:38:59.093 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/bin/dd if=/dev/nbd0 of=/dev/null bs=1M 2024-09-18T04:38:59.111 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:59.111 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:59.116 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:59.109+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:59.121 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:38:59.217 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:38:59.209+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:38:59.272 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:59.272 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:59.281 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:38:59.330 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:59.325+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:59.401 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T04:38:59.402 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:59.402 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:59.412 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:38:59.419 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:59.417+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:59.470 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-18T04:38:59.521 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:38:59.513+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:38:59.525 INFO:tasks.workunit.client.0.smithi115.stderr:64+0 records in 2024-09-18T04:38:59.525 INFO:tasks.workunit.client.0.smithi115.stderr:64+0 records out 2024-09-18T04:38:59.525 INFO:tasks.workunit.client.0.smithi115.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.421903 s, 159 MB/s 2024-09-18T04:38:59.526 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false _sudo dd if=/tmp/tmp.WJMNJxIdAN/data of=/dev/nbd0 bs=1M oflag=direct 2024-09-18T04:38:59.526 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo dd if=/tmp/tmp.WJMNJxIdAN/data of=/dev/nbd0 bs=1M oflag=direct 2024-09-18T04:38:59.526 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:38:59.526 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:38:59.528 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:38:59.529 INFO:tasks.workunit.client.0.smithi115.stderr:++ which dd 2024-09-18T04:38:59.531 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/bin/dd 2024-09-18T04:38:59.531 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:38:59.531 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.WJMNJxIdAN/data of=/dev/nbd0 bs=1M oflag=direct 2024-09-18T04:38:59.536 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:38:59.537 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:38:59.541 INFO:tasks.workunit.client.0.smithi115.stderr:/usr/bin/dd: error writing '/dev/nbd0': Operation not permitted 2024-09-18T04:38:59.542 INFO:tasks.workunit.client.0.smithi115.stderr:1+0 records in 2024-09-18T04:38:59.542 INFO:tasks.workunit.client.0.smithi115.stderr:0+0 records out 2024-09-18T04:38:59.542 INFO:tasks.workunit.client.0.smithi115.stderr:0 bytes copied, 0.00118125 s, 0.0 kB/s 2024-09-18T04:38:59.543 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:38:59.543 INFO:tasks.workunit.client.0.smithi115.stderr:+ unmap_device /dev/nbd0 18295 2024-09-18T04:38:59.543 INFO:tasks.workunit.client.0.smithi115.stderr:+ local args=/dev/nbd0 2024-09-18T04:38:59.544 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pid=18295 2024-09-18T04:38:59.545 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-18T04:38:59.545 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:38:59.545 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:38:59.545 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:38:59.546 INFO:tasks.workunit.client.0.smithi115.stderr:++ which rbd 2024-09-18T04:38:59.547 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:38:59.548 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/bin/rbd 2024-09-18T04:38:59.548 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:38:59.548 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-18T04:38:59.554 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:38:59.582 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:38:59.605 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:38:59.622 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:59.617+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:59.634 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:38:59.639 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd device --device-type nbd list 2024-09-18T04:38:59.639 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false grep '^18295\b' 2024-09-18T04:38:59.639 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep '^18295\b' 2024-09-18T04:38:59.662 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:38:59.664 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:59.664 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:38:59.664 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:38:59.664 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:38:59.664 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.664 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:37 2.5 e51/28 2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:head v 51'1, mlcod=0'0)", 2024-09-18T04:38:59.664 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:38:59.664 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.369314955, 2024-09-18T04:38:59.665 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012279973, 2024-09-18T04:38:59.665 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:59.665 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:59.665 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:59.665 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.665 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:59.665 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:38:59.665 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:59.665 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.665 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.665 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:59.665 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:38:59.665 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:59.666 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.666 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.666 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:59.666 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292468+0000", 2024-09-18T04:38:59.666 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:38:59.666 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.666 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.666 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:59.666 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295837+0000", 2024-09-18T04:38:59.666 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0033691699999999999 2024-09-18T04:38:59.666 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.666 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.666 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:59.667 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295860+0000", 2024-09-18T04:38:59.667 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2756999999999999e-05 2024-09-18T04:38:59.667 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.667 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.667 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:59.667 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295872+0000", 2024-09-18T04:38:59.667 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2442e-05 2024-09-18T04:38:59.667 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.667 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.667 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:59.667 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295934+0000", 2024-09-18T04:38:59.667 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1187999999999998e-05 2024-09-18T04:38:59.667 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.668 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.668 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:59.668 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295967+0000", 2024-09-18T04:38:59.668 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3185999999999997e-05 2024-09-18T04:38:59.668 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.668 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.668 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:59.668 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304697+0000", 2024-09-18T04:38:59.668 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0087298059999999997 2024-09-18T04:38:59.668 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.668 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.668 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:59.668 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304749+0000", 2024-09-18T04:38:59.668 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1922000000000002e-05 2024-09-18T04:38:59.669 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:59.669 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:59.669 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:59.669 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.669 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.669 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:49 2.5 e51/28 2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head v 51'2, mlcod=0'0)", 2024-09-18T04:38:59.669 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:38:59.669 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.335988116999999, 2024-09-18T04:38:59.669 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018033041999999999, 2024-09-18T04:38:59.669 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:59.669 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:59.669 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:59.669 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.670 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:59.670 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:38:59.670 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:59.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.670 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.670 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:59.670 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:38:59.670 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:59.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.670 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.670 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:59.670 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325793+0000", 2024-09-18T04:38:59.670 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:59.671 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.671 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.671 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:59.671 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329038+0000", 2024-09-18T04:38:59.671 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0032444010000000001 2024-09-18T04:38:59.671 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.671 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.671 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:59.671 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329065+0000", 2024-09-18T04:38:59.671 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7500000000000001e-05 2024-09-18T04:38:59.671 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.671 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.671 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:59.671 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329077+0000", 2024-09-18T04:38:59.671 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1919999999999999e-05 2024-09-18T04:38:59.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.672 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.672 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:59.672 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329149+0000", 2024-09-18T04:38:59.672 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2349999999999997e-05 2024-09-18T04:38:59.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.672 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.672 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:59.672 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329192+0000", 2024-09-18T04:38:59.672 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2744000000000003e-05 2024-09-18T04:38:59.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.672 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.672 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:59.672 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343773+0000", 2024-09-18T04:38:59.672 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014581246000000001 2024-09-18T04:38:59.673 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.673 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.673 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:59.673 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343829+0000", 2024-09-18T04:38:59.673 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5143999999999999e-05 2024-09-18T04:38:59.673 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:59.673 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:59.673 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:59.673 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.673 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.673 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:47 2.4 e51/17 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 51'3, mlcod=51'1)", 2024-09-18T04:38:59.673 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:38:59.673 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.335407851999999, 2024-09-18T04:38:59.674 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018960186, 2024-09-18T04:38:59.674 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:59.674 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:59.674 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:59.674 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.674 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:59.674 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:38:59.674 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:59.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.674 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.674 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:59.674 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:38:59.674 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:59.675 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.675 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.675 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:59.675 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326374+0000", 2024-09-18T04:38:59.675 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:59.675 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.675 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.675 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:59.675 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330883+0000", 2024-09-18T04:38:59.675 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0045084740000000002 2024-09-18T04:38:59.675 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.675 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.675 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:59.675 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330901+0000", 2024-09-18T04:38:59.675 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7890999999999999e-05 2024-09-18T04:38:59.676 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.676 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.676 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:59.676 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330912+0000", 2024-09-18T04:38:59.676 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1041e-05 2024-09-18T04:38:59.676 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.676 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.676 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:59.676 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.331003+0000", 2024-09-18T04:38:59.676 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1323999999999995e-05 2024-09-18T04:38:59.676 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.676 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.676 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:59.677 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.331030+0000", 2024-09-18T04:38:59.677 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7351999999999998e-05 2024-09-18T04:38:59.677 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.677 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.677 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:59.677 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345287+0000", 2024-09-18T04:38:59.677 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014256332 2024-09-18T04:38:59.677 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.677 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.677 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:59.677 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345336+0000", 2024-09-18T04:38:59.677 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9356e-05 2024-09-18T04:38:59.677 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:59.677 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:59.678 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:59.678 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.678 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.678 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:58 2.5 e51/28 2:b227e6b1:::rbd_data.125fc09df6d1.000000000000000e:head v 51'3, mlcod=51'1)", 2024-09-18T04:38:59.678 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:38:59.678 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.294981225000001, 2024-09-18T04:38:59.678 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016513678, 2024-09-18T04:38:59.678 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:59.678 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:59.678 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:59.678 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.678 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:59.678 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:38:59.678 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:59.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.679 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.679 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:59.679 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:38:59.679 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:59.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.679 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.679 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:59.679 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366800+0000", 2024-09-18T04:38:59.679 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:38:59.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.679 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.679 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:59.679 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368644+0000", 2024-09-18T04:38:59.679 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001844399 2024-09-18T04:38:59.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.680 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.680 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:59.680 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368681+0000", 2024-09-18T04:38:59.680 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6526999999999998e-05 2024-09-18T04:38:59.680 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.680 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.680 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:59.680 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368698+0000", 2024-09-18T04:38:59.680 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7272999999999999e-05 2024-09-18T04:38:59.680 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.680 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.680 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:59.680 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368773+0000", 2024-09-18T04:38:59.681 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4862000000000003e-05 2024-09-18T04:38:59.681 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.681 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.681 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:59.681 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368801+0000", 2024-09-18T04:38:59.681 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7925000000000001e-05 2024-09-18T04:38:59.681 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.681 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.681 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:59.681 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383261+0000", 2024-09-18T04:38:59.681 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014460377999999999 2024-09-18T04:38:59.681 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.681 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.681 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:59.682 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383316+0000", 2024-09-18T04:38:59.682 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4623999999999997e-05 2024-09-18T04:38:59.682 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:59.682 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:59.682 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:59.682 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.682 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.682 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:56 2.4 e51/17 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 51'4, mlcod=51'2)", 2024-09-18T04:38:59.682 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:38:59.682 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.292977198999999, 2024-09-18T04:38:59.682 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015105221, 2024-09-18T04:38:59.682 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:59.682 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:59.682 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:59.682 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.683 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:59.683 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:38:59.683 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:59.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.683 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.683 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:59.683 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:38:59.683 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:59.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.683 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.683 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:59.683 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368805+0000", 2024-09-18T04:38:59.683 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:59.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.683 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.683 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:59.684 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371603+0000", 2024-09-18T04:38:59.684 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027982660000000002 2024-09-18T04:38:59.684 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.684 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.684 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:59.684 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371633+0000", 2024-09-18T04:38:59.684 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0073000000000002e-05 2024-09-18T04:38:59.684 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.684 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.684 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:59.684 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371652+0000", 2024-09-18T04:38:59.684 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8597000000000002e-05 2024-09-18T04:38:59.684 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.684 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.684 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:59.685 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371752+0000", 2024-09-18T04:38:59.685 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9772999999999995e-05 2024-09-18T04:38:59.685 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.685 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.685 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:59.685 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371793+0000", 2024-09-18T04:38:59.685 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1173e-05 2024-09-18T04:38:59.685 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.685 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.685 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:59.685 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383875+0000", 2024-09-18T04:38:59.685 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012081852000000001 2024-09-18T04:38:59.685 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.685 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.686 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:59.686 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383912+0000", 2024-09-18T04:38:59.686 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7045000000000002e-05 2024-09-18T04:38:59.686 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:59.686 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:59.686 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:59.686 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.686 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.686 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:36 2.5 e51/28 2:bd5b8dd8:ns::rbd_data.133e813ab5bc.0000000000000002:head v 51'4, mlcod=51'2)", 2024-09-18T04:38:59.686 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:38:59.686 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.124258208000001, 2024-09-18T04:38:59.686 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020758161000000001, 2024-09-18T04:38:59.686 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:59.686 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:59.686 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:59.687 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.687 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:59.687 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:38:59.687 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:59.687 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.687 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.687 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:59.687 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:38:59.687 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:59.687 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.687 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.687 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:59.687 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537523+0000", 2024-09-18T04:38:59.687 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:59.688 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.688 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.688 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:59.688 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541578+0000", 2024-09-18T04:38:59.688 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0040549710000000001 2024-09-18T04:38:59.688 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.688 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.688 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:59.688 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541608+0000", 2024-09-18T04:38:59.688 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9349e-05 2024-09-18T04:38:59.688 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.688 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.688 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:59.688 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541621+0000", 2024-09-18T04:38:59.689 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2736e-05 2024-09-18T04:38:59.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.689 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.689 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:59.689 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541715+0000", 2024-09-18T04:38:59.689 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4197999999999995e-05 2024-09-18T04:38:59.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.689 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.689 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:59.689 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541755+0000", 2024-09-18T04:38:59.689 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9818999999999999e-05 2024-09-18T04:38:59.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.689 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.689 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:59.690 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558217+0000", 2024-09-18T04:38:59.690 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016462234999999999 2024-09-18T04:38:59.690 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.690 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.690 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:59.690 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558284+0000", 2024-09-18T04:38:59.690 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6774999999999997e-05 2024-09-18T04:38:59.690 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:59.690 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:59.690 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:59.690 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.690 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.690 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:39 2.4 e51/17 2:304364d1:ns::rbd_data.133e813ab5bc.0000000000000004:head v 51'5, mlcod=51'3)", 2024-09-18T04:38:59.690 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:38:59.691 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.116119081000001, 2024-09-18T04:38:59.691 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016777558000000001, 2024-09-18T04:38:59.691 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:59.691 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:59.691 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:59.691 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.691 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:59.691 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:38:59.691 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:59.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.691 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.691 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:59.691 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:38:59.691 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:59.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.692 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.692 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:59.692 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545663+0000", 2024-09-18T04:38:59.692 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:59.692 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.692 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.692 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:59.692 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548286+0000", 2024-09-18T04:38:59.692 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026233390000000001 2024-09-18T04:38:59.692 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.692 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.692 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:59.692 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548305+0000", 2024-09-18T04:38:59.693 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8425000000000001e-05 2024-09-18T04:38:59.693 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.693 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.693 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:59.693 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548316+0000", 2024-09-18T04:38:59.693 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1163e-05 2024-09-18T04:38:59.693 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.693 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.693 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:59.693 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548379+0000", 2024-09-18T04:38:59.693 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2780000000000005e-05 2024-09-18T04:38:59.693 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.693 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.693 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:59.694 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548399+0000", 2024-09-18T04:38:59.694 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0080000000000001e-05 2024-09-18T04:38:59.694 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.694 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.694 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:59.694 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562383+0000", 2024-09-18T04:38:59.694 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013984671000000001 2024-09-18T04:38:59.694 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.694 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.694 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:59.694 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562442+0000", 2024-09-18T04:38:59.694 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8700999999999999e-05 2024-09-18T04:38:59.694 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:59.694 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:59.695 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:59.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.695 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.695 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:37 2.5 e51/28 2:bf39b8e4:ns::rbd_data.133e813ab5bc.0000000000000003:head v 51'5, mlcod=51'2)", 2024-09-18T04:38:59.695 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:38:59.695 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.113250812, 2024-09-18T04:38:59.695 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014264492, 2024-09-18T04:38:59.695 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:59.695 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:59.695 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:59.695 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.695 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:59.695 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:38:59.696 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:59.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.696 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.696 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:59.696 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:38:59.696 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:59.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.696 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.696 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:59.696 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548531+0000", 2024-09-18T04:38:59.696 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:59.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.696 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.696 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:59.697 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551500+0000", 2024-09-18T04:38:59.697 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0029688129999999998 2024-09-18T04:38:59.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.697 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.697 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:59.697 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551516+0000", 2024-09-18T04:38:59.697 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5994e-05 2024-09-18T04:38:59.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.697 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.697 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:59.697 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551526+0000", 2024-09-18T04:38:59.697 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9299999999999998e-06 2024-09-18T04:38:59.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.697 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.698 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:59.698 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551577+0000", 2024-09-18T04:38:59.698 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0871000000000001e-05 2024-09-18T04:38:59.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.698 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.698 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:59.698 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551602+0000", 2024-09-18T04:38:59.698 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4813000000000001e-05 2024-09-18T04:38:59.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.698 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.698 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:59.699 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562741+0000", 2024-09-18T04:38:59.699 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011139428999999999 2024-09-18T04:38:59.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.699 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.699 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:59.699 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562797+0000", 2024-09-18T04:38:59.699 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6329000000000002e-05 2024-09-18T04:38:59.699 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:59.699 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:59.699 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:59.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.699 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.700 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:49 2.4 e51/17 2:20f0b0c9:ns::rbd_data.133e813ab5bc.0000000000000009:head v 51'6, mlcod=51'4)", 2024-09-18T04:38:59.700 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:38:59.700 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.081816254, 2024-09-18T04:38:59.700 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013849046, 2024-09-18T04:38:59.700 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:59.700 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:59.700 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:59.700 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.700 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:59.700 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:38:59.700 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:59.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.701 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.701 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:59.701 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:38:59.701 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:59.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.701 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.701 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:59.701 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579966+0000", 2024-09-18T04:38:59.701 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:59.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.701 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.701 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:59.701 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582324+0000", 2024-09-18T04:38:59.701 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023578890000000002 2024-09-18T04:38:59.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.702 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.702 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:59.702 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582355+0000", 2024-09-18T04:38:59.702 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1485000000000001e-05 2024-09-18T04:38:59.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.702 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.702 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:59.702 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582371+0000", 2024-09-18T04:38:59.702 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6147999999999999e-05 2024-09-18T04:38:59.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.702 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.702 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:59.702 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582467+0000", 2024-09-18T04:38:59.703 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6059999999999998e-05 2024-09-18T04:38:59.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.703 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.703 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:59.703 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582505+0000", 2024-09-18T04:38:59.703 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7679000000000003e-05 2024-09-18T04:38:59.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.703 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.703 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:59.703 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593743+0000", 2024-09-18T04:38:59.703 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011238356 2024-09-18T04:38:59.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.703 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.703 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:59.704 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593816+0000", 2024-09-18T04:38:59.704 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2935000000000006e-05 2024-09-18T04:38:59.704 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:59.704 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:59.704 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:59.704 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.704 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.704 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:53 2.4 e51/17 2:39f7f174:ns::rbd_data.133e813ab5bc.000000000000000b:head v 51'7, mlcod=51'4)", 2024-09-18T04:38:59.704 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:38:59.704 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.067115098999999, 2024-09-18T04:38:59.704 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0079914159999999995, 2024-09-18T04:38:59.704 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:59.704 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:59.704 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:59.705 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.705 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:59.705 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:38:59.705 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:59.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.705 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.705 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:59.705 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:38:59.705 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:59.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.705 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.705 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:59.705 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594667+0000", 2024-09-18T04:38:59.706 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:59.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.706 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.706 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:59.706 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596446+0000", 2024-09-18T04:38:59.706 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001779033 2024-09-18T04:38:59.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.706 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.706 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:59.706 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596472+0000", 2024-09-18T04:38:59.706 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6120000000000001e-05 2024-09-18T04:38:59.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.706 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.707 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:59.707 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596500+0000", 2024-09-18T04:38:59.707 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7518e-05 2024-09-18T04:38:59.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.707 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.707 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:59.707 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596564+0000", 2024-09-18T04:38:59.707 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3880999999999994e-05 2024-09-18T04:38:59.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.707 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.707 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:59.707 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596596+0000", 2024-09-18T04:38:59.707 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2638999999999997e-05 2024-09-18T04:38:59.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.707 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.708 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:59.708 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602605+0000", 2024-09-18T04:38:59.708 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0060080699999999999 2024-09-18T04:38:59.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.708 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.708 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:59.708 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602660+0000", 2024-09-18T04:38:59.708 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5396000000000003e-05 2024-09-18T04:38:59.708 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:59.708 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:59.708 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:59.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.708 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.708 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:60 2.5 e51/28 2:a06f51e4:ns::rbd_data.133e813ab5bc.000000000000000f:head v 51'6, mlcod=51'4)", 2024-09-18T04:38:59.709 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:38:59.709 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.052298113999999, 2024-09-18T04:38:59.709 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0093954989999999999, 2024-09-18T04:38:59.709 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:59.709 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:59.709 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:59.709 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.709 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:59.709 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:38:59.709 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:59.709 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.709 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.709 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:59.709 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:38:59.710 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:59.710 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.710 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.710 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:59.710 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609483+0000", 2024-09-18T04:38:59.710 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:59.710 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.710 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.710 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:59.710 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611767+0000", 2024-09-18T04:38:59.710 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002283724 2024-09-18T04:38:59.710 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.710 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.710 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:59.710 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611791+0000", 2024-09-18T04:38:59.711 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3547e-05 2024-09-18T04:38:59.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.711 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.711 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:59.711 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611800+0000", 2024-09-18T04:38:59.711 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4979999999999995e-06 2024-09-18T04:38:59.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.711 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.711 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:59.711 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611864+0000", 2024-09-18T04:38:59.711 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3974999999999996e-05 2024-09-18T04:38:59.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.711 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.711 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:59.711 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611887+0000", 2024-09-18T04:38:59.712 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2633000000000002e-05 2024-09-18T04:38:59.712 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.712 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.712 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:59.712 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.618777+0000", 2024-09-18T04:38:59.712 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0068897979999999999 2024-09-18T04:38:59.712 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.712 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.712 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:59.712 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.618881+0000", 2024-09-18T04:38:59.712 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010437699999999999 2024-09-18T04:38:59.712 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:59.712 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:59.713 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:59.713 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.713 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.713 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:627 2.5 e57/28 2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:head v 57'8, mlcod=51'6)", 2024-09-18T04:38:59.713 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.163545+0000", 2024-09-18T04:38:59.713 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 8.4982386339999998, 2024-09-18T04:38:59.713 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0043737209999999997, 2024-09-18T04:38:59.713 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:59.713 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:59.713 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:59.713 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.713 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:59.713 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.163545+0000", 2024-09-18T04:38:59.713 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:59.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.714 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.714 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:59.714 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.163545+0000", 2024-09-18T04:38:59.714 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:59.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.714 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.714 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:59.714 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.163544+0000", 2024-09-18T04:38:59.714 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:38:59.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.714 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.714 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:59.714 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.164269+0000", 2024-09-18T04:38:59.714 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00072451899999999997 2024-09-18T04:38:59.715 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.715 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.715 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:59.715 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.164288+0000", 2024-09-18T04:38:59.715 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8824000000000001e-05 2024-09-18T04:38:59.715 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.715 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.715 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:59.715 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.164297+0000", 2024-09-18T04:38:59.715 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4229999999999994e-06 2024-09-18T04:38:59.715 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.715 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.715 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:59.715 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.164377+0000", 2024-09-18T04:38:59.715 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0007999999999997e-05 2024-09-18T04:38:59.715 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.716 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.716 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:59.716 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.164422+0000", 2024-09-18T04:38:59.716 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5222000000000002e-05 2024-09-18T04:38:59.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.716 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.716 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:59.716 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.167863+0000", 2024-09-18T04:38:59.716 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0034412050000000001 2024-09-18T04:38:59.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.716 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.716 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:59.716 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.167919+0000", 2024-09-18T04:38:59.716 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5296e-05 2024-09-18T04:38:59.717 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:59.717 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:59.717 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:59.717 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.717 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.717 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:629 2.5 e57/28 2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:head v 57'10, mlcod=57'8)", 2024-09-18T04:38:59.717 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.173967+0000", 2024-09-18T04:38:59.717 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 8.4878166559999997, 2024-09-18T04:38:59.717 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0043585999999999998, 2024-09-18T04:38:59.717 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:59.717 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:59.717 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:59.717 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.718 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:59.718 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.173967+0000", 2024-09-18T04:38:59.718 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:59.718 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.718 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.718 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:59.718 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.173967+0000", 2024-09-18T04:38:59.718 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:59.718 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.718 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.718 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:59.718 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.173965+0000", 2024-09-18T04:38:59.718 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:59.718 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.719 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.719 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:59.719 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.174507+0000", 2024-09-18T04:38:59.719 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00054151999999999998 2024-09-18T04:38:59.719 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.719 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.719 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:59.719 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.174526+0000", 2024-09-18T04:38:59.719 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8683999999999999e-05 2024-09-18T04:38:59.719 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.719 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.719 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:59.719 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.174538+0000", 2024-09-18T04:38:59.719 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2561000000000001e-05 2024-09-18T04:38:59.719 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.720 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.720 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:59.720 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.174679+0000", 2024-09-18T04:38:59.720 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000140841 2024-09-18T04:38:59.720 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.720 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.720 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:59.720 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.174742+0000", 2024-09-18T04:38:59.720 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2800999999999997e-05 2024-09-18T04:38:59.720 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.720 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.720 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:59.720 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.178265+0000", 2024-09-18T04:38:59.721 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0035236849999999999 2024-09-18T04:38:59.721 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.721 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.721 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:59.721 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.178326+0000", 2024-09-18T04:38:59.721 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0078000000000003e-05 2024-09-18T04:38:59.721 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:59.721 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:59.721 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:59.721 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.721 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.721 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:663 2.4 e57/17 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 57'13, mlcod=57'11)", 2024-09-18T04:38:59.721 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.355300+0000", 2024-09-18T04:38:59.721 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 8.3064831790000007, 2024-09-18T04:38:59.722 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0049192760000000002, 2024-09-18T04:38:59.722 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:59.722 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:59.722 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:59.722 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.722 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:59.722 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355300+0000", 2024-09-18T04:38:59.722 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:59.722 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.722 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.722 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:59.722 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355300+0000", 2024-09-18T04:38:59.722 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:59.723 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.723 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.723 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:59.723 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355299+0000", 2024-09-18T04:38:59.723 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:59.723 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.723 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.723 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:59.723 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355876+0000", 2024-09-18T04:38:59.723 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00057730900000000003 2024-09-18T04:38:59.723 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.723 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.723 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:59.723 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355897+0000", 2024-09-18T04:38:59.723 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0477000000000001e-05 2024-09-18T04:38:59.724 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.724 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.724 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:59.724 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355905+0000", 2024-09-18T04:38:59.724 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5739999999999996e-06 2024-09-18T04:38:59.724 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.724 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.724 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:59.724 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355989+0000", 2024-09-18T04:38:59.724 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4002e-05 2024-09-18T04:38:59.724 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.724 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.724 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:59.724 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.356041+0000", 2024-09-18T04:38:59.725 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1347000000000002e-05 2024-09-18T04:38:59.725 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.725 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.725 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:59.725 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.360172+0000", 2024-09-18T04:38:59.725 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0041315090000000002 2024-09-18T04:38:59.725 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.725 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.725 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:59.725 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.360220+0000", 2024-09-18T04:38:59.725 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7584999999999999e-05 2024-09-18T04:38:59.725 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:59.725 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:59.726 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:59.726 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.726 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.726 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:677 2.5 e57/28 2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head v 57'18, mlcod=57'16)", 2024-09-18T04:38:59.726 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.434699+0000", 2024-09-18T04:38:59.726 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 8.2270842599999998, 2024-09-18T04:38:59.726 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0043950960000000002, 2024-09-18T04:38:59.726 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:59.726 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:59.726 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:59.726 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.726 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:59.726 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.434699+0000", 2024-09-18T04:38:59.726 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:59.727 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.727 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.727 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:59.727 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.434699+0000", 2024-09-18T04:38:59.727 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:59.727 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.727 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.727 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:59.727 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.434697+0000", 2024-09-18T04:38:59.727 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:59.727 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.727 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.727 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:59.727 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.435430+0000", 2024-09-18T04:38:59.728 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00073206900000000001 2024-09-18T04:38:59.728 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.728 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.728 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:59.728 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.435457+0000", 2024-09-18T04:38:59.728 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7039e-05 2024-09-18T04:38:59.728 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.728 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.728 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:59.728 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.435467+0000", 2024-09-18T04:38:59.728 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0482999999999999e-05 2024-09-18T04:38:59.728 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.728 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.728 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:59.729 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.435535+0000", 2024-09-18T04:38:59.729 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8308999999999999e-05 2024-09-18T04:38:59.729 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.729 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.729 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:59.729 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.435575+0000", 2024-09-18T04:38:59.729 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0018999999999997e-05 2024-09-18T04:38:59.729 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.729 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.729 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:59.729 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.439029+0000", 2024-09-18T04:38:59.729 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0034535019999999998 2024-09-18T04:38:59.729 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.729 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.729 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:59.730 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.439094+0000", 2024-09-18T04:38:59.730 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5517000000000003e-05 2024-09-18T04:38:59.730 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:59.730 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:59.730 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:59.730 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.730 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.730 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:679 2.5 e57/28 2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head v 57'20, mlcod=57'18)", 2024-09-18T04:38:59.730 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.445316+0000", 2024-09-18T04:38:59.730 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 8.2164677019999992, 2024-09-18T04:38:59.730 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0047016760000000001, 2024-09-18T04:38:59.730 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:59.730 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:59.730 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:59.730 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.731 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:59.731 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445316+0000", 2024-09-18T04:38:59.731 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:59.731 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.731 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.731 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:59.731 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445316+0000", 2024-09-18T04:38:59.731 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:59.731 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.731 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.731 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:59.731 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445314+0000", 2024-09-18T04:38:59.731 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:59.731 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.731 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.732 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:59.732 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445947+0000", 2024-09-18T04:38:59.732 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00063298900000000003 2024-09-18T04:38:59.732 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.732 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.732 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:59.732 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445968+0000", 2024-09-18T04:38:59.732 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.054e-05 2024-09-18T04:38:59.732 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.732 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.732 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:59.732 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445981+0000", 2024-09-18T04:38:59.732 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2697000000000001e-05 2024-09-18T04:38:59.733 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.733 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.733 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:59.733 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.446102+0000", 2024-09-18T04:38:59.733 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012189300000000001 2024-09-18T04:38:59.733 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.733 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.733 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:59.733 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.446151+0000", 2024-09-18T04:38:59.733 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8279000000000003e-05 2024-09-18T04:38:59.733 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.733 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.733 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:59.733 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.449953+0000", 2024-09-18T04:38:59.734 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003801979 2024-09-18T04:38:59.734 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.734 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.734 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:59.734 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.450018+0000", 2024-09-18T04:38:59.734 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4873e-05 2024-09-18T04:38:59.734 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:59.734 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:59.734 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:59.734 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.734 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.734 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:699 2.4 e57/17 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 57'17, mlcod=57'15)", 2024-09-18T04:38:59.734 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.566073+0000", 2024-09-18T04:38:59.734 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 8.0957101890000001, 2024-09-18T04:38:59.734 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055398380000000001, 2024-09-18T04:38:59.735 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:59.735 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:59.735 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:59.735 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.735 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:59.735 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566073+0000", 2024-09-18T04:38:59.735 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:59.735 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.735 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.735 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:59.735 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566073+0000", 2024-09-18T04:38:59.735 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:59.735 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.735 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.735 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:59.736 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566071+0000", 2024-09-18T04:38:59.736 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:59.736 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.736 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.736 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:59.736 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566637+0000", 2024-09-18T04:38:59.736 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00056516699999999999 2024-09-18T04:38:59.736 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.736 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.736 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:59.736 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566668+0000", 2024-09-18T04:38:59.736 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1544999999999997e-05 2024-09-18T04:38:59.736 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.736 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.736 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:59.737 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566685+0000", 2024-09-18T04:38:59.737 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6857000000000001e-05 2024-09-18T04:38:59.737 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.737 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.737 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:59.737 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566816+0000", 2024-09-18T04:38:59.737 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013092799999999999 2024-09-18T04:38:59.737 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.737 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.737 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:59.737 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566875+0000", 2024-09-18T04:38:59.737 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9583999999999997e-05 2024-09-18T04:38:59.737 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.737 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.737 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:59.738 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.571567+0000", 2024-09-18T04:38:59.738 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.004691682 2024-09-18T04:38:59.738 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.738 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.738 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:59.738 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.571613+0000", 2024-09-18T04:38:59.738 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6125000000000003e-05 2024-09-18T04:38:59.738 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:59.738 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:59.738 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:59.738 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.738 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.738 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:703 2.4 e57/17 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 57'21, mlcod=57'19)", 2024-09-18T04:38:59.738 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.588442+0000", 2024-09-18T04:38:59.739 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 8.073341653, 2024-09-18T04:38:59.739 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0044132429999999999, 2024-09-18T04:38:59.739 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:59.739 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:59.739 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:59.739 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.739 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:59.739 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.588442+0000", 2024-09-18T04:38:59.739 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:59.739 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.739 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.739 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:59.739 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.588442+0000", 2024-09-18T04:38:59.739 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:59.739 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.739 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.740 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:59.740 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.588439+0000", 2024-09-18T04:38:59.740 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999971 2024-09-18T04:38:59.740 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.740 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.740 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:59.740 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.589194+0000", 2024-09-18T04:38:59.740 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000754447 2024-09-18T04:38:59.740 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.740 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.740 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:59.740 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.589231+0000", 2024-09-18T04:38:59.740 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6912000000000002e-05 2024-09-18T04:38:59.740 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.741 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.741 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:59.741 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.589249+0000", 2024-09-18T04:38:59.741 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8712999999999999e-05 2024-09-18T04:38:59.741 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.741 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.741 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:59.741 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.589336+0000", 2024-09-18T04:38:59.741 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.6991999999999997e-05 2024-09-18T04:38:59.741 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.741 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.741 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:59.741 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.589387+0000", 2024-09-18T04:38:59.742 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0698000000000001e-05 2024-09-18T04:38:59.742 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.742 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.742 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:59.742 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.592782+0000", 2024-09-18T04:38:59.742 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003394925 2024-09-18T04:38:59.742 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.742 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.742 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:59.742 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.592855+0000", 2024-09-18T04:38:59.742 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3245999999999996e-05 2024-09-18T04:38:59.742 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:59.742 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:59.742 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:59.743 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.743 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.743 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:717 2.5 e57/28 2:b227e6b1:::rbd_data.125fc09df6d1.000000000000000e:head v 57'26, mlcod=57'24)", 2024-09-18T04:38:59.743 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.667326+0000", 2024-09-18T04:38:59.743 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 7.9944575669999995, 2024-09-18T04:38:59.743 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0042492520000000002, 2024-09-18T04:38:59.743 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:59.743 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:59.743 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:59.743 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.743 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:59.743 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.667326+0000", 2024-09-18T04:38:59.743 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:59.743 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.743 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.743 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:59.744 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.667326+0000", 2024-09-18T04:38:59.744 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:59.744 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.744 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.744 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:59.744 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.667324+0000", 2024-09-18T04:38:59.744 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:38:59.744 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.744 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.744 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:59.744 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.667849+0000", 2024-09-18T04:38:59.744 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00052490600000000001 2024-09-18T04:38:59.744 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.745 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.745 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:59.745 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.667889+0000", 2024-09-18T04:38:59.745 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9632999999999998e-05 2024-09-18T04:38:59.745 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.745 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.745 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:59.745 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.667907+0000", 2024-09-18T04:38:59.745 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7564999999999999e-05 2024-09-18T04:38:59.745 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.745 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.745 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:59.745 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.668059+0000", 2024-09-18T04:38:59.745 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000152263 2024-09-18T04:38:59.746 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.746 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.746 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:59.746 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.668099+0000", 2024-09-18T04:38:59.746 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0413000000000001e-05 2024-09-18T04:38:59.746 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.746 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.746 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:59.746 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.671512+0000", 2024-09-18T04:38:59.746 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0034132339999999998 2024-09-18T04:38:59.746 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.746 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.746 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:59.746 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.671575+0000", 2024-09-18T04:38:59.746 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2848999999999994e-05 2024-09-18T04:38:59.746 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:59.747 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:59.747 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:59.747 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.747 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.747 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:721 2.5 e57/28 2:b227e6b1:::rbd_data.125fc09df6d1.000000000000000e:head v 57'30, mlcod=57'28)", 2024-09-18T04:38:59.747 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.686880+0000", 2024-09-18T04:38:59.747 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 7.974903748, 2024-09-18T04:38:59.747 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0044157340000000002, 2024-09-18T04:38:59.747 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:38:59.747 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:38:59.747 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:38:59.747 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.747 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:38:59.748 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.686880+0000", 2024-09-18T04:38:59.748 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:59.748 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.748 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.748 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:38:59.748 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.686880+0000", 2024-09-18T04:38:59.748 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:38:59.748 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.748 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.748 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:38:59.748 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.686878+0000", 2024-09-18T04:38:59.748 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:38:59.748 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.748 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.749 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:38:59.749 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.687586+0000", 2024-09-18T04:38:59.749 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00070759499999999999 2024-09-18T04:38:59.749 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.749 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.749 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:38:59.749 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.687610+0000", 2024-09-18T04:38:59.749 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4304e-05 2024-09-18T04:38:59.749 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.749 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.749 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:38:59.749 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.687619+0000", 2024-09-18T04:38:59.749 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3710000000000006e-06 2024-09-18T04:38:59.749 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.750 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.750 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:38:59.750 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.687700+0000", 2024-09-18T04:38:59.750 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0916999999999998e-05 2024-09-18T04:38:59.750 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.750 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.750 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:38:59.750 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.687735+0000", 2024-09-18T04:38:59.750 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5138999999999997e-05 2024-09-18T04:38:59.750 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.750 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.750 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:38:59.750 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.691244+0000", 2024-09-18T04:38:59.750 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0035090189999999999 2024-09-18T04:38:59.751 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:38:59.751 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:38:59.751 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:38:59.751 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.691296+0000", 2024-09-18T04:38:59.751 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1267000000000002e-05 2024-09-18T04:38:59.751 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:59.751 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:59.751 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:59.751 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:38:59.751 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:38:59.751 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:59.753 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:38:59.753 INFO:tasks.workunit.client.0.smithi115.stderr:+ ps -C rbd-nbd 2024-09-18T04:38:59.754 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false grep '^ *18295\b' 2024-09-18T04:38:59.754 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep '^ *18295\b' 2024-09-18T04:38:59.754 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:38:59.754 INFO:tasks.workunit.client.0.smithi115.stderr:+ sleep 0.5 2024-09-18T04:38:59.754 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:38:59.754 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:38:59.755 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:38:59.755 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:38:59.755 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:38:59.756 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:38:59.756 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:38:59.756 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:38:59.756 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:38:59.745+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:38:59.854 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:38:59.849+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:38:59.911 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:38:59.911 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:38:59.911 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:38:59.911 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:38:59.921 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:38:59.955 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:38:59.949+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:00.027 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:00.027 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:00.027 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:39:00.027 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:39:00.027 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:00.037 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:39:00.056 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:00.053+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:00.147 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:00.147 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:39:00.147 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:39:00.147 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:39:00.147 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.147 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:33 2.1 e51/24 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head v 51'4, mlcod=51'2)", 2024-09-18T04:39:00.147 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:39:00.148 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.863285109, 2024-09-18T04:39:00.148 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018702106, 2024-09-18T04:39:00.148 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:00.148 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:00.148 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:00.148 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.148 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:00.148 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:39:00.148 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.148 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.148 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.148 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:00.148 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:39:00.149 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.149 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.149 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:00.149 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279197+0000", 2024-09-18T04:39:00.149 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:00.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.149 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.149 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:00.149 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282682+0000", 2024-09-18T04:39:00.149 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0034848689999999998 2024-09-18T04:39:00.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.149 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.149 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:00.150 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282709+0000", 2024-09-18T04:39:00.150 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7599000000000001e-05 2024-09-18T04:39:00.150 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.150 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.150 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:00.150 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282720+0000", 2024-09-18T04:39:00.150 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.092e-05 2024-09-18T04:39:00.150 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.150 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.150 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:00.150 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282791+0000", 2024-09-18T04:39:00.150 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1268999999999998e-05 2024-09-18T04:39:00.150 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.151 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.151 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:00.151 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282843+0000", 2024-09-18T04:39:00.151 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1771999999999998e-05 2024-09-18T04:39:00.151 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.151 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.151 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:00.151 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.297830+0000", 2024-09-18T04:39:00.151 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014987228999999999 2024-09-18T04:39:00.151 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.151 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.151 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:00.151 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.297901+0000", 2024-09-18T04:39:00.151 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0209999999999994e-05 2024-09-18T04:39:00.152 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.152 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:00.152 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.152 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.152 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.152 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:38 2.0 e51/17 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head v 51'10, mlcod=51'8)", 2024-09-18T04:39:00.152 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:39:00.152 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.861866678999998, 2024-09-18T04:39:00.152 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.023984472, 2024-09-18T04:39:00.152 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:00.152 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:00.152 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:00.152 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.153 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:00.153 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:39:00.153 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.153 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.153 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.153 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:00.153 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:39:00.153 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.153 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.153 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.153 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:00.153 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280614+0000", 2024-09-18T04:39:00.153 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:00.153 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.154 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.154 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:00.154 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293094+0000", 2024-09-18T04:39:00.154 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012479571 2024-09-18T04:39:00.154 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.154 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.154 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:00.154 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293111+0000", 2024-09-18T04:39:00.154 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7354999999999999e-05 2024-09-18T04:39:00.154 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.154 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.154 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:00.154 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293121+0000", 2024-09-18T04:39:00.154 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6979999999999993e-06 2024-09-18T04:39:00.154 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.154 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.155 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:00.155 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293180+0000", 2024-09-18T04:39:00.155 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9333000000000003e-05 2024-09-18T04:39:00.155 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.155 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.155 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:00.155 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293203+0000", 2024-09-18T04:39:00.155 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2633000000000002e-05 2024-09-18T04:39:00.155 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.155 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.155 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:00.155 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304534+0000", 2024-09-18T04:39:00.155 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011330761 2024-09-18T04:39:00.155 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.156 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.156 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:00.156 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304601+0000", 2024-09-18T04:39:00.156 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7657000000000006e-05 2024-09-18T04:39:00.156 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.156 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:00.156 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.156 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.156 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:46 2.6 2:7ba25245:::rbd_data.125fc09df6d1.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:00.156 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:39:00.156 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.834323878999999, 2024-09-18T04:39:00.156 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.035604595000000003, 2024-09-18T04:39:00.157 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:00.157 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:00.157 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:00.157 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:00.157 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:00.157 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 46 2024-09-18T04:39:00.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.157 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:00.157 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.157 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:00.157 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:39:00.157 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.157 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.158 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:00.158 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:39:00.158 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.158 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.158 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.158 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:00.158 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308158+0000", 2024-09-18T04:39:00.158 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:00.158 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.158 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.158 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:00.158 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313191+0000", 2024-09-18T04:39:00.158 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0050323929999999996 2024-09-18T04:39:00.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.159 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.159 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:00.159 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313210+0000", 2024-09-18T04:39:00.159 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9673e-05 2024-09-18T04:39:00.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.159 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.159 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:00.159 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313221+0000", 2024-09-18T04:39:00.159 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0444e-05 2024-09-18T04:39:00.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.159 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.159 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:00.159 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313294+0000", 2024-09-18T04:39:00.159 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2650000000000004e-05 2024-09-18T04:39:00.160 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.160 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.160 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:00.160 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313493+0000", 2024-09-18T04:39:00.160 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019968300000000001 2024-09-18T04:39:00.160 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.160 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.160 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:00.160 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313666+0000", 2024-09-18T04:39:00.160 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00017248599999999999 2024-09-18T04:39:00.160 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.160 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.160 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:00.161 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.323175+0000", 2024-09-18T04:39:00.161 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0095088819999999998 2024-09-18T04:39:00.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.161 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.161 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:00.161 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343651+0000", 2024-09-18T04:39:00.161 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020475892999999998 2024-09-18T04:39:00.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.161 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.161 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:00.161 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343715+0000", 2024-09-18T04:39:00.161 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4443999999999995e-05 2024-09-18T04:39:00.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.162 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.162 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:00.162 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343764+0000", 2024-09-18T04:39:00.162 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9280000000000003e-05 2024-09-18T04:39:00.162 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.162 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:00.162 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.162 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.162 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.162 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:47 2.4 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:00.162 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:39:00.162 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.829241841999998, 2024-09-18T04:39:00.162 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.032570693999999997, 2024-09-18T04:39:00.162 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:00.163 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:00.163 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:00.163 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:00.163 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:00.163 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 47 2024-09-18T04:39:00.163 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.163 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:00.163 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.163 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:00.163 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:39:00.163 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.163 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.163 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.164 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:00.164 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:39:00.164 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.164 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.164 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.164 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:00.164 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313240+0000", 2024-09-18T04:39:00.164 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:00.164 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.164 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.164 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:00.164 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325825+0000", 2024-09-18T04:39:00.164 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012584425999999999 2024-09-18T04:39:00.164 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.165 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.165 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:00.165 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325848+0000", 2024-09-18T04:39:00.165 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3025999999999999e-05 2024-09-18T04:39:00.165 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.165 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.165 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:00.165 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325863+0000", 2024-09-18T04:39:00.165 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5152e-05 2024-09-18T04:39:00.165 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.165 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.165 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:00.165 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325938+0000", 2024-09-18T04:39:00.166 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4982999999999997e-05 2024-09-18T04:39:00.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.166 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.166 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:00.166 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326090+0000", 2024-09-18T04:39:00.166 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015207900000000001 2024-09-18T04:39:00.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.166 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.166 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:00.166 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326242+0000", 2024-09-18T04:39:00.166 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015163500000000001 2024-09-18T04:39:00.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.167 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.167 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:00.167 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343015+0000", 2024-09-18T04:39:00.167 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016773414 2024-09-18T04:39:00.167 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.167 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.167 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:00.167 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345664+0000", 2024-09-18T04:39:00.167 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026487870000000001 2024-09-18T04:39:00.167 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.167 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.167 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:00.167 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345739+0000", 2024-09-18T04:39:00.167 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5018999999999994e-05 2024-09-18T04:39:00.168 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.168 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.168 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:00.168 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345812+0000", 2024-09-18T04:39:00.168 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3547000000000006e-05 2024-09-18T04:39:00.168 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.168 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:00.168 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.168 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.168 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.168 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:48 2.6 2:61db7a3a:::rbd_data.125fc09df6d1.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:00.168 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:39:00.168 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.81658517, 2024-09-18T04:39:00.168 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.025064388, 2024-09-18T04:39:00.168 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:00.169 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:00.169 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:00.169 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:00.169 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:00.169 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 48 2024-09-18T04:39:00.169 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.169 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:00.169 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.169 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:00.169 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:39:00.169 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.169 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.169 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.169 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:00.170 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:39:00.170 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.170 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.170 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.170 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:00.170 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325897+0000", 2024-09-18T04:39:00.170 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:00.170 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.170 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.170 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:00.170 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333697+0000", 2024-09-18T04:39:00.170 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0078002649999999998 2024-09-18T04:39:00.170 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.171 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.171 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:00.171 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333716+0000", 2024-09-18T04:39:00.171 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8910999999999999e-05 2024-09-18T04:39:00.171 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.171 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.171 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:00.171 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333728+0000", 2024-09-18T04:39:00.171 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1936000000000001e-05 2024-09-18T04:39:00.171 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.171 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.171 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:00.171 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333792+0000", 2024-09-18T04:39:00.171 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4010999999999994e-05 2024-09-18T04:39:00.171 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.172 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.172 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:00.172 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.334082+0000", 2024-09-18T04:39:00.172 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00028955799999999998 2024-09-18T04:39:00.172 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.172 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.172 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:00.172 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.334326+0000", 2024-09-18T04:39:00.172 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00024399099999999999 2024-09-18T04:39:00.172 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.172 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.172 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:00.172 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350745+0000", 2024-09-18T04:39:00.172 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016419474999999999 2024-09-18T04:39:00.173 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.173 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.173 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:00.173 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350808+0000", 2024-09-18T04:39:00.173 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3244999999999995e-05 2024-09-18T04:39:00.173 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.173 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.173 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:00.173 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350892+0000", 2024-09-18T04:39:00.173 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4074999999999997e-05 2024-09-18T04:39:00.173 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.173 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.173 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:00.173 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350963+0000", 2024-09-18T04:39:00.174 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0412000000000003e-05 2024-09-18T04:39:00.174 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.174 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:00.174 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.174 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.174 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.174 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:51 2.6 2:775d7b11:::rbd_data.125fc09df6d1.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:00.174 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:39:00.174 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.808733033999999, 2024-09-18T04:39:00.174 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.023824221, 2024-09-18T04:39:00.174 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:00.174 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:00.174 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:00.174 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:00.175 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:00.175 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 51 2024-09-18T04:39:00.175 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.175 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:00.175 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.175 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:00.175 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:39:00.175 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.175 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.175 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.175 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:00.175 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:39:00.175 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.175 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.176 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.176 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:00.176 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333749+0000", 2024-09-18T04:39:00.176 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:00.176 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.176 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.176 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:00.176 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340899+0000", 2024-09-18T04:39:00.176 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0071500340000000004 2024-09-18T04:39:00.176 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.176 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.176 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:00.176 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340929+0000", 2024-09-18T04:39:00.177 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0093000000000001e-05 2024-09-18T04:39:00.177 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.177 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.177 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:00.177 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340943+0000", 2024-09-18T04:39:00.177 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4019000000000001e-05 2024-09-18T04:39:00.177 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.177 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.177 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:00.177 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343834+0000", 2024-09-18T04:39:00.177 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028918120000000001 2024-09-18T04:39:00.177 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.177 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.177 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:00.178 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344020+0000", 2024-09-18T04:39:00.178 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000185892 2024-09-18T04:39:00.178 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.178 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.178 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:00.178 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344190+0000", 2024-09-18T04:39:00.178 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016946800000000001 2024-09-18T04:39:00.178 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.178 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.178 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:00.178 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.351291+0000", 2024-09-18T04:39:00.178 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.007100805 2024-09-18T04:39:00.178 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.178 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.179 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:00.179 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357470+0000", 2024-09-18T04:39:00.179 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0061797570000000001 2024-09-18T04:39:00.179 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.179 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.179 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:00.179 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357533+0000", 2024-09-18T04:39:00.179 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2906999999999999e-05 2024-09-18T04:39:00.179 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.179 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.179 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:00.179 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357575+0000", 2024-09-18T04:39:00.179 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1403999999999999e-05 2024-09-18T04:39:00.179 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.180 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:00.180 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.180 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.180 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.180 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:56 2.4 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:00.180 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:39:00.180 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.792687941000001, 2024-09-18T04:39:00.180 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.034295870999999999, 2024-09-18T04:39:00.180 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:00.180 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:00.180 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:00.180 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:00.180 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:00.181 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 56 2024-09-18T04:39:00.181 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.181 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:00.181 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.181 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:00.181 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:39:00.181 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.181 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.181 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.181 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:00.181 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:39:00.181 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.181 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.181 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.182 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:00.182 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349794+0000", 2024-09-18T04:39:00.182 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:00.182 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.182 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.182 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:00.182 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366632+0000", 2024-09-18T04:39:00.182 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01683755 2024-09-18T04:39:00.182 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.182 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.182 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:00.182 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366669+0000", 2024-09-18T04:39:00.182 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7351000000000002e-05 2024-09-18T04:39:00.183 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.183 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.183 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:00.183 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366686+0000", 2024-09-18T04:39:00.183 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6507999999999998e-05 2024-09-18T04:39:00.183 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.183 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.183 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:00.183 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366962+0000", 2024-09-18T04:39:00.183 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00027655900000000001 2024-09-18T04:39:00.183 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.183 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.183 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:00.183 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.367452+0000", 2024-09-18T04:39:00.184 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00049025299999999996 2024-09-18T04:39:00.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.184 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.184 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:00.184 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368113+0000", 2024-09-18T04:39:00.184 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00066093399999999996 2024-09-18T04:39:00.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.184 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.184 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:00.184 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383719+0000", 2024-09-18T04:39:00.184 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015606106999999999 2024-09-18T04:39:00.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.184 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.184 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:00.185 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384035+0000", 2024-09-18T04:39:00.185 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00031590399999999999 2024-09-18T04:39:00.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.185 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.185 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:00.185 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384067+0000", 2024-09-18T04:39:00.185 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.129e-05 2024-09-18T04:39:00.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.185 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.185 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:00.185 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384091+0000", 2024-09-18T04:39:00.185 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4788000000000001e-05 2024-09-18T04:39:00.185 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.186 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:00.186 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.186 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.186 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:57 2.0 e51/17 2:0cdd070e:::rbd_data.125fc09df6d1.000000000000000d:head v 51'21, mlcod=51'19)", 2024-09-18T04:39:00.186 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:39:00.186 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.784860454, 2024-09-18T04:39:00.186 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.025216311000000002, 2024-09-18T04:39:00.186 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:00.186 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:00.186 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:00.186 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.186 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:00.186 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:39:00.187 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.187 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.187 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:00.187 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:39:00.187 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.187 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.187 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:00.187 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:39:00.187 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:00.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.187 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.187 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:00.187 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363844+0000", 2024-09-18T04:39:00.188 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0062218450000000002 2024-09-18T04:39:00.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.188 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.188 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:00.188 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363879+0000", 2024-09-18T04:39:00.188 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4382999999999999e-05 2024-09-18T04:39:00.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.188 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.188 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:00.188 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363893+0000", 2024-09-18T04:39:00.188 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4348999999999999e-05 2024-09-18T04:39:00.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.188 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.188 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:00.188 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363977+0000", 2024-09-18T04:39:00.189 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3499999999999997e-05 2024-09-18T04:39:00.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.189 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.189 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:00.189 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364009+0000", 2024-09-18T04:39:00.189 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2292000000000002e-05 2024-09-18T04:39:00.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.189 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.189 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:00.189 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.382781+0000", 2024-09-18T04:39:00.189 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018771626 2024-09-18T04:39:00.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.189 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.190 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:00.190 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.382839+0000", 2024-09-18T04:39:00.190 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8758000000000002e-05 2024-09-18T04:39:00.190 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.190 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:00.190 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.190 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.190 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:59 2.0 e51/17 2:19c52cde:::rbd_data.125fc09df6d1.000000000000000f:head v 51'22, mlcod=51'19)", 2024-09-18T04:39:00.190 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:39:00.190 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.778200527999999, 2024-09-18T04:39:00.190 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.019497112, 2024-09-18T04:39:00.190 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:00.191 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:00.191 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:00.191 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.191 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:00.191 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:39:00.191 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.191 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.191 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:00.191 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:39:00.191 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.191 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.191 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:00.192 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364282+0000", 2024-09-18T04:39:00.192 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:00.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.192 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.192 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:00.192 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371823+0000", 2024-09-18T04:39:00.192 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0075411829999999999 2024-09-18T04:39:00.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.192 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.192 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:00.192 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371837+0000", 2024-09-18T04:39:00.192 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4336e-05 2024-09-18T04:39:00.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.193 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.193 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:00.193 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371846+0000", 2024-09-18T04:39:00.193 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9770000000000006e-06 2024-09-18T04:39:00.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.193 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.193 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:00.193 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.373566+0000", 2024-09-18T04:39:00.193 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017192539999999999 2024-09-18T04:39:00.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.193 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.193 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:00.194 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.373595+0000", 2024-09-18T04:39:00.194 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8975999999999998e-05 2024-09-18T04:39:00.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.194 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.194 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:00.194 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383740+0000", 2024-09-18T04:39:00.194 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01014525 2024-09-18T04:39:00.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.194 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.194 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:00.194 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383780+0000", 2024-09-18T04:39:00.194 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0275999999999999e-05 2024-09-18T04:39:00.194 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.195 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:00.195 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.195 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.195 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:34 2.6 2:66289171:ns::rbd_data.133e813ab5bc.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:00.195 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.524015+0000", 2024-09-18T04:39:00.195 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.618468919000001, 2024-09-18T04:39:00.195 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027582111999999999, 2024-09-18T04:39:00.195 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:00.195 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:00.195 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:00.195 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:00.195 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:00.195 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 34 2024-09-18T04:39:00.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.196 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:00.196 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.196 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:00.196 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524015+0000", 2024-09-18T04:39:00.196 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.196 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.196 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:00.196 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524015+0000", 2024-09-18T04:39:00.196 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.196 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.196 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:00.197 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524012+0000", 2024-09-18T04:39:00.197 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:00.197 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.197 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.197 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:00.197 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534956+0000", 2024-09-18T04:39:00.197 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010943403000000001 2024-09-18T04:39:00.197 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.197 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.197 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:00.197 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534978+0000", 2024-09-18T04:39:00.197 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1965999999999999e-05 2024-09-18T04:39:00.197 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.197 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.197 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:00.198 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534990+0000", 2024-09-18T04:39:00.198 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2065999999999999e-05 2024-09-18T04:39:00.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.198 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.198 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:00.198 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535065+0000", 2024-09-18T04:39:00.198 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5523000000000002e-05 2024-09-18T04:39:00.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.198 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.198 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:00.198 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535208+0000", 2024-09-18T04:39:00.198 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014309500000000001 2024-09-18T04:39:00.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.198 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.198 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:00.199 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535343+0000", 2024-09-18T04:39:00.199 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013437400000000001 2024-09-18T04:39:00.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.199 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.199 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:00.199 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.546576+0000", 2024-09-18T04:39:00.199 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011233425999999999 2024-09-18T04:39:00.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.199 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.199 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:00.199 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551498+0000", 2024-09-18T04:39:00.199 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0049219349999999997 2024-09-18T04:39:00.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.199 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.199 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:00.199 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551554+0000", 2024-09-18T04:39:00.200 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6400999999999998e-05 2024-09-18T04:39:00.200 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.200 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.200 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:00.200 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551597+0000", 2024-09-18T04:39:00.200 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2283000000000002e-05 2024-09-18T04:39:00.200 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.200 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:00.200 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.200 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.200 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.200 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:33 2.1 e51/24 2:8b40f1c5:ns::rbd_data.133e813ab5bc.0000000000000000:head v 51'7, mlcod=51'5)", 2024-09-18T04:39:00.200 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.531767+0000", 2024-09-18T04:39:00.200 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.610717002000001, 2024-09-18T04:39:00.201 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015370295000000001, 2024-09-18T04:39:00.201 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:00.201 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:00.201 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:00.201 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.201 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:00.201 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531767+0000", 2024-09-18T04:39:00.201 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.201 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.201 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.201 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:00.201 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531767+0000", 2024-09-18T04:39:00.201 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.201 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.201 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.202 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:00.202 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531765+0000", 2024-09-18T04:39:00.202 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:00.202 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.202 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.202 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:00.202 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534258+0000", 2024-09-18T04:39:00.202 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002493561 2024-09-18T04:39:00.202 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.202 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.202 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:00.202 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534288+0000", 2024-09-18T04:39:00.202 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.976e-05 2024-09-18T04:39:00.202 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.203 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.203 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:00.203 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534300+0000", 2024-09-18T04:39:00.203 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1928e-05 2024-09-18T04:39:00.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.203 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.203 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:00.203 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534367+0000", 2024-09-18T04:39:00.203 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7045999999999995e-05 2024-09-18T04:39:00.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.203 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.203 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:00.203 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534394+0000", 2024-09-18T04:39:00.203 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6903999999999999e-05 2024-09-18T04:39:00.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.204 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.204 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:00.204 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.547073+0000", 2024-09-18T04:39:00.204 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012679365 2024-09-18T04:39:00.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.204 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.204 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:00.204 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.547137+0000", 2024-09-18T04:39:00.204 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3691999999999999e-05 2024-09-18T04:39:00.204 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.204 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:00.204 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.204 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.205 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:39 2.4 2:304364d1:ns::rbd_data.133e813ab5bc.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:00.205 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.535013+0000", 2024-09-18T04:39:00.205 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.607470067000001, 2024-09-18T04:39:00.205 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027764487000000001, 2024-09-18T04:39:00.205 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:00.205 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:00.205 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:00.205 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:00.205 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:00.205 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 39 2024-09-18T04:39:00.205 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.205 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:00.205 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.205 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:00.206 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535013+0000", 2024-09-18T04:39:00.206 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.206 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.206 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.206 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:00.206 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535013+0000", 2024-09-18T04:39:00.206 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.206 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.206 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.206 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:00.206 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535012+0000", 2024-09-18T04:39:00.206 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:00.206 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.206 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.207 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:00.207 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545016+0000", 2024-09-18T04:39:00.207 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01000322 2024-09-18T04:39:00.207 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.207 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.207 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:00.207 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545035+0000", 2024-09-18T04:39:00.207 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9845999999999999e-05 2024-09-18T04:39:00.207 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.207 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.207 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:00.207 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545045+0000", 2024-09-18T04:39:00.207 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3640000000000005e-06 2024-09-18T04:39:00.207 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.208 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.208 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:00.208 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545152+0000", 2024-09-18T04:39:00.208 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000107025 2024-09-18T04:39:00.208 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.208 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.208 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:00.208 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545338+0000", 2024-09-18T04:39:00.208 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018643500000000001 2024-09-18T04:39:00.208 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.208 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.208 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:00.208 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545492+0000", 2024-09-18T04:39:00.208 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015393199999999999 2024-09-18T04:39:00.209 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.209 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.209 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:00.209 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.553549+0000", 2024-09-18T04:39:00.209 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0080569170000000002 2024-09-18T04:39:00.209 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.209 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.209 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:00.209 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562693+0000", 2024-09-18T04:39:00.209 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0091441609999999996 2024-09-18T04:39:00.209 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.209 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.209 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:00.209 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562748+0000", 2024-09-18T04:39:00.209 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5047000000000003e-05 2024-09-18T04:39:00.210 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.210 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.210 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:00.210 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562778+0000", 2024-09-18T04:39:00.210 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9652e-05 2024-09-18T04:39:00.210 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.210 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:00.210 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.210 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.210 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.210 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:40 2.6 2:784d6cd1:ns::rbd_data.133e813ab5bc.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:00.210 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.545062+0000", 2024-09-18T04:39:00.210 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.597421015999998, 2024-09-18T04:39:00.210 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.032771623, 2024-09-18T04:39:00.210 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:00.210 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:00.211 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:00.211 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:00.211 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:00.211 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 40 2024-09-18T04:39:00.211 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.211 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:00.211 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.211 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:00.211 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545062+0000", 2024-09-18T04:39:00.211 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.211 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.211 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.211 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:00.211 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545062+0000", 2024-09-18T04:39:00.211 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.212 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.212 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.212 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:00.212 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545061+0000", 2024-09-18T04:39:00.212 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:00.212 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.212 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.212 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:00.212 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552477+0000", 2024-09-18T04:39:00.212 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.007415914 2024-09-18T04:39:00.212 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.212 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.212 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:00.213 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552500+0000", 2024-09-18T04:39:00.213 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3546000000000001e-05 2024-09-18T04:39:00.213 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.213 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.213 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:00.213 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552515+0000", 2024-09-18T04:39:00.213 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4606999999999999e-05 2024-09-18T04:39:00.213 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.213 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.213 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:00.213 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552661+0000", 2024-09-18T04:39:00.213 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000146007 2024-09-18T04:39:00.213 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.213 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.214 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:00.214 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552876+0000", 2024-09-18T04:39:00.214 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000214618 2024-09-18T04:39:00.214 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.214 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.214 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:00.214 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.553061+0000", 2024-09-18T04:39:00.214 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018564100000000001 2024-09-18T04:39:00.214 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.214 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.214 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:00.214 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.565612+0000", 2024-09-18T04:39:00.214 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012550301999999999 2024-09-18T04:39:00.214 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.215 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.215 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:00.215 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577695+0000", 2024-09-18T04:39:00.215 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012083493000000001 2024-09-18T04:39:00.215 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.215 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.215 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:00.215 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577772+0000", 2024-09-18T04:39:00.215 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7158999999999998e-05 2024-09-18T04:39:00.215 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.215 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.215 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:00.215 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577834+0000", 2024-09-18T04:39:00.215 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1851000000000001e-05 2024-09-18T04:39:00.215 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.215 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:00.216 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.216 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.216 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.216 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:43 2.6 2:759f27b6:ns::rbd_data.133e813ab5bc.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:00.216 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.555868+0000", 2024-09-18T04:39:00.216 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.586615521999999, 2024-09-18T04:39:00.216 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.022332497999999999, 2024-09-18T04:39:00.216 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:00.216 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:00.216 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:00.216 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:00.216 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:00.216 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 43 2024-09-18T04:39:00.217 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.217 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:00.217 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.217 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:00.217 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555868+0000", 2024-09-18T04:39:00.217 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.217 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.217 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.217 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:00.217 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555868+0000", 2024-09-18T04:39:00.217 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.217 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.217 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.217 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:00.217 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555865+0000", 2024-09-18T04:39:00.218 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999971 2024-09-18T04:39:00.218 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.218 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.218 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:00.218 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561031+0000", 2024-09-18T04:39:00.218 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0051658770000000001 2024-09-18T04:39:00.218 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.218 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.218 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:00.218 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561048+0000", 2024-09-18T04:39:00.218 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6569e-05 2024-09-18T04:39:00.218 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.218 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.218 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:00.219 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561055+0000", 2024-09-18T04:39:00.219 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.328e-06 2024-09-18T04:39:00.219 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.219 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.219 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:00.219 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561114+0000", 2024-09-18T04:39:00.219 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9540000000000003e-05 2024-09-18T04:39:00.219 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.219 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.219 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:00.219 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561314+0000", 2024-09-18T04:39:00.219 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019965799999999999 2024-09-18T04:39:00.219 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.219 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.219 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:00.220 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561473+0000", 2024-09-18T04:39:00.220 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000158904 2024-09-18T04:39:00.220 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.220 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.220 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:00.220 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.567015+0000", 2024-09-18T04:39:00.220 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055420859999999999 2024-09-18T04:39:00.220 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.220 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.220 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:00.220 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577982+0000", 2024-09-18T04:39:00.220 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010966872000000001 2024-09-18T04:39:00.220 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.221 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.221 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:00.221 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578162+0000", 2024-09-18T04:39:00.221 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018050999999999999 2024-09-18T04:39:00.221 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.221 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.221 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:00.221 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578200+0000", 2024-09-18T04:39:00.221 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8000999999999998e-05 2024-09-18T04:39:00.221 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.221 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:00.221 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.221 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.222 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.222 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:48 2.6 2:7c225365:ns::rbd_data.133e813ab5bc.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:00.222 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:39:00.222 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.578816299, 2024-09-18T04:39:00.222 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.024576377, 2024-09-18T04:39:00.222 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:00.222 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:00.222 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:00.222 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:00.222 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:00.222 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 48 2024-09-18T04:39:00.222 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.222 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:00.222 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.222 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:00.223 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:39:00.223 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.223 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.223 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.223 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:00.223 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:39:00.223 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.223 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.223 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.223 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:00.223 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.563666+0000", 2024-09-18T04:39:00.223 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:00.223 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.223 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.223 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:00.223 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570331+0000", 2024-09-18T04:39:00.224 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0066653329999999998 2024-09-18T04:39:00.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.224 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.224 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:00.224 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570348+0000", 2024-09-18T04:39:00.224 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6507e-05 2024-09-18T04:39:00.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.224 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.224 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:00.224 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570355+0000", 2024-09-18T04:39:00.224 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2749999999999998e-06 2024-09-18T04:39:00.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.224 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.224 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:00.225 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570396+0000", 2024-09-18T04:39:00.225 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0809000000000002e-05 2024-09-18T04:39:00.225 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.225 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.225 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:00.225 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570571+0000", 2024-09-18T04:39:00.225 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000174823 2024-09-18T04:39:00.225 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.225 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.225 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:00.225 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570722+0000", 2024-09-18T04:39:00.225 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015154599999999999 2024-09-18T04:39:00.225 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.225 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.226 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:00.226 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579756+0000", 2024-09-18T04:39:00.226 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0090337000000000004 2024-09-18T04:39:00.226 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.226 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.226 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:00.226 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588093+0000", 2024-09-18T04:39:00.226 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.008336553 2024-09-18T04:39:00.226 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.226 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.226 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:00.226 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588173+0000", 2024-09-18T04:39:00.226 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0333999999999994e-05 2024-09-18T04:39:00.227 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.227 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.227 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:00.227 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588244+0000", 2024-09-18T04:39:00.227 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0573000000000003e-05 2024-09-18T04:39:00.227 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.227 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:00.227 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.227 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.227 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.227 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:49 2.4 2:20f0b0c9:ns::rbd_data.133e813ab5bc.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:00.227 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.570370+0000", 2024-09-18T04:39:00.227 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.572113972, 2024-09-18T04:39:00.227 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.029633981, 2024-09-18T04:39:00.227 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:00.228 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:00.228 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:00.228 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:00.228 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:00.228 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 49 2024-09-18T04:39:00.228 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.228 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:00.228 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.228 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:00.228 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570370+0000", 2024-09-18T04:39:00.228 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.228 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.228 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.228 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:00.228 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570370+0000", 2024-09-18T04:39:00.229 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.229 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.229 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.229 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:00.229 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570369+0000", 2024-09-18T04:39:00.229 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:00.229 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.229 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.229 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:00.229 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579090+0000", 2024-09-18T04:39:00.229 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0087212339999999996 2024-09-18T04:39:00.229 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.229 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.229 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:00.230 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579109+0000", 2024-09-18T04:39:00.230 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9120999999999999e-05 2024-09-18T04:39:00.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.230 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.230 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:00.230 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579123+0000", 2024-09-18T04:39:00.230 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4382000000000001e-05 2024-09-18T04:39:00.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.230 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.230 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:00.231 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579221+0000", 2024-09-18T04:39:00.231 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.7503999999999993e-05 2024-09-18T04:39:00.231 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.231 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.231 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:00.231 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579553+0000", 2024-09-18T04:39:00.231 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00033254200000000002 2024-09-18T04:39:00.231 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.231 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.231 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:00.231 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579817+0000", 2024-09-18T04:39:00.231 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00026317099999999998 2024-09-18T04:39:00.232 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.232 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.232 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:00.232 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.599905+0000", 2024-09-18T04:39:00.232 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020088575000000001 2024-09-18T04:39:00.232 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.232 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.232 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:00.232 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.599923+0000", 2024-09-18T04:39:00.232 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7425999999999999e-05 2024-09-18T04:39:00.232 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.232 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.233 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:00.233 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.599973+0000", 2024-09-18T04:39:00.233 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0649999999999998e-05 2024-09-18T04:39:00.233 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.233 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.233 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:00.233 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.600004+0000", 2024-09-18T04:39:00.233 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0361999999999998e-05 2024-09-18T04:39:00.233 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.233 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:00.233 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.233 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.234 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.234 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:50 2.1 e51/24 2:8c706374:ns::rbd_data.133e813ab5bc.000000000000000a:head v 51'8, mlcod=51'6)", 2024-09-18T04:39:00.234 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.577029+0000", 2024-09-18T04:39:00.234 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.565454600999999, 2024-09-18T04:39:00.234 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.017362855, 2024-09-18T04:39:00.234 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:00.234 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:00.234 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:00.234 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.234 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:00.234 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577029+0000", 2024-09-18T04:39:00.234 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.235 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.235 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.235 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:00.235 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577029+0000", 2024-09-18T04:39:00.235 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.235 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.235 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.235 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:00.235 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577027+0000", 2024-09-18T04:39:00.235 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:00.235 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.235 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.235 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:00.235 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578880+0000", 2024-09-18T04:39:00.236 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0018532080000000001 2024-09-18T04:39:00.236 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.236 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.236 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:00.236 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578906+0000", 2024-09-18T04:39:00.236 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5573999999999999e-05 2024-09-18T04:39:00.236 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.236 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.236 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:00.236 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578920+0000", 2024-09-18T04:39:00.236 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4483e-05 2024-09-18T04:39:00.236 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.236 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.237 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:00.237 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578983+0000", 2024-09-18T04:39:00.237 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2248000000000004e-05 2024-09-18T04:39:00.237 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.237 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.237 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:00.237 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579016+0000", 2024-09-18T04:39:00.237 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3364000000000001e-05 2024-09-18T04:39:00.237 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.237 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.237 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:00.237 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594342+0000", 2024-09-18T04:39:00.237 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01532564 2024-09-18T04:39:00.238 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.238 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.238 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:00.238 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594392+0000", 2024-09-18T04:39:00.238 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0068999999999998e-05 2024-09-18T04:39:00.238 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.238 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:00.238 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.238 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.238 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.238 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:45 2.0 e51/17 2:0fbfa3dc:ns::rbd_data.133e813ab5bc.0000000000000007:head v 51'24, mlcod=51'22)", 2024-09-18T04:39:00.238 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.581892+0000", 2024-09-18T04:39:00.238 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.560591929000001, 2024-09-18T04:39:00.239 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015646006, 2024-09-18T04:39:00.239 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:00.239 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:00.239 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:00.239 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.239 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:00.239 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581892+0000", 2024-09-18T04:39:00.239 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.239 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.239 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.239 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:00.239 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581892+0000", 2024-09-18T04:39:00.239 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.239 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.239 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.240 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:00.240 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581890+0000", 2024-09-18T04:39:00.240 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:00.240 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.240 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.240 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:00.240 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586188+0000", 2024-09-18T04:39:00.240 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0042985769999999996 2024-09-18T04:39:00.240 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.240 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.240 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:00.240 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586210+0000", 2024-09-18T04:39:00.240 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1069000000000001e-05 2024-09-18T04:39:00.241 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.241 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.241 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:00.241 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586221+0000", 2024-09-18T04:39:00.241 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1287000000000001e-05 2024-09-18T04:39:00.241 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.241 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.241 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:00.241 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586313+0000", 2024-09-18T04:39:00.241 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1935999999999995e-05 2024-09-18T04:39:00.241 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.241 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.241 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:00.241 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586368+0000", 2024-09-18T04:39:00.242 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5578000000000002e-05 2024-09-18T04:39:00.242 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.242 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.242 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:00.242 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.597473+0000", 2024-09-18T04:39:00.242 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011105139999999999 2024-09-18T04:39:00.242 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.242 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.242 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:00.242 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.597538+0000", 2024-09-18T04:39:00.242 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4133000000000004e-05 2024-09-18T04:39:00.242 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.242 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:00.242 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.243 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.243 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.243 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:53 2.4 2:39f7f174:ns::rbd_data.133e813ab5bc.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:00.243 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.584263+0000", 2024-09-18T04:39:00.243 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.558220055, 2024-09-18T04:39:00.243 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018701180000000001, 2024-09-18T04:39:00.243 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:00.243 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:00.243 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:00.243 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:00.243 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:00.243 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 53 2024-09-18T04:39:00.243 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.244 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:00.244 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.244 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:00.244 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584263+0000", 2024-09-18T04:39:00.244 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.244 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.244 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.244 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:00.244 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584263+0000", 2024-09-18T04:39:00.244 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.244 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.244 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.244 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:00.244 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584262+0000", 2024-09-18T04:39:00.245 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:00.245 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.245 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.245 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:00.245 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.591617+0000", 2024-09-18T04:39:00.245 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0073550600000000001 2024-09-18T04:39:00.245 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.245 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.245 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:00.245 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.591637+0000", 2024-09-18T04:39:00.245 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0432999999999999e-05 2024-09-18T04:39:00.245 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.245 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.246 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:00.246 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.591646+0000", 2024-09-18T04:39:00.246 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7849999999999997e-06 2024-09-18T04:39:00.246 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.246 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.246 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:00.246 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593478+0000", 2024-09-18T04:39:00.246 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001831495 2024-09-18T04:39:00.246 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.246 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.246 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:00.246 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593661+0000", 2024-09-18T04:39:00.246 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018364999999999999 2024-09-18T04:39:00.246 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.247 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.247 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:00.247 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593834+0000", 2024-09-18T04:39:00.247 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000173065 2024-09-18T04:39:00.247 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.247 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.247 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:00.247 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.600670+0000", 2024-09-18T04:39:00.247 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0068354970000000003 2024-09-18T04:39:00.247 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.247 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.247 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:00.247 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602851+0000", 2024-09-18T04:39:00.247 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002181653 2024-09-18T04:39:00.247 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.248 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.248 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:00.248 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602930+0000", 2024-09-18T04:39:00.248 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8045000000000002e-05 2024-09-18T04:39:00.248 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.248 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.248 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:00.248 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602965+0000", 2024-09-18T04:39:00.248 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5102999999999999e-05 2024-09-18T04:39:00.248 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.248 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:00.248 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.248 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.249 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.249 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:56 2.6 2:72453c3e:ns::rbd_data.133e813ab5bc.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:00.249 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.595988+0000", 2024-09-18T04:39:00.249 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.546495653000001, 2024-09-18T04:39:00.249 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.030296172, 2024-09-18T04:39:00.249 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:00.249 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:00.249 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:00.249 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:00.249 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:00.249 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 56 2024-09-18T04:39:00.249 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.249 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:00.249 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.250 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:00.250 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.595988+0000", 2024-09-18T04:39:00.250 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.250 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.250 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.250 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:00.250 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.595988+0000", 2024-09-18T04:39:00.250 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.250 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.250 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.250 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:00.250 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.595986+0000", 2024-09-18T04:39:00.250 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:00.250 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.250 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.251 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:00.251 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605262+0000", 2024-09-18T04:39:00.251 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0092767240000000001 2024-09-18T04:39:00.251 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.251 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.251 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:00.251 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605282+0000", 2024-09-18T04:39:00.251 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9307e-05 2024-09-18T04:39:00.251 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.251 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.251 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:00.251 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605292+0000", 2024-09-18T04:39:00.251 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9510000000000001e-06 2024-09-18T04:39:00.251 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.252 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.252 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:00.252 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605369+0000", 2024-09-18T04:39:00.252 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7846e-05 2024-09-18T04:39:00.252 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.252 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.252 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:00.252 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605668+0000", 2024-09-18T04:39:00.252 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00029896099999999999 2024-09-18T04:39:00.252 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.252 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.252 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:00.252 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605906+0000", 2024-09-18T04:39:00.252 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000237758 2024-09-18T04:39:00.253 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.253 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.253 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:00.253 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.613532+0000", 2024-09-18T04:39:00.253 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0076260590000000001 2024-09-18T04:39:00.253 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.253 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.253 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:00.253 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.626116+0000", 2024-09-18T04:39:00.253 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012584092999999999 2024-09-18T04:39:00.253 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.253 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.253 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:00.253 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.626216+0000", 2024-09-18T04:39:00.253 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9630999999999995e-05 2024-09-18T04:39:00.254 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.254 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.254 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:00.254 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.626284+0000", 2024-09-18T04:39:00.254 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8034999999999995e-05 2024-09-18T04:39:00.254 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.254 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:00.254 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.254 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.254 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:00.254 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:00.257 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:39:00.257 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:00.249+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:00.257 INFO:tasks.workunit.client.0.smithi115.stderr:++ _sudo rbd --device-type nbd map --exclusive rbd/testrbdnbd17342 2024-09-18T04:39:00.257 INFO:tasks.workunit.client.0.smithi115.stderr:++ local cmd 2024-09-18T04:39:00.257 INFO:tasks.workunit.client.0.smithi115.stderr:+++ id -u 2024-09-18T04:39:00.257 INFO:tasks.workunit.client.0.smithi115.stderr:++ '[' 1000 -eq 0 ']' 2024-09-18T04:39:00.257 INFO:tasks.workunit.client.0.smithi115.stderr:+++ which rbd 2024-09-18T04:39:00.258 INFO:tasks.workunit.client.0.smithi115.stderr:++ cmd=/usr/bin/rbd 2024-09-18T04:39:00.258 INFO:tasks.workunit.client.0.smithi115.stderr:++ shift 2024-09-18T04:39:00.258 INFO:tasks.workunit.client.0.smithi115.stderr:++ sudo -nE /usr/bin/rbd --device-type nbd map --exclusive rbd/testrbdnbd17342 2024-09-18T04:39:00.357 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:00.349+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:00.372 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:00.372 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:00.372 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:39:00.372 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:00.382 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:39:00.391 INFO:tasks.workunit.client.0.smithi115.stderr:+ DEV=/dev/nbd0 2024-09-18T04:39:00.391 INFO:tasks.workunit.client.0.smithi115.stderr:+ get_pid rbd 2024-09-18T04:39:00.391 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pool=rbd 2024-09-18T04:39:00.391 INFO:tasks.workunit.client.0.smithi115.stderr:+ local ns= 2024-09-18T04:39:00.392 INFO:tasks.workunit.client.0.smithi115.stderr:++ rbd device --device-type nbd --format xml list 2024-09-18T04:39:00.393 INFO:tasks.workunit.client.0.smithi115.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd17342'\''][device='\''/dev/nbd0'\'']/id' 2024-09-18T04:39:00.458 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:00.453+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:00.479 INFO:tasks.workunit.client.0.smithi115.stderr:+ PID=18369 2024-09-18T04:39:00.479 INFO:tasks.workunit.client.0.smithi115.stderr:+ test -n 18369 2024-09-18T04:39:00.479 INFO:tasks.workunit.client.0.smithi115.stderr:+ ps -p 18369 -C rbd-nbd 2024-09-18T04:39:00.479 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-09-18T04:39:00.489 INFO:tasks.workunit.client.0.smithi115.stdout: PID TTY TIME CMD 2024-09-18T04:39:00.489 INFO:tasks.workunit.client.0.smithi115.stdout: 18369 ? 00:00:00 rbd-nbd 2024-09-18T04:39:00.490 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo dd if=/tmp/tmp.WJMNJxIdAN/data of=/dev/nbd0 bs=1M oflag=direct 2024-09-18T04:39:00.490 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:39:00.490 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:39:00.490 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:00.490 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:00.490 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:39:00.490 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:39:00.490 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:00.491 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:39:00.491 INFO:tasks.workunit.client.0.smithi115.stderr:++ which dd 2024-09-18T04:39:00.492 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/bin/dd 2024-09-18T04:39:00.493 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:39:00.493 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.WJMNJxIdAN/data of=/dev/nbd0 bs=1M oflag=direct 2024-09-18T04:39:00.499 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:39:00.560 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:00.553+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:00.619 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:00.619 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:39:00.619 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:39:00.619 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:39:00.619 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.619 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:33 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:00.619 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.261182+0000", 2024-09-18T04:39:00.619 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 18.352750918000002, 2024-09-18T04:39:00.619 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.037082208999999998, 2024-09-18T04:39:00.620 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:00.620 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:00.620 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:00.620 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:00.620 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:00.620 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 33 2024-09-18T04:39:00.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.620 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:00.620 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.620 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:00.620 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.261182+0000", 2024-09-18T04:39:00.620 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.620 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.621 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:00.621 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.261182+0000", 2024-09-18T04:39:00.621 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.621 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.621 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.621 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:00.621 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.261180+0000", 2024-09-18T04:39:00.621 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:00.621 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.621 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.621 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:00.621 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278669+0000", 2024-09-18T04:39:00.621 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.017488877999999999 2024-09-18T04:39:00.621 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.622 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.622 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:00.622 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278689+0000", 2024-09-18T04:39:00.622 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0047999999999998e-05 2024-09-18T04:39:00.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.622 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.622 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:00.622 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278699+0000", 2024-09-18T04:39:00.622 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6800000000000005e-06 2024-09-18T04:39:00.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.622 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.622 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:00.622 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278775+0000", 2024-09-18T04:39:00.622 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6087999999999994e-05 2024-09-18T04:39:00.623 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.623 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.623 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:00.623 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278935+0000", 2024-09-18T04:39:00.623 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000159834 2024-09-18T04:39:00.623 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.623 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.623 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:00.623 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279064+0000", 2024-09-18T04:39:00.623 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012954199999999999 2024-09-18T04:39:00.623 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.623 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.623 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:00.624 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295562+0000", 2024-09-18T04:39:00.624 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016497583999999999 2024-09-18T04:39:00.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.624 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.624 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:00.624 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.298147+0000", 2024-09-18T04:39:00.624 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002584982 2024-09-18T04:39:00.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.624 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.624 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:00.624 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.298217+0000", 2024-09-18T04:39:00.624 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0400000000000004e-05 2024-09-18T04:39:00.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.624 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.625 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:00.625 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.298265+0000", 2024-09-18T04:39:00.625 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7580000000000002e-05 2024-09-18T04:39:00.625 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.625 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:00.625 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.625 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.625 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.625 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:34 2.3 e51/24 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head v 51'3, mlcod=18'1)", 2024-09-18T04:39:00.625 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.273067+0000", 2024-09-18T04:39:00.625 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 18.340866127999998, 2024-09-18T04:39:00.625 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.021784103999999999, 2024-09-18T04:39:00.626 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:00.626 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:00.626 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:00.626 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.626 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:00.626 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.273067+0000", 2024-09-18T04:39:00.626 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.626 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.626 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.626 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:00.626 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.273067+0000", 2024-09-18T04:39:00.626 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.626 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.627 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.627 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:00.627 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.273060+0000", 2024-09-18T04:39:00.627 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999924 2024-09-18T04:39:00.627 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.627 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.627 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:00.627 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278596+0000", 2024-09-18T04:39:00.627 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055365839999999998 2024-09-18T04:39:00.628 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.628 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.628 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:00.628 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278637+0000", 2024-09-18T04:39:00.628 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0596000000000002e-05 2024-09-18T04:39:00.628 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.628 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.628 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:00.628 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278653+0000", 2024-09-18T04:39:00.628 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6626999999999999e-05 2024-09-18T04:39:00.628 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.628 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.628 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:00.629 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278748+0000", 2024-09-18T04:39:00.629 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4483999999999999e-05 2024-09-18T04:39:00.629 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.629 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.629 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:00.629 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278799+0000", 2024-09-18T04:39:00.629 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1484999999999999e-05 2024-09-18T04:39:00.629 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.629 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.629 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:00.629 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.294785+0000", 2024-09-18T04:39:00.629 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015985256999999999 2024-09-18T04:39:00.629 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.630 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.630 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:00.630 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.294851+0000", 2024-09-18T04:39:00.630 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6728000000000002e-05 2024-09-18T04:39:00.630 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.630 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:00.630 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.630 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.630 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.630 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:37 2.5 2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:00.630 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.278722+0000", 2024-09-18T04:39:00.630 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 18.335211079, 2024-09-18T04:39:00.630 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.026323801000000001, 2024-09-18T04:39:00.630 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:00.631 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:00.631 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:00.631 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:00.631 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:00.631 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 37 2024-09-18T04:39:00.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.631 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:00.631 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.631 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:00.631 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278722+0000", 2024-09-18T04:39:00.631 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.631 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.632 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:00.632 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278722+0000", 2024-09-18T04:39:00.632 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.632 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.632 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.632 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:00.632 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278720+0000", 2024-09-18T04:39:00.632 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:00.632 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.632 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.632 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:00.632 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.291875+0000", 2024-09-18T04:39:00.632 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013154414 2024-09-18T04:39:00.632 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.633 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.633 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:00.633 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.291901+0000", 2024-09-18T04:39:00.633 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6339999999999999e-05 2024-09-18T04:39:00.633 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.633 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.633 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:00.633 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.291914+0000", 2024-09-18T04:39:00.633 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3043e-05 2024-09-18T04:39:00.633 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.633 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.633 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.291975+0000", 2024-09-18T04:39:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0949999999999998e-05 2024-09-18T04:39:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292173+0000", 2024-09-18T04:39:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019825499999999999 2024-09-18T04:39:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292300+0000", 2024-09-18T04:39:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000126616 2024-09-18T04:39:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304334+0000", 2024-09-18T04:39:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012033934 2024-09-18T04:39:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304901+0000", 2024-09-18T04:39:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00056694099999999995 2024-09-18T04:39:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304981+0000", 2024-09-18T04:39:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9974000000000004e-05 2024-09-18T04:39:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.305046+0000", 2024-09-18T04:39:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5184000000000005e-05 2024-09-18T04:39:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:49 2.5 2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.314271+0000", 2024-09-18T04:39:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 18.299662476999998, 2024-09-18T04:39:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.029779953000000001, 2024-09-18T04:39:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 49 2024-09-18T04:39:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.314271+0000", 2024-09-18T04:39:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.314271+0000", 2024-09-18T04:39:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.314269+0000", 2024-09-18T04:39:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.324780+0000", 2024-09-18T04:39:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010511609999999999 2024-09-18T04:39:00.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.639 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.639 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:00.639 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.324821+0000", 2024-09-18T04:39:00.639 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0899000000000003e-05 2024-09-18T04:39:00.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.639 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.639 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:00.639 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.324840+0000", 2024-09-18T04:39:00.639 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8947e-05 2024-09-18T04:39:00.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.639 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.639 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:00.640 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.324936+0000", 2024-09-18T04:39:00.640 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6342999999999996e-05 2024-09-18T04:39:00.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.640 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.640 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:00.640 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325284+0000", 2024-09-18T04:39:00.640 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000347146 2024-09-18T04:39:00.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.640 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.640 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:00.640 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325586+0000", 2024-09-18T04:39:00.640 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00030199799999999999 2024-09-18T04:39:00.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.640 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.641 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:00.641 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343710+0000", 2024-09-18T04:39:00.641 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018124583999999999 2024-09-18T04:39:00.641 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.641 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.641 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:00.641 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343991+0000", 2024-09-18T04:39:00.641 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00028085600000000001 2024-09-18T04:39:00.641 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.641 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.641 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:00.641 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344028+0000", 2024-09-18T04:39:00.641 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6637000000000002e-05 2024-09-18T04:39:00.642 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.642 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.642 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:00.642 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344051+0000", 2024-09-18T04:39:00.642 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3260000000000001e-05 2024-09-18T04:39:00.642 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.642 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:00.642 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.642 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.642 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.642 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:50 2.3 e51/24 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head v 51'4, mlcod=51'2)", 2024-09-18T04:39:00.642 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.323066+0000", 2024-09-18T04:39:00.642 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 18.290867083999998, 2024-09-18T04:39:00.642 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.022135774, 2024-09-18T04:39:00.643 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:00.643 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:00.643 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:00.643 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.643 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:00.643 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.323066+0000", 2024-09-18T04:39:00.643 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.643 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.643 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.643 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:00.643 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.323066+0000", 2024-09-18T04:39:00.643 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.643 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.643 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.643 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:00.644 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.323065+0000", 2024-09-18T04:39:00.644 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:00.644 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.644 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.644 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:00.644 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.328664+0000", 2024-09-18T04:39:00.644 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055993600000000003 2024-09-18T04:39:00.644 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.644 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.644 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:00.644 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.328692+0000", 2024-09-18T04:39:00.644 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7654e-05 2024-09-18T04:39:00.644 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.645 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.645 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:00.645 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.328706+0000", 2024-09-18T04:39:00.645 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4097000000000001e-05 2024-09-18T04:39:00.645 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.645 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.645 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:00.645 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.328816+0000", 2024-09-18T04:39:00.645 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000110281 2024-09-18T04:39:00.645 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.645 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.645 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:00.645 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.328848+0000", 2024-09-18T04:39:00.645 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1837e-05 2024-09-18T04:39:00.645 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.646 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.646 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:00.646 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345160+0000", 2024-09-18T04:39:00.646 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016312099 2024-09-18T04:39:00.646 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.646 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.646 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:00.646 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345202+0000", 2024-09-18T04:39:00.646 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1857999999999998e-05 2024-09-18T04:39:00.646 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.646 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:00.646 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.646 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.646 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.646 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:58 2.5 2:b227e6b1:::rbd_data.125fc09df6d1.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:00.647 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.351564+0000", 2024-09-18T04:39:00.647 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 18.262369631999999, 2024-09-18T04:39:00.647 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.032125956999999997, 2024-09-18T04:39:00.647 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:00.647 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:00.647 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:00.647 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:00.647 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:00.647 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 58 2024-09-18T04:39:00.647 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.647 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:00.647 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.647 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:00.647 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.351564+0000", 2024-09-18T04:39:00.648 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.648 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.648 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.648 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:00.648 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.351564+0000", 2024-09-18T04:39:00.648 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.648 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.648 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.648 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:00.648 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.351561+0000", 2024-09-18T04:39:00.648 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:00.648 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.648 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.648 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:00.649 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364390+0000", 2024-09-18T04:39:00.649 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012829063999999999 2024-09-18T04:39:00.649 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.649 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.649 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:00.649 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364418+0000", 2024-09-18T04:39:00.649 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7307000000000001e-05 2024-09-18T04:39:00.649 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.649 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.649 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:00.649 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364428+0000", 2024-09-18T04:39:00.649 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0675e-05 2024-09-18T04:39:00.649 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.649 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.649 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:00.650 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364523+0000", 2024-09-18T04:39:00.650 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4698999999999996e-05 2024-09-18T04:39:00.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.650 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.650 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:00.650 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364785+0000", 2024-09-18T04:39:00.650 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00026170099999999998 2024-09-18T04:39:00.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.650 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.650 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:00.650 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.365039+0000", 2024-09-18T04:39:00.650 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00025462299999999999 2024-09-18T04:39:00.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.650 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.650 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:00.651 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383143+0000", 2024-09-18T04:39:00.651 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018103768999999999 2024-09-18T04:39:00.651 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.651 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.651 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:00.651 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383582+0000", 2024-09-18T04:39:00.651 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00043860600000000002 2024-09-18T04:39:00.651 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.651 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.651 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:00.651 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383644+0000", 2024-09-18T04:39:00.651 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2682999999999996e-05 2024-09-18T04:39:00.651 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.651 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.652 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:00.652 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383690+0000", 2024-09-18T04:39:00.652 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5247999999999998e-05 2024-09-18T04:39:00.652 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.652 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:00.652 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.652 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.652 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.652 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:33 2.1 2:8b40f1c5:ns::rbd_data.133e813ab5bc.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:00.652 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.523765+0000", 2024-09-18T04:39:00.652 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 18.090168592000001, 2024-09-18T04:39:00.652 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027472973000000001, 2024-09-18T04:39:00.652 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:00.652 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:00.653 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:00.653 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:00.653 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:00.653 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 33 2024-09-18T04:39:00.653 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.653 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:00.653 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.653 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:00.653 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.523765+0000", 2024-09-18T04:39:00.653 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.653 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.653 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.653 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:00.653 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.523765+0000", 2024-09-18T04:39:00.654 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.654 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.654 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:00.654 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.523762+0000", 2024-09-18T04:39:00.654 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:00.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.654 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.654 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:00.654 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531127+0000", 2024-09-18T04:39:00.654 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0073644560000000001 2024-09-18T04:39:00.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.655 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.655 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:00.655 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531189+0000", 2024-09-18T04:39:00.655 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2315000000000002e-05 2024-09-18T04:39:00.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.655 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.655 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:00.655 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531211+0000", 2024-09-18T04:39:00.655 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1542000000000001e-05 2024-09-18T04:39:00.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.655 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.655 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:00.655 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531314+0000", 2024-09-18T04:39:00.655 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000103582 2024-09-18T04:39:00.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.656 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.656 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:00.656 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531493+0000", 2024-09-18T04:39:00.656 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000178433 2024-09-18T04:39:00.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.656 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.656 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:00.656 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531653+0000", 2024-09-18T04:39:00.656 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000160388 2024-09-18T04:39:00.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.656 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.656 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:00.656 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.547345+0000", 2024-09-18T04:39:00.657 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015691742000000002 2024-09-18T04:39:00.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.657 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.657 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:00.657 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551076+0000", 2024-09-18T04:39:00.657 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003730957 2024-09-18T04:39:00.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.657 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.657 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:00.657 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551176+0000", 2024-09-18T04:39:00.657 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010002599999999999 2024-09-18T04:39:00.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.657 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.658 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:00.658 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551238+0000", 2024-09-18T04:39:00.658 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1945000000000004e-05 2024-09-18T04:39:00.658 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.658 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:00.658 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.658 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.658 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.658 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:36 2.5 2:bd5b8dd8:ns::rbd_data.133e813ab5bc.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:00.658 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.531253+0000", 2024-09-18T04:39:00.658 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 18.082680282999998, 2024-09-18T04:39:00.658 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027295212999999999, 2024-09-18T04:39:00.658 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:00.659 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:00.659 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:00.659 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:00.659 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:00.659 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 36 2024-09-18T04:39:00.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.659 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:00.659 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.659 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:00.659 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531253+0000", 2024-09-18T04:39:00.659 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.660 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.660 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.660 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:00.660 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531253+0000", 2024-09-18T04:39:00.660 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.660 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.660 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.660 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:00.660 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531251+0000", 2024-09-18T04:39:00.660 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:00.660 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.660 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.660 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:00.660 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536540+0000", 2024-09-18T04:39:00.661 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0052885839999999998 2024-09-18T04:39:00.661 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.661 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.661 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:00.661 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536604+0000", 2024-09-18T04:39:00.661 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4824000000000001e-05 2024-09-18T04:39:00.661 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.661 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.661 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:00.661 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536621+0000", 2024-09-18T04:39:00.661 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6215999999999999e-05 2024-09-18T04:39:00.661 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.661 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.662 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:00.662 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536768+0000", 2024-09-18T04:39:00.662 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014767900000000001 2024-09-18T04:39:00.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.662 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.662 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:00.662 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537084+0000", 2024-09-18T04:39:00.662 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00031553499999999998 2024-09-18T04:39:00.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.662 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.662 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:00.662 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537300+0000", 2024-09-18T04:39:00.662 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000216145 2024-09-18T04:39:00.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.663 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.663 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:00.663 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555326+0000", 2024-09-18T04:39:00.663 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018025861000000001 2024-09-18T04:39:00.663 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.663 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.663 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:00.663 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558449+0000", 2024-09-18T04:39:00.663 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0031229750000000001 2024-09-18T04:39:00.663 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.663 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.663 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:00.663 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558502+0000", 2024-09-18T04:39:00.663 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3572000000000001e-05 2024-09-18T04:39:00.663 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.663 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.664 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:00.664 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558548+0000", 2024-09-18T04:39:00.664 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5921999999999998e-05 2024-09-18T04:39:00.664 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.664 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:00.664 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.664 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.664 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.664 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:37 2.5 2:bf39b8e4:ns::rbd_data.133e813ab5bc.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:00.664 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.536664+0000", 2024-09-18T04:39:00.664 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 18.077268867000001, 2024-09-18T04:39:00.664 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.026331765, 2024-09-18T04:39:00.664 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:00.664 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:00.665 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:00.665 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:00.665 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:00.665 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 37 2024-09-18T04:39:00.665 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.665 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:00.665 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.665 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:00.665 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536664+0000", 2024-09-18T04:39:00.665 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.665 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.665 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.665 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:00.666 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536664+0000", 2024-09-18T04:39:00.666 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.666 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.666 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.666 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:00.666 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536662+0000", 2024-09-18T04:39:00.666 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:00.666 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.666 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.667 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:00.667 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.543699+0000", 2024-09-18T04:39:00.667 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0070367119999999997 2024-09-18T04:39:00.667 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.667 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.667 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:00.667 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.543749+0000", 2024-09-18T04:39:00.667 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0476999999999998e-05 2024-09-18T04:39:00.667 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.667 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.667 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:00.667 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.543800+0000", 2024-09-18T04:39:00.667 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0822999999999998e-05 2024-09-18T04:39:00.668 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.668 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.668 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:00.668 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.547868+0000", 2024-09-18T04:39:00.668 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0040679339999999996 2024-09-18T04:39:00.668 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.668 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.668 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:00.668 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548087+0000", 2024-09-18T04:39:00.668 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000218994 2024-09-18T04:39:00.668 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.668 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.668 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:00.669 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548383+0000", 2024-09-18T04:39:00.669 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000295588 2024-09-18T04:39:00.669 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.669 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.669 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:00.669 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562622+0000", 2024-09-18T04:39:00.669 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014239915000000001 2024-09-18T04:39:00.669 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.669 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.669 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:00.669 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562924+0000", 2024-09-18T04:39:00.669 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00030127600000000001 2024-09-18T04:39:00.669 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.670 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.670 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:00.670 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562957+0000", 2024-09-18T04:39:00.670 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3365999999999999e-05 2024-09-18T04:39:00.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.670 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.670 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:00.670 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562996+0000", 2024-09-18T04:39:00.670 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9153999999999998e-05 2024-09-18T04:39:00.670 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.670 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:00.670 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.670 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.671 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:50 2.1 2:8c706374:ns::rbd_data.133e813ab5bc.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:00.671 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.570513+0000", 2024-09-18T04:39:00.671 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 18.043420585, 2024-09-18T04:39:00.671 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.024182995999999998, 2024-09-18T04:39:00.671 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:00.671 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:00.671 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:00.671 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:00.671 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:00.671 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 50 2024-09-18T04:39:00.671 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.671 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:00.671 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.671 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:00.672 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570513+0000", 2024-09-18T04:39:00.672 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.672 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.672 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:00.672 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570513+0000", 2024-09-18T04:39:00.672 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.672 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.672 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:00.672 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570511+0000", 2024-09-18T04:39:00.672 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:00.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.672 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.673 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:00.673 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.576327+0000", 2024-09-18T04:39:00.673 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0058162530000000004 2024-09-18T04:39:00.673 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.673 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.673 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:00.673 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.576355+0000", 2024-09-18T04:39:00.673 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8030000000000001e-05 2024-09-18T04:39:00.673 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.673 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.673 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:00.673 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.576366+0000", 2024-09-18T04:39:00.673 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1139e-05 2024-09-18T04:39:00.673 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.674 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.674 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:00.674 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.576458+0000", 2024-09-18T04:39:00.674 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1780999999999994e-05 2024-09-18T04:39:00.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.674 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.674 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:00.674 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.576701+0000", 2024-09-18T04:39:00.674 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00024303899999999999 2024-09-18T04:39:00.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.674 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.674 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:00.674 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.576876+0000", 2024-09-18T04:39:00.674 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000174637 2024-09-18T04:39:00.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.675 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.675 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:00.675 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588432+0000", 2024-09-18T04:39:00.675 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011555943000000001 2024-09-18T04:39:00.675 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.675 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.675 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:00.675 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594573+0000", 2024-09-18T04:39:00.675 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0061408670000000004 2024-09-18T04:39:00.675 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.675 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.675 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:00.675 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594640+0000", 2024-09-18T04:39:00.675 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7404999999999996e-05 2024-09-18T04:39:00.676 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.676 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.676 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:00.676 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594696+0000", 2024-09-18T04:39:00.676 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5612999999999998e-05 2024-09-18T04:39:00.676 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.676 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:00.676 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.676 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.676 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.676 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:60 2.5 2:a06f51e4:ns::rbd_data.133e813ab5bc.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:00.676 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.604753+0000", 2024-09-18T04:39:00.676 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 18.009180466, 2024-09-18T04:39:00.676 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014400181, 2024-09-18T04:39:00.676 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:00.677 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:00.677 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:00.677 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:00.677 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:00.677 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 60 2024-09-18T04:39:00.677 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.677 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:00.677 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.677 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:00.677 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.604753+0000", 2024-09-18T04:39:00.677 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.677 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.677 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.677 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:00.677 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.604753+0000", 2024-09-18T04:39:00.678 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.678 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.678 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.678 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:00.678 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.604751+0000", 2024-09-18T04:39:00.678 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:00.678 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.678 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.678 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:00.678 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.608980+0000", 2024-09-18T04:39:00.678 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0042292420000000002 2024-09-18T04:39:00.678 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.678 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.678 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:00.678 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609014+0000", 2024-09-18T04:39:00.679 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3572999999999999e-05 2024-09-18T04:39:00.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.679 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.679 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:00.679 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609032+0000", 2024-09-18T04:39:00.679 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8150000000000001e-05 2024-09-18T04:39:00.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.679 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.679 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:00.679 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609090+0000", 2024-09-18T04:39:00.679 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8214999999999997e-05 2024-09-18T04:39:00.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.679 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.679 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:00.680 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609235+0000", 2024-09-18T04:39:00.680 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014437799999999999 2024-09-18T04:39:00.680 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.680 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.680 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:00.680 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609360+0000", 2024-09-18T04:39:00.680 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012526099999999999 2024-09-18T04:39:00.680 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.680 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.680 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:00.680 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.614952+0000", 2024-09-18T04:39:00.680 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055922300000000001 2024-09-18T04:39:00.681 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.681 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.681 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:00.681 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.619070+0000", 2024-09-18T04:39:00.681 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0041178250000000003 2024-09-18T04:39:00.681 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.681 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.681 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:00.681 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.619117+0000", 2024-09-18T04:39:00.681 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7275999999999999e-05 2024-09-18T04:39:00.681 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.681 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.681 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:00.681 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.619153+0000", 2024-09-18T04:39:00.682 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5970000000000003e-05 2024-09-18T04:39:00.682 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.682 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:00.682 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.682 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.682 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.682 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4805.0:592 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [write 393216~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:00.682 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.023386+0000", 2024-09-18T04:39:00.682 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 9.5905477609999998, 2024-09-18T04:39:00.682 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0086813870000000005, 2024-09-18T04:39:00.682 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:00.682 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:00.682 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:00.683 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4805", 2024-09-18T04:39:00.683 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:00.683 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 592 2024-09-18T04:39:00.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.683 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:00.683 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.683 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:00.683 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023386+0000", 2024-09-18T04:39:00.683 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.683 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.683 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:00.683 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023386+0000", 2024-09-18T04:39:00.683 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.683 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.684 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:00.684 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023384+0000", 2024-09-18T04:39:00.684 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:00.684 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.684 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.684 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:00.684 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023616+0000", 2024-09-18T04:39:00.684 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00023165600000000001 2024-09-18T04:39:00.684 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.684 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.684 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:00.684 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023637+0000", 2024-09-18T04:39:00.684 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1163e-05 2024-09-18T04:39:00.684 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.685 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.685 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:00.685 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023652+0000", 2024-09-18T04:39:00.685 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4562000000000001e-05 2024-09-18T04:39:00.685 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.685 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.685 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:00.685 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.027732+0000", 2024-09-18T04:39:00.685 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0040803929999999999 2024-09-18T04:39:00.685 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.685 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.685 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:00.685 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.027841+0000", 2024-09-18T04:39:00.686 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000108978 2024-09-18T04:39:00.686 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.686 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.686 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:00.686 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.027990+0000", 2024-09-18T04:39:00.686 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014852999999999999 2024-09-18T04:39:00.686 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.686 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.686 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:00.686 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.031930+0000", 2024-09-18T04:39:00.686 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0039406299999999997 2024-09-18T04:39:00.686 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.686 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.687 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:00.687 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.031978+0000", 2024-09-18T04:39:00.687 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7817e-05 2024-09-18T04:39:00.687 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.687 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.687 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:00.687 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.032019+0000", 2024-09-18T04:39:00.687 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0531999999999998e-05 2024-09-18T04:39:00.687 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.687 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.687 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:00.687 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.032067+0000", 2024-09-18T04:39:00.687 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8371000000000002e-05 2024-09-18T04:39:00.687 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.688 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:00.688 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.688 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.688 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.688 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4805.0:593 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [write 524288~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:00.688 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.023665+0000", 2024-09-18T04:39:00.688 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 9.590267914, 2024-09-18T04:39:00.688 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0085518090000000005, 2024-09-18T04:39:00.688 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:00.688 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:00.688 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:00.688 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4805", 2024-09-18T04:39:00.688 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:00.689 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 593 2024-09-18T04:39:00.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.689 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:00.689 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.689 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:00.689 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023665+0000", 2024-09-18T04:39:00.689 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.689 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.689 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:00.689 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023665+0000", 2024-09-18T04:39:00.689 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.689 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.689 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:00.690 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023664+0000", 2024-09-18T04:39:00.690 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:00.690 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.690 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.690 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:00.690 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023875+0000", 2024-09-18T04:39:00.690 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000210862 2024-09-18T04:39:00.690 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.690 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.690 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:00.690 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023906+0000", 2024-09-18T04:39:00.690 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0660000000000001e-05 2024-09-18T04:39:00.690 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.691 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.691 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:00.691 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023922+0000", 2024-09-18T04:39:00.691 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5831000000000001e-05 2024-09-18T04:39:00.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.691 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.691 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:00.691 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.029603+0000", 2024-09-18T04:39:00.691 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0056816719999999996 2024-09-18T04:39:00.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.691 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.691 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:00.691 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.029766+0000", 2024-09-18T04:39:00.692 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000162903 2024-09-18T04:39:00.692 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.692 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.692 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:00.692 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.030042+0000", 2024-09-18T04:39:00.692 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00027561399999999998 2024-09-18T04:39:00.692 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.692 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.692 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:00.692 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.032127+0000", 2024-09-18T04:39:00.692 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020850920000000002 2024-09-18T04:39:00.692 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.692 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.692 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:00.692 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.032137+0000", 2024-09-18T04:39:00.693 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0312000000000001e-05 2024-09-18T04:39:00.693 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.693 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.693 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:00.693 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.032183+0000", 2024-09-18T04:39:00.693 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6020999999999999e-05 2024-09-18T04:39:00.693 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.693 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.693 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:00.693 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.032217+0000", 2024-09-18T04:39:00.693 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4109999999999997e-05 2024-09-18T04:39:00.693 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.693 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:00.693 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.694 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.694 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.694 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4805.0:594 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [write 655360~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:00.694 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.023941+0000", 2024-09-18T04:39:00.694 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 9.5899923759999997, 2024-09-18T04:39:00.694 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010767616000000001, 2024-09-18T04:39:00.694 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:00.694 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:00.694 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:00.694 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4805", 2024-09-18T04:39:00.694 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:00.694 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 594 2024-09-18T04:39:00.694 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.694 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:00.695 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.695 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:00.695 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023941+0000", 2024-09-18T04:39:00.695 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.695 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.695 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:00.695 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023941+0000", 2024-09-18T04:39:00.695 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.695 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.695 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:00.695 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023940+0000", 2024-09-18T04:39:00.695 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:00.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.696 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.696 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:00.696 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024163+0000", 2024-09-18T04:39:00.696 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000223017 2024-09-18T04:39:00.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.696 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.696 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:00.696 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024188+0000", 2024-09-18T04:39:00.696 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5174e-05 2024-09-18T04:39:00.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.696 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.696 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:00.697 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024214+0000", 2024-09-18T04:39:00.697 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5782000000000002e-05 2024-09-18T04:39:00.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.697 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.697 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:00.697 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.031116+0000", 2024-09-18T04:39:00.697 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.006902282 2024-09-18T04:39:00.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.697 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.697 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:00.697 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.031199+0000", 2024-09-18T04:39:00.697 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3203999999999998e-05 2024-09-18T04:39:00.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.697 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.698 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:00.698 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.031315+0000", 2024-09-18T04:39:00.698 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000115742 2024-09-18T04:39:00.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.698 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.698 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:00.698 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.033780+0000", 2024-09-18T04:39:00.698 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002465058 2024-09-18T04:39:00.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.698 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.698 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:00.698 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.034630+0000", 2024-09-18T04:39:00.698 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00084966800000000004 2024-09-18T04:39:00.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.699 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.699 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:00.699 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.034682+0000", 2024-09-18T04:39:00.699 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2562999999999998e-05 2024-09-18T04:39:00.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.699 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.699 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:00.699 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.034709+0000", 2024-09-18T04:39:00.699 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.62e-05 2024-09-18T04:39:00.699 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.699 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:00.699 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.699 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.700 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4805.0:595 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [write 786432~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:00.700 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.024233+0000", 2024-09-18T04:39:00.700 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 9.5897000010000006, 2024-09-18T04:39:00.700 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010599558, 2024-09-18T04:39:00.700 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:00.700 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:00.700 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:00.700 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4805", 2024-09-18T04:39:00.700 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:00.700 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 595 2024-09-18T04:39:00.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.700 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:00.700 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.701 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:00.701 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024233+0000", 2024-09-18T04:39:00.701 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.701 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.701 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:00.701 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024233+0000", 2024-09-18T04:39:00.701 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.701 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.701 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:00.701 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024232+0000", 2024-09-18T04:39:00.701 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:00.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.701 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.702 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:00.702 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024449+0000", 2024-09-18T04:39:00.702 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000216868 2024-09-18T04:39:00.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.702 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.702 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:00.702 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024474+0000", 2024-09-18T04:39:00.702 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4598999999999999e-05 2024-09-18T04:39:00.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.702 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.702 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:00.702 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024487+0000", 2024-09-18T04:39:00.703 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3451999999999999e-05 2024-09-18T04:39:00.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.703 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.703 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:00.703 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.032272+0000", 2024-09-18T04:39:00.703 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0077845709999999997 2024-09-18T04:39:00.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.703 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.703 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:00.703 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.032332+0000", 2024-09-18T04:39:00.703 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0736999999999998e-05 2024-09-18T04:39:00.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.703 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.703 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:00.703 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.032421+0000", 2024-09-18T04:39:00.704 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8993000000000007e-05 2024-09-18T04:39:00.704 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.704 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.704 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:00.704 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.033786+0000", 2024-09-18T04:39:00.704 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001364451 2024-09-18T04:39:00.704 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.704 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.704 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:00.704 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.034752+0000", 2024-09-18T04:39:00.704 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00096642400000000002 2024-09-18T04:39:00.704 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.704 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.705 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:00.705 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.034789+0000", 2024-09-18T04:39:00.705 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6899e-05 2024-09-18T04:39:00.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.705 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.705 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:00.705 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.034833+0000", 2024-09-18T04:39:00.705 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3846e-05 2024-09-18T04:39:00.705 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.705 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:00.705 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.705 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.705 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4805.0:596 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [write 917504~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:00.706 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.024500+0000", 2024-09-18T04:39:00.706 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 9.5894332159999998, 2024-09-18T04:39:00.706 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010770534, 2024-09-18T04:39:00.706 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:00.706 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:00.706 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:00.706 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4805", 2024-09-18T04:39:00.706 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:00.706 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 596 2024-09-18T04:39:00.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.706 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:00.706 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.706 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:00.707 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024500+0000", 2024-09-18T04:39:00.707 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.707 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.707 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:00.707 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024500+0000", 2024-09-18T04:39:00.707 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.707 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.707 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:00.707 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024499+0000", 2024-09-18T04:39:00.707 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:00.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.708 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.708 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:00.708 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024700+0000", 2024-09-18T04:39:00.708 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00020038599999999999 2024-09-18T04:39:00.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.708 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.708 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:00.708 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024729+0000", 2024-09-18T04:39:00.708 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9224999999999999e-05 2024-09-18T04:39:00.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.708 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.708 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:00.708 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024776+0000", 2024-09-18T04:39:00.709 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7385000000000001e-05 2024-09-18T04:39:00.709 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.709 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.709 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:00.709 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.032998+0000", 2024-09-18T04:39:00.709 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0082216189999999995 2024-09-18T04:39:00.709 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.709 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.709 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:00.709 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.033076+0000", 2024-09-18T04:39:00.709 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8668e-05 2024-09-18T04:39:00.709 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.709 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.710 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:00.710 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.033167+0000", 2024-09-18T04:39:00.710 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0728000000000003e-05 2024-09-18T04:39:00.710 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.710 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.710 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:00.710 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.034399+0000", 2024-09-18T04:39:00.710 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00123234 2024-09-18T04:39:00.710 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.710 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.710 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:00.710 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.035206+0000", 2024-09-18T04:39:00.710 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00080701799999999995 2024-09-18T04:39:00.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.711 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.711 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:00.711 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.035244+0000", 2024-09-18T04:39:00.711 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7487999999999998e-05 2024-09-18T04:39:00.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.711 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.711 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:00.711 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.035271+0000", 2024-09-18T04:39:00.711 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6709000000000001e-05 2024-09-18T04:39:00.711 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.711 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:00.711 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.712 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.712 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.712 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4805.0:597 2.1 2:9370a598:::rbd_header.125fc09df6d1:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:00.712 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.025206+0000", 2024-09-18T04:39:00.712 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 9.5887271149999993, 2024-09-18T04:39:00.712 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0091536900000000008, 2024-09-18T04:39:00.712 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:00.712 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:00.712 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:00.712 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4805", 2024-09-18T04:39:00.712 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:00.712 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 597 2024-09-18T04:39:00.712 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.713 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:00.713 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.713 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:00.713 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.025206+0000", 2024-09-18T04:39:00.713 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.713 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.713 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.713 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:00.713 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.025206+0000", 2024-09-18T04:39:00.713 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.713 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.713 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.713 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:00.713 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.025204+0000", 2024-09-18T04:39:00.714 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:00.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.714 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.714 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:00.714 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.025216+0000", 2024-09-18T04:39:00.714 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2150999999999999e-05 2024-09-18T04:39:00.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.714 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.714 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:00.714 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.025237+0000", 2024-09-18T04:39:00.714 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0740000000000001e-05 2024-09-18T04:39:00.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.714 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.715 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:00.715 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.025254+0000", 2024-09-18T04:39:00.715 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6731e-05 2024-09-18T04:39:00.715 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.715 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.715 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:00.715 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.034049+0000", 2024-09-18T04:39:00.715 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0087952810000000003 2024-09-18T04:39:00.715 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.715 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.715 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:00.715 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.034202+0000", 2024-09-18T04:39:00.715 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0001526 2024-09-18T04:39:00.715 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.716 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.716 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:00.716 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.034360+0000", 2024-09-18T04:39:00.716 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015809299999999999 2024-09-18T04:39:00.716 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.716 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:00.716 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.716 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.716 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4972.0:14 2.5 2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:head [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:00.716 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.949251+0000", 2024-09-18T04:39:00.716 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 8.6646821690000007, 2024-09-18T04:39:00.716 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0098324480000000006, 2024-09-18T04:39:00.716 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:00.717 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:00.717 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:00.717 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4972", 2024-09-18T04:39:00.717 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2280251472", 2024-09-18T04:39:00.717 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 14 2024-09-18T04:39:00.717 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.717 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:00.717 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.717 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:00.717 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949251+0000", 2024-09-18T04:39:00.717 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.717 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.717 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.717 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:00.718 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949251+0000", 2024-09-18T04:39:00.718 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.718 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.718 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.718 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:00.718 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949250+0000", 2024-09-18T04:39:00.718 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:00.718 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.718 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.718 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:00.718 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949258+0000", 2024-09-18T04:39:00.718 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0590000000000003e-06 2024-09-18T04:39:00.718 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.718 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.719 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:00.719 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949269+0000", 2024-09-18T04:39:00.719 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0607e-05 2024-09-18T04:39:00.719 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.719 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.719 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:00.719 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949277+0000", 2024-09-18T04:39:00.719 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8269999999999998e-06 2024-09-18T04:39:00.719 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.719 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.719 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:00.719 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949394+0000", 2024-09-18T04:39:00.719 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000117113 2024-09-18T04:39:00.719 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.719 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.719 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:00.720 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949527+0000", 2024-09-18T04:39:00.720 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013288300000000001 2024-09-18T04:39:00.720 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.720 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.720 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:00.720 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.959084+0000", 2024-09-18T04:39:00.720 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0095569550000000007 2024-09-18T04:39:00.720 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.720 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:00.720 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.720 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.720 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.720 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4972.0:20 2.5 2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:00.720 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.949296+0000", 2024-09-18T04:39:00.721 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 8.6646378629999994, 2024-09-18T04:39:00.721 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014709732, 2024-09-18T04:39:00.721 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:00.721 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:00.721 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:00.721 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4972", 2024-09-18T04:39:00.721 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2280251472", 2024-09-18T04:39:00.721 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 20 2024-09-18T04:39:00.721 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.721 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:00.721 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.721 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:00.721 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949296+0000", 2024-09-18T04:39:00.721 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.721 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.722 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.722 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:00.722 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949296+0000", 2024-09-18T04:39:00.722 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.722 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.722 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.722 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:00.722 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949295+0000", 2024-09-18T04:39:00.722 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:00.722 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.722 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.722 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:00.722 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949301+0000", 2024-09-18T04:39:00.722 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4829999999999996e-06 2024-09-18T04:39:00.722 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.723 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.723 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:00.723 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949317+0000", 2024-09-18T04:39:00.723 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5866999999999999e-05 2024-09-18T04:39:00.723 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.723 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.723 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:00.723 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949331+0000", 2024-09-18T04:39:00.723 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3485999999999999e-05 2024-09-18T04:39:00.723 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.723 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.723 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:00.723 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.959137+0000", 2024-09-18T04:39:00.723 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0098066520000000008 2024-09-18T04:39:00.723 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.723 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.724 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:00.724 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.959251+0000", 2024-09-18T04:39:00.724 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000113783 2024-09-18T04:39:00.724 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.724 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.724 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:00.724 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.964005+0000", 2024-09-18T04:39:00.724 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0047542080000000002 2024-09-18T04:39:00.724 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.724 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:00.724 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.724 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.724 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.725 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4970.0:37 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [write 1966080~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:00.725 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.516050+0000", 2024-09-18T04:39:00.725 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.097883005999999995, 2024-09-18T04:39:00.725 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0083751499999999996, 2024-09-18T04:39:00.725 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:00.725 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:00.725 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:00.725 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4970", 2024-09-18T04:39:00.725 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:00.725 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 37 2024-09-18T04:39:00.725 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.725 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:00.726 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.726 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:00.726 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.516050+0000", 2024-09-18T04:39:00.726 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.726 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.726 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.726 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:00.726 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.516050+0000", 2024-09-18T04:39:00.726 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:00.726 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.726 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.726 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:00.726 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.516050+0000", 2024-09-18T04:39:00.726 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:00.726 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.727 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.727 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:00.727 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.516150+0000", 2024-09-18T04:39:00.727 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000100159 2024-09-18T04:39:00.727 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.727 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.727 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:00.727 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.516157+0000", 2024-09-18T04:39:00.727 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7710000000000001e-06 2024-09-18T04:39:00.727 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.727 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.727 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:00.727 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.516163+0000", 2024-09-18T04:39:00.728 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.596e-06 2024-09-18T04:39:00.728 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.728 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.728 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:00.728 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.522975+0000", 2024-09-18T04:39:00.728 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0068118140000000002 2024-09-18T04:39:00.728 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.728 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.728 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:00.728 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.523028+0000", 2024-09-18T04:39:00.728 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2998999999999999e-05 2024-09-18T04:39:00.728 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.728 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.728 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:00.729 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.523107+0000", 2024-09-18T04:39:00.729 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9357000000000006e-05 2024-09-18T04:39:00.729 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.729 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.729 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:00.729 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.524167+0000", 2024-09-18T04:39:00.729 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0010592329999999999 2024-09-18T04:39:00.729 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.729 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.729 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:00.729 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.524369+0000", 2024-09-18T04:39:00.729 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00020209000000000001 2024-09-18T04:39:00.729 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.729 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.730 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:00.730 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.524402+0000", 2024-09-18T04:39:00.730 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3685e-05 2024-09-18T04:39:00.730 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:00.730 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:00.730 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:00.730 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.524426+0000", 2024-09-18T04:39:00.730 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3178000000000001e-05 2024-09-18T04:39:00.730 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.730 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:00.730 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.730 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:00.730 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:00.730 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:00.732 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:39:00.732 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:00.725+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:39:00.833 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:00.825+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:00.850 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:00.850 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:00.850 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:39:00.850 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:00.861 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:39:00.934 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:00.929+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:00.975 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:00.975 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:00.976 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:39:00.976 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:39:00.976 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:00.986 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:39:01.035 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:01.029+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:01.099 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:01.099 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:39:01.099 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:39:01.100 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:39:01.100 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.100 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:39:01.100 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:39:01.100 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 93.307217649999998, 2024-09-18T04:39:01.100 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017399218000000001, 2024-09-18T04:39:01.100 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:01.100 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:01.100 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:01.100 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:39:01.100 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:39:01.100 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 114 2024-09-18T04:39:01.100 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.101 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:01.101 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.101 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:01.101 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:39:01.101 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:01.101 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.101 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.101 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:01.101 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:39:01.101 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:01.101 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.101 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.101 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:01.101 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786900+0000", 2024-09-18T04:39:01.102 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:01.102 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.102 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.102 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:01.102 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786908+0000", 2024-09-18T04:39:01.102 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8199999999999997e-06 2024-09-18T04:39:01.102 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.102 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.102 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:01.102 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786925+0000", 2024-09-18T04:39:01.102 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6849000000000001e-05 2024-09-18T04:39:01.102 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.102 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.102 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:01.103 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786934+0000", 2024-09-18T04:39:01.103 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6299999999999993e-06 2024-09-18T04:39:01.103 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.103 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.103 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:01.103 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800926+0000", 2024-09-18T04:39:01.103 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013991632 2024-09-18T04:39:01.103 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.103 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.103 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:01.103 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801060+0000", 2024-09-18T04:39:01.103 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013378199999999999 2024-09-18T04:39:01.103 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.103 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.104 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:01.104 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801166+0000", 2024-09-18T04:39:01.104 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010607099999999999 2024-09-18T04:39:01.104 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.104 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.104 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:01.104 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803026+0000", 2024-09-18T04:39:01.104 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001860469 2024-09-18T04:39:01.104 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.104 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.104 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:01.104 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804245+0000", 2024-09-18T04:39:01.104 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0012185900000000001 2024-09-18T04:39:01.104 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.105 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.105 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:01.105 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804278+0000", 2024-09-18T04:39:01.105 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3272000000000003e-05 2024-09-18T04:39:01.105 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.105 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.105 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:01.105 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804300+0000", 2024-09-18T04:39:01.105 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2266e-05 2024-09-18T04:39:01.105 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:01.105 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:01.105 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:01.106 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.106 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.106 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:39:01.106 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:39:01.106 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 93.306304752000003, 2024-09-18T04:39:01.106 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017521662, 2024-09-18T04:39:01.106 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:01.106 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:01.106 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:01.106 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:39:01.106 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:39:01.106 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 116 2024-09-18T04:39:01.106 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.107 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:01.107 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.107 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:01.107 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:39:01.107 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:01.107 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.107 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.107 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:01.107 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:39:01.107 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:01.107 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.107 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.107 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:01.107 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787813+0000", 2024-09-18T04:39:01.108 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:01.108 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.108 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.108 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:01.108 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787824+0000", 2024-09-18T04:39:01.108 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1157e-05 2024-09-18T04:39:01.108 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.108 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.108 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:01.108 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787840+0000", 2024-09-18T04:39:01.109 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6269e-05 2024-09-18T04:39:01.109 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.109 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.109 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:01.109 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787853+0000", 2024-09-18T04:39:01.109 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3156000000000001e-05 2024-09-18T04:39:01.109 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.109 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.109 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:01.109 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803110+0000", 2024-09-18T04:39:01.109 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015256736 2024-09-18T04:39:01.109 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.109 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.110 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:01.110 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803197+0000", 2024-09-18T04:39:01.110 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7195999999999997e-05 2024-09-18T04:39:01.110 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.110 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.110 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:01.110 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803308+0000", 2024-09-18T04:39:01.110 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111005 2024-09-18T04:39:01.110 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.110 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.110 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:01.110 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804316+0000", 2024-09-18T04:39:01.110 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010073949999999999 2024-09-18T04:39:01.110 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.110 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.111 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:01.111 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805232+0000", 2024-09-18T04:39:01.111 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091636300000000001 2024-09-18T04:39:01.111 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.111 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.111 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:01.111 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805277+0000", 2024-09-18T04:39:01.111 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4864000000000003e-05 2024-09-18T04:39:01.111 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.111 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.111 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:01.112 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805336+0000", 2024-09-18T04:39:01.112 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8968000000000002e-05 2024-09-18T04:39:01.112 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:01.112 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:01.112 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:01.112 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.112 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.112 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:39:01.112 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:39:01.112 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 93.306252697000005, 2024-09-18T04:39:01.112 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017983453, 2024-09-18T04:39:01.112 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:01.112 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:01.113 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:01.113 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:39:01.113 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:39:01.113 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 117 2024-09-18T04:39:01.113 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.113 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:01.113 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.113 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:01.113 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:39:01.113 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:01.113 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.113 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.113 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:01.113 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:39:01.113 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:01.114 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.114 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.114 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:01.114 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787865+0000", 2024-09-18T04:39:01.114 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:01.114 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.114 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.114 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:01.114 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787889+0000", 2024-09-18T04:39:01.114 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4244999999999999e-05 2024-09-18T04:39:01.114 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.114 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.114 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:01.114 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787905+0000", 2024-09-18T04:39:01.115 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5739999999999998e-05 2024-09-18T04:39:01.115 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.115 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.115 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:01.115 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787918+0000", 2024-09-18T04:39:01.115 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.252e-05 2024-09-18T04:39:01.115 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.115 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.115 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:01.115 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803693+0000", 2024-09-18T04:39:01.115 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015774872999999998 2024-09-18T04:39:01.115 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.115 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.115 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:01.116 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803775+0000", 2024-09-18T04:39:01.116 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2761000000000003e-05 2024-09-18T04:39:01.116 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.116 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.116 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:01.116 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803887+0000", 2024-09-18T04:39:01.116 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111708 2024-09-18T04:39:01.116 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.116 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.116 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:01.116 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804823+0000", 2024-09-18T04:39:01.116 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00093625100000000003 2024-09-18T04:39:01.116 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.116 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.117 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:01.117 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805804+0000", 2024-09-18T04:39:01.117 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00098114699999999992 2024-09-18T04:39:01.117 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.117 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.117 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:01.117 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805831+0000", 2024-09-18T04:39:01.117 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6401999999999999e-05 2024-09-18T04:39:01.117 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.117 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.117 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:01.117 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805850+0000", 2024-09-18T04:39:01.117 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8862e-05 2024-09-18T04:39:01.117 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:01.118 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:01.118 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:01.118 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.118 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.118 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:36 2.7 2:fe1b02ad:::rbd_data.125fc09df6d1.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:01.118 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:39:01.118 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 18.828396587, 2024-09-18T04:39:01.118 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.040387804999999999, 2024-09-18T04:39:01.118 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:01.118 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:01.118 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:01.118 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:01.118 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:01.118 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 36 2024-09-18T04:39:01.119 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.119 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:01.119 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.119 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:01.119 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:39:01.119 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:01.119 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.119 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.119 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:01.119 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:39:01.119 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:01.119 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.119 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.119 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:01.120 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265720+0000", 2024-09-18T04:39:01.120 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:01.120 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.120 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.120 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:01.120 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272590+0000", 2024-09-18T04:39:01.120 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.006870508 2024-09-18T04:39:01.120 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.120 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.120 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:01.120 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272623+0000", 2024-09-18T04:39:01.120 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2716000000000003e-05 2024-09-18T04:39:01.120 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.121 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.121 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:01.121 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272637+0000", 2024-09-18T04:39:01.121 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4175999999999999e-05 2024-09-18T04:39:01.121 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.121 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.121 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:01.121 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272779+0000", 2024-09-18T04:39:01.121 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000141392 2024-09-18T04:39:01.121 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.121 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.121 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:01.121 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273031+0000", 2024-09-18T04:39:01.121 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00025249099999999998 2024-09-18T04:39:01.121 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.122 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.122 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:39:01.122 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273219+0000", 2024-09-18T04:39:01.122 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000187531 2024-09-18T04:39:01.122 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.122 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.122 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:01.122 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.302806+0000", 2024-09-18T04:39:01.122 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.029587553999999999 2024-09-18T04:39:01.122 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.122 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.122 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:01.122 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306004+0000", 2024-09-18T04:39:01.123 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0031982629999999998 2024-09-18T04:39:01.123 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.123 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.123 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:01.123 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306071+0000", 2024-09-18T04:39:01.123 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6829999999999995e-05 2024-09-18T04:39:01.123 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.123 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.123 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:01.123 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306110+0000", 2024-09-18T04:39:01.123 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8874000000000001e-05 2024-09-18T04:39:01.123 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:01.123 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:01.123 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:01.124 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.124 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.124 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:38 2.0 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:01.124 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:39:01.124 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 18.821443049999999, 2024-09-18T04:39:01.124 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.033754348000000003, 2024-09-18T04:39:01.124 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:01.124 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:01.124 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:01.124 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:01.124 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:01.124 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 38 2024-09-18T04:39:01.125 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.125 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:01.125 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.125 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:01.125 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:39:01.125 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:01.125 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.125 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.125 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:01.125 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:39:01.125 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:01.125 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.125 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.126 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:01.126 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272673+0000", 2024-09-18T04:39:01.126 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:01.126 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.126 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.126 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:01.126 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277166+0000", 2024-09-18T04:39:01.126 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044922249999999999 2024-09-18T04:39:01.126 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.126 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.126 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:01.126 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277189+0000", 2024-09-18T04:39:01.126 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3283000000000001e-05 2024-09-18T04:39:01.126 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.127 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.127 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:01.127 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277205+0000", 2024-09-18T04:39:01.127 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6305000000000001e-05 2024-09-18T04:39:01.127 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.127 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.127 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:01.127 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277290+0000", 2024-09-18T04:39:01.127 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5135000000000004e-05 2024-09-18T04:39:01.127 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.127 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.127 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:01.127 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277531+0000", 2024-09-18T04:39:01.128 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00024014699999999999 2024-09-18T04:39:01.128 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.128 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.128 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:01.128 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277793+0000", 2024-09-18T04:39:01.128 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00026224499999999998 2024-09-18T04:39:01.128 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.128 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.129 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:01.129 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303990+0000", 2024-09-18T04:39:01.129 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.026197295999999998 2024-09-18T04:39:01.129 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.129 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.129 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:01.129 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306340+0000", 2024-09-18T04:39:01.129 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.002349524 2024-09-18T04:39:01.129 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.129 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.129 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:01.130 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306394+0000", 2024-09-18T04:39:01.130 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3934999999999998e-05 2024-09-18T04:39:01.130 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.130 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.130 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:01.130 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306430+0000", 2024-09-18T04:39:01.130 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6690000000000003e-05 2024-09-18T04:39:01.130 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:01.130 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:01.130 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:01.130 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.130 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.130 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:39 2.7 2:e9c909c5:::rbd_data.125fc09df6d1.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:01.130 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:39:01.131 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 18.816883444999998, 2024-09-18T04:39:01.131 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.029110673, 2024-09-18T04:39:01.131 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:01.131 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:01.131 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:01.131 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:01.131 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:01.131 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 39 2024-09-18T04:39:01.131 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.131 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:01.131 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.131 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:01.131 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:39:01.132 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:01.132 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.132 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.132 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:01.132 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:39:01.132 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:01.132 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.132 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.132 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:01.132 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277234+0000", 2024-09-18T04:39:01.132 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:01.132 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.132 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.133 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:01.133 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281351+0000", 2024-09-18T04:39:01.133 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0041170920000000001 2024-09-18T04:39:01.133 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.133 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.133 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:01.133 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281375+0000", 2024-09-18T04:39:01.133 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.446e-05 2024-09-18T04:39:01.133 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.133 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.133 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:01.133 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281390+0000", 2024-09-18T04:39:01.133 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4309e-05 2024-09-18T04:39:01.133 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.134 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.134 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:01.134 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.282306+0000", 2024-09-18T04:39:01.134 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091662000000000004 2024-09-18T04:39:01.134 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.134 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.134 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:01.134 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.282613+0000", 2024-09-18T04:39:01.134 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00030663799999999997 2024-09-18T04:39:01.134 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.134 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.134 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:39:01.134 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283001+0000", 2024-09-18T04:39:01.134 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00038801599999999998 2024-09-18T04:39:01.134 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.135 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.135 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:01.135 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306239+0000", 2024-09-18T04:39:01.135 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023238466999999999 2024-09-18T04:39:01.135 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.135 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.135 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:01.135 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306259+0000", 2024-09-18T04:39:01.135 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9837999999999999e-05 2024-09-18T04:39:01.135 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.135 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.135 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:01.135 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306299+0000", 2024-09-18T04:39:01.135 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0194000000000002e-05 2024-09-18T04:39:01.136 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.136 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.136 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:01.136 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306346+0000", 2024-09-18T04:39:01.136 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6845000000000003e-05 2024-09-18T04:39:01.136 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:01.136 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:01.136 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:01.136 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.136 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.136 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:46 2.6 e51/17 2:7ba25245:::rbd_data.125fc09df6d1.0000000000000006:head v 51'2, mlcod=0'0)", 2024-09-18T04:39:01.136 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:39:01.136 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 18.780962313, 2024-09-18T04:39:01.137 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019843098, 2024-09-18T04:39:01.137 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:01.137 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:01.137 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:01.137 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.137 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:01.137 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:39:01.137 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:01.137 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.137 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.137 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:01.137 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:39:01.137 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:01.137 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.137 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.138 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:01.138 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313156+0000", 2024-09-18T04:39:01.138 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:01.138 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.138 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.138 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:01.138 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320144+0000", 2024-09-18T04:39:01.138 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.006988345 2024-09-18T04:39:01.138 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.138 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.138 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:01.138 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320175+0000", 2024-09-18T04:39:01.138 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.0861999999999997e-05 2024-09-18T04:39:01.139 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.139 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.139 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:01.139 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320188+0000", 2024-09-18T04:39:01.139 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2714999999999999e-05 2024-09-18T04:39:01.139 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.139 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.139 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:01.139 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320267+0000", 2024-09-18T04:39:01.139 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9586999999999995e-05 2024-09-18T04:39:01.139 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.139 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.139 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:01.139 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320311+0000", 2024-09-18T04:39:01.139 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3529000000000003e-05 2024-09-18T04:39:01.140 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.140 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.140 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:01.140 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.332949+0000", 2024-09-18T04:39:01.140 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012638267999999999 2024-09-18T04:39:01.140 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.140 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.140 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:01.140 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.333000+0000", 2024-09-18T04:39:01.140 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0736999999999999e-05 2024-09-18T04:39:01.140 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:01.140 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:01.140 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:01.140 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.141 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.141 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:57 2.0 2:0cdd070e:::rbd_data.125fc09df6d1.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:01.141 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:39:01.141 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 18.745104737999998, 2024-09-18T04:39:01.141 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.033329974999999998, 2024-09-18T04:39:01.141 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:01.141 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:01.141 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:01.141 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:01.141 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:01.141 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 57 2024-09-18T04:39:01.141 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.141 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:01.141 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.141 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:01.142 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:39:01.142 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:01.142 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.142 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.142 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:01.142 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:39:01.142 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:01.142 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.142 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.142 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:01.142 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349013+0000", 2024-09-18T04:39:01.142 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:01.142 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.142 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.143 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:01.143 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356138+0000", 2024-09-18T04:39:01.143 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071258010000000002 2024-09-18T04:39:01.143 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.143 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.143 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:01.143 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356168+0000", 2024-09-18T04:39:01.143 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9679000000000002e-05 2024-09-18T04:39:01.143 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.143 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.143 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:01.143 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356181+0000", 2024-09-18T04:39:01.143 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3164e-05 2024-09-18T04:39:01.143 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.144 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.144 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:01.144 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356254+0000", 2024-09-18T04:39:01.144 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3182999999999994e-05 2024-09-18T04:39:01.144 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.144 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.144 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:01.144 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356453+0000", 2024-09-18T04:39:01.144 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019876599999999999 2024-09-18T04:39:01.144 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.144 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.144 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:01.144 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356627+0000", 2024-09-18T04:39:01.144 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017340199999999999 2024-09-18T04:39:01.145 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.145 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.145 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:01.145 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.369200+0000", 2024-09-18T04:39:01.145 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012573302 2024-09-18T04:39:01.145 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.145 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.145 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:01.145 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382262+0000", 2024-09-18T04:39:01.145 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013062549 2024-09-18T04:39:01.145 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.145 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.145 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:01.145 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382310+0000", 2024-09-18T04:39:01.146 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7366e-05 2024-09-18T04:39:01.146 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.146 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.146 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:01.146 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382344+0000", 2024-09-18T04:39:01.146 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4319999999999997e-05 2024-09-18T04:39:01.146 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:01.146 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:01.146 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:01.146 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.146 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.146 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:59 2.0 2:19c52cde:::rbd_data.125fc09df6d1.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:01.146 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:39:01.146 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 18.737913573, 2024-09-18T04:39:01.147 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.027137247999999999, 2024-09-18T04:39:01.147 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:01.147 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:01.147 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:01.147 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:01.147 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:01.147 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 59 2024-09-18T04:39:01.147 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.147 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:01.148 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.148 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:01.148 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:39:01.148 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:01.148 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.148 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.148 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:01.148 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:39:01.148 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:01.148 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.148 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.148 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:01.148 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356204+0000", 2024-09-18T04:39:01.149 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:01.149 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.149 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.149 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:01.149 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362661+0000", 2024-09-18T04:39:01.149 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0064565129999999997 2024-09-18T04:39:01.149 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.149 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.149 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:01.149 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362684+0000", 2024-09-18T04:39:01.149 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3688000000000001e-05 2024-09-18T04:39:01.149 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.149 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.149 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:01.150 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362699+0000", 2024-09-18T04:39:01.150 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4798e-05 2024-09-18T04:39:01.150 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.150 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.150 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:01.150 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363016+0000", 2024-09-18T04:39:01.150 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031737300000000003 2024-09-18T04:39:01.150 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.150 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.150 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:01.150 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363201+0000", 2024-09-18T04:39:01.150 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000184131 2024-09-18T04:39:01.150 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.150 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.151 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:01.151 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363343+0000", 2024-09-18T04:39:01.151 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014236699999999999 2024-09-18T04:39:01.151 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.151 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.151 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:01.151 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.372189+0000", 2024-09-18T04:39:01.151 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0088464819999999993 2024-09-18T04:39:01.151 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.151 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.151 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:01.151 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383269+0000", 2024-09-18T04:39:01.151 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011079594 2024-09-18T04:39:01.152 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.152 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.152 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:01.152 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383314+0000", 2024-09-18T04:39:01.152 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4932000000000003e-05 2024-09-18T04:39:01.152 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.152 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.152 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:01.152 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383343+0000", 2024-09-18T04:39:01.152 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8600999999999999e-05 2024-09-18T04:39:01.152 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:01.152 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:01.152 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:01.153 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.153 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.153 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:40 2.6 e51/17 2:784d6cd1:ns::rbd_data.133e813ab5bc.0000000000000005:head v 51'7, mlcod=51'5)", 2024-09-18T04:39:01.153 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:39:01.153 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 18.541663839000002, 2024-09-18T04:39:01.153 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020497496, 2024-09-18T04:39:01.153 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:01.153 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:01.153 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:01.153 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.153 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:01.153 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:39:01.153 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:01.153 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.153 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.154 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:01.154 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:39:01.154 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:01.154 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.154 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.154 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:01.154 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552454+0000", 2024-09-18T04:39:01.154 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:01.154 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.154 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.154 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:01.154 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556448+0000", 2024-09-18T04:39:01.154 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0039940990000000001 2024-09-18T04:39:01.154 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.155 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.155 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:01.155 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556481+0000", 2024-09-18T04:39:01.155 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3778000000000001e-05 2024-09-18T04:39:01.155 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.155 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.155 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:01.155 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556493+0000", 2024-09-18T04:39:01.155 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1622e-05 2024-09-18T04:39:01.155 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.155 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.155 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:01.155 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556556+0000", 2024-09-18T04:39:01.155 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2582999999999994e-05 2024-09-18T04:39:01.155 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.156 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.156 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:01.156 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556592+0000", 2024-09-18T04:39:01.156 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5880999999999997e-05 2024-09-18T04:39:01.156 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.156 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.156 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:01.156 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.572786+0000", 2024-09-18T04:39:01.156 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016194571000000001 2024-09-18T04:39:01.156 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.156 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.156 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:01.156 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.572953+0000", 2024-09-18T04:39:01.157 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000166436 2024-09-18T04:39:01.157 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:01.157 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:01.157 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:01.157 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.157 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.157 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:45 2.0 2:0fbfa3dc:ns::rbd_data.133e813ab5bc.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:01.157 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:39:01.157 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 18.530449539999999, 2024-09-18T04:39:01.157 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.036558168000000002, 2024-09-18T04:39:01.157 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:01.157 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:01.157 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:01.157 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:01.157 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:01.158 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 45 2024-09-18T04:39:01.158 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.158 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:01.158 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.158 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:01.158 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:39:01.158 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:01.158 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.158 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.158 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:01.158 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:39:01.158 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:01.158 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.158 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.159 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:01.159 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:39:01.159 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:01.159 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.159 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.159 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:01.159 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578594+0000", 2024-09-18T04:39:01.159 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014927147999999999 2024-09-18T04:39:01.159 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.159 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.159 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:01.159 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578612+0000", 2024-09-18T04:39:01.159 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7623999999999999e-05 2024-09-18T04:39:01.159 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.160 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.160 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:01.160 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578620+0000", 2024-09-18T04:39:01.160 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9070000000000007e-06 2024-09-18T04:39:01.160 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.160 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.160 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:01.160 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578702+0000", 2024-09-18T04:39:01.160 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2349999999999996e-05 2024-09-18T04:39:01.160 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.160 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.160 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:01.160 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.579028+0000", 2024-09-18T04:39:01.161 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032622899999999998 2024-09-18T04:39:01.161 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.161 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.161 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:01.161 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.579343+0000", 2024-09-18T04:39:01.161 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031448899999999999 2024-09-18T04:39:01.161 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.161 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.161 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:01.161 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.590078+0000", 2024-09-18T04:39:01.161 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010734946 2024-09-18T04:39:01.161 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.162 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.162 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:01.162 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600089+0000", 2024-09-18T04:39:01.162 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010011195000000001 2024-09-18T04:39:01.162 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.162 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.162 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:01.162 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600177+0000", 2024-09-18T04:39:01.162 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8425999999999997e-05 2024-09-18T04:39:01.162 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.162 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.162 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:01.162 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600228+0000", 2024-09-18T04:39:01.162 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0216999999999997e-05 2024-09-18T04:39:01.163 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:01.163 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:01.163 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:01.163 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.163 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.163 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:54 2.0 2:04f17ef6:ns::rbd_data.133e813ab5bc.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:01.163 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:39:01.163 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 18.508877703, 2024-09-18T04:39:01.163 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.024612999, 2024-09-18T04:39:01.163 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:01.163 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:01.163 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:01.163 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:01.163 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:01.164 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 54 2024-09-18T04:39:01.164 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.164 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:01.164 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.164 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:01.164 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:39:01.164 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:01.164 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.164 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.164 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:01.164 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:39:01.164 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:01.164 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.165 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.165 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:01.165 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585239+0000", 2024-09-18T04:39:01.165 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:01.165 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.165 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.165 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:01.165 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593702+0000", 2024-09-18T04:39:01.165 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0084632990000000005 2024-09-18T04:39:01.165 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.165 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.165 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:01.165 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593719+0000", 2024-09-18T04:39:01.165 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.664e-05 2024-09-18T04:39:01.165 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.166 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.166 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:01.166 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593727+0000", 2024-09-18T04:39:01.166 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6839999999999996e-06 2024-09-18T04:39:01.166 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.166 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.166 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:01.166 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593767+0000", 2024-09-18T04:39:01.166 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0330999999999997e-05 2024-09-18T04:39:01.166 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.166 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.166 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:01.166 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593905+0000", 2024-09-18T04:39:01.167 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013754800000000001 2024-09-18T04:39:01.167 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.167 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.167 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:01.167 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.594013+0000", 2024-09-18T04:39:01.167 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000108805 2024-09-18T04:39:01.167 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.167 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.167 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:01.167 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604678+0000", 2024-09-18T04:39:01.167 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010664391 2024-09-18T04:39:01.167 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.168 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.168 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:01.168 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609662+0000", 2024-09-18T04:39:01.168 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0049846100000000004 2024-09-18T04:39:01.168 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.168 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.168 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:01.168 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609788+0000", 2024-09-18T04:39:01.168 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012518800000000001 2024-09-18T04:39:01.169 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.169 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.169 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:01.169 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609854+0000", 2024-09-18T04:39:01.169 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6556000000000004e-05 2024-09-18T04:39:01.169 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:01.169 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:01.169 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:01.169 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.169 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.169 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:58 2.7 2:fc7c6cd3:ns::rbd_data.133e813ab5bc.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:01.169 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:39:01.169 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 18.496035274, 2024-09-18T04:39:01.170 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.027690566999999999, 2024-09-18T04:39:01.170 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:01.170 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:01.170 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:01.170 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:01.170 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:01.170 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 58 2024-09-18T04:39:01.170 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.170 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:01.170 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.170 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:01.170 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:39:01.171 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:01.171 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.171 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.171 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:01.171 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:39:01.171 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:01.171 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.171 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.171 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:01.171 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598082+0000", 2024-09-18T04:39:01.171 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:01.171 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.171 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.171 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:01.172 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604832+0000", 2024-09-18T04:39:01.172 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0067504510000000002 2024-09-18T04:39:01.172 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.172 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.172 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:01.172 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604852+0000", 2024-09-18T04:39:01.172 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0228999999999999e-05 2024-09-18T04:39:01.172 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.172 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.172 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:01.172 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604870+0000", 2024-09-18T04:39:01.172 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7952e-05 2024-09-18T04:39:01.172 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.172 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.172 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:01.173 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604936+0000", 2024-09-18T04:39:01.173 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5431000000000004e-05 2024-09-18T04:39:01.173 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.173 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.173 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:01.173 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605085+0000", 2024-09-18T04:39:01.173 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014910300000000001 2024-09-18T04:39:01.173 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.173 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.173 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:39:01.173 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605205+0000", 2024-09-18T04:39:01.173 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000119887 2024-09-18T04:39:01.173 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.174 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.174 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:01.174 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625586+0000", 2024-09-18T04:39:01.174 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020381399000000001 2024-09-18T04:39:01.174 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.174 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.174 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:01.174 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625628+0000", 2024-09-18T04:39:01.174 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1873999999999999e-05 2024-09-18T04:39:01.174 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.174 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.174 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:01.174 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625722+0000", 2024-09-18T04:39:01.174 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3787000000000002e-05 2024-09-18T04:39:01.174 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.175 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.175 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:01.175 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625774+0000", 2024-09-18T04:39:01.175 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2503999999999997e-05 2024-09-18T04:39:01.175 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:01.175 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:01.175 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:01.175 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.175 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.175 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:56 2.6 e51/17 2:72453c3e:ns::rbd_data.133e813ab5bc.000000000000000d:head v 51'10, mlcod=51'8)", 2024-09-18T04:39:01.175 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:39:01.175 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 18.488838166000001, 2024-09-18T04:39:01.175 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019766552999999999, 2024-09-18T04:39:01.175 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:01.176 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:01.176 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:01.176 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.176 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:01.176 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:39:01.176 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:01.176 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.176 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.176 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:01.176 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:39:01.176 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:01.176 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.176 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.176 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:01.177 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605279+0000", 2024-09-18T04:39:01.177 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:01.177 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.177 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.177 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:01.177 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609417+0000", 2024-09-18T04:39:01.177 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0041378059999999999 2024-09-18T04:39:01.177 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.177 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.177 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:01.177 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609443+0000", 2024-09-18T04:39:01.177 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6098e-05 2024-09-18T04:39:01.177 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.177 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.178 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:01.178 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609458+0000", 2024-09-18T04:39:01.178 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4671000000000001e-05 2024-09-18T04:39:01.178 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.178 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.178 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:01.178 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609553+0000", 2024-09-18T04:39:01.178 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5265000000000004e-05 2024-09-18T04:39:01.178 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.178 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.178 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:01.178 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609583+0000", 2024-09-18T04:39:01.178 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9839e-05 2024-09-18T04:39:01.178 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.179 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.179 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:01.179 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625004+0000", 2024-09-18T04:39:01.179 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015420639999999999 2024-09-18T04:39:01.179 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.179 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.179 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:01.179 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625047+0000", 2024-09-18T04:39:01.179 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3665999999999998e-05 2024-09-18T04:39:01.179 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:01.179 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:01.179 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:01.179 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.179 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.179 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:99 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:01.180 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.582718+0000", 2024-09-18T04:39:01.180 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 4.5114012570000002, 2024-09-18T04:39:01.180 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019362575, 2024-09-18T04:39:01.180 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:01.180 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:01.180 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:01.180 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:01.180 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:01.180 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 99 2024-09-18T04:39:01.180 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.180 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:01.180 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.180 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:01.181 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582718+0000", 2024-09-18T04:39:01.181 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:01.181 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.181 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.181 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:01.181 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582718+0000", 2024-09-18T04:39:01.181 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:01.181 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.181 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.181 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:01.181 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582717+0000", 2024-09-18T04:39:01.181 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:01.182 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.182 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.182 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:01.182 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582720+0000", 2024-09-18T04:39:01.182 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2239999999999998e-06 2024-09-18T04:39:01.182 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.182 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.182 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:01.182 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582723+0000", 2024-09-18T04:39:01.182 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.258e-06 2024-09-18T04:39:01.182 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.182 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.183 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:01.183 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582726+0000", 2024-09-18T04:39:01.183 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8870000000000001e-06 2024-09-18T04:39:01.183 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.183 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.183 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:01.183 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.592018+0000", 2024-09-18T04:39:01.183 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0092926420000000003 2024-09-18T04:39:01.183 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.183 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.183 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:01.183 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.592214+0000", 2024-09-18T04:39:01.183 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000195267 2024-09-18T04:39:01.183 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.183 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.184 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:01.184 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.592913+0000", 2024-09-18T04:39:01.184 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00069924600000000005 2024-09-18T04:39:01.184 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.184 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.184 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:01.184 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597493+0000", 2024-09-18T04:39:01.184 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045797629999999997 2024-09-18T04:39:01.184 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.184 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.184 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:01.184 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.602009+0000", 2024-09-18T04:39:01.184 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045159559999999998 2024-09-18T04:39:01.185 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.185 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.185 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:01.185 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.602042+0000", 2024-09-18T04:39:01.185 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3084999999999999e-05 2024-09-18T04:39:01.185 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.185 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.185 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:01.185 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.602080+0000", 2024-09-18T04:39:01.185 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8600000000000003e-05 2024-09-18T04:39:01.185 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:01.185 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:01.185 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:01.185 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.185 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.186 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:104 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:01.186 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.583165+0000", 2024-09-18T04:39:01.186 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 4.5109539590000001, 2024-09-18T04:39:01.186 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023679603, 2024-09-18T04:39:01.186 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:01.186 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:01.186 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:01.186 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:01.186 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:01.186 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 104 2024-09-18T04:39:01.186 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.186 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:01.187 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.187 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:01.187 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583165+0000", 2024-09-18T04:39:01.187 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:01.187 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.187 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.187 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:01.187 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583165+0000", 2024-09-18T04:39:01.187 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:01.187 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.187 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.187 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:01.187 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583164+0000", 2024-09-18T04:39:01.187 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:01.188 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.188 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.188 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:01.188 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583167+0000", 2024-09-18T04:39:01.188 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9210000000000001e-06 2024-09-18T04:39:01.188 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.188 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.188 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:01.188 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583171+0000", 2024-09-18T04:39:01.188 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9759999999999997e-06 2024-09-18T04:39:01.188 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.188 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.188 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:01.189 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583174+0000", 2024-09-18T04:39:01.189 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9960000000000002e-06 2024-09-18T04:39:01.189 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.189 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.189 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:01.189 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.594373+0000", 2024-09-18T04:39:01.189 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011198665999999999 2024-09-18T04:39:01.189 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.189 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.189 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:01.189 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.594434+0000", 2024-09-18T04:39:01.189 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.1446999999999996e-05 2024-09-18T04:39:01.189 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.190 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.190 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:01.190 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.594692+0000", 2024-09-18T04:39:01.190 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00025756599999999998 2024-09-18T04:39:01.190 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.190 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.190 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:01.190 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599302+0000", 2024-09-18T04:39:01.190 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0046099640000000002 2024-09-18T04:39:01.190 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.190 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.190 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:01.190 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606710+0000", 2024-09-18T04:39:01.190 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0074077739999999998 2024-09-18T04:39:01.191 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.191 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.191 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:01.191 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606798+0000", 2024-09-18T04:39:01.191 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8133999999999993e-05 2024-09-18T04:39:01.191 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.191 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.191 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:01.191 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606845+0000", 2024-09-18T04:39:01.191 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6653000000000002e-05 2024-09-18T04:39:01.191 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:01.191 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:01.191 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:01.191 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.192 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.192 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:105 2.0 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:01.192 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.583193+0000", 2024-09-18T04:39:01.192 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 4.5109263400000001, 2024-09-18T04:39:01.192 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023713649, 2024-09-18T04:39:01.192 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:01.192 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:01.192 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:01.192 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:01.192 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:01.192 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 105 2024-09-18T04:39:01.192 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.192 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:01.193 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.193 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:01.193 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583193+0000", 2024-09-18T04:39:01.193 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:01.193 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.193 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.193 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:01.193 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583193+0000", 2024-09-18T04:39:01.193 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:01.193 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.193 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.193 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:01.193 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583192+0000", 2024-09-18T04:39:01.193 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:01.193 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.194 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.194 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:01.194 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583194+0000", 2024-09-18T04:39:01.194 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.305e-06 2024-09-18T04:39:01.194 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.194 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.194 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:01.194 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583198+0000", 2024-09-18T04:39:01.194 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9369999999999997e-06 2024-09-18T04:39:01.194 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.194 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.194 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:01.194 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583202+0000", 2024-09-18T04:39:01.194 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6480000000000001e-06 2024-09-18T04:39:01.195 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.195 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.195 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:01.195 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.595304+0000", 2024-09-18T04:39:01.195 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012102063999999999 2024-09-18T04:39:01.195 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.195 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.195 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:01.195 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.595390+0000", 2024-09-18T04:39:01.195 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5493000000000003e-05 2024-09-18T04:39:01.195 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.195 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.195 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:01.195 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.595509+0000", 2024-09-18T04:39:01.196 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011966199999999999 2024-09-18T04:39:01.196 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.196 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.196 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:01.196 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599358+0000", 2024-09-18T04:39:01.196 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0038486689999999999 2024-09-18T04:39:01.196 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.196 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.196 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:01.196 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606859+0000", 2024-09-18T04:39:01.196 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0075010789999999999 2024-09-18T04:39:01.196 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.196 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.197 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:01.197 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606887+0000", 2024-09-18T04:39:01.197 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7773e-05 2024-09-18T04:39:01.197 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.197 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.197 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:01.197 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606906+0000", 2024-09-18T04:39:01.197 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9417000000000001e-05 2024-09-18T04:39:01.197 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:01.197 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:01.197 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:01.197 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.197 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.198 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:106 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:01.198 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.584425+0000", 2024-09-18T04:39:01.198 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 4.5096940439999997, 2024-09-18T04:39:01.198 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.022529293999999998, 2024-09-18T04:39:01.198 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:01.198 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:01.198 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:01.198 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:01.198 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:01.198 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 106 2024-09-18T04:39:01.198 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.198 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:01.198 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.199 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:01.199 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584425+0000", 2024-09-18T04:39:01.199 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:01.199 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.199 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.199 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:01.199 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584425+0000", 2024-09-18T04:39:01.199 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:01.199 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.199 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.199 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:01.199 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584424+0000", 2024-09-18T04:39:01.199 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:01.199 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.199 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.200 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:01.200 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584430+0000", 2024-09-18T04:39:01.200 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6389999999999997e-06 2024-09-18T04:39:01.200 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.200 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.200 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:01.200 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584456+0000", 2024-09-18T04:39:01.200 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6188999999999999e-05 2024-09-18T04:39:01.200 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.200 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.200 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:01.200 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584463+0000", 2024-09-18T04:39:01.200 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7340000000000002e-06 2024-09-18T04:39:01.201 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.201 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.201 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:01.201 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597537+0000", 2024-09-18T04:39:01.201 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013074161000000001 2024-09-18T04:39:01.201 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.201 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.201 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:01.201 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597598+0000", 2024-09-18T04:39:01.201 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0930000000000001e-05 2024-09-18T04:39:01.201 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.201 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.201 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:01.201 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597922+0000", 2024-09-18T04:39:01.202 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032473000000000002 2024-09-18T04:39:01.202 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.202 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.202 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:01.202 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.601103+0000", 2024-09-18T04:39:01.202 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0031809849999999999 2024-09-18T04:39:01.202 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.202 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.202 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:01.202 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606912+0000", 2024-09-18T04:39:01.202 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058083550000000003 2024-09-18T04:39:01.202 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.202 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.202 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:01.203 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606935+0000", 2024-09-18T04:39:01.203 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2759999999999999e-05 2024-09-18T04:39:01.203 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.203 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.203 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:01.203 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606954+0000", 2024-09-18T04:39:01.203 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9602999999999998e-05 2024-09-18T04:39:01.203 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:01.203 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:01.203 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:01.203 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.203 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.203 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:109 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:01.204 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.585231+0000", 2024-09-18T04:39:01.204 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 4.5088874240000001, 2024-09-18T04:39:01.204 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.021767496000000001, 2024-09-18T04:39:01.204 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:01.204 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:01.204 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:01.204 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:01.204 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:01.204 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 109 2024-09-18T04:39:01.204 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.204 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:01.204 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.204 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:01.204 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585231+0000", 2024-09-18T04:39:01.205 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:01.205 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.205 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.205 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:01.205 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585231+0000", 2024-09-18T04:39:01.205 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:01.205 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.205 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.205 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:01.205 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585230+0000", 2024-09-18T04:39:01.205 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:01.205 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.205 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.205 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:01.206 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585237+0000", 2024-09-18T04:39:01.206 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.618e-06 2024-09-18T04:39:01.206 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.206 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.206 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:01.206 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585256+0000", 2024-09-18T04:39:01.206 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9048000000000001e-05 2024-09-18T04:39:01.206 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.206 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.206 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:01.206 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585274+0000", 2024-09-18T04:39:01.206 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8153000000000001e-05 2024-09-18T04:39:01.206 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.206 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.207 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:01.207 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.598284+0000", 2024-09-18T04:39:01.207 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013009682999999999 2024-09-18T04:39:01.207 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.207 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.207 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:01.207 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.598383+0000", 2024-09-18T04:39:01.207 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.8722999999999997e-05 2024-09-18T04:39:01.207 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.207 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.207 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:01.207 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.598829+0000", 2024-09-18T04:39:01.207 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00044623999999999998 2024-09-18T04:39:01.207 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.208 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.208 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:01.208 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.601149+0000", 2024-09-18T04:39:01.208 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0023204150000000002 2024-09-18T04:39:01.208 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.208 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.208 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:01.208 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606960+0000", 2024-09-18T04:39:01.208 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058106390000000003 2024-09-18T04:39:01.208 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.208 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.208 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:01.208 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606982+0000", 2024-09-18T04:39:01.209 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2252e-05 2024-09-18T04:39:01.209 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.209 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.209 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:01.209 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606999+0000", 2024-09-18T04:39:01.209 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6871000000000002e-05 2024-09-18T04:39:01.209 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:01.209 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:01.209 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:01.209 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.209 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.210 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:110 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:01.210 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.586640+0000", 2024-09-18T04:39:01.210 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 4.5074791049999998, 2024-09-18T04:39:01.210 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020400911000000001, 2024-09-18T04:39:01.210 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:01.210 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:01.210 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:01.210 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:01.210 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:01.210 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 110 2024-09-18T04:39:01.210 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.210 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:01.210 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.210 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:01.211 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586640+0000", 2024-09-18T04:39:01.211 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:01.211 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.211 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.211 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:01.211 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586640+0000", 2024-09-18T04:39:01.211 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:01.211 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.211 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.211 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:01.211 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586639+0000", 2024-09-18T04:39:01.211 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:01.211 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.211 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.212 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:01.212 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586652+0000", 2024-09-18T04:39:01.212 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2482e-05 2024-09-18T04:39:01.212 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.212 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.212 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:01.212 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586660+0000", 2024-09-18T04:39:01.212 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1659999999999994e-06 2024-09-18T04:39:01.212 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.212 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.212 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:01.212 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586666+0000", 2024-09-18T04:39:01.212 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6900000000000003e-06 2024-09-18T04:39:01.212 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.213 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.213 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:01.213 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599381+0000", 2024-09-18T04:39:01.213 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012714328 2024-09-18T04:39:01.213 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.213 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.213 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:01.213 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599436+0000", 2024-09-18T04:39:01.213 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4802999999999997e-05 2024-09-18T04:39:01.213 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.213 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.213 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:01.213 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599629+0000", 2024-09-18T04:39:01.213 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000193921 2024-09-18T04:39:01.213 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.214 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.214 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:01.214 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.601174+0000", 2024-09-18T04:39:01.214 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001544857 2024-09-18T04:39:01.214 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.214 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.214 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:01.214 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.607004+0000", 2024-09-18T04:39:01.214 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058300820000000003 2024-09-18T04:39:01.214 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.214 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.214 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:01.214 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.607024+0000", 2024-09-18T04:39:01.214 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9888e-05 2024-09-18T04:39:01.214 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:01.215 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:01.215 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:01.215 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.607041+0000", 2024-09-18T04:39:01.215 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6450999999999999e-05 2024-09-18T04:39:01.215 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:01.215 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:01.215 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:01.215 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:01.215 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:01.215 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:01.217 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:39:01.217 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:01.209+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:01.277 INFO:tasks.workunit.client.0.smithi115.stderr:64+0 records in 2024-09-18T04:39:01.277 INFO:tasks.workunit.client.0.smithi115.stderr:64+0 records out 2024-09-18T04:39:01.277 INFO:tasks.workunit.client.0.smithi115.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.778623 s, 86.2 MB/s 2024-09-18T04:39:01.279 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false timeout 10 rbd bench testrbdnbd17342 --io-type write --io-size=1024 --io-total=1024 2024-09-18T04:39:01.279 INFO:tasks.workunit.client.0.smithi115.stderr:+ timeout 10 rbd bench testrbdnbd17342 --io-type write --io-size=1024 --io-total=1024 2024-09-18T04:39:01.317 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:01.309+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:01.357 INFO:tasks.workunit.client.0.smithi115.stderr:2024-09-18T04:39:01.351+0000 7f76eda08700 -1 librbd::exclusive_lock::ImageDispatch: 0x7f76c8004b40 handle_acquire_lock: failed to acquire exclusive lock: (30) Read-only file system 2024-09-18T04:39:01.358 INFO:tasks.workunit.client.0.smithi115.stderr:rbd: failed to flush: (30) Read-only file system 2024-09-18T04:39:01.358 INFO:tasks.workunit.client.0.smithi115.stderr:2024-09-18T04:39:01.351+0000 7f76eda08700 -1 librbd::io::AioCompletion: 0x55ca7d5dfb30 fail: (30) Read-only file system 2024-09-18T04:39:01.358 INFO:tasks.workunit.client.0.smithi115.stderr:bench failed: (30) Read-only file system 2024-09-18T04:39:01.367 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:39:01.367 INFO:tasks.workunit.client.0.smithi115.stderr:+ unmap_device /dev/nbd0 18369 2024-09-18T04:39:01.368 INFO:tasks.workunit.client.0.smithi115.stderr:+ local args=/dev/nbd0 2024-09-18T04:39:01.368 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pid=18369 2024-09-18T04:39:01.368 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-18T04:39:01.368 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:39:01.368 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:39:01.370 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:39:01.370 INFO:tasks.workunit.client.0.smithi115.stderr:++ which rbd 2024-09-18T04:39:01.373 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/bin/rbd 2024-09-18T04:39:01.373 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:39:01.373 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-18T04:39:01.377 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:01.377 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:01.377 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:39:01.377 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:01.387 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:39:01.418 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:01.413+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:01.486 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd device --device-type nbd list 2024-09-18T04:39:01.486 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false grep '^18369\b' 2024-09-18T04:39:01.486 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep '^18369\b' 2024-09-18T04:39:01.494 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:01.494 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:01.494 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:39:01.494 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:39:01.494 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:01.504 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:39:01.519 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:01.513+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:01.558 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:39:01.559 INFO:tasks.workunit.client.0.smithi115.stderr:+ ps -C rbd-nbd 2024-09-18T04:39:01.559 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false grep '^ *18369\b' 2024-09-18T04:39:01.559 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep '^ *18369\b' 2024-09-18T04:39:01.570 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:39:01.571 INFO:tasks.workunit.client.0.smithi115.stderr:+ sleep 0.5 2024-09-18T04:39:01.620 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:01.613+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:01.655 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:01.655 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:39:01.655 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:39:01.656 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [] 2024-09-18T04:39:01.656 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:01.665 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:39:01.721 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:01.713+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:01.757 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T04:39:01.767 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:01.767 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:01.767 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:39:01.767 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:01.777 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:39:01.822 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:01.817+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:01.911 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:39:01.923 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:01.917+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:01.929 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:01.929 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:01.929 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:39:01.929 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:39:01.929 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:01.932 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:39:01.939 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:39:01.951 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:39:01.973 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:39:01.994 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:39:02.015 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:39:02.017 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:39:02.018 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:39:02.018 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:39:02.018 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:39:02.018 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:39:02.018 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:39:02.023 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:02.017+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:02.048 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:02.048 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:02.058 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:39:02.072 INFO:tasks.workunit.client.0.smithi115.stderr:+ DEV= 2024-09-18T04:39:02.072 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd bench testrbdnbd17342 --io-type write --io-size=1024 --io-total=1024 2024-09-18T04:39:02.124 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:02.117+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:02.150 INFO:tasks.workunit.client.0.smithi115.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2024-09-18T04:39:02.153 INFO:tasks.workunit.client.0.smithi115.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-18T04:39:02.153 INFO:tasks.workunit.client.0.smithi115.stdout:elapsed: 0 ops: 1 ops/sec: 250.004 bytes/sec: 250 KiB/s 2024-09-18T04:39:02.169 INFO:tasks.workunit.client.0.smithi115.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd17342 2024-09-18T04:39:02.169 INFO:tasks.workunit.client.0.smithi115.stderr:++ local cmd 2024-09-18T04:39:02.170 INFO:tasks.workunit.client.0.smithi115.stderr:+++ id -u 2024-09-18T04:39:02.171 INFO:tasks.workunit.client.0.smithi115.stderr:++ '[' 1000 -eq 0 ']' 2024-09-18T04:39:02.172 INFO:tasks.workunit.client.0.smithi115.stderr:+++ which rbd 2024-09-18T04:39:02.174 INFO:tasks.workunit.client.0.smithi115.stderr:++ cmd=/usr/bin/rbd 2024-09-18T04:39:02.175 INFO:tasks.workunit.client.0.smithi115.stderr:++ shift 2024-09-18T04:39:02.175 INFO:tasks.workunit.client.0.smithi115.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd17342 2024-09-18T04:39:02.185 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:02.186 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:02.196 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:39:02.224 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:02.217+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:39:02.302 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:02.302 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:02.312 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:39:02.325 INFO:tasks.workunit.client.0.smithi115.stderr:+ DEV=/dev/nbd0 2024-09-18T04:39:02.325 INFO:tasks.workunit.client.0.smithi115.stderr:+ get_pid rbd 2024-09-18T04:39:02.325 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pool=rbd 2024-09-18T04:39:02.325 INFO:tasks.workunit.client.0.smithi115.stderr:+ local ns= 2024-09-18T04:39:02.326 INFO:tasks.workunit.client.0.smithi115.stderr:++ rbd device --device-type nbd --format xml list 2024-09-18T04:39:02.326 INFO:tasks.workunit.client.0.smithi115.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd17342'\''][device='\''/dev/nbd0'\'']/id' 2024-09-18T04:39:02.358 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:02.353+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:02.409 INFO:tasks.workunit.client.0.smithi115.stderr:+ PID=18482 2024-09-18T04:39:02.409 INFO:tasks.workunit.client.0.smithi115.stderr:+ test -n 18482 2024-09-18T04:39:02.409 INFO:tasks.workunit.client.0.smithi115.stderr:+ ps -p 18482 -C rbd-nbd 2024-09-18T04:39:02.418 INFO:tasks.workunit.client.0.smithi115.stdout: PID TTY TIME CMD 2024-09-18T04:39:02.418 INFO:tasks.workunit.client.0.smithi115.stdout: 18482 ? 00:00:00 rbd-nbd 2024-09-18T04:39:02.418 INFO:tasks.workunit.client.0.smithi115.stderr:+ unmap_device testrbdnbd17342 18482 2024-09-18T04:39:02.418 INFO:tasks.workunit.client.0.smithi115.stderr:+ local args=testrbdnbd17342 2024-09-18T04:39:02.419 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pid=18482 2024-09-18T04:39:02.419 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo rbd device --device-type nbd unmap testrbdnbd17342 2024-09-18T04:39:02.419 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:39:02.419 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:39:02.420 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:39:02.421 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:02.421 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:02.421 INFO:tasks.workunit.client.0.smithi115.stderr:++ which rbd 2024-09-18T04:39:02.422 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/bin/rbd 2024-09-18T04:39:02.422 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:39:02.422 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap testrbdnbd17342 2024-09-18T04:39:02.426 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:02.421+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:02.431 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:39:02.492 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-18T04:39:02.524 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd device --device-type nbd list 2024-09-18T04:39:02.524 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false grep '^18482\b' 2024-09-18T04:39:02.524 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep '^18482\b' 2024-09-18T04:39:02.538 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:02.533+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:02.556 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:02.556 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:02.566 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:39:02.592 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:39:02.593 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false grep '^ *18482\b' 2024-09-18T04:39:02.593 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep '^ *18482\b' 2024-09-18T04:39:02.593 INFO:tasks.workunit.client.0.smithi115.stderr:+ ps -C rbd-nbd 2024-09-18T04:39:02.605 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:39:02.605 INFO:tasks.workunit.client.0.smithi115.stderr:+ sleep 0.5 2024-09-18T04:39:02.627 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:02.621+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:39:02.676 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:02.677 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:02.687 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:39:02.729 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:02.725+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:02.789 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:02.789 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:02.799 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:39:02.830 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:02.825+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:02.902 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:02.902 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:02.912 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:39:02.931 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:02.925+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:03.031 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:03.025+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:03.063 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:03.064 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:03.073 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:39:03.107 INFO:tasks.workunit.client.0.smithi115.stderr:+ DEV= 2024-09-18T04:39:03.107 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd snap create rbd/testrbdnbd17342@snap 2024-09-18T04:39:03.132 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:03.125+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:03.184 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:03.184 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:03.194 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:39:03.238 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:03.233+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:03.314 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:03.314 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:03.325 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:39:03.334 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:03.329+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:03.435 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:03.429+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:03.438 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:03.438 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:03.448 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:39:03.473 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-18T04:39:03.473 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T04:39:03.473 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-18T04:39:03.474 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-18T04:39:03.496 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-09-18T04:39:03.508 INFO:tasks.workunit.client.0.smithi115.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-18T04:39:03.523 INFO:tasks.workunit.client.0.smithi115.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd17342@snap 2024-09-18T04:39:03.523 INFO:tasks.workunit.client.0.smithi115.stderr:++ local cmd 2024-09-18T04:39:03.524 INFO:tasks.workunit.client.0.smithi115.stderr:+++ id -u 2024-09-18T04:39:03.526 INFO:tasks.workunit.client.0.smithi115.stderr:++ '[' 1000 -eq 0 ']' 2024-09-18T04:39:03.526 INFO:tasks.workunit.client.0.smithi115.stderr:+++ which rbd 2024-09-18T04:39:03.529 INFO:tasks.workunit.client.0.smithi115.stderr:++ cmd=/usr/bin/rbd 2024-09-18T04:39:03.529 INFO:tasks.workunit.client.0.smithi115.stderr:++ shift 2024-09-18T04:39:03.529 INFO:tasks.workunit.client.0.smithi115.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd17342@snap 2024-09-18T04:39:03.535 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:03.529+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:39:03.597 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:03.597 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:03.604 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:39:03.637 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:03.629+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:03.700 INFO:tasks.workunit.client.0.smithi115.stderr:+ DEV=/dev/nbd0 2024-09-18T04:39:03.700 INFO:tasks.workunit.client.0.smithi115.stderr:+ get_pid rbd 2024-09-18T04:39:03.701 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pool=rbd 2024-09-18T04:39:03.701 INFO:tasks.workunit.client.0.smithi115.stderr:+ local ns= 2024-09-18T04:39:03.701 INFO:tasks.workunit.client.0.smithi115.stderr:++ rbd device --device-type nbd --format xml list 2024-09-18T04:39:03.702 INFO:tasks.workunit.client.0.smithi115.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd17342'\''][device='\''/dev/nbd0'\'']/id' 2024-09-18T04:39:03.710 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:03.710 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:03.725 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:39:03.738 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:03.733+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:39:03.784 INFO:tasks.workunit.client.0.smithi115.stderr:+ PID=18581 2024-09-18T04:39:03.784 INFO:tasks.workunit.client.0.smithi115.stderr:+ test -n 18581 2024-09-18T04:39:03.784 INFO:tasks.workunit.client.0.smithi115.stderr:+ ps -p 18581 -C rbd-nbd 2024-09-18T04:39:03.793 INFO:tasks.workunit.client.0.smithi115.stdout: PID TTY TIME CMD 2024-09-18T04:39:03.793 INFO:tasks.workunit.client.0.smithi115.stdout: 18581 ? 00:00:00 rbd-nbd 2024-09-18T04:39:03.794 INFO:tasks.workunit.client.0.smithi115.stderr:+ unmap_device testrbdnbd17342@snap 18581 2024-09-18T04:39:03.794 INFO:tasks.workunit.client.0.smithi115.stderr:+ local args=testrbdnbd17342@snap 2024-09-18T04:39:03.794 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pid=18581 2024-09-18T04:39:03.794 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo rbd device --device-type nbd unmap testrbdnbd17342@snap 2024-09-18T04:39:03.794 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:39:03.794 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:39:03.795 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:39:03.795 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:03.795 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 65, 2024-09-18T04:39:03.795 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "be6146ba-a991-4f6e-b359-99cbedc9f9b1", 2024-09-18T04:39:03.796 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-09-18T04:37:15.676634+0000", 2024-09-18T04:39:03.796 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-09-18T04:39:03.477907+0000", 2024-09-18T04:39:03.796 INFO:teuthology.orchestra.run.smithi028.stdout: "last_up_change": "2024-09-18T04:37:25.403175+0000", 2024-09-18T04:39:03.796 INFO:teuthology.orchestra.run.smithi028.stdout: "last_in_change": "2024-09-18T04:37:50.652169+0000", 2024-09-18T04:39:03.796 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-18T04:39:03.796 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_num": 5804032, 2024-09-18T04:39:03.796 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_set": [ 2024-09-18T04:39:03.796 INFO:teuthology.orchestra.run.smithi028.stdout: "nodeep-scrub", 2024-09-18T04:39:03.796 INFO:teuthology.orchestra.run.smithi028.stdout: "pglog_hardlimit", 2024-09-18T04:39:03.796 INFO:teuthology.orchestra.run.smithi028.stdout: "purged_snapdirs", 2024-09-18T04:39:03.796 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_deletes", 2024-09-18T04:39:03.796 INFO:teuthology.orchestra.run.smithi028.stdout: "sortbitwise" 2024-09-18T04:39:03.796 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-18T04:39:03.797 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_version": 5, 2024-09-18T04:39:03.797 INFO:teuthology.orchestra.run.smithi028.stdout: "full_ratio": 0.94999998807907104, 2024-09-18T04:39:03.797 INFO:teuthology.orchestra.run.smithi028.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-18T04:39:03.797 INFO:teuthology.orchestra.run.smithi028.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-18T04:39:03.797 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_snapshot": "", 2024-09-18T04:39:03.797 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_max": 3, 2024-09-18T04:39:03.797 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd": 6, 2024-09-18T04:39:03.797 INFO:teuthology.orchestra.run.smithi028.stdout: "require_min_compat_client": "luminous", 2024-09-18T04:39:03.797 INFO:teuthology.orchestra.run.smithi028.stdout: "min_compat_client": "luminous", 2024-09-18T04:39:03.797 INFO:teuthology.orchestra.run.smithi028.stdout: "require_osd_release": "reef", 2024-09-18T04:39:03.797 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_crimson": false, 2024-09-18T04:39:03.797 INFO:teuthology.orchestra.run.smithi028.stdout: "pools": [ 2024-09-18T04:39:03.797 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.798 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 1, 2024-09-18T04:39:03.798 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": ".mgr", 2024-09-18T04:39:03.798 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-18T04:37:25.654950+0000", 2024-09-18T04:39:03.798 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-18T04:39:03.798 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-09-18T04:39:03.798 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-18T04:39:03.798 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 2, 2024-09-18T04:39:03.798 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 1, 2024-09-18T04:39:03.798 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-18T04:39:03.798 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-18T04:39:03.798 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-18T04:39:03.798 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T04:39:03.798 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T04:39:03.798 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-18T04:39:03.799 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-18T04:39:03.799 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 1, 2024-09-18T04:39:03.799 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 1, 2024-09-18T04:39:03.799 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 1, 2024-09-18T04:39:03.799 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 1, 2024-09-18T04:39:03.799 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 1, 2024-09-18T04:39:03.799 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-18T04:39:03.799 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-18T04:39:03.799 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-18T04:39:03.799 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-18T04:39:03.799 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-18T04:39:03.799 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-18T04:39:03.799 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-18T04:39:03.799 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.800 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "16", 2024-09-18T04:39:03.800 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-18T04:39:03.800 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T04:39:03.800 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T04:39:03.800 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-18T04:39:03.800 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-18T04:39:03.800 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-09-18T04:39:03.800 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-09-18T04:39:03.800 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-18T04:39:03.800 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-18T04:39:03.800 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-18T04:39:03.800 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-18T04:39:03.801 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-18T04:39:03.801 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-18T04:39:03.801 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-18T04:39:03.801 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-18T04:39:03.801 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-18T04:39:03.801 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-18T04:39:03.801 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-18T04:39:03.802 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T04:39:03.802 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T04:39:03.802 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T04:39:03.802 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-18T04:39:03.802 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-18T04:39:03.802 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-18T04:39:03.802 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-18T04:39:03.802 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-18T04:39:03.803 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.803 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-18T04:39:03.803 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-18T04:39:03.803 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-18T04:39:03.803 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-18T04:39:03.803 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-18T04:39:03.803 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T04:39:03.803 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-18T04:39:03.803 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-18T04:39:03.803 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-18T04:39:03.803 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-18T04:39:03.803 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-18T04:39:03.803 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-09-18T04:39:03.804 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_max": 32, 2024-09-18T04:39:03.804 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_min": 1 2024-09-18T04:39:03.804 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.804 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-18T04:39:03.804 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": {} 2024-09-18T04:39:03.804 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.804 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-09-18T04:39:03.804 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 6.059999942779541, 2024-09-18T04:39:03.804 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 6.059999942779541, 2024-09-18T04:39:03.804 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.33000001311302185, 2024-09-18T04:39:03.804 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2, 2024-09-18T04:39:03.804 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2, 2024-09-18T04:39:03.804 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 1, 2024-09-18T04:39:03.805 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 1, 2024-09-18T04:39:03.805 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 1 2024-09-18T04:39:03.805 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:03.805 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.805 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.805 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 2, 2024-09-18T04:39:03.805 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "rbd", 2024-09-18T04:39:03.805 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-18T04:37:32.672462+0000", 2024-09-18T04:39:03.805 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 8193, 2024-09-18T04:39:03.805 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-18T04:39:03.805 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-18T04:39:03.805 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 2, 2024-09-18T04:39:03.805 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 1, 2024-09-18T04:39:03.805 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-18T04:39:03.806 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-18T04:39:03.806 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-18T04:39:03.806 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T04:39:03.806 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T04:39:03.806 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-18T04:39:03.806 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-18T04:39:03.806 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 8, 2024-09-18T04:39:03.806 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 8, 2024-09-18T04:39:03.806 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 8, 2024-09-18T04:39:03.806 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 8, 2024-09-18T04:39:03.806 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 8, 2024-09-18T04:39:03.806 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-18T04:39:03.806 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-18T04:39:03.806 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-18T04:39:03.807 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-18T04:39:03.807 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-18T04:39:03.807 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-18T04:39:03.807 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-18T04:39:03.807 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.807 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "65", 2024-09-18T04:39:03.807 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-18T04:39:03.807 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T04:39:03.807 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T04:39:03.807 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-18T04:39:03.807 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-18T04:39:03.807 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 4, 2024-09-18T04:39:03.807 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 65, 2024-09-18T04:39:03.807 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-18T04:39:03.807 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-18T04:39:03.808 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-18T04:39:03.808 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-18T04:39:03.808 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-18T04:39:03.808 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-18T04:39:03.808 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-18T04:39:03.808 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-18T04:39:03.808 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-18T04:39:03.808 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-18T04:39:03.808 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-18T04:39:03.808 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T04:39:03.808 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T04:39:03.808 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T04:39:03.808 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-18T04:39:03.809 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-18T04:39:03.809 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-18T04:39:03.809 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-18T04:39:03.809 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-18T04:39:03.809 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.809 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-18T04:39:03.809 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-18T04:39:03.809 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-18T04:39:03.809 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-18T04:39:03.809 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-18T04:39:03.809 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T04:39:03.809 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-18T04:39:03.809 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-18T04:39:03.809 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-18T04:39:03.810 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-18T04:39:03.810 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-18T04:39:03.810 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-09-18T04:39:03.810 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-18T04:39:03.810 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd": {} 2024-09-18T04:39:03.810 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.810 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-09-18T04:39:03.810 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2.2699999809265137, 2024-09-18T04:39:03.810 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2.2699999809265137, 2024-09-18T04:39:03.810 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.82999998331069946, 2024-09-18T04:39:03.810 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.8799999952316284, 2024-09-18T04:39:03.810 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.8799999952316284, 2024-09-18T04:39:03.810 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 1, 2024-09-18T04:39:03.810 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 1, 2024-09-18T04:39:03.810 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 1 2024-09-18T04:39:03.811 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:03.811 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.811 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.811 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 3, 2024-09-18T04:39:03.811 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "new_default_pool17342", 2024-09-18T04:39:03.811 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-18T04:38:42.986132+0000", 2024-09-18T04:39:03.811 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 8193, 2024-09-18T04:39:03.811 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-18T04:39:03.811 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-18T04:39:03.811 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 2, 2024-09-18T04:39:03.811 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 1, 2024-09-18T04:39:03.811 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-18T04:39:03.811 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-18T04:39:03.811 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-18T04:39:03.812 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T04:39:03.812 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T04:39:03.812 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-18T04:39:03.812 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-18T04:39:03.812 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 8, 2024-09-18T04:39:03.812 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 8, 2024-09-18T04:39:03.812 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 8, 2024-09-18T04:39:03.812 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 8, 2024-09-18T04:39:03.812 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 8, 2024-09-18T04:39:03.812 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-18T04:39:03.812 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-18T04:39:03.812 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-18T04:39:03.812 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-18T04:39:03.812 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-18T04:39:03.813 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-18T04:39:03.813 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-18T04:39:03.813 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.813 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "55", 2024-09-18T04:39:03.813 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-18T04:39:03.813 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T04:39:03.813 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T04:39:03.813 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-18T04:39:03.813 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-18T04:39:03.813 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 3, 2024-09-18T04:39:03.813 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 55, 2024-09-18T04:39:03.813 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-18T04:39:03.813 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-18T04:39:03.814 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-18T04:39:03.814 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-18T04:39:03.814 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-18T04:39:03.814 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-18T04:39:03.814 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-18T04:39:03.814 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-18T04:39:03.814 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-18T04:39:03.814 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-18T04:39:03.814 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-18T04:39:03.814 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T04:39:03.814 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T04:39:03.814 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T04:39:03.814 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-18T04:39:03.814 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-18T04:39:03.815 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-18T04:39:03.815 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-18T04:39:03.815 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-18T04:39:03.815 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.815 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-18T04:39:03.815 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-18T04:39:03.815 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-18T04:39:03.815 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-18T04:39:03.815 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-18T04:39:03.815 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T04:39:03.815 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-18T04:39:03.815 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-18T04:39:03.815 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-18T04:39:03.815 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-18T04:39:03.816 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-18T04:39:03.816 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-09-18T04:39:03.816 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-18T04:39:03.816 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd": {} 2024-09-18T04:39:03.816 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.816 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-09-18T04:39:03.816 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2.2699999809265137, 2024-09-18T04:39:03.816 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2.2699999809265137, 2024-09-18T04:39:03.816 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.82999998331069946, 2024-09-18T04:39:03.816 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.8799999952316284, 2024-09-18T04:39:03.816 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.8799999952316284, 2024-09-18T04:39:03.816 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 1, 2024-09-18T04:39:03.816 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 1, 2024-09-18T04:39:03.816 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 1 2024-09-18T04:39:03.817 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:03.817 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:03.817 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-18T04:39:03.817 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-09-18T04:39:03.817 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.817 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-09-18T04:39:03.817 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "6177002d-72e6-4888-bcde-29b6a92d9dc4", 2024-09-18T04:39:03.817 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-18T04:39:03.817 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-18T04:39:03.817 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-18T04:39:03.817 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-18T04:39:03.817 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-18T04:39:03.817 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-18T04:39:03.817 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 13, 2024-09-18T04:39:03.818 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 52, 2024-09-18T04:39:03.818 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-18T04:39:03.818 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-18T04:39:03.818 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-18T04:39:03.818 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:39:03.818 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.818 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:39:03.818 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6801", 2024-09-18T04:39:03.818 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1554407851 2024-09-18T04:39:03.818 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.818 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.818 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:39:03.818 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6803", 2024-09-18T04:39:03.818 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1554407851 2024-09-18T04:39:03.819 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:03.819 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:03.819 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.819 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-18T04:39:03.819 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:39:03.819 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.819 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:39:03.819 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6805", 2024-09-18T04:39:03.819 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1554407851 2024-09-18T04:39:03.819 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.819 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.819 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:39:03.819 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6807", 2024-09-18T04:39:03.819 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1554407851 2024-09-18T04:39:03.820 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:03.820 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:03.820 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.820 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-18T04:39:03.820 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:39:03.820 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.820 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:39:03.820 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6813", 2024-09-18T04:39:03.820 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1554407851 2024-09-18T04:39:03.820 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.820 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.820 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:39:03.820 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6815", 2024-09-18T04:39:03.820 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1554407851 2024-09-18T04:39:03.821 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:03.821 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:03.821 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.821 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-18T04:39:03.821 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:39:03.821 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.821 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:39:03.821 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6809", 2024-09-18T04:39:03.821 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1554407851 2024-09-18T04:39:03.821 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.821 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.821 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:39:03.821 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6811", 2024-09-18T04:39:03.821 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1554407851 2024-09-18T04:39:03.822 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:03.822 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:03.822 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.822 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6803/1554407851", 2024-09-18T04:39:03.822 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6807/1554407851", 2024-09-18T04:39:03.822 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6815/1554407851", 2024-09-18T04:39:03.822 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6811/1554407851", 2024-09-18T04:39:03.822 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-18T04:39:03.822 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-18T04:39:03.822 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-18T04:39:03.822 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:03.822 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.822 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.823 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-09-18T04:39:03.823 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "4f62cb09-3457-418c-bb69-491d9b70914c", 2024-09-18T04:39:03.823 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-18T04:39:03.823 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-18T04:39:03.823 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-18T04:39:03.823 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-18T04:39:03.823 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-18T04:39:03.823 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-18T04:39:03.823 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 12, 2024-09-18T04:39:03.823 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 52, 2024-09-18T04:39:03.823 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-18T04:39:03.823 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-18T04:39:03.823 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-18T04:39:03.823 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:39:03.823 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.824 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:39:03.824 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6800", 2024-09-18T04:39:03.824 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 511141401 2024-09-18T04:39:03.824 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.824 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.824 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:39:03.824 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6802", 2024-09-18T04:39:03.824 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 511141401 2024-09-18T04:39:03.824 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:03.824 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:03.824 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.824 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-18T04:39:03.824 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:39:03.824 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.825 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:39:03.825 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6804", 2024-09-18T04:39:03.825 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 511141401 2024-09-18T04:39:03.825 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.825 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.825 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:39:03.825 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6806", 2024-09-18T04:39:03.825 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 511141401 2024-09-18T04:39:03.825 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:03.825 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:03.825 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.825 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-18T04:39:03.825 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:39:03.825 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.826 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:39:03.826 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6812", 2024-09-18T04:39:03.826 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 511141401 2024-09-18T04:39:03.826 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.826 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.826 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:39:03.826 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6814", 2024-09-18T04:39:03.826 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 511141401 2024-09-18T04:39:03.826 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:03.826 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:03.826 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.826 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-18T04:39:03.826 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:39:03.826 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.827 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:39:03.827 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6808", 2024-09-18T04:39:03.827 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 511141401 2024-09-18T04:39:03.827 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.827 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.827 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:39:03.827 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6810", 2024-09-18T04:39:03.827 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 511141401 2024-09-18T04:39:03.827 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:03.827 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:03.827 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.827 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6802/511141401", 2024-09-18T04:39:03.827 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6806/511141401", 2024-09-18T04:39:03.827 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6814/511141401", 2024-09-18T04:39:03.828 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6810/511141401", 2024-09-18T04:39:03.828 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-18T04:39:03.828 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-18T04:39:03.828 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-18T04:39:03.828 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:03.828 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.828 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.828 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-09-18T04:39:03.828 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "854fe031-3b87-4290-9c2f-3fb196fb806c", 2024-09-18T04:39:03.828 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-18T04:39:03.828 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-18T04:39:03.828 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-18T04:39:03.828 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-18T04:39:03.828 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-18T04:39:03.828 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-18T04:39:03.829 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 12, 2024-09-18T04:39:03.829 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 52, 2024-09-18T04:39:03.829 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-18T04:39:03.829 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-18T04:39:03.829 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-18T04:39:03.829 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:39:03.829 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.829 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:39:03.829 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6816", 2024-09-18T04:39:03.829 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2623426735 2024-09-18T04:39:03.829 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.829 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.829 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:39:03.830 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6817", 2024-09-18T04:39:03.830 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2623426735 2024-09-18T04:39:03.830 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:03.830 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:03.830 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.830 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-18T04:39:03.830 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:39:03.830 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.830 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:39:03.830 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6818", 2024-09-18T04:39:03.830 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2623426735 2024-09-18T04:39:03.830 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.830 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.830 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:39:03.830 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6819", 2024-09-18T04:39:03.831 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2623426735 2024-09-18T04:39:03.831 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:03.831 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:03.831 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.831 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-18T04:39:03.831 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:39:03.831 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.831 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:39:03.831 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6822", 2024-09-18T04:39:03.831 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2623426735 2024-09-18T04:39:03.831 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.831 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.831 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:39:03.832 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6823", 2024-09-18T04:39:03.832 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2623426735 2024-09-18T04:39:03.832 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:03.832 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:03.832 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.832 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-18T04:39:03.832 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:39:03.832 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.832 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:39:03.832 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6820", 2024-09-18T04:39:03.832 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2623426735 2024-09-18T04:39:03.832 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.832 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.832 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:39:03.833 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6821", 2024-09-18T04:39:03.833 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2623426735 2024-09-18T04:39:03.833 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:03.833 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:03.833 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.833 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6817/2623426735", 2024-09-18T04:39:03.833 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6819/2623426735", 2024-09-18T04:39:03.833 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6823/2623426735", 2024-09-18T04:39:03.833 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6821/2623426735", 2024-09-18T04:39:03.833 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-18T04:39:03.833 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-18T04:39:03.833 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-18T04:39:03.833 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:03.834 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.834 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.834 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-09-18T04:39:03.834 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "c2625f98-780b-4f86-bf36-a48d2a227756", 2024-09-18T04:39:03.834 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-18T04:39:03.834 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-18T04:39:03.834 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-18T04:39:03.834 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-18T04:39:03.834 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-18T04:39:03.834 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-18T04:39:03.834 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 12, 2024-09-18T04:39:03.834 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 32, 2024-09-18T04:39:03.834 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-18T04:39:03.834 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-18T04:39:03.835 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-18T04:39:03.835 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:39:03.835 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.835 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:39:03.835 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6808", 2024-09-18T04:39:03.835 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2746051132 2024-09-18T04:39:03.835 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.835 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.835 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:39:03.835 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6809", 2024-09-18T04:39:03.835 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2746051132 2024-09-18T04:39:03.835 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:03.835 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:03.835 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.836 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-18T04:39:03.836 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:39:03.836 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.836 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:39:03.836 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6810", 2024-09-18T04:39:03.836 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2746051132 2024-09-18T04:39:03.836 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.836 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.836 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:39:03.836 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6811", 2024-09-18T04:39:03.836 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2746051132 2024-09-18T04:39:03.836 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:03.836 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:03.837 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.837 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-18T04:39:03.837 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:39:03.837 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.837 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:39:03.837 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6814", 2024-09-18T04:39:03.837 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2746051132 2024-09-18T04:39:03.837 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.837 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.837 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:39:03.837 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6815", 2024-09-18T04:39:03.837 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2746051132 2024-09-18T04:39:03.837 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:03.837 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:03.838 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.838 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-18T04:39:03.838 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:39:03.838 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.838 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:39:03.838 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6812", 2024-09-18T04:39:03.838 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2746051132 2024-09-18T04:39:03.838 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.838 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.838 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:39:03.838 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6813", 2024-09-18T04:39:03.838 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2746051132 2024-09-18T04:39:03.838 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:03.838 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:03.839 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.839 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.82:6809/2746051132", 2024-09-18T04:39:03.839 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.82:6811/2746051132", 2024-09-18T04:39:03.839 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.82:6815/2746051132", 2024-09-18T04:39:03.839 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.82:6813/2746051132", 2024-09-18T04:39:03.839 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-18T04:39:03.839 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-18T04:39:03.839 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-18T04:39:03.839 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:03.839 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.839 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.839 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-09-18T04:39:03.839 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "deb22d86-4081-484d-96eb-9cb9d96b60bc", 2024-09-18T04:39:03.839 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-18T04:39:03.840 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-18T04:39:03.840 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-18T04:39:03.840 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-18T04:39:03.840 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-18T04:39:03.840 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-18T04:39:03.840 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 13, 2024-09-18T04:39:03.840 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 52, 2024-09-18T04:39:03.840 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-18T04:39:03.840 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-18T04:39:03.840 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-18T04:39:03.840 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:39:03.840 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.840 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:39:03.840 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6800", 2024-09-18T04:39:03.841 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2005712460 2024-09-18T04:39:03.841 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.841 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.841 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:39:03.841 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6801", 2024-09-18T04:39:03.841 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2005712460 2024-09-18T04:39:03.841 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:03.841 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:03.841 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.841 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-18T04:39:03.841 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:39:03.841 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.841 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:39:03.841 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6802", 2024-09-18T04:39:03.842 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2005712460 2024-09-18T04:39:03.842 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.842 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.842 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:39:03.842 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6803", 2024-09-18T04:39:03.842 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2005712460 2024-09-18T04:39:03.842 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:03.842 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:03.842 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.842 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-18T04:39:03.842 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:39:03.842 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.842 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:39:03.842 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6806", 2024-09-18T04:39:03.843 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2005712460 2024-09-18T04:39:03.843 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.843 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.843 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:39:03.843 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6807", 2024-09-18T04:39:03.843 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2005712460 2024-09-18T04:39:03.843 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:03.843 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:03.843 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.843 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-18T04:39:03.843 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:39:03.843 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.843 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:39:03.843 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6804", 2024-09-18T04:39:03.844 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2005712460 2024-09-18T04:39:03.844 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.844 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.844 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:39:03.844 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6805", 2024-09-18T04:39:03.844 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2005712460 2024-09-18T04:39:03.844 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:03.844 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:03.844 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.844 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.82:6801/2005712460", 2024-09-18T04:39:03.844 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.82:6803/2005712460", 2024-09-18T04:39:03.844 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.82:6807/2005712460", 2024-09-18T04:39:03.844 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.82:6805/2005712460", 2024-09-18T04:39:03.844 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-18T04:39:03.845 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-18T04:39:03.845 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-18T04:39:03.845 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:03.845 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.845 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.845 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-09-18T04:39:03.845 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "061d332c-f51b-403d-bb18-eba415c97e7f", 2024-09-18T04:39:03.845 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-18T04:39:03.845 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-18T04:39:03.845 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-18T04:39:03.845 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-18T04:39:03.845 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-18T04:39:03.845 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-18T04:39:03.845 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 13, 2024-09-18T04:39:03.846 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 62, 2024-09-18T04:39:03.846 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-18T04:39:03.846 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-18T04:39:03.846 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-18T04:39:03.846 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:39:03.846 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.846 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:39:03.846 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6816", 2024-09-18T04:39:03.846 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1267644232 2024-09-18T04:39:03.846 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.846 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.846 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:39:03.846 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6817", 2024-09-18T04:39:03.846 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1267644232 2024-09-18T04:39:03.847 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:03.847 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:03.847 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.847 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-18T04:39:03.847 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:39:03.847 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.847 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:39:03.847 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6818", 2024-09-18T04:39:03.847 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1267644232 2024-09-18T04:39:03.847 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.847 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.847 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:39:03.847 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6819", 2024-09-18T04:39:03.847 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1267644232 2024-09-18T04:39:03.847 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:03.848 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:03.848 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.848 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-18T04:39:03.848 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:39:03.848 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.848 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:39:03.848 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6822", 2024-09-18T04:39:03.848 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1267644232 2024-09-18T04:39:03.848 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.848 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.848 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:39:03.848 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6823", 2024-09-18T04:39:03.848 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1267644232 2024-09-18T04:39:03.848 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:03.849 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:03.849 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.849 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-18T04:39:03.849 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:39:03.849 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.849 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:39:03.849 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6820", 2024-09-18T04:39:03.849 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1267644232 2024-09-18T04:39:03.849 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.849 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.849 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:39:03.849 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6821", 2024-09-18T04:39:03.849 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1267644232 2024-09-18T04:39:03.850 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:03.850 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:03.850 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.850 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.82:6817/1267644232", 2024-09-18T04:39:03.850 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.82:6819/1267644232", 2024-09-18T04:39:03.850 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.82:6823/1267644232", 2024-09-18T04:39:03.850 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.82:6821/1267644232", 2024-09-18T04:39:03.850 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-18T04:39:03.850 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-18T04:39:03.850 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-18T04:39:03.850 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:03.850 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:03.850 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-18T04:39:03.851 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_xinfo": [ 2024-09-18T04:39:03.851 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.851 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-09-18T04:39:03.851 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-18T04:39:03.851 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-18T04:39:03.851 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-18T04:39:03.851 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138322906710015, 2024-09-18T04:39:03.851 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-18T04:39:03.851 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-18T04:37:22.192087+0000", 2024-09-18T04:39:03.851 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-18T04:39:03.851 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.851 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.851 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-09-18T04:39:03.852 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-18T04:39:03.852 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-18T04:39:03.852 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-18T04:39:03.852 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138322906710015, 2024-09-18T04:39:03.852 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-18T04:39:03.852 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-18T04:37:22.017833+0000", 2024-09-18T04:39:03.852 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-18T04:39:03.852 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.852 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.852 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-09-18T04:39:03.852 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-18T04:39:03.852 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-18T04:39:03.852 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-18T04:39:03.852 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138322906710015, 2024-09-18T04:39:03.853 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-18T04:39:03.853 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-18T04:37:22.755774+0000", 2024-09-18T04:39:03.853 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-18T04:39:03.853 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.853 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.853 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-09-18T04:39:03.853 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-18T04:39:03.853 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-18T04:39:03.853 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-18T04:39:03.853 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138322906710015, 2024-09-18T04:39:03.853 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-18T04:39:03.853 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-18T04:37:22.663785+0000", 2024-09-18T04:39:03.853 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-18T04:39:03.854 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.854 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.854 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-09-18T04:39:03.854 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-18T04:39:03.854 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-18T04:39:03.854 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-18T04:39:03.854 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138322906710015, 2024-09-18T04:39:03.854 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-18T04:39:03.854 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-18T04:37:22.531758+0000", 2024-09-18T04:39:03.854 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-18T04:39:03.854 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.854 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.854 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-09-18T04:39:03.854 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-18T04:39:03.854 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-18T04:39:03.855 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-18T04:39:03.855 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138322906710015, 2024-09-18T04:39:03.855 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-18T04:39:03.855 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-18T04:37:22.400774+0000", 2024-09-18T04:39:03.855 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-18T04:39:03.855 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:03.855 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-18T04:39:03.855 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap": [ 2024-09-18T04:39:03.855 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.855 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "2.5", 2024-09-18T04:39:03.855 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-09-18T04:39:03.855 INFO:teuthology.orchestra.run.smithi028.stdout: 2, 2024-09-18T04:39:03.855 INFO:teuthology.orchestra.run.smithi028.stdout: 0 2024-09-18T04:39:03.855 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:03.856 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.856 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.856 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "2.7", 2024-09-18T04:39:03.856 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-09-18T04:39:03.856 INFO:teuthology.orchestra.run.smithi028.stdout: 3, 2024-09-18T04:39:03.856 INFO:teuthology.orchestra.run.smithi028.stdout: 5 2024-09-18T04:39:03.856 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:03.856 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:03.856 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-18T04:39:03.856 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap_items": [ 2024-09-18T04:39:03.856 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.856 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "2.1", 2024-09-18T04:39:03.856 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-09-18T04:39:03.856 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.857 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 2, 2024-09-18T04:39:03.857 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 1 2024-09-18T04:39:03.857 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:03.857 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:03.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.857 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.857 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "2.4", 2024-09-18T04:39:03.857 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-09-18T04:39:03.857 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:03.857 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-09-18T04:39:03.857 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 5 2024-09-18T04:39:03.857 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:03.857 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:03.857 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:03.858 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-18T04:39:03.858 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap_primaries": [], 2024-09-18T04:39:03.858 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_temp": [], 2024-09-18T04:39:03.858 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_temp": [], 2024-09-18T04:39:03.858 INFO:teuthology.orchestra.run.smithi028.stdout: "blocklist": {}, 2024-09-18T04:39:03.858 INFO:teuthology.orchestra.run.smithi028.stdout: "range_blocklist": {}, 2024-09-18T04:39:03.858 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profiles": { 2024-09-18T04:39:03.858 INFO:teuthology.orchestra.run.smithi028.stdout: "default": { 2024-09-18T04:39:03.858 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-failure-domain": "osd", 2024-09-18T04:39:03.858 INFO:teuthology.orchestra.run.smithi028.stdout: "k": "2", 2024-09-18T04:39:03.858 INFO:teuthology.orchestra.run.smithi028.stdout: "m": "1", 2024-09-18T04:39:03.858 INFO:teuthology.orchestra.run.smithi028.stdout: "plugin": "jerasure", 2024-09-18T04:39:03.858 INFO:teuthology.orchestra.run.smithi028.stdout: "technique": "reed_sol_van" 2024-09-18T04:39:03.858 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:03.858 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:03.859 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps_queue": [], 2024-09-18T04:39:03.859 INFO:teuthology.orchestra.run.smithi028.stdout: "new_removed_snaps": [], 2024-09-18T04:39:03.859 INFO:teuthology.orchestra.run.smithi028.stdout: "new_purged_snaps": [], 2024-09-18T04:39:03.859 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_node_flags": {}, 2024-09-18T04:39:03.859 INFO:teuthology.orchestra.run.smithi028.stdout: "device_class_flags": {}, 2024-09-18T04:39:03.859 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": { 2024-09-18T04:39:03.859 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_enabled": false, 2024-09-18T04:39:03.859 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_bucket_count": 0, 2024-09-18T04:39:03.859 INFO:teuthology.orchestra.run.smithi028.stdout: "degraded_stretch_mode": 0, 2024-09-18T04:39:03.859 INFO:teuthology.orchestra.run.smithi028.stdout: "recovering_stretch_mode": 0, 2024-09-18T04:39:03.859 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_bucket": 0 2024-09-18T04:39:03.859 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:03.859 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:03.861 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:39:03.862 INFO:tasks.workunit.client.0.smithi115.stderr:++ which rbd 2024-09-18T04:39:03.862 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/bin/rbd 2024-09-18T04:39:03.862 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:39:03.862 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap testrbdnbd17342@snap 2024-09-18T04:39:03.862 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:03.862 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:03.863 INFO:tasks.thrashosds.thrasher:j is {'epoch': 65, 'fsid': 'be6146ba-a991-4f6e-b359-99cbedc9f9b1', 'created': '2024-09-18T04:37:15.676634+0000', 'modified': '2024-09-18T04:39:03.477907+0000', 'last_up_change': '2024-09-18T04:37:25.403175+0000', 'last_in_change': '2024-09-18T04:37:50.652169+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-18T04:37:25.654950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-18T04:37:32.672462+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 4, 'snap_epoch': 65, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2.2699999809265137, 'score_stable': 2.2699999809265137, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'new_default_pool17342', 'create_time': '2024-09-18T04:38:42.986132+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 55, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2.2699999809265137, 'score_stable': 2.2699999809265137, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '6177002d-72e6-4888-bcde-29b6a92d9dc4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 52, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6801', 'nonce': 1554407851}, {'type': 'v1', 'addr': '172.21.15.28:6803', 'nonce': 1554407851}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6805', 'nonce': 1554407851}, {'type': 'v1', 'addr': '172.21.15.28:6807', 'nonce': 1554407851}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6813', 'nonce': 1554407851}, {'type': 'v1', 'addr': '172.21.15.28:6815', 'nonce': 1554407851}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6809', 'nonce': 1554407851}, {'type': 'v1', 'addr': '172.21.15.28:6811', 'nonce': 1554407851}]}, 'public_addr': '172.21.15.28:6803/1554407851', 'cluster_addr': '172.21.15.28:6807/1554407851', 'heartbeat_back_addr': '172.21.15.28:6815/1554407851', 'heartbeat_front_addr': '172.21.15.28:6811/1554407851', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '4f62cb09-3457-418c-bb69-491d9b70914c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 52, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6800', 'nonce': 511141401}, {'type': 'v1', 'addr': '172.21.15.28:6802', 'nonce': 511141401}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6804', 'nonce': 511141401}, {'type': 'v1', 'addr': '172.21.15.28:6806', 'nonce': 511141401}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6812', 'nonce': 511141401}, {'type': 'v1', 'addr': '172.21.15.28:6814', 'nonce': 511141401}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6808', 'nonce': 511141401}, {'type': 'v1', 'addr': '172.21.15.28:6810', 'nonce': 511141401}]}, 'public_addr': '172.21.15.28:6802/511141401', 'cluster_addr': '172.21.15.28:6806/511141401', 'heartbeat_back_addr': '172.21.15.28:6814/511141401', 'heartbeat_front_addr': '172.21.15.28:6810/511141401', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '854fe031-3b87-4290-9c2f-3fb196fb806c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 52, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6816', 'nonce': 2623426735}, {'type': 'v1', 'addr': '172.21.15.28:6817', 'nonce': 2623426735}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6818', 'nonce': 2623426735}, {'type': 'v1', 'addr': '172.21.15.28:6819', 'nonce': 2623426735}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6822', 'nonce': 2623426735}, {'type': 'v1', 'addr': '172.21.15.28:6823', 'nonce': 2623426735}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6820', 'nonce': 2623426735}, {'type': 'v1', 'addr': '172.21.15.28:6821', 'nonce': 2623426735}]}, 'public_addr': '172.21.15.28:6817/2623426735', 'cluster_addr': '172.21.15.28:6819/2623426735', 'heartbeat_back_addr': '172.21.15.28:6823/2623426735', 'heartbeat_front_addr': '172.21.15.28:6821/2623426735', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'c2625f98-780b-4f86-bf36-a48d2a227756', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 32, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6808', 'nonce': 2746051132}, {'type': 'v1', 'addr': '172.21.15.82:6809', 'nonce': 2746051132}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6810', 'nonce': 2746051132}, {'type': 'v1', 'addr': '172.21.15.82:6811', 'nonce': 2746051132}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6814', 'nonce': 2746051132}, {'type': 'v1', 'addr': '172.21.15.82:6815', 'nonce': 2746051132}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6812', 'nonce': 2746051132}, {'type': 'v1', 'addr': '172.21.15.82:6813', 'nonce': 2746051132}]}, 'public_addr': '172.21.15.82:6809/2746051132', 'cluster_addr': '172.21.15.82:6811/2746051132', 'heartbeat_back_addr': '172.21.15.82:6815/2746051132', 'heartbeat_front_addr': '172.21.15.82:6813/2746051132', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'deb22d86-4081-484d-96eb-9cb9d96b60bc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 52, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6800', 'nonce': 2005712460}, {'type': 'v1', 'addr': '172.21.15.82:6801', 'nonce': 2005712460}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6802', 'nonce': 2005712460}, {'type': 'v1', 'addr': '172.21.15.82:6803', 'nonce': 2005712460}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6806', 'nonce': 2005712460}, {'type': 'v1', 'addr': '172.21.15.82:6807', 'nonce': 2005712460}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6804', 'nonce': 2005712460}, {'type': 'v1', 'addr': '172.21.15.82:6805', 'nonce': 2005712460}]}, 'public_addr': '172.21.15.82:6801/2005712460', 'cluster_addr': '172.21.15.82:6803/2005712460', 'heartbeat_back_addr': '172.21.15.82:6807/2005712460', 'heartbeat_front_addr': '172.21.15.82:6805/2005712460', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '061d332c-f51b-403d-bb18-eba415c97e7f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 62, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6816', 'nonce': 1267644232}, {'type': 'v1', 'addr': '172.21.15.82:6817', 'nonce': 1267644232}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6818', 'nonce': 1267644232}, {'type': 'v1', 'addr': '172.21.15.82:6819', 'nonce': 1267644232}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6822', 'nonce': 1267644232}, {'type': 'v1', 'addr': '172.21.15.82:6823', 'nonce': 1267644232}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6820', 'nonce': 1267644232}, {'type': 'v1', 'addr': '172.21.15.82:6821', 'nonce': 1267644232}]}, 'public_addr': '172.21.15.82:6817/1267644232', 'cluster_addr': '172.21.15.82:6819/1267644232', 'heartbeat_back_addr': '172.21.15.82:6823/1267644232', 'heartbeat_front_addr': '172.21.15.82:6821/1267644232', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T04:37:22.192087+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T04:37:22.017833+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T04:37:22.755774+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T04:37:22.663785+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T04:37:22.531758+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T04:37:22.400774+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.5', 'osds': [2, 0]}, {'pgid': '2.7', 'osds': [3, 5]}], 'pg_upmap_items': [{'pgid': '2.1', 'mappings': [{'from': 2, 'to': 1}]}, {'pgid': '2.4', 'mappings': [{'from': 1, 'to': 5}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-18T04:39:03.864 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T04:39:03.864 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:39:03.864 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:03.853+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:03.901 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd device --device-type nbd list 2024-09-18T04:39:03.901 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false grep '^18581\b' 2024-09-18T04:39:03.901 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep '^18581\b' 2024-09-18T04:39:03.962 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:03.957+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:03.969 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:39:03.970 INFO:tasks.workunit.client.0.smithi115.stderr:+ ps -C rbd-nbd 2024-09-18T04:39:03.971 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false grep '^ *18581\b' 2024-09-18T04:39:03.971 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep '^ *18581\b' 2024-09-18T04:39:03.981 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:39:03.981 INFO:tasks.workunit.client.0.smithi115.stderr:+ sleep 0.5 2024-09-18T04:39:04.022 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:04.022 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:04.029 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T04:39:04.032 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:39:04.062 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:04.057+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:04.138 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:04.139 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:04.148 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:39:04.164 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:04.157+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:39:04.168 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:39:04.188 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:39:04.209 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:39:04.222 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:39:04.226 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-18T04:39:04.229 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:39:04.237 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":109,"stamp":"2024-09-18T04:39:03.617035+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":82379442,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":2308,"num_read_kb":147482,"num_write":545,"num_write_kb":213506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":12582912,"num_keys_recovered":3,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":474,"ondisk_log_size":474,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":351988,"kb_used_data":157080,"kb_used_omap":9,"kb_used_meta":194806,"kb_avail":565879052,"statfs":{"total":579820584960,"available":579460149248,"internally_reserved":0,"allocated":160849920,"data_stored":159153236,"data_compressed":66040,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":9561,"internal_metadata":199482023},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-52428800,"num_objects":-12,"num_object_clones":0,"num_object_copies":-24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-12,"num_whiteouts":0,"num_read":686,"num_read_kb":1641,"num_write":124,"num_write_kb":14404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":12582912,"num_keys_recovered":3,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.163479"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":206,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T04:38:58.456842+0000","last_change":"2024-09-18T04:37:27.423069+0000","last_active":"2024-09-18T04:38:58.456842+0000","last_peered":"2024-09-18T04:38:58.456842+0000","last_clean":"2024-09-18T04:38:58.456842+0000","last_became_active":"2024-09-18T04:37:27.422350+0000","last_became_peered":"2024-09-18T04:37:27.422350+0000","last_unstale":"2024-09-18T04:38:58.456842+0000","last_undegraded":"2024-09-18T04:38:58.456842+0000","last_fullsized":"2024-09-18T04:38:58.456842+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:26.406292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:26.406292+0000","last_clean_scrub_stamp":"2024-09-18T04:37:26.406292+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:31:45.852149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"64'57","reported_seq":268,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-18T04:39:01.020378+0000","last_change":"2024-09-18T04:37:51.681357+0000","last_active":"2024-09-18T04:39:01.020378+0000","last_peered":"2024-09-18T04:39:01.020378+0000","last_clean":"2024-09-18T04:39:01.020378+0000","last_became_active":"2024-09-18T04:37:51.676131+0000","last_became_peered":"2024-09-18T04:37:51.676131+0000","last_unstale":"2024-09-18T04:39:01.020378+0000","last_undegraded":"2024-09-18T04:39:01.020378+0000","last_fullsized":"2024-09-18T04:39:01.020378+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:45:22.605009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018293420000000001,"stat_sum":{"num_bytes":8388662,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":255,"num_read_kb":16777,"num_write":72,"num_write_kb":24610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"55'2","reported_seq":26,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T04:38:58.453620+0000","last_change":"2024-09-18T04:38:46.966799+0000","last_active":"2024-09-18T04:38:58.453620+0000","last_peered":"2024-09-18T04:38:58.453620+0000","last_clean":"2024-09-18T04:38:58.453620+0000","last_became_active":"2024-09-18T04:38:44.953092+0000","last_became_peered":"2024-09-18T04:38:44.953092+0000","last_unstale":"2024-09-18T04:38:58.453620+0000","last_undegraded":"2024-09-18T04:38:58.453620+0000","last_fullsized":"2024-09-18T04:38:58.453620+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_clean_scrub_stamp":"2024-09-18T04:38:43.928423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:04:46.888906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010424099999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"61'94","reported_seq":462,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T04:38:58.456915+0000","last_change":"2024-09-18T04:37:36.484950+0000","last_active":"2024-09-18T04:38:58.456915+0000","last_peered":"2024-09-18T04:38:58.456915+0000","last_clean":"2024-09-18T04:38:58.456915+0000","last_became_active":"2024-09-18T04:37:34.470841+0000","last_became_peered":"2024-09-18T04:37:34.470841+0000","last_unstale":"2024-09-18T04:38:58.456915+0000","last_undegraded":"2024-09-18T04:38:58.456915+0000","last_fullsized":"2024-09-18T04:38:58.456915+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:16:34.528783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090513900000000001,"stat_sum":{"num_bytes":8388646,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":628,"num_read_kb":28086,"num_write":141,"num_write_kb":32867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T04:38:59.456727+0000","last_change":"2024-09-18T04:38:47.337598+0000","last_active":"2024-09-18T04:38:59.456727+0000","last_peered":"2024-09-18T04:38:59.456727+0000","last_clean":"2024-09-18T04:38:59.456727+0000","last_became_active":"2024-09-18T04:38:44.952820+0000","last_became_peered":"2024-09-18T04:38:44.952820+0000","last_unstale":"2024-09-18T04:38:59.456727+0000","last_undegraded":"2024-09-18T04:38:59.456727+0000","last_fullsized":"2024-09-18T04:38:59.456727+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_clean_scrub_stamp":"2024-09-18T04:38:43.928423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:10:04.992053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075989799999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"63'67","reported_seq":636,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T04:38:59.455549+0000","last_change":"2024-09-18T04:37:51.679120+0000","last_active":"2024-09-18T04:38:59.455549+0000","last_peered":"2024-09-18T04:38:59.455549+0000","last_clean":"2024-09-18T04:38:59.455549+0000","last_became_active":"2024-09-18T04:37:51.678711+0000","last_became_peered":"2024-09-18T04:37:51.678711+0000","last_unstale":"2024-09-18T04:38:59.455549+0000","last_undegraded":"2024-09-18T04:38:59.455549+0000","last_fullsized":"2024-09-18T04:38:59.455549+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T04:46:09.459905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":882,"num_read_kb":12074,"num_write":37,"num_write_kb":16387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T04:38:58.453598+0000","last_change":"2024-09-18T04:38:46.967459+0000","last_active":"2024-09-18T04:38:58.453598+0000","last_peered":"2024-09-18T04:38:58.453598+0000","last_clean":"2024-09-18T04:38:58.453598+0000","last_became_active":"2024-09-18T04:38:44.950578+0000","last_became_peered":"2024-09-18T04:38:44.950578+0000","last_unstale":"2024-09-18T04:38:58.453598+0000","last_undegraded":"2024-09-18T04:38:58.453598+0000","last_fullsized":"2024-09-18T04:38:58.453598+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_clean_scrub_stamp":"2024-09-18T04:38:43.928423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:45:13.410610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011574549999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"20'2","reported_seq":100,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T04:39:00.348519+0000","last_change":"2024-09-18T04:37:36.490461+0000","last_active":"2024-09-18T04:39:00.348519+0000","last_peered":"2024-09-18T04:39:00.348519+0000","last_clean":"2024-09-18T04:39:00.348519+0000","last_became_active":"2024-09-18T04:37:34.473088+0000","last_became_peered":"2024-09-18T04:37:34.473088+0000","last_unstale":"2024-09-18T04:39:00.348519+0000","last_undegraded":"2024-09-18T04:39:00.348519+0000","last_fullsized":"2024-09-18T04:39:00.348519+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:17:57.352397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001879932,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":40,"num_read_kb":40,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"53'1","reported_seq":25,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T04:38:59.434699+0000","last_change":"2024-09-18T04:38:47.545918+0000","last_active":"2024-09-18T04:38:59.434699+0000","last_peered":"2024-09-18T04:38:59.434699+0000","last_clean":"2024-09-18T04:38:59.434699+0000","last_became_active":"2024-09-18T04:38:44.950045+0000","last_became_peered":"2024-09-18T04:38:44.950045+0000","last_unstale":"2024-09-18T04:38:59.434699+0000","last_undegraded":"2024-09-18T04:38:59.434699+0000","last_fullsized":"2024-09-18T04:38:59.434699+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_clean_scrub_stamp":"2024-09-18T04:38:43.928423+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:25:54.288210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010091900000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T04:38:59.459498+0000","last_change":"2024-09-18T04:38:46.974127+0000","last_active":"2024-09-18T04:38:59.459498+0000","last_peered":"2024-09-18T04:38:59.459498+0000","last_clean":"2024-09-18T04:38:59.459498+0000","last_became_active":"2024-09-18T04:38:44.949428+0000","last_became_peered":"2024-09-18T04:38:44.949428+0000","last_unstale":"2024-09-18T04:38:59.459498+0000","last_undegraded":"2024-09-18T04:38:59.459498+0000","last_fullsized":"2024-09-18T04:38:59.459498+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_clean_scrub_stamp":"2024-09-18T04:38:43.928423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:24:58.364051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065509499999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"64'37","reported_seq":38,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-18T04:39:01.096405+0000","last_change":"2024-09-18T04:38:59.528402+0000","last_active":"2024-09-18T04:39:01.096405+0000","last_peered":"2024-09-18T04:39:01.096405+0000","last_clean":"2024-09-18T04:39:01.096405+0000","last_became_active":"2024-09-18T04:38:59.462279+0000","last_became_peered":"2024-09-18T04:38:59.462279+0000","last_unstale":"2024-09-18T04:39:01.096405+0000","last_undegraded":"2024-09-18T04:39:01.096405+0000","last_fullsized":"2024-09-18T04:39:01.096405+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:32:58.894411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081514499999999998,"stat_sum":{"num_bytes":18874368,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":83,"num_read_kb":16649,"num_write":37,"num_write_kb":34818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":12582912,"num_keys_recovered":3,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T04:38:58.453913+0000","last_change":"2024-09-18T04:38:46.967504+0000","last_active":"2024-09-18T04:38:58.453913+0000","last_peered":"2024-09-18T04:38:58.453913+0000","last_clean":"2024-09-18T04:38:58.453913+0000","last_became_active":"2024-09-18T04:38:44.950268+0000","last_became_peered":"2024-09-18T04:38:44.950268+0000","last_unstale":"2024-09-18T04:38:58.453913+0000","last_undegraded":"2024-09-18T04:38:58.453913+0000","last_fullsized":"2024-09-18T04:38:58.453913+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_clean_scrub_stamp":"2024-09-18T04:38:43.928423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:07:16.193764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016757619999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"60'33","reported_seq":177,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T04:38:58.454515+0000","last_change":"2024-09-18T04:37:59.284861+0000","last_active":"2024-09-18T04:38:58.454515+0000","last_peered":"2024-09-18T04:38:58.454515+0000","last_clean":"2024-09-18T04:38:58.454515+0000","last_became_active":"2024-09-18T04:37:59.283691+0000","last_became_peered":"2024-09-18T04:37:59.283691+0000","last_unstale":"2024-09-18T04:38:58.454515+0000","last_undegraded":"2024-09-18T04:38:58.454515+0000","last_fullsized":"2024-09-18T04:38:58.454515+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:19:36.536686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":100,"num_read_kb":28672,"num_write":33,"num_write_kb":36864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":24,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T04:38:58.454383+0000","last_change":"2024-09-18T04:38:46.969605+0000","last_active":"2024-09-18T04:38:58.454383+0000","last_peered":"2024-09-18T04:38:58.454383+0000","last_clean":"2024-09-18T04:38:58.454383+0000","last_became_active":"2024-09-18T04:38:44.948788+0000","last_became_peered":"2024-09-18T04:38:44.948788+0000","last_unstale":"2024-09-18T04:38:58.454383+0000","last_undegraded":"2024-09-18T04:38:58.454383+0000","last_fullsized":"2024-09-18T04:38:58.454383+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_clean_scrub_stamp":"2024-09-18T04:38:43.928423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:18:24.302150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014497589999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"60'37","reported_seq":197,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T04:38:58.453967+0000","last_change":"2024-09-18T04:37:36.494193+0000","last_active":"2024-09-18T04:38:58.453967+0000","last_peered":"2024-09-18T04:38:58.453967+0000","last_clean":"2024-09-18T04:38:58.453967+0000","last_became_active":"2024-09-18T04:37:34.472940+0000","last_became_peered":"2024-09-18T04:37:34.472940+0000","last_unstale":"2024-09-18T04:38:58.453967+0000","last_undegraded":"2024-09-18T04:38:58.453967+0000","last_fullsized":"2024-09-18T04:38:58.453967+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:18:29.266815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070553900000000004,"stat_sum":{"num_bytes":20971520,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":105,"num_read_kb":28676,"num_write":37,"num_write_kb":45058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":24,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-18T04:38:59.456855+0000","last_change":"2024-09-18T04:38:47.337380+0000","last_active":"2024-09-18T04:38:59.456855+0000","last_peered":"2024-09-18T04:38:59.456855+0000","last_clean":"2024-09-18T04:38:59.456855+0000","last_became_active":"2024-09-18T04:38:44.948732+0000","last_became_peered":"2024-09-18T04:38:44.948732+0000","last_unstale":"2024-09-18T04:38:59.456855+0000","last_undegraded":"2024-09-18T04:38:59.456855+0000","last_fullsized":"2024-09-18T04:38:59.456855+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_clean_scrub_stamp":"2024-09-18T04:38:43.928423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:07:02.430621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058611500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"60'27","reported_seq":174,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-18T04:38:58.456941+0000","last_change":"2024-09-18T04:38:05.754820+0000","last_active":"2024-09-18T04:38:58.456941+0000","last_peered":"2024-09-18T04:38:58.456941+0000","last_clean":"2024-09-18T04:38:58.456941+0000","last_became_active":"2024-09-18T04:38:05.753962+0000","last_became_peered":"2024-09-18T04:38:05.753962+0000","last_unstale":"2024-09-18T04:38:58.456941+0000","last_undegraded":"2024-09-18T04:38:58.456941+0000","last_fullsized":"2024-09-18T04:38:58.456941+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:53:52.224098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047343899999999998,"stat_sum":{"num_bytes":4194320,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":99,"num_read_kb":16408,"num_write":25,"num_write_kb":20482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":5},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":81789055,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":2192,"num_read_kb":147382,"num_write":384,"num_write_kb":211088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":12582912,"num_keys_recovered":3,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":171089920,"data_stored":171049214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":354,"ondisk_log_size":354,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":66040,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":13,"seq":55834574871,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57572,"kb_used_data":26460,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94314268,"statfs":{"total":96636764160,"available":96577810432,"internally_reserved":0,"allocated":27095040,"data_stored":26617857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:38:26 2024","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.873,"15min":0.873},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.5249999999999999,"5min":1.5249999999999999,"15min":1.5249999999999999},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.369,"5min":1.369,"15min":1.369},"last":0.59099999999999997}]},{"osd":1,"last update":"Wed Sep 18 04:38:26 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.68100000000000005}]},{"osd":2,"last update":"Wed Sep 18 04:38:26 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.4930000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.83799999999999997}]},{"osd":3,"last update":"Wed Sep 18 04:38:26 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.22,"5min":1.22,"15min":1.22},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.109,"5min":1.109,"15min":1.109},"last":0.60999999999999999}]},{"osd":4,"last update":"Wed Sep 18 04:38:26 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.121,"5min":1.121,"15min":1.121},"last":0.88500000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574871,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27720,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344120,"statfs":{"total":96636764160,"available":96608378880,"internally_reserved":0,"allocated":716800,"data_stored":267220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.175,"5min":1.175,"15min":1.175},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.258,"5min":1.258,"15min":1.258},"last":0.72999999999999998}]},{"osd":1,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.67100000000000004}]},{"osd":2,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.619,"5min":1.619,"15min":1.619},"last":0.62}]},{"osd":3,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.84899999999999998}]},{"osd":5,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":1.0329999999999999},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.214,"5min":1.214,"15min":1.214},"last":0.80400000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574872,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60560,"kb_used_data":25340,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94311280,"statfs":{"total":96636764160,"available":96574750720,"internally_reserved":0,"allocated":25948160,"data_stored":26023412,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 04:38:28 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.113,"5min":1.113,"15min":1.113},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.51900000000000002}]},{"osd":2,"last update":"Wed Sep 18 04:38:28 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.8180000000000001,"5min":1.8180000000000001,"15min":1.8180000000000001},"last":0.48799999999999999}]},{"osd":3,"last update":"Wed Sep 18 04:38:28 2024","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":1.169,"5min":1.169,"15min":1.169},"last":0.872},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.72699999999999998}]},{"osd":4,"last update":"Wed Sep 18 04:38:28 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.82399999999999995}]},{"osd":5,"last update":"Wed Sep 18 04:38:28 2024","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.8919999999999999,"5min":1.8919999999999999,"15min":1.8919999999999999},"last":0.751},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":1.224,"5min":1.224,"15min":1.224},"last":0.90500000000000003}]}]},{"osd":3,"up_from":12,"seq":51539607575,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64664,"kb_used_data":33540,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94307176,"statfs":{"total":96636764160,"available":96570548224,"internally_reserved":0,"allocated":34344960,"data_stored":34412074,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":2.665,"5min":2.665,"15min":2.665},"last":1.119},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.99099999999999999}]},{"osd":1,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":2.4510000000000001,"5min":2.4510000000000001,"15min":2.4510000000000001},"last":1.0600000000000001},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.71699999999999997}]},{"osd":2,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":2.577,"5min":2.577,"15min":2.577},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.76000000000000001}]},{"osd":4,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.94699999999999995}]},{"osd":5,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.626},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":1.0169999999999999}]}]},{"osd":2,"up_from":12,"seq":51539607573,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56404,"kb_used_data":21188,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94315436,"statfs":{"total":96636764160,"available":96579006464,"internally_reserved":0,"allocated":21696512,"data_stored":21238794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.55500000000000005}]},{"osd":1,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.623},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.60099999999999998}]},{"osd":3,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.748}]},{"osd":4,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.879,"5min":0.879,"15min":0.879},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.71499999999999997}]},{"osd":5,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.92400000000000004}]}]},{"osd":1,"up_from":12,"seq":51539607575,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85068,"kb_used_data":49852,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94286772,"statfs":{"total":96636764160,"available":96549654528,"internally_reserved":0,"allocated":51048448,"data_stored":50593879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.54700000000000004}]},{"osd":2,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.327,"5min":1.327,"15min":1.327},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.70399999999999996}]},{"osd":3,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.121,"5min":1.121,"15min":1.121},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.873,"5min":0.873,"15min":0.873},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.92400000000000004}]},{"osd":4,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.117,"5min":1.117,"15min":1.117},"last":0.99199999999999999}]},{"osd":5,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":1.3859999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.89800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":38928384,"data_stored":38928384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":37756928,"data_stored":37748793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20979712,"data_stored":20971574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":33562624,"data_stored":33554486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":39862272,"data_stored":39845977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T04:39:04.239 INFO:tasks.thrashosds.thrasher:Setting 3.6 to [5, 0, 1, 4] 2024-09-18T04:39:04.239 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '3.6', '5', '0', '1', '4'] 2024-09-18T04:39:04.240 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 3.6 5 0 1 4 2024-09-18T04:39:04.250 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:39:04.262 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:04.262 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:04.264 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:04.257+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:04.271 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:39:04.272 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:39:04.272 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:39:04.272 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:39:04.273 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:39:04.273 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:39:04.273 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:39:04.273 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:39:04.366 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:04.361+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:39:04.382 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:04.382 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:39:04.382 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:39:04.383 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:39:04.383 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.383 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-18T04:39:04.383 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:39:04.383 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 89.907286381000006, 2024-09-18T04:39:04.383 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0095781189999999995, 2024-09-18T04:39:04.383 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:04.384 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:04.384 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:04.384 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:39:04.384 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:39:04.384 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 1 2024-09-18T04:39:04.384 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.384 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:04.384 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.384 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:04.384 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:39:04.385 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:04.385 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.385 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.385 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:04.385 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:39:04.385 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:04.385 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.385 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.385 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:04.385 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469862+0000", 2024-09-18T04:39:04.385 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:04.385 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.386 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.386 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:04.386 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469879+0000", 2024-09-18T04:39:04.386 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7431999999999998e-05 2024-09-18T04:39:04.386 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.386 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.386 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:04.386 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469896+0000", 2024-09-18T04:39:04.386 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6742999999999999e-05 2024-09-18T04:39:04.386 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.386 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.386 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:04.387 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469915+0000", 2024-09-18T04:39:04.387 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8991000000000002e-05 2024-09-18T04:39:04.387 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.387 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.387 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:04.387 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470397+0000", 2024-09-18T04:39:04.387 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048167799999999998 2024-09-18T04:39:04.387 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.387 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.387 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-09-18T04:39:04.387 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470412+0000", 2024-09-18T04:39:04.387 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5526e-05 2024-09-18T04:39:04.387 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.388 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.388 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:04.388 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473279+0000", 2024-09-18T04:39:04.388 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0028667469999999998 2024-09-18T04:39:04.388 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.388 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.388 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:04.388 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473441+0000", 2024-09-18T04:39:04.388 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000162082 2024-09-18T04:39:04.388 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.388 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.388 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:04.388 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473573+0000", 2024-09-18T04:39:04.389 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000132032 2024-09-18T04:39:04.389 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.389 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.389 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:04.389 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474826+0000", 2024-09-18T04:39:04.389 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001252717 2024-09-18T04:39:04.389 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.389 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.389 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:04.389 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479307+0000", 2024-09-18T04:39:04.389 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044813220000000003 2024-09-18T04:39:04.389 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.390 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.390 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:04.390 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479366+0000", 2024-09-18T04:39:04.390 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9024999999999998e-05 2024-09-18T04:39:04.390 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.390 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.390 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:04.390 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479443+0000", 2024-09-18T04:39:04.390 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6339000000000002e-05 2024-09-18T04:39:04.390 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:04.390 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:04.391 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:04.391 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.391 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.391 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-18T04:39:04.391 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:39:04.391 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 87.894277961, 2024-09-18T04:39:04.391 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082699560000000002, 2024-09-18T04:39:04.391 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:04.391 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:04.391 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:04.391 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:39:04.391 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:39:04.391 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 7 2024-09-18T04:39:04.391 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.392 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:04.392 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.392 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:04.392 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:39:04.392 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:04.392 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.392 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.392 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:04.392 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:39:04.392 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:04.392 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.392 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.392 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:04.393 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482870+0000", 2024-09-18T04:39:04.393 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999967 2024-09-18T04:39:04.393 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.393 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.393 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:04.393 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482889+0000", 2024-09-18T04:39:04.393 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.965e-05 2024-09-18T04:39:04.393 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.393 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.393 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:04.393 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482938+0000", 2024-09-18T04:39:04.393 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8548999999999999e-05 2024-09-18T04:39:04.393 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.394 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.394 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:04.394 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482963+0000", 2024-09-18T04:39:04.394 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5075999999999998e-05 2024-09-18T04:39:04.394 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.394 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.394 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:04.394 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483121+0000", 2024-09-18T04:39:04.394 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015819400000000001 2024-09-18T04:39:04.394 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.394 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.394 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op must wait for map", 2024-09-18T04:39:04.394 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483138+0000", 2024-09-18T04:39:04.395 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7068999999999999e-05 2024-09-18T04:39:04.395 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.395 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.395 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:04.395 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488666+0000", 2024-09-18T04:39:04.395 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0055282049999999996 2024-09-18T04:39:04.395 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.395 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.395 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:04.395 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488783+0000", 2024-09-18T04:39:04.395 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000116737 2024-09-18T04:39:04.395 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.395 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.395 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:04.396 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488991+0000", 2024-09-18T04:39:04.396 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020781400000000001 2024-09-18T04:39:04.396 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.396 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.396 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:04.396 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490554+0000", 2024-09-18T04:39:04.396 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0015628370000000001 2024-09-18T04:39:04.396 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.396 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.396 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:04.396 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491019+0000", 2024-09-18T04:39:04.396 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000465608 2024-09-18T04:39:04.396 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.397 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.397 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:04.397 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491087+0000", 2024-09-18T04:39:04.397 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7988e-05 2024-09-18T04:39:04.397 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.397 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.397 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:04.397 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491143+0000", 2024-09-18T04:39:04.397 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5538999999999998e-05 2024-09-18T04:39:04.397 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:04.397 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:04.398 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:04.398 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.398 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.398 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:34 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:04.398 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.261758+0000", 2024-09-18T04:39:04.398 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 22.115392809999999, 2024-09-18T04:39:04.398 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.042024191000000002, 2024-09-18T04:39:04.398 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:04.398 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:04.398 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:04.399 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:04.399 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:04.399 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 34 2024-09-18T04:39:04.399 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.399 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:04.399 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.399 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:04.399 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.261758+0000", 2024-09-18T04:39:04.399 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:04.399 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.399 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.400 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:04.400 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.261758+0000", 2024-09-18T04:39:04.400 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:04.400 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.400 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.400 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:04.400 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.261756+0000", 2024-09-18T04:39:04.400 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:04.400 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.400 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.400 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:04.401 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.270785+0000", 2024-09-18T04:39:04.401 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0090292340000000006 2024-09-18T04:39:04.401 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.401 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.401 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:04.401 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.270890+0000", 2024-09-18T04:39:04.401 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000104541 2024-09-18T04:39:04.401 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.401 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.401 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:04.402 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.270991+0000", 2024-09-18T04:39:04.402 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000100778 2024-09-18T04:39:04.402 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.402 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.402 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:04.402 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.271157+0000", 2024-09-18T04:39:04.402 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016653999999999999 2024-09-18T04:39:04.402 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.402 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.403 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:04.403 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.271591+0000", 2024-09-18T04:39:04.403 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00043409500000000001 2024-09-18T04:39:04.403 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.403 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.403 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:04.404 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272017+0000", 2024-09-18T04:39:04.404 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000425769 2024-09-18T04:39:04.404 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.404 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.404 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:04.404 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.294246+0000", 2024-09-18T04:39:04.404 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.022229237999999998 2024-09-18T04:39:04.404 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.405 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.405 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:04.405 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303571+0000", 2024-09-18T04:39:04.405 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0093250239999999995 2024-09-18T04:39:04.405 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.406 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.406 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:04.406 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303705+0000", 2024-09-18T04:39:04.406 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000133831 2024-09-18T04:39:04.406 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.407 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.407 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:04.407 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303782+0000", 2024-09-18T04:39:04.407 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7156000000000005e-05 2024-09-18T04:39:04.407 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:04.407 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:04.407 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:04.408 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.408 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.408 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:36 2.7 e51/32 2:fe1b02ad:::rbd_data.125fc09df6d1.0000000000000002:head v 51'1, mlcod=0'0)", 2024-09-18T04:39:04.408 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.273441+0000", 2024-09-18T04:39:04.408 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 22.103709829, 2024-09-18T04:39:04.408 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.032400768000000003, 2024-09-18T04:39:04.408 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:04.409 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:04.409 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:04.409 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.409 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:04.409 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273441+0000", 2024-09-18T04:39:04.409 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:04.410 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.410 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.410 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:04.410 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273441+0000", 2024-09-18T04:39:04.410 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:04.410 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.410 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.410 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:04.411 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273440+0000", 2024-09-18T04:39:04.411 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:04.411 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.411 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.411 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:04.411 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278723+0000", 2024-09-18T04:39:04.411 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0052829909999999999 2024-09-18T04:39:04.411 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.411 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.411 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:04.412 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278752+0000", 2024-09-18T04:39:04.412 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9178000000000001e-05 2024-09-18T04:39:04.412 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.412 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.412 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:04.412 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278761+0000", 2024-09-18T04:39:04.412 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7900000000000005e-06 2024-09-18T04:39:04.412 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.413 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.413 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:04.413 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278864+0000", 2024-09-18T04:39:04.413 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000102699 2024-09-18T04:39:04.413 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.413 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.413 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:04.413 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278900+0000", 2024-09-18T04:39:04.413 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6001999999999999e-05 2024-09-18T04:39:04.414 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.414 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.414 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:04.414 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.305793+0000", 2024-09-18T04:39:04.414 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.026893344 2024-09-18T04:39:04.414 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.414 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.414 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:04.414 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.305842+0000", 2024-09-18T04:39:04.414 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8720000000000001e-05 2024-09-18T04:39:04.415 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:04.415 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:04.415 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:04.415 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.415 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.415 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:39 2.7 e51/32 2:e9c909c5:::rbd_data.125fc09df6d1.0000000000000005:head v 51'2, mlcod=0'0)", 2024-09-18T04:39:04.415 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.283175+0000", 2024-09-18T04:39:04.415 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 22.093976139999999, 2024-09-18T04:39:04.415 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.022864102000000001, 2024-09-18T04:39:04.415 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:04.416 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:04.416 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:04.416 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.416 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:04.416 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283175+0000", 2024-09-18T04:39:04.416 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:04.416 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.416 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.416 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:04.417 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283175+0000", 2024-09-18T04:39:04.417 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:04.417 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.417 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.417 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:04.417 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283173+0000", 2024-09-18T04:39:04.417 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:04.417 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.417 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.417 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:04.417 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.285951+0000", 2024-09-18T04:39:04.418 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0027784540000000001 2024-09-18T04:39:04.418 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.418 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.418 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:04.418 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.285970+0000", 2024-09-18T04:39:04.418 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8855000000000002e-05 2024-09-18T04:39:04.419 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.419 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.419 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:04.419 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.285984+0000", 2024-09-18T04:39:04.419 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4252e-05 2024-09-18T04:39:04.419 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.419 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.419 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:04.419 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.286041+0000", 2024-09-18T04:39:04.419 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6431999999999998e-05 2024-09-18T04:39:04.419 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.420 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.420 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:04.420 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.286086+0000", 2024-09-18T04:39:04.420 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5423000000000002e-05 2024-09-18T04:39:04.420 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.420 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.420 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:04.420 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.305998+0000", 2024-09-18T04:39:04.420 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019911697999999999 2024-09-18T04:39:04.420 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.420 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.421 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:04.421 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306039+0000", 2024-09-18T04:39:04.421 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0973999999999998e-05 2024-09-18T04:39:04.421 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:04.421 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:04.421 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:04.421 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.421 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.421 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:50 2.3 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:04.421 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.312563+0000", 2024-09-18T04:39:04.422 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 22.064588112999999, 2024-09-18T04:39:04.422 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.032636001999999997, 2024-09-18T04:39:04.422 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:04.422 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:04.422 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:04.422 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:04.422 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:04.422 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 50 2024-09-18T04:39:04.422 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.422 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:04.422 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.422 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:04.422 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.312563+0000", 2024-09-18T04:39:04.422 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:04.423 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.423 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.423 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:04.423 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.312563+0000", 2024-09-18T04:39:04.423 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:04.423 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.423 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.423 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:04.423 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.312561+0000", 2024-09-18T04:39:04.423 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:04.423 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.423 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.424 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:04.424 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321709+0000", 2024-09-18T04:39:04.424 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0091486650000000003 2024-09-18T04:39:04.424 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.424 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.424 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:04.424 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321739+0000", 2024-09-18T04:39:04.424 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9666e-05 2024-09-18T04:39:04.424 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.424 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.424 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:04.425 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321751+0000", 2024-09-18T04:39:04.425 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2086000000000001e-05 2024-09-18T04:39:04.425 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.425 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.425 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:04.425 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321833+0000", 2024-09-18T04:39:04.425 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1842999999999996e-05 2024-09-18T04:39:04.425 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.425 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.425 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:04.425 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321981+0000", 2024-09-18T04:39:04.425 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000147441 2024-09-18T04:39:04.425 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.425 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.426 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:04.426 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.322109+0000", 2024-09-18T04:39:04.426 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012851200000000001 2024-09-18T04:39:04.426 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.426 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.426 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:04.426 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.332829+0000", 2024-09-18T04:39:04.426 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010720225999999999 2024-09-18T04:39:04.426 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.426 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.426 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:04.426 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.345039+0000", 2024-09-18T04:39:04.427 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012209831000000001 2024-09-18T04:39:04.427 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.427 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.427 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:04.427 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.345120+0000", 2024-09-18T04:39:04.427 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1145999999999999e-05 2024-09-18T04:39:04.427 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.427 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.427 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:04.428 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.345199+0000", 2024-09-18T04:39:04.428 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8546000000000003e-05 2024-09-18T04:39:04.428 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:04.428 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:04.428 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:04.428 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.428 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.428 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:30 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:04.428 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.516122+0000", 2024-09-18T04:39:04.428 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 21.861028951000002, 2024-09-18T04:39:04.428 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0091865030000000004, 2024-09-18T04:39:04.429 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:04.429 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:04.429 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:04.429 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:04.429 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:04.429 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 30 2024-09-18T04:39:04.429 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.429 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:04.429 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.430 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:04.430 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516122+0000", 2024-09-18T04:39:04.430 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:04.430 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.430 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.430 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:04.430 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516122+0000", 2024-09-18T04:39:04.430 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:04.430 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.430 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.431 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:04.431 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516120+0000", 2024-09-18T04:39:04.431 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:04.431 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.431 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.431 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:04.431 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516129+0000", 2024-09-18T04:39:04.431 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.85e-06 2024-09-18T04:39:04.431 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.432 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.432 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:04.432 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516155+0000", 2024-09-18T04:39:04.432 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5882000000000001e-05 2024-09-18T04:39:04.432 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.432 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.432 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:04.432 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516175+0000", 2024-09-18T04:39:04.432 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.022e-05 2024-09-18T04:39:04.432 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.433 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.433 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:04.433 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519780+0000", 2024-09-18T04:39:04.433 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003604187 2024-09-18T04:39:04.433 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.433 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.433 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:04.433 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519964+0000", 2024-09-18T04:39:04.433 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00018391999999999999 2024-09-18T04:39:04.434 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.434 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.434 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:04.434 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.520794+0000", 2024-09-18T04:39:04.434 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00083078100000000005 2024-09-18T04:39:04.434 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.434 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.435 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:04.435 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.524129+0000", 2024-09-18T04:39:04.435 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033342419999999998 2024-09-18T04:39:04.435 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.435 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.435 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:04.435 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525239+0000", 2024-09-18T04:39:04.435 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001110778 2024-09-18T04:39:04.435 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.436 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.436 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:04.436 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525277+0000", 2024-09-18T04:39:04.436 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7596999999999999e-05 2024-09-18T04:39:04.436 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.436 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.436 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:04.437 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525308+0000", 2024-09-18T04:39:04.437 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.1535999999999998e-05 2024-09-18T04:39:04.437 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:04.437 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:04.437 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:04.437 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.437 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.437 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:32 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:04.438 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.519217+0000", 2024-09-18T04:39:04.438 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 21.857933631000002, 2024-09-18T04:39:04.438 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082266370000000002, 2024-09-18T04:39:04.438 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:04.438 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:04.438 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:04.438 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:04.438 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:04.439 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 32 2024-09-18T04:39:04.439 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.439 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:04.439 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.439 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:04.439 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519217+0000", 2024-09-18T04:39:04.439 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:04.439 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.439 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.440 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:04.440 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519217+0000", 2024-09-18T04:39:04.440 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:04.440 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.440 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.440 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:04.440 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519215+0000", 2024-09-18T04:39:04.440 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:04.441 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.441 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.441 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:04.441 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519229+0000", 2024-09-18T04:39:04.441 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3329999999999999e-05 2024-09-18T04:39:04.441 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.442 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.442 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:04.442 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519268+0000", 2024-09-18T04:39:04.442 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9533000000000002e-05 2024-09-18T04:39:04.442 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.443 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.443 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:04.443 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519285+0000", 2024-09-18T04:39:04.443 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6446000000000001e-05 2024-09-18T04:39:04.443 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.443 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.443 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:04.444 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.522358+0000", 2024-09-18T04:39:04.444 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030734310000000002 2024-09-18T04:39:04.444 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.444 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.444 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:04.445 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.522551+0000", 2024-09-18T04:39:04.445 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019317600000000001 2024-09-18T04:39:04.445 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.445 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.445 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:04.446 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.523262+0000", 2024-09-18T04:39:04.446 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00071069999999999998 2024-09-18T04:39:04.446 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.446 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.446 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:04.447 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525318+0000", 2024-09-18T04:39:04.447 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0020561469999999999 2024-09-18T04:39:04.447 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.447 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.447 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:04.447 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.527376+0000", 2024-09-18T04:39:04.447 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0020584969999999998 2024-09-18T04:39:04.447 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.448 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.448 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:04.448 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.527416+0000", 2024-09-18T04:39:04.448 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9054000000000003e-05 2024-09-18T04:39:04.448 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.448 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.448 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:04.448 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.527444+0000", 2024-09-18T04:39:04.448 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8347000000000002e-05 2024-09-18T04:39:04.448 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:04.448 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:04.448 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:04.449 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.449 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.449 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:42 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:04.449 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.551160+0000", 2024-09-18T04:39:04.449 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 21.825991239, 2024-09-18T04:39:04.449 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010692757000000001, 2024-09-18T04:39:04.449 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:04.449 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:04.449 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:04.449 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:04.449 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:04.449 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 42 2024-09-18T04:39:04.450 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.450 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:04.450 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.450 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:04.450 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551160+0000", 2024-09-18T04:39:04.450 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:04.450 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.450 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.450 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:04.450 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551160+0000", 2024-09-18T04:39:04.450 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:04.450 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.451 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.451 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:04.451 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551159+0000", 2024-09-18T04:39:04.451 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:04.451 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.451 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.451 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:04.451 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551166+0000", 2024-09-18T04:39:04.451 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1029999999999996e-06 2024-09-18T04:39:04.452 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.452 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.452 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:04.452 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551177+0000", 2024-09-18T04:39:04.452 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1384e-05 2024-09-18T04:39:04.452 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.452 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.452 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:04.452 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551185+0000", 2024-09-18T04:39:04.452 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1990000000000006e-06 2024-09-18T04:39:04.452 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.452 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.453 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:04.453 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552021+0000", 2024-09-18T04:39:04.453 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00083574400000000003 2024-09-18T04:39:04.453 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.453 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.453 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:04.453 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552339+0000", 2024-09-18T04:39:04.453 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031771399999999997 2024-09-18T04:39:04.453 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.453 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.453 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:04.453 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.553140+0000", 2024-09-18T04:39:04.454 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00080088100000000003 2024-09-18T04:39:04.454 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.454 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.454 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:04.454 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.554585+0000", 2024-09-18T04:39:04.454 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001445801 2024-09-18T04:39:04.454 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.454 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.454 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:04.454 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.561711+0000", 2024-09-18T04:39:04.454 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071260109999999998 2024-09-18T04:39:04.454 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.454 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.455 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:04.455 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.561790+0000", 2024-09-18T04:39:04.455 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8967000000000005e-05 2024-09-18T04:39:04.455 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.455 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.455 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:04.455 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.561852+0000", 2024-09-18T04:39:04.455 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2045999999999995e-05 2024-09-18T04:39:04.455 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:04.455 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:04.456 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:04.456 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.456 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.456 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:58 2.7 e51/32 2:fc7c6cd3:ns::rbd_data.133e813ab5bc.000000000000000e:head v 51'7, mlcod=51'5)", 2024-09-18T04:39:04.456 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.605333+0000", 2024-09-18T04:39:04.456 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 21.771818033999999, 2024-09-18T04:39:04.456 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020085902999999999, 2024-09-18T04:39:04.456 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:04.456 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:04.456 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:04.457 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.457 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:04.457 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605333+0000", 2024-09-18T04:39:04.457 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:04.457 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.457 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.457 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:04.457 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605333+0000", 2024-09-18T04:39:04.457 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:04.457 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.457 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.458 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:04.458 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605332+0000", 2024-09-18T04:39:04.458 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:04.458 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.458 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.458 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:04.458 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608667+0000", 2024-09-18T04:39:04.458 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033350300000000001 2024-09-18T04:39:04.458 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.458 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.459 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:04.459 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608706+0000", 2024-09-18T04:39:04.459 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8951e-05 2024-09-18T04:39:04.459 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.459 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.459 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:04.459 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608725+0000", 2024-09-18T04:39:04.459 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9675999999999999e-05 2024-09-18T04:39:04.459 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.459 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.459 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:04.459 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608798+0000", 2024-09-18T04:39:04.459 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3135999999999999e-05 2024-09-18T04:39:04.460 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.460 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.460 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:04.460 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608824+0000", 2024-09-18T04:39:04.460 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5284000000000001e-05 2024-09-18T04:39:04.460 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.460 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.460 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:04.460 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625370+0000", 2024-09-18T04:39:04.460 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016545975000000001 2024-09-18T04:39:04.460 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.460 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.460 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:04.461 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625419+0000", 2024-09-18T04:39:04.461 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9076999999999998e-05 2024-09-18T04:39:04.461 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:04.461 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:04.461 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:04.461 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.461 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.461 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:611 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:04.461 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.069561+0000", 2024-09-18T04:39:04.461 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 13.307589480000001, 2024-09-18T04:39:04.461 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0072330299999999997, 2024-09-18T04:39:04.461 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:04.461 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:04.461 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:04.462 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:39:04.462 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:04.462 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 611 2024-09-18T04:39:04.462 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.462 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:04.462 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.462 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:04.462 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.069561+0000", 2024-09-18T04:39:04.462 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:04.462 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.462 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.463 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:04.463 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.069561+0000", 2024-09-18T04:39:04.463 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:04.463 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.463 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.463 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:04.463 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.069559+0000", 2024-09-18T04:39:04.463 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:04.463 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.463 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.464 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:04.464 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070801+0000", 2024-09-18T04:39:04.464 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0012423359999999999 2024-09-18T04:39:04.464 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.464 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.464 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:04.464 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070828+0000", 2024-09-18T04:39:04.464 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6387e-05 2024-09-18T04:39:04.464 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.465 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.465 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:04.465 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070840+0000", 2024-09-18T04:39:04.465 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2689e-05 2024-09-18T04:39:04.465 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.465 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.465 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:04.465 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070990+0000", 2024-09-18T04:39:04.465 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014946800000000001 2024-09-18T04:39:04.466 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.466 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.466 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:04.466 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.071128+0000", 2024-09-18T04:39:04.466 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013796699999999999 2024-09-18T04:39:04.466 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.466 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.466 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:04.467 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.071318+0000", 2024-09-18T04:39:04.467 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000190106 2024-09-18T04:39:04.467 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.467 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.467 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:04.467 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.075711+0000", 2024-09-18T04:39:04.467 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.004393002 2024-09-18T04:39:04.467 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.468 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.468 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:04.468 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.076665+0000", 2024-09-18T04:39:04.468 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00095390399999999999 2024-09-18T04:39:04.468 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.468 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.468 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:04.468 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.076742+0000", 2024-09-18T04:39:04.469 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7112000000000003e-05 2024-09-18T04:39:04.469 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.469 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.469 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:04.469 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.076794+0000", 2024-09-18T04:39:04.469 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2475999999999997e-05 2024-09-18T04:39:04.469 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:04.469 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:04.470 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:04.470 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.470 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.470 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:687 2.3 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:04.470 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.487994+0000", 2024-09-18T04:39:04.470 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 12.889156497, 2024-09-18T04:39:04.470 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0074167690000000001, 2024-09-18T04:39:04.470 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:04.470 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:04.471 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:04.471 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:39:04.471 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:04.471 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 687 2024-09-18T04:39:04.471 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.471 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:04.471 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.471 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:04.471 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.487994+0000", 2024-09-18T04:39:04.471 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:04.472 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.472 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.472 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:04.472 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.487994+0000", 2024-09-18T04:39:04.472 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:04.472 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.472 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.472 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:04.472 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.487993+0000", 2024-09-18T04:39:04.472 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:04.473 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.473 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.473 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:04.473 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.488979+0000", 2024-09-18T04:39:04.473 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00098620700000000001 2024-09-18T04:39:04.473 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.473 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.473 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:04.473 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489002+0000", 2024-09-18T04:39:04.473 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2500000000000001e-05 2024-09-18T04:39:04.474 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.474 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.474 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:04.474 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489020+0000", 2024-09-18T04:39:04.474 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7691000000000001e-05 2024-09-18T04:39:04.474 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.474 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.475 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:04.475 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489091+0000", 2024-09-18T04:39:04.475 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1307e-05 2024-09-18T04:39:04.475 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.475 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.475 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:04.475 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489219+0000", 2024-09-18T04:39:04.475 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012766000000000001 2024-09-18T04:39:04.475 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.475 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.476 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:04.476 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489387+0000", 2024-09-18T04:39:04.476 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016855400000000001 2024-09-18T04:39:04.476 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.476 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.476 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:04.476 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495243+0000", 2024-09-18T04:39:04.476 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058563189999999996 2024-09-18T04:39:04.476 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.476 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.476 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:04.476 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495266+0000", 2024-09-18T04:39:04.476 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2810999999999999e-05 2024-09-18T04:39:04.476 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.477 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.477 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:04.477 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495332+0000", 2024-09-18T04:39:04.477 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5469999999999995e-05 2024-09-18T04:39:04.477 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.477 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.477 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:04.477 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495411+0000", 2024-09-18T04:39:04.477 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9395999999999997e-05 2024-09-18T04:39:04.477 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:04.477 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:04.477 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:04.478 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.478 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.478 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:689 2.3 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:04.478 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.500070+0000", 2024-09-18T04:39:04.478 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 12.877080681000001, 2024-09-18T04:39:04.478 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071096650000000003, 2024-09-18T04:39:04.478 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:04.478 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:04.478 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:04.478 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:39:04.478 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:04.478 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 689 2024-09-18T04:39:04.478 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.479 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:04.479 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.479 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:04.479 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.500070+0000", 2024-09-18T04:39:04.479 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:04.479 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.479 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.479 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:04.479 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.500070+0000", 2024-09-18T04:39:04.479 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:04.479 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.479 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.479 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:04.479 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.500069+0000", 2024-09-18T04:39:04.480 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:04.480 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.480 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.480 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:04.480 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501082+0000", 2024-09-18T04:39:04.480 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010136570000000001 2024-09-18T04:39:04.480 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.480 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.480 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:04.480 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501104+0000", 2024-09-18T04:39:04.480 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1883e-05 2024-09-18T04:39:04.480 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.480 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.480 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:04.481 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501115+0000", 2024-09-18T04:39:04.481 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0414000000000001e-05 2024-09-18T04:39:04.481 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.481 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.481 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:04.481 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501178+0000", 2024-09-18T04:39:04.481 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3128999999999998e-05 2024-09-18T04:39:04.481 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.481 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.481 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:04.481 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501257+0000", 2024-09-18T04:39:04.482 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9592999999999995e-05 2024-09-18T04:39:04.482 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.482 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.482 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:04.482 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501366+0000", 2024-09-18T04:39:04.482 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000108643 2024-09-18T04:39:04.482 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.482 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.482 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:04.482 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.504041+0000", 2024-09-18T04:39:04.482 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0026750889999999999 2024-09-18T04:39:04.482 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.482 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.483 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:04.483 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.507097+0000", 2024-09-18T04:39:04.483 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030556070000000001 2024-09-18T04:39:04.483 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.483 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.483 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:04.483 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.507140+0000", 2024-09-18T04:39:04.483 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3745000000000002e-05 2024-09-18T04:39:04.483 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.483 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.483 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:04.484 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.507180+0000", 2024-09-18T04:39:04.484 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9475999999999999e-05 2024-09-18T04:39:04.484 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:04.484 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:04.484 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:04.484 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.484 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.484 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4972.0:13 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:04.484 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.948263+0000", 2024-09-18T04:39:04.484 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 12.42888789, 2024-09-18T04:39:04.484 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0091358970000000005, 2024-09-18T04:39:04.484 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:04.485 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:39:04.485 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:04.485 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4972", 2024-09-18T04:39:04.485 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2280251472", 2024-09-18T04:39:04.485 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 13 2024-09-18T04:39:04.485 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.485 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:04.485 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.485 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:04.485 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948263+0000", 2024-09-18T04:39:04.485 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:04.485 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.485 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.486 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:04.486 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948263+0000", 2024-09-18T04:39:04.486 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:04.486 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.486 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.486 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:04.486 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948261+0000", 2024-09-18T04:39:04.486 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:04.486 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.486 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.487 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:04.487 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948273+0000", 2024-09-18T04:39:04.487 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1576999999999999e-05 2024-09-18T04:39:04.487 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.487 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.487 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:04.487 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948305+0000", 2024-09-18T04:39:04.487 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2209000000000003e-05 2024-09-18T04:39:04.487 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.487 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.487 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:04.487 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948325+0000", 2024-09-18T04:39:04.488 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.022e-05 2024-09-18T04:39:04.488 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.488 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.488 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:04.488 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948455+0000", 2024-09-18T04:39:04.488 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000129624 2024-09-18T04:39:04.488 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.488 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.488 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:04.488 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948598+0000", 2024-09-18T04:39:04.488 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014376699999999999 2024-09-18T04:39:04.489 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.489 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.489 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:04.489 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.957399+0000", 2024-09-18T04:39:04.489 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0088004299999999997 2024-09-18T04:39:04.489 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:04.489 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:04.489 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:04.489 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.490 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.490 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:101 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:04.490 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.576367+0000", 2024-09-18T04:39:04.490 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 7.8007841820000001, 2024-09-18T04:39:04.490 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0078671020000000008, 2024-09-18T04:39:04.490 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:04.490 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:04.490 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:04.490 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:04.490 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:04.490 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 101 2024-09-18T04:39:04.491 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.491 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:04.491 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.491 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:04.491 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576367+0000", 2024-09-18T04:39:04.491 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:04.491 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.491 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.492 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:04.492 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576367+0000", 2024-09-18T04:39:04.492 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:04.492 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.492 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.492 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:04.492 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576364+0000", 2024-09-18T04:39:04.492 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999971 2024-09-18T04:39:04.493 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.493 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.493 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:04.493 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576378+0000", 2024-09-18T04:39:04.493 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3879999999999999e-05 2024-09-18T04:39:04.493 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.493 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.493 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:04.493 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576410+0000", 2024-09-18T04:39:04.494 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2286000000000002e-05 2024-09-18T04:39:04.494 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.494 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.494 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:04.494 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576425+0000", 2024-09-18T04:39:04.494 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5290000000000001e-05 2024-09-18T04:39:04.494 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.494 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.495 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:04.495 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576551+0000", 2024-09-18T04:39:04.495 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012555199999999999 2024-09-18T04:39:04.495 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.495 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.495 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:04.495 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576719+0000", 2024-09-18T04:39:04.495 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016787400000000001 2024-09-18T04:39:04.495 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.495 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.496 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:04.496 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576951+0000", 2024-09-18T04:39:04.496 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00023252400000000001 2024-09-18T04:39:04.496 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.496 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.496 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:04.496 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.581459+0000", 2024-09-18T04:39:04.496 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045074410000000001 2024-09-18T04:39:04.497 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.497 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.497 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:04.497 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584076+0000", 2024-09-18T04:39:04.497 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0026173860000000002 2024-09-18T04:39:04.497 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.497 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.497 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:04.497 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584160+0000", 2024-09-18T04:39:04.498 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4191999999999996e-05 2024-09-18T04:39:04.498 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.498 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.498 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:04.498 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584234+0000", 2024-09-18T04:39:04.498 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.339e-05 2024-09-18T04:39:04.498 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:04.498 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:04.498 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:04.498 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.499 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.499 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:44 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:04.499 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.550448+0000", 2024-09-18T04:39:04.499 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 3.8267029930000001, 2024-09-18T04:39:04.499 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082117590000000008, 2024-09-18T04:39:04.499 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:04.499 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:04.499 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:04.499 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:39:04.499 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:04.499 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 44 2024-09-18T04:39:04.500 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.500 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:04.500 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.500 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:04.500 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.550448+0000", 2024-09-18T04:39:04.500 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:04.500 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.500 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.500 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:04.500 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.550448+0000", 2024-09-18T04:39:04.501 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:04.501 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.501 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.501 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:04.501 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.550445+0000", 2024-09-18T04:39:04.501 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999971 2024-09-18T04:39:04.501 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.501 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.501 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:04.502 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.551904+0000", 2024-09-18T04:39:04.502 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001458322 2024-09-18T04:39:04.502 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.502 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.502 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:04.502 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.551938+0000", 2024-09-18T04:39:04.502 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4359000000000001e-05 2024-09-18T04:39:04.502 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.502 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.502 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:04.503 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.551959+0000", 2024-09-18T04:39:04.503 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1403000000000002e-05 2024-09-18T04:39:04.503 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.503 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.503 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:04.503 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.552049+0000", 2024-09-18T04:39:04.503 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9970999999999996e-05 2024-09-18T04:39:04.503 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.503 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.503 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:04.503 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.552190+0000", 2024-09-18T04:39:04.504 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000140271 2024-09-18T04:39:04.504 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.504 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.504 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:04.504 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.552385+0000", 2024-09-18T04:39:04.504 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019498700000000001 2024-09-18T04:39:04.504 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.504 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.504 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:04.504 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.556308+0000", 2024-09-18T04:39:04.504 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00392333 2024-09-18T04:39:04.505 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.505 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.505 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:04.505 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.558479+0000", 2024-09-18T04:39:04.505 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0021715889999999998 2024-09-18T04:39:04.505 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.505 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.505 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:04.505 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.558616+0000", 2024-09-18T04:39:04.505 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013613399999999999 2024-09-18T04:39:04.505 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.505 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.505 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:04.505 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.558660+0000", 2024-09-18T04:39:04.506 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4053e-05 2024-09-18T04:39:04.506 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:04.506 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:04.506 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:04.506 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.506 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.506 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:46 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:04.506 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.564089+0000", 2024-09-18T04:39:04.506 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 3.8130618639999998, 2024-09-18T04:39:04.506 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0069437090000000002, 2024-09-18T04:39:04.507 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:04.507 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:04.507 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:04.507 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:39:04.507 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:04.507 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 46 2024-09-18T04:39:04.507 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.507 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:04.507 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.507 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:04.507 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.564089+0000", 2024-09-18T04:39:04.507 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:04.507 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.508 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.508 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:04.508 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.564089+0000", 2024-09-18T04:39:04.508 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:04.508 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.509 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.509 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:04.509 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.564087+0000", 2024-09-18T04:39:04.509 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:04.509 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.509 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.510 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:04.510 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565126+0000", 2024-09-18T04:39:04.510 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001038902 2024-09-18T04:39:04.510 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.510 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.510 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:04.510 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565148+0000", 2024-09-18T04:39:04.511 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1911000000000001e-05 2024-09-18T04:39:04.511 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.511 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.511 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:04.511 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565157+0000", 2024-09-18T04:39:04.511 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6220000000000004e-06 2024-09-18T04:39:04.511 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.511 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.511 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:04.511 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565261+0000", 2024-09-18T04:39:04.511 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000103063 2024-09-18T04:39:04.512 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.512 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.512 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:04.512 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565394+0000", 2024-09-18T04:39:04.512 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000133032 2024-09-18T04:39:04.512 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.512 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.512 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:04.512 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565564+0000", 2024-09-18T04:39:04.512 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017068000000000001 2024-09-18T04:39:04.512 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.512 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.512 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:04.512 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.570556+0000", 2024-09-18T04:39:04.513 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0049920620000000002 2024-09-18T04:39:04.513 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.513 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.513 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:04.513 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.570885+0000", 2024-09-18T04:39:04.513 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032827900000000001 2024-09-18T04:39:04.513 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.513 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.513 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:04.513 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.570959+0000", 2024-09-18T04:39:04.513 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3955999999999995e-05 2024-09-18T04:39:04.513 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.514 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.514 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:04.514 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.571033+0000", 2024-09-18T04:39:04.514 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4191999999999997e-05 2024-09-18T04:39:04.514 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:04.514 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:04.514 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:04.514 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.514 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.514 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:100 2.4 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:04.514 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.856389+0000", 2024-09-18T04:39:04.514 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 3.5207617519999999, 2024-09-18T04:39:04.514 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0081480800000000003, 2024-09-18T04:39:04.515 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:04.515 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:04.515 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:04.515 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:39:04.515 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:04.515 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 100 2024-09-18T04:39:04.515 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.515 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:04.515 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.515 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:04.515 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.856389+0000", 2024-09-18T04:39:04.515 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:04.516 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.516 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.516 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:04.516 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.856389+0000", 2024-09-18T04:39:04.516 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:04.516 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.516 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.516 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:04.516 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.856388+0000", 2024-09-18T04:39:04.516 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:04.516 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.517 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.517 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:04.517 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857295+0000", 2024-09-18T04:39:04.517 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00090725599999999995 2024-09-18T04:39:04.517 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.517 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.517 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:04.517 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857311+0000", 2024-09-18T04:39:04.517 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6092000000000001e-05 2024-09-18T04:39:04.518 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.518 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.518 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:04.518 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857319+0000", 2024-09-18T04:39:04.518 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9200000000000004e-06 2024-09-18T04:39:04.518 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.518 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.519 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:04.519 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857388+0000", 2024-09-18T04:39:04.519 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8591999999999997e-05 2024-09-18T04:39:04.519 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.519 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.519 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:04.519 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857535+0000", 2024-09-18T04:39:04.519 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014684299999999999 2024-09-18T04:39:04.520 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.520 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.520 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:04.520 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857747+0000", 2024-09-18T04:39:04.520 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000212361 2024-09-18T04:39:04.520 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.520 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.520 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:04.521 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.860831+0000", 2024-09-18T04:39:04.521 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030841789999999998 2024-09-18T04:39:04.521 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.521 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.521 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:04.521 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.864361+0000", 2024-09-18T04:39:04.521 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0035298780000000002 2024-09-18T04:39:04.521 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.521 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.522 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:04.522 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.864456+0000", 2024-09-18T04:39:04.522 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4557999999999998e-05 2024-09-18T04:39:04.522 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.522 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.522 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:04.522 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.864537+0000", 2024-09-18T04:39:04.522 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1577999999999997e-05 2024-09-18T04:39:04.522 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:04.522 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:04.522 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:04.522 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.522 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.523 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:102 2.4 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:04.523 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.869773+0000", 2024-09-18T04:39:04.523 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 3.507378095, 2024-09-18T04:39:04.523 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0084430359999999992, 2024-09-18T04:39:04.523 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:04.523 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:04.523 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:04.523 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:39:04.523 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:04.523 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 102 2024-09-18T04:39:04.523 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.523 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:04.524 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.524 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:04.524 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.869773+0000", 2024-09-18T04:39:04.524 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:04.524 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.524 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.524 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:04.525 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.869773+0000", 2024-09-18T04:39:04.525 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:04.525 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.525 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.525 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:04.525 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.869771+0000", 2024-09-18T04:39:04.525 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:04.525 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.525 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.525 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:04.526 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870629+0000", 2024-09-18T04:39:04.526 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00085760999999999997 2024-09-18T04:39:04.526 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.526 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.526 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:04.526 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870647+0000", 2024-09-18T04:39:04.526 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8363999999999999e-05 2024-09-18T04:39:04.526 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.526 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.527 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:04.527 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870655+0000", 2024-09-18T04:39:04.527 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0099999999999995e-06 2024-09-18T04:39:04.527 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.527 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.527 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:04.527 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870803+0000", 2024-09-18T04:39:04.527 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014798 2024-09-18T04:39:04.528 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.528 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.528 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:04.528 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870951+0000", 2024-09-18T04:39:04.528 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014715999999999999 2024-09-18T04:39:04.528 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.528 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.528 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:04.529 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.871145+0000", 2024-09-18T04:39:04.529 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000194253 2024-09-18T04:39:04.529 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.529 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.529 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:04.530 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.877033+0000", 2024-09-18T04:39:04.530 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058883099999999999 2024-09-18T04:39:04.530 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.530 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.530 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:04.531 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.878078+0000", 2024-09-18T04:39:04.531 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001044747 2024-09-18T04:39:04.531 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.531 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.532 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:04.532 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.878160+0000", 2024-09-18T04:39:04.532 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1904999999999996e-05 2024-09-18T04:39:04.532 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.532 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.532 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:04.532 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.878216+0000", 2024-09-18T04:39:04.533 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6082000000000003e-05 2024-09-18T04:39:04.533 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:04.533 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:04.533 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:04.533 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.533 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.533 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:145 2.4 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:04.534 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:01.096935+0000", 2024-09-18T04:39:04.534 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 3.2802163150000001, 2024-09-18T04:39:04.534 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0069937869999999996, 2024-09-18T04:39:04.534 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:04.534 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:04.534 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:04.534 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:39:04.534 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:04.535 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 145 2024-09-18T04:39:04.535 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.535 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:04.535 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.535 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:04.535 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.096935+0000", 2024-09-18T04:39:04.535 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:04.535 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.535 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.535 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:04.535 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.096935+0000", 2024-09-18T04:39:04.535 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:04.536 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.536 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.536 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:04.536 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.096932+0000", 2024-09-18T04:39:04.536 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:04.536 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.536 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.536 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:04.536 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097815+0000", 2024-09-18T04:39:04.536 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00088276000000000001 2024-09-18T04:39:04.536 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.537 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.537 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:04.537 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097842+0000", 2024-09-18T04:39:04.537 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6806999999999999e-05 2024-09-18T04:39:04.537 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.537 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.537 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:04.537 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097852+0000", 2024-09-18T04:39:04.537 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0548e-05 2024-09-18T04:39:04.537 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.537 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.537 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:04.538 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097931+0000", 2024-09-18T04:39:04.538 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8417000000000005e-05 2024-09-18T04:39:04.538 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.538 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.538 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:04.538 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.098047+0000", 2024-09-18T04:39:04.538 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011611 2024-09-18T04:39:04.538 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.538 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.538 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:04.538 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.098172+0000", 2024-09-18T04:39:04.538 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000125482 2024-09-18T04:39:04.538 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.539 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.539 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:04.539 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.100888+0000", 2024-09-18T04:39:04.539 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0027154430000000001 2024-09-18T04:39:04.539 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.539 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.539 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:04.539 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.103814+0000", 2024-09-18T04:39:04.539 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0029259619999999998 2024-09-18T04:39:04.539 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.539 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.539 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:04.539 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.103891+0000", 2024-09-18T04:39:04.539 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7619000000000003e-05 2024-09-18T04:39:04.540 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:04.540 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:04.540 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:04.540 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.103928+0000", 2024-09-18T04:39:04.540 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6900999999999998e-05 2024-09-18T04:39:04.540 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:04.540 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:04.540 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:04.540 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:04.540 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:04.540 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:04.542 INFO:tasks.workunit.client.0.smithi115.stderr:+ DEV= 2024-09-18T04:39:04.542 INFO:tasks.workunit.client.0.smithi115.stderr:++ rbd snap ls rbd/testrbdnbd17342 2024-09-18T04:39:04.542 INFO:tasks.workunit.client.0.smithi115.stderr:++ awk '$2 == "snap" {print $1}' 2024-09-18T04:39:04.542 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:04.533+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:04.543 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:39:04.556 INFO:tasks.workunit.client.0.smithi115.stderr:+ SNAPID=4 2024-09-18T04:39:04.557 INFO:tasks.workunit.client.0.smithi115.stderr:++ _sudo rbd device --device-type nbd map --snap-id 4 rbd/testrbdnbd17342 2024-09-18T04:39:04.557 INFO:tasks.workunit.client.0.smithi115.stderr:++ local cmd 2024-09-18T04:39:04.558 INFO:tasks.workunit.client.0.smithi115.stderr:+++ id -u 2024-09-18T04:39:04.559 INFO:tasks.workunit.client.0.smithi115.stderr:++ '[' 1000 -eq 0 ']' 2024-09-18T04:39:04.559 INFO:tasks.workunit.client.0.smithi115.stderr:+++ which rbd 2024-09-18T04:39:04.562 INFO:tasks.workunit.client.0.smithi115.stderr:++ cmd=/usr/bin/rbd 2024-09-18T04:39:04.562 INFO:tasks.workunit.client.0.smithi115.stderr:++ shift 2024-09-18T04:39:04.562 INFO:tasks.workunit.client.0.smithi115.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map --snap-id 4 rbd/testrbdnbd17342 2024-09-18T04:39:04.642 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:04.637+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:04.651 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:04.651 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:04.651 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:39:04.651 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:04.661 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:39:04.733 INFO:tasks.workunit.client.0.smithi115.stderr:+ DEV=/dev/nbd0 2024-09-18T04:39:04.733 INFO:tasks.workunit.client.0.smithi115.stderr:+ get_pid rbd 2024-09-18T04:39:04.733 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pool=rbd 2024-09-18T04:39:04.734 INFO:tasks.workunit.client.0.smithi115.stderr:+ local ns= 2024-09-18T04:39:04.734 INFO:tasks.workunit.client.0.smithi115.stderr:++ rbd device --device-type nbd --format xml list 2024-09-18T04:39:04.734 INFO:tasks.workunit.client.0.smithi115.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd17342'\''][device='\''/dev/nbd0'\'']/id' 2024-09-18T04:39:04.743 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:04.737+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:04.821 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:04.822 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:04.822 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:39:04.822 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:39:04.822 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:04.823 INFO:tasks.workunit.client.0.smithi115.stderr:+ PID=18682 2024-09-18T04:39:04.823 INFO:tasks.workunit.client.0.smithi115.stderr:+ test -n 18682 2024-09-18T04:39:04.823 INFO:tasks.workunit.client.0.smithi115.stderr:+ ps -p 18682 -C rbd-nbd 2024-09-18T04:39:04.827 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:39:04.833 INFO:tasks.workunit.client.0.smithi115.stdout: PID TTY TIME CMD 2024-09-18T04:39:04.833 INFO:tasks.workunit.client.0.smithi115.stdout: 18682 ? 00:00:00 rbd-nbd 2024-09-18T04:39:04.834 INFO:tasks.workunit.client.0.smithi115.stderr:+ unmap_device '--snap-id 4 testrbdnbd17342' 18682 2024-09-18T04:39:04.834 INFO:tasks.workunit.client.0.smithi115.stderr:+ local 'args=--snap-id 4 testrbdnbd17342' 2024-09-18T04:39:04.834 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pid=18682 2024-09-18T04:39:04.834 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo rbd device --device-type nbd unmap --snap-id 4 testrbdnbd17342 2024-09-18T04:39:04.834 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:39:04.834 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:39:04.835 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:39:04.835 INFO:tasks.workunit.client.0.smithi115.stderr:++ which rbd 2024-09-18T04:39:04.836 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/bin/rbd 2024-09-18T04:39:04.836 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:39:04.836 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap --snap-id 4 testrbdnbd17342 2024-09-18T04:39:04.844 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:04.837+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:04.938 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:04.938 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:39:04.939 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:39:04.939 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:39:04.939 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.939 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:37 2.5 e51/28 2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:head v 51'1, mlcod=0'0)", 2024-09-18T04:39:04.939 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:39:04.939 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.643315918999999, 2024-09-18T04:39:04.939 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012279973, 2024-09-18T04:39:04.939 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:04.939 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:04.939 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:04.939 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.939 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:04.939 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:39:04.940 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:04.940 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.940 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.940 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:04.940 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:39:04.940 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:04.940 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.940 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.940 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:04.940 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292468+0000", 2024-09-18T04:39:04.940 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:04.940 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.940 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.941 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:04.941 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295837+0000", 2024-09-18T04:39:04.941 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0033691699999999999 2024-09-18T04:39:04.941 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.941 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.941 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:04.941 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295860+0000", 2024-09-18T04:39:04.941 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2756999999999999e-05 2024-09-18T04:39:04.941 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.941 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.941 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:04.941 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295872+0000", 2024-09-18T04:39:04.941 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2442e-05 2024-09-18T04:39:04.942 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.942 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.942 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:04.942 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295934+0000", 2024-09-18T04:39:04.942 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1187999999999998e-05 2024-09-18T04:39:04.942 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.942 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.942 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:04.942 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295967+0000", 2024-09-18T04:39:04.942 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3185999999999997e-05 2024-09-18T04:39:04.942 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.942 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.942 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:04.943 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304697+0000", 2024-09-18T04:39:04.943 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0087298059999999997 2024-09-18T04:39:04.943 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.943 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.943 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:04.943 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304749+0000", 2024-09-18T04:39:04.943 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1922000000000002e-05 2024-09-18T04:39:04.943 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:04.943 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:04.943 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:04.943 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.943 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.943 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:49 2.5 e51/28 2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head v 51'2, mlcod=0'0)", 2024-09-18T04:39:04.943 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:39:04.944 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.609989080999998, 2024-09-18T04:39:04.944 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018033041999999999, 2024-09-18T04:39:04.944 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:04.944 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:04.944 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:04.944 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.944 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:04.944 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:39:04.944 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:04.944 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.944 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.944 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:04.944 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:39:04.944 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:04.945 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.945 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.945 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:04.945 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325793+0000", 2024-09-18T04:39:04.945 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:04.945 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.945 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.945 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:04.945 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329038+0000", 2024-09-18T04:39:04.945 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0032444010000000001 2024-09-18T04:39:04.945 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.945 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.945 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:04.945 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329065+0000", 2024-09-18T04:39:04.946 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7500000000000001e-05 2024-09-18T04:39:04.946 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.946 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.946 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:04.946 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329077+0000", 2024-09-18T04:39:04.946 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1919999999999999e-05 2024-09-18T04:39:04.946 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.946 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.946 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:04.946 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329149+0000", 2024-09-18T04:39:04.946 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2349999999999997e-05 2024-09-18T04:39:04.946 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.946 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.946 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:04.946 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329192+0000", 2024-09-18T04:39:04.947 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2744000000000003e-05 2024-09-18T04:39:04.947 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.947 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.947 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:04.947 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343773+0000", 2024-09-18T04:39:04.947 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014581246000000001 2024-09-18T04:39:04.947 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.947 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.947 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:04.947 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343829+0000", 2024-09-18T04:39:04.947 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5143999999999999e-05 2024-09-18T04:39:04.947 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:04.947 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:04.947 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:04.948 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.948 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.948 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:47 2.4 e51/17 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 51'3, mlcod=51'1)", 2024-09-18T04:39:04.948 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:39:04.948 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.609408815999998, 2024-09-18T04:39:04.948 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018960186, 2024-09-18T04:39:04.948 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:04.948 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:04.948 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:04.948 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.948 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:04.948 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:39:04.948 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:04.948 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.948 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.949 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:04.949 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:39:04.949 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:04.949 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.949 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.949 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:04.949 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326374+0000", 2024-09-18T04:39:04.949 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:04.949 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.949 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.949 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:04.949 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330883+0000", 2024-09-18T04:39:04.949 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0045084740000000002 2024-09-18T04:39:04.950 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.950 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.950 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:04.950 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330901+0000", 2024-09-18T04:39:04.950 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7890999999999999e-05 2024-09-18T04:39:04.950 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.950 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.950 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:04.950 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330912+0000", 2024-09-18T04:39:04.950 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1041e-05 2024-09-18T04:39:04.950 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.950 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.950 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:04.950 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.331003+0000", 2024-09-18T04:39:04.951 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1323999999999995e-05 2024-09-18T04:39:04.951 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.951 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.951 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:04.951 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.331030+0000", 2024-09-18T04:39:04.951 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7351999999999998e-05 2024-09-18T04:39:04.951 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.951 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.951 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:04.951 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345287+0000", 2024-09-18T04:39:04.951 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014256332 2024-09-18T04:39:04.951 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.951 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.952 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:04.952 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345336+0000", 2024-09-18T04:39:04.952 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9356e-05 2024-09-18T04:39:04.952 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:04.952 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:04.952 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:04.952 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.952 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.952 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:58 2.5 e51/28 2:b227e6b1:::rbd_data.125fc09df6d1.000000000000000e:head v 51'3, mlcod=51'1)", 2024-09-18T04:39:04.952 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:39:04.952 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.568982189, 2024-09-18T04:39:04.952 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016513678, 2024-09-18T04:39:04.952 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:04.952 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:04.953 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:04.953 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.953 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:04.953 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:39:04.953 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:04.953 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.953 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.953 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:04.953 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:39:04.953 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:04.953 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.953 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.953 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:04.953 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366800+0000", 2024-09-18T04:39:04.954 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:04.954 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.954 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.954 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:04.954 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368644+0000", 2024-09-18T04:39:04.954 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001844399 2024-09-18T04:39:04.954 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.954 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.954 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:04.954 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368681+0000", 2024-09-18T04:39:04.954 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6526999999999998e-05 2024-09-18T04:39:04.954 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.954 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.954 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:04.955 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368698+0000", 2024-09-18T04:39:04.955 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7272999999999999e-05 2024-09-18T04:39:04.955 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.955 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.955 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:04.955 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368773+0000", 2024-09-18T04:39:04.955 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4862000000000003e-05 2024-09-18T04:39:04.955 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.955 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.955 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:04.955 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368801+0000", 2024-09-18T04:39:04.955 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7925000000000001e-05 2024-09-18T04:39:04.955 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.956 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.956 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:04.956 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383261+0000", 2024-09-18T04:39:04.956 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014460377999999999 2024-09-18T04:39:04.956 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.956 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.956 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:04.956 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383316+0000", 2024-09-18T04:39:04.956 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4623999999999997e-05 2024-09-18T04:39:04.956 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:04.956 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:04.956 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:04.956 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.956 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.957 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:56 2.4 e51/17 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 51'4, mlcod=51'2)", 2024-09-18T04:39:04.957 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:39:04.957 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.566978163000002, 2024-09-18T04:39:04.957 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015105221, 2024-09-18T04:39:04.957 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:04.957 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:04.957 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:04.957 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.957 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:04.957 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:39:04.957 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:04.957 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.957 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.958 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:04.958 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:39:04.958 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:04.958 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.958 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.958 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:04.958 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368805+0000", 2024-09-18T04:39:04.958 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:04.958 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.958 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.958 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:04.958 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371603+0000", 2024-09-18T04:39:04.958 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027982660000000002 2024-09-18T04:39:04.958 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.958 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.959 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:04.959 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371633+0000", 2024-09-18T04:39:04.959 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0073000000000002e-05 2024-09-18T04:39:04.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.959 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.959 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:04.959 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371652+0000", 2024-09-18T04:39:04.959 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8597000000000002e-05 2024-09-18T04:39:04.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.959 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.959 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:04.959 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371752+0000", 2024-09-18T04:39:04.959 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9772999999999995e-05 2024-09-18T04:39:04.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.960 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.960 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:04.960 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371793+0000", 2024-09-18T04:39:04.960 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1173e-05 2024-09-18T04:39:04.960 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.960 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.960 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:04.960 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383875+0000", 2024-09-18T04:39:04.961 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012081852000000001 2024-09-18T04:39:04.961 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.961 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.961 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:04.961 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383912+0000", 2024-09-18T04:39:04.961 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7045000000000002e-05 2024-09-18T04:39:04.961 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:04.961 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:04.961 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:04.961 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.961 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.961 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:36 2.5 e51/28 2:bd5b8dd8:ns::rbd_data.133e813ab5bc.0000000000000002:head v 51'4, mlcod=51'2)", 2024-09-18T04:39:04.961 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:39:04.961 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.398259171999999, 2024-09-18T04:39:04.962 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020758161000000001, 2024-09-18T04:39:04.962 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:04.962 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:04.962 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:04.962 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.962 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:04.962 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:39:04.962 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:04.962 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.962 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.962 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:04.962 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:39:04.962 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:04.962 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.963 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.963 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:04.963 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537523+0000", 2024-09-18T04:39:04.963 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:04.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.963 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.963 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:04.963 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541578+0000", 2024-09-18T04:39:04.963 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0040549710000000001 2024-09-18T04:39:04.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.963 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.963 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:04.963 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541608+0000", 2024-09-18T04:39:04.963 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9349e-05 2024-09-18T04:39:04.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.964 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.964 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:04.964 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541621+0000", 2024-09-18T04:39:04.964 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2736e-05 2024-09-18T04:39:04.964 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.964 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.964 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:04.964 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541715+0000", 2024-09-18T04:39:04.964 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4197999999999995e-05 2024-09-18T04:39:04.964 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.964 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.964 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:04.964 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541755+0000", 2024-09-18T04:39:04.964 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9818999999999999e-05 2024-09-18T04:39:04.965 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.965 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.965 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:04.965 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558217+0000", 2024-09-18T04:39:04.965 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016462234999999999 2024-09-18T04:39:04.965 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.965 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.965 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:04.965 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558284+0000", 2024-09-18T04:39:04.965 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6774999999999997e-05 2024-09-18T04:39:04.965 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:04.965 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:04.965 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:04.965 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.966 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.966 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:39 2.4 e51/17 2:304364d1:ns::rbd_data.133e813ab5bc.0000000000000004:head v 51'5, mlcod=51'3)", 2024-09-18T04:39:04.966 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:39:04.966 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.390120045, 2024-09-18T04:39:04.966 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016777558000000001, 2024-09-18T04:39:04.966 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:04.966 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:04.966 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:04.966 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.966 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:04.966 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:39:04.966 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:04.966 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.966 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.966 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:04.967 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:39:04.967 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:04.967 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.967 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.967 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:04.967 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545663+0000", 2024-09-18T04:39:04.967 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:04.967 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.967 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.967 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:04.967 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548286+0000", 2024-09-18T04:39:04.967 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026233390000000001 2024-09-18T04:39:04.967 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.968 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.968 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:04.968 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548305+0000", 2024-09-18T04:39:04.968 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8425000000000001e-05 2024-09-18T04:39:04.968 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.968 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.968 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:04.968 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548316+0000", 2024-09-18T04:39:04.968 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1163e-05 2024-09-18T04:39:04.968 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.968 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.968 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:04.968 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548379+0000", 2024-09-18T04:39:04.968 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2780000000000005e-05 2024-09-18T04:39:04.969 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.969 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.969 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:04.969 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548399+0000", 2024-09-18T04:39:04.969 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0080000000000001e-05 2024-09-18T04:39:04.969 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.969 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.969 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:04.969 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562383+0000", 2024-09-18T04:39:04.969 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013984671000000001 2024-09-18T04:39:04.969 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.969 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.969 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:04.969 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562442+0000", 2024-09-18T04:39:04.970 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8700999999999999e-05 2024-09-18T04:39:04.970 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:04.970 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:04.970 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:04.970 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.970 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.970 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:37 2.5 e51/28 2:bf39b8e4:ns::rbd_data.133e813ab5bc.0000000000000003:head v 51'5, mlcod=51'2)", 2024-09-18T04:39:04.970 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:39:04.970 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.387251775999999, 2024-09-18T04:39:04.970 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014264492, 2024-09-18T04:39:04.970 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:04.970 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:04.970 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:04.970 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.970 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:04.971 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:39:04.971 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:04.971 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.971 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.971 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:04.971 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:39:04.971 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:04.971 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.971 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.971 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:04.971 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548531+0000", 2024-09-18T04:39:04.971 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:04.971 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.971 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.972 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:04.972 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551500+0000", 2024-09-18T04:39:04.972 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0029688129999999998 2024-09-18T04:39:04.972 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.972 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.972 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:04.972 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551516+0000", 2024-09-18T04:39:04.972 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5994e-05 2024-09-18T04:39:04.972 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.972 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.972 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:04.972 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551526+0000", 2024-09-18T04:39:04.972 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9299999999999998e-06 2024-09-18T04:39:04.973 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.973 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.973 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:04.973 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551577+0000", 2024-09-18T04:39:04.973 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0871000000000001e-05 2024-09-18T04:39:04.973 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.973 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.973 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:04.973 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551602+0000", 2024-09-18T04:39:04.973 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4813000000000001e-05 2024-09-18T04:39:04.973 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.973 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.973 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:04.973 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562741+0000", 2024-09-18T04:39:04.974 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011139428999999999 2024-09-18T04:39:04.974 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.974 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.974 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:04.974 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562797+0000", 2024-09-18T04:39:04.974 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6329000000000002e-05 2024-09-18T04:39:04.974 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:04.974 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:04.974 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:04.974 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.974 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.974 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:49 2.4 e51/17 2:20f0b0c9:ns::rbd_data.133e813ab5bc.0000000000000009:head v 51'6, mlcod=51'4)", 2024-09-18T04:39:04.974 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:39:04.974 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.355817217999999, 2024-09-18T04:39:04.975 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013849046, 2024-09-18T04:39:04.975 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:04.975 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:04.975 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:04.975 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.975 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:04.975 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:39:04.975 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:04.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.975 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.975 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:04.975 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:39:04.975 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:04.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.976 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.976 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:04.976 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579966+0000", 2024-09-18T04:39:04.976 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:04.976 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.976 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.976 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:04.976 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582324+0000", 2024-09-18T04:39:04.976 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023578890000000002 2024-09-18T04:39:04.976 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.976 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.976 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:04.976 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582355+0000", 2024-09-18T04:39:04.976 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1485000000000001e-05 2024-09-18T04:39:04.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.977 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.977 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:04.977 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582371+0000", 2024-09-18T04:39:04.977 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6147999999999999e-05 2024-09-18T04:39:04.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.977 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.977 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:04.977 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582467+0000", 2024-09-18T04:39:04.977 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6059999999999998e-05 2024-09-18T04:39:04.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.977 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.977 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:04.977 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582505+0000", 2024-09-18T04:39:04.978 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7679000000000003e-05 2024-09-18T04:39:04.978 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.978 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.978 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:04.978 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593743+0000", 2024-09-18T04:39:04.978 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011238356 2024-09-18T04:39:04.978 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.978 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.978 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:04.978 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593816+0000", 2024-09-18T04:39:04.978 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2935000000000006e-05 2024-09-18T04:39:04.978 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:04.978 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:04.978 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:04.979 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.979 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.979 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:53 2.4 e51/17 2:39f7f174:ns::rbd_data.133e813ab5bc.000000000000000b:head v 51'7, mlcod=51'4)", 2024-09-18T04:39:04.979 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:39:04.979 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.341116063000001, 2024-09-18T04:39:04.979 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0079914159999999995, 2024-09-18T04:39:04.979 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:04.979 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:04.979 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:04.979 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.979 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:04.979 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:39:04.979 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:04.979 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.979 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.980 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:04.980 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:39:04.980 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:04.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.980 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.980 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:04.980 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594667+0000", 2024-09-18T04:39:04.980 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:04.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.980 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.980 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:04.980 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596446+0000", 2024-09-18T04:39:04.980 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001779033 2024-09-18T04:39:04.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.981 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.981 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:04.981 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596472+0000", 2024-09-18T04:39:04.981 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6120000000000001e-05 2024-09-18T04:39:04.981 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.981 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.981 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:04.981 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596500+0000", 2024-09-18T04:39:04.981 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7518e-05 2024-09-18T04:39:04.981 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.981 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.981 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:04.981 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596564+0000", 2024-09-18T04:39:04.981 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3880999999999994e-05 2024-09-18T04:39:04.982 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.982 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.982 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:04.982 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596596+0000", 2024-09-18T04:39:04.982 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2638999999999997e-05 2024-09-18T04:39:04.982 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.982 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.982 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:04.982 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602605+0000", 2024-09-18T04:39:04.982 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0060080699999999999 2024-09-18T04:39:04.982 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.982 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.982 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:04.982 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602660+0000", 2024-09-18T04:39:04.983 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5396000000000003e-05 2024-09-18T04:39:04.983 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:04.983 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:04.983 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:04.983 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.983 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.983 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:60 2.5 e51/28 2:a06f51e4:ns::rbd_data.133e813ab5bc.000000000000000f:head v 51'6, mlcod=51'4)", 2024-09-18T04:39:04.983 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:39:04.983 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.326299078000002, 2024-09-18T04:39:04.983 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0093954989999999999, 2024-09-18T04:39:04.983 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:04.983 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:04.983 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:04.984 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.984 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:04.984 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:39:04.984 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:04.984 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.984 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.984 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:04.984 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:39:04.984 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:04.984 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.984 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.984 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:04.984 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609483+0000", 2024-09-18T04:39:04.984 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:04.985 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.985 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.985 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:04.985 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611767+0000", 2024-09-18T04:39:04.985 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002283724 2024-09-18T04:39:04.985 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.985 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.985 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:04.985 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611791+0000", 2024-09-18T04:39:04.985 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3547e-05 2024-09-18T04:39:04.985 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.985 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.985 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:04.985 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611800+0000", 2024-09-18T04:39:04.986 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4979999999999995e-06 2024-09-18T04:39:04.986 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.986 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.986 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:04.986 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611864+0000", 2024-09-18T04:39:04.986 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3974999999999996e-05 2024-09-18T04:39:04.986 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.986 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.986 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:04.986 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611887+0000", 2024-09-18T04:39:04.986 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2633000000000002e-05 2024-09-18T04:39:04.986 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.986 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.987 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:04.987 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.618777+0000", 2024-09-18T04:39:04.987 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0068897979999999999 2024-09-18T04:39:04.987 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.987 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.987 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:04.987 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.618881+0000", 2024-09-18T04:39:04.987 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010437699999999999 2024-09-18T04:39:04.987 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:04.987 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:04.987 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:04.987 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.987 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.987 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:663 2.4 e57/17 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 57'13, mlcod=57'11)", 2024-09-18T04:39:04.988 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.355300+0000", 2024-09-18T04:39:04.988 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 13.580484143, 2024-09-18T04:39:04.988 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0049192760000000002, 2024-09-18T04:39:04.988 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:04.988 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:04.988 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:04.988 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.988 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:04.988 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355300+0000", 2024-09-18T04:39:04.988 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:04.988 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.988 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.988 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:04.988 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355300+0000", 2024-09-18T04:39:04.989 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:04.989 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.989 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.989 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:04.989 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355299+0000", 2024-09-18T04:39:04.989 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:04.989 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.989 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.989 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:04.989 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355876+0000", 2024-09-18T04:39:04.989 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00057730900000000003 2024-09-18T04:39:04.989 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.989 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.989 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:04.989 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355897+0000", 2024-09-18T04:39:04.990 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0477000000000001e-05 2024-09-18T04:39:04.990 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.990 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.990 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:04.990 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355905+0000", 2024-09-18T04:39:04.990 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5739999999999996e-06 2024-09-18T04:39:04.990 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.990 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.990 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:04.990 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355989+0000", 2024-09-18T04:39:04.990 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4002e-05 2024-09-18T04:39:04.990 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.990 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.990 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:04.990 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.356041+0000", 2024-09-18T04:39:04.991 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1347000000000002e-05 2024-09-18T04:39:04.991 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.991 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.991 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:04.991 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.360172+0000", 2024-09-18T04:39:04.991 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0041315090000000002 2024-09-18T04:39:04.991 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.991 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.991 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:04.991 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.360220+0000", 2024-09-18T04:39:04.991 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7584999999999999e-05 2024-09-18T04:39:04.991 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:04.991 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:04.991 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:04.992 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.992 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.992 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:679 2.5 e57/28 2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head v 57'20, mlcod=57'18)", 2024-09-18T04:39:04.992 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.445316+0000", 2024-09-18T04:39:04.992 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 13.490468666, 2024-09-18T04:39:04.992 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0047016760000000001, 2024-09-18T04:39:04.992 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:04.992 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:04.992 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:04.992 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.992 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:04.992 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445316+0000", 2024-09-18T04:39:04.992 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:04.992 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.992 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.993 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:04.993 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445316+0000", 2024-09-18T04:39:04.993 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:04.993 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.993 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.993 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:04.993 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445314+0000", 2024-09-18T04:39:04.993 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:04.993 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.993 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.993 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:04.993 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445947+0000", 2024-09-18T04:39:04.993 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00063298900000000003 2024-09-18T04:39:04.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.994 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.994 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:04.994 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445968+0000", 2024-09-18T04:39:04.994 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.054e-05 2024-09-18T04:39:04.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.994 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.994 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:04.994 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445981+0000", 2024-09-18T04:39:04.994 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2697000000000001e-05 2024-09-18T04:39:04.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.994 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.994 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:04.994 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.446102+0000", 2024-09-18T04:39:04.994 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012189300000000001 2024-09-18T04:39:04.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.995 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.995 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:04.995 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.446151+0000", 2024-09-18T04:39:04.995 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8279000000000003e-05 2024-09-18T04:39:04.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.995 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.995 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:04.995 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.449953+0000", 2024-09-18T04:39:04.995 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003801979 2024-09-18T04:39:04.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.995 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.995 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:04.996 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.450018+0000", 2024-09-18T04:39:04.996 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4873e-05 2024-09-18T04:39:04.996 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:04.996 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:04.996 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:04.996 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.996 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.996 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:699 2.4 e57/17 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 57'17, mlcod=57'15)", 2024-09-18T04:39:04.996 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.566073+0000", 2024-09-18T04:39:04.996 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 13.369711153000001, 2024-09-18T04:39:04.996 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055398380000000001, 2024-09-18T04:39:04.996 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:04.996 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:04.997 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:04.997 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.997 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:04.997 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566073+0000", 2024-09-18T04:39:04.997 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:04.997 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.997 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.997 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:04.997 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566073+0000", 2024-09-18T04:39:04.997 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:04.997 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.997 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.997 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:04.997 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566071+0000", 2024-09-18T04:39:04.998 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:04.998 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.998 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.998 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:04.998 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566637+0000", 2024-09-18T04:39:04.998 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00056516699999999999 2024-09-18T04:39:04.998 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.998 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.998 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:04.998 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566668+0000", 2024-09-18T04:39:04.998 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1544999999999997e-05 2024-09-18T04:39:04.998 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.998 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.998 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:04.999 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566685+0000", 2024-09-18T04:39:04.999 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6857000000000001e-05 2024-09-18T04:39:04.999 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.999 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.999 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:04.999 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566816+0000", 2024-09-18T04:39:04.999 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013092799999999999 2024-09-18T04:39:04.999 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.999 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:04.999 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:04.999 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566875+0000", 2024-09-18T04:39:04.999 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9583999999999997e-05 2024-09-18T04:39:04.999 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:04.999 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.000 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:05.000 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.571567+0000", 2024-09-18T04:39:05.000 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.004691682 2024-09-18T04:39:05.000 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.000 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.000 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:05.000 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.571613+0000", 2024-09-18T04:39:05.000 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6125000000000003e-05 2024-09-18T04:39:05.000 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.000 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:05.000 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.000 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.000 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.000 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:98 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'27, mlcod=60'25)", 2024-09-18T04:39:05.001 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.847978+0000", 2024-09-18T04:39:05.001 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.0878065939999999, 2024-09-18T04:39:05.001 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0044617709999999998, 2024-09-18T04:39:05.001 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:05.001 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:05.001 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:05.001 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.001 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:05.001 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.847978+0000", 2024-09-18T04:39:05.001 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.001 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.001 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.001 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:05.001 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.847978+0000", 2024-09-18T04:39:05.002 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.002 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.002 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:05.002 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.847977+0000", 2024-09-18T04:39:05.002 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:05.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.002 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.002 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:05.002 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849038+0000", 2024-09-18T04:39:05.002 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0010614820000000001 2024-09-18T04:39:05.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.002 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.002 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:05.002 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849065+0000", 2024-09-18T04:39:05.003 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7127999999999998e-05 2024-09-18T04:39:05.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.003 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.003 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:05.003 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849082+0000", 2024-09-18T04:39:05.003 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7170999999999999e-05 2024-09-18T04:39:05.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.003 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.003 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:05.003 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849188+0000", 2024-09-18T04:39:05.003 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010605800000000001 2024-09-18T04:39:05.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.003 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.003 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:05.004 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849233+0000", 2024-09-18T04:39:05.004 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4579000000000001e-05 2024-09-18T04:39:05.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.004 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.004 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:05.004 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.852370+0000", 2024-09-18T04:39:05.004 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00313751 2024-09-18T04:39:05.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.004 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.004 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:05.004 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.852440+0000", 2024-09-18T04:39:05.004 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9238000000000004e-05 2024-09-18T04:39:05.004 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.004 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:05.005 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.005 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.005 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.005 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:100 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'29, mlcod=64'27)", 2024-09-18T04:39:05.005 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.858766+0000", 2024-09-18T04:39:05.005 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.0770182650000004, 2024-09-18T04:39:05.005 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0061964840000000004, 2024-09-18T04:39:05.005 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:05.005 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:05.005 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:05.005 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.005 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:05.005 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.858766+0000", 2024-09-18T04:39:05.005 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.005 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.006 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.006 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:05.006 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.858766+0000", 2024-09-18T04:39:05.006 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.006 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.006 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.006 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:05.006 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.858765+0000", 2024-09-18T04:39:05.006 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:05.006 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.006 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.006 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:05.006 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861612+0000", 2024-09-18T04:39:05.006 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028473169999999998 2024-09-18T04:39:05.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.007 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.007 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:05.007 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861643+0000", 2024-09-18T04:39:05.007 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1356000000000003e-05 2024-09-18T04:39:05.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.007 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.007 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:05.007 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861656+0000", 2024-09-18T04:39:05.007 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2899e-05 2024-09-18T04:39:05.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.007 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.007 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:05.007 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861773+0000", 2024-09-18T04:39:05.007 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011637999999999999 2024-09-18T04:39:05.008 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.008 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.008 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:05.008 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861808+0000", 2024-09-18T04:39:05.008 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5002999999999997e-05 2024-09-18T04:39:05.008 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.008 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.008 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:05.008 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.864876+0000", 2024-09-18T04:39:05.008 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030680719999999998 2024-09-18T04:39:05.008 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.008 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.008 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:05.008 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.864963+0000", 2024-09-18T04:39:05.009 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7114000000000007e-05 2024-09-18T04:39:05.009 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.009 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:05.009 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.009 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.009 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.009 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:102 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'31, mlcod=64'29)", 2024-09-18T04:39:05.009 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.872098+0000", 2024-09-18T04:39:05.009 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.0636870079999996, 2024-09-18T04:39:05.009 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0064676450000000002, 2024-09-18T04:39:05.009 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:05.009 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:05.009 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:05.009 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.010 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:05.010 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.872098+0000", 2024-09-18T04:39:05.010 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.010 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.010 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:05.010 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.872098+0000", 2024-09-18T04:39:05.010 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.010 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.010 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:05.010 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.872096+0000", 2024-09-18T04:39:05.010 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:05.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.011 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.011 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:05.011 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875176+0000", 2024-09-18T04:39:05.011 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030799809999999999 2024-09-18T04:39:05.011 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.011 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.011 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:05.011 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875204+0000", 2024-09-18T04:39:05.011 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8025999999999999e-05 2024-09-18T04:39:05.011 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.011 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.011 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:05.011 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875216+0000", 2024-09-18T04:39:05.011 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2072999999999999e-05 2024-09-18T04:39:05.011 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.012 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.012 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:05.012 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875307+0000", 2024-09-18T04:39:05.012 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0373999999999999e-05 2024-09-18T04:39:05.012 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.012 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.012 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:05.012 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875349+0000", 2024-09-18T04:39:05.012 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2299000000000003e-05 2024-09-18T04:39:05.012 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.012 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.012 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:05.012 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.878496+0000", 2024-09-18T04:39:05.012 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0031468960000000002 2024-09-18T04:39:05.013 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.013 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.013 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:05.013 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.878565+0000", 2024-09-18T04:39:05.013 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9465e-05 2024-09-18T04:39:05.013 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.013 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:05.013 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.013 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.013 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.013 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:104 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'33, mlcod=64'31)", 2024-09-18T04:39:05.013 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.885100+0000", 2024-09-18T04:39:05.013 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.0506844729999996, 2024-09-18T04:39:05.013 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0051888159999999997, 2024-09-18T04:39:05.014 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:05.014 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:05.014 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:05.014 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.014 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:05.014 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.885100+0000", 2024-09-18T04:39:05.014 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.014 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.014 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.014 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:05.014 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.885100+0000", 2024-09-18T04:39:05.014 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.014 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.014 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.014 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:05.015 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.885098+0000", 2024-09-18T04:39:05.015 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:05.015 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.015 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.015 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:05.015 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886809+0000", 2024-09-18T04:39:05.015 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017110459999999999 2024-09-18T04:39:05.015 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.015 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.015 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:05.015 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886842+0000", 2024-09-18T04:39:05.016 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2638999999999997e-05 2024-09-18T04:39:05.016 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.016 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.016 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:05.016 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886855+0000", 2024-09-18T04:39:05.016 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3380000000000001e-05 2024-09-18T04:39:05.016 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.016 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.016 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:05.016 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886939+0000", 2024-09-18T04:39:05.016 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3733999999999995e-05 2024-09-18T04:39:05.016 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.016 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.017 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:05.017 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886981+0000", 2024-09-18T04:39:05.017 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1956000000000003e-05 2024-09-18T04:39:05.017 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.017 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.017 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:05.017 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.890201+0000", 2024-09-18T04:39:05.017 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003219927 2024-09-18T04:39:05.017 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.017 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.017 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:05.017 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.890289+0000", 2024-09-18T04:39:05.017 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7997999999999993e-05 2024-09-18T04:39:05.017 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.017 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:05.018 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.018 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.018 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.018 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:145 2.4 e64/62 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 64'37, mlcod=64'35)", 2024-09-18T04:39:05.018 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:01.099398+0000", 2024-09-18T04:39:05.018 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 3.8363869849999999, 2024-09-18T04:39:05.018 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0050078459999999998, 2024-09-18T04:39:05.018 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:05.018 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:05.018 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:05.018 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.018 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:05.018 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.099398+0000", 2024-09-18T04:39:05.018 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.019 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.019 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.019 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:05.019 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.099398+0000", 2024-09-18T04:39:05.019 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.019 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.019 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.019 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:05.019 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.099396+0000", 2024-09-18T04:39:05.019 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:05.019 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.019 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.019 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:05.019 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100199+0000", 2024-09-18T04:39:05.020 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00080338000000000002 2024-09-18T04:39:05.020 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.020 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.020 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:05.020 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100230+0000", 2024-09-18T04:39:05.020 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0716000000000002e-05 2024-09-18T04:39:05.020 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.020 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.020 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:05.020 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100243+0000", 2024-09-18T04:39:05.020 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3359e-05 2024-09-18T04:39:05.020 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.020 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.020 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:05.020 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100337+0000", 2024-09-18T04:39:05.021 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4387000000000003e-05 2024-09-18T04:39:05.021 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.021 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.021 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:05.021 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100396+0000", 2024-09-18T04:39:05.021 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8164000000000001e-05 2024-09-18T04:39:05.021 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.021 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.021 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:05.021 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.104309+0000", 2024-09-18T04:39:05.022 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0039137800000000004 2024-09-18T04:39:05.022 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.022 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.022 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:05.022 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.104405+0000", 2024-09-18T04:39:05.022 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6039999999999995e-05 2024-09-18T04:39:05.022 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.022 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:05.022 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.022 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.022 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.022 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:147 2.4 e64/62 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 64'39, mlcod=64'37)", 2024-09-18T04:39:05.022 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:01.110658+0000", 2024-09-18T04:39:05.022 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 3.825126896, 2024-09-18T04:39:05.023 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0046316910000000003, 2024-09-18T04:39:05.023 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:05.023 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:05.023 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:05.023 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.023 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:05.023 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.110658+0000", 2024-09-18T04:39:05.023 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.023 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.023 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.023 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:05.023 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.110658+0000", 2024-09-18T04:39:05.023 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.024 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.024 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:05.024 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.110656+0000", 2024-09-18T04:39:05.024 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:05.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.024 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.024 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:05.024 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111444+0000", 2024-09-18T04:39:05.024 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00078762400000000001 2024-09-18T04:39:05.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.024 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.024 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:05.024 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111474+0000", 2024-09-18T04:39:05.025 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0085000000000001e-05 2024-09-18T04:39:05.025 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.025 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.025 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:05.025 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111486+0000", 2024-09-18T04:39:05.025 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2645e-05 2024-09-18T04:39:05.025 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.025 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.025 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:05.025 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111581+0000", 2024-09-18T04:39:05.025 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4432000000000007e-05 2024-09-18T04:39:05.025 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.025 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.025 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:05.025 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111631+0000", 2024-09-18T04:39:05.026 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9901000000000002e-05 2024-09-18T04:39:05.026 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.026 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.026 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:05.026 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.115209+0000", 2024-09-18T04:39:05.026 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003577916 2024-09-18T04:39:05.026 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.026 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.026 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:05.026 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.115289+0000", 2024-09-18T04:39:05.026 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0704999999999994e-05 2024-09-18T04:39:05.026 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.026 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:05.026 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.027 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.027 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:05.027 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:05.029 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:39:05.029 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:05.025+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:05.030 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd device --device-type nbd list 2024-09-18T04:39:05.030 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false grep '^18682\b' 2024-09-18T04:39:05.030 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep '^18682\b' 2024-09-18T04:39:05.030 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:39:05.030 INFO:tasks.workunit.client.0.smithi115.stderr:+ ps -C rbd-nbd 2024-09-18T04:39:05.030 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false grep '^ *18682\b' 2024-09-18T04:39:05.030 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep '^ *18682\b' 2024-09-18T04:39:05.030 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:39:05.030 INFO:tasks.workunit.client.0.smithi115.stderr:+ sleep 0.5 2024-09-18T04:39:05.129 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:05.121+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:05.183 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:05.183 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:05.183 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:39:05.183 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:05.193 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:39:05.230 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:05.225+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:05.302 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:05.302 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:05.302 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:39:05.303 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:39:05.303 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:05.312 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:39:05.331 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:05.325+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:05.425 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:05.425 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:39:05.425 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:39:05.425 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:39:05.425 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.425 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:33 2.1 e51/24 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head v 51'4, mlcod=51'2)", 2024-09-18T04:39:05.426 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:39:05.426 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 23.141431670999999, 2024-09-18T04:39:05.426 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018702106, 2024-09-18T04:39:05.426 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:05.426 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:05.426 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:05.426 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.426 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:05.426 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:39:05.426 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.426 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.426 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:05.427 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:39:05.427 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.427 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.427 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.427 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:05.427 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279197+0000", 2024-09-18T04:39:05.427 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:05.427 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.427 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.427 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:05.427 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282682+0000", 2024-09-18T04:39:05.427 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0034848689999999998 2024-09-18T04:39:05.427 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.427 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.427 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:05.428 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282709+0000", 2024-09-18T04:39:05.428 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7599000000000001e-05 2024-09-18T04:39:05.428 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.428 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.428 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:05.428 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282720+0000", 2024-09-18T04:39:05.428 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.092e-05 2024-09-18T04:39:05.428 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.428 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.428 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:05.428 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282791+0000", 2024-09-18T04:39:05.428 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1268999999999998e-05 2024-09-18T04:39:05.428 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.428 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.429 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:05.429 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282843+0000", 2024-09-18T04:39:05.429 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1771999999999998e-05 2024-09-18T04:39:05.429 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.429 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.429 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:05.429 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.297830+0000", 2024-09-18T04:39:05.429 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014987228999999999 2024-09-18T04:39:05.429 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.429 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.429 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:05.429 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.297901+0000", 2024-09-18T04:39:05.429 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0209999999999994e-05 2024-09-18T04:39:05.430 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.430 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:05.430 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.430 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.430 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.430 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:38 2.0 e51/17 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head v 51'10, mlcod=51'8)", 2024-09-18T04:39:05.430 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:39:05.430 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 23.140013240999998, 2024-09-18T04:39:05.430 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.023984472, 2024-09-18T04:39:05.430 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:05.430 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:05.430 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:05.430 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.430 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:05.431 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:39:05.431 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.431 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.431 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.431 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:05.431 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:39:05.431 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.431 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.431 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.431 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:05.431 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280614+0000", 2024-09-18T04:39:05.431 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:05.431 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.431 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.431 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:05.432 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293094+0000", 2024-09-18T04:39:05.432 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012479571 2024-09-18T04:39:05.432 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.432 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.432 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:05.432 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293111+0000", 2024-09-18T04:39:05.432 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7354999999999999e-05 2024-09-18T04:39:05.432 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.432 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.432 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:05.432 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293121+0000", 2024-09-18T04:39:05.432 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6979999999999993e-06 2024-09-18T04:39:05.432 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.433 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.433 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:05.433 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293180+0000", 2024-09-18T04:39:05.433 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9333000000000003e-05 2024-09-18T04:39:05.433 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.433 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.433 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:05.433 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293203+0000", 2024-09-18T04:39:05.433 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2633000000000002e-05 2024-09-18T04:39:05.433 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.433 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.433 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:05.433 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304534+0000", 2024-09-18T04:39:05.433 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011330761 2024-09-18T04:39:05.434 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.434 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.434 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:05.434 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304601+0000", 2024-09-18T04:39:05.434 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7657000000000006e-05 2024-09-18T04:39:05.434 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.434 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:05.434 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.434 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.434 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.434 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:46 2.6 2:7ba25245:::rbd_data.125fc09df6d1.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:05.434 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:39:05.434 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 23.112470440999999, 2024-09-18T04:39:05.434 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.035604595000000003, 2024-09-18T04:39:05.434 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:05.435 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:05.435 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:05.435 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:05.435 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:05.435 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 46 2024-09-18T04:39:05.435 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.435 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:05.435 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.435 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:05.435 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:39:05.435 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.435 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.435 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.436 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:05.436 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:39:05.436 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.436 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.436 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.436 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:05.436 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308158+0000", 2024-09-18T04:39:05.436 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:05.436 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.436 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.436 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:05.436 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313191+0000", 2024-09-18T04:39:05.436 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0050323929999999996 2024-09-18T04:39:05.436 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.437 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.437 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:05.437 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313210+0000", 2024-09-18T04:39:05.437 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9673e-05 2024-09-18T04:39:05.437 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.437 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.437 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:05.437 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313221+0000", 2024-09-18T04:39:05.437 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0444e-05 2024-09-18T04:39:05.437 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.437 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.437 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:05.437 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313294+0000", 2024-09-18T04:39:05.438 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2650000000000004e-05 2024-09-18T04:39:05.438 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.438 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.438 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:05.438 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313493+0000", 2024-09-18T04:39:05.438 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019968300000000001 2024-09-18T04:39:05.438 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.438 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.438 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:05.438 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313666+0000", 2024-09-18T04:39:05.438 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00017248599999999999 2024-09-18T04:39:05.438 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.438 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.438 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:05.439 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.323175+0000", 2024-09-18T04:39:05.439 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0095088819999999998 2024-09-18T04:39:05.439 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.439 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.439 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:05.439 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343651+0000", 2024-09-18T04:39:05.439 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020475892999999998 2024-09-18T04:39:05.439 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.439 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.439 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:05.439 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343715+0000", 2024-09-18T04:39:05.439 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4443999999999995e-05 2024-09-18T04:39:05.439 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.439 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.440 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:05.440 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343764+0000", 2024-09-18T04:39:05.440 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9280000000000003e-05 2024-09-18T04:39:05.440 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.440 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:05.440 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.440 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.440 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.440 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:47 2.4 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:05.440 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:39:05.440 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 23.107388404000002, 2024-09-18T04:39:05.440 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.032570693999999997, 2024-09-18T04:39:05.440 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:05.441 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:05.441 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:05.441 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:05.441 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:05.441 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 47 2024-09-18T04:39:05.441 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.441 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:05.441 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.441 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:05.441 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:39:05.441 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.441 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.441 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.441 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:05.442 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:39:05.442 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.442 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.442 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.442 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:05.442 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313240+0000", 2024-09-18T04:39:05.442 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:05.442 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.442 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.442 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:05.442 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325825+0000", 2024-09-18T04:39:05.442 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012584425999999999 2024-09-18T04:39:05.442 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.442 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.443 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:05.443 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325848+0000", 2024-09-18T04:39:05.443 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3025999999999999e-05 2024-09-18T04:39:05.443 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.443 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.443 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:05.443 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325863+0000", 2024-09-18T04:39:05.443 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5152e-05 2024-09-18T04:39:05.443 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.443 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.443 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:05.443 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325938+0000", 2024-09-18T04:39:05.443 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4982999999999997e-05 2024-09-18T04:39:05.443 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.444 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.444 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:05.444 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326090+0000", 2024-09-18T04:39:05.444 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015207900000000001 2024-09-18T04:39:05.444 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.444 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.444 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:05.444 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326242+0000", 2024-09-18T04:39:05.444 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015163500000000001 2024-09-18T04:39:05.444 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.444 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.444 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:05.444 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343015+0000", 2024-09-18T04:39:05.445 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016773414 2024-09-18T04:39:05.445 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.445 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.445 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:05.445 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345664+0000", 2024-09-18T04:39:05.445 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026487870000000001 2024-09-18T04:39:05.445 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.445 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.445 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:05.445 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345739+0000", 2024-09-18T04:39:05.445 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5018999999999994e-05 2024-09-18T04:39:05.445 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.445 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.445 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:05.446 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345812+0000", 2024-09-18T04:39:05.446 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3547000000000006e-05 2024-09-18T04:39:05.446 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.446 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:05.446 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.446 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.446 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.446 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:48 2.6 2:61db7a3a:::rbd_data.125fc09df6d1.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:05.446 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:39:05.446 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 23.094731732, 2024-09-18T04:39:05.446 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.025064388, 2024-09-18T04:39:05.446 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:05.446 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:05.446 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:05.447 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:05.447 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:05.447 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 48 2024-09-18T04:39:05.447 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.447 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:05.447 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.447 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:05.447 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:39:05.447 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.447 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.447 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.447 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:05.447 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:39:05.448 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.448 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.448 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.448 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:05.448 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325897+0000", 2024-09-18T04:39:05.448 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:05.448 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.448 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.448 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:05.448 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333697+0000", 2024-09-18T04:39:05.448 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0078002649999999998 2024-09-18T04:39:05.448 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.448 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.448 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:05.449 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333716+0000", 2024-09-18T04:39:05.449 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8910999999999999e-05 2024-09-18T04:39:05.449 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.449 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.449 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:05.449 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333728+0000", 2024-09-18T04:39:05.449 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1936000000000001e-05 2024-09-18T04:39:05.449 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.449 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.449 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:05.449 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333792+0000", 2024-09-18T04:39:05.449 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4010999999999994e-05 2024-09-18T04:39:05.449 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.449 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.450 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:05.450 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.334082+0000", 2024-09-18T04:39:05.450 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00028955799999999998 2024-09-18T04:39:05.450 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.450 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.450 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:05.450 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.334326+0000", 2024-09-18T04:39:05.450 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00024399099999999999 2024-09-18T04:39:05.450 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.450 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.450 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:05.450 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350745+0000", 2024-09-18T04:39:05.450 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016419474999999999 2024-09-18T04:39:05.450 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.451 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.451 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:05.451 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350808+0000", 2024-09-18T04:39:05.451 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3244999999999995e-05 2024-09-18T04:39:05.451 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.451 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.451 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:05.451 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350892+0000", 2024-09-18T04:39:05.451 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4074999999999997e-05 2024-09-18T04:39:05.451 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.451 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.451 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:05.451 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350963+0000", 2024-09-18T04:39:05.451 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0412000000000003e-05 2024-09-18T04:39:05.451 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.452 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:05.452 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.452 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.452 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.452 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:51 2.6 2:775d7b11:::rbd_data.125fc09df6d1.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:05.452 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:39:05.452 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 23.086879595999999, 2024-09-18T04:39:05.452 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.023824221, 2024-09-18T04:39:05.452 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:05.452 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:05.452 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:05.452 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:05.452 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:05.453 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 51 2024-09-18T04:39:05.453 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.453 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:05.453 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.453 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:05.453 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:39:05.453 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.453 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.453 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.453 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:05.453 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:39:05.453 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.453 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.453 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.454 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:05.454 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333749+0000", 2024-09-18T04:39:05.454 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:05.454 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.454 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.454 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:05.454 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340899+0000", 2024-09-18T04:39:05.454 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0071500340000000004 2024-09-18T04:39:05.454 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.454 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.454 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:05.454 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340929+0000", 2024-09-18T04:39:05.454 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0093000000000001e-05 2024-09-18T04:39:05.455 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.455 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.455 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:05.455 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340943+0000", 2024-09-18T04:39:05.455 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4019000000000001e-05 2024-09-18T04:39:05.455 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.455 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.455 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:05.455 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343834+0000", 2024-09-18T04:39:05.455 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028918120000000001 2024-09-18T04:39:05.455 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.455 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.455 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:05.456 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344020+0000", 2024-09-18T04:39:05.456 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000185892 2024-09-18T04:39:05.456 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.456 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.456 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:05.456 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344190+0000", 2024-09-18T04:39:05.456 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016946800000000001 2024-09-18T04:39:05.456 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.456 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.456 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:05.456 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.351291+0000", 2024-09-18T04:39:05.456 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.007100805 2024-09-18T04:39:05.456 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.456 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.457 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:05.457 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357470+0000", 2024-09-18T04:39:05.457 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0061797570000000001 2024-09-18T04:39:05.457 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.457 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.457 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:05.457 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357533+0000", 2024-09-18T04:39:05.457 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2906999999999999e-05 2024-09-18T04:39:05.457 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.457 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.457 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:05.457 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357575+0000", 2024-09-18T04:39:05.457 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1403999999999999e-05 2024-09-18T04:39:05.457 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.458 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:05.458 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.458 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.458 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.458 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:56 2.4 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:05.458 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:39:05.458 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 23.070834503, 2024-09-18T04:39:05.458 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.034295870999999999, 2024-09-18T04:39:05.458 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:05.458 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:05.458 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:05.458 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:05.458 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:05.458 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 56 2024-09-18T04:39:05.459 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.459 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:05.459 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.459 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:05.459 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:39:05.459 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.459 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.459 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.459 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:05.459 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:39:05.459 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.459 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.459 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.459 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:05.460 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349794+0000", 2024-09-18T04:39:05.460 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:05.460 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.460 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.460 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:05.460 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366632+0000", 2024-09-18T04:39:05.460 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01683755 2024-09-18T04:39:05.460 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.460 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.460 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:05.460 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366669+0000", 2024-09-18T04:39:05.460 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7351000000000002e-05 2024-09-18T04:39:05.460 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.460 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.461 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:05.461 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366686+0000", 2024-09-18T04:39:05.461 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6507999999999998e-05 2024-09-18T04:39:05.461 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.461 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.461 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:05.461 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366962+0000", 2024-09-18T04:39:05.461 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00027655900000000001 2024-09-18T04:39:05.461 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.461 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.461 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:05.461 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.367452+0000", 2024-09-18T04:39:05.461 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00049025299999999996 2024-09-18T04:39:05.461 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.462 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.462 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:05.462 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368113+0000", 2024-09-18T04:39:05.462 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00066093399999999996 2024-09-18T04:39:05.462 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.462 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.462 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:05.462 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383719+0000", 2024-09-18T04:39:05.462 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015606106999999999 2024-09-18T04:39:05.462 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.462 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.462 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:05.462 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384035+0000", 2024-09-18T04:39:05.462 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00031590399999999999 2024-09-18T04:39:05.462 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.463 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.463 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:05.463 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384067+0000", 2024-09-18T04:39:05.463 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.129e-05 2024-09-18T04:39:05.463 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.463 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.463 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:05.463 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384091+0000", 2024-09-18T04:39:05.463 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4788000000000001e-05 2024-09-18T04:39:05.463 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.463 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:05.463 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.463 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.463 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.464 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:57 2.0 e51/17 2:0cdd070e:::rbd_data.125fc09df6d1.000000000000000d:head v 51'21, mlcod=51'19)", 2024-09-18T04:39:05.464 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:39:05.464 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 23.063007016, 2024-09-18T04:39:05.464 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.025216311000000002, 2024-09-18T04:39:05.464 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:05.464 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:05.464 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:05.464 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.464 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:05.464 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:39:05.464 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.464 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.464 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.464 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:05.465 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:39:05.465 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.465 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.465 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.465 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:05.465 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:39:05.465 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:05.465 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.465 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.465 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:05.465 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363844+0000", 2024-09-18T04:39:05.465 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0062218450000000002 2024-09-18T04:39:05.465 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.465 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.466 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:05.466 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363879+0000", 2024-09-18T04:39:05.466 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4382999999999999e-05 2024-09-18T04:39:05.466 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.466 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.466 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:05.466 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363893+0000", 2024-09-18T04:39:05.466 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4348999999999999e-05 2024-09-18T04:39:05.466 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.466 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.466 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:05.466 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363977+0000", 2024-09-18T04:39:05.466 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3499999999999997e-05 2024-09-18T04:39:05.466 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.467 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.467 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:05.467 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364009+0000", 2024-09-18T04:39:05.467 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2292000000000002e-05 2024-09-18T04:39:05.467 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.467 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.467 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:05.467 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.382781+0000", 2024-09-18T04:39:05.467 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018771626 2024-09-18T04:39:05.467 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.467 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.467 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:05.467 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.382839+0000", 2024-09-18T04:39:05.467 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8758000000000002e-05 2024-09-18T04:39:05.468 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.468 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:05.468 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.468 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.468 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.468 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:59 2.0 e51/17 2:19c52cde:::rbd_data.125fc09df6d1.000000000000000f:head v 51'22, mlcod=51'19)", 2024-09-18T04:39:05.468 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:39:05.468 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 23.056347089999999, 2024-09-18T04:39:05.468 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.019497112, 2024-09-18T04:39:05.468 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:05.468 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:05.468 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:05.468 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.468 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:05.469 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:39:05.469 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.469 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.469 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.469 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:05.469 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:39:05.469 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.469 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.469 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.469 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:05.469 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364282+0000", 2024-09-18T04:39:05.469 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:05.469 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.469 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.470 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:05.470 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371823+0000", 2024-09-18T04:39:05.470 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0075411829999999999 2024-09-18T04:39:05.470 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.470 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.470 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:05.470 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371837+0000", 2024-09-18T04:39:05.470 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4336e-05 2024-09-18T04:39:05.470 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.470 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.470 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:05.470 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371846+0000", 2024-09-18T04:39:05.470 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9770000000000006e-06 2024-09-18T04:39:05.470 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.471 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.471 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:05.471 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.373566+0000", 2024-09-18T04:39:05.471 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017192539999999999 2024-09-18T04:39:05.471 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.471 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.471 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:05.471 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.373595+0000", 2024-09-18T04:39:05.471 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8975999999999998e-05 2024-09-18T04:39:05.471 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.471 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.471 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:05.471 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383740+0000", 2024-09-18T04:39:05.472 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01014525 2024-09-18T04:39:05.472 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.472 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.472 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:05.472 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383780+0000", 2024-09-18T04:39:05.472 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0275999999999999e-05 2024-09-18T04:39:05.472 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.472 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:05.472 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.472 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.472 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.472 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:34 2.6 2:66289171:ns::rbd_data.133e813ab5bc.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:05.472 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.524015+0000", 2024-09-18T04:39:05.472 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.896615481000001, 2024-09-18T04:39:05.473 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027582111999999999, 2024-09-18T04:39:05.473 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:05.473 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:05.473 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:05.473 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:05.473 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:05.473 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 34 2024-09-18T04:39:05.473 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.473 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:05.473 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.473 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:05.473 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524015+0000", 2024-09-18T04:39:05.473 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.474 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.474 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.474 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:05.474 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524015+0000", 2024-09-18T04:39:05.474 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.474 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.474 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.474 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:05.474 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524012+0000", 2024-09-18T04:39:05.474 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:05.474 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.474 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.474 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:05.474 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534956+0000", 2024-09-18T04:39:05.475 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010943403000000001 2024-09-18T04:39:05.475 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.475 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.475 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:05.475 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534978+0000", 2024-09-18T04:39:05.475 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1965999999999999e-05 2024-09-18T04:39:05.475 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.475 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.475 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:05.475 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534990+0000", 2024-09-18T04:39:05.475 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2065999999999999e-05 2024-09-18T04:39:05.475 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.475 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.475 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:05.476 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535065+0000", 2024-09-18T04:39:05.476 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5523000000000002e-05 2024-09-18T04:39:05.476 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.476 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.476 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:05.476 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535208+0000", 2024-09-18T04:39:05.476 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014309500000000001 2024-09-18T04:39:05.476 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.476 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.476 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:05.476 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535343+0000", 2024-09-18T04:39:05.476 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013437400000000001 2024-09-18T04:39:05.476 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.476 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.477 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:05.477 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.546576+0000", 2024-09-18T04:39:05.477 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011233425999999999 2024-09-18T04:39:05.477 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.477 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.477 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:05.477 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551498+0000", 2024-09-18T04:39:05.477 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0049219349999999997 2024-09-18T04:39:05.477 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.477 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.477 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:05.477 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551554+0000", 2024-09-18T04:39:05.477 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6400999999999998e-05 2024-09-18T04:39:05.477 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.478 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.478 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:05.478 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551597+0000", 2024-09-18T04:39:05.478 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2283000000000002e-05 2024-09-18T04:39:05.478 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.478 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:05.478 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.478 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.478 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.478 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:33 2.1 e51/24 2:8b40f1c5:ns::rbd_data.133e813ab5bc.0000000000000000:head v 51'7, mlcod=51'5)", 2024-09-18T04:39:05.478 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.531767+0000", 2024-09-18T04:39:05.478 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.888863564000001, 2024-09-18T04:39:05.478 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015370295000000001, 2024-09-18T04:39:05.478 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:05.478 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:05.479 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:05.479 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.479 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:05.479 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531767+0000", 2024-09-18T04:39:05.479 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.479 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.479 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.479 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:05.479 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531767+0000", 2024-09-18T04:39:05.479 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.479 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.479 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.479 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:05.479 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531765+0000", 2024-09-18T04:39:05.479 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:05.480 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.480 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.480 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:05.480 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534258+0000", 2024-09-18T04:39:05.480 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002493561 2024-09-18T04:39:05.480 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.480 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.480 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:05.480 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534288+0000", 2024-09-18T04:39:05.480 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.976e-05 2024-09-18T04:39:05.480 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.480 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.480 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:05.481 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534300+0000", 2024-09-18T04:39:05.481 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1928e-05 2024-09-18T04:39:05.481 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.481 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.481 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:05.481 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534367+0000", 2024-09-18T04:39:05.481 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7045999999999995e-05 2024-09-18T04:39:05.481 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.481 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.481 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:05.481 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534394+0000", 2024-09-18T04:39:05.481 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6903999999999999e-05 2024-09-18T04:39:05.481 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.482 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.482 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:05.482 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.547073+0000", 2024-09-18T04:39:05.482 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012679365 2024-09-18T04:39:05.482 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.482 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.482 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:05.482 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.547137+0000", 2024-09-18T04:39:05.482 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3691999999999999e-05 2024-09-18T04:39:05.482 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.482 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:05.482 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.482 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.482 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.482 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:39 2.4 2:304364d1:ns::rbd_data.133e813ab5bc.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:05.483 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.535013+0000", 2024-09-18T04:39:05.483 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.885616629000001, 2024-09-18T04:39:05.483 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027764487000000001, 2024-09-18T04:39:05.483 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:05.483 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:05.483 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:05.483 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:05.483 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:05.483 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 39 2024-09-18T04:39:05.483 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.483 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:05.483 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.483 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:05.484 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535013+0000", 2024-09-18T04:39:05.484 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.484 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.484 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.484 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:05.484 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535013+0000", 2024-09-18T04:39:05.484 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.484 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.484 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.484 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:05.484 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535012+0000", 2024-09-18T04:39:05.484 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:05.484 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.484 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.485 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:05.485 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545016+0000", 2024-09-18T04:39:05.485 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01000322 2024-09-18T04:39:05.485 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.485 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.485 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:05.485 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545035+0000", 2024-09-18T04:39:05.485 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9845999999999999e-05 2024-09-18T04:39:05.485 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.485 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.485 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:05.485 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545045+0000", 2024-09-18T04:39:05.485 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3640000000000005e-06 2024-09-18T04:39:05.486 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.486 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.486 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:05.486 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545152+0000", 2024-09-18T04:39:05.486 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000107025 2024-09-18T04:39:05.486 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.486 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.486 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:05.486 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545338+0000", 2024-09-18T04:39:05.486 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018643500000000001 2024-09-18T04:39:05.486 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.486 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.486 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:05.486 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545492+0000", 2024-09-18T04:39:05.487 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015393199999999999 2024-09-18T04:39:05.487 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.487 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.487 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:05.487 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.553549+0000", 2024-09-18T04:39:05.487 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0080569170000000002 2024-09-18T04:39:05.487 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.487 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.487 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:05.487 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562693+0000", 2024-09-18T04:39:05.487 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0091441609999999996 2024-09-18T04:39:05.487 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.487 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.487 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:05.488 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562748+0000", 2024-09-18T04:39:05.488 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5047000000000003e-05 2024-09-18T04:39:05.488 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.488 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.488 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:05.488 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562778+0000", 2024-09-18T04:39:05.488 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9652e-05 2024-09-18T04:39:05.488 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.488 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:05.488 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.488 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.488 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.488 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:40 2.6 2:784d6cd1:ns::rbd_data.133e813ab5bc.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:05.488 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.545062+0000", 2024-09-18T04:39:05.489 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.875567577999998, 2024-09-18T04:39:05.489 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.032771623, 2024-09-18T04:39:05.489 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:05.489 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:05.489 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:05.489 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:05.489 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:05.489 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 40 2024-09-18T04:39:05.489 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.489 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:05.489 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.489 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:05.489 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545062+0000", 2024-09-18T04:39:05.489 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.490 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.490 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.490 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:05.490 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545062+0000", 2024-09-18T04:39:05.490 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.490 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.490 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.490 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:05.490 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545061+0000", 2024-09-18T04:39:05.490 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:05.490 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.490 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.490 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:05.490 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552477+0000", 2024-09-18T04:39:05.491 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.007415914 2024-09-18T04:39:05.491 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.491 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.491 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:05.491 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552500+0000", 2024-09-18T04:39:05.491 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3546000000000001e-05 2024-09-18T04:39:05.491 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.491 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.491 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:05.491 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552515+0000", 2024-09-18T04:39:05.491 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4606999999999999e-05 2024-09-18T04:39:05.491 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.491 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.491 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:05.492 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552661+0000", 2024-09-18T04:39:05.492 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000146007 2024-09-18T04:39:05.492 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.492 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.492 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:05.492 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552876+0000", 2024-09-18T04:39:05.492 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000214618 2024-09-18T04:39:05.492 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.492 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.492 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:05.492 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.553061+0000", 2024-09-18T04:39:05.492 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018564100000000001 2024-09-18T04:39:05.492 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.492 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.493 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:05.493 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.565612+0000", 2024-09-18T04:39:05.493 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012550301999999999 2024-09-18T04:39:05.493 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.493 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.493 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:05.493 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577695+0000", 2024-09-18T04:39:05.493 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012083493000000001 2024-09-18T04:39:05.493 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.493 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.493 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:05.493 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577772+0000", 2024-09-18T04:39:05.493 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7158999999999998e-05 2024-09-18T04:39:05.493 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.494 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.494 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:05.494 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577834+0000", 2024-09-18T04:39:05.494 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1851000000000001e-05 2024-09-18T04:39:05.494 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.494 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:05.494 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.494 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.494 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.494 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:43 2.6 2:759f27b6:ns::rbd_data.133e813ab5bc.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:05.494 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.555868+0000", 2024-09-18T04:39:05.494 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.864762083999999, 2024-09-18T04:39:05.494 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.022332497999999999, 2024-09-18T04:39:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 43 2024-09-18T04:39:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555868+0000", 2024-09-18T04:39:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.496 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:05.496 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555868+0000", 2024-09-18T04:39:05.496 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.496 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.496 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.496 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:05.496 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555865+0000", 2024-09-18T04:39:05.496 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999971 2024-09-18T04:39:05.496 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.496 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.496 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:05.496 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561031+0000", 2024-09-18T04:39:05.496 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0051658770000000001 2024-09-18T04:39:05.496 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.497 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.497 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:05.497 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561048+0000", 2024-09-18T04:39:05.497 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6569e-05 2024-09-18T04:39:05.497 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.497 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.497 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:05.497 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561055+0000", 2024-09-18T04:39:05.497 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.328e-06 2024-09-18T04:39:05.497 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.497 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.497 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:05.497 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561114+0000", 2024-09-18T04:39:05.497 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9540000000000003e-05 2024-09-18T04:39:05.498 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.498 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.498 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:05.498 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561314+0000", 2024-09-18T04:39:05.498 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019965799999999999 2024-09-18T04:39:05.498 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.498 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.498 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:05.498 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561473+0000", 2024-09-18T04:39:05.498 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000158904 2024-09-18T04:39:05.498 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.498 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.498 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:05.498 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.567015+0000", 2024-09-18T04:39:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055420859999999999 2024-09-18T04:39:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577982+0000", 2024-09-18T04:39:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010966872000000001 2024-09-18T04:39:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578162+0000", 2024-09-18T04:39:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018050999999999999 2024-09-18T04:39:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:05.500 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578200+0000", 2024-09-18T04:39:05.500 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8000999999999998e-05 2024-09-18T04:39:05.500 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.500 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:05.500 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.500 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.500 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.500 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:48 2.6 2:7c225365:ns::rbd_data.133e813ab5bc.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:05.500 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:39:05.500 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.856962861, 2024-09-18T04:39:05.500 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.024576377, 2024-09-18T04:39:05.500 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:05.500 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:05.500 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:05.500 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:05.501 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:05.501 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 48 2024-09-18T04:39:05.501 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.501 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:05.501 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.501 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:05.501 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:39:05.501 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.501 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.501 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.501 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:05.501 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:39:05.501 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.502 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.502 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.502 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:05.502 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.563666+0000", 2024-09-18T04:39:05.502 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:05.502 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.502 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.502 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:05.502 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570331+0000", 2024-09-18T04:39:05.502 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0066653329999999998 2024-09-18T04:39:05.502 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.502 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.502 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:05.502 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570348+0000", 2024-09-18T04:39:05.502 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6507e-05 2024-09-18T04:39:05.503 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.503 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.503 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:05.503 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570355+0000", 2024-09-18T04:39:05.503 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2749999999999998e-06 2024-09-18T04:39:05.503 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.503 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.503 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:05.503 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570396+0000", 2024-09-18T04:39:05.503 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0809000000000002e-05 2024-09-18T04:39:05.503 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.503 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.503 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:05.504 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570571+0000", 2024-09-18T04:39:05.504 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000174823 2024-09-18T04:39:05.504 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.504 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.504 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:05.504 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570722+0000", 2024-09-18T04:39:05.504 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015154599999999999 2024-09-18T04:39:05.504 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.504 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.504 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:05.504 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579756+0000", 2024-09-18T04:39:05.504 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0090337000000000004 2024-09-18T04:39:05.504 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.505 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.505 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:05.505 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588093+0000", 2024-09-18T04:39:05.505 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.008336553 2024-09-18T04:39:05.505 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.505 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.505 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:05.505 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588173+0000", 2024-09-18T04:39:05.505 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0333999999999994e-05 2024-09-18T04:39:05.505 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.505 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.505 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:05.505 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588244+0000", 2024-09-18T04:39:05.505 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0573000000000003e-05 2024-09-18T04:39:05.506 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.506 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:05.506 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.506 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.506 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.506 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:49 2.4 2:20f0b0c9:ns::rbd_data.133e813ab5bc.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:05.506 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.570370+0000", 2024-09-18T04:39:05.506 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.850260534, 2024-09-18T04:39:05.506 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.029633981, 2024-09-18T04:39:05.506 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:05.506 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:05.506 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:05.506 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:05.506 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:05.507 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 49 2024-09-18T04:39:05.507 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.507 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:05.507 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.507 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:05.507 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570370+0000", 2024-09-18T04:39:05.507 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.507 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.507 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.507 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:05.507 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570370+0000", 2024-09-18T04:39:05.507 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.507 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.507 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.508 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:05.508 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570369+0000", 2024-09-18T04:39:05.508 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:05.508 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.508 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.508 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:05.508 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579090+0000", 2024-09-18T04:39:05.508 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0087212339999999996 2024-09-18T04:39:05.508 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.508 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.508 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:05.508 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579109+0000", 2024-09-18T04:39:05.508 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9120999999999999e-05 2024-09-18T04:39:05.508 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.509 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.509 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:05.509 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579123+0000", 2024-09-18T04:39:05.509 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4382000000000001e-05 2024-09-18T04:39:05.509 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.509 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.509 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:05.509 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579221+0000", 2024-09-18T04:39:05.509 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.7503999999999993e-05 2024-09-18T04:39:05.509 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.509 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.509 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:05.509 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579553+0000", 2024-09-18T04:39:05.509 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00033254200000000002 2024-09-18T04:39:05.510 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.510 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.510 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:05.510 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579817+0000", 2024-09-18T04:39:05.510 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00026317099999999998 2024-09-18T04:39:05.510 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.510 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.510 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:05.510 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.599905+0000", 2024-09-18T04:39:05.510 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020088575000000001 2024-09-18T04:39:05.510 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.510 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.510 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:05.510 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.599923+0000", 2024-09-18T04:39:05.510 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7425999999999999e-05 2024-09-18T04:39:05.511 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.511 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.511 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:05.511 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.599973+0000", 2024-09-18T04:39:05.511 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0649999999999998e-05 2024-09-18T04:39:05.511 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.511 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.511 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:05.511 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.600004+0000", 2024-09-18T04:39:05.511 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0361999999999998e-05 2024-09-18T04:39:05.511 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.511 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:05.511 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.511 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.512 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.512 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:50 2.1 e51/24 2:8c706374:ns::rbd_data.133e813ab5bc.000000000000000a:head v 51'8, mlcod=51'6)", 2024-09-18T04:39:05.512 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.577029+0000", 2024-09-18T04:39:05.512 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.843601162999999, 2024-09-18T04:39:05.512 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.017362855, 2024-09-18T04:39:05.512 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:05.512 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:05.512 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:05.512 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.512 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:05.512 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577029+0000", 2024-09-18T04:39:05.512 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.512 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.513 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.513 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:05.513 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577029+0000", 2024-09-18T04:39:05.513 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.513 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.513 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.513 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:05.513 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577027+0000", 2024-09-18T04:39:05.513 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:05.513 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.513 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.513 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:05.513 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578880+0000", 2024-09-18T04:39:05.513 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0018532080000000001 2024-09-18T04:39:05.514 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.514 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.514 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:05.514 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578906+0000", 2024-09-18T04:39:05.514 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5573999999999999e-05 2024-09-18T04:39:05.514 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.514 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.514 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:05.514 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578920+0000", 2024-09-18T04:39:05.514 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4483e-05 2024-09-18T04:39:05.514 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.514 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.514 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:05.515 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578983+0000", 2024-09-18T04:39:05.515 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2248000000000004e-05 2024-09-18T04:39:05.515 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.515 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.515 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:05.515 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579016+0000", 2024-09-18T04:39:05.515 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3364000000000001e-05 2024-09-18T04:39:05.515 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.515 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.515 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:05.515 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594342+0000", 2024-09-18T04:39:05.515 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01532564 2024-09-18T04:39:05.515 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.515 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.515 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:05.516 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594392+0000", 2024-09-18T04:39:05.516 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0068999999999998e-05 2024-09-18T04:39:05.516 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.516 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:05.516 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.516 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.516 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.516 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:45 2.0 e51/17 2:0fbfa3dc:ns::rbd_data.133e813ab5bc.0000000000000007:head v 51'24, mlcod=51'22)", 2024-09-18T04:39:05.516 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.581892+0000", 2024-09-18T04:39:05.516 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.838738491000001, 2024-09-18T04:39:05.516 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015646006, 2024-09-18T04:39:05.516 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:05.516 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:05.516 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:05.517 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.517 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:05.517 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581892+0000", 2024-09-18T04:39:05.517 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.517 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.517 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.517 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:05.517 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581892+0000", 2024-09-18T04:39:05.517 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.518 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.518 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.518 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:05.518 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581890+0000", 2024-09-18T04:39:05.518 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:05.518 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.518 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.518 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:05.518 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586188+0000", 2024-09-18T04:39:05.518 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0042985769999999996 2024-09-18T04:39:05.518 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.518 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.518 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:05.518 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586210+0000", 2024-09-18T04:39:05.519 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1069000000000001e-05 2024-09-18T04:39:05.519 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.519 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.519 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:05.519 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586221+0000", 2024-09-18T04:39:05.519 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1287000000000001e-05 2024-09-18T04:39:05.519 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.519 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.519 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:05.519 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586313+0000", 2024-09-18T04:39:05.519 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1935999999999995e-05 2024-09-18T04:39:05.519 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.519 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.519 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:05.520 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586368+0000", 2024-09-18T04:39:05.520 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5578000000000002e-05 2024-09-18T04:39:05.520 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.520 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.520 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:05.520 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.597473+0000", 2024-09-18T04:39:05.520 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011105139999999999 2024-09-18T04:39:05.520 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.520 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.520 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:05.520 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.597538+0000", 2024-09-18T04:39:05.520 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4133000000000004e-05 2024-09-18T04:39:05.520 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.520 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:05.521 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.521 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.521 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.521 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:53 2.4 2:39f7f174:ns::rbd_data.133e813ab5bc.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:05.521 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.584263+0000", 2024-09-18T04:39:05.521 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.836366616999999, 2024-09-18T04:39:05.521 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018701180000000001, 2024-09-18T04:39:05.521 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:05.521 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:05.521 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:05.521 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:05.521 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:05.521 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 53 2024-09-18T04:39:05.521 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.522 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:05.522 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.522 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:05.522 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584263+0000", 2024-09-18T04:39:05.522 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.522 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.522 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.522 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:05.522 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584263+0000", 2024-09-18T04:39:05.522 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.522 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.522 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.522 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:05.522 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584262+0000", 2024-09-18T04:39:05.523 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:05.523 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.523 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.523 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:05.523 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.591617+0000", 2024-09-18T04:39:05.523 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0073550600000000001 2024-09-18T04:39:05.523 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.523 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.523 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:05.523 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.591637+0000", 2024-09-18T04:39:05.524 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0432999999999999e-05 2024-09-18T04:39:05.524 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.524 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.524 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:05.524 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.591646+0000", 2024-09-18T04:39:05.524 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7849999999999997e-06 2024-09-18T04:39:05.524 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.524 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.524 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:05.524 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593478+0000", 2024-09-18T04:39:05.524 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001831495 2024-09-18T04:39:05.524 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.524 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.524 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:05.525 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593661+0000", 2024-09-18T04:39:05.525 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018364999999999999 2024-09-18T04:39:05.525 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.525 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.525 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:05.525 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593834+0000", 2024-09-18T04:39:05.525 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000173065 2024-09-18T04:39:05.525 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.525 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.525 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:05.525 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.600670+0000", 2024-09-18T04:39:05.525 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0068354970000000003 2024-09-18T04:39:05.525 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.525 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.526 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:05.526 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602851+0000", 2024-09-18T04:39:05.526 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002181653 2024-09-18T04:39:05.526 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.526 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.526 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:05.526 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602930+0000", 2024-09-18T04:39:05.526 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8045000000000002e-05 2024-09-18T04:39:05.526 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.526 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.526 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:05.526 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602965+0000", 2024-09-18T04:39:05.526 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5102999999999999e-05 2024-09-18T04:39:05.526 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.527 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:05.527 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.527 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.527 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.527 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:56 2.6 2:72453c3e:ns::rbd_data.133e813ab5bc.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:05.527 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.595988+0000", 2024-09-18T04:39:05.527 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.824642215000001, 2024-09-18T04:39:05.527 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.030296172, 2024-09-18T04:39:05.527 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:05.527 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:05.527 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:05.527 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:05.527 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:05.527 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 56 2024-09-18T04:39:05.528 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.528 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:05.528 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.528 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:05.528 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.595988+0000", 2024-09-18T04:39:05.528 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.528 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.528 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.528 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:05.528 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.595988+0000", 2024-09-18T04:39:05.528 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.528 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.528 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.529 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:05.529 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.595986+0000", 2024-09-18T04:39:05.529 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:05.529 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.529 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.529 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:05.529 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605262+0000", 2024-09-18T04:39:05.529 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0092767240000000001 2024-09-18T04:39:05.529 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.529 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.529 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:05.529 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605282+0000", 2024-09-18T04:39:05.529 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9307e-05 2024-09-18T04:39:05.529 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.529 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.530 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:05.530 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605292+0000", 2024-09-18T04:39:05.530 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9510000000000001e-06 2024-09-18T04:39:05.530 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.530 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.530 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:05.530 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605369+0000", 2024-09-18T04:39:05.530 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7846e-05 2024-09-18T04:39:05.530 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.530 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.530 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:05.530 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605668+0000", 2024-09-18T04:39:05.530 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00029896099999999999 2024-09-18T04:39:05.530 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.531 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.531 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:05.531 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605906+0000", 2024-09-18T04:39:05.531 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000237758 2024-09-18T04:39:05.531 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.531 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.531 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:05.531 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.613532+0000", 2024-09-18T04:39:05.531 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0076260590000000001 2024-09-18T04:39:05.531 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.531 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.531 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:05.531 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.626116+0000", 2024-09-18T04:39:05.531 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012584092999999999 2024-09-18T04:39:05.532 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.532 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.532 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:05.532 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.626216+0000", 2024-09-18T04:39:05.532 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9630999999999995e-05 2024-09-18T04:39:05.532 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.532 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.532 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:05.532 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.626284+0000", 2024-09-18T04:39:05.532 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8034999999999995e-05 2024-09-18T04:39:05.532 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.532 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:05.532 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.532 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.533 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:05.533 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:05.533 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-18T04:39:05.534 INFO:tasks.workunit.client.0.smithi115.stderr:+ DEV= 2024-09-18T04:39:05.534 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd snap create rbd/ns/testrbdnbd17342@snap 2024-09-18T04:39:05.535 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:39:05.536 INFO:teuthology.orchestra.run.smithi028.stderr:set 3.6 pg_upmap_items mapping to [5->0,1->4] 2024-09-18T04:39:05.536 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:05.529+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:05.638 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:05.633+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:05.651 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:05.652 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:05.652 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:39:05.652 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:05.663 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:39:05.738 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:05.733+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:05.768 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:05.768 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:05.768 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:39:05.768 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:39:05.768 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:05.778 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:39:05.839 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:05.833+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:39:05.930 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:05.930 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:39:05.930 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:39:05.930 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:39:05.930 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.930 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:33 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:05.931 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.261182+0000", 2024-09-18T04:39:05.931 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 23.664658063000001, 2024-09-18T04:39:05.931 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.037082208999999998, 2024-09-18T04:39:05.931 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:05.931 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:05.931 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:05.931 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:05.931 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:05.931 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 33 2024-09-18T04:39:05.931 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.931 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:05.931 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.932 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:05.932 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.261182+0000", 2024-09-18T04:39:05.932 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.932 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.932 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.932 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:05.932 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.261182+0000", 2024-09-18T04:39:05.932 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.932 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.932 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.932 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:05.932 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.261180+0000", 2024-09-18T04:39:05.933 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:05.933 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.933 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.933 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:05.933 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278669+0000", 2024-09-18T04:39:05.933 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.017488877999999999 2024-09-18T04:39:05.933 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.933 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.933 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:05.933 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278689+0000", 2024-09-18T04:39:05.933 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0047999999999998e-05 2024-09-18T04:39:05.933 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.934 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.934 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:05.934 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278699+0000", 2024-09-18T04:39:05.934 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6800000000000005e-06 2024-09-18T04:39:05.934 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.934 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.934 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:05.934 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278775+0000", 2024-09-18T04:39:05.934 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6087999999999994e-05 2024-09-18T04:39:05.934 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.934 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.934 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:05.934 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278935+0000", 2024-09-18T04:39:05.935 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000159834 2024-09-18T04:39:05.935 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.935 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.935 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:05.935 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279064+0000", 2024-09-18T04:39:05.935 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012954199999999999 2024-09-18T04:39:05.935 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.935 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.935 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:05.935 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295562+0000", 2024-09-18T04:39:05.935 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016497583999999999 2024-09-18T04:39:05.935 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.935 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.936 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:05.936 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.298147+0000", 2024-09-18T04:39:05.936 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002584982 2024-09-18T04:39:05.936 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.936 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.936 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:05.936 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.298217+0000", 2024-09-18T04:39:05.936 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0400000000000004e-05 2024-09-18T04:39:05.936 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.936 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.936 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:05.936 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.298265+0000", 2024-09-18T04:39:05.936 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7580000000000002e-05 2024-09-18T04:39:05.936 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.937 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:05.937 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.937 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.937 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.937 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:34 2.3 e51/24 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head v 51'3, mlcod=18'1)", 2024-09-18T04:39:05.937 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.273067+0000", 2024-09-18T04:39:05.937 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 23.652773273000001, 2024-09-18T04:39:05.937 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.021784103999999999, 2024-09-18T04:39:05.937 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:05.937 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:05.937 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:05.937 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.937 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:05.938 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.273067+0000", 2024-09-18T04:39:05.938 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.938 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.938 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.938 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:05.938 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.273067+0000", 2024-09-18T04:39:05.938 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.938 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.938 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.938 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:05.938 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.273060+0000", 2024-09-18T04:39:05.938 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999924 2024-09-18T04:39:05.938 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.938 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.939 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:05.939 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278596+0000", 2024-09-18T04:39:05.939 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055365839999999998 2024-09-18T04:39:05.939 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.939 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.939 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:05.939 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278637+0000", 2024-09-18T04:39:05.939 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0596000000000002e-05 2024-09-18T04:39:05.939 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.939 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.939 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:05.939 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278653+0000", 2024-09-18T04:39:05.939 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6626999999999999e-05 2024-09-18T04:39:05.939 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.940 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.940 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:05.940 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278748+0000", 2024-09-18T04:39:05.940 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4483999999999999e-05 2024-09-18T04:39:05.940 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.940 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.940 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:05.940 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278799+0000", 2024-09-18T04:39:05.940 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1484999999999999e-05 2024-09-18T04:39:05.940 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.940 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.940 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:05.940 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.294785+0000", 2024-09-18T04:39:05.941 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015985256999999999 2024-09-18T04:39:05.941 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.941 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.941 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:05.941 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.294851+0000", 2024-09-18T04:39:05.941 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6728000000000002e-05 2024-09-18T04:39:05.941 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.941 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:05.941 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.941 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.941 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.941 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:37 2.5 2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:05.941 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.278722+0000", 2024-09-18T04:39:05.941 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 23.647118224, 2024-09-18T04:39:05.942 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.026323801000000001, 2024-09-18T04:39:05.942 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:05.942 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:05.942 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:05.942 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:05.942 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:05.942 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 37 2024-09-18T04:39:05.942 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.942 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:05.942 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.943 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:05.943 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278722+0000", 2024-09-18T04:39:05.943 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.943 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.943 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.943 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:05.943 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278722+0000", 2024-09-18T04:39:05.943 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.943 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.943 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.943 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:05.943 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278720+0000", 2024-09-18T04:39:05.943 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:05.943 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.944 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.944 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:05.944 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.291875+0000", 2024-09-18T04:39:05.944 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013154414 2024-09-18T04:39:05.944 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.944 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.944 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:05.944 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.291901+0000", 2024-09-18T04:39:05.944 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6339999999999999e-05 2024-09-18T04:39:05.944 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.944 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.944 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:05.944 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.291914+0000", 2024-09-18T04:39:05.945 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3043e-05 2024-09-18T04:39:05.945 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.945 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.945 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:05.945 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.291975+0000", 2024-09-18T04:39:05.945 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0949999999999998e-05 2024-09-18T04:39:05.945 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.945 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.945 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:05.945 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292173+0000", 2024-09-18T04:39:05.945 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019825499999999999 2024-09-18T04:39:05.945 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.945 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.946 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:05.946 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292300+0000", 2024-09-18T04:39:05.946 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000126616 2024-09-18T04:39:05.946 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.946 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.946 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:05.946 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304334+0000", 2024-09-18T04:39:05.946 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012033934 2024-09-18T04:39:05.946 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.946 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.946 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:05.946 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304901+0000", 2024-09-18T04:39:05.946 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00056694099999999995 2024-09-18T04:39:05.947 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.947 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.947 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:05.947 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304981+0000", 2024-09-18T04:39:05.947 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9974000000000004e-05 2024-09-18T04:39:05.947 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.947 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.947 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:05.947 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.305046+0000", 2024-09-18T04:39:05.947 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5184000000000005e-05 2024-09-18T04:39:05.947 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.947 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:05.947 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.948 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.948 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.948 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:49 2.5 2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:05.948 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.314271+0000", 2024-09-18T04:39:05.948 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 23.611569622000001, 2024-09-18T04:39:05.948 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.029779953000000001, 2024-09-18T04:39:05.948 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:05.948 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:05.948 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:05.948 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:05.948 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:05.948 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 49 2024-09-18T04:39:05.949 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.949 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:05.949 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.949 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:05.949 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.314271+0000", 2024-09-18T04:39:05.949 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.949 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.949 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.949 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:05.949 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.314271+0000", 2024-09-18T04:39:05.949 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.949 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.949 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.950 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:05.950 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.314269+0000", 2024-09-18T04:39:05.950 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:05.950 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.950 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.950 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:05.950 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.324780+0000", 2024-09-18T04:39:05.950 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010511609999999999 2024-09-18T04:39:05.950 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.950 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.950 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:05.950 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.324821+0000", 2024-09-18T04:39:05.951 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0899000000000003e-05 2024-09-18T04:39:05.951 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.951 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.951 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:05.951 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.324840+0000", 2024-09-18T04:39:05.951 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8947e-05 2024-09-18T04:39:05.951 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.951 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.951 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:05.951 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.324936+0000", 2024-09-18T04:39:05.951 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6342999999999996e-05 2024-09-18T04:39:05.951 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.951 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.951 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:05.952 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325284+0000", 2024-09-18T04:39:05.952 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000347146 2024-09-18T04:39:05.952 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.952 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.952 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:05.952 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325586+0000", 2024-09-18T04:39:05.952 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00030199799999999999 2024-09-18T04:39:05.952 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.952 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.952 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:05.952 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343710+0000", 2024-09-18T04:39:05.952 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018124583999999999 2024-09-18T04:39:05.953 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.953 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.953 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:05.953 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343991+0000", 2024-09-18T04:39:05.953 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00028085600000000001 2024-09-18T04:39:05.953 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.953 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.953 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:05.953 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344028+0000", 2024-09-18T04:39:05.953 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6637000000000002e-05 2024-09-18T04:39:05.953 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.953 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.954 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:05.954 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344051+0000", 2024-09-18T04:39:05.954 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3260000000000001e-05 2024-09-18T04:39:05.954 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.954 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:05.954 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.954 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.954 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.954 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:50 2.3 e51/24 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head v 51'4, mlcod=51'2)", 2024-09-18T04:39:05.954 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.323066+0000", 2024-09-18T04:39:05.954 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 23.602774229000001, 2024-09-18T04:39:05.954 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.022135774, 2024-09-18T04:39:05.955 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:05.955 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:05.955 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:05.955 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.955 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:05.955 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.323066+0000", 2024-09-18T04:39:05.955 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.955 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.955 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.955 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:05.955 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.323066+0000", 2024-09-18T04:39:05.955 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.955 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.956 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.956 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:05.956 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.323065+0000", 2024-09-18T04:39:05.956 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:05.956 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.956 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.956 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:05.956 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.328664+0000", 2024-09-18T04:39:05.956 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055993600000000003 2024-09-18T04:39:05.956 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.956 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.956 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:05.956 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.328692+0000", 2024-09-18T04:39:05.957 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7654e-05 2024-09-18T04:39:05.957 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.957 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.957 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:05.957 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.328706+0000", 2024-09-18T04:39:05.957 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4097000000000001e-05 2024-09-18T04:39:05.957 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.957 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.957 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:05.957 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.328816+0000", 2024-09-18T04:39:05.957 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000110281 2024-09-18T04:39:05.957 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.957 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.957 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:05.958 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.328848+0000", 2024-09-18T04:39:05.958 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1837e-05 2024-09-18T04:39:05.958 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.958 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.958 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:05.958 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345160+0000", 2024-09-18T04:39:05.958 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016312099 2024-09-18T04:39:05.958 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.958 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.958 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:05.958 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345202+0000", 2024-09-18T04:39:05.958 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1857999999999998e-05 2024-09-18T04:39:05.958 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.958 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:05.959 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.959 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.959 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:58 2.5 2:b227e6b1:::rbd_data.125fc09df6d1.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:05.959 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.351564+0000", 2024-09-18T04:39:05.959 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 23.574276777000001, 2024-09-18T04:39:05.959 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.032125956999999997, 2024-09-18T04:39:05.959 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:05.959 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:05.959 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:05.959 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:05.959 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:05.960 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 58 2024-09-18T04:39:05.960 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.960 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:05.960 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.960 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:05.960 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.351564+0000", 2024-09-18T04:39:05.960 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.960 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.960 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.960 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:05.960 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.351564+0000", 2024-09-18T04:39:05.960 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.960 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.960 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.961 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:05.961 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.351561+0000", 2024-09-18T04:39:05.961 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:05.961 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.961 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.961 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:05.961 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364390+0000", 2024-09-18T04:39:05.961 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012829063999999999 2024-09-18T04:39:05.961 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.961 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.961 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:05.961 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364418+0000", 2024-09-18T04:39:05.961 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7307000000000001e-05 2024-09-18T04:39:05.961 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.962 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.962 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:05.962 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364428+0000", 2024-09-18T04:39:05.962 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0675e-05 2024-09-18T04:39:05.962 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.962 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.962 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:05.962 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364523+0000", 2024-09-18T04:39:05.962 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4698999999999996e-05 2024-09-18T04:39:05.962 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.962 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.962 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:05.962 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364785+0000", 2024-09-18T04:39:05.962 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00026170099999999998 2024-09-18T04:39:05.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.963 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.963 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:05.963 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.365039+0000", 2024-09-18T04:39:05.963 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00025462299999999999 2024-09-18T04:39:05.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.963 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.963 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:05.963 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383143+0000", 2024-09-18T04:39:05.963 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018103768999999999 2024-09-18T04:39:05.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.963 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.963 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:05.963 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383582+0000", 2024-09-18T04:39:05.964 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00043860600000000002 2024-09-18T04:39:05.964 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.964 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.964 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:05.964 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383644+0000", 2024-09-18T04:39:05.964 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2682999999999996e-05 2024-09-18T04:39:05.964 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.964 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.964 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:05.964 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383690+0000", 2024-09-18T04:39:05.964 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5247999999999998e-05 2024-09-18T04:39:05.964 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.964 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:05.965 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.965 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.965 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.965 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:33 2.1 2:8b40f1c5:ns::rbd_data.133e813ab5bc.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:05.965 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.523765+0000", 2024-09-18T04:39:05.965 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 23.402075737000001, 2024-09-18T04:39:05.965 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027472973000000001, 2024-09-18T04:39:05.965 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:05.965 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:05.965 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:05.965 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:05.965 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:05.965 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 33 2024-09-18T04:39:05.966 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.966 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:05.966 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.966 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:05.966 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.523765+0000", 2024-09-18T04:39:05.966 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.966 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.966 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.966 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:05.966 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.523765+0000", 2024-09-18T04:39:05.966 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.966 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.966 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.966 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:05.967 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.523762+0000", 2024-09-18T04:39:05.967 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:05.967 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.967 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.967 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:05.967 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531127+0000", 2024-09-18T04:39:05.967 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0073644560000000001 2024-09-18T04:39:05.967 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.967 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.967 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:05.967 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531189+0000", 2024-09-18T04:39:05.967 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2315000000000002e-05 2024-09-18T04:39:05.967 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.968 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.968 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:05.968 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531211+0000", 2024-09-18T04:39:05.968 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1542000000000001e-05 2024-09-18T04:39:05.968 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.968 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.968 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:05.968 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531314+0000", 2024-09-18T04:39:05.968 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000103582 2024-09-18T04:39:05.968 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.968 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.968 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:05.968 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531493+0000", 2024-09-18T04:39:05.968 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000178433 2024-09-18T04:39:05.969 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.969 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.969 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:05.969 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531653+0000", 2024-09-18T04:39:05.969 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000160388 2024-09-18T04:39:05.969 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.969 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.969 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:05.969 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.547345+0000", 2024-09-18T04:39:05.969 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015691742000000002 2024-09-18T04:39:05.969 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.969 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.970 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:05.970 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551076+0000", 2024-09-18T04:39:05.970 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003730957 2024-09-18T04:39:05.970 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.970 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.970 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:05.970 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551176+0000", 2024-09-18T04:39:05.970 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010002599999999999 2024-09-18T04:39:05.970 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.970 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.970 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:05.970 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551238+0000", 2024-09-18T04:39:05.970 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1945000000000004e-05 2024-09-18T04:39:05.971 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.971 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:05.971 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.971 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.971 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.971 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:36 2.5 2:bd5b8dd8:ns::rbd_data.133e813ab5bc.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:05.971 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.531253+0000", 2024-09-18T04:39:05.971 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 23.394587428000001, 2024-09-18T04:39:05.971 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027295212999999999, 2024-09-18T04:39:05.971 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:05.971 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:05.971 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:05.971 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:05.972 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:05.972 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 36 2024-09-18T04:39:05.972 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.972 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:05.972 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.972 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:05.972 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531253+0000", 2024-09-18T04:39:05.972 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.972 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.972 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.972 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:05.972 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531253+0000", 2024-09-18T04:39:05.972 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.972 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.973 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.973 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:05.973 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531251+0000", 2024-09-18T04:39:05.973 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:05.973 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.973 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.973 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:05.973 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536540+0000", 2024-09-18T04:39:05.973 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0052885839999999998 2024-09-18T04:39:05.973 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.973 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.973 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:05.973 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536604+0000", 2024-09-18T04:39:05.974 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4824000000000001e-05 2024-09-18T04:39:05.974 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.974 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.974 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:05.974 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536621+0000", 2024-09-18T04:39:05.974 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6215999999999999e-05 2024-09-18T04:39:05.974 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.974 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.974 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:05.974 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536768+0000", 2024-09-18T04:39:05.974 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014767900000000001 2024-09-18T04:39:05.974 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.974 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.975 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:05.975 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537084+0000", 2024-09-18T04:39:05.975 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00031553499999999998 2024-09-18T04:39:05.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.975 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.975 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:05.975 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537300+0000", 2024-09-18T04:39:05.975 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000216145 2024-09-18T04:39:05.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.975 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.975 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:05.975 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555326+0000", 2024-09-18T04:39:05.976 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018025861000000001 2024-09-18T04:39:05.976 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.976 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.976 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:05.976 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558449+0000", 2024-09-18T04:39:05.976 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0031229750000000001 2024-09-18T04:39:05.976 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.976 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.976 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:05.976 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558502+0000", 2024-09-18T04:39:05.976 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3572000000000001e-05 2024-09-18T04:39:05.976 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.976 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.977 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:05.977 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558548+0000", 2024-09-18T04:39:05.977 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5921999999999998e-05 2024-09-18T04:39:05.977 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.977 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:05.977 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.977 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.977 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:37 2.5 2:bf39b8e4:ns::rbd_data.133e813ab5bc.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:05.977 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.536664+0000", 2024-09-18T04:39:05.977 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 23.389176012, 2024-09-18T04:39:05.977 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.026331765, 2024-09-18T04:39:05.977 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:05.977 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:05.978 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:05.978 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:05.978 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:05.978 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 37 2024-09-18T04:39:05.978 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.978 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:05.978 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.978 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:05.978 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536664+0000", 2024-09-18T04:39:05.978 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.978 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.978 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.978 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:05.979 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536664+0000", 2024-09-18T04:39:05.979 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.979 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.979 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.979 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:05.979 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536662+0000", 2024-09-18T04:39:05.979 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:05.979 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.979 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.979 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:05.979 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.543699+0000", 2024-09-18T04:39:05.979 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0070367119999999997 2024-09-18T04:39:05.979 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.979 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.980 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:05.980 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.543749+0000", 2024-09-18T04:39:05.980 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0476999999999998e-05 2024-09-18T04:39:05.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.980 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.980 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:05.980 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.543800+0000", 2024-09-18T04:39:05.980 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0822999999999998e-05 2024-09-18T04:39:05.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.980 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.980 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:05.980 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.547868+0000", 2024-09-18T04:39:05.980 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0040679339999999996 2024-09-18T04:39:05.981 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.981 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.981 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:05.981 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548087+0000", 2024-09-18T04:39:05.981 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000218994 2024-09-18T04:39:05.981 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.981 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.981 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:05.981 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548383+0000", 2024-09-18T04:39:05.981 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000295588 2024-09-18T04:39:05.981 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.981 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.981 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:05.981 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562622+0000", 2024-09-18T04:39:05.982 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014239915000000001 2024-09-18T04:39:05.982 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.982 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.982 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:05.982 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562924+0000", 2024-09-18T04:39:05.982 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00030127600000000001 2024-09-18T04:39:05.982 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.982 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.982 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:05.982 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562957+0000", 2024-09-18T04:39:05.982 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3365999999999999e-05 2024-09-18T04:39:05.982 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.982 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.983 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:05.983 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562996+0000", 2024-09-18T04:39:05.983 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9153999999999998e-05 2024-09-18T04:39:05.983 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.983 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:05.983 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.983 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.983 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.983 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:50 2.1 2:8c706374:ns::rbd_data.133e813ab5bc.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:05.983 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.570513+0000", 2024-09-18T04:39:05.983 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 23.355327729999999, 2024-09-18T04:39:05.983 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.024182995999999998, 2024-09-18T04:39:05.983 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:05.983 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:05.984 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:05.984 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:05.984 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:05.984 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 50 2024-09-18T04:39:05.984 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.984 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:05.984 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.984 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:05.984 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570513+0000", 2024-09-18T04:39:05.984 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.984 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.984 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.985 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:05.985 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570513+0000", 2024-09-18T04:39:05.985 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.985 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.985 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.985 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:05.985 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570511+0000", 2024-09-18T04:39:05.985 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:05.985 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.985 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.985 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:05.985 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.576327+0000", 2024-09-18T04:39:05.985 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0058162530000000004 2024-09-18T04:39:05.985 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.986 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.986 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:05.986 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.576355+0000", 2024-09-18T04:39:05.986 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8030000000000001e-05 2024-09-18T04:39:05.986 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.986 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.986 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:05.986 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.576366+0000", 2024-09-18T04:39:05.986 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1139e-05 2024-09-18T04:39:05.986 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.986 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.986 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:05.986 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.576458+0000", 2024-09-18T04:39:05.987 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1780999999999994e-05 2024-09-18T04:39:05.987 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.987 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.987 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:05.987 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.576701+0000", 2024-09-18T04:39:05.987 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00024303899999999999 2024-09-18T04:39:05.987 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.987 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.987 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:05.987 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.576876+0000", 2024-09-18T04:39:05.987 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000174637 2024-09-18T04:39:05.987 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.987 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.988 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:05.988 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588432+0000", 2024-09-18T04:39:05.988 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011555943000000001 2024-09-18T04:39:05.988 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.988 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.988 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:05.988 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594573+0000", 2024-09-18T04:39:05.988 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0061408670000000004 2024-09-18T04:39:05.988 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.988 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.988 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:05.988 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594640+0000", 2024-09-18T04:39:05.988 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7404999999999996e-05 2024-09-18T04:39:05.989 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.989 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.989 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:05.989 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594696+0000", 2024-09-18T04:39:05.989 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5612999999999998e-05 2024-09-18T04:39:05.989 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.989 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:05.989 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.989 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.989 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.989 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:60 2.5 2:a06f51e4:ns::rbd_data.133e813ab5bc.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:05.989 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.604753+0000", 2024-09-18T04:39:05.989 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 23.321087610999999, 2024-09-18T04:39:05.990 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014400181, 2024-09-18T04:39:05.990 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:05.990 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:05.990 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:05.990 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:05.990 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:05.990 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 60 2024-09-18T04:39:05.990 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.990 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:05.990 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.990 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:05.990 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.604753+0000", 2024-09-18T04:39:05.990 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.991 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.991 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.991 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:05.991 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.604753+0000", 2024-09-18T04:39:05.991 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.991 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.991 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.991 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:05.991 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.604751+0000", 2024-09-18T04:39:05.991 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:05.991 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.991 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.991 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:05.991 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.608980+0000", 2024-09-18T04:39:05.992 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0042292420000000002 2024-09-18T04:39:05.992 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.992 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.992 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:05.992 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609014+0000", 2024-09-18T04:39:05.992 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3572999999999999e-05 2024-09-18T04:39:05.992 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.992 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.992 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:05.992 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609032+0000", 2024-09-18T04:39:05.992 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8150000000000001e-05 2024-09-18T04:39:05.992 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.992 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.993 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:05.993 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609090+0000", 2024-09-18T04:39:05.993 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8214999999999997e-05 2024-09-18T04:39:05.993 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.993 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.993 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:05.993 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609235+0000", 2024-09-18T04:39:05.993 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014437799999999999 2024-09-18T04:39:05.993 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.993 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.993 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:05.993 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609360+0000", 2024-09-18T04:39:05.993 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012526099999999999 2024-09-18T04:39:05.993 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.994 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.994 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:05.994 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.614952+0000", 2024-09-18T04:39:05.994 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055922300000000001 2024-09-18T04:39:05.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.994 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.994 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:05.994 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.619070+0000", 2024-09-18T04:39:05.994 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0041178250000000003 2024-09-18T04:39:05.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.994 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.994 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:05.994 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.619117+0000", 2024-09-18T04:39:05.994 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7275999999999999e-05 2024-09-18T04:39:05.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.995 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.995 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:05.995 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.619153+0000", 2024-09-18T04:39:05.995 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5970000000000003e-05 2024-09-18T04:39:05.995 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.995 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:05.995 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:05.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.995 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.995 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4805.0:592 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [write 393216~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:05.995 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.023386+0000", 2024-09-18T04:39:05.995 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 14.902454905999999, 2024-09-18T04:39:05.996 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0086813870000000005, 2024-09-18T04:39:05.996 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:05.996 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:05.996 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:05.996 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4805", 2024-09-18T04:39:05.996 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:05.996 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 592 2024-09-18T04:39:05.996 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.996 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:05.996 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.996 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:05.996 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023386+0000", 2024-09-18T04:39:05.996 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.997 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.997 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.997 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:05.997 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023386+0000", 2024-09-18T04:39:05.997 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:05.997 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.997 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.997 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:05.997 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023384+0000", 2024-09-18T04:39:05.997 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:05.997 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.997 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.997 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:05.997 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023616+0000", 2024-09-18T04:39:05.998 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00023165600000000001 2024-09-18T04:39:05.998 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.998 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.998 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:05.998 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023637+0000", 2024-09-18T04:39:05.998 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1163e-05 2024-09-18T04:39:05.998 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.998 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.998 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:05.998 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023652+0000", 2024-09-18T04:39:05.998 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4562000000000001e-05 2024-09-18T04:39:05.998 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.998 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.998 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:05.998 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.027732+0000", 2024-09-18T04:39:05.999 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0040803929999999999 2024-09-18T04:39:05.999 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.999 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.999 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:05.999 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.027841+0000", 2024-09-18T04:39:05.999 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000108978 2024-09-18T04:39:05.999 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.999 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.999 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:05.999 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.027990+0000", 2024-09-18T04:39:05.999 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014852999999999999 2024-09-18T04:39:05.999 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:05.999 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:05.999 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:06.000 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.031930+0000", 2024-09-18T04:39:06.000 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0039406299999999997 2024-09-18T04:39:06.000 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.000 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.000 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:06.000 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.031978+0000", 2024-09-18T04:39:06.000 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7817e-05 2024-09-18T04:39:06.000 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.000 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.000 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:06.000 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.032019+0000", 2024-09-18T04:39:06.000 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0531999999999998e-05 2024-09-18T04:39:06.000 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.001 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.001 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:06.001 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.032067+0000", 2024-09-18T04:39:06.001 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8371000000000002e-05 2024-09-18T04:39:06.001 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:06.001 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:06.001 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:06.001 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.001 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.001 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4805.0:593 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [write 524288~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:06.001 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.023665+0000", 2024-09-18T04:39:06.001 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 14.902175058999999, 2024-09-18T04:39:06.001 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0085518090000000005, 2024-09-18T04:39:06.002 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:06.002 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:06.002 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:06.002 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4805", 2024-09-18T04:39:06.002 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:06.002 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 593 2024-09-18T04:39:06.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.002 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:06.002 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.002 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:06.002 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023665+0000", 2024-09-18T04:39:06.002 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:06.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.002 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.003 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:06.003 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023665+0000", 2024-09-18T04:39:06.003 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:06.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.003 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.003 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:06.003 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023664+0000", 2024-09-18T04:39:06.003 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:06.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.003 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.003 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:06.003 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023875+0000", 2024-09-18T04:39:06.003 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000210862 2024-09-18T04:39:06.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.004 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.004 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:06.004 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023906+0000", 2024-09-18T04:39:06.004 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0660000000000001e-05 2024-09-18T04:39:06.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.004 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.004 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:06.004 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023922+0000", 2024-09-18T04:39:06.004 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5831000000000001e-05 2024-09-18T04:39:06.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.004 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.004 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:06.005 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.029603+0000", 2024-09-18T04:39:06.005 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0056816719999999996 2024-09-18T04:39:06.005 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.005 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.005 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:06.005 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.029766+0000", 2024-09-18T04:39:06.005 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000162903 2024-09-18T04:39:06.005 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.005 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.005 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:06.005 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.030042+0000", 2024-09-18T04:39:06.005 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00027561399999999998 2024-09-18T04:39:06.005 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.006 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.006 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:06.006 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.032127+0000", 2024-09-18T04:39:06.006 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020850920000000002 2024-09-18T04:39:06.006 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.006 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.006 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:06.006 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.032137+0000", 2024-09-18T04:39:06.006 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0312000000000001e-05 2024-09-18T04:39:06.006 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.006 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.006 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:06.006 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.032183+0000", 2024-09-18T04:39:06.006 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6020999999999999e-05 2024-09-18T04:39:06.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.007 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.007 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:06.007 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.032217+0000", 2024-09-18T04:39:06.007 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4109999999999997e-05 2024-09-18T04:39:06.007 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:06.007 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:06.007 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:06.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.007 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.007 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4805.0:594 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [write 655360~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:06.007 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.023941+0000", 2024-09-18T04:39:06.007 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 14.901899521000001, 2024-09-18T04:39:06.007 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010767616000000001, 2024-09-18T04:39:06.008 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:06.008 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:06.008 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:06.008 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4805", 2024-09-18T04:39:06.008 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:06.008 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 594 2024-09-18T04:39:06.008 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.008 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:06.008 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.008 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:06.008 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023941+0000", 2024-09-18T04:39:06.008 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:06.008 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.009 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.009 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:06.009 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023941+0000", 2024-09-18T04:39:06.009 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:06.009 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.009 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.009 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:06.009 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023940+0000", 2024-09-18T04:39:06.009 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:06.009 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.009 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.009 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:06.009 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024163+0000", 2024-09-18T04:39:06.009 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000223017 2024-09-18T04:39:06.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.010 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.010 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:06.010 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024188+0000", 2024-09-18T04:39:06.010 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5174e-05 2024-09-18T04:39:06.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.010 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.010 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:06.010 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024214+0000", 2024-09-18T04:39:06.010 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5782000000000002e-05 2024-09-18T04:39:06.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.010 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.010 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:06.010 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.031116+0000", 2024-09-18T04:39:06.011 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.006902282 2024-09-18T04:39:06.011 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.011 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.011 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:06.011 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.031199+0000", 2024-09-18T04:39:06.011 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3203999999999998e-05 2024-09-18T04:39:06.011 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.011 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.011 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:06.011 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.031315+0000", 2024-09-18T04:39:06.011 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000115742 2024-09-18T04:39:06.011 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.011 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.012 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:06.012 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.033780+0000", 2024-09-18T04:39:06.012 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002465058 2024-09-18T04:39:06.012 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.012 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.012 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:06.012 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.034630+0000", 2024-09-18T04:39:06.012 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00084966800000000004 2024-09-18T04:39:06.012 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.012 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.012 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:06.012 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.034682+0000", 2024-09-18T04:39:06.013 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2562999999999998e-05 2024-09-18T04:39:06.013 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.013 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.013 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:06.013 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.034709+0000", 2024-09-18T04:39:06.013 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.62e-05 2024-09-18T04:39:06.013 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:06.013 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:06.013 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:06.013 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.013 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.013 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4805.0:595 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [write 786432~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:06.013 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.024233+0000", 2024-09-18T04:39:06.014 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 14.901607146, 2024-09-18T04:39:06.014 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010599558, 2024-09-18T04:39:06.014 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:06.014 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:06.014 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:06.014 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4805", 2024-09-18T04:39:06.014 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:06.014 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 595 2024-09-18T04:39:06.014 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.014 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:06.014 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.014 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:06.014 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024233+0000", 2024-09-18T04:39:06.014 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:06.014 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.015 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.015 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:06.015 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024233+0000", 2024-09-18T04:39:06.015 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:06.015 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.015 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.015 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:06.015 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024232+0000", 2024-09-18T04:39:06.015 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:06.015 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.015 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.015 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:06.015 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024449+0000", 2024-09-18T04:39:06.015 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000216868 2024-09-18T04:39:06.016 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.016 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.016 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:06.016 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024474+0000", 2024-09-18T04:39:06.016 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4598999999999999e-05 2024-09-18T04:39:06.016 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.016 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.016 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:06.016 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024487+0000", 2024-09-18T04:39:06.016 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3451999999999999e-05 2024-09-18T04:39:06.016 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.016 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.016 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:06.016 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.032272+0000", 2024-09-18T04:39:06.017 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0077845709999999997 2024-09-18T04:39:06.017 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.017 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.017 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:06.017 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.032332+0000", 2024-09-18T04:39:06.017 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0736999999999998e-05 2024-09-18T04:39:06.017 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.017 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.017 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:06.017 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.032421+0000", 2024-09-18T04:39:06.017 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8993000000000007e-05 2024-09-18T04:39:06.017 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.017 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.018 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:06.018 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.033786+0000", 2024-09-18T04:39:06.018 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001364451 2024-09-18T04:39:06.018 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.018 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.018 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:06.018 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.034752+0000", 2024-09-18T04:39:06.018 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00096642400000000002 2024-09-18T04:39:06.018 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.018 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.018 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:06.018 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.034789+0000", 2024-09-18T04:39:06.018 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6899e-05 2024-09-18T04:39:06.018 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.019 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.019 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:06.019 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.034833+0000", 2024-09-18T04:39:06.019 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3846e-05 2024-09-18T04:39:06.019 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:06.019 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:06.019 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:06.019 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.019 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.019 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4805.0:596 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [write 917504~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:06.019 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.024500+0000", 2024-09-18T04:39:06.019 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 14.901340361000001, 2024-09-18T04:39:06.019 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010770534, 2024-09-18T04:39:06.019 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:06.020 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:06.020 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:06.020 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4805", 2024-09-18T04:39:06.020 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:06.020 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 596 2024-09-18T04:39:06.020 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.020 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:06.020 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.020 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:06.020 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024500+0000", 2024-09-18T04:39:06.020 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:06.020 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.020 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.020 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:06.021 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024500+0000", 2024-09-18T04:39:06.021 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:06.021 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.021 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.021 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:06.021 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024499+0000", 2024-09-18T04:39:06.021 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:06.021 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.021 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.021 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:06.021 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024700+0000", 2024-09-18T04:39:06.021 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00020038599999999999 2024-09-18T04:39:06.021 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.021 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.022 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:06.022 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024729+0000", 2024-09-18T04:39:06.022 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9224999999999999e-05 2024-09-18T04:39:06.022 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.022 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.022 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:06.022 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024776+0000", 2024-09-18T04:39:06.022 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7385000000000001e-05 2024-09-18T04:39:06.022 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.022 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.022 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:06.022 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.032998+0000", 2024-09-18T04:39:06.022 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0082216189999999995 2024-09-18T04:39:06.022 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.023 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.023 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:06.023 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.033076+0000", 2024-09-18T04:39:06.023 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8668e-05 2024-09-18T04:39:06.023 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.023 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.023 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:06.023 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.033167+0000", 2024-09-18T04:39:06.023 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0728000000000003e-05 2024-09-18T04:39:06.023 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.023 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.023 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:06.023 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.034399+0000", 2024-09-18T04:39:06.024 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00123234 2024-09-18T04:39:06.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.024 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.024 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:06.024 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.035206+0000", 2024-09-18T04:39:06.024 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00080701799999999995 2024-09-18T04:39:06.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.024 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.024 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:06.024 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.035244+0000", 2024-09-18T04:39:06.024 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7487999999999998e-05 2024-09-18T04:39:06.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.024 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.024 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:06.025 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.035271+0000", 2024-09-18T04:39:06.025 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6709000000000001e-05 2024-09-18T04:39:06.025 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:06.025 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:06.025 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:06.025 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.025 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.025 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4805.0:597 2.1 2:9370a598:::rbd_header.125fc09df6d1:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:06.025 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.025206+0000", 2024-09-18T04:39:06.025 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 14.90063426, 2024-09-18T04:39:06.025 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0091536900000000008, 2024-09-18T04:39:06.025 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:06.025 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:06.026 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:06.026 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4805", 2024-09-18T04:39:06.026 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:06.026 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 597 2024-09-18T04:39:06.026 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.026 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:06.026 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.026 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:06.026 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.025206+0000", 2024-09-18T04:39:06.026 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:06.026 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.026 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.026 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:06.026 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.025206+0000", 2024-09-18T04:39:06.027 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:06.027 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.027 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.027 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:06.027 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.025204+0000", 2024-09-18T04:39:06.027 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:06.027 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.027 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.027 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:06.027 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.025216+0000", 2024-09-18T04:39:06.027 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2150999999999999e-05 2024-09-18T04:39:06.027 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.027 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.027 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:06.028 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.025237+0000", 2024-09-18T04:39:06.028 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0740000000000001e-05 2024-09-18T04:39:06.028 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.028 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.028 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:06.028 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.025254+0000", 2024-09-18T04:39:06.028 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6731e-05 2024-09-18T04:39:06.028 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.028 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.028 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:06.028 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.034049+0000", 2024-09-18T04:39:06.028 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0087952810000000003 2024-09-18T04:39:06.028 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.029 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.029 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:06.029 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.034202+0000", 2024-09-18T04:39:06.029 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0001526 2024-09-18T04:39:06.029 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.029 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.029 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:06.029 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.034360+0000", 2024-09-18T04:39:06.029 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015809299999999999 2024-09-18T04:39:06.029 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:06.029 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:06.030 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:06.030 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.030 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.030 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4972.0:14 2.5 2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:head [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:06.030 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.949251+0000", 2024-09-18T04:39:06.030 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 13.976589314, 2024-09-18T04:39:06.030 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0098324480000000006, 2024-09-18T04:39:06.030 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:06.030 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:06.030 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:06.030 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4972", 2024-09-18T04:39:06.030 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2280251472", 2024-09-18T04:39:06.030 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 14 2024-09-18T04:39:06.030 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.031 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:06.031 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.031 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:06.031 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949251+0000", 2024-09-18T04:39:06.031 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:06.031 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.031 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.031 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:06.031 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949251+0000", 2024-09-18T04:39:06.031 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:06.031 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.031 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.031 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:06.031 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949250+0000", 2024-09-18T04:39:06.032 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:06.032 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.032 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.032 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:06.032 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949258+0000", 2024-09-18T04:39:06.032 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0590000000000003e-06 2024-09-18T04:39:06.032 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.032 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.032 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:06.032 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949269+0000", 2024-09-18T04:39:06.032 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0607e-05 2024-09-18T04:39:06.032 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.032 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.033 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:06.033 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949277+0000", 2024-09-18T04:39:06.033 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8269999999999998e-06 2024-09-18T04:39:06.033 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.033 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.033 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:06.033 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949394+0000", 2024-09-18T04:39:06.033 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000117113 2024-09-18T04:39:06.033 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.033 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.033 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:06.033 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949527+0000", 2024-09-18T04:39:06.033 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013288300000000001 2024-09-18T04:39:06.033 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.034 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.034 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:06.034 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.959084+0000", 2024-09-18T04:39:06.034 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0095569550000000007 2024-09-18T04:39:06.034 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:06.034 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:06.034 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:06.034 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.034 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.034 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4972.0:20 2.5 2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:06.034 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.949296+0000", 2024-09-18T04:39:06.034 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 13.976545008, 2024-09-18T04:39:06.034 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014709732, 2024-09-18T04:39:06.035 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:06.035 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:06.035 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:06.035 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4972", 2024-09-18T04:39:06.035 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2280251472", 2024-09-18T04:39:06.035 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 20 2024-09-18T04:39:06.035 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.035 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:06.035 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.035 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:06.035 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949296+0000", 2024-09-18T04:39:06.035 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:06.035 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.035 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.036 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:06.036 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949296+0000", 2024-09-18T04:39:06.036 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:06.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.036 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.036 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:06.036 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949295+0000", 2024-09-18T04:39:06.036 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:06.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.036 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.036 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:06.036 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949301+0000", 2024-09-18T04:39:06.036 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4829999999999996e-06 2024-09-18T04:39:06.037 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.037 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.037 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:06.037 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949317+0000", 2024-09-18T04:39:06.037 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5866999999999999e-05 2024-09-18T04:39:06.037 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.037 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.037 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:06.037 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949331+0000", 2024-09-18T04:39:06.037 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3485999999999999e-05 2024-09-18T04:39:06.037 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.037 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.037 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:06.037 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.959137+0000", 2024-09-18T04:39:06.038 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0098066520000000008 2024-09-18T04:39:06.038 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.038 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.038 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:06.038 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.959251+0000", 2024-09-18T04:39:06.038 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000113783 2024-09-18T04:39:06.038 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.038 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.038 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:06.038 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.964005+0000", 2024-09-18T04:39:06.038 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0047542080000000002 2024-09-18T04:39:06.038 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:06.038 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:06.038 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:06.039 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.039 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.039 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4970.0:37 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [write 1966080~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:06.039 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.516050+0000", 2024-09-18T04:39:06.039 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 5.4097901510000002, 2024-09-18T04:39:06.039 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0083751499999999996, 2024-09-18T04:39:06.039 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:06.039 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:06.039 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:06.039 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4970", 2024-09-18T04:39:06.040 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:06.040 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 37 2024-09-18T04:39:06.040 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.040 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:06.040 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.040 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:06.040 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.516050+0000", 2024-09-18T04:39:06.040 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:06.040 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.040 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.041 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:06.041 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.516050+0000", 2024-09-18T04:39:06.041 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:06.041 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.041 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.041 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:06.041 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.516050+0000", 2024-09-18T04:39:06.041 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:06.041 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.041 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.041 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:06.042 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.516150+0000", 2024-09-18T04:39:06.042 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000100159 2024-09-18T04:39:06.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.042 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.042 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:06.042 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.516157+0000", 2024-09-18T04:39:06.042 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7710000000000001e-06 2024-09-18T04:39:06.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.042 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.042 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:06.042 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.516163+0000", 2024-09-18T04:39:06.042 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.596e-06 2024-09-18T04:39:06.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.042 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.043 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:06.043 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.522975+0000", 2024-09-18T04:39:06.043 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0068118140000000002 2024-09-18T04:39:06.043 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.043 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.043 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:06.043 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.523028+0000", 2024-09-18T04:39:06.043 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2998999999999999e-05 2024-09-18T04:39:06.043 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.043 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.043 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:06.043 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.523107+0000", 2024-09-18T04:39:06.043 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9357000000000006e-05 2024-09-18T04:39:06.043 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.044 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.044 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:06.044 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.524167+0000", 2024-09-18T04:39:06.044 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0010592329999999999 2024-09-18T04:39:06.044 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.044 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.044 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:06.044 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.524369+0000", 2024-09-18T04:39:06.044 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00020209000000000001 2024-09-18T04:39:06.044 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.044 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.044 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:06.044 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.524402+0000", 2024-09-18T04:39:06.045 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3685e-05 2024-09-18T04:39:06.045 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:06.045 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:06.045 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:06.045 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.524426+0000", 2024-09-18T04:39:06.045 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3178000000000001e-05 2024-09-18T04:39:06.045 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:06.045 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:06.045 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:06.045 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:06.045 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:06.045 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:06.050 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:39:06.050 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:06.041+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:06.150 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:06.145+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:06.169 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:06.169 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:06.169 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:39:06.169 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:06.179 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:39:06.251 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:06.245+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:06.284 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:06.284 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:06.284 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:39:06.285 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:39:06.285 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:06.285 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T04:39:06.294 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:39:06.351 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:06.345+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:39:06.402 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:06.403 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:39:06.403 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:39:06.403 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:39:06.403 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.403 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:39:06.404 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:39:06.404 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 98.609430490999998, 2024-09-18T04:39:06.404 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017399218000000001, 2024-09-18T04:39:06.404 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:06.404 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:06.404 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:06.405 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:39:06.405 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:39:06.405 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 114 2024-09-18T04:39:06.405 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.405 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:06.406 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.406 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:06.406 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:39:06.406 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:06.406 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.406 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.407 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:06.407 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:39:06.407 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:06.407 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.407 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.408 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:06.408 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786900+0000", 2024-09-18T04:39:06.408 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:06.408 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.408 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.409 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:06.409 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786908+0000", 2024-09-18T04:39:06.409 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8199999999999997e-06 2024-09-18T04:39:06.409 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.409 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.409 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:06.410 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786925+0000", 2024-09-18T04:39:06.410 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6849000000000001e-05 2024-09-18T04:39:06.410 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.410 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.410 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:06.411 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786934+0000", 2024-09-18T04:39:06.411 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6299999999999993e-06 2024-09-18T04:39:06.411 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.411 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.411 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:06.411 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800926+0000", 2024-09-18T04:39:06.412 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013991632 2024-09-18T04:39:06.412 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.412 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.412 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:06.412 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801060+0000", 2024-09-18T04:39:06.413 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013378199999999999 2024-09-18T04:39:06.413 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.413 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.413 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:06.413 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801166+0000", 2024-09-18T04:39:06.413 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010607099999999999 2024-09-18T04:39:06.414 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.414 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.414 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:06.414 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803026+0000", 2024-09-18T04:39:06.414 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001860469 2024-09-18T04:39:06.414 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.415 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.415 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:06.415 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804245+0000", 2024-09-18T04:39:06.415 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0012185900000000001 2024-09-18T04:39:06.415 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.415 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.416 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:06.416 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804278+0000", 2024-09-18T04:39:06.416 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3272000000000003e-05 2024-09-18T04:39:06.416 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.416 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.417 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:06.417 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804300+0000", 2024-09-18T04:39:06.417 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2266e-05 2024-09-18T04:39:06.417 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:06.417 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:06.417 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:06.418 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.418 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.418 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:39:06.418 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:39:06.419 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 98.608517593000002, 2024-09-18T04:39:06.419 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017521662, 2024-09-18T04:39:06.419 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:06.419 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:06.419 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:06.420 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:39:06.420 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:39:06.420 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 116 2024-09-18T04:39:06.420 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.420 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:06.421 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.421 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:06.421 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:39:06.421 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:06.421 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.422 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.422 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:06.422 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:39:06.422 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:06.422 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.422 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.423 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:06.423 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787813+0000", 2024-09-18T04:39:06.423 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:06.423 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.423 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.424 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:06.424 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787824+0000", 2024-09-18T04:39:06.424 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1157e-05 2024-09-18T04:39:06.424 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.424 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.424 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:06.425 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787840+0000", 2024-09-18T04:39:06.425 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6269e-05 2024-09-18T04:39:06.425 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.425 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.425 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:06.426 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787853+0000", 2024-09-18T04:39:06.426 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3156000000000001e-05 2024-09-18T04:39:06.426 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.426 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.426 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:06.426 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803110+0000", 2024-09-18T04:39:06.427 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015256736 2024-09-18T04:39:06.427 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.427 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.427 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:06.427 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803197+0000", 2024-09-18T04:39:06.427 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7195999999999997e-05 2024-09-18T04:39:06.428 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.428 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.428 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:06.428 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803308+0000", 2024-09-18T04:39:06.428 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111005 2024-09-18T04:39:06.429 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.429 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.429 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:06.429 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804316+0000", 2024-09-18T04:39:06.429 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010073949999999999 2024-09-18T04:39:06.429 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.430 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.430 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:06.430 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805232+0000", 2024-09-18T04:39:06.430 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091636300000000001 2024-09-18T04:39:06.430 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.431 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.431 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:06.431 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805277+0000", 2024-09-18T04:39:06.431 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4864000000000003e-05 2024-09-18T04:39:06.431 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.431 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.431 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:06.432 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805336+0000", 2024-09-18T04:39:06.432 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8968000000000002e-05 2024-09-18T04:39:06.432 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:06.432 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:06.432 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:06.433 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.433 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.433 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:39:06.433 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:39:06.433 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 98.608465538000004, 2024-09-18T04:39:06.433 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017983453, 2024-09-18T04:39:06.434 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:06.434 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:06.434 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:06.434 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:39:06.434 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:39:06.435 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 117 2024-09-18T04:39:06.435 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.435 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:06.435 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.435 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:06.435 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:39:06.436 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:06.436 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.436 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.436 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:06.436 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:39:06.437 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:06.437 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.437 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.437 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:06.437 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787865+0000", 2024-09-18T04:39:06.438 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:06.438 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.438 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.438 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:06.438 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787889+0000", 2024-09-18T04:39:06.438 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4244999999999999e-05 2024-09-18T04:39:06.439 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.439 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.439 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:06.439 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787905+0000", 2024-09-18T04:39:06.439 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5739999999999998e-05 2024-09-18T04:39:06.439 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.440 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.440 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:06.440 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787918+0000", 2024-09-18T04:39:06.440 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.252e-05 2024-09-18T04:39:06.440 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.441 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.441 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:06.441 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803693+0000", 2024-09-18T04:39:06.441 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015774872999999998 2024-09-18T04:39:06.441 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.442 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.442 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:06.442 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803775+0000", 2024-09-18T04:39:06.442 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2761000000000003e-05 2024-09-18T04:39:06.442 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.442 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.443 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:06.443 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803887+0000", 2024-09-18T04:39:06.443 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111708 2024-09-18T04:39:06.444 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.444 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.444 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:06.444 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804823+0000", 2024-09-18T04:39:06.444 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00093625100000000003 2024-09-18T04:39:06.444 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.444 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.444 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:06.444 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805804+0000", 2024-09-18T04:39:06.444 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00098114699999999992 2024-09-18T04:39:06.445 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.445 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.445 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:06.445 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805831+0000", 2024-09-18T04:39:06.445 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6401999999999999e-05 2024-09-18T04:39:06.445 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.445 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.445 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:06.445 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805850+0000", 2024-09-18T04:39:06.445 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8862e-05 2024-09-18T04:39:06.445 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:06.446 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:06.446 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:06.446 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.446 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.446 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:36 2.7 2:fe1b02ad:::rbd_data.125fc09df6d1.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:06.446 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:39:06.446 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 24.130609428, 2024-09-18T04:39:06.446 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.040387804999999999, 2024-09-18T04:39:06.446 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:06.446 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:06.446 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:06.446 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:06.447 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:06.447 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 36 2024-09-18T04:39:06.447 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.447 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:06.447 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.447 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:06.447 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:39:06.447 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:06.447 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.448 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.448 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:06.448 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:39:06.448 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:06.448 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.448 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.448 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:06.448 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265720+0000", 2024-09-18T04:39:06.448 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:06.448 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.448 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.449 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:06.449 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272590+0000", 2024-09-18T04:39:06.449 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.006870508 2024-09-18T04:39:06.449 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.449 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.449 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:06.449 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272623+0000", 2024-09-18T04:39:06.449 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2716000000000003e-05 2024-09-18T04:39:06.449 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.449 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.449 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:06.450 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272637+0000", 2024-09-18T04:39:06.450 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4175999999999999e-05 2024-09-18T04:39:06.450 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.450 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.450 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:06.450 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272779+0000", 2024-09-18T04:39:06.450 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000141392 2024-09-18T04:39:06.451 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.451 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.451 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:06.451 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273031+0000", 2024-09-18T04:39:06.451 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00025249099999999998 2024-09-18T04:39:06.451 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.451 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.451 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:39:06.451 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273219+0000", 2024-09-18T04:39:06.451 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000187531 2024-09-18T04:39:06.451 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.451 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.452 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:06.452 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.302806+0000", 2024-09-18T04:39:06.452 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.029587553999999999 2024-09-18T04:39:06.452 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.452 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.452 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:06.452 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306004+0000", 2024-09-18T04:39:06.452 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0031982629999999998 2024-09-18T04:39:06.452 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.452 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.452 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:06.452 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306071+0000", 2024-09-18T04:39:06.453 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6829999999999995e-05 2024-09-18T04:39:06.453 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.453 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.453 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:06.453 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306110+0000", 2024-09-18T04:39:06.453 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8874000000000001e-05 2024-09-18T04:39:06.453 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:06.453 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:06.453 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:06.453 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.456 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.456 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:38 2.0 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:06.456 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:39:06.456 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 24.123655890999999, 2024-09-18T04:39:06.456 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.033754348000000003, 2024-09-18T04:39:06.456 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:06.457 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:06.457 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:06.457 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:06.457 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:06.457 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 38 2024-09-18T04:39:06.457 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.457 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:06.457 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.457 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:06.457 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:39:06.457 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:06.457 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.458 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.458 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:06.458 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:39:06.458 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:06.458 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.458 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.458 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:06.458 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272673+0000", 2024-09-18T04:39:06.458 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:06.458 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.458 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.458 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:06.459 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277166+0000", 2024-09-18T04:39:06.459 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044922249999999999 2024-09-18T04:39:06.459 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.459 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.459 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:06.459 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277189+0000", 2024-09-18T04:39:06.459 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3283000000000001e-05 2024-09-18T04:39:06.459 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.459 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.459 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:06.459 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277205+0000", 2024-09-18T04:39:06.459 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6305000000000001e-05 2024-09-18T04:39:06.460 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.460 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.460 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:06.460 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277290+0000", 2024-09-18T04:39:06.460 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5135000000000004e-05 2024-09-18T04:39:06.460 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.460 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.460 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:06.460 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277531+0000", 2024-09-18T04:39:06.460 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00024014699999999999 2024-09-18T04:39:06.460 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.461 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.461 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:06.461 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277793+0000", 2024-09-18T04:39:06.461 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00026224499999999998 2024-09-18T04:39:06.461 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.461 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.461 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:06.461 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303990+0000", 2024-09-18T04:39:06.461 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.026197295999999998 2024-09-18T04:39:06.461 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.461 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.461 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:06.462 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306340+0000", 2024-09-18T04:39:06.462 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.002349524 2024-09-18T04:39:06.462 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.462 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.462 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:06.462 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306394+0000", 2024-09-18T04:39:06.462 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3934999999999998e-05 2024-09-18T04:39:06.462 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.462 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.462 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:06.462 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306430+0000", 2024-09-18T04:39:06.463 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6690000000000003e-05 2024-09-18T04:39:06.463 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:06.463 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:06.463 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:06.463 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.463 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.463 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:39 2.7 2:e9c909c5:::rbd_data.125fc09df6d1.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:06.463 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:39:06.463 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 24.119096286000001, 2024-09-18T04:39:06.463 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.029110673, 2024-09-18T04:39:06.463 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:06.464 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:06.464 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:06.464 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:06.464 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:06.464 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 39 2024-09-18T04:39:06.464 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.464 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:06.464 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.464 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:06.464 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:39:06.464 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:06.465 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.465 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.465 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:06.465 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:39:06.465 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:06.465 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.465 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.465 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:06.465 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277234+0000", 2024-09-18T04:39:06.465 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:06.465 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.465 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.466 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:06.466 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281351+0000", 2024-09-18T04:39:06.466 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0041170920000000001 2024-09-18T04:39:06.466 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.466 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.466 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:06.466 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281375+0000", 2024-09-18T04:39:06.466 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.446e-05 2024-09-18T04:39:06.466 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.466 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.466 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:06.467 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281390+0000", 2024-09-18T04:39:06.467 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4309e-05 2024-09-18T04:39:06.467 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.467 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.467 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:06.467 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.282306+0000", 2024-09-18T04:39:06.467 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091662000000000004 2024-09-18T04:39:06.467 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.467 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.467 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:06.467 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.282613+0000", 2024-09-18T04:39:06.467 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00030663799999999997 2024-09-18T04:39:06.468 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.468 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.468 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:39:06.468 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283001+0000", 2024-09-18T04:39:06.468 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00038801599999999998 2024-09-18T04:39:06.468 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.468 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.469 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:06.469 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306239+0000", 2024-09-18T04:39:06.469 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023238466999999999 2024-09-18T04:39:06.469 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.469 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.469 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:06.469 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306259+0000", 2024-09-18T04:39:06.469 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9837999999999999e-05 2024-09-18T04:39:06.469 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.469 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.469 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:06.470 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306299+0000", 2024-09-18T04:39:06.470 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0194000000000002e-05 2024-09-18T04:39:06.470 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.470 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.470 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:06.470 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306346+0000", 2024-09-18T04:39:06.470 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6845000000000003e-05 2024-09-18T04:39:06.470 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:06.470 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:06.470 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:06.471 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.471 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.471 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:46 2.6 e51/17 2:7ba25245:::rbd_data.125fc09df6d1.0000000000000006:head v 51'2, mlcod=0'0)", 2024-09-18T04:39:06.471 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:39:06.471 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 24.083175153999999, 2024-09-18T04:39:06.471 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019843098, 2024-09-18T04:39:06.471 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:06.472 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:06.472 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:06.472 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.472 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:06.472 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:39:06.472 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:06.472 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.472 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.472 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:06.472 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:39:06.473 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:06.473 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.473 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.473 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:06.473 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313156+0000", 2024-09-18T04:39:06.473 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:06.473 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.473 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.473 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:06.473 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320144+0000", 2024-09-18T04:39:06.473 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.006988345 2024-09-18T04:39:06.473 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.474 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.474 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:06.474 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320175+0000", 2024-09-18T04:39:06.474 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.0861999999999997e-05 2024-09-18T04:39:06.474 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.474 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.474 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:06.474 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320188+0000", 2024-09-18T04:39:06.474 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2714999999999999e-05 2024-09-18T04:39:06.474 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.474 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.474 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:06.475 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320267+0000", 2024-09-18T04:39:06.475 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9586999999999995e-05 2024-09-18T04:39:06.475 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.475 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.475 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:06.475 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320311+0000", 2024-09-18T04:39:06.475 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3529000000000003e-05 2024-09-18T04:39:06.475 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.475 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.475 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:06.475 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.332949+0000", 2024-09-18T04:39:06.475 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012638267999999999 2024-09-18T04:39:06.476 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.476 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.476 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:06.476 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.333000+0000", 2024-09-18T04:39:06.476 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0736999999999999e-05 2024-09-18T04:39:06.476 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:06.476 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:06.476 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:06.476 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.476 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.476 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:57 2.0 2:0cdd070e:::rbd_data.125fc09df6d1.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:06.477 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:39:06.477 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 24.047317579000001, 2024-09-18T04:39:06.477 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.033329974999999998, 2024-09-18T04:39:06.477 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:06.477 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:06.477 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:06.477 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:06.477 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:06.477 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 57 2024-09-18T04:39:06.477 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.477 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:06.477 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.478 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:06.478 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:39:06.478 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:06.478 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.478 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.478 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:06.478 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:39:06.478 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:06.478 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.478 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.478 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:06.479 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349013+0000", 2024-09-18T04:39:06.479 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:06.479 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.479 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.479 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:06.479 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356138+0000", 2024-09-18T04:39:06.479 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071258010000000002 2024-09-18T04:39:06.479 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.479 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.479 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:06.479 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356168+0000", 2024-09-18T04:39:06.479 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9679000000000002e-05 2024-09-18T04:39:06.480 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.480 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.480 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:06.480 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356181+0000", 2024-09-18T04:39:06.480 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3164e-05 2024-09-18T04:39:06.480 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.480 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.480 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:06.480 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356254+0000", 2024-09-18T04:39:06.480 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3182999999999994e-05 2024-09-18T04:39:06.480 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.481 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.481 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:06.481 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356453+0000", 2024-09-18T04:39:06.481 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019876599999999999 2024-09-18T04:39:06.481 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.481 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.481 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:06.481 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356627+0000", 2024-09-18T04:39:06.481 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017340199999999999 2024-09-18T04:39:06.481 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.481 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.481 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:06.482 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.369200+0000", 2024-09-18T04:39:06.482 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012573302 2024-09-18T04:39:06.482 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.482 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.482 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:06.482 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382262+0000", 2024-09-18T04:39:06.482 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013062549 2024-09-18T04:39:06.482 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.482 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.482 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:06.482 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382310+0000", 2024-09-18T04:39:06.482 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7366e-05 2024-09-18T04:39:06.483 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.483 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.483 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:06.483 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382344+0000", 2024-09-18T04:39:06.483 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4319999999999997e-05 2024-09-18T04:39:06.483 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:06.483 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:06.483 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:06.483 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.483 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.483 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:59 2.0 2:19c52cde:::rbd_data.125fc09df6d1.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:06.483 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:39:06.484 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 24.040126413999999, 2024-09-18T04:39:06.484 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.027137247999999999, 2024-09-18T04:39:06.484 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:06.484 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:06.484 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:06.485 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:06.485 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:06.486 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 59 2024-09-18T04:39:06.486 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.486 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:06.486 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.486 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:06.486 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:39:06.486 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:06.486 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.486 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.486 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:06.486 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:39:06.486 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:06.487 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.487 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.487 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:06.487 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356204+0000", 2024-09-18T04:39:06.487 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:06.487 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.487 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.487 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:06.487 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362661+0000", 2024-09-18T04:39:06.487 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0064565129999999997 2024-09-18T04:39:06.487 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.487 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.488 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:06.488 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362684+0000", 2024-09-18T04:39:06.488 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3688000000000001e-05 2024-09-18T04:39:06.488 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.488 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.488 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:06.488 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362699+0000", 2024-09-18T04:39:06.488 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4798e-05 2024-09-18T04:39:06.488 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.488 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.488 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:06.488 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363016+0000", 2024-09-18T04:39:06.489 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031737300000000003 2024-09-18T04:39:06.489 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.489 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.489 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:06.489 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363201+0000", 2024-09-18T04:39:06.489 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000184131 2024-09-18T04:39:06.489 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.489 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.489 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:06.489 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363343+0000", 2024-09-18T04:39:06.489 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014236699999999999 2024-09-18T04:39:06.489 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.490 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.490 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:06.490 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.372189+0000", 2024-09-18T04:39:06.490 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0088464819999999993 2024-09-18T04:39:06.490 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.490 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.490 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:06.490 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383269+0000", 2024-09-18T04:39:06.491 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011079594 2024-09-18T04:39:06.491 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.491 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.491 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:06.491 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383314+0000", 2024-09-18T04:39:06.491 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4932000000000003e-05 2024-09-18T04:39:06.491 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.492 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.492 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:06.492 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383343+0000", 2024-09-18T04:39:06.492 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8600999999999999e-05 2024-09-18T04:39:06.492 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:06.492 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:06.492 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:06.492 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.492 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.492 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:40 2.6 e51/17 2:784d6cd1:ns::rbd_data.133e813ab5bc.0000000000000005:head v 51'7, mlcod=51'5)", 2024-09-18T04:39:06.492 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:39:06.492 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 23.843876680000001, 2024-09-18T04:39:06.493 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020497496, 2024-09-18T04:39:06.493 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:06.493 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:06.493 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:06.493 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.493 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:06.493 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:39:06.493 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:06.493 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.493 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.493 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:06.493 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:39:06.494 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:06.494 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.494 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.494 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:06.494 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552454+0000", 2024-09-18T04:39:06.494 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:06.494 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.494 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.494 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:06.494 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556448+0000", 2024-09-18T04:39:06.494 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0039940990000000001 2024-09-18T04:39:06.494 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.495 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.495 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:06.495 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556481+0000", 2024-09-18T04:39:06.495 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3778000000000001e-05 2024-09-18T04:39:06.495 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.495 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.495 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:06.495 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556493+0000", 2024-09-18T04:39:06.495 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1622e-05 2024-09-18T04:39:06.495 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.495 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.495 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:06.496 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556556+0000", 2024-09-18T04:39:06.496 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2582999999999994e-05 2024-09-18T04:39:06.496 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.496 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.496 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:06.496 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556592+0000", 2024-09-18T04:39:06.496 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5880999999999997e-05 2024-09-18T04:39:06.496 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.496 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.496 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:06.496 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.572786+0000", 2024-09-18T04:39:06.497 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016194571000000001 2024-09-18T04:39:06.497 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.497 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.497 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:06.497 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.572953+0000", 2024-09-18T04:39:06.497 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000166436 2024-09-18T04:39:06.497 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:06.497 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:06.497 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:06.497 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.497 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.497 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:45 2.0 2:0fbfa3dc:ns::rbd_data.133e813ab5bc.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:06.498 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:39:06.498 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 23.832662380999999, 2024-09-18T04:39:06.498 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.036558168000000002, 2024-09-18T04:39:06.498 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:06.498 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:06.498 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:06.498 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:06.498 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:06.498 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 45 2024-09-18T04:39:06.498 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.498 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:06.498 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.499 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:06.499 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:39:06.499 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:06.499 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.499 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.499 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:06.499 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:39:06.499 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:06.499 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.499 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.499 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:06.500 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:39:06.500 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:06.500 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.500 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.500 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:06.500 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578594+0000", 2024-09-18T04:39:06.500 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014927147999999999 2024-09-18T04:39:06.500 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.500 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.500 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:06.500 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578612+0000", 2024-09-18T04:39:06.501 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7623999999999999e-05 2024-09-18T04:39:06.501 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.501 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.501 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:06.501 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578620+0000", 2024-09-18T04:39:06.501 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9070000000000007e-06 2024-09-18T04:39:06.501 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.501 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.501 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:06.501 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578702+0000", 2024-09-18T04:39:06.501 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2349999999999996e-05 2024-09-18T04:39:06.501 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.502 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.502 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:06.502 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.579028+0000", 2024-09-18T04:39:06.502 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032622899999999998 2024-09-18T04:39:06.502 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.502 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.502 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:06.502 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.579343+0000", 2024-09-18T04:39:06.502 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031448899999999999 2024-09-18T04:39:06.502 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.502 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.502 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:06.503 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.590078+0000", 2024-09-18T04:39:06.503 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010734946 2024-09-18T04:39:06.503 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.503 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.503 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:06.503 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600089+0000", 2024-09-18T04:39:06.503 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010011195000000001 2024-09-18T04:39:06.503 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.503 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.503 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:06.503 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600177+0000", 2024-09-18T04:39:06.503 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8425999999999997e-05 2024-09-18T04:39:06.504 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.504 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.504 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:06.504 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600228+0000", 2024-09-18T04:39:06.504 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0216999999999997e-05 2024-09-18T04:39:06.504 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:06.504 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:06.504 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:06.504 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.504 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.504 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:54 2.0 2:04f17ef6:ns::rbd_data.133e813ab5bc.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:06.505 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:39:06.505 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 23.811090543999999, 2024-09-18T04:39:06.505 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.024612999, 2024-09-18T04:39:06.505 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:06.505 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:06.505 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:06.505 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:06.505 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:06.505 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 54 2024-09-18T04:39:06.505 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.505 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:06.505 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.506 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:06.506 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:39:06.506 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:06.506 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.506 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.506 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:06.506 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:39:06.506 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:06.506 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.506 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.506 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:06.506 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585239+0000", 2024-09-18T04:39:06.507 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:06.507 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.507 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.507 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:06.507 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593702+0000", 2024-09-18T04:39:06.507 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0084632990000000005 2024-09-18T04:39:06.507 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.507 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.507 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:06.508 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593719+0000", 2024-09-18T04:39:06.508 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.664e-05 2024-09-18T04:39:06.508 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.508 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.508 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:06.508 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593727+0000", 2024-09-18T04:39:06.508 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6839999999999996e-06 2024-09-18T04:39:06.508 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.508 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.508 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:06.508 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593767+0000", 2024-09-18T04:39:06.509 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0330999999999997e-05 2024-09-18T04:39:06.509 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.509 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.509 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:06.509 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593905+0000", 2024-09-18T04:39:06.509 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013754800000000001 2024-09-18T04:39:06.509 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.509 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.509 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:06.509 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.594013+0000", 2024-09-18T04:39:06.509 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000108805 2024-09-18T04:39:06.509 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.510 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.510 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:06.510 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604678+0000", 2024-09-18T04:39:06.510 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010664391 2024-09-18T04:39:06.510 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.510 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.510 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:06.510 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609662+0000", 2024-09-18T04:39:06.510 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0049846100000000004 2024-09-18T04:39:06.510 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.510 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.510 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:06.511 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609788+0000", 2024-09-18T04:39:06.511 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012518800000000001 2024-09-18T04:39:06.511 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.511 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.511 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:06.511 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609854+0000", 2024-09-18T04:39:06.511 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6556000000000004e-05 2024-09-18T04:39:06.511 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:06.511 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:06.511 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:06.511 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.511 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.512 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:58 2.7 2:fc7c6cd3:ns::rbd_data.133e813ab5bc.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:06.512 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:39:06.512 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 23.798248115, 2024-09-18T04:39:06.512 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.027690566999999999, 2024-09-18T04:39:06.512 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:06.512 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:06.512 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:06.512 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:06.512 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:06.512 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 58 2024-09-18T04:39:06.513 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.513 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:06.513 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.513 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:06.513 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:39:06.513 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:06.513 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.513 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.513 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:06.513 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:39:06.513 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:06.513 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.514 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.514 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:06.514 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598082+0000", 2024-09-18T04:39:06.514 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:06.514 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.514 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.514 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:06.514 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604832+0000", 2024-09-18T04:39:06.514 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0067504510000000002 2024-09-18T04:39:06.514 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.514 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.515 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:06.515 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604852+0000", 2024-09-18T04:39:06.515 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0228999999999999e-05 2024-09-18T04:39:06.515 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.515 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.515 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:06.515 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604870+0000", 2024-09-18T04:39:06.515 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7952e-05 2024-09-18T04:39:06.515 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.515 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.515 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:06.516 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604936+0000", 2024-09-18T04:39:06.516 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5431000000000004e-05 2024-09-18T04:39:06.516 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.516 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.516 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:06.516 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605085+0000", 2024-09-18T04:39:06.516 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014910300000000001 2024-09-18T04:39:06.516 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.516 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.516 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:39:06.516 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605205+0000", 2024-09-18T04:39:06.517 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000119887 2024-09-18T04:39:06.517 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.517 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.517 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:06.517 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625586+0000", 2024-09-18T04:39:06.517 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020381399000000001 2024-09-18T04:39:06.517 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.517 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.517 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:06.517 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625628+0000", 2024-09-18T04:39:06.517 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1873999999999999e-05 2024-09-18T04:39:06.517 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.518 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.518 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:06.518 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625722+0000", 2024-09-18T04:39:06.518 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3787000000000002e-05 2024-09-18T04:39:06.518 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.518 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.518 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:06.518 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625774+0000", 2024-09-18T04:39:06.518 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2503999999999997e-05 2024-09-18T04:39:06.518 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:06.518 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:06.519 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:06.519 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.519 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.519 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:56 2.6 e51/17 2:72453c3e:ns::rbd_data.133e813ab5bc.000000000000000d:head v 51'10, mlcod=51'8)", 2024-09-18T04:39:06.519 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:39:06.519 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 23.791051007, 2024-09-18T04:39:06.519 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019766552999999999, 2024-09-18T04:39:06.519 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:06.519 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:06.519 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:06.519 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.519 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:06.520 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:39:06.520 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:06.520 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.520 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.520 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:06.520 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:39:06.520 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:06.520 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.520 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.520 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:06.520 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605279+0000", 2024-09-18T04:39:06.521 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:06.521 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.521 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.521 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:06.521 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609417+0000", 2024-09-18T04:39:06.521 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0041378059999999999 2024-09-18T04:39:06.521 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.521 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.521 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:06.521 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609443+0000", 2024-09-18T04:39:06.521 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6098e-05 2024-09-18T04:39:06.521 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.522 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.522 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:06.522 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609458+0000", 2024-09-18T04:39:06.522 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4671000000000001e-05 2024-09-18T04:39:06.522 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.522 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.522 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:06.522 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609553+0000", 2024-09-18T04:39:06.522 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5265000000000004e-05 2024-09-18T04:39:06.522 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.522 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.523 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:06.523 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609583+0000", 2024-09-18T04:39:06.523 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9839e-05 2024-09-18T04:39:06.523 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.523 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.523 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:06.523 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625004+0000", 2024-09-18T04:39:06.523 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015420639999999999 2024-09-18T04:39:06.523 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.523 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.523 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:06.523 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625047+0000", 2024-09-18T04:39:06.524 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3665999999999998e-05 2024-09-18T04:39:06.524 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:06.524 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:06.524 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:06.524 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.524 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.524 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:99 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:06.524 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.582718+0000", 2024-09-18T04:39:06.524 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 9.8136140980000004, 2024-09-18T04:39:06.524 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019362575, 2024-09-18T04:39:06.525 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:06.525 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:06.525 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:06.525 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:06.525 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:06.525 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 99 2024-09-18T04:39:06.525 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.525 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:06.525 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.525 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:06.526 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582718+0000", 2024-09-18T04:39:06.526 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:06.526 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.526 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.526 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:06.526 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582718+0000", 2024-09-18T04:39:06.526 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:06.526 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.526 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.526 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:06.526 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582717+0000", 2024-09-18T04:39:06.526 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:06.526 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.527 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.527 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:06.527 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582720+0000", 2024-09-18T04:39:06.527 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2239999999999998e-06 2024-09-18T04:39:06.527 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.527 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.527 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:06.527 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582723+0000", 2024-09-18T04:39:06.527 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.258e-06 2024-09-18T04:39:06.527 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.527 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.527 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:06.527 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582726+0000", 2024-09-18T04:39:06.527 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8870000000000001e-06 2024-09-18T04:39:06.528 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.528 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.528 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:06.528 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.592018+0000", 2024-09-18T04:39:06.528 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0092926420000000003 2024-09-18T04:39:06.528 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.528 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.528 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:06.528 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.592214+0000", 2024-09-18T04:39:06.528 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000195267 2024-09-18T04:39:06.528 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.528 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.528 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:06.529 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.592913+0000", 2024-09-18T04:39:06.529 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00069924600000000005 2024-09-18T04:39:06.529 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.529 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.529 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:06.529 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597493+0000", 2024-09-18T04:39:06.529 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045797629999999997 2024-09-18T04:39:06.529 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.529 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.529 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:06.529 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.602009+0000", 2024-09-18T04:39:06.529 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045159559999999998 2024-09-18T04:39:06.529 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.530 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.530 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:06.530 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.602042+0000", 2024-09-18T04:39:06.530 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3084999999999999e-05 2024-09-18T04:39:06.530 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.530 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.530 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:06.530 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.602080+0000", 2024-09-18T04:39:06.530 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8600000000000003e-05 2024-09-18T04:39:06.530 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:06.530 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:06.530 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:06.530 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.530 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.531 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:104 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:06.531 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.583165+0000", 2024-09-18T04:39:06.531 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 9.8131667999999994, 2024-09-18T04:39:06.531 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023679603, 2024-09-18T04:39:06.531 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:06.531 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:06.531 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:06.531 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:06.531 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:06.531 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 104 2024-09-18T04:39:06.531 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.531 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:06.531 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.531 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:06.532 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583165+0000", 2024-09-18T04:39:06.532 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:06.532 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.532 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.532 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:06.532 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583165+0000", 2024-09-18T04:39:06.532 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:06.532 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.532 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.532 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:06.532 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583164+0000", 2024-09-18T04:39:06.532 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:06.532 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.533 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.533 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:06.533 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583167+0000", 2024-09-18T04:39:06.533 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9210000000000001e-06 2024-09-18T04:39:06.533 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.533 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.533 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:06.533 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583171+0000", 2024-09-18T04:39:06.533 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9759999999999997e-06 2024-09-18T04:39:06.533 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.533 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.533 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:06.533 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583174+0000", 2024-09-18T04:39:06.534 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9960000000000002e-06 2024-09-18T04:39:06.534 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.534 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.534 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:06.534 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.594373+0000", 2024-09-18T04:39:06.534 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011198665999999999 2024-09-18T04:39:06.534 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.534 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.534 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:06.534 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.594434+0000", 2024-09-18T04:39:06.534 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.1446999999999996e-05 2024-09-18T04:39:06.534 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.534 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.534 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:06.535 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.594692+0000", 2024-09-18T04:39:06.535 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00025756599999999998 2024-09-18T04:39:06.535 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.535 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.535 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:06.535 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599302+0000", 2024-09-18T04:39:06.535 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0046099640000000002 2024-09-18T04:39:06.535 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.535 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.535 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:06.535 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606710+0000", 2024-09-18T04:39:06.535 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0074077739999999998 2024-09-18T04:39:06.535 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.536 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.536 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:06.536 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606798+0000", 2024-09-18T04:39:06.536 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8133999999999993e-05 2024-09-18T04:39:06.536 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.536 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.536 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:06.536 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606845+0000", 2024-09-18T04:39:06.536 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6653000000000002e-05 2024-09-18T04:39:06.536 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:06.536 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:06.536 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:06.536 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.537 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.537 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:105 2.0 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:06.537 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.583193+0000", 2024-09-18T04:39:06.537 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 9.8131391810000004, 2024-09-18T04:39:06.537 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023713649, 2024-09-18T04:39:06.537 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:06.537 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:06.537 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:06.537 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:06.537 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:06.537 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 105 2024-09-18T04:39:06.537 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.537 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:06.538 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.538 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:06.538 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583193+0000", 2024-09-18T04:39:06.538 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:06.538 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.538 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.538 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:06.538 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583193+0000", 2024-09-18T04:39:06.538 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:06.538 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.538 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.538 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:06.538 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583192+0000", 2024-09-18T04:39:06.538 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:06.539 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.539 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.539 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:06.539 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583194+0000", 2024-09-18T04:39:06.539 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.305e-06 2024-09-18T04:39:06.539 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.539 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.539 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:06.539 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583198+0000", 2024-09-18T04:39:06.539 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9369999999999997e-06 2024-09-18T04:39:06.539 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.539 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.539 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:06.539 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583202+0000", 2024-09-18T04:39:06.540 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6480000000000001e-06 2024-09-18T04:39:06.540 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.540 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.540 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:06.540 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.595304+0000", 2024-09-18T04:39:06.540 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012102063999999999 2024-09-18T04:39:06.540 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.540 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.540 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:06.540 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.595390+0000", 2024-09-18T04:39:06.540 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5493000000000003e-05 2024-09-18T04:39:06.540 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.540 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.541 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:06.541 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.595509+0000", 2024-09-18T04:39:06.541 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011966199999999999 2024-09-18T04:39:06.541 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.541 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.541 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:06.541 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599358+0000", 2024-09-18T04:39:06.541 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0038486689999999999 2024-09-18T04:39:06.541 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.541 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.541 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:06.541 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606859+0000", 2024-09-18T04:39:06.541 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0075010789999999999 2024-09-18T04:39:06.541 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.542 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.542 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:06.542 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606887+0000", 2024-09-18T04:39:06.542 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7773e-05 2024-09-18T04:39:06.542 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.542 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.542 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:06.542 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606906+0000", 2024-09-18T04:39:06.542 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9417000000000001e-05 2024-09-18T04:39:06.542 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:06.542 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:06.542 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:06.542 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.543 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.543 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:106 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:06.543 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.584425+0000", 2024-09-18T04:39:06.543 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 9.8119068849999991, 2024-09-18T04:39:06.543 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.022529293999999998, 2024-09-18T04:39:06.543 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:06.543 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:06.543 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:06.543 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:06.543 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:06.543 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 106 2024-09-18T04:39:06.543 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.543 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:06.543 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.544 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:06.544 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584425+0000", 2024-09-18T04:39:06.544 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:06.544 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.544 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.544 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:06.544 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584425+0000", 2024-09-18T04:39:06.544 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:06.544 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.544 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.544 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:06.544 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584424+0000", 2024-09-18T04:39:06.544 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:06.545 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.545 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.545 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:06.545 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584430+0000", 2024-09-18T04:39:06.545 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6389999999999997e-06 2024-09-18T04:39:06.545 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.545 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.545 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:06.545 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584456+0000", 2024-09-18T04:39:06.545 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6188999999999999e-05 2024-09-18T04:39:06.545 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.545 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.545 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:06.545 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584463+0000", 2024-09-18T04:39:06.546 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7340000000000002e-06 2024-09-18T04:39:06.546 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.546 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.546 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:06.546 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597537+0000", 2024-09-18T04:39:06.546 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013074161000000001 2024-09-18T04:39:06.546 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.546 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.546 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:06.546 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597598+0000", 2024-09-18T04:39:06.546 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0930000000000001e-05 2024-09-18T04:39:06.546 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.546 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.546 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:06.547 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597922+0000", 2024-09-18T04:39:06.547 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032473000000000002 2024-09-18T04:39:06.547 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.547 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.547 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:06.547 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.601103+0000", 2024-09-18T04:39:06.547 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0031809849999999999 2024-09-18T04:39:06.547 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.547 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.547 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:06.547 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606912+0000", 2024-09-18T04:39:06.547 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058083550000000003 2024-09-18T04:39:06.547 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.547 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.548 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:06.548 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606935+0000", 2024-09-18T04:39:06.548 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2759999999999999e-05 2024-09-18T04:39:06.548 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.548 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.548 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:06.548 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606954+0000", 2024-09-18T04:39:06.548 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9602999999999998e-05 2024-09-18T04:39:06.548 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:06.548 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:06.548 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:06.548 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.548 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.549 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:109 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:06.549 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.585231+0000", 2024-09-18T04:39:06.549 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 9.8111002650000003, 2024-09-18T04:39:06.549 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.021767496000000001, 2024-09-18T04:39:06.549 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:06.549 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:06.549 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:06.549 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:06.549 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:06.549 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 109 2024-09-18T04:39:06.549 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.549 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:06.549 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.549 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:06.550 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585231+0000", 2024-09-18T04:39:06.550 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:06.550 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.550 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.550 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:06.550 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585231+0000", 2024-09-18T04:39:06.550 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:06.550 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.550 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.550 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:06.551 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585230+0000", 2024-09-18T04:39:06.551 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:06.551 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.551 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.551 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:06.551 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585237+0000", 2024-09-18T04:39:06.551 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.618e-06 2024-09-18T04:39:06.551 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.551 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.551 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:06.551 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585256+0000", 2024-09-18T04:39:06.551 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9048000000000001e-05 2024-09-18T04:39:06.551 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.552 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.552 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:06.552 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585274+0000", 2024-09-18T04:39:06.552 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8153000000000001e-05 2024-09-18T04:39:06.552 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.552 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.552 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:06.552 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.598284+0000", 2024-09-18T04:39:06.552 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013009682999999999 2024-09-18T04:39:06.552 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.552 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.552 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:06.552 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.598383+0000", 2024-09-18T04:39:06.552 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.8722999999999997e-05 2024-09-18T04:39:06.553 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.553 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.553 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:06.553 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.598829+0000", 2024-09-18T04:39:06.553 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00044623999999999998 2024-09-18T04:39:06.553 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.553 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.553 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:06.553 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.601149+0000", 2024-09-18T04:39:06.553 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0023204150000000002 2024-09-18T04:39:06.553 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.553 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.553 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:06.553 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606960+0000", 2024-09-18T04:39:06.554 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058106390000000003 2024-09-18T04:39:06.554 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.554 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.554 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:06.554 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606982+0000", 2024-09-18T04:39:06.554 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2252e-05 2024-09-18T04:39:06.554 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.554 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.554 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:06.554 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606999+0000", 2024-09-18T04:39:06.554 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6871000000000002e-05 2024-09-18T04:39:06.554 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:06.554 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:06.555 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:06.555 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.555 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.555 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:110 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:06.555 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.586640+0000", 2024-09-18T04:39:06.555 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 9.8096919460000009, 2024-09-18T04:39:06.555 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020400911000000001, 2024-09-18T04:39:06.555 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:06.555 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:06.555 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:06.555 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:06.555 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:06.555 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 110 2024-09-18T04:39:06.556 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.556 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:06.556 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.556 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:06.556 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586640+0000", 2024-09-18T04:39:06.556 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:06.556 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.556 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.556 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:06.556 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586640+0000", 2024-09-18T04:39:06.556 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:06.556 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.556 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.557 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:06.557 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586639+0000", 2024-09-18T04:39:06.557 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:06.557 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.557 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.557 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:06.557 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586652+0000", 2024-09-18T04:39:06.557 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2482e-05 2024-09-18T04:39:06.557 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.557 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.557 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:06.557 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586660+0000", 2024-09-18T04:39:06.557 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1659999999999994e-06 2024-09-18T04:39:06.558 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.558 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.558 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:06.558 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586666+0000", 2024-09-18T04:39:06.558 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6900000000000003e-06 2024-09-18T04:39:06.558 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.558 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.558 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:06.558 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599381+0000", 2024-09-18T04:39:06.558 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012714328 2024-09-18T04:39:06.558 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.558 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.558 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:06.558 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599436+0000", 2024-09-18T04:39:06.559 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4802999999999997e-05 2024-09-18T04:39:06.559 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.559 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.559 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:06.559 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599629+0000", 2024-09-18T04:39:06.559 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000193921 2024-09-18T04:39:06.559 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.559 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.559 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:06.559 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.601174+0000", 2024-09-18T04:39:06.559 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001544857 2024-09-18T04:39:06.559 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.559 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.559 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:06.560 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.607004+0000", 2024-09-18T04:39:06.560 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058300820000000003 2024-09-18T04:39:06.560 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.560 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.560 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:06.560 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.607024+0000", 2024-09-18T04:39:06.560 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9888e-05 2024-09-18T04:39:06.560 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:06.560 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:06.560 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:06.560 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.607041+0000", 2024-09-18T04:39:06.560 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6450999999999999e-05 2024-09-18T04:39:06.560 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:06.561 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:06.561 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:06.561 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:06.561 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:06.561 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:06.563 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:39:06.563 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:39:06.563 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:39:06.563 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:39:06.564 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:39:06.565 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:06.557+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:06.565 INFO:tasks.workunit.client.0.smithi115.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-18T04:39:06.565 INFO:tasks.workunit.client.0.smithi115.stderr:++ _sudo rbd device --device-type nbd map rbd/ns/testrbdnbd17342@snap 2024-09-18T04:39:06.565 INFO:tasks.workunit.client.0.smithi115.stderr:++ local cmd 2024-09-18T04:39:06.565 INFO:tasks.workunit.client.0.smithi115.stderr:+++ id -u 2024-09-18T04:39:06.565 INFO:tasks.workunit.client.0.smithi115.stderr:++ '[' 1000 -eq 0 ']' 2024-09-18T04:39:06.566 INFO:tasks.workunit.client.0.smithi115.stderr:+++ which rbd 2024-09-18T04:39:06.566 INFO:tasks.workunit.client.0.smithi115.stderr:++ cmd=/usr/bin/rbd 2024-09-18T04:39:06.566 INFO:tasks.workunit.client.0.smithi115.stderr:++ shift 2024-09-18T04:39:06.566 INFO:tasks.workunit.client.0.smithi115.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/ns/testrbdnbd17342@snap 2024-09-18T04:39:06.566 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-09-18T04:39:06.567 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:39:06.567 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:39:06.567 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:39:06.567 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:39:06.567 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:39:06.567 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:39:06.567 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:39:06.567 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:39:06.663 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:06.657+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:06.684 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:06.684 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:06.694 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:39:06.703 INFO:tasks.workunit.client.0.smithi115.stderr:+ DEV=/dev/nbd0 2024-09-18T04:39:06.703 INFO:tasks.workunit.client.0.smithi115.stderr:+ get_pid rbd ns 2024-09-18T04:39:06.703 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pool=rbd 2024-09-18T04:39:06.703 INFO:tasks.workunit.client.0.smithi115.stderr:+ local ns=ns 2024-09-18T04:39:06.704 INFO:tasks.workunit.client.0.smithi115.stderr:++ rbd device --device-type nbd --format xml list 2024-09-18T04:39:06.704 INFO:tasks.workunit.client.0.smithi115.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\''ns'\''][image='\''testrbdnbd17342'\''][device='\''/dev/nbd0'\'']/id' 2024-09-18T04:39:06.765 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:06.757+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:06.777 INFO:tasks.workunit.client.0.smithi115.stderr:+ PID=18781 2024-09-18T04:39:06.778 INFO:tasks.workunit.client.0.smithi115.stderr:+ test -n 18781 2024-09-18T04:39:06.778 INFO:tasks.workunit.client.0.smithi115.stderr:+ ps -p 18781 -C rbd-nbd 2024-09-18T04:39:06.783 INFO:tasks.workunit.client.0.smithi115.stdout: PID TTY TIME CMD 2024-09-18T04:39:06.783 INFO:tasks.workunit.client.0.smithi115.stdout: 18781 ? 00:00:00 rbd-nbd 2024-09-18T04:39:06.784 INFO:tasks.workunit.client.0.smithi115.stderr:+ unmap_device rbd/ns/testrbdnbd17342@snap 18781 2024-09-18T04:39:06.784 INFO:tasks.workunit.client.0.smithi115.stderr:+ local args=rbd/ns/testrbdnbd17342@snap 2024-09-18T04:39:06.784 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pid=18781 2024-09-18T04:39:06.784 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo rbd device --device-type nbd unmap rbd/ns/testrbdnbd17342@snap 2024-09-18T04:39:06.785 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:39:06.785 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:39:06.785 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:39:06.786 INFO:tasks.workunit.client.0.smithi115.stderr:++ which rbd 2024-09-18T04:39:06.788 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/bin/rbd 2024-09-18T04:39:06.789 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:39:06.789 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap rbd/ns/testrbdnbd17342@snap 2024-09-18T04:39:06.795 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:06.795 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:06.805 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:39:06.866 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:06.857+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:06.882 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd device --device-type nbd list 2024-09-18T04:39:06.883 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false grep '^18781\b' 2024-09-18T04:39:06.883 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep '^18781\b' 2024-09-18T04:39:06.910 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:06.910 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:06.919 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:39:06.948 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:39:06.949 INFO:tasks.workunit.client.0.smithi115.stderr:+ ps -C rbd-nbd 2024-09-18T04:39:06.949 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false grep '^ *18781\b' 2024-09-18T04:39:06.949 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep '^ *18781\b' 2024-09-18T04:39:06.963 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:39:06.963 INFO:tasks.workunit.client.0.smithi115.stderr:+ sleep 0.5 2024-09-18T04:39:06.967 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:06.961+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:07.020 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:07.020 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:07.030 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:39:07.068 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:07.061+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:07.131 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:07.131 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:07.140 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:39:07.169 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:07.165+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:07.247 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:07.247 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:07.256 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:39:07.270 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:07.265+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:07.371 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:07.365+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:07.377 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:07.377 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:07.387 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:39:07.465 INFO:tasks.workunit.client.0.smithi115.stderr:+ DEV= 2024-09-18T04:39:07.466 INFO:tasks.workunit.client.0.smithi115.stderr:++ rbd snap ls rbd/ns/testrbdnbd17342 2024-09-18T04:39:07.466 INFO:tasks.workunit.client.0.smithi115.stderr:++ awk '$2 == "snap" {print $1}' 2024-09-18T04:39:07.472 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:07.465+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:39:07.493 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:07.493 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:07.504 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:39:07.544 INFO:tasks.workunit.client.0.smithi115.stderr:+ SNAPID=5 2024-09-18T04:39:07.544 INFO:tasks.workunit.client.0.smithi115.stderr:++ _sudo rbd device --device-type nbd map --snap-id 5 rbd/ns/testrbdnbd17342 2024-09-18T04:39:07.544 INFO:tasks.workunit.client.0.smithi115.stderr:++ local cmd 2024-09-18T04:39:07.544 INFO:tasks.workunit.client.0.smithi115.stderr:+++ id -u 2024-09-18T04:39:07.544 INFO:tasks.workunit.client.0.smithi115.stderr:++ '[' 1000 -eq 0 ']' 2024-09-18T04:39:07.545 INFO:tasks.workunit.client.0.smithi115.stderr:+++ which rbd 2024-09-18T04:39:07.547 INFO:tasks.workunit.client.0.smithi115.stderr:++ cmd=/usr/bin/rbd 2024-09-18T04:39:07.547 INFO:tasks.workunit.client.0.smithi115.stderr:++ shift 2024-09-18T04:39:07.547 INFO:tasks.workunit.client.0.smithi115.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map --snap-id 5 rbd/ns/testrbdnbd17342 2024-09-18T04:39:07.573 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:07.569+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:07.606 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:08.279 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:08.281 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:39:08.281 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:08.277+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:39:08.282 INFO:tasks.workunit.client.0.smithi115.stderr:+ DEV=/dev/nbd0 2024-09-18T04:39:08.282 INFO:tasks.workunit.client.0.smithi115.stderr:+ get_pid rbd ns 2024-09-18T04:39:08.282 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pool=rbd 2024-09-18T04:39:08.282 INFO:tasks.workunit.client.0.smithi115.stderr:+ local ns=ns 2024-09-18T04:39:08.282 INFO:tasks.workunit.client.0.smithi115.stderr:++ rbd device --device-type nbd --format xml list 2024-09-18T04:39:08.282 INFO:tasks.workunit.client.0.smithi115.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\''ns'\''][image='\''testrbdnbd17342'\''][device='\''/dev/nbd0'\'']/id' 2024-09-18T04:39:08.282 INFO:tasks.workunit.client.0.smithi115.stderr:+ PID=18867 2024-09-18T04:39:08.282 INFO:tasks.workunit.client.0.smithi115.stderr:+ test -n 18867 2024-09-18T04:39:08.282 INFO:tasks.workunit.client.0.smithi115.stderr:+ ps -p 18867 -C rbd-nbd 2024-09-18T04:39:08.283 INFO:tasks.workunit.client.0.smithi115.stdout: PID TTY TIME CMD 2024-09-18T04:39:08.283 INFO:tasks.workunit.client.0.smithi115.stdout: 18867 ? 00:00:00 rbd-nbd 2024-09-18T04:39:08.284 INFO:tasks.workunit.client.0.smithi115.stderr:+ unmap_device '--snap-id 5 rbd/ns/testrbdnbd17342' 18867 2024-09-18T04:39:08.285 INFO:tasks.workunit.client.0.smithi115.stderr:+ local 'args=--snap-id 5 rbd/ns/testrbdnbd17342' 2024-09-18T04:39:08.285 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pid=18867 2024-09-18T04:39:08.285 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo rbd device --device-type nbd unmap --snap-id 5 rbd/ns/testrbdnbd17342 2024-09-18T04:39:08.285 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:39:08.285 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:39:08.285 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:39:08.285 INFO:tasks.workunit.client.0.smithi115.stderr:++ which rbd 2024-09-18T04:39:08.285 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/bin/rbd 2024-09-18T04:39:08.285 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:39:08.285 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap --snap-id 5 rbd/ns/testrbdnbd17342 2024-09-18T04:39:08.286 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd device --device-type nbd list 2024-09-18T04:39:08.286 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false grep '^18867\b' 2024-09-18T04:39:08.286 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep '^18867\b' 2024-09-18T04:39:08.286 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:39:08.286 INFO:tasks.workunit.client.0.smithi115.stderr:+ ps -C rbd-nbd 2024-09-18T04:39:08.286 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false grep '^ *18867\b' 2024-09-18T04:39:08.286 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep '^ *18867\b' 2024-09-18T04:39:08.286 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:39:08.286 INFO:tasks.workunit.client.0.smithi115.stderr:+ sleep 0.5 2024-09-18T04:39:08.381 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:08.373+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:39:08.441 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:08.441 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:08.452 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:39:08.482 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:08.477+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:08.493 INFO:tasks.workunit.client.0.smithi115.stderr:+ DEV= 2024-09-18T04:39:08.494 INFO:tasks.workunit.client.0.smithi115.stderr:++ _sudo rbd device --device-type nbd map --pool rbd --namespace ns --image testrbdnbd17342 2024-09-18T04:39:08.494 INFO:tasks.workunit.client.0.smithi115.stderr:++ local cmd 2024-09-18T04:39:08.494 INFO:tasks.workunit.client.0.smithi115.stderr:+++ id -u 2024-09-18T04:39:08.496 INFO:tasks.workunit.client.0.smithi115.stderr:++ '[' 1000 -eq 0 ']' 2024-09-18T04:39:08.497 INFO:tasks.workunit.client.0.smithi115.stderr:+++ which rbd 2024-09-18T04:39:08.499 INFO:tasks.workunit.client.0.smithi115.stderr:++ cmd=/usr/bin/rbd 2024-09-18T04:39:08.499 INFO:tasks.workunit.client.0.smithi115.stderr:++ shift 2024-09-18T04:39:08.499 INFO:tasks.workunit.client.0.smithi115.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map --pool rbd --namespace ns --image testrbdnbd17342 2024-09-18T04:39:08.561 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:08.562 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:08.567 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-18T04:39:08.571 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:39:08.575 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T04:39:08.582 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:08.577+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:08.669 INFO:tasks.workunit.client.0.smithi115.stderr:+ DEV=/dev/nbd0 2024-09-18T04:39:08.669 INFO:tasks.workunit.client.0.smithi115.stderr:+ get_pid rbd ns 2024-09-18T04:39:08.669 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pool=rbd 2024-09-18T04:39:08.670 INFO:tasks.workunit.client.0.smithi115.stderr:+ local ns=ns 2024-09-18T04:39:08.670 INFO:tasks.workunit.client.0.smithi115.stderr:++ rbd device --device-type nbd --format xml list 2024-09-18T04:39:08.670 INFO:tasks.workunit.client.0.smithi115.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\''ns'\''][image='\''testrbdnbd17342'\''][device='\''/dev/nbd0'\'']/id' 2024-09-18T04:39:08.683 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:08.677+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:08.744 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:08.745 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:08.752 INFO:tasks.workunit.client.0.smithi115.stderr:+ PID=18931 2024-09-18T04:39:08.753 INFO:tasks.workunit.client.0.smithi115.stderr:+ test -n 18931 2024-09-18T04:39:08.753 INFO:tasks.workunit.client.0.smithi115.stderr:+ ps -p 18931 -C rbd-nbd 2024-09-18T04:39:08.754 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:39:08.755 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:39:08.763 INFO:tasks.workunit.client.0.smithi115.stdout: PID TTY TIME CMD 2024-09-18T04:39:08.763 INFO:tasks.workunit.client.0.smithi115.stdout: 18931 ? 00:00:00 rbd-nbd 2024-09-18T04:39:08.764 INFO:tasks.workunit.client.0.smithi115.stderr:+ unmap_device '--pool rbd --namespace ns --image testrbdnbd17342' 18931 2024-09-18T04:39:08.764 INFO:tasks.workunit.client.0.smithi115.stderr:+ local 'args=--pool rbd --namespace ns --image testrbdnbd17342' 2024-09-18T04:39:08.764 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pid=18931 2024-09-18T04:39:08.764 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo rbd device --device-type nbd unmap --pool rbd --namespace ns --image testrbdnbd17342 2024-09-18T04:39:08.764 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:39:08.764 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:39:08.765 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:39:08.766 INFO:tasks.workunit.client.0.smithi115.stderr:++ which rbd 2024-09-18T04:39:08.768 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/bin/rbd 2024-09-18T04:39:08.768 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:39:08.768 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap --pool rbd --namespace ns --image testrbdnbd17342 2024-09-18T04:39:08.777 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:39:08.784 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:08.777+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:08.797 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:39:08.821 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:39:08.850 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:39:08.863 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:08.863 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:08.863 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:39:08.863 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:08.867 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd device --device-type nbd list 2024-09-18T04:39:08.867 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false grep '^18931\b' 2024-09-18T04:39:08.867 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep '^18931\b' 2024-09-18T04:39:08.872 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:39:08.873 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:39:08.874 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:39:08.874 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:39:08.874 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:39:08.874 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:39:08.874 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:39:08.874 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:39:08.884 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:08.877+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:08.934 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:39:08.935 INFO:tasks.workunit.client.0.smithi115.stderr:+ ps -C rbd-nbd 2024-09-18T04:39:08.935 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false grep '^ *18931\b' 2024-09-18T04:39:08.935 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep '^ *18931\b' 2024-09-18T04:39:08.941 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:39:08.941 INFO:tasks.workunit.client.0.smithi115.stderr:+ sleep 0.5 2024-09-18T04:39:08.984 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:08.985 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:08.985 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:39:08.985 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:39:08.985 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:08.985 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:08.981+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:08.994 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:39:09.086 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:09.081+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:39:09.109 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:09.110 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:39:09.110 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:39:09.111 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:39:09.111 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.111 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:33 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:09.111 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.261182+0000", 2024-09-18T04:39:09.111 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 26.841863278000002, 2024-09-18T04:39:09.112 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.037082208999999998, 2024-09-18T04:39:09.112 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:09.112 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:09.112 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:09.112 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:09.113 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:09.113 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 33 2024-09-18T04:39:09.113 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.113 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:09.113 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.114 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:09.114 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.261182+0000", 2024-09-18T04:39:09.114 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:09.114 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.114 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.115 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:09.115 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.261182+0000", 2024-09-18T04:39:09.115 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:09.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.115 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.115 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:09.115 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.261180+0000", 2024-09-18T04:39:09.115 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:09.116 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.116 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.116 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:09.116 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278669+0000", 2024-09-18T04:39:09.116 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.017488877999999999 2024-09-18T04:39:09.116 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.116 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.116 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:09.116 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278689+0000", 2024-09-18T04:39:09.117 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0047999999999998e-05 2024-09-18T04:39:09.117 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.117 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.117 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:09.117 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278699+0000", 2024-09-18T04:39:09.117 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6800000000000005e-06 2024-09-18T04:39:09.117 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.118 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.118 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:09.118 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278775+0000", 2024-09-18T04:39:09.118 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6087999999999994e-05 2024-09-18T04:39:09.118 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.118 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.118 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:09.118 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278935+0000", 2024-09-18T04:39:09.118 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000159834 2024-09-18T04:39:09.118 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.119 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.119 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:09.119 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279064+0000", 2024-09-18T04:39:09.119 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012954199999999999 2024-09-18T04:39:09.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.119 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.120 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:09.120 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295562+0000", 2024-09-18T04:39:09.120 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016497583999999999 2024-09-18T04:39:09.120 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.120 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.120 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:09.120 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.298147+0000", 2024-09-18T04:39:09.120 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002584982 2024-09-18T04:39:09.120 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.120 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.120 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:09.121 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.298217+0000", 2024-09-18T04:39:09.121 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0400000000000004e-05 2024-09-18T04:39:09.121 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.121 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.121 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:09.121 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.298265+0000", 2024-09-18T04:39:09.121 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7580000000000002e-05 2024-09-18T04:39:09.121 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:09.121 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:09.121 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:09.122 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.122 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.122 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:34 2.3 e51/24 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head v 51'3, mlcod=18'1)", 2024-09-18T04:39:09.122 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.273067+0000", 2024-09-18T04:39:09.122 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 26.829978487999998, 2024-09-18T04:39:09.122 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.021784103999999999, 2024-09-18T04:39:09.122 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:09.122 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:09.122 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:09.122 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.123 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:09.123 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.273067+0000", 2024-09-18T04:39:09.123 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:09.123 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.123 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.123 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:09.123 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.273067+0000", 2024-09-18T04:39:09.123 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:09.123 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.123 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.123 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:09.123 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.273060+0000", 2024-09-18T04:39:09.124 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999924 2024-09-18T04:39:09.124 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.124 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.124 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:09.124 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278596+0000", 2024-09-18T04:39:09.124 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055365839999999998 2024-09-18T04:39:09.124 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.124 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.124 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:09.124 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278637+0000", 2024-09-18T04:39:09.124 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0596000000000002e-05 2024-09-18T04:39:09.124 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.124 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.124 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:09.125 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278653+0000", 2024-09-18T04:39:09.125 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6626999999999999e-05 2024-09-18T04:39:09.125 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.125 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.125 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:09.125 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278748+0000", 2024-09-18T04:39:09.125 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4483999999999999e-05 2024-09-18T04:39:09.125 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.125 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.125 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:09.125 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278799+0000", 2024-09-18T04:39:09.125 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1484999999999999e-05 2024-09-18T04:39:09.125 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.126 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.126 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:09.126 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.294785+0000", 2024-09-18T04:39:09.126 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015985256999999999 2024-09-18T04:39:09.126 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.126 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.126 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:09.126 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.294851+0000", 2024-09-18T04:39:09.126 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6728000000000002e-05 2024-09-18T04:39:09.126 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:09.127 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:09.127 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:09.127 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.127 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.127 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:37 2.5 2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:09.127 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.278722+0000", 2024-09-18T04:39:09.128 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 26.824323439, 2024-09-18T04:39:09.128 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.026323801000000001, 2024-09-18T04:39:09.128 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:09.128 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:09.128 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:09.129 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:09.129 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:09.129 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 37 2024-09-18T04:39:09.129 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.129 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:09.129 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.129 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:09.129 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278722+0000", 2024-09-18T04:39:09.129 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:09.130 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.130 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.130 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:09.130 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278722+0000", 2024-09-18T04:39:09.131 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:09.131 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.131 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.131 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:09.131 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278720+0000", 2024-09-18T04:39:09.131 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:09.131 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.131 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.132 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:09.132 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.291875+0000", 2024-09-18T04:39:09.132 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013154414 2024-09-18T04:39:09.132 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.132 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.132 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:09.132 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.291901+0000", 2024-09-18T04:39:09.132 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6339999999999999e-05 2024-09-18T04:39:09.132 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.132 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.132 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:09.132 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.291914+0000", 2024-09-18T04:39:09.132 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3043e-05 2024-09-18T04:39:09.133 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.133 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.133 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:09.133 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.291975+0000", 2024-09-18T04:39:09.133 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0949999999999998e-05 2024-09-18T04:39:09.133 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.133 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.133 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:09.133 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292173+0000", 2024-09-18T04:39:09.133 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019825499999999999 2024-09-18T04:39:09.133 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.134 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.134 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:09.134 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292300+0000", 2024-09-18T04:39:09.134 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000126616 2024-09-18T04:39:09.134 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.134 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.134 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:09.134 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304334+0000", 2024-09-18T04:39:09.134 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012033934 2024-09-18T04:39:09.134 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.134 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.134 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:09.134 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304901+0000", 2024-09-18T04:39:09.134 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00056694099999999995 2024-09-18T04:39:09.135 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.135 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.135 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:09.135 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304981+0000", 2024-09-18T04:39:09.135 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9974000000000004e-05 2024-09-18T04:39:09.135 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.135 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.135 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:09.135 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.305046+0000", 2024-09-18T04:39:09.135 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5184000000000005e-05 2024-09-18T04:39:09.136 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:09.136 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:09.136 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:09.136 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.136 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.136 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:49 2.5 2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:09.136 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.314271+0000", 2024-09-18T04:39:09.137 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 26.788774836999998, 2024-09-18T04:39:09.137 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.029779953000000001, 2024-09-18T04:39:09.137 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:09.137 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:09.137 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:09.137 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:09.137 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:09.137 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 49 2024-09-18T04:39:09.138 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.138 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:09.138 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.138 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:09.139 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.314271+0000", 2024-09-18T04:39:09.139 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:09.139 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.139 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.139 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:09.139 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.314271+0000", 2024-09-18T04:39:09.139 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:09.139 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.139 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.139 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:09.139 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.314269+0000", 2024-09-18T04:39:09.140 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:09.140 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.140 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.140 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:09.140 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.324780+0000", 2024-09-18T04:39:09.140 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010511609999999999 2024-09-18T04:39:09.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.141 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.141 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:09.141 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.324821+0000", 2024-09-18T04:39:09.141 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0899000000000003e-05 2024-09-18T04:39:09.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.141 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.141 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:09.141 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.324840+0000", 2024-09-18T04:39:09.142 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8947e-05 2024-09-18T04:39:09.142 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.142 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.142 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:09.142 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.324936+0000", 2024-09-18T04:39:09.143 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6342999999999996e-05 2024-09-18T04:39:09.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.143 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.143 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:09.143 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325284+0000", 2024-09-18T04:39:09.143 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000347146 2024-09-18T04:39:09.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.143 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.143 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:09.143 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325586+0000", 2024-09-18T04:39:09.144 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00030199799999999999 2024-09-18T04:39:09.144 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.144 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.144 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:09.144 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343710+0000", 2024-09-18T04:39:09.144 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018124583999999999 2024-09-18T04:39:09.144 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.144 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.144 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:09.145 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343991+0000", 2024-09-18T04:39:09.145 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00028085600000000001 2024-09-18T04:39:09.145 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.145 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.145 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:09.145 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344028+0000", 2024-09-18T04:39:09.145 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6637000000000002e-05 2024-09-18T04:39:09.145 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.146 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.146 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:09.146 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344051+0000", 2024-09-18T04:39:09.146 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3260000000000001e-05 2024-09-18T04:39:09.146 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:09.146 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:09.147 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:09.147 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.147 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.147 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:50 2.3 e51/24 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head v 51'4, mlcod=51'2)", 2024-09-18T04:39:09.147 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.323066+0000", 2024-09-18T04:39:09.147 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 26.779979443999999, 2024-09-18T04:39:09.147 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.022135774, 2024-09-18T04:39:09.148 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:09.148 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:09.148 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:09.148 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.148 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:09.148 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.323066+0000", 2024-09-18T04:39:09.148 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:09.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.149 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.149 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:09.149 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.323066+0000", 2024-09-18T04:39:09.149 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:09.150 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.150 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.150 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:09.150 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.323065+0000", 2024-09-18T04:39:09.150 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:09.150 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.151 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.151 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:09.151 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.328664+0000", 2024-09-18T04:39:09.151 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055993600000000003 2024-09-18T04:39:09.151 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.151 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.151 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:09.151 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.328692+0000", 2024-09-18T04:39:09.151 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7654e-05 2024-09-18T04:39:09.151 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.152 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.152 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:09.152 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.328706+0000", 2024-09-18T04:39:09.152 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4097000000000001e-05 2024-09-18T04:39:09.152 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.152 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.152 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:09.152 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.328816+0000", 2024-09-18T04:39:09.152 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000110281 2024-09-18T04:39:09.152 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.152 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.153 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:09.153 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.328848+0000", 2024-09-18T04:39:09.153 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1837e-05 2024-09-18T04:39:09.153 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.153 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.153 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:09.153 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345160+0000", 2024-09-18T04:39:09.153 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016312099 2024-09-18T04:39:09.153 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.154 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.154 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:09.154 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345202+0000", 2024-09-18T04:39:09.154 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1857999999999998e-05 2024-09-18T04:39:09.154 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:09.154 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:09.154 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:09.154 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.154 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.154 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:58 2.5 2:b227e6b1:::rbd_data.125fc09df6d1.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:09.154 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.351564+0000", 2024-09-18T04:39:09.155 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 26.751481991999999, 2024-09-18T04:39:09.155 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.032125956999999997, 2024-09-18T04:39:09.155 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:09.155 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:09.155 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:09.155 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:09.155 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:09.155 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 58 2024-09-18T04:39:09.155 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.155 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:09.155 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.156 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:09.156 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.351564+0000", 2024-09-18T04:39:09.156 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:09.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.156 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.156 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:09.156 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.351564+0000", 2024-09-18T04:39:09.156 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:09.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.156 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.156 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:09.156 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.351561+0000", 2024-09-18T04:39:09.156 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:09.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.157 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.157 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:09.157 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364390+0000", 2024-09-18T04:39:09.157 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012829063999999999 2024-09-18T04:39:09.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.157 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.157 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:09.157 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364418+0000", 2024-09-18T04:39:09.157 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7307000000000001e-05 2024-09-18T04:39:09.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.157 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.157 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:09.158 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364428+0000", 2024-09-18T04:39:09.158 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0675e-05 2024-09-18T04:39:09.158 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.158 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.158 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:09.158 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364523+0000", 2024-09-18T04:39:09.158 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4698999999999996e-05 2024-09-18T04:39:09.158 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.158 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.158 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:09.158 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364785+0000", 2024-09-18T04:39:09.158 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00026170099999999998 2024-09-18T04:39:09.158 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.159 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.159 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:09.159 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.365039+0000", 2024-09-18T04:39:09.159 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00025462299999999999 2024-09-18T04:39:09.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.159 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.159 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:09.159 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383143+0000", 2024-09-18T04:39:09.159 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018103768999999999 2024-09-18T04:39:09.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.159 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.159 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:09.159 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383582+0000", 2024-09-18T04:39:09.159 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00043860600000000002 2024-09-18T04:39:09.160 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.160 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.160 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:09.160 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383644+0000", 2024-09-18T04:39:09.160 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2682999999999996e-05 2024-09-18T04:39:09.160 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.160 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.160 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:09.160 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383690+0000", 2024-09-18T04:39:09.160 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5247999999999998e-05 2024-09-18T04:39:09.160 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:09.160 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:09.161 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:09.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.161 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.161 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:33 2.1 2:8b40f1c5:ns::rbd_data.133e813ab5bc.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:09.161 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.523765+0000", 2024-09-18T04:39:09.161 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 26.579280952000001, 2024-09-18T04:39:09.161 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027472973000000001, 2024-09-18T04:39:09.161 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:09.161 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:09.161 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:09.161 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:09.161 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:09.161 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 33 2024-09-18T04:39:09.162 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.162 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:09.162 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.162 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:09.162 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.523765+0000", 2024-09-18T04:39:09.162 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:09.162 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.162 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.162 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:09.162 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.523765+0000", 2024-09-18T04:39:09.162 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:09.162 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.162 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.162 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:09.163 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.523762+0000", 2024-09-18T04:39:09.163 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:09.163 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.163 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.163 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:09.163 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531127+0000", 2024-09-18T04:39:09.163 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0073644560000000001 2024-09-18T04:39:09.163 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.163 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.163 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:09.163 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531189+0000", 2024-09-18T04:39:09.163 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2315000000000002e-05 2024-09-18T04:39:09.163 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.164 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.164 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:09.164 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531211+0000", 2024-09-18T04:39:09.164 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1542000000000001e-05 2024-09-18T04:39:09.164 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.164 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.164 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:09.164 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531314+0000", 2024-09-18T04:39:09.164 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000103582 2024-09-18T04:39:09.164 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.164 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.164 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:09.165 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531493+0000", 2024-09-18T04:39:09.165 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000178433 2024-09-18T04:39:09.165 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.165 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.165 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:09.165 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531653+0000", 2024-09-18T04:39:09.165 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000160388 2024-09-18T04:39:09.165 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.165 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.165 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:09.165 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.547345+0000", 2024-09-18T04:39:09.166 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015691742000000002 2024-09-18T04:39:09.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.166 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.166 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:09.166 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551076+0000", 2024-09-18T04:39:09.166 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003730957 2024-09-18T04:39:09.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.166 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.166 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:09.166 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551176+0000", 2024-09-18T04:39:09.166 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010002599999999999 2024-09-18T04:39:09.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.167 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.167 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:09.167 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551238+0000", 2024-09-18T04:39:09.167 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1945000000000004e-05 2024-09-18T04:39:09.167 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:09.167 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:09.167 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:09.167 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.167 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.167 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:36 2.5 2:bd5b8dd8:ns::rbd_data.133e813ab5bc.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:09.167 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.531253+0000", 2024-09-18T04:39:09.167 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 26.571792642999998, 2024-09-18T04:39:09.168 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027295212999999999, 2024-09-18T04:39:09.168 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:09.168 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:09.169 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:09.169 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:09.169 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:09.169 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 36 2024-09-18T04:39:09.169 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.169 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:09.170 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.170 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:09.170 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531253+0000", 2024-09-18T04:39:09.170 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:09.170 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.170 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.171 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:09.171 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531253+0000", 2024-09-18T04:39:09.171 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:09.171 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.171 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.171 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:09.172 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531251+0000", 2024-09-18T04:39:09.172 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:09.172 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.173 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.173 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:09.173 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536540+0000", 2024-09-18T04:39:09.173 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0052885839999999998 2024-09-18T04:39:09.173 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.173 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.173 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:09.173 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536604+0000", 2024-09-18T04:39:09.173 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4824000000000001e-05 2024-09-18T04:39:09.173 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.174 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.174 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:09.174 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536621+0000", 2024-09-18T04:39:09.174 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6215999999999999e-05 2024-09-18T04:39:09.174 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.174 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.174 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:09.174 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536768+0000", 2024-09-18T04:39:09.175 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014767900000000001 2024-09-18T04:39:09.175 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.175 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.175 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:09.175 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537084+0000", 2024-09-18T04:39:09.175 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00031553499999999998 2024-09-18T04:39:09.175 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.175 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.175 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:09.176 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537300+0000", 2024-09-18T04:39:09.176 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000216145 2024-09-18T04:39:09.176 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.176 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.176 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:09.177 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555326+0000", 2024-09-18T04:39:09.177 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018025861000000001 2024-09-18T04:39:09.177 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.177 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.177 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:09.177 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558449+0000", 2024-09-18T04:39:09.177 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0031229750000000001 2024-09-18T04:39:09.177 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.177 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.178 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:09.178 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558502+0000", 2024-09-18T04:39:09.178 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3572000000000001e-05 2024-09-18T04:39:09.178 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.178 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.178 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:09.178 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558548+0000", 2024-09-18T04:39:09.179 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5921999999999998e-05 2024-09-18T04:39:09.179 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:09.179 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:09.179 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:09.179 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.179 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.179 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:37 2.5 2:bf39b8e4:ns::rbd_data.133e813ab5bc.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:09.179 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.536664+0000", 2024-09-18T04:39:09.179 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 26.566381227000001, 2024-09-18T04:39:09.179 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.026331765, 2024-09-18T04:39:09.179 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:09.180 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:09.180 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:09.180 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:09.180 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:09.180 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 37 2024-09-18T04:39:09.180 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.180 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:09.180 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.181 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:09.181 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536664+0000", 2024-09-18T04:39:09.181 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:09.181 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.181 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.181 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:09.181 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536664+0000", 2024-09-18T04:39:09.181 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:09.182 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.182 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.182 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:09.182 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536662+0000", 2024-09-18T04:39:09.182 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:09.182 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.183 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.183 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:09.183 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.543699+0000", 2024-09-18T04:39:09.183 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0070367119999999997 2024-09-18T04:39:09.183 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.183 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.183 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:09.183 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.543749+0000", 2024-09-18T04:39:09.184 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0476999999999998e-05 2024-09-18T04:39:09.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.184 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.184 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:09.184 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.543800+0000", 2024-09-18T04:39:09.184 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0822999999999998e-05 2024-09-18T04:39:09.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.185 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.185 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:09.185 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.547868+0000", 2024-09-18T04:39:09.185 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0040679339999999996 2024-09-18T04:39:09.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.185 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.185 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:09.185 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548087+0000", 2024-09-18T04:39:09.185 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000218994 2024-09-18T04:39:09.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.186 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.186 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:09.186 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548383+0000", 2024-09-18T04:39:09.186 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000295588 2024-09-18T04:39:09.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.186 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.187 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:09.187 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562622+0000", 2024-09-18T04:39:09.187 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014239915000000001 2024-09-18T04:39:09.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.187 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.187 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:09.187 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562924+0000", 2024-09-18T04:39:09.187 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00030127600000000001 2024-09-18T04:39:09.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.187 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.187 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:09.187 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562957+0000", 2024-09-18T04:39:09.187 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3365999999999999e-05 2024-09-18T04:39:09.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.187 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.188 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:09.188 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562996+0000", 2024-09-18T04:39:09.188 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9153999999999998e-05 2024-09-18T04:39:09.188 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:09.188 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:09.189 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:09.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.189 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.189 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:50 2.1 2:8c706374:ns::rbd_data.133e813ab5bc.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:09.189 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.570513+0000", 2024-09-18T04:39:09.190 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 26.532532945, 2024-09-18T04:39:09.190 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.024182995999999998, 2024-09-18T04:39:09.190 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:09.190 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:09.190 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:09.191 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:09.191 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:09.191 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 50 2024-09-18T04:39:09.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.191 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:09.192 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.192 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:09.192 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570513+0000", 2024-09-18T04:39:09.192 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:09.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.192 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.192 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:09.193 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570513+0000", 2024-09-18T04:39:09.193 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:09.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.193 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.193 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:09.193 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570511+0000", 2024-09-18T04:39:09.193 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:09.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.193 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.193 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:09.193 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.576327+0000", 2024-09-18T04:39:09.193 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0058162530000000004 2024-09-18T04:39:09.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.194 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.194 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:09.194 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.576355+0000", 2024-09-18T04:39:09.194 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8030000000000001e-05 2024-09-18T04:39:09.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.194 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.194 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:09.194 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.576366+0000", 2024-09-18T04:39:09.194 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1139e-05 2024-09-18T04:39:09.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.195 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.195 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:09.195 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.576458+0000", 2024-09-18T04:39:09.195 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1780999999999994e-05 2024-09-18T04:39:09.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.195 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.196 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:09.196 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.576701+0000", 2024-09-18T04:39:09.196 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00024303899999999999 2024-09-18T04:39:09.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.196 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.196 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:09.196 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.576876+0000", 2024-09-18T04:39:09.196 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000174637 2024-09-18T04:39:09.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.196 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.196 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:09.197 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588432+0000", 2024-09-18T04:39:09.197 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011555943000000001 2024-09-18T04:39:09.197 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.197 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.197 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:09.197 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594573+0000", 2024-09-18T04:39:09.197 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0061408670000000004 2024-09-18T04:39:09.197 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.197 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.197 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:09.197 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594640+0000", 2024-09-18T04:39:09.197 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7404999999999996e-05 2024-09-18T04:39:09.197 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.198 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.198 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:09.198 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594696+0000", 2024-09-18T04:39:09.198 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5612999999999998e-05 2024-09-18T04:39:09.198 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:09.198 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:09.198 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:09.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.198 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.199 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:60 2.5 2:a06f51e4:ns::rbd_data.133e813ab5bc.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:09.199 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.604753+0000", 2024-09-18T04:39:09.199 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 26.498292826, 2024-09-18T04:39:09.199 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014400181, 2024-09-18T04:39:09.199 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:09.199 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:09.199 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:09.199 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:09.199 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:09.199 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 60 2024-09-18T04:39:09.200 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.200 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:09.200 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.200 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:09.200 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.604753+0000", 2024-09-18T04:39:09.200 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:09.200 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.200 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.200 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:09.200 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.604753+0000", 2024-09-18T04:39:09.200 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:09.201 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.201 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.201 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:09.201 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.604751+0000", 2024-09-18T04:39:09.201 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:09.202 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.202 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.202 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:09.202 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.608980+0000", 2024-09-18T04:39:09.202 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0042292420000000002 2024-09-18T04:39:09.202 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.202 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.202 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:09.202 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609014+0000", 2024-09-18T04:39:09.203 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3572999999999999e-05 2024-09-18T04:39:09.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.203 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.203 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:09.203 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609032+0000", 2024-09-18T04:39:09.203 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8150000000000001e-05 2024-09-18T04:39:09.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.203 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.203 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:09.203 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609090+0000", 2024-09-18T04:39:09.204 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8214999999999997e-05 2024-09-18T04:39:09.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.204 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.204 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:09.204 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609235+0000", 2024-09-18T04:39:09.204 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014437799999999999 2024-09-18T04:39:09.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.204 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.204 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:09.205 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609360+0000", 2024-09-18T04:39:09.205 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012526099999999999 2024-09-18T04:39:09.205 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.205 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.205 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:09.205 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.614952+0000", 2024-09-18T04:39:09.205 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055922300000000001 2024-09-18T04:39:09.206 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.206 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.206 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:09.206 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.619070+0000", 2024-09-18T04:39:09.206 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0041178250000000003 2024-09-18T04:39:09.207 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.207 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.207 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:09.207 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.619117+0000", 2024-09-18T04:39:09.207 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7275999999999999e-05 2024-09-18T04:39:09.207 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.207 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.207 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:09.208 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.619153+0000", 2024-09-18T04:39:09.208 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5970000000000003e-05 2024-09-18T04:39:09.208 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:09.208 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:09.208 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:09.209 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.209 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.209 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4805.0:592 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [write 393216~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:09.209 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.023386+0000", 2024-09-18T04:39:09.209 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 18.079660121, 2024-09-18T04:39:09.209 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0086813870000000005, 2024-09-18T04:39:09.209 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:09.210 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:09.210 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:09.210 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4805", 2024-09-18T04:39:09.210 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:09.210 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 592 2024-09-18T04:39:09.210 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.210 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:09.211 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.211 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:09.211 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023386+0000", 2024-09-18T04:39:09.211 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:09.211 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.211 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.211 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:09.211 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023386+0000", 2024-09-18T04:39:09.211 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:09.211 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.212 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.212 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:09.212 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023384+0000", 2024-09-18T04:39:09.213 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:09.213 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.213 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.213 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:09.214 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023616+0000", 2024-09-18T04:39:09.214 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00023165600000000001 2024-09-18T04:39:09.214 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.214 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.214 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:09.214 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023637+0000", 2024-09-18T04:39:09.214 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1163e-05 2024-09-18T04:39:09.214 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.215 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.215 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:09.215 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023652+0000", 2024-09-18T04:39:09.215 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4562000000000001e-05 2024-09-18T04:39:09.215 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.216 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.216 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:09.216 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.027732+0000", 2024-09-18T04:39:09.216 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0040803929999999999 2024-09-18T04:39:09.216 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.216 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.217 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:09.217 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.027841+0000", 2024-09-18T04:39:09.217 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000108978 2024-09-18T04:39:09.217 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.217 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.217 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:09.217 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.027990+0000", 2024-09-18T04:39:09.217 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014852999999999999 2024-09-18T04:39:09.217 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.217 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.217 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:09.217 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.031930+0000", 2024-09-18T04:39:09.218 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0039406299999999997 2024-09-18T04:39:09.218 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.218 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.218 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:09.218 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.031978+0000", 2024-09-18T04:39:09.219 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7817e-05 2024-09-18T04:39:09.219 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.220 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.220 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:09.220 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.032019+0000", 2024-09-18T04:39:09.220 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0531999999999998e-05 2024-09-18T04:39:09.220 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.220 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.220 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:09.220 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.032067+0000", 2024-09-18T04:39:09.220 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8371000000000002e-05 2024-09-18T04:39:09.221 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:09.221 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:09.221 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:09.221 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.221 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.221 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4805.0:593 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [write 524288~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:09.221 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.023665+0000", 2024-09-18T04:39:09.221 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 18.079380273999998, 2024-09-18T04:39:09.222 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0085518090000000005, 2024-09-18T04:39:09.222 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:09.222 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:09.222 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:09.222 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4805", 2024-09-18T04:39:09.222 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:09.222 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 593 2024-09-18T04:39:09.222 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.222 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:09.222 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.222 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:09.222 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023665+0000", 2024-09-18T04:39:09.222 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:09.223 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.223 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.223 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:09.223 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023665+0000", 2024-09-18T04:39:09.223 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:09.223 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.223 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.223 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:09.223 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023664+0000", 2024-09-18T04:39:09.223 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:09.223 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.223 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.223 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:09.223 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023875+0000", 2024-09-18T04:39:09.224 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000210862 2024-09-18T04:39:09.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.224 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.224 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:09.224 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023906+0000", 2024-09-18T04:39:09.224 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0660000000000001e-05 2024-09-18T04:39:09.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.224 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.224 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:09.224 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023922+0000", 2024-09-18T04:39:09.224 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5831000000000001e-05 2024-09-18T04:39:09.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.224 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.225 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:09.225 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.029603+0000", 2024-09-18T04:39:09.225 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0056816719999999996 2024-09-18T04:39:09.225 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.225 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.225 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:09.225 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.029766+0000", 2024-09-18T04:39:09.225 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000162903 2024-09-18T04:39:09.225 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.225 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.225 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:09.225 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.030042+0000", 2024-09-18T04:39:09.225 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00027561399999999998 2024-09-18T04:39:09.226 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.226 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.226 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:09.226 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.032127+0000", 2024-09-18T04:39:09.226 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020850920000000002 2024-09-18T04:39:09.226 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.226 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.226 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:09.226 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.032137+0000", 2024-09-18T04:39:09.226 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0312000000000001e-05 2024-09-18T04:39:09.226 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.227 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.227 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:09.227 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.032183+0000", 2024-09-18T04:39:09.227 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6020999999999999e-05 2024-09-18T04:39:09.227 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.227 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.227 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:09.227 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.032217+0000", 2024-09-18T04:39:09.227 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4109999999999997e-05 2024-09-18T04:39:09.227 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:09.227 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:09.228 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:09.228 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.228 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.228 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4805.0:594 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [write 655360~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:09.228 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.023941+0000", 2024-09-18T04:39:09.228 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 18.079104736000001, 2024-09-18T04:39:09.228 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010767616000000001, 2024-09-18T04:39:09.228 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:09.228 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:09.229 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:09.229 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4805", 2024-09-18T04:39:09.229 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:09.229 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 594 2024-09-18T04:39:09.229 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.229 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:09.229 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.229 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:09.229 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023941+0000", 2024-09-18T04:39:09.229 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:09.229 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.229 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.230 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:09.230 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023941+0000", 2024-09-18T04:39:09.230 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:09.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.230 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.230 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:09.231 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023940+0000", 2024-09-18T04:39:09.231 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:09.231 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.231 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.231 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:09.231 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024163+0000", 2024-09-18T04:39:09.231 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000223017 2024-09-18T04:39:09.231 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.232 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.232 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:09.232 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024188+0000", 2024-09-18T04:39:09.232 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5174e-05 2024-09-18T04:39:09.232 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.232 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.232 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:09.232 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024214+0000", 2024-09-18T04:39:09.233 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5782000000000002e-05 2024-09-18T04:39:09.233 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.233 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.233 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:09.233 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.031116+0000", 2024-09-18T04:39:09.233 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.006902282 2024-09-18T04:39:09.233 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.233 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.233 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:09.234 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.031199+0000", 2024-09-18T04:39:09.234 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3203999999999998e-05 2024-09-18T04:39:09.234 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.234 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.234 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:09.234 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.031315+0000", 2024-09-18T04:39:09.234 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000115742 2024-09-18T04:39:09.234 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.234 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.234 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:09.235 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.033780+0000", 2024-09-18T04:39:09.235 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002465058 2024-09-18T04:39:09.235 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.235 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.235 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:09.235 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.034630+0000", 2024-09-18T04:39:09.235 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00084966800000000004 2024-09-18T04:39:09.235 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.235 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.236 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:09.236 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.034682+0000", 2024-09-18T04:39:09.236 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2562999999999998e-05 2024-09-18T04:39:09.236 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.236 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.236 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:09.237 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.034709+0000", 2024-09-18T04:39:09.237 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.62e-05 2024-09-18T04:39:09.237 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:09.237 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:09.237 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:09.237 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.237 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.238 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4805.0:595 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [write 786432~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:09.238 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.024233+0000", 2024-09-18T04:39:09.238 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 18.078812361000001, 2024-09-18T04:39:09.238 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010599558, 2024-09-18T04:39:09.238 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:09.239 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:09.239 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:09.239 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4805", 2024-09-18T04:39:09.239 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:09.239 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 595 2024-09-18T04:39:09.239 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.240 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:09.240 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.240 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:09.240 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024233+0000", 2024-09-18T04:39:09.240 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:09.241 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.241 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.241 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:09.241 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024233+0000", 2024-09-18T04:39:09.241 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:09.241 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.242 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.242 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:09.242 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024232+0000", 2024-09-18T04:39:09.242 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:09.242 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.242 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.242 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:09.243 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024449+0000", 2024-09-18T04:39:09.243 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000216868 2024-09-18T04:39:09.243 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.243 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.243 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:09.243 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024474+0000", 2024-09-18T04:39:09.243 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4598999999999999e-05 2024-09-18T04:39:09.244 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.244 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.244 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:09.244 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024487+0000", 2024-09-18T04:39:09.244 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3451999999999999e-05 2024-09-18T04:39:09.244 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.244 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.245 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:09.245 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.032272+0000", 2024-09-18T04:39:09.245 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0077845709999999997 2024-09-18T04:39:09.245 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.245 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.245 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:09.245 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.032332+0000", 2024-09-18T04:39:09.246 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0736999999999998e-05 2024-09-18T04:39:09.246 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.246 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.246 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:09.246 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.032421+0000", 2024-09-18T04:39:09.246 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8993000000000007e-05 2024-09-18T04:39:09.246 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.246 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.246 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:09.246 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.033786+0000", 2024-09-18T04:39:09.246 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001364451 2024-09-18T04:39:09.246 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.247 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.247 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:09.247 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.034752+0000", 2024-09-18T04:39:09.247 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00096642400000000002 2024-09-18T04:39:09.247 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.247 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.247 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:09.247 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.034789+0000", 2024-09-18T04:39:09.248 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6899e-05 2024-09-18T04:39:09.248 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.248 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.248 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:09.248 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.034833+0000", 2024-09-18T04:39:09.248 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3846e-05 2024-09-18T04:39:09.248 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:09.248 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:09.248 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:09.248 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.249 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.249 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4805.0:596 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [write 917504~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:09.249 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.024500+0000", 2024-09-18T04:39:09.249 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 18.078545576, 2024-09-18T04:39:09.249 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010770534, 2024-09-18T04:39:09.249 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:09.249 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:09.249 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:09.250 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4805", 2024-09-18T04:39:09.250 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:09.250 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 596 2024-09-18T04:39:09.250 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.250 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:09.250 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.250 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:09.250 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024500+0000", 2024-09-18T04:39:09.250 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:09.250 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.250 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.250 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:09.251 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024500+0000", 2024-09-18T04:39:09.251 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:09.251 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.251 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.251 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:09.251 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024499+0000", 2024-09-18T04:39:09.251 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:09.251 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.251 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.251 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:09.251 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024700+0000", 2024-09-18T04:39:09.251 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00020038599999999999 2024-09-18T04:39:09.252 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.252 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.252 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:09.252 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024729+0000", 2024-09-18T04:39:09.252 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9224999999999999e-05 2024-09-18T04:39:09.252 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.252 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.253 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:09.253 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024776+0000", 2024-09-18T04:39:09.253 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7385000000000001e-05 2024-09-18T04:39:09.253 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.253 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.253 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:09.253 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.032998+0000", 2024-09-18T04:39:09.253 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0082216189999999995 2024-09-18T04:39:09.254 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.254 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.254 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:09.254 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.033076+0000", 2024-09-18T04:39:09.254 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8668e-05 2024-09-18T04:39:09.254 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.254 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.255 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:09.255 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.033167+0000", 2024-09-18T04:39:09.255 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0728000000000003e-05 2024-09-18T04:39:09.255 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.255 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.255 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:09.255 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.034399+0000", 2024-09-18T04:39:09.255 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00123234 2024-09-18T04:39:09.255 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.255 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.256 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:09.256 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.035206+0000", 2024-09-18T04:39:09.256 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00080701799999999995 2024-09-18T04:39:09.256 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.256 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.256 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:09.256 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.035244+0000", 2024-09-18T04:39:09.256 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7487999999999998e-05 2024-09-18T04:39:09.256 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.257 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.257 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:09.257 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.035271+0000", 2024-09-18T04:39:09.257 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6709000000000001e-05 2024-09-18T04:39:09.257 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:09.257 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:09.257 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:09.257 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.257 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.257 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4805.0:597 2.1 2:9370a598:::rbd_header.125fc09df6d1:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:09.257 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.025206+0000", 2024-09-18T04:39:09.258 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 18.077839475000001, 2024-09-18T04:39:09.258 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0091536900000000008, 2024-09-18T04:39:09.258 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:09.258 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:09.258 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:09.258 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4805", 2024-09-18T04:39:09.258 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:09.258 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 597 2024-09-18T04:39:09.258 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.258 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:09.258 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.259 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:09.259 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.025206+0000", 2024-09-18T04:39:09.259 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:09.259 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.259 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.259 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:09.259 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.025206+0000", 2024-09-18T04:39:09.259 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:09.259 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.259 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.259 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:09.260 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.025204+0000", 2024-09-18T04:39:09.260 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:09.260 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.260 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.260 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:09.260 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.025216+0000", 2024-09-18T04:39:09.260 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2150999999999999e-05 2024-09-18T04:39:09.260 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.260 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.260 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:09.260 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.025237+0000", 2024-09-18T04:39:09.260 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0740000000000001e-05 2024-09-18T04:39:09.260 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.261 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.261 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:09.261 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.025254+0000", 2024-09-18T04:39:09.261 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6731e-05 2024-09-18T04:39:09.261 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.261 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.261 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:09.261 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.034049+0000", 2024-09-18T04:39:09.261 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0087952810000000003 2024-09-18T04:39:09.261 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.261 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.261 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:09.261 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.034202+0000", 2024-09-18T04:39:09.262 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0001526 2024-09-18T04:39:09.262 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.262 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.262 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:09.262 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.034360+0000", 2024-09-18T04:39:09.262 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015809299999999999 2024-09-18T04:39:09.262 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:09.262 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:09.262 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:09.262 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.262 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.262 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4972.0:14 2.5 2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:head [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:09.263 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.949251+0000", 2024-09-18T04:39:09.263 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.153794528999999, 2024-09-18T04:39:09.263 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0098324480000000006, 2024-09-18T04:39:09.263 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:09.263 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:09.263 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:09.263 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4972", 2024-09-18T04:39:09.263 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2280251472", 2024-09-18T04:39:09.263 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 14 2024-09-18T04:39:09.263 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.263 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:09.263 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.263 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:09.264 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949251+0000", 2024-09-18T04:39:09.264 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:09.264 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.264 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.264 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:09.264 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949251+0000", 2024-09-18T04:39:09.264 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:09.264 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.264 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.264 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:09.264 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949250+0000", 2024-09-18T04:39:09.265 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:09.265 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.265 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.265 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:09.265 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949258+0000", 2024-09-18T04:39:09.265 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0590000000000003e-06 2024-09-18T04:39:09.265 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.265 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.265 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:09.265 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949269+0000", 2024-09-18T04:39:09.266 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0607e-05 2024-09-18T04:39:09.266 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.266 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.266 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:09.266 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949277+0000", 2024-09-18T04:39:09.266 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8269999999999998e-06 2024-09-18T04:39:09.266 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.266 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.266 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:09.266 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949394+0000", 2024-09-18T04:39:09.266 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000117113 2024-09-18T04:39:09.267 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.267 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.267 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:09.267 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949527+0000", 2024-09-18T04:39:09.267 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013288300000000001 2024-09-18T04:39:09.267 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.267 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.267 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:09.267 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.959084+0000", 2024-09-18T04:39:09.267 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0095569550000000007 2024-09-18T04:39:09.267 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:09.267 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:09.268 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:09.268 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.268 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.268 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4972.0:20 2.5 2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:09.268 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.949296+0000", 2024-09-18T04:39:09.268 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.153750222999999, 2024-09-18T04:39:09.268 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014709732, 2024-09-18T04:39:09.268 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:09.269 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:09.269 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:09.269 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4972", 2024-09-18T04:39:09.269 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2280251472", 2024-09-18T04:39:09.269 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 20 2024-09-18T04:39:09.269 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.269 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:09.269 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.269 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:09.270 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949296+0000", 2024-09-18T04:39:09.270 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:09.270 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.270 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.270 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:09.270 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949296+0000", 2024-09-18T04:39:09.270 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:09.270 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.270 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.271 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:09.271 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949295+0000", 2024-09-18T04:39:09.271 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:09.271 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.271 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.271 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:09.271 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949301+0000", 2024-09-18T04:39:09.271 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4829999999999996e-06 2024-09-18T04:39:09.272 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.272 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.272 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:09.272 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949317+0000", 2024-09-18T04:39:09.272 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5866999999999999e-05 2024-09-18T04:39:09.272 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.272 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.273 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:09.273 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949331+0000", 2024-09-18T04:39:09.273 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3485999999999999e-05 2024-09-18T04:39:09.273 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.273 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.273 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:09.273 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.959137+0000", 2024-09-18T04:39:09.274 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0098066520000000008 2024-09-18T04:39:09.274 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.274 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.274 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:09.274 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.959251+0000", 2024-09-18T04:39:09.274 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000113783 2024-09-18T04:39:09.274 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.274 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.274 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:09.274 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.964005+0000", 2024-09-18T04:39:09.275 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0047542080000000002 2024-09-18T04:39:09.275 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:09.275 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:09.275 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:09.275 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.275 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.275 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4970.0:37 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [write 1966080~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:09.275 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.516050+0000", 2024-09-18T04:39:09.275 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 8.586995366, 2024-09-18T04:39:09.275 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0083751499999999996, 2024-09-18T04:39:09.276 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:09.276 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:09.276 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:09.276 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4970", 2024-09-18T04:39:09.276 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:09.277 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 37 2024-09-18T04:39:09.277 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.277 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:09.277 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.277 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:09.277 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.516050+0000", 2024-09-18T04:39:09.277 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:09.277 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.277 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.277 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:09.277 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.516050+0000", 2024-09-18T04:39:09.277 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:09.278 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.278 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.278 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:09.278 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.516050+0000", 2024-09-18T04:39:09.278 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:09.278 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.278 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.278 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:09.278 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.516150+0000", 2024-09-18T04:39:09.278 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000100159 2024-09-18T04:39:09.279 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.279 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.279 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:09.279 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.516157+0000", 2024-09-18T04:39:09.279 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7710000000000001e-06 2024-09-18T04:39:09.279 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.279 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.279 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:09.279 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.516163+0000", 2024-09-18T04:39:09.279 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.596e-06 2024-09-18T04:39:09.279 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.279 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.279 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:09.280 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.522975+0000", 2024-09-18T04:39:09.280 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0068118140000000002 2024-09-18T04:39:09.280 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.280 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.280 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:09.280 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.523028+0000", 2024-09-18T04:39:09.280 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2998999999999999e-05 2024-09-18T04:39:09.280 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.280 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.280 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:09.281 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.523107+0000", 2024-09-18T04:39:09.281 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9357000000000006e-05 2024-09-18T04:39:09.281 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.281 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.281 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:09.281 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.524167+0000", 2024-09-18T04:39:09.281 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0010592329999999999 2024-09-18T04:39:09.281 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.281 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.281 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:09.281 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.524369+0000", 2024-09-18T04:39:09.281 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00020209000000000001 2024-09-18T04:39:09.282 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.282 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.282 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:09.282 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.524402+0000", 2024-09-18T04:39:09.282 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3685e-05 2024-09-18T04:39:09.282 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:09.282 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:09.282 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:09.282 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.524426+0000", 2024-09-18T04:39:09.283 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3178000000000001e-05 2024-09-18T04:39:09.283 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:09.283 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:09.283 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:09.283 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:09.283 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:09.283 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:09.285 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:39:09.286 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:09.281+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:09.386 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:09.381+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:39:09.401 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:09.401 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:09.401 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:39:09.401 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:09.411 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:39:09.444 INFO:tasks.workunit.client.0.smithi115.stderr:++ _sudo rbd device --device-type nbd map --pool rbd --namespace ns --image testrbdnbd17342 --snap snap 2024-09-18T04:39:09.445 INFO:tasks.workunit.client.0.smithi115.stderr:++ local cmd 2024-09-18T04:39:09.445 INFO:tasks.workunit.client.0.smithi115.stderr:+++ id -u 2024-09-18T04:39:09.446 INFO:tasks.workunit.client.0.smithi115.stderr:++ '[' 1000 -eq 0 ']' 2024-09-18T04:39:09.447 INFO:tasks.workunit.client.0.smithi115.stderr:+++ which rbd 2024-09-18T04:39:09.449 INFO:tasks.workunit.client.0.smithi115.stderr:++ cmd=/usr/bin/rbd 2024-09-18T04:39:09.449 INFO:tasks.workunit.client.0.smithi115.stderr:++ shift 2024-09-18T04:39:09.449 INFO:tasks.workunit.client.0.smithi115.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map --pool rbd --namespace ns --image testrbdnbd17342 --snap snap 2024-09-18T04:39:09.487 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:09.481+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:09.518 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:09.518 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:09.518 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:39:09.518 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:39:09.518 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:09.525 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-09-18T04:39:09.528 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:39:09.588 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:09.581+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:39:09.620 INFO:tasks.workunit.client.0.smithi115.stderr:+ DEV=/dev/nbd0 2024-09-18T04:39:09.621 INFO:tasks.workunit.client.0.smithi115.stderr:+ get_pid rbd ns 2024-09-18T04:39:09.621 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pool=rbd 2024-09-18T04:39:09.621 INFO:tasks.workunit.client.0.smithi115.stderr:+ local ns=ns 2024-09-18T04:39:09.621 INFO:tasks.workunit.client.0.smithi115.stderr:++ rbd device --device-type nbd --format xml list 2024-09-18T04:39:09.622 INFO:tasks.workunit.client.0.smithi115.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\''ns'\''][image='\''testrbdnbd17342'\''][device='\''/dev/nbd0'\'']/id' 2024-09-18T04:39:09.639 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:09.639 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:39:09.640 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:39:09.640 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:39:09.640 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.640 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:39:09.640 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:39:09.640 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 101.846957199, 2024-09-18T04:39:09.640 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017399218000000001, 2024-09-18T04:39:09.640 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:09.640 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:09.640 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:09.640 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:39:09.641 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:39:09.641 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 114 2024-09-18T04:39:09.641 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.641 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:09.641 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.641 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:09.641 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:39:09.641 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:09.641 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.641 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.641 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:09.641 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:39:09.641 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:09.641 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.642 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.642 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:09.642 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786900+0000", 2024-09-18T04:39:09.642 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:09.642 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.642 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.642 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:09.642 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786908+0000", 2024-09-18T04:39:09.642 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8199999999999997e-06 2024-09-18T04:39:09.642 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.642 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.642 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:09.642 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786925+0000", 2024-09-18T04:39:09.643 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6849000000000001e-05 2024-09-18T04:39:09.643 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.643 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.643 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:09.643 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786934+0000", 2024-09-18T04:39:09.643 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6299999999999993e-06 2024-09-18T04:39:09.643 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.643 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.643 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:09.643 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800926+0000", 2024-09-18T04:39:09.643 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013991632 2024-09-18T04:39:09.643 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.643 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.644 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:09.644 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801060+0000", 2024-09-18T04:39:09.644 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013378199999999999 2024-09-18T04:39:09.644 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.644 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.644 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:09.644 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801166+0000", 2024-09-18T04:39:09.644 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010607099999999999 2024-09-18T04:39:09.644 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.644 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.644 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:09.644 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803026+0000", 2024-09-18T04:39:09.644 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001860469 2024-09-18T04:39:09.645 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.645 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.645 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:09.645 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804245+0000", 2024-09-18T04:39:09.645 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0012185900000000001 2024-09-18T04:39:09.645 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.645 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.645 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:09.645 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804278+0000", 2024-09-18T04:39:09.645 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3272000000000003e-05 2024-09-18T04:39:09.645 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.645 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.646 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:09.646 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804300+0000", 2024-09-18T04:39:09.646 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2266e-05 2024-09-18T04:39:09.646 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:09.646 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:09.646 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:09.646 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.646 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.646 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:39:09.646 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:39:09.646 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 101.84604430100001, 2024-09-18T04:39:09.646 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017521662, 2024-09-18T04:39:09.646 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:09.646 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:09.647 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:09.647 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:39:09.647 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:39:09.647 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 116 2024-09-18T04:39:09.647 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.647 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:09.647 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.647 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:09.647 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:39:09.647 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:09.647 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.647 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.647 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:09.648 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:39:09.648 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:09.648 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.648 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.648 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:09.648 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787813+0000", 2024-09-18T04:39:09.648 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:09.648 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.648 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.648 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:09.648 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787824+0000", 2024-09-18T04:39:09.648 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1157e-05 2024-09-18T04:39:09.648 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.648 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.649 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:09.649 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787840+0000", 2024-09-18T04:39:09.649 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6269e-05 2024-09-18T04:39:09.649 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.649 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.649 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:09.649 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787853+0000", 2024-09-18T04:39:09.649 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3156000000000001e-05 2024-09-18T04:39:09.649 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.649 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.649 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:09.649 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803110+0000", 2024-09-18T04:39:09.649 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015256736 2024-09-18T04:39:09.649 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.650 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.650 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:09.650 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803197+0000", 2024-09-18T04:39:09.650 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7195999999999997e-05 2024-09-18T04:39:09.650 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.650 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.650 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:09.650 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803308+0000", 2024-09-18T04:39:09.650 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111005 2024-09-18T04:39:09.650 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.650 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.650 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:09.650 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804316+0000", 2024-09-18T04:39:09.650 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010073949999999999 2024-09-18T04:39:09.650 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.651 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.651 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:09.651 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805232+0000", 2024-09-18T04:39:09.651 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091636300000000001 2024-09-18T04:39:09.651 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.651 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.651 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:09.651 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805277+0000", 2024-09-18T04:39:09.651 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4864000000000003e-05 2024-09-18T04:39:09.651 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.651 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.651 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:09.651 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805336+0000", 2024-09-18T04:39:09.651 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8968000000000002e-05 2024-09-18T04:39:09.652 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:09.652 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:09.652 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:09.652 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.652 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.652 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:39:09.652 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:39:09.652 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 101.84599224599999, 2024-09-18T04:39:09.652 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017983453, 2024-09-18T04:39:09.652 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:09.652 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:09.652 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:09.652 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:39:09.652 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:39:09.653 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 117 2024-09-18T04:39:09.653 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.653 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:09.653 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.653 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:09.653 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:39:09.653 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:09.653 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.653 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.653 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:09.653 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:39:09.653 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:09.653 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.653 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.654 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:09.654 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787865+0000", 2024-09-18T04:39:09.654 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:09.654 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.654 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.654 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:09.654 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787889+0000", 2024-09-18T04:39:09.654 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4244999999999999e-05 2024-09-18T04:39:09.654 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.654 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.654 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:09.654 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787905+0000", 2024-09-18T04:39:09.654 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5739999999999998e-05 2024-09-18T04:39:09.655 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.655 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.655 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:09.655 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787918+0000", 2024-09-18T04:39:09.655 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.252e-05 2024-09-18T04:39:09.655 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.655 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.655 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:09.655 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803693+0000", 2024-09-18T04:39:09.655 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015774872999999998 2024-09-18T04:39:09.655 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.655 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.655 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:09.655 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803775+0000", 2024-09-18T04:39:09.656 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2761000000000003e-05 2024-09-18T04:39:09.656 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.656 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.656 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:09.656 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803887+0000", 2024-09-18T04:39:09.656 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111708 2024-09-18T04:39:09.656 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.656 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.656 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:09.656 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804823+0000", 2024-09-18T04:39:09.656 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00093625100000000003 2024-09-18T04:39:09.656 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.656 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.656 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:09.657 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805804+0000", 2024-09-18T04:39:09.657 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00098114699999999992 2024-09-18T04:39:09.657 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.657 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.657 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:09.657 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805831+0000", 2024-09-18T04:39:09.657 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6401999999999999e-05 2024-09-18T04:39:09.657 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.657 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.657 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:09.657 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805850+0000", 2024-09-18T04:39:09.657 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8862e-05 2024-09-18T04:39:09.657 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:09.658 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:09.658 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:09.658 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.658 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.658 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:36 2.7 2:fe1b02ad:::rbd_data.125fc09df6d1.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:09.658 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:39:09.658 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 27.368136136, 2024-09-18T04:39:09.658 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.040387804999999999, 2024-09-18T04:39:09.658 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:09.658 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:09.658 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:09.658 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:09.658 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:09.658 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 36 2024-09-18T04:39:09.659 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.659 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:09.659 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.659 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:09.659 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:39:09.659 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:09.659 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.659 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.659 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:09.659 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:39:09.659 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:09.659 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.659 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.659 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:09.659 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265720+0000", 2024-09-18T04:39:09.659 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:09.660 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.660 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.660 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:09.660 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272590+0000", 2024-09-18T04:39:09.660 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.006870508 2024-09-18T04:39:09.660 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.660 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.660 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:09.660 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272623+0000", 2024-09-18T04:39:09.660 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2716000000000003e-05 2024-09-18T04:39:09.660 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.660 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.660 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:09.660 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272637+0000", 2024-09-18T04:39:09.661 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4175999999999999e-05 2024-09-18T04:39:09.661 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.661 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.661 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:09.661 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272779+0000", 2024-09-18T04:39:09.661 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000141392 2024-09-18T04:39:09.661 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.661 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.661 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:09.661 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273031+0000", 2024-09-18T04:39:09.661 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00025249099999999998 2024-09-18T04:39:09.661 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.661 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.661 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:39:09.662 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273219+0000", 2024-09-18T04:39:09.662 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000187531 2024-09-18T04:39:09.662 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.662 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.662 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:09.662 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.302806+0000", 2024-09-18T04:39:09.662 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.029587553999999999 2024-09-18T04:39:09.662 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.662 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.662 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:09.662 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306004+0000", 2024-09-18T04:39:09.662 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0031982629999999998 2024-09-18T04:39:09.662 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.662 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.663 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:09.663 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306071+0000", 2024-09-18T04:39:09.663 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6829999999999995e-05 2024-09-18T04:39:09.663 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.663 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.663 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:09.663 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306110+0000", 2024-09-18T04:39:09.663 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8874000000000001e-05 2024-09-18T04:39:09.663 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:09.663 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:09.663 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:09.663 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.663 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.664 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:38 2.0 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:09.664 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:39:09.664 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 27.361182598999999, 2024-09-18T04:39:09.664 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.033754348000000003, 2024-09-18T04:39:09.664 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:09.664 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:09.664 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:09.664 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:09.664 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:09.664 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 38 2024-09-18T04:39:09.664 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.664 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:09.664 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.664 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:09.665 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:39:09.665 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:09.665 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.665 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.665 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:09.665 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:39:09.665 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:09.665 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.665 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.665 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:09.665 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272673+0000", 2024-09-18T04:39:09.665 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:09.665 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.665 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.665 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:09.666 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277166+0000", 2024-09-18T04:39:09.666 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044922249999999999 2024-09-18T04:39:09.666 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.666 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.666 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:09.666 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277189+0000", 2024-09-18T04:39:09.666 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3283000000000001e-05 2024-09-18T04:39:09.666 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.666 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.666 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:09.666 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277205+0000", 2024-09-18T04:39:09.666 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6305000000000001e-05 2024-09-18T04:39:09.666 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.666 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.667 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:09.667 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277290+0000", 2024-09-18T04:39:09.667 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5135000000000004e-05 2024-09-18T04:39:09.667 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.667 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.667 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:09.667 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277531+0000", 2024-09-18T04:39:09.667 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00024014699999999999 2024-09-18T04:39:09.667 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.667 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.667 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:09.667 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277793+0000", 2024-09-18T04:39:09.667 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00026224499999999998 2024-09-18T04:39:09.667 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.668 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.668 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:09.668 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303990+0000", 2024-09-18T04:39:09.668 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.026197295999999998 2024-09-18T04:39:09.668 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.668 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.668 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:09.668 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306340+0000", 2024-09-18T04:39:09.668 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.002349524 2024-09-18T04:39:09.668 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.668 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.668 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:09.668 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306394+0000", 2024-09-18T04:39:09.668 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3934999999999998e-05 2024-09-18T04:39:09.669 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.669 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.669 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:09.669 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306430+0000", 2024-09-18T04:39:09.669 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6690000000000003e-05 2024-09-18T04:39:09.669 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:09.669 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:09.669 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:09.669 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.669 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.669 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:39 2.7 2:e9c909c5:::rbd_data.125fc09df6d1.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:09.669 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:39:09.669 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 27.356622993999999, 2024-09-18T04:39:09.669 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.029110673, 2024-09-18T04:39:09.670 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:09.670 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:09.670 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:09.670 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:09.670 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:09.670 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 39 2024-09-18T04:39:09.670 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.670 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:09.670 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.670 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:09.670 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:39:09.670 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:09.670 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.670 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.671 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:09.671 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:39:09.671 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:09.671 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.671 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.671 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:09.671 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277234+0000", 2024-09-18T04:39:09.671 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:09.671 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.671 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.671 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:09.671 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281351+0000", 2024-09-18T04:39:09.671 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0041170920000000001 2024-09-18T04:39:09.672 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.672 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.672 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:09.672 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281375+0000", 2024-09-18T04:39:09.672 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.446e-05 2024-09-18T04:39:09.672 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.672 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.672 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:09.672 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281390+0000", 2024-09-18T04:39:09.672 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4309e-05 2024-09-18T04:39:09.672 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.672 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.672 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:09.672 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.282306+0000", 2024-09-18T04:39:09.673 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091662000000000004 2024-09-18T04:39:09.673 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.673 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.673 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:09.673 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.282613+0000", 2024-09-18T04:39:09.673 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00030663799999999997 2024-09-18T04:39:09.673 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.673 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.673 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:39:09.673 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283001+0000", 2024-09-18T04:39:09.673 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00038801599999999998 2024-09-18T04:39:09.673 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.673 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.674 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:09.674 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306239+0000", 2024-09-18T04:39:09.674 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023238466999999999 2024-09-18T04:39:09.674 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.674 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.674 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:09.674 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306259+0000", 2024-09-18T04:39:09.674 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9837999999999999e-05 2024-09-18T04:39:09.674 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.674 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.674 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:09.674 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306299+0000", 2024-09-18T04:39:09.674 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0194000000000002e-05 2024-09-18T04:39:09.674 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.675 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.675 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:09.675 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306346+0000", 2024-09-18T04:39:09.675 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6845000000000003e-05 2024-09-18T04:39:09.675 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:09.675 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:09.675 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:09.675 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.675 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.675 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:46 2.6 e51/17 2:7ba25245:::rbd_data.125fc09df6d1.0000000000000006:head v 51'2, mlcod=0'0)", 2024-09-18T04:39:09.675 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:39:09.675 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 27.320701862, 2024-09-18T04:39:09.675 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019843098, 2024-09-18T04:39:09.675 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:09.676 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:09.676 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:09.676 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.676 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:09.676 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:39:09.676 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:09.676 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.676 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.676 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:09.676 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:39:09.676 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:09.676 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.676 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.676 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:09.677 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313156+0000", 2024-09-18T04:39:09.677 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:09.677 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.677 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.677 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:09.677 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320144+0000", 2024-09-18T04:39:09.677 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.006988345 2024-09-18T04:39:09.677 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.677 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.677 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:09.677 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320175+0000", 2024-09-18T04:39:09.677 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.0861999999999997e-05 2024-09-18T04:39:09.677 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.678 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.678 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:09.678 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320188+0000", 2024-09-18T04:39:09.678 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2714999999999999e-05 2024-09-18T04:39:09.678 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.678 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.678 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:09.678 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320267+0000", 2024-09-18T04:39:09.678 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9586999999999995e-05 2024-09-18T04:39:09.678 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.678 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.678 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:09.678 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320311+0000", 2024-09-18T04:39:09.678 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3529000000000003e-05 2024-09-18T04:39:09.679 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.679 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.679 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:09.679 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.332949+0000", 2024-09-18T04:39:09.679 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012638267999999999 2024-09-18T04:39:09.679 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.679 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.679 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:09.679 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.333000+0000", 2024-09-18T04:39:09.679 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0736999999999999e-05 2024-09-18T04:39:09.679 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:09.679 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:09.679 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:09.679 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.680 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.680 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:57 2.0 2:0cdd070e:::rbd_data.125fc09df6d1.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:09.680 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:39:09.680 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 27.284844286999999, 2024-09-18T04:39:09.680 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.033329974999999998, 2024-09-18T04:39:09.680 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:09.680 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:09.680 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:09.680 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:09.680 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:09.680 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 57 2024-09-18T04:39:09.680 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.680 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:09.680 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.681 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:09.681 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:39:09.681 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:09.681 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.681 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.681 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:09.681 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:39:09.681 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:09.681 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.681 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.681 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:09.681 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349013+0000", 2024-09-18T04:39:09.681 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:09.682 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.682 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.682 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:09.682 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356138+0000", 2024-09-18T04:39:09.682 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071258010000000002 2024-09-18T04:39:09.682 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.682 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.682 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:09.682 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356168+0000", 2024-09-18T04:39:09.682 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9679000000000002e-05 2024-09-18T04:39:09.682 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.682 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.682 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:09.682 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356181+0000", 2024-09-18T04:39:09.682 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3164e-05 2024-09-18T04:39:09.683 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.683 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.683 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:09.683 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356254+0000", 2024-09-18T04:39:09.683 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3182999999999994e-05 2024-09-18T04:39:09.683 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.683 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.683 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:09.683 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356453+0000", 2024-09-18T04:39:09.683 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019876599999999999 2024-09-18T04:39:09.683 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.683 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.683 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:09.684 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356627+0000", 2024-09-18T04:39:09.684 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017340199999999999 2024-09-18T04:39:09.684 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.684 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.684 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:09.684 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.369200+0000", 2024-09-18T04:39:09.684 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012573302 2024-09-18T04:39:09.684 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.684 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.684 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:09.684 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382262+0000", 2024-09-18T04:39:09.684 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013062549 2024-09-18T04:39:09.684 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.684 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.685 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:09.685 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382310+0000", 2024-09-18T04:39:09.685 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7366e-05 2024-09-18T04:39:09.685 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.685 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.685 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:09.685 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382344+0000", 2024-09-18T04:39:09.685 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4319999999999997e-05 2024-09-18T04:39:09.685 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:09.685 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:09.685 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:09.685 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.685 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.685 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:59 2.0 2:19c52cde:::rbd_data.125fc09df6d1.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:09.686 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:39:09.686 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 27.277653122, 2024-09-18T04:39:09.686 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.027137247999999999, 2024-09-18T04:39:09.686 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:09.686 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:09.686 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:09.686 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:09.686 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:09.686 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 59 2024-09-18T04:39:09.686 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.686 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:09.686 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.686 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:09.686 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:39:09.687 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:09.687 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.687 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.687 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:09.687 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:39:09.687 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:09.687 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.687 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.687 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:09.687 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356204+0000", 2024-09-18T04:39:09.688 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:09.688 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.688 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.688 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:09.688 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362661+0000", 2024-09-18T04:39:09.688 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0064565129999999997 2024-09-18T04:39:09.688 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.688 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.688 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:09.688 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362684+0000", 2024-09-18T04:39:09.688 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3688000000000001e-05 2024-09-18T04:39:09.688 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.688 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.688 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:09.689 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362699+0000", 2024-09-18T04:39:09.689 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4798e-05 2024-09-18T04:39:09.689 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.689 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.689 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:09.689 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363016+0000", 2024-09-18T04:39:09.689 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031737300000000003 2024-09-18T04:39:09.689 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.689 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.689 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:09.689 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363201+0000", 2024-09-18T04:39:09.689 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000184131 2024-09-18T04:39:09.689 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.689 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.690 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:09.690 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363343+0000", 2024-09-18T04:39:09.690 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014236699999999999 2024-09-18T04:39:09.690 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.690 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.690 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:09.690 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.372189+0000", 2024-09-18T04:39:09.690 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0088464819999999993 2024-09-18T04:39:09.690 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.690 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.690 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:09.690 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383269+0000", 2024-09-18T04:39:09.690 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011079594 2024-09-18T04:39:09.690 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.691 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.691 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:09.691 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383314+0000", 2024-09-18T04:39:09.691 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4932000000000003e-05 2024-09-18T04:39:09.691 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.691 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.691 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:09.691 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383343+0000", 2024-09-18T04:39:09.691 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8600999999999999e-05 2024-09-18T04:39:09.691 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:09.691 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:09.691 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:09.691 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.691 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.692 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:40 2.6 e51/17 2:784d6cd1:ns::rbd_data.133e813ab5bc.0000000000000005:head v 51'7, mlcod=51'5)", 2024-09-18T04:39:09.692 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:39:09.692 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 27.081403387999998, 2024-09-18T04:39:09.692 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020497496, 2024-09-18T04:39:09.692 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:09.692 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:09.692 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:09.692 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.692 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:09.692 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:39:09.692 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:09.692 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.692 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.693 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:09.693 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:39:09.693 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:09.693 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.693 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.693 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:09.693 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552454+0000", 2024-09-18T04:39:09.693 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:09.693 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.693 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.693 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:09.693 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556448+0000", 2024-09-18T04:39:09.693 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0039940990000000001 2024-09-18T04:39:09.693 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.693 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.694 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:09.694 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556481+0000", 2024-09-18T04:39:09.694 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3778000000000001e-05 2024-09-18T04:39:09.694 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.694 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.694 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:09.694 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556493+0000", 2024-09-18T04:39:09.694 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1622e-05 2024-09-18T04:39:09.694 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.694 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.694 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:09.694 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556556+0000", 2024-09-18T04:39:09.694 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2582999999999994e-05 2024-09-18T04:39:09.694 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.695 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.695 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:09.695 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556592+0000", 2024-09-18T04:39:09.695 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5880999999999997e-05 2024-09-18T04:39:09.695 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.695 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.695 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:09.695 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.572786+0000", 2024-09-18T04:39:09.695 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016194571000000001 2024-09-18T04:39:09.695 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.695 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.695 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:09.695 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.572953+0000", 2024-09-18T04:39:09.695 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000166436 2024-09-18T04:39:09.696 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:09.696 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:09.696 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:09.696 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.696 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.696 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:45 2.0 2:0fbfa3dc:ns::rbd_data.133e813ab5bc.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:09.696 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:39:09.696 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 27.070189088999999, 2024-09-18T04:39:09.696 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.036558168000000002, 2024-09-18T04:39:09.696 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:09.696 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:09.696 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:09.696 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:09.696 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:09.697 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 45 2024-09-18T04:39:09.697 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.697 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:09.697 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.697 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:09.697 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:39:09.697 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:09.697 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.697 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.697 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:09.697 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:39:09.697 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:09.697 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.698 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.698 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:09.698 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:39:09.698 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:09.698 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.698 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.698 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:09.698 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578594+0000", 2024-09-18T04:39:09.698 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014927147999999999 2024-09-18T04:39:09.698 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.698 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.698 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:09.698 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578612+0000", 2024-09-18T04:39:09.698 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7623999999999999e-05 2024-09-18T04:39:09.698 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.699 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.699 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:09.699 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578620+0000", 2024-09-18T04:39:09.699 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9070000000000007e-06 2024-09-18T04:39:09.699 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.699 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.699 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:09.699 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578702+0000", 2024-09-18T04:39:09.699 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2349999999999996e-05 2024-09-18T04:39:09.699 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.699 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.699 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:09.699 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.579028+0000", 2024-09-18T04:39:09.700 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032622899999999998 2024-09-18T04:39:09.700 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.700 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.700 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:09.700 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.579343+0000", 2024-09-18T04:39:09.700 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031448899999999999 2024-09-18T04:39:09.700 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.700 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.700 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:09.700 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.590078+0000", 2024-09-18T04:39:09.700 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010734946 2024-09-18T04:39:09.700 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.700 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.701 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:09.701 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600089+0000", 2024-09-18T04:39:09.701 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010011195000000001 2024-09-18T04:39:09.701 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.701 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.701 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:09.701 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600177+0000", 2024-09-18T04:39:09.701 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8425999999999997e-05 2024-09-18T04:39:09.701 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.701 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.701 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:09.701 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600228+0000", 2024-09-18T04:39:09.701 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0216999999999997e-05 2024-09-18T04:39:09.701 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:09.701 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:09.702 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:09.702 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.702 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.702 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:54 2.0 2:04f17ef6:ns::rbd_data.133e813ab5bc.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:09.702 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:39:09.702 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 27.048617252, 2024-09-18T04:39:09.702 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.024612999, 2024-09-18T04:39:09.702 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:09.702 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:09.702 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:09.702 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:09.702 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:09.702 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 54 2024-09-18T04:39:09.703 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.703 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:09.703 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.703 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:09.703 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:39:09.703 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:09.703 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.703 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.703 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:09.703 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:39:09.703 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:09.703 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.703 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.703 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:09.704 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585239+0000", 2024-09-18T04:39:09.704 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:09.704 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.704 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.704 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:09.704 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593702+0000", 2024-09-18T04:39:09.704 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0084632990000000005 2024-09-18T04:39:09.704 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.704 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.704 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:09.704 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593719+0000", 2024-09-18T04:39:09.704 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.664e-05 2024-09-18T04:39:09.704 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.705 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.705 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:09.705 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593727+0000", 2024-09-18T04:39:09.705 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6839999999999996e-06 2024-09-18T04:39:09.705 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.705 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.705 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:09.705 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593767+0000", 2024-09-18T04:39:09.705 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0330999999999997e-05 2024-09-18T04:39:09.705 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.705 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.705 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:09.705 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593905+0000", 2024-09-18T04:39:09.705 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013754800000000001 2024-09-18T04:39:09.706 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.706 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.706 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:09.706 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.594013+0000", 2024-09-18T04:39:09.706 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000108805 2024-09-18T04:39:09.706 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.706 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.706 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:09.706 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604678+0000", 2024-09-18T04:39:09.706 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010664391 2024-09-18T04:39:09.706 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.706 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.706 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:09.707 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609662+0000", 2024-09-18T04:39:09.707 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0049846100000000004 2024-09-18T04:39:09.707 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.707 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.707 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:09.707 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609788+0000", 2024-09-18T04:39:09.707 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012518800000000001 2024-09-18T04:39:09.707 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.707 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.707 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:09.707 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609854+0000", 2024-09-18T04:39:09.707 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6556000000000004e-05 2024-09-18T04:39:09.707 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:09.707 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:09.708 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:09.708 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.708 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.708 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:58 2.7 2:fc7c6cd3:ns::rbd_data.133e813ab5bc.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:09.708 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:39:09.708 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 27.035774823000001, 2024-09-18T04:39:09.708 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.027690566999999999, 2024-09-18T04:39:09.708 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:09.708 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:09.708 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:09.708 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:09.708 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:09.708 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 58 2024-09-18T04:39:09.709 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.709 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:09.709 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.709 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:09.709 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:39:09.709 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:09.709 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.709 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.709 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:09.709 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:39:09.709 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:09.709 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.709 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.710 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:09.710 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598082+0000", 2024-09-18T04:39:09.710 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:09.710 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.710 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.710 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:09.710 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604832+0000", 2024-09-18T04:39:09.710 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0067504510000000002 2024-09-18T04:39:09.710 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.710 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.710 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:09.710 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604852+0000", 2024-09-18T04:39:09.710 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0228999999999999e-05 2024-09-18T04:39:09.710 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.711 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.711 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:09.711 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604870+0000", 2024-09-18T04:39:09.711 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7952e-05 2024-09-18T04:39:09.711 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.711 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.711 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:09.711 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604936+0000", 2024-09-18T04:39:09.711 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5431000000000004e-05 2024-09-18T04:39:09.711 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.711 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.711 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:09.711 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605085+0000", 2024-09-18T04:39:09.711 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014910300000000001 2024-09-18T04:39:09.712 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.712 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.712 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:39:09.712 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605205+0000", 2024-09-18T04:39:09.712 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000119887 2024-09-18T04:39:09.712 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.712 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.712 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:09.712 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625586+0000", 2024-09-18T04:39:09.712 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020381399000000001 2024-09-18T04:39:09.712 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.712 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.712 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:09.713 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625628+0000", 2024-09-18T04:39:09.713 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1873999999999999e-05 2024-09-18T04:39:09.713 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.713 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.713 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:09.713 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625722+0000", 2024-09-18T04:39:09.713 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3787000000000002e-05 2024-09-18T04:39:09.713 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.713 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.713 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:09.713 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625774+0000", 2024-09-18T04:39:09.713 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2503999999999997e-05 2024-09-18T04:39:09.713 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:09.714 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:09.714 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:09.714 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.714 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.714 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:56 2.6 e51/17 2:72453c3e:ns::rbd_data.133e813ab5bc.000000000000000d:head v 51'10, mlcod=51'8)", 2024-09-18T04:39:09.714 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:39:09.714 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 27.028577715000001, 2024-09-18T04:39:09.714 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019766552999999999, 2024-09-18T04:39:09.714 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:09.714 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:09.714 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:09.714 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.714 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:09.715 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:39:09.715 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:09.715 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.715 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.715 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:09.715 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:39:09.715 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:09.715 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.715 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.715 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:09.715 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605279+0000", 2024-09-18T04:39:09.715 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:09.715 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.715 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.716 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:09.716 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609417+0000", 2024-09-18T04:39:09.716 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0041378059999999999 2024-09-18T04:39:09.716 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.716 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.716 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:09.716 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609443+0000", 2024-09-18T04:39:09.716 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6098e-05 2024-09-18T04:39:09.716 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.716 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.716 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:09.716 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609458+0000", 2024-09-18T04:39:09.716 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4671000000000001e-05 2024-09-18T04:39:09.716 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.717 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.717 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:09.717 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609553+0000", 2024-09-18T04:39:09.717 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5265000000000004e-05 2024-09-18T04:39:09.717 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.717 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.717 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:09.717 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609583+0000", 2024-09-18T04:39:09.717 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9839e-05 2024-09-18T04:39:09.717 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.717 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.717 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:09.717 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625004+0000", 2024-09-18T04:39:09.717 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015420639999999999 2024-09-18T04:39:09.718 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.718 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.718 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:09.718 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625047+0000", 2024-09-18T04:39:09.718 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3665999999999998e-05 2024-09-18T04:39:09.718 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:09.718 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:09.718 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:09.718 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.718 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.718 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:99 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:09.718 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.582718+0000", 2024-09-18T04:39:09.718 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 13.051140805999999, 2024-09-18T04:39:09.719 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019362575, 2024-09-18T04:39:09.719 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:09.719 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:09.719 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:09.719 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:09.719 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:09.719 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 99 2024-09-18T04:39:09.719 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.719 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:09.719 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.719 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:09.719 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582718+0000", 2024-09-18T04:39:09.719 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:09.720 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.720 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.720 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:09.720 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582718+0000", 2024-09-18T04:39:09.720 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:09.720 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.720 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.720 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:09.720 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582717+0000", 2024-09-18T04:39:09.720 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:09.720 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.720 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.720 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:09.721 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582720+0000", 2024-09-18T04:39:09.721 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2239999999999998e-06 2024-09-18T04:39:09.721 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.721 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.721 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:09.721 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582723+0000", 2024-09-18T04:39:09.721 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.258e-06 2024-09-18T04:39:09.721 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.721 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.721 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:09.721 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582726+0000", 2024-09-18T04:39:09.721 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8870000000000001e-06 2024-09-18T04:39:09.721 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.721 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.722 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:09.722 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.592018+0000", 2024-09-18T04:39:09.722 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0092926420000000003 2024-09-18T04:39:09.722 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.722 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.722 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:09.722 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.592214+0000", 2024-09-18T04:39:09.722 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000195267 2024-09-18T04:39:09.722 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.722 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.722 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:09.722 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.592913+0000", 2024-09-18T04:39:09.722 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00069924600000000005 2024-09-18T04:39:09.723 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.723 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.723 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:09.723 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597493+0000", 2024-09-18T04:39:09.723 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045797629999999997 2024-09-18T04:39:09.723 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.723 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.723 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:09.723 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.602009+0000", 2024-09-18T04:39:09.723 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045159559999999998 2024-09-18T04:39:09.723 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.723 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.723 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:09.724 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.602042+0000", 2024-09-18T04:39:09.724 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3084999999999999e-05 2024-09-18T04:39:09.724 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.724 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.724 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:09.724 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.602080+0000", 2024-09-18T04:39:09.724 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8600000000000003e-05 2024-09-18T04:39:09.724 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:09.724 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:09.724 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:09.724 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.724 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.724 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:104 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:09.725 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.583165+0000", 2024-09-18T04:39:09.725 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 13.050693508, 2024-09-18T04:39:09.725 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023679603, 2024-09-18T04:39:09.725 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:09.725 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:09.725 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:09.725 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:09.725 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:09.725 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 104 2024-09-18T04:39:09.725 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.725 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:09.725 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.725 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:09.725 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583165+0000", 2024-09-18T04:39:09.726 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:09.726 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.726 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.726 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:09.726 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583165+0000", 2024-09-18T04:39:09.726 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:09.726 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.726 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.726 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:09.726 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583164+0000", 2024-09-18T04:39:09.726 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:09.726 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.726 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.727 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:09.727 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583167+0000", 2024-09-18T04:39:09.727 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9210000000000001e-06 2024-09-18T04:39:09.727 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.727 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.727 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:09.727 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583171+0000", 2024-09-18T04:39:09.727 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9759999999999997e-06 2024-09-18T04:39:09.727 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.727 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.727 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:09.727 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583174+0000", 2024-09-18T04:39:09.727 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9960000000000002e-06 2024-09-18T04:39:09.727 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.727 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.728 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:09.728 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.594373+0000", 2024-09-18T04:39:09.728 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011198665999999999 2024-09-18T04:39:09.728 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.728 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.728 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:09.728 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.594434+0000", 2024-09-18T04:39:09.728 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.1446999999999996e-05 2024-09-18T04:39:09.728 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.728 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.728 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:09.728 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.594692+0000", 2024-09-18T04:39:09.728 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00025756599999999998 2024-09-18T04:39:09.728 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.729 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.729 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:09.729 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599302+0000", 2024-09-18T04:39:09.729 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0046099640000000002 2024-09-18T04:39:09.729 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.729 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.729 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:09.729 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606710+0000", 2024-09-18T04:39:09.729 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0074077739999999998 2024-09-18T04:39:09.729 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.729 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.729 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:09.729 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606798+0000", 2024-09-18T04:39:09.729 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8133999999999993e-05 2024-09-18T04:39:09.730 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.730 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.730 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:09.730 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606845+0000", 2024-09-18T04:39:09.730 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6653000000000002e-05 2024-09-18T04:39:09.730 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:09.730 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:09.730 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:09.730 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.730 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.730 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:105 2.0 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:09.730 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.583193+0000", 2024-09-18T04:39:09.730 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 13.050665888999999, 2024-09-18T04:39:09.730 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023713649, 2024-09-18T04:39:09.731 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:09.731 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:09.731 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:09.731 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:09.731 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:09.731 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 105 2024-09-18T04:39:09.731 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.731 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:09.731 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.731 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:09.731 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583193+0000", 2024-09-18T04:39:09.731 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:09.731 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.731 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.732 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:09.732 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583193+0000", 2024-09-18T04:39:09.732 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:09.732 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.732 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.732 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:09.732 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583192+0000", 2024-09-18T04:39:09.732 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:09.732 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.732 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.732 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:09.732 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583194+0000", 2024-09-18T04:39:09.732 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.305e-06 2024-09-18T04:39:09.732 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.733 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.733 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:09.733 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583198+0000", 2024-09-18T04:39:09.733 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9369999999999997e-06 2024-09-18T04:39:09.733 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.733 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.733 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:09.733 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583202+0000", 2024-09-18T04:39:09.733 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6480000000000001e-06 2024-09-18T04:39:09.733 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.733 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.733 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:09.733 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.595304+0000", 2024-09-18T04:39:09.733 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012102063999999999 2024-09-18T04:39:09.734 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.734 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.734 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:09.734 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.595390+0000", 2024-09-18T04:39:09.734 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5493000000000003e-05 2024-09-18T04:39:09.734 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.734 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.734 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:09.734 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.595509+0000", 2024-09-18T04:39:09.734 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011966199999999999 2024-09-18T04:39:09.734 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.734 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.734 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:09.734 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599358+0000", 2024-09-18T04:39:09.735 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0038486689999999999 2024-09-18T04:39:09.735 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.735 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.735 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:09.735 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606859+0000", 2024-09-18T04:39:09.735 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0075010789999999999 2024-09-18T04:39:09.735 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.735 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.735 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:09.735 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606887+0000", 2024-09-18T04:39:09.735 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7773e-05 2024-09-18T04:39:09.735 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.735 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.735 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:09.736 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606906+0000", 2024-09-18T04:39:09.736 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9417000000000001e-05 2024-09-18T04:39:09.736 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:09.736 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:09.736 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:09.736 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.736 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.736 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:106 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:09.736 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.584425+0000", 2024-09-18T04:39:09.736 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 13.049433593, 2024-09-18T04:39:09.736 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.022529293999999998, 2024-09-18T04:39:09.736 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:09.736 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:09.737 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:09.737 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:09.737 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:09.737 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 106 2024-09-18T04:39:09.737 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.737 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:09.737 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.737 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:09.737 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584425+0000", 2024-09-18T04:39:09.737 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:09.737 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.737 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.737 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:09.737 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584425+0000", 2024-09-18T04:39:09.737 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:09.738 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.738 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.738 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:09.738 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584424+0000", 2024-09-18T04:39:09.738 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:09.738 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.738 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.738 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:09.738 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584430+0000", 2024-09-18T04:39:09.738 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6389999999999997e-06 2024-09-18T04:39:09.738 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.738 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.738 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:09.739 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584456+0000", 2024-09-18T04:39:09.739 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6188999999999999e-05 2024-09-18T04:39:09.739 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.739 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.739 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:09.739 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584463+0000", 2024-09-18T04:39:09.739 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7340000000000002e-06 2024-09-18T04:39:09.739 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.739 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.739 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:09.739 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597537+0000", 2024-09-18T04:39:09.739 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013074161000000001 2024-09-18T04:39:09.739 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.739 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.740 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:09.740 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597598+0000", 2024-09-18T04:39:09.740 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0930000000000001e-05 2024-09-18T04:39:09.740 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.740 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.740 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:09.740 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597922+0000", 2024-09-18T04:39:09.740 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032473000000000002 2024-09-18T04:39:09.740 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.740 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.740 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:09.740 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.601103+0000", 2024-09-18T04:39:09.740 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0031809849999999999 2024-09-18T04:39:09.740 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.741 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.741 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:09.741 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606912+0000", 2024-09-18T04:39:09.741 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058083550000000003 2024-09-18T04:39:09.741 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.741 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.741 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:09.741 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606935+0000", 2024-09-18T04:39:09.741 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2759999999999999e-05 2024-09-18T04:39:09.741 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.741 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.741 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:09.741 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606954+0000", 2024-09-18T04:39:09.742 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9602999999999998e-05 2024-09-18T04:39:09.742 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:09.742 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:09.742 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:09.742 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.742 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.742 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:109 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:09.742 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.585231+0000", 2024-09-18T04:39:09.742 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 13.048626972999999, 2024-09-18T04:39:09.742 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.021767496000000001, 2024-09-18T04:39:09.742 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:09.742 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:09.742 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:09.742 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:09.743 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:09.743 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 109 2024-09-18T04:39:09.743 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.743 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:09.743 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.743 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:09.743 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585231+0000", 2024-09-18T04:39:09.743 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:09.743 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.743 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.743 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:09.743 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585231+0000", 2024-09-18T04:39:09.743 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:09.743 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.744 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.744 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:09.744 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585230+0000", 2024-09-18T04:39:09.744 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:09.744 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.744 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.744 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:09.744 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585237+0000", 2024-09-18T04:39:09.744 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.618e-06 2024-09-18T04:39:09.744 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.744 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.744 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:09.744 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585256+0000", 2024-09-18T04:39:09.744 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9048000000000001e-05 2024-09-18T04:39:09.745 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.745 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.745 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:09.745 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585274+0000", 2024-09-18T04:39:09.745 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8153000000000001e-05 2024-09-18T04:39:09.745 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.745 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.745 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:09.745 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.598284+0000", 2024-09-18T04:39:09.745 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013009682999999999 2024-09-18T04:39:09.745 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.745 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.745 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:09.746 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.598383+0000", 2024-09-18T04:39:09.746 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.8722999999999997e-05 2024-09-18T04:39:09.746 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.746 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.746 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:09.746 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.598829+0000", 2024-09-18T04:39:09.746 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00044623999999999998 2024-09-18T04:39:09.746 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.746 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.746 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:09.746 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.601149+0000", 2024-09-18T04:39:09.746 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0023204150000000002 2024-09-18T04:39:09.746 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.746 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.747 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:09.747 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606960+0000", 2024-09-18T04:39:09.747 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058106390000000003 2024-09-18T04:39:09.747 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.747 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.747 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:09.747 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606982+0000", 2024-09-18T04:39:09.747 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2252e-05 2024-09-18T04:39:09.747 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.747 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.747 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:09.747 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606999+0000", 2024-09-18T04:39:09.747 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6871000000000002e-05 2024-09-18T04:39:09.747 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:09.748 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:09.748 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:09.748 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.748 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.748 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:110 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:09.748 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.586640+0000", 2024-09-18T04:39:09.748 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 13.047218654, 2024-09-18T04:39:09.748 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020400911000000001, 2024-09-18T04:39:09.748 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:09.748 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:09.748 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:09.748 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:09.748 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:09.748 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 110 2024-09-18T04:39:09.749 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.749 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:09.749 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.749 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:09.749 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586640+0000", 2024-09-18T04:39:09.749 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:09.749 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.749 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.749 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:09.749 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586640+0000", 2024-09-18T04:39:09.749 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:09.749 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.749 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.749 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:09.750 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586639+0000", 2024-09-18T04:39:09.750 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:09.750 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.750 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.750 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:09.750 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586652+0000", 2024-09-18T04:39:09.750 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2482e-05 2024-09-18T04:39:09.750 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.750 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.750 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:09.750 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586660+0000", 2024-09-18T04:39:09.750 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1659999999999994e-06 2024-09-18T04:39:09.750 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.750 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.751 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:09.751 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586666+0000", 2024-09-18T04:39:09.751 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6900000000000003e-06 2024-09-18T04:39:09.751 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.751 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.751 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:09.751 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599381+0000", 2024-09-18T04:39:09.751 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012714328 2024-09-18T04:39:09.751 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.751 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.751 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:09.751 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599436+0000", 2024-09-18T04:39:09.751 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4802999999999997e-05 2024-09-18T04:39:09.751 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.752 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.752 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:09.752 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599629+0000", 2024-09-18T04:39:09.752 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000193921 2024-09-18T04:39:09.752 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.752 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.752 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:09.752 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.601174+0000", 2024-09-18T04:39:09.752 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001544857 2024-09-18T04:39:09.752 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.752 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.752 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:09.752 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.607004+0000", 2024-09-18T04:39:09.753 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058300820000000003 2024-09-18T04:39:09.753 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.753 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.753 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:09.753 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.607024+0000", 2024-09-18T04:39:09.753 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9888e-05 2024-09-18T04:39:09.753 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:09.753 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:09.753 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:09.753 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.607041+0000", 2024-09-18T04:39:09.753 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6450999999999999e-05 2024-09-18T04:39:09.753 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:09.753 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:09.753 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:09.754 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:09.754 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:09.754 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:09.755 INFO:tasks.workunit.client.0.smithi115.stderr:+ PID=18995 2024-09-18T04:39:09.755 INFO:tasks.workunit.client.0.smithi115.stderr:+ test -n 18995 2024-09-18T04:39:09.755 INFO:tasks.workunit.client.0.smithi115.stderr:+ ps -p 18995 -C rbd-nbd 2024-09-18T04:39:09.755 INFO:tasks.workunit.client.0.smithi115.stdout: PID TTY TIME CMD 2024-09-18T04:39:09.756 INFO:tasks.workunit.client.0.smithi115.stdout: 18995 ? 00:00:00 rbd-nbd 2024-09-18T04:39:09.756 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:39:09.758 INFO:tasks.workunit.client.0.smithi115.stderr:+ unmap_device '--pool rbd --namespace ns --image testrbdnbd17342 --snap snap' 18995 2024-09-18T04:39:09.758 INFO:tasks.workunit.client.0.smithi115.stderr:+ local 'args=--pool rbd --namespace ns --image testrbdnbd17342 --snap snap' 2024-09-18T04:39:09.758 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pid=18995 2024-09-18T04:39:09.758 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo rbd device --device-type nbd unmap --pool rbd --namespace ns --image testrbdnbd17342 --snap snap 2024-09-18T04:39:09.758 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:39:09.758 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:39:09.758 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:39:09.758 INFO:tasks.workunit.client.0.smithi115.stderr:++ which rbd 2024-09-18T04:39:09.758 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/bin/rbd 2024-09-18T04:39:09.758 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:39:09.758 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap --pool rbd --namespace ns --image testrbdnbd17342 --snap snap 2024-09-18T04:39:09.759 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:09.749+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:09.825 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd device --device-type nbd list 2024-09-18T04:39:09.825 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false grep '^18995\b' 2024-09-18T04:39:09.825 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep '^18995\b' 2024-09-18T04:39:09.856 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:09.849+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:39:09.871 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:09.871 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:09.871 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:39:09.871 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:09.881 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:39:09.890 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:39:09.891 INFO:tasks.workunit.client.0.smithi115.stderr:+ ps -C rbd-nbd 2024-09-18T04:39:09.891 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false grep '^ *18995\b' 2024-09-18T04:39:09.891 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep '^ *18995\b' 2024-09-18T04:39:09.901 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:39:09.901 INFO:tasks.workunit.client.0.smithi115.stderr:+ sleep 0.5 2024-09-18T04:39:09.957 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:09.949+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:09.990 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:09.990 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:09.990 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:39:09.990 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:39:09.991 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:10.000 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:39:10.058 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:10.053+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:39:10.108 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:10.108 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:39:10.108 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:39:10.108 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [] 2024-09-18T04:39:10.108 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:10.118 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:39:10.162 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:10.157+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:10.222 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:10.222 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:10.222 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:39:10.222 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:10.231 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:39:10.260 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:10.253+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:10.341 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:10.342 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:10.342 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:39:10.342 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:39:10.342 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:10.351 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:39:10.361 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:10.357+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:39:10.403 INFO:tasks.workunit.client.0.smithi115.stderr:+ DEV= 2024-09-18T04:39:10.404 INFO:tasks.workunit.client.0.smithi115.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd17342 2024-09-18T04:39:10.404 INFO:tasks.workunit.client.0.smithi115.stderr:++ local cmd 2024-09-18T04:39:10.404 INFO:tasks.workunit.client.0.smithi115.stderr:+++ id -u 2024-09-18T04:39:10.406 INFO:tasks.workunit.client.0.smithi115.stderr:++ '[' 1000 -eq 0 ']' 2024-09-18T04:39:10.407 INFO:tasks.workunit.client.0.smithi115.stderr:+++ which rbd 2024-09-18T04:39:10.409 INFO:tasks.workunit.client.0.smithi115.stderr:++ cmd=/usr/bin/rbd 2024-09-18T04:39:10.410 INFO:tasks.workunit.client.0.smithi115.stderr:++ shift 2024-09-18T04:39:10.410 INFO:tasks.workunit.client.0.smithi115.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd17342 2024-09-18T04:39:10.462 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:10.462 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:39:10.462 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:39:10.462 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:39:10.462 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.462 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-18T04:39:10.463 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:39:10.463 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 95.986437180999999, 2024-09-18T04:39:10.463 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0095781189999999995, 2024-09-18T04:39:10.463 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:10.463 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:10.463 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:10.463 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:39:10.463 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:39:10.463 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 1 2024-09-18T04:39:10.463 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.463 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:10.463 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.463 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:10.464 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:39:10.464 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:10.464 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.464 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.464 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:10.464 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:39:10.464 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:10.464 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.464 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.464 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:10.464 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469862+0000", 2024-09-18T04:39:10.464 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:10.464 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.465 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.465 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:10.465 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469879+0000", 2024-09-18T04:39:10.465 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7431999999999998e-05 2024-09-18T04:39:10.465 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.465 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.465 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:10.465 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469896+0000", 2024-09-18T04:39:10.465 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6742999999999999e-05 2024-09-18T04:39:10.465 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.465 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.465 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:10.465 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469915+0000", 2024-09-18T04:39:10.465 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8991000000000002e-05 2024-09-18T04:39:10.466 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.466 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.466 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:10.466 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470397+0000", 2024-09-18T04:39:10.466 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048167799999999998 2024-09-18T04:39:10.466 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.466 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.466 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-09-18T04:39:10.466 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470412+0000", 2024-09-18T04:39:10.466 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5526e-05 2024-09-18T04:39:10.466 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.466 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.466 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:10.466 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473279+0000", 2024-09-18T04:39:10.467 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0028667469999999998 2024-09-18T04:39:10.467 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.467 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.467 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:10.467 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473441+0000", 2024-09-18T04:39:10.467 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000162082 2024-09-18T04:39:10.467 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.467 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.467 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:10.467 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473573+0000", 2024-09-18T04:39:10.467 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000132032 2024-09-18T04:39:10.467 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.467 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.468 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:10.468 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474826+0000", 2024-09-18T04:39:10.468 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001252717 2024-09-18T04:39:10.468 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.468 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.468 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:10.468 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479307+0000", 2024-09-18T04:39:10.468 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044813220000000003 2024-09-18T04:39:10.468 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.468 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.468 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:10.468 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479366+0000", 2024-09-18T04:39:10.468 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9024999999999998e-05 2024-09-18T04:39:10.468 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.469 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.469 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:10.469 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479443+0000", 2024-09-18T04:39:10.469 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6339000000000002e-05 2024-09-18T04:39:10.469 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:10.469 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:10.469 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:10.469 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.469 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.469 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-18T04:39:10.469 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:39:10.469 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 93.973428760999994, 2024-09-18T04:39:10.469 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082699560000000002, 2024-09-18T04:39:10.469 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:10.470 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:10.470 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:10.470 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:39:10.470 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:39:10.470 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 7 2024-09-18T04:39:10.470 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.470 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:10.470 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.470 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:10.470 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:39:10.470 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:10.470 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.470 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.471 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:10.471 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:39:10.471 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:10.471 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.471 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.471 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:10.471 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482870+0000", 2024-09-18T04:39:10.471 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999967 2024-09-18T04:39:10.471 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.471 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.471 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:10.471 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482889+0000", 2024-09-18T04:39:10.471 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.965e-05 2024-09-18T04:39:10.471 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.472 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.472 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:10.472 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482938+0000", 2024-09-18T04:39:10.472 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8548999999999999e-05 2024-09-18T04:39:10.472 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.472 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.472 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:10.472 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482963+0000", 2024-09-18T04:39:10.472 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5075999999999998e-05 2024-09-18T04:39:10.472 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.472 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.472 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:10.472 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483121+0000", 2024-09-18T04:39:10.472 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015819400000000001 2024-09-18T04:39:10.473 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.473 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.473 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op must wait for map", 2024-09-18T04:39:10.473 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483138+0000", 2024-09-18T04:39:10.473 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7068999999999999e-05 2024-09-18T04:39:10.473 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.473 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.473 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:10.473 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488666+0000", 2024-09-18T04:39:10.473 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0055282049999999996 2024-09-18T04:39:10.473 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.473 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.473 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:10.473 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488783+0000", 2024-09-18T04:39:10.474 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000116737 2024-09-18T04:39:10.474 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.474 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.474 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:10.474 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488991+0000", 2024-09-18T04:39:10.474 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020781400000000001 2024-09-18T04:39:10.474 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.474 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.474 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:10.474 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490554+0000", 2024-09-18T04:39:10.474 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0015628370000000001 2024-09-18T04:39:10.474 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.474 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.475 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:10.475 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491019+0000", 2024-09-18T04:39:10.475 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000465608 2024-09-18T04:39:10.475 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.475 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.475 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:10.475 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491087+0000", 2024-09-18T04:39:10.475 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7988e-05 2024-09-18T04:39:10.475 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.475 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.475 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:10.475 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491143+0000", 2024-09-18T04:39:10.476 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5538999999999998e-05 2024-09-18T04:39:10.476 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:10.476 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:10.476 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:10.476 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.476 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.476 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:34 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:10.476 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.261758+0000", 2024-09-18T04:39:10.476 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 28.19454361, 2024-09-18T04:39:10.476 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.042024191000000002, 2024-09-18T04:39:10.476 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:10.477 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:10.477 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:10.477 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:10.477 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:10.477 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 34 2024-09-18T04:39:10.477 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.477 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:10.478 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.478 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:10.478 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.261758+0000", 2024-09-18T04:39:10.478 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:10.478 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.478 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.478 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:10.478 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.261758+0000", 2024-09-18T04:39:10.478 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:10.478 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.479 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.479 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:10.479 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.261756+0000", 2024-09-18T04:39:10.479 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:10.479 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.479 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.479 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:10.479 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.270785+0000", 2024-09-18T04:39:10.479 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0090292340000000006 2024-09-18T04:39:10.479 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.480 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.480 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:10.480 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.270890+0000", 2024-09-18T04:39:10.480 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000104541 2024-09-18T04:39:10.480 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.480 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.480 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:10.480 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.270991+0000", 2024-09-18T04:39:10.480 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000100778 2024-09-18T04:39:10.480 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.480 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.480 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:10.480 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.271157+0000", 2024-09-18T04:39:10.481 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016653999999999999 2024-09-18T04:39:10.481 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.481 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.481 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:10.481 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.271591+0000", 2024-09-18T04:39:10.481 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00043409500000000001 2024-09-18T04:39:10.481 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.481 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.481 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:10.481 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272017+0000", 2024-09-18T04:39:10.482 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000425769 2024-09-18T04:39:10.482 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.482 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.482 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:10.482 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.294246+0000", 2024-09-18T04:39:10.482 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.022229237999999998 2024-09-18T04:39:10.482 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.482 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.482 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:10.482 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303571+0000", 2024-09-18T04:39:10.482 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0093250239999999995 2024-09-18T04:39:10.482 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.482 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.482 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:10.483 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303705+0000", 2024-09-18T04:39:10.483 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000133831 2024-09-18T04:39:10.483 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.483 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.483 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:10.483 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303782+0000", 2024-09-18T04:39:10.483 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7156000000000005e-05 2024-09-18T04:39:10.483 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:10.483 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:10.483 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:10.483 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.483 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.483 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:36 2.7 e51/32 2:fe1b02ad:::rbd_data.125fc09df6d1.0000000000000002:head v 51'1, mlcod=0'0)", 2024-09-18T04:39:10.483 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.273441+0000", 2024-09-18T04:39:10.484 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 28.182860629, 2024-09-18T04:39:10.484 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.032400768000000003, 2024-09-18T04:39:10.484 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:10.484 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:10.484 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:10.484 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.484 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:10.484 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273441+0000", 2024-09-18T04:39:10.484 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:10.484 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.485 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.485 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:10.485 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273441+0000", 2024-09-18T04:39:10.485 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:10.485 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.485 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.485 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:10.485 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273440+0000", 2024-09-18T04:39:10.485 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:10.485 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.485 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.485 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:10.485 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278723+0000", 2024-09-18T04:39:10.486 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0052829909999999999 2024-09-18T04:39:10.486 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.486 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.486 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:10.486 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278752+0000", 2024-09-18T04:39:10.486 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9178000000000001e-05 2024-09-18T04:39:10.486 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.486 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.486 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:10.486 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278761+0000", 2024-09-18T04:39:10.486 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7900000000000005e-06 2024-09-18T04:39:10.486 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.486 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.486 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:10.487 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278864+0000", 2024-09-18T04:39:10.487 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000102699 2024-09-18T04:39:10.487 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.487 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.487 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:10.487 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278900+0000", 2024-09-18T04:39:10.487 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6001999999999999e-05 2024-09-18T04:39:10.487 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.487 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.487 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:10.487 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.305793+0000", 2024-09-18T04:39:10.487 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.026893344 2024-09-18T04:39:10.487 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.487 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.488 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:10.488 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.305842+0000", 2024-09-18T04:39:10.488 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8720000000000001e-05 2024-09-18T04:39:10.488 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:10.488 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:10.488 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:10.488 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.488 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.488 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:39 2.7 e51/32 2:e9c909c5:::rbd_data.125fc09df6d1.0000000000000005:head v 51'2, mlcod=0'0)", 2024-09-18T04:39:10.488 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.283175+0000", 2024-09-18T04:39:10.488 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 28.17312694, 2024-09-18T04:39:10.488 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.022864102000000001, 2024-09-18T04:39:10.488 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:10.488 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:10.489 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:10.489 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.489 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:10.489 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283175+0000", 2024-09-18T04:39:10.489 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:10.489 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.489 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.489 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:10.489 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283175+0000", 2024-09-18T04:39:10.489 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:10.489 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.489 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.490 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:10.490 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283173+0000", 2024-09-18T04:39:10.490 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:10.490 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.490 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.490 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:10.490 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.285951+0000", 2024-09-18T04:39:10.490 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0027784540000000001 2024-09-18T04:39:10.490 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.490 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.490 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:10.490 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.285970+0000", 2024-09-18T04:39:10.491 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8855000000000002e-05 2024-09-18T04:39:10.491 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.491 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.491 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:10.491 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.285984+0000", 2024-09-18T04:39:10.491 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4252e-05 2024-09-18T04:39:10.491 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.491 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.491 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:10.491 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.286041+0000", 2024-09-18T04:39:10.491 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6431999999999998e-05 2024-09-18T04:39:10.491 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.492 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.492 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:10.492 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.286086+0000", 2024-09-18T04:39:10.492 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5423000000000002e-05 2024-09-18T04:39:10.492 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.492 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.492 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:10.492 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.305998+0000", 2024-09-18T04:39:10.492 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019911697999999999 2024-09-18T04:39:10.492 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.492 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.492 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:10.492 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306039+0000", 2024-09-18T04:39:10.492 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0973999999999998e-05 2024-09-18T04:39:10.493 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:10.493 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:10.493 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:10.493 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.493 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.493 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:50 2.3 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:10.493 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.312563+0000", 2024-09-18T04:39:10.493 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 28.143738913, 2024-09-18T04:39:10.493 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.032636001999999997, 2024-09-18T04:39:10.493 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:10.493 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:10.493 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:10.493 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:10.493 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:10.494 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 50 2024-09-18T04:39:10.494 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.494 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:10.494 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.494 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:10.494 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.312563+0000", 2024-09-18T04:39:10.494 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:10.494 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.494 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.494 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:10.494 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.312563+0000", 2024-09-18T04:39:10.494 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:10.494 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.494 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.495 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:10.495 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.312561+0000", 2024-09-18T04:39:10.495 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:10.495 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.495 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.495 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:10.495 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321709+0000", 2024-09-18T04:39:10.495 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0091486650000000003 2024-09-18T04:39:10.495 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.495 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.495 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:10.495 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321739+0000", 2024-09-18T04:39:10.495 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9666e-05 2024-09-18T04:39:10.495 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.495 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.496 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:10.496 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321751+0000", 2024-09-18T04:39:10.496 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2086000000000001e-05 2024-09-18T04:39:10.496 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.496 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.496 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:10.496 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321833+0000", 2024-09-18T04:39:10.496 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1842999999999996e-05 2024-09-18T04:39:10.496 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.496 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.496 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:10.496 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321981+0000", 2024-09-18T04:39:10.496 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000147441 2024-09-18T04:39:10.496 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.497 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.497 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:10.497 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.322109+0000", 2024-09-18T04:39:10.497 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012851200000000001 2024-09-18T04:39:10.497 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.497 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.497 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:10.497 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.332829+0000", 2024-09-18T04:39:10.497 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010720225999999999 2024-09-18T04:39:10.497 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.497 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.497 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:10.497 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.345039+0000", 2024-09-18T04:39:10.497 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012209831000000001 2024-09-18T04:39:10.497 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.498 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.498 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:10.498 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.345120+0000", 2024-09-18T04:39:10.498 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1145999999999999e-05 2024-09-18T04:39:10.498 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.498 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.498 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:10.498 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.345199+0000", 2024-09-18T04:39:10.498 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8546000000000003e-05 2024-09-18T04:39:10.498 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:10.498 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:10.498 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:10.498 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.499 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.499 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:30 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:10.499 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.516122+0000", 2024-09-18T04:39:10.499 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 27.940179750999999, 2024-09-18T04:39:10.499 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0091865030000000004, 2024-09-18T04:39:10.499 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:10.499 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:10.499 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:10.499 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:10.499 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:10.499 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 30 2024-09-18T04:39:10.499 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.499 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:10.499 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.500 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:10.500 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516122+0000", 2024-09-18T04:39:10.500 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:10.500 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.500 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.500 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:10.500 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516122+0000", 2024-09-18T04:39:10.500 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:10.500 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.500 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.500 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:10.500 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516120+0000", 2024-09-18T04:39:10.500 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:10.501 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.501 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.501 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:10.501 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516129+0000", 2024-09-18T04:39:10.501 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.85e-06 2024-09-18T04:39:10.501 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.501 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.501 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:10.501 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516155+0000", 2024-09-18T04:39:10.501 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5882000000000001e-05 2024-09-18T04:39:10.501 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.501 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.501 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:10.501 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516175+0000", 2024-09-18T04:39:10.501 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.022e-05 2024-09-18T04:39:10.502 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.502 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.502 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:10.502 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519780+0000", 2024-09-18T04:39:10.502 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003604187 2024-09-18T04:39:10.502 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.502 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.502 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:10.502 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519964+0000", 2024-09-18T04:39:10.502 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00018391999999999999 2024-09-18T04:39:10.502 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.502 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.502 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:10.503 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.520794+0000", 2024-09-18T04:39:10.503 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00083078100000000005 2024-09-18T04:39:10.503 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.503 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.503 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:10.503 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.524129+0000", 2024-09-18T04:39:10.503 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033342419999999998 2024-09-18T04:39:10.504 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.504 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.504 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:10.504 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525239+0000", 2024-09-18T04:39:10.504 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001110778 2024-09-18T04:39:10.504 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.504 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.504 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:10.505 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525277+0000", 2024-09-18T04:39:10.505 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7596999999999999e-05 2024-09-18T04:39:10.505 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.505 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.505 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:10.505 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525308+0000", 2024-09-18T04:39:10.505 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.1535999999999998e-05 2024-09-18T04:39:10.505 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:10.505 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:10.505 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:10.505 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.505 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.505 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:32 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:10.505 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.519217+0000", 2024-09-18T04:39:10.506 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 27.937084430999999, 2024-09-18T04:39:10.506 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082266370000000002, 2024-09-18T04:39:10.506 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:10.506 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:10.506 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:10.506 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:10.506 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:10.506 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 32 2024-09-18T04:39:10.506 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.506 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:10.506 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.506 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:10.506 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519217+0000", 2024-09-18T04:39:10.507 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:10.507 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.507 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.507 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:10.507 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519217+0000", 2024-09-18T04:39:10.507 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:10.507 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.507 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.507 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:10.507 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519215+0000", 2024-09-18T04:39:10.507 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:10.507 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.507 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.508 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:10.508 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519229+0000", 2024-09-18T04:39:10.508 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3329999999999999e-05 2024-09-18T04:39:10.508 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.508 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.508 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:10.508 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519268+0000", 2024-09-18T04:39:10.508 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9533000000000002e-05 2024-09-18T04:39:10.508 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.508 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.508 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:10.508 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519285+0000", 2024-09-18T04:39:10.508 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6446000000000001e-05 2024-09-18T04:39:10.509 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.509 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.509 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:10.509 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.522358+0000", 2024-09-18T04:39:10.509 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030734310000000002 2024-09-18T04:39:10.509 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.509 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.509 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:10.509 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.522551+0000", 2024-09-18T04:39:10.509 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019317600000000001 2024-09-18T04:39:10.509 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.509 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.509 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:10.509 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.523262+0000", 2024-09-18T04:39:10.510 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00071069999999999998 2024-09-18T04:39:10.510 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.510 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.510 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:10.510 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525318+0000", 2024-09-18T04:39:10.510 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0020561469999999999 2024-09-18T04:39:10.510 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.510 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.510 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:10.510 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.527376+0000", 2024-09-18T04:39:10.510 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0020584969999999998 2024-09-18T04:39:10.510 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.510 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.510 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:10.511 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.527416+0000", 2024-09-18T04:39:10.511 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9054000000000003e-05 2024-09-18T04:39:10.511 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.511 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.511 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:10.511 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.527444+0000", 2024-09-18T04:39:10.511 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8347000000000002e-05 2024-09-18T04:39:10.511 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:10.511 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:10.511 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:10.511 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.511 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.511 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:42 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:10.512 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.551160+0000", 2024-09-18T04:39:10.512 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 27.905142039000001, 2024-09-18T04:39:10.512 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010692757000000001, 2024-09-18T04:39:10.512 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:10.512 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:10.512 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:10.512 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:10.512 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:10.512 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 42 2024-09-18T04:39:10.512 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.513 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:10.513 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.513 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:10.513 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551160+0000", 2024-09-18T04:39:10.513 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:10.513 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.513 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.513 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:10.513 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551160+0000", 2024-09-18T04:39:10.513 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:10.513 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.513 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.514 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:10.514 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551159+0000", 2024-09-18T04:39:10.514 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:10.514 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.514 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.514 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:10.514 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551166+0000", 2024-09-18T04:39:10.514 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1029999999999996e-06 2024-09-18T04:39:10.514 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.515 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.515 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:10.515 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551177+0000", 2024-09-18T04:39:10.515 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1384e-05 2024-09-18T04:39:10.515 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.515 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.515 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:10.515 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551185+0000", 2024-09-18T04:39:10.515 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1990000000000006e-06 2024-09-18T04:39:10.515 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.515 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.515 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:10.516 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552021+0000", 2024-09-18T04:39:10.516 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00083574400000000003 2024-09-18T04:39:10.516 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.516 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.516 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:10.516 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552339+0000", 2024-09-18T04:39:10.516 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031771399999999997 2024-09-18T04:39:10.516 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.516 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.516 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:10.517 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.553140+0000", 2024-09-18T04:39:10.517 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00080088100000000003 2024-09-18T04:39:10.517 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.517 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.517 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:10.517 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.554585+0000", 2024-09-18T04:39:10.517 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001445801 2024-09-18T04:39:10.517 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.517 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.517 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:10.517 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.561711+0000", 2024-09-18T04:39:10.517 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071260109999999998 2024-09-18T04:39:10.517 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.518 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.518 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:10.518 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.561790+0000", 2024-09-18T04:39:10.518 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8967000000000005e-05 2024-09-18T04:39:10.518 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.518 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.518 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:10.518 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.561852+0000", 2024-09-18T04:39:10.518 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2045999999999995e-05 2024-09-18T04:39:10.518 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:10.519 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:10.519 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:10.519 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.519 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.519 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:58 2.7 e51/32 2:fc7c6cd3:ns::rbd_data.133e813ab5bc.000000000000000e:head v 51'7, mlcod=51'5)", 2024-09-18T04:39:10.519 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.605333+0000", 2024-09-18T04:39:10.519 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 27.850968834, 2024-09-18T04:39:10.519 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020085902999999999, 2024-09-18T04:39:10.519 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:10.519 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:10.519 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:10.519 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.520 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:10.520 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605333+0000", 2024-09-18T04:39:10.520 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:10.520 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.520 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.520 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:10.520 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605333+0000", 2024-09-18T04:39:10.520 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:10.520 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.520 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.520 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:10.521 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605332+0000", 2024-09-18T04:39:10.521 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:10.521 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.521 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.521 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:10.521 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608667+0000", 2024-09-18T04:39:10.521 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033350300000000001 2024-09-18T04:39:10.521 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.521 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.522 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:10.522 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608706+0000", 2024-09-18T04:39:10.522 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8951e-05 2024-09-18T04:39:10.522 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.522 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.522 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:10.522 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608725+0000", 2024-09-18T04:39:10.522 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9675999999999999e-05 2024-09-18T04:39:10.522 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.522 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.522 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:10.523 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608798+0000", 2024-09-18T04:39:10.523 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3135999999999999e-05 2024-09-18T04:39:10.523 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.523 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.523 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:10.523 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608824+0000", 2024-09-18T04:39:10.523 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5284000000000001e-05 2024-09-18T04:39:10.523 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.523 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.524 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:10.524 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625370+0000", 2024-09-18T04:39:10.524 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016545975000000001 2024-09-18T04:39:10.524 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.524 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.524 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:10.524 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625419+0000", 2024-09-18T04:39:10.524 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9076999999999998e-05 2024-09-18T04:39:10.524 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:10.524 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:10.525 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:10.525 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.525 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.525 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:611 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:10.525 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.069561+0000", 2024-09-18T04:39:10.525 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 19.386740280000001, 2024-09-18T04:39:10.525 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0072330299999999997, 2024-09-18T04:39:10.525 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:10.525 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:10.525 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:10.525 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:39:10.525 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:10.525 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 611 2024-09-18T04:39:10.526 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.526 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:10.526 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.526 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:10.526 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.069561+0000", 2024-09-18T04:39:10.526 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:10.526 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.526 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.526 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:10.526 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.069561+0000", 2024-09-18T04:39:10.526 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:10.526 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.526 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.526 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:10.527 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.069559+0000", 2024-09-18T04:39:10.527 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:10.527 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.527 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.527 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:10.527 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070801+0000", 2024-09-18T04:39:10.527 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0012423359999999999 2024-09-18T04:39:10.527 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.527 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.527 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:10.527 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070828+0000", 2024-09-18T04:39:10.527 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6387e-05 2024-09-18T04:39:10.528 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.528 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.528 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:10.528 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070840+0000", 2024-09-18T04:39:10.528 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2689e-05 2024-09-18T04:39:10.528 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.528 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.528 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:10.528 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070990+0000", 2024-09-18T04:39:10.528 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014946800000000001 2024-09-18T04:39:10.528 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.528 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.528 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:10.529 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.071128+0000", 2024-09-18T04:39:10.529 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013796699999999999 2024-09-18T04:39:10.529 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.529 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.529 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:10.529 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.071318+0000", 2024-09-18T04:39:10.529 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000190106 2024-09-18T04:39:10.529 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.529 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.529 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:10.529 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.075711+0000", 2024-09-18T04:39:10.529 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.004393002 2024-09-18T04:39:10.529 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.530 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.530 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:10.530 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.076665+0000", 2024-09-18T04:39:10.530 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00095390399999999999 2024-09-18T04:39:10.530 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.530 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.530 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:10.530 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.076742+0000", 2024-09-18T04:39:10.530 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7112000000000003e-05 2024-09-18T04:39:10.530 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.530 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.530 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:10.530 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.076794+0000", 2024-09-18T04:39:10.530 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2475999999999997e-05 2024-09-18T04:39:10.531 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:10.531 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:10.531 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:10.531 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.531 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.531 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:687 2.3 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:10.531 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.487994+0000", 2024-09-18T04:39:10.531 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 18.968307296999999, 2024-09-18T04:39:10.531 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0074167690000000001, 2024-09-18T04:39:10.531 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:10.531 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:10.531 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:10.532 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:39:10.532 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:10.532 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 687 2024-09-18T04:39:10.532 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.532 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:10.532 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.532 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:10.532 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.487994+0000", 2024-09-18T04:39:10.532 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:10.532 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.532 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.532 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:10.532 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.487994+0000", 2024-09-18T04:39:10.532 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:10.533 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.533 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.533 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:10.533 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.487993+0000", 2024-09-18T04:39:10.533 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:10.533 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.533 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.533 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:10.533 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.488979+0000", 2024-09-18T04:39:10.533 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00098620700000000001 2024-09-18T04:39:10.533 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.533 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.533 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:10.534 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489002+0000", 2024-09-18T04:39:10.534 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2500000000000001e-05 2024-09-18T04:39:10.534 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.534 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.534 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:10.534 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489020+0000", 2024-09-18T04:39:10.534 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7691000000000001e-05 2024-09-18T04:39:10.534 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.534 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.534 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:10.534 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489091+0000", 2024-09-18T04:39:10.534 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1307e-05 2024-09-18T04:39:10.534 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.534 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.535 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:10.535 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489219+0000", 2024-09-18T04:39:10.535 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012766000000000001 2024-09-18T04:39:10.535 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.535 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.535 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:10.535 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489387+0000", 2024-09-18T04:39:10.535 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016855400000000001 2024-09-18T04:39:10.535 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.535 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.535 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:10.535 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495243+0000", 2024-09-18T04:39:10.535 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058563189999999996 2024-09-18T04:39:10.536 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.536 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.536 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:10.536 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495266+0000", 2024-09-18T04:39:10.536 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2810999999999999e-05 2024-09-18T04:39:10.536 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.536 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.536 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:10.536 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495332+0000", 2024-09-18T04:39:10.536 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5469999999999995e-05 2024-09-18T04:39:10.536 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.536 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.537 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:10.537 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495411+0000", 2024-09-18T04:39:10.537 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9395999999999997e-05 2024-09-18T04:39:10.537 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:10.537 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:10.537 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:10.537 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.537 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.537 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:689 2.3 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:10.537 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.500070+0000", 2024-09-18T04:39:10.538 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 18.956231481, 2024-09-18T04:39:10.538 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071096650000000003, 2024-09-18T04:39:10.538 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:10.538 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:10.538 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:10.538 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:39:10.538 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:10.538 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 689 2024-09-18T04:39:10.538 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.538 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:10.538 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.538 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:10.538 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.500070+0000", 2024-09-18T04:39:10.539 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:10.539 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.539 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.539 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:10.539 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.500070+0000", 2024-09-18T04:39:10.539 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:10.539 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.539 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.539 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:10.539 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.500069+0000", 2024-09-18T04:39:10.539 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:10.539 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.539 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.540 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:10.540 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501082+0000", 2024-09-18T04:39:10.540 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010136570000000001 2024-09-18T04:39:10.540 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.540 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.540 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:10.540 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501104+0000", 2024-09-18T04:39:10.540 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1883e-05 2024-09-18T04:39:10.540 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.540 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.540 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:10.540 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501115+0000", 2024-09-18T04:39:10.540 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0414000000000001e-05 2024-09-18T04:39:10.541 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.541 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.541 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:10.541 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501178+0000", 2024-09-18T04:39:10.541 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3128999999999998e-05 2024-09-18T04:39:10.541 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.541 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.541 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:10.541 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501257+0000", 2024-09-18T04:39:10.541 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9592999999999995e-05 2024-09-18T04:39:10.541 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.541 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.541 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:10.542 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501366+0000", 2024-09-18T04:39:10.542 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000108643 2024-09-18T04:39:10.542 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.542 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.542 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:10.542 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.504041+0000", 2024-09-18T04:39:10.542 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0026750889999999999 2024-09-18T04:39:10.542 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.542 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.542 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:10.542 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.507097+0000", 2024-09-18T04:39:10.542 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030556070000000001 2024-09-18T04:39:10.542 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.542 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.543 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:10.543 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.507140+0000", 2024-09-18T04:39:10.543 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3745000000000002e-05 2024-09-18T04:39:10.543 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.543 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.543 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:10.543 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.507180+0000", 2024-09-18T04:39:10.543 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9475999999999999e-05 2024-09-18T04:39:10.543 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:10.543 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:10.543 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:10.543 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.544 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.544 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4972.0:13 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:10.544 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.948263+0000", 2024-09-18T04:39:10.544 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 18.508038689999999, 2024-09-18T04:39:10.544 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0091358970000000005, 2024-09-18T04:39:10.544 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:10.544 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:39:10.544 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:10.544 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4972", 2024-09-18T04:39:10.544 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2280251472", 2024-09-18T04:39:10.544 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 13 2024-09-18T04:39:10.545 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.545 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:10.545 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.545 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:10.545 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948263+0000", 2024-09-18T04:39:10.545 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:10.545 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.545 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.545 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:10.546 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948263+0000", 2024-09-18T04:39:10.546 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:10.546 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.546 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.546 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:10.546 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948261+0000", 2024-09-18T04:39:10.546 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:10.546 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.546 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.547 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:10.547 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948273+0000", 2024-09-18T04:39:10.547 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1576999999999999e-05 2024-09-18T04:39:10.547 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.547 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.547 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:10.547 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948305+0000", 2024-09-18T04:39:10.547 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2209000000000003e-05 2024-09-18T04:39:10.547 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.547 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.548 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:10.548 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948325+0000", 2024-09-18T04:39:10.548 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.022e-05 2024-09-18T04:39:10.548 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.548 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.548 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:10.548 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948455+0000", 2024-09-18T04:39:10.548 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000129624 2024-09-18T04:39:10.548 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.548 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.548 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:10.548 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948598+0000", 2024-09-18T04:39:10.548 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014376699999999999 2024-09-18T04:39:10.549 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.549 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.549 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:10.549 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.957399+0000", 2024-09-18T04:39:10.549 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0088004299999999997 2024-09-18T04:39:10.549 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:10.549 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:10.549 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:10.549 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.550 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.550 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:101 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:10.550 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.576367+0000", 2024-09-18T04:39:10.550 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 13.879934982, 2024-09-18T04:39:10.550 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0078671020000000008, 2024-09-18T04:39:10.550 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:10.550 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:10.550 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:10.550 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:10.550 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:10.550 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 101 2024-09-18T04:39:10.550 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.551 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:10.551 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.551 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:10.551 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576367+0000", 2024-09-18T04:39:10.551 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:10.551 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.551 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.551 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:10.551 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576367+0000", 2024-09-18T04:39:10.552 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:10.552 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.552 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.552 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:10.552 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576364+0000", 2024-09-18T04:39:10.552 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999971 2024-09-18T04:39:10.552 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.552 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.552 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:10.552 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576378+0000", 2024-09-18T04:39:10.552 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3879999999999999e-05 2024-09-18T04:39:10.553 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.553 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.553 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:10.553 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576410+0000", 2024-09-18T04:39:10.553 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2286000000000002e-05 2024-09-18T04:39:10.553 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.553 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.553 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:10.553 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576425+0000", 2024-09-18T04:39:10.553 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5290000000000001e-05 2024-09-18T04:39:10.553 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.553 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.553 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:10.554 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576551+0000", 2024-09-18T04:39:10.554 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012555199999999999 2024-09-18T04:39:10.554 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.554 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.554 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:10.554 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576719+0000", 2024-09-18T04:39:10.554 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016787400000000001 2024-09-18T04:39:10.554 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.554 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.554 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:10.554 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576951+0000", 2024-09-18T04:39:10.554 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00023252400000000001 2024-09-18T04:39:10.554 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.554 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.555 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:10.555 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.581459+0000", 2024-09-18T04:39:10.555 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045074410000000001 2024-09-18T04:39:10.555 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.555 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.555 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:10.555 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584076+0000", 2024-09-18T04:39:10.555 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0026173860000000002 2024-09-18T04:39:10.555 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.555 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.555 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:10.555 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584160+0000", 2024-09-18T04:39:10.555 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4191999999999996e-05 2024-09-18T04:39:10.555 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.556 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.556 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:10.556 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584234+0000", 2024-09-18T04:39:10.556 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.339e-05 2024-09-18T04:39:10.556 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:10.556 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:10.556 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:10.556 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.556 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.556 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:44 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:10.557 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.550448+0000", 2024-09-18T04:39:10.557 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 9.9058537930000004, 2024-09-18T04:39:10.557 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082117590000000008, 2024-09-18T04:39:10.557 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:10.557 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:10.557 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:10.557 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:39:10.557 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:10.557 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 44 2024-09-18T04:39:10.557 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.558 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:10.558 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.558 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:10.558 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.550448+0000", 2024-09-18T04:39:10.558 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:10.558 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.558 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.558 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:10.558 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.550448+0000", 2024-09-18T04:39:10.559 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:10.559 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.559 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.559 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:10.559 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.550445+0000", 2024-09-18T04:39:10.559 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999971 2024-09-18T04:39:10.559 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.559 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.559 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:10.559 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.551904+0000", 2024-09-18T04:39:10.560 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001458322 2024-09-18T04:39:10.560 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.560 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.560 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:10.560 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.551938+0000", 2024-09-18T04:39:10.560 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4359000000000001e-05 2024-09-18T04:39:10.560 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.560 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.560 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:10.560 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.551959+0000", 2024-09-18T04:39:10.560 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1403000000000002e-05 2024-09-18T04:39:10.560 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.561 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.561 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:10.561 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.552049+0000", 2024-09-18T04:39:10.561 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9970999999999996e-05 2024-09-18T04:39:10.561 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.561 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.561 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:10.561 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.552190+0000", 2024-09-18T04:39:10.561 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000140271 2024-09-18T04:39:10.561 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.561 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.561 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:10.561 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.552385+0000", 2024-09-18T04:39:10.561 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019498700000000001 2024-09-18T04:39:10.562 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.562 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.562 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:10.562 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.556308+0000", 2024-09-18T04:39:10.562 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00392333 2024-09-18T04:39:10.562 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.562 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.562 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:10.562 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.558479+0000", 2024-09-18T04:39:10.562 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0021715889999999998 2024-09-18T04:39:10.562 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.562 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.562 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:10.563 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.558616+0000", 2024-09-18T04:39:10.563 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013613399999999999 2024-09-18T04:39:10.563 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.563 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.563 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:10.563 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.558660+0000", 2024-09-18T04:39:10.563 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4053e-05 2024-09-18T04:39:10.563 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:10.563 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:10.563 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:10.563 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.563 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.563 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:46 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:10.564 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.564089+0000", 2024-09-18T04:39:10.564 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 9.8922126640000005, 2024-09-18T04:39:10.564 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0069437090000000002, 2024-09-18T04:39:10.564 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:10.564 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:10.564 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:10.564 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:39:10.564 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:10.564 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 46 2024-09-18T04:39:10.564 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.564 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:10.564 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.564 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:10.564 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.564089+0000", 2024-09-18T04:39:10.565 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:10.565 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.565 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.565 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:10.565 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.564089+0000", 2024-09-18T04:39:10.565 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:10.565 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.565 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.565 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:10.565 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.564087+0000", 2024-09-18T04:39:10.565 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:10.565 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.565 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.566 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:10.566 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565126+0000", 2024-09-18T04:39:10.566 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001038902 2024-09-18T04:39:10.566 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.566 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.566 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:10.566 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565148+0000", 2024-09-18T04:39:10.566 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1911000000000001e-05 2024-09-18T04:39:10.566 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.566 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.566 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:10.566 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565157+0000", 2024-09-18T04:39:10.566 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6220000000000004e-06 2024-09-18T04:39:10.566 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.567 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.567 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:10.567 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565261+0000", 2024-09-18T04:39:10.567 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000103063 2024-09-18T04:39:10.567 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.567 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.567 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:10.567 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565394+0000", 2024-09-18T04:39:10.567 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000133032 2024-09-18T04:39:10.567 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.567 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.567 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:10.567 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565564+0000", 2024-09-18T04:39:10.568 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017068000000000001 2024-09-18T04:39:10.568 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.568 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.568 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:10.568 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.570556+0000", 2024-09-18T04:39:10.568 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0049920620000000002 2024-09-18T04:39:10.568 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.568 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.568 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:10.568 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.570885+0000", 2024-09-18T04:39:10.568 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032827900000000001 2024-09-18T04:39:10.568 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.568 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.568 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:10.569 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.570959+0000", 2024-09-18T04:39:10.569 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3955999999999995e-05 2024-09-18T04:39:10.569 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.569 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.569 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:10.569 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.571033+0000", 2024-09-18T04:39:10.569 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4191999999999997e-05 2024-09-18T04:39:10.569 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:10.569 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:10.569 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:10.569 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.569 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.570 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:100 2.4 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:10.570 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.856389+0000", 2024-09-18T04:39:10.570 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 9.5999125519999993, 2024-09-18T04:39:10.570 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0081480800000000003, 2024-09-18T04:39:10.570 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:10.570 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:10.570 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:10.570 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:39:10.570 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:10.570 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 100 2024-09-18T04:39:10.570 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.570 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:10.570 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.570 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:10.571 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.856389+0000", 2024-09-18T04:39:10.571 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:10.571 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.571 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.571 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:10.571 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.856389+0000", 2024-09-18T04:39:10.571 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:10.571 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.571 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.571 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:10.571 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.856388+0000", 2024-09-18T04:39:10.571 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:10.571 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.572 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.572 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:10.572 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857295+0000", 2024-09-18T04:39:10.572 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00090725599999999995 2024-09-18T04:39:10.572 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.572 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.572 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:10.572 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857311+0000", 2024-09-18T04:39:10.572 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6092000000000001e-05 2024-09-18T04:39:10.572 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.572 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.572 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:10.573 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857319+0000", 2024-09-18T04:39:10.573 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9200000000000004e-06 2024-09-18T04:39:10.573 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.573 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.573 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:10.573 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857388+0000", 2024-09-18T04:39:10.573 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8591999999999997e-05 2024-09-18T04:39:10.573 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.573 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.573 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:10.573 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857535+0000", 2024-09-18T04:39:10.573 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014684299999999999 2024-09-18T04:39:10.574 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.574 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.574 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:10.574 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857747+0000", 2024-09-18T04:39:10.574 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000212361 2024-09-18T04:39:10.574 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.574 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.574 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:10.574 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.860831+0000", 2024-09-18T04:39:10.574 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030841789999999998 2024-09-18T04:39:10.574 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.574 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.574 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:10.575 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.864361+0000", 2024-09-18T04:39:10.575 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0035298780000000002 2024-09-18T04:39:10.575 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.575 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.575 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:10.575 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.864456+0000", 2024-09-18T04:39:10.575 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4557999999999998e-05 2024-09-18T04:39:10.575 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.575 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.575 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:10.575 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.864537+0000", 2024-09-18T04:39:10.576 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1577999999999997e-05 2024-09-18T04:39:10.576 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:10.576 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:10.576 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:10.576 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.576 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.576 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:102 2.4 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:10.576 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.869773+0000", 2024-09-18T04:39:10.576 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 9.5865288950000007, 2024-09-18T04:39:10.576 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0084430359999999992, 2024-09-18T04:39:10.576 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:10.576 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:10.576 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:10.577 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:39:10.577 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:10.577 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 102 2024-09-18T04:39:10.577 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.577 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:10.577 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.577 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:10.577 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.869773+0000", 2024-09-18T04:39:10.577 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:10.577 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.577 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.578 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:10.578 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.869773+0000", 2024-09-18T04:39:10.578 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:10.578 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.578 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.578 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:10.578 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.869771+0000", 2024-09-18T04:39:10.578 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:10.578 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.578 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.578 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:10.578 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870629+0000", 2024-09-18T04:39:10.578 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00085760999999999997 2024-09-18T04:39:10.579 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.579 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.579 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:10.579 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870647+0000", 2024-09-18T04:39:10.579 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8363999999999999e-05 2024-09-18T04:39:10.579 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.579 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.579 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:10.579 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870655+0000", 2024-09-18T04:39:10.579 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0099999999999995e-06 2024-09-18T04:39:10.579 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.579 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.579 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:10.579 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870803+0000", 2024-09-18T04:39:10.580 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014798 2024-09-18T04:39:10.580 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.580 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.580 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:10.580 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870951+0000", 2024-09-18T04:39:10.580 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014715999999999999 2024-09-18T04:39:10.580 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.580 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.580 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:10.581 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.871145+0000", 2024-09-18T04:39:10.581 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000194253 2024-09-18T04:39:10.581 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.581 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.581 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:10.581 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.877033+0000", 2024-09-18T04:39:10.581 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058883099999999999 2024-09-18T04:39:10.581 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.581 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.581 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:10.581 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.878078+0000", 2024-09-18T04:39:10.581 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001044747 2024-09-18T04:39:10.581 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.582 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.582 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:10.582 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.878160+0000", 2024-09-18T04:39:10.582 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1904999999999996e-05 2024-09-18T04:39:10.582 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.582 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.582 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:10.582 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.878216+0000", 2024-09-18T04:39:10.582 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6082000000000003e-05 2024-09-18T04:39:10.582 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:10.583 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:10.583 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:10.583 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.583 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.583 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:145 2.4 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:10.583 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:01.096935+0000", 2024-09-18T04:39:10.583 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 9.3593671149999995, 2024-09-18T04:39:10.583 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0069937869999999996, 2024-09-18T04:39:10.583 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:10.583 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:10.583 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:10.583 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:39:10.583 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:10.584 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 145 2024-09-18T04:39:10.584 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.584 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:10.584 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.584 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:10.584 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.096935+0000", 2024-09-18T04:39:10.584 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:10.584 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.584 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.584 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:10.585 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.096935+0000", 2024-09-18T04:39:10.585 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:10.585 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.585 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.585 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:10.585 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.096932+0000", 2024-09-18T04:39:10.585 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:10.585 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.585 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.585 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:10.585 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097815+0000", 2024-09-18T04:39:10.585 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00088276000000000001 2024-09-18T04:39:10.585 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.585 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.586 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:10.586 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097842+0000", 2024-09-18T04:39:10.586 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6806999999999999e-05 2024-09-18T04:39:10.586 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.586 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.586 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:10.586 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097852+0000", 2024-09-18T04:39:10.586 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0548e-05 2024-09-18T04:39:10.586 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.587 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.587 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:10.587 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097931+0000", 2024-09-18T04:39:10.587 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8417000000000005e-05 2024-09-18T04:39:10.587 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.587 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.587 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:10.587 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.098047+0000", 2024-09-18T04:39:10.587 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011611 2024-09-18T04:39:10.587 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.588 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.588 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:10.588 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.098172+0000", 2024-09-18T04:39:10.588 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000125482 2024-09-18T04:39:10.588 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.588 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.588 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:10.588 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.100888+0000", 2024-09-18T04:39:10.588 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0027154430000000001 2024-09-18T04:39:10.588 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.588 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.589 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:10.589 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.103814+0000", 2024-09-18T04:39:10.589 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0029259619999999998 2024-09-18T04:39:10.589 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.589 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.589 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:10.589 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.103891+0000", 2024-09-18T04:39:10.589 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7619000000000003e-05 2024-09-18T04:39:10.589 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:10.589 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:10.589 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:10.590 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.103928+0000", 2024-09-18T04:39:10.590 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6900999999999998e-05 2024-09-18T04:39:10.590 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:10.590 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:10.590 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:10.590 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:10.590 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:10.590 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:10.590 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-18T04:39:10.591 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T04:39:10.591 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-09-18T04:39:10.591 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2024-09-18T04:39:10.593 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:10.457+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:10.593 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:10.585+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:10.594 INFO:tasks.workunit.client.0.smithi115.stderr:+ DEV=/dev/nbd0 2024-09-18T04:39:10.594 INFO:tasks.workunit.client.0.smithi115.stderr:+ get_pid rbd 2024-09-18T04:39:10.594 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pool=rbd 2024-09-18T04:39:10.594 INFO:tasks.workunit.client.0.smithi115.stderr:+ local ns= 2024-09-18T04:39:10.594 INFO:tasks.workunit.client.0.smithi115.stderr:++ rbd device --device-type nbd --format xml list 2024-09-18T04:39:10.595 INFO:tasks.workunit.client.0.smithi115.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd17342'\''][device='\''/dev/nbd0'\'']/id' 2024-09-18T04:39:10.595 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:39:10.640 INFO:tasks.workunit.client.0.smithi115.stderr:+ PID=19059 2024-09-18T04:39:10.640 INFO:tasks.workunit.client.0.smithi115.stderr:+ test -n 19059 2024-09-18T04:39:10.641 INFO:tasks.workunit.client.0.smithi115.stderr:+ ps -p 19059 -C rbd-nbd 2024-09-18T04:39:10.650 INFO:tasks.workunit.client.0.smithi115.stdout: PID TTY TIME CMD 2024-09-18T04:39:10.650 INFO:tasks.workunit.client.0.smithi115.stdout: 19059 ? 00:00:00 rbd-nbd 2024-09-18T04:39:10.651 INFO:tasks.workunit.client.0.smithi115.stderr:+ pid=19059 2024-09-18T04:39:10.651 INFO:tasks.workunit.client.0.smithi115.stderr:++ _sudo rbd device --device-type nbd map rbd/ns/testrbdnbd17342 2024-09-18T04:39:10.651 INFO:tasks.workunit.client.0.smithi115.stderr:++ local cmd 2024-09-18T04:39:10.651 INFO:tasks.workunit.client.0.smithi115.stderr:+++ id -u 2024-09-18T04:39:10.652 INFO:tasks.workunit.client.0.smithi115.stderr:++ '[' 1000 -eq 0 ']' 2024-09-18T04:39:10.652 INFO:tasks.workunit.client.0.smithi115.stderr:+++ which rbd 2024-09-18T04:39:10.654 INFO:tasks.workunit.client.0.smithi115.stderr:++ cmd=/usr/bin/rbd 2024-09-18T04:39:10.654 INFO:tasks.workunit.client.0.smithi115.stderr:++ shift 2024-09-18T04:39:10.654 INFO:tasks.workunit.client.0.smithi115.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/ns/testrbdnbd17342 2024-09-18T04:39:10.693 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:10.685+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:10.755 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:10.755 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:10.755 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:39:10.755 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:10.763 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2024-09-18T04:39:10.765 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:39:10.795 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:10.789+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:39:10.841 INFO:tasks.workunit.client.0.smithi115.stderr:+ DEV=/dev/nbd1 2024-09-18T04:39:10.841 INFO:tasks.workunit.client.0.smithi115.stderr:+ get_pid rbd ns 2024-09-18T04:39:10.841 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pool=rbd 2024-09-18T04:39:10.841 INFO:tasks.workunit.client.0.smithi115.stderr:+ local ns=ns 2024-09-18T04:39:10.842 INFO:tasks.workunit.client.0.smithi115.stderr:++ rbd device --device-type nbd --format xml list 2024-09-18T04:39:10.842 INFO:tasks.workunit.client.0.smithi115.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\''ns'\''][image='\''testrbdnbd17342'\''][device='\''/dev/nbd1'\'']/id' 2024-09-18T04:39:10.868 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:10.868 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:10.868 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:39:10.868 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:39:10.868 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:10.878 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:39:10.886 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T04:39:10.896 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:10.889+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:10.910 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2024-09-18T04:39:10.936 INFO:tasks.workunit.client.0.smithi115.stderr:+ PID=19107 2024-09-18T04:39:10.936 INFO:tasks.workunit.client.0.smithi115.stderr:+ test -n 19107 2024-09-18T04:39:10.936 INFO:tasks.workunit.client.0.smithi115.stderr:+ ps -p 19107 -C rbd-nbd 2024-09-18T04:39:10.944 INFO:tasks.workunit.client.0.smithi115.stdout: PID TTY TIME CMD 2024-09-18T04:39:10.944 INFO:tasks.workunit.client.0.smithi115.stdout: 19059 ? 00:00:00 rbd-nbd 2024-09-18T04:39:10.944 INFO:tasks.workunit.client.0.smithi115.stdout: 19107 ? 00:00:00 rbd-nbd 2024-09-18T04:39:10.945 INFO:tasks.workunit.client.0.smithi115.stderr:+ unmap_device rbd/ns/testrbdnbd17342 19107 2024-09-18T04:39:10.945 INFO:tasks.workunit.client.0.smithi115.stderr:+ local args=rbd/ns/testrbdnbd17342 2024-09-18T04:39:10.945 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pid=19107 2024-09-18T04:39:10.946 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo rbd device --device-type nbd unmap rbd/ns/testrbdnbd17342 2024-09-18T04:39:10.946 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:39:10.946 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:39:10.946 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:39:10.946 INFO:tasks.workunit.client.0.smithi115.stderr:++ which rbd 2024-09-18T04:39:10.948 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/bin/rbd 2024-09-18T04:39:10.948 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:39:10.949 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap rbd/ns/testrbdnbd17342 2024-09-18T04:39:10.997 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:10.989+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:11.050 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd device --device-type nbd list 2024-09-18T04:39:11.050 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false grep '^19107\b' 2024-09-18T04:39:11.051 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep '^19107\b' 2024-09-18T04:39:11.066 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:11.066 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:39:11.066 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:39:11.066 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:39:11.066 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.066 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:37 2.5 e51/28 2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:head v 51'1, mlcod=0'0)", 2024-09-18T04:39:11.067 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:39:11.067 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 28.769244102999998, 2024-09-18T04:39:11.067 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012279973, 2024-09-18T04:39:11.067 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:11.067 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:11.067 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:11.067 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.067 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:11.067 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:39:11.067 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:11.068 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.068 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.068 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:11.068 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:39:11.068 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:11.068 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.068 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.068 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:11.068 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292468+0000", 2024-09-18T04:39:11.068 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:11.069 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.069 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.069 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:11.069 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295837+0000", 2024-09-18T04:39:11.069 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0033691699999999999 2024-09-18T04:39:11.069 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.069 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.069 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:11.069 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295860+0000", 2024-09-18T04:39:11.069 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2756999999999999e-05 2024-09-18T04:39:11.069 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.070 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.070 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:11.070 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295872+0000", 2024-09-18T04:39:11.070 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2442e-05 2024-09-18T04:39:11.070 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.070 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.070 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:11.070 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295934+0000", 2024-09-18T04:39:11.071 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1187999999999998e-05 2024-09-18T04:39:11.071 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.071 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.071 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:11.071 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295967+0000", 2024-09-18T04:39:11.071 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3185999999999997e-05 2024-09-18T04:39:11.071 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.071 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.071 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:11.071 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304697+0000", 2024-09-18T04:39:11.072 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0087298059999999997 2024-09-18T04:39:11.072 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.072 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.072 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:11.072 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304749+0000", 2024-09-18T04:39:11.072 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1922000000000002e-05 2024-09-18T04:39:11.072 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:11.072 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:11.072 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:11.072 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.072 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.072 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:49 2.5 e51/28 2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head v 51'2, mlcod=0'0)", 2024-09-18T04:39:11.073 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:39:11.073 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 28.735917265000001, 2024-09-18T04:39:11.073 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018033041999999999, 2024-09-18T04:39:11.073 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:11.073 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:11.073 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:11.073 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.073 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:11.073 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:39:11.073 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:11.073 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.073 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.073 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:11.074 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:39:11.074 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:11.074 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.074 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.074 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:11.074 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325793+0000", 2024-09-18T04:39:11.074 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:11.074 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.074 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.074 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:11.074 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329038+0000", 2024-09-18T04:39:11.074 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0032444010000000001 2024-09-18T04:39:11.074 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.074 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.075 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:11.075 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329065+0000", 2024-09-18T04:39:11.075 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7500000000000001e-05 2024-09-18T04:39:11.075 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.075 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.075 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:11.075 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329077+0000", 2024-09-18T04:39:11.075 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1919999999999999e-05 2024-09-18T04:39:11.076 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.076 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.076 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:11.076 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329149+0000", 2024-09-18T04:39:11.076 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2349999999999997e-05 2024-09-18T04:39:11.076 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.076 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.076 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:11.076 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329192+0000", 2024-09-18T04:39:11.076 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2744000000000003e-05 2024-09-18T04:39:11.076 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.077 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.077 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:11.077 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343773+0000", 2024-09-18T04:39:11.077 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014581246000000001 2024-09-18T04:39:11.077 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.077 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.077 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:11.077 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343829+0000", 2024-09-18T04:39:11.077 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5143999999999999e-05 2024-09-18T04:39:11.077 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:11.077 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:11.077 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:11.077 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.078 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.078 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:47 2.4 e51/17 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 51'3, mlcod=51'1)", 2024-09-18T04:39:11.078 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:39:11.078 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 28.735337000000001, 2024-09-18T04:39:11.078 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018960186, 2024-09-18T04:39:11.078 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:11.078 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:11.078 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:11.078 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.078 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:11.078 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:39:11.078 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:11.078 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.078 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.079 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:11.079 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:39:11.079 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:11.079 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.079 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.079 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:11.079 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326374+0000", 2024-09-18T04:39:11.079 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:11.079 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.079 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.079 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:11.079 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330883+0000", 2024-09-18T04:39:11.079 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0045084740000000002 2024-09-18T04:39:11.079 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.080 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.080 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:11.080 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330901+0000", 2024-09-18T04:39:11.080 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7890999999999999e-05 2024-09-18T04:39:11.080 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.080 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.080 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:11.080 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330912+0000", 2024-09-18T04:39:11.080 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1041e-05 2024-09-18T04:39:11.080 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.080 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.080 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:11.080 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.331003+0000", 2024-09-18T04:39:11.080 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1323999999999995e-05 2024-09-18T04:39:11.081 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.081 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.081 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:11.081 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.331030+0000", 2024-09-18T04:39:11.081 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7351999999999998e-05 2024-09-18T04:39:11.081 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.081 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.081 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:11.081 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345287+0000", 2024-09-18T04:39:11.081 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014256332 2024-09-18T04:39:11.081 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.081 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.081 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:11.081 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345336+0000", 2024-09-18T04:39:11.082 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9356e-05 2024-09-18T04:39:11.082 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:11.082 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:11.082 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:11.082 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.082 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.082 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:58 2.5 e51/28 2:b227e6b1:::rbd_data.125fc09df6d1.000000000000000e:head v 51'3, mlcod=51'1)", 2024-09-18T04:39:11.082 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:39:11.082 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 28.694910372999999, 2024-09-18T04:39:11.082 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016513678, 2024-09-18T04:39:11.082 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:11.082 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:11.082 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:11.082 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.083 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:11.083 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:39:11.083 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:11.083 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.083 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.083 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:11.083 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:39:11.083 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:11.083 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.083 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.083 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:11.083 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366800+0000", 2024-09-18T04:39:11.083 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:11.084 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.084 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.084 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:11.084 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368644+0000", 2024-09-18T04:39:11.084 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001844399 2024-09-18T04:39:11.084 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.084 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.084 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:11.084 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368681+0000", 2024-09-18T04:39:11.084 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6526999999999998e-05 2024-09-18T04:39:11.084 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.084 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.085 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:11.085 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368698+0000", 2024-09-18T04:39:11.085 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7272999999999999e-05 2024-09-18T04:39:11.085 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.085 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.085 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:11.085 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368773+0000", 2024-09-18T04:39:11.085 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4862000000000003e-05 2024-09-18T04:39:11.085 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.085 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.085 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:11.085 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368801+0000", 2024-09-18T04:39:11.085 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7925000000000001e-05 2024-09-18T04:39:11.085 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.086 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.086 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:11.086 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383261+0000", 2024-09-18T04:39:11.086 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014460377999999999 2024-09-18T04:39:11.086 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.086 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.086 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:11.086 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383316+0000", 2024-09-18T04:39:11.086 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4623999999999997e-05 2024-09-18T04:39:11.086 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:11.086 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:11.086 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:11.086 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.087 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.087 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:56 2.4 e51/17 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 51'4, mlcod=51'2)", 2024-09-18T04:39:11.087 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:39:11.087 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 28.692906347000001, 2024-09-18T04:39:11.087 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015105221, 2024-09-18T04:39:11.087 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:11.087 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:11.087 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:11.087 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.087 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:11.087 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:39:11.087 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:11.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.087 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.088 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:11.088 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:39:11.088 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:11.088 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.088 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.088 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:11.088 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368805+0000", 2024-09-18T04:39:11.088 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:11.088 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.088 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.088 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:11.088 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371603+0000", 2024-09-18T04:39:11.088 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027982660000000002 2024-09-18T04:39:11.089 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.089 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.089 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:11.089 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371633+0000", 2024-09-18T04:39:11.089 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0073000000000002e-05 2024-09-18T04:39:11.089 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.089 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.089 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:11.089 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371652+0000", 2024-09-18T04:39:11.089 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8597000000000002e-05 2024-09-18T04:39:11.089 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.089 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.089 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:11.089 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371752+0000", 2024-09-18T04:39:11.090 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9772999999999995e-05 2024-09-18T04:39:11.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.090 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.090 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:11.090 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371793+0000", 2024-09-18T04:39:11.090 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1173e-05 2024-09-18T04:39:11.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.090 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.090 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:11.090 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383875+0000", 2024-09-18T04:39:11.090 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012081852000000001 2024-09-18T04:39:11.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.090 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.090 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:11.090 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383912+0000", 2024-09-18T04:39:11.091 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7045000000000002e-05 2024-09-18T04:39:11.091 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:11.091 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:11.091 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:11.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.091 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.091 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:36 2.5 e51/28 2:bd5b8dd8:ns::rbd_data.133e813ab5bc.0000000000000002:head v 51'4, mlcod=51'2)", 2024-09-18T04:39:11.091 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:39:11.091 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 28.524187355999999, 2024-09-18T04:39:11.091 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020758161000000001, 2024-09-18T04:39:11.091 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:11.091 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:11.091 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:11.091 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.092 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:11.092 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:39:11.092 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:11.092 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.092 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.092 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:11.092 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:39:11.092 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:11.092 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.092 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.092 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:11.092 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537523+0000", 2024-09-18T04:39:11.092 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:11.092 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.093 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.093 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:11.093 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541578+0000", 2024-09-18T04:39:11.093 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0040549710000000001 2024-09-18T04:39:11.093 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.093 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.093 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:11.093 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541608+0000", 2024-09-18T04:39:11.093 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9349e-05 2024-09-18T04:39:11.093 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.093 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.093 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:11.093 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541621+0000", 2024-09-18T04:39:11.093 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2736e-05 2024-09-18T04:39:11.094 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.094 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.094 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:11.094 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541715+0000", 2024-09-18T04:39:11.094 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4197999999999995e-05 2024-09-18T04:39:11.094 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.094 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.094 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:11.094 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541755+0000", 2024-09-18T04:39:11.094 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9818999999999999e-05 2024-09-18T04:39:11.094 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.094 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.094 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:11.095 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558217+0000", 2024-09-18T04:39:11.095 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016462234999999999 2024-09-18T04:39:11.095 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.095 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.095 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:11.095 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558284+0000", 2024-09-18T04:39:11.095 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6774999999999997e-05 2024-09-18T04:39:11.095 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:11.095 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:11.095 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:11.095 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.095 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.095 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:39 2.4 e51/17 2:304364d1:ns::rbd_data.133e813ab5bc.0000000000000004:head v 51'5, mlcod=51'3)", 2024-09-18T04:39:11.095 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:39:11.096 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 28.516048228999999, 2024-09-18T04:39:11.096 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016777558000000001, 2024-09-18T04:39:11.096 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:11.096 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:11.096 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:11.096 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.096 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:11.096 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:39:11.096 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:11.096 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.096 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.096 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:11.097 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:39:11.097 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:11.097 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.097 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.097 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:11.097 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545663+0000", 2024-09-18T04:39:11.097 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:11.097 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.097 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.098 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:11.098 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548286+0000", 2024-09-18T04:39:11.098 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026233390000000001 2024-09-18T04:39:11.098 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.098 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.098 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:11.098 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548305+0000", 2024-09-18T04:39:11.098 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8425000000000001e-05 2024-09-18T04:39:11.098 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.098 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.098 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:11.098 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548316+0000", 2024-09-18T04:39:11.098 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1163e-05 2024-09-18T04:39:11.098 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.099 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.099 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:11.099 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548379+0000", 2024-09-18T04:39:11.099 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2780000000000005e-05 2024-09-18T04:39:11.099 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.099 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.099 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:11.099 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548399+0000", 2024-09-18T04:39:11.099 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0080000000000001e-05 2024-09-18T04:39:11.099 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.100 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.100 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:11.100 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562383+0000", 2024-09-18T04:39:11.100 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013984671000000001 2024-09-18T04:39:11.100 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.100 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.100 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:11.100 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562442+0000", 2024-09-18T04:39:11.100 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8700999999999999e-05 2024-09-18T04:39:11.100 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:11.100 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:11.100 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:11.101 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.101 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.101 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:37 2.5 e51/28 2:bf39b8e4:ns::rbd_data.133e813ab5bc.0000000000000003:head v 51'5, mlcod=51'2)", 2024-09-18T04:39:11.101 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:39:11.101 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 28.513179959999999, 2024-09-18T04:39:11.101 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014264492, 2024-09-18T04:39:11.101 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:11.101 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:11.101 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:11.102 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.102 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:11.102 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:39:11.102 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:11.102 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.102 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.102 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:11.102 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:39:11.102 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:11.102 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.102 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.102 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:11.102 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548531+0000", 2024-09-18T04:39:11.102 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:11.103 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.103 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.103 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:11.103 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551500+0000", 2024-09-18T04:39:11.103 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0029688129999999998 2024-09-18T04:39:11.103 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.103 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.103 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:11.103 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551516+0000", 2024-09-18T04:39:11.103 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5994e-05 2024-09-18T04:39:11.104 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.104 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.104 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:11.104 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551526+0000", 2024-09-18T04:39:11.104 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9299999999999998e-06 2024-09-18T04:39:11.104 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.104 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.104 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:11.104 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551577+0000", 2024-09-18T04:39:11.104 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0871000000000001e-05 2024-09-18T04:39:11.104 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.105 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.105 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:11.105 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551602+0000", 2024-09-18T04:39:11.105 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4813000000000001e-05 2024-09-18T04:39:11.105 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.105 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.105 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:11.105 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562741+0000", 2024-09-18T04:39:11.105 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011139428999999999 2024-09-18T04:39:11.105 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.105 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.105 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:11.106 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562797+0000", 2024-09-18T04:39:11.106 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6329000000000002e-05 2024-09-18T04:39:11.106 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:11.106 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:11.106 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:11.106 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.106 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.106 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:49 2.4 e51/17 2:20f0b0c9:ns::rbd_data.133e813ab5bc.0000000000000009:head v 51'6, mlcod=51'4)", 2024-09-18T04:39:11.106 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:39:11.107 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 28.481745402000001, 2024-09-18T04:39:11.107 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013849046, 2024-09-18T04:39:11.107 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:11.107 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:11.107 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:11.107 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.107 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:11.107 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:39:11.107 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:11.107 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.108 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.108 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:11.108 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:39:11.108 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:11.108 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.108 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.108 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:11.108 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579966+0000", 2024-09-18T04:39:11.108 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:11.108 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.109 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.109 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:11.109 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582324+0000", 2024-09-18T04:39:11.109 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023578890000000002 2024-09-18T04:39:11.109 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.109 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.109 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:11.109 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582355+0000", 2024-09-18T04:39:11.109 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1485000000000001e-05 2024-09-18T04:39:11.109 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.109 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.110 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:11.110 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582371+0000", 2024-09-18T04:39:11.110 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6147999999999999e-05 2024-09-18T04:39:11.110 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.110 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.110 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:11.110 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582467+0000", 2024-09-18T04:39:11.110 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6059999999999998e-05 2024-09-18T04:39:11.110 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.110 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.110 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:11.110 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582505+0000", 2024-09-18T04:39:11.110 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7679000000000003e-05 2024-09-18T04:39:11.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.111 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.111 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:11.111 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593743+0000", 2024-09-18T04:39:11.111 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011238356 2024-09-18T04:39:11.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.111 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.111 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:11.111 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593816+0000", 2024-09-18T04:39:11.111 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2935000000000006e-05 2024-09-18T04:39:11.111 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:11.111 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:11.111 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:11.112 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.112 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.112 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:53 2.4 e51/17 2:39f7f174:ns::rbd_data.133e813ab5bc.000000000000000b:head v 51'7, mlcod=51'4)", 2024-09-18T04:39:11.112 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:39:11.112 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 28.467044247, 2024-09-18T04:39:11.112 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0079914159999999995, 2024-09-18T04:39:11.112 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:11.112 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:11.112 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:11.112 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.112 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:11.112 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:39:11.112 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:11.112 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.113 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.113 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:11.113 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:39:11.113 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:11.113 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.113 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.113 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:11.113 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594667+0000", 2024-09-18T04:39:11.113 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:11.113 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.113 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.113 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:11.113 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596446+0000", 2024-09-18T04:39:11.114 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001779033 2024-09-18T04:39:11.114 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.114 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.114 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:11.114 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596472+0000", 2024-09-18T04:39:11.114 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6120000000000001e-05 2024-09-18T04:39:11.114 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.114 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.114 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:11.114 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596500+0000", 2024-09-18T04:39:11.114 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7518e-05 2024-09-18T04:39:11.114 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.114 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.114 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:11.115 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596564+0000", 2024-09-18T04:39:11.115 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3880999999999994e-05 2024-09-18T04:39:11.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.115 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.115 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:11.115 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596596+0000", 2024-09-18T04:39:11.115 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2638999999999997e-05 2024-09-18T04:39:11.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.115 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.115 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:11.115 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602605+0000", 2024-09-18T04:39:11.115 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0060080699999999999 2024-09-18T04:39:11.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.115 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.116 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:11.116 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602660+0000", 2024-09-18T04:39:11.116 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5396000000000003e-05 2024-09-18T04:39:11.116 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:11.116 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:11.116 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:11.116 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.116 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.116 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:60 2.5 e51/28 2:a06f51e4:ns::rbd_data.133e813ab5bc.000000000000000f:head v 51'6, mlcod=51'4)", 2024-09-18T04:39:11.116 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:39:11.116 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 28.452227262000001, 2024-09-18T04:39:11.116 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0093954989999999999, 2024-09-18T04:39:11.117 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:11.117 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:11.117 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:11.117 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.117 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:11.117 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:39:11.117 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:11.117 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.117 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.117 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:11.117 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:39:11.117 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:11.117 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.118 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.118 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:11.118 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609483+0000", 2024-09-18T04:39:11.118 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:11.118 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.118 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.118 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:11.118 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611767+0000", 2024-09-18T04:39:11.118 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002283724 2024-09-18T04:39:11.118 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.118 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.118 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:11.118 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611791+0000", 2024-09-18T04:39:11.118 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3547e-05 2024-09-18T04:39:11.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.119 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.119 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:11.119 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611800+0000", 2024-09-18T04:39:11.119 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4979999999999995e-06 2024-09-18T04:39:11.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.119 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.119 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:11.119 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611864+0000", 2024-09-18T04:39:11.119 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3974999999999996e-05 2024-09-18T04:39:11.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.119 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.119 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:11.120 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611887+0000", 2024-09-18T04:39:11.120 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2633000000000002e-05 2024-09-18T04:39:11.120 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.120 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.120 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:11.120 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.618777+0000", 2024-09-18T04:39:11.120 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0068897979999999999 2024-09-18T04:39:11.120 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.120 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.120 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:11.120 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.618881+0000", 2024-09-18T04:39:11.120 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010437699999999999 2024-09-18T04:39:11.120 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:11.120 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:11.121 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:11.121 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.121 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.121 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:663 2.4 e57/17 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 57'13, mlcod=57'11)", 2024-09-18T04:39:11.121 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.355300+0000", 2024-09-18T04:39:11.121 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 19.706412326999999, 2024-09-18T04:39:11.121 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0049192760000000002, 2024-09-18T04:39:11.121 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:11.121 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:11.121 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:11.121 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.121 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:11.121 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355300+0000", 2024-09-18T04:39:11.122 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:11.122 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.122 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.122 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:11.122 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355300+0000", 2024-09-18T04:39:11.122 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:11.122 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.122 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.122 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:11.122 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355299+0000", 2024-09-18T04:39:11.122 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:11.122 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.122 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.122 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:11.123 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355876+0000", 2024-09-18T04:39:11.123 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00057730900000000003 2024-09-18T04:39:11.123 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.123 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.123 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:11.123 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355897+0000", 2024-09-18T04:39:11.123 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0477000000000001e-05 2024-09-18T04:39:11.123 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.123 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.123 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:11.123 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355905+0000", 2024-09-18T04:39:11.123 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5739999999999996e-06 2024-09-18T04:39:11.123 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.123 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.124 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:11.124 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355989+0000", 2024-09-18T04:39:11.124 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4002e-05 2024-09-18T04:39:11.124 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.124 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.124 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:11.124 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.356041+0000", 2024-09-18T04:39:11.124 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1347000000000002e-05 2024-09-18T04:39:11.124 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.124 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.124 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:11.124 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.360172+0000", 2024-09-18T04:39:11.124 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0041315090000000002 2024-09-18T04:39:11.124 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.125 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.125 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:11.125 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.360220+0000", 2024-09-18T04:39:11.125 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7584999999999999e-05 2024-09-18T04:39:11.125 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:11.125 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:11.125 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:11.125 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.125 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.125 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:679 2.5 e57/28 2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head v 57'20, mlcod=57'18)", 2024-09-18T04:39:11.125 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.445316+0000", 2024-09-18T04:39:11.125 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 19.616396850000001, 2024-09-18T04:39:11.125 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0047016760000000001, 2024-09-18T04:39:11.125 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:11.126 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:11.126 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:11.126 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.126 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:11.126 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445316+0000", 2024-09-18T04:39:11.126 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:11.126 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.126 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.126 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:11.126 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445316+0000", 2024-09-18T04:39:11.126 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:11.126 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.126 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.126 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:11.127 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445314+0000", 2024-09-18T04:39:11.127 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:11.127 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.127 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.127 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:11.127 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445947+0000", 2024-09-18T04:39:11.127 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00063298900000000003 2024-09-18T04:39:11.127 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.127 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.127 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:11.127 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445968+0000", 2024-09-18T04:39:11.127 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.054e-05 2024-09-18T04:39:11.127 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.127 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.128 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:11.128 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445981+0000", 2024-09-18T04:39:11.128 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2697000000000001e-05 2024-09-18T04:39:11.128 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.128 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.128 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:11.128 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.446102+0000", 2024-09-18T04:39:11.128 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012189300000000001 2024-09-18T04:39:11.128 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.128 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.128 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:11.128 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.446151+0000", 2024-09-18T04:39:11.128 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8279000000000003e-05 2024-09-18T04:39:11.128 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.129 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.129 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:11.129 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.449953+0000", 2024-09-18T04:39:11.129 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003801979 2024-09-18T04:39:11.129 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.129 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.129 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:11.129 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.450018+0000", 2024-09-18T04:39:11.129 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4873e-05 2024-09-18T04:39:11.129 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:11.129 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:11.129 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:11.129 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.130 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.130 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:699 2.4 e57/17 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 57'17, mlcod=57'15)", 2024-09-18T04:39:11.130 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.566073+0000", 2024-09-18T04:39:11.130 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 19.495639337, 2024-09-18T04:39:11.130 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055398380000000001, 2024-09-18T04:39:11.130 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:11.130 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:11.130 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:11.130 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.131 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:11.131 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566073+0000", 2024-09-18T04:39:11.131 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:11.131 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.131 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.131 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:11.131 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566073+0000", 2024-09-18T04:39:11.131 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:11.131 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.131 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.131 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:11.131 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566071+0000", 2024-09-18T04:39:11.131 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:11.131 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.131 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.132 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:11.132 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566637+0000", 2024-09-18T04:39:11.132 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00056516699999999999 2024-09-18T04:39:11.132 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.132 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.132 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:11.132 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566668+0000", 2024-09-18T04:39:11.132 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1544999999999997e-05 2024-09-18T04:39:11.132 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.132 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.133 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:11.133 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566685+0000", 2024-09-18T04:39:11.133 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6857000000000001e-05 2024-09-18T04:39:11.133 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.133 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.133 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:11.133 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566816+0000", 2024-09-18T04:39:11.133 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013092799999999999 2024-09-18T04:39:11.133 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.133 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.133 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:11.134 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566875+0000", 2024-09-18T04:39:11.134 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9583999999999997e-05 2024-09-18T04:39:11.134 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.134 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.134 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:11.134 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.571567+0000", 2024-09-18T04:39:11.134 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.004691682 2024-09-18T04:39:11.134 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.134 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.134 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:11.134 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.571613+0000", 2024-09-18T04:39:11.135 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6125000000000003e-05 2024-09-18T04:39:11.135 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:11.135 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:11.135 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:11.135 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.135 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.135 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:98 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'27, mlcod=60'25)", 2024-09-18T04:39:11.135 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.847978+0000", 2024-09-18T04:39:11.135 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 10.213734777999999, 2024-09-18T04:39:11.135 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0044617709999999998, 2024-09-18T04:39:11.135 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:11.136 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:11.136 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:11.136 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.136 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:11.136 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.847978+0000", 2024-09-18T04:39:11.136 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:11.136 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.136 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.136 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:11.136 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.847978+0000", 2024-09-18T04:39:11.137 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:11.137 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.137 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.137 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:11.137 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.847977+0000", 2024-09-18T04:39:11.137 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:11.137 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.137 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.137 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:11.137 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849038+0000", 2024-09-18T04:39:11.137 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0010614820000000001 2024-09-18T04:39:11.138 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.138 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.138 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:11.138 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849065+0000", 2024-09-18T04:39:11.138 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7127999999999998e-05 2024-09-18T04:39:11.138 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.138 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.138 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:11.138 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849082+0000", 2024-09-18T04:39:11.138 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7170999999999999e-05 2024-09-18T04:39:11.138 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.139 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.139 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:11.139 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849188+0000", 2024-09-18T04:39:11.139 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010605800000000001 2024-09-18T04:39:11.139 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.139 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.139 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:11.139 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849233+0000", 2024-09-18T04:39:11.139 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4579000000000001e-05 2024-09-18T04:39:11.140 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.140 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.140 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:11.140 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.852370+0000", 2024-09-18T04:39:11.140 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00313751 2024-09-18T04:39:11.140 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.140 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.140 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:11.141 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.852440+0000", 2024-09-18T04:39:11.141 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9238000000000004e-05 2024-09-18T04:39:11.141 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:11.141 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:11.141 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:11.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.141 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.141 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:100 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'29, mlcod=64'27)", 2024-09-18T04:39:11.141 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.858766+0000", 2024-09-18T04:39:11.141 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 10.202946449000001, 2024-09-18T04:39:11.142 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0061964840000000004, 2024-09-18T04:39:11.142 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:11.142 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:11.142 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:11.142 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.142 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:11.142 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.858766+0000", 2024-09-18T04:39:11.142 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:11.142 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.142 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.142 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:11.142 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.858766+0000", 2024-09-18T04:39:11.142 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:11.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.143 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.143 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:11.143 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.858765+0000", 2024-09-18T04:39:11.143 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:11.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.143 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.143 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:11.143 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861612+0000", 2024-09-18T04:39:11.143 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028473169999999998 2024-09-18T04:39:11.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.143 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.143 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:11.143 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861643+0000", 2024-09-18T04:39:11.144 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1356000000000003e-05 2024-09-18T04:39:11.144 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.144 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.144 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:11.144 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861656+0000", 2024-09-18T04:39:11.144 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2899e-05 2024-09-18T04:39:11.144 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.144 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.144 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:11.145 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861773+0000", 2024-09-18T04:39:11.145 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011637999999999999 2024-09-18T04:39:11.145 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.145 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.145 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:11.145 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861808+0000", 2024-09-18T04:39:11.145 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5002999999999997e-05 2024-09-18T04:39:11.145 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.145 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.145 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:11.145 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.864876+0000", 2024-09-18T04:39:11.145 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030680719999999998 2024-09-18T04:39:11.145 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.146 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.146 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:11.146 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.864963+0000", 2024-09-18T04:39:11.146 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7114000000000007e-05 2024-09-18T04:39:11.146 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:11.146 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:11.146 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:11.146 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.146 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.146 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:102 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'31, mlcod=64'29)", 2024-09-18T04:39:11.146 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.872098+0000", 2024-09-18T04:39:11.146 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 10.189615192, 2024-09-18T04:39:11.146 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0064676450000000002, 2024-09-18T04:39:11.146 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:11.147 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:11.147 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:11.147 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.147 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:11.147 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.872098+0000", 2024-09-18T04:39:11.147 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:11.147 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.147 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.147 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:11.147 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.872098+0000", 2024-09-18T04:39:11.147 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:11.147 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.147 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.148 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:11.148 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.872096+0000", 2024-09-18T04:39:11.148 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:11.148 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.148 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.148 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:11.148 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875176+0000", 2024-09-18T04:39:11.148 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030799809999999999 2024-09-18T04:39:11.148 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.148 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.148 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:11.148 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875204+0000", 2024-09-18T04:39:11.148 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8025999999999999e-05 2024-09-18T04:39:11.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.149 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.149 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:11.149 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875216+0000", 2024-09-18T04:39:11.149 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2072999999999999e-05 2024-09-18T04:39:11.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.149 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.149 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:11.149 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875307+0000", 2024-09-18T04:39:11.149 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0373999999999999e-05 2024-09-18T04:39:11.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.149 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.149 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:11.149 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875349+0000", 2024-09-18T04:39:11.150 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2299000000000003e-05 2024-09-18T04:39:11.150 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.150 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.150 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:11.150 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.878496+0000", 2024-09-18T04:39:11.150 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0031468960000000002 2024-09-18T04:39:11.150 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.150 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.150 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:11.151 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.878565+0000", 2024-09-18T04:39:11.151 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9465e-05 2024-09-18T04:39:11.151 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:11.151 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:11.151 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:11.151 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.151 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.151 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:104 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'33, mlcod=64'31)", 2024-09-18T04:39:11.151 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.885100+0000", 2024-09-18T04:39:11.151 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 10.176612657, 2024-09-18T04:39:11.151 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0051888159999999997, 2024-09-18T04:39:11.151 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:11.152 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:11.152 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:11.152 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.152 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:11.152 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.885100+0000", 2024-09-18T04:39:11.152 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:11.152 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.152 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.152 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:11.152 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.885100+0000", 2024-09-18T04:39:11.152 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:11.152 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.152 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.152 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:11.153 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.885098+0000", 2024-09-18T04:39:11.153 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:11.153 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.153 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.153 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:11.153 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886809+0000", 2024-09-18T04:39:11.153 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017110459999999999 2024-09-18T04:39:11.153 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.153 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.153 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:11.153 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886842+0000", 2024-09-18T04:39:11.153 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2638999999999997e-05 2024-09-18T04:39:11.153 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.154 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.154 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:11.154 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886855+0000", 2024-09-18T04:39:11.154 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3380000000000001e-05 2024-09-18T04:39:11.154 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.154 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.154 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:11.154 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886939+0000", 2024-09-18T04:39:11.154 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3733999999999995e-05 2024-09-18T04:39:11.154 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.154 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.154 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:11.154 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886981+0000", 2024-09-18T04:39:11.155 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1956000000000003e-05 2024-09-18T04:39:11.155 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.155 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.155 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:11.155 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.890201+0000", 2024-09-18T04:39:11.155 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003219927 2024-09-18T04:39:11.155 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.155 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.155 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:11.155 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.890289+0000", 2024-09-18T04:39:11.155 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7997999999999993e-05 2024-09-18T04:39:11.155 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:11.155 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:11.155 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:11.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.156 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.156 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:145 2.4 e64/62 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 64'37, mlcod=64'35)", 2024-09-18T04:39:11.156 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:01.099398+0000", 2024-09-18T04:39:11.156 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 9.962315169, 2024-09-18T04:39:11.156 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0050078459999999998, 2024-09-18T04:39:11.156 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:11.156 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:11.156 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:11.156 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.156 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:11.156 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.099398+0000", 2024-09-18T04:39:11.156 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:11.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.157 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.157 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:11.157 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.099398+0000", 2024-09-18T04:39:11.157 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:11.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.157 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.157 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:11.157 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.099396+0000", 2024-09-18T04:39:11.157 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:11.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.157 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.157 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:11.157 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100199+0000", 2024-09-18T04:39:11.157 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00080338000000000002 2024-09-18T04:39:11.158 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.158 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.158 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:11.158 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100230+0000", 2024-09-18T04:39:11.158 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0716000000000002e-05 2024-09-18T04:39:11.158 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.158 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.158 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:11.158 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100243+0000", 2024-09-18T04:39:11.158 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3359e-05 2024-09-18T04:39:11.158 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.158 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.158 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:11.158 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100337+0000", 2024-09-18T04:39:11.159 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4387000000000003e-05 2024-09-18T04:39:11.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.159 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.159 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:11.159 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100396+0000", 2024-09-18T04:39:11.159 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8164000000000001e-05 2024-09-18T04:39:11.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.159 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.159 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:11.159 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.104309+0000", 2024-09-18T04:39:11.159 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0039137800000000004 2024-09-18T04:39:11.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.159 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.160 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:11.160 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.104405+0000", 2024-09-18T04:39:11.160 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6039999999999995e-05 2024-09-18T04:39:11.160 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:11.160 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:11.160 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:11.160 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.160 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.160 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:147 2.4 e64/62 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 64'39, mlcod=64'37)", 2024-09-18T04:39:11.160 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:01.110658+0000", 2024-09-18T04:39:11.160 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 9.9510550799999997, 2024-09-18T04:39:11.160 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0046316910000000003, 2024-09-18T04:39:11.161 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:11.161 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:11.161 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:11.161 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.161 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:11.161 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.110658+0000", 2024-09-18T04:39:11.161 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:11.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.161 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.161 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:11.161 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.110658+0000", 2024-09-18T04:39:11.161 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:11.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.161 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.162 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:11.162 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.110656+0000", 2024-09-18T04:39:11.162 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:11.162 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.162 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.162 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:11.162 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111444+0000", 2024-09-18T04:39:11.162 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00078762400000000001 2024-09-18T04:39:11.162 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.162 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.162 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:11.162 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111474+0000", 2024-09-18T04:39:11.162 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0085000000000001e-05 2024-09-18T04:39:11.163 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.163 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.163 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:11.163 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111486+0000", 2024-09-18T04:39:11.163 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2645e-05 2024-09-18T04:39:11.163 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.163 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.163 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:11.163 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111581+0000", 2024-09-18T04:39:11.163 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4432000000000007e-05 2024-09-18T04:39:11.163 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.163 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.163 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:11.163 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111631+0000", 2024-09-18T04:39:11.163 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9901000000000002e-05 2024-09-18T04:39:11.164 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.164 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.164 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:11.164 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.115209+0000", 2024-09-18T04:39:11.164 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003577916 2024-09-18T04:39:11.164 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:11.164 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:11.164 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:11.164 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.115289+0000", 2024-09-18T04:39:11.164 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0704999999999994e-05 2024-09-18T04:39:11.164 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:11.164 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:11.164 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:11.165 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:11.165 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:11.165 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:11.166 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:11.161+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:11.167 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:39:11.167 INFO:tasks.workunit.client.0.smithi115.stderr:+ ps -C rbd-nbd 2024-09-18T04:39:11.167 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false grep '^ *19107\b' 2024-09-18T04:39:11.167 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep '^ *19107\b' 2024-09-18T04:39:11.168 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:39:11.168 INFO:tasks.workunit.client.0.smithi115.stderr:+ sleep 0.5 2024-09-18T04:39:11.168 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:39:11.169 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:39:11.169 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 reset_purged_snaps_last 2024-09-18T04:39:11.170 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:39:11.170 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:39:11.171 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:39:11.184 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:39:11.224 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:39:11.226 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:39:11.226 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:39:11.226 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:39:11.227 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:39:11.227 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:39:11.227 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:39:11.267 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:11.261+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:11.281 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:11.281 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:11.290 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:39:11.345 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 reset_purged_snaps_last 2024-09-18T04:39:11.368 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:11.361+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:11.450 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:11.450 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:11.460 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:39:11.502 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:11.497+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:11.515 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2024-09-18T04:39:11.538 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-18T04:39:11.569 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:11.561+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:11.620 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:11.621 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:11.630 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:39:11.640 INFO:tasks.workunit.client.0.smithi115.stderr:+ DEV= 2024-09-18T04:39:11.640 INFO:tasks.workunit.client.0.smithi115.stderr:+ unmap_device rbd/testrbdnbd17342 19059 2024-09-18T04:39:11.640 INFO:tasks.workunit.client.0.smithi115.stderr:+ local args=rbd/testrbdnbd17342 2024-09-18T04:39:11.640 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pid=19059 2024-09-18T04:39:11.640 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo rbd device --device-type nbd unmap rbd/testrbdnbd17342 2024-09-18T04:39:11.641 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:39:11.641 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:39:11.642 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:39:11.642 INFO:tasks.workunit.client.0.smithi115.stderr:++ which rbd 2024-09-18T04:39:11.645 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/bin/rbd 2024-09-18T04:39:11.645 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:39:11.645 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap rbd/testrbdnbd17342 2024-09-18T04:39:11.670 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:11.665+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:39:11.735 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:11.735 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:11.746 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:39:11.747 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd device --device-type nbd list 2024-09-18T04:39:11.747 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false grep '^19059\b' 2024-09-18T04:39:11.747 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep '^19059\b' 2024-09-18T04:39:11.771 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:11.765+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:11.811 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:39:11.813 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false grep '^ *19059\b' 2024-09-18T04:39:11.813 INFO:tasks.workunit.client.0.smithi115.stderr:+ ps -C rbd-nbd 2024-09-18T04:39:11.813 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep '^ *19059\b' 2024-09-18T04:39:11.824 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:39:11.825 INFO:tasks.workunit.client.0.smithi115.stderr:+ sleep 0.5 2024-09-18T04:39:11.872 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:11.865+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:11.898 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:11.898 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:11.908 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:39:11.973 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:11.965+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:12.013 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:12.013 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:12.021 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:39:12.073 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:12.065+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:12.135 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:12.136 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:12.145 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:39:12.174 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:12.169+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:12.257 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:12.257 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:12.267 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:39:12.310 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:12.305+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:12.326 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' rbd = rbd ']' 2024-09-18T04:39:12.327 INFO:tasks.workunit.client.0.smithi115.stderr:++ _sudo rbd device --device-type nbd map testrbdnbd17342 2024-09-18T04:39:12.327 INFO:tasks.workunit.client.0.smithi115.stderr:++ local cmd 2024-09-18T04:39:12.327 INFO:tasks.workunit.client.0.smithi115.stderr:+++ id -u 2024-09-18T04:39:12.329 INFO:tasks.workunit.client.0.smithi115.stderr:++ '[' 1000 -eq 0 ']' 2024-09-18T04:39:12.330 INFO:tasks.workunit.client.0.smithi115.stderr:+++ which rbd 2024-09-18T04:39:12.333 INFO:tasks.workunit.client.0.smithi115.stderr:++ cmd=/usr/bin/rbd 2024-09-18T04:39:12.333 INFO:tasks.workunit.client.0.smithi115.stderr:++ shift 2024-09-18T04:39:12.333 INFO:tasks.workunit.client.0.smithi115.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map testrbdnbd17342 2024-09-18T04:39:12.375 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:12.373+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:39:12.421 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:12.422 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:12.432 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:39:12.477 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:12.469+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:12.502 INFO:tasks.workunit.client.0.smithi115.stderr:+ DEV=/dev/nbd0 2024-09-18T04:39:12.502 INFO:tasks.workunit.client.0.smithi115.stderr:+ get_pid rbd 2024-09-18T04:39:12.502 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pool=rbd 2024-09-18T04:39:12.502 INFO:tasks.workunit.client.0.smithi115.stderr:+ local ns= 2024-09-18T04:39:12.503 INFO:tasks.workunit.client.0.smithi115.stderr:++ rbd device --device-type nbd --format xml list 2024-09-18T04:39:12.503 INFO:tasks.workunit.client.0.smithi115.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd17342'\''][device='\''/dev/nbd0'\'']/id' 2024-09-18T04:39:12.536 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-09-18T04:39:12.541 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:12.541 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:12.551 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:39:12.578 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:12.573+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:12.584 INFO:tasks.workunit.client.0.smithi115.stderr:+ PID=19203 2024-09-18T04:39:12.584 INFO:tasks.workunit.client.0.smithi115.stderr:+ test -n 19203 2024-09-18T04:39:12.584 INFO:tasks.workunit.client.0.smithi115.stderr:+ ps -p 19203 -C rbd-nbd 2024-09-18T04:39:12.594 INFO:tasks.workunit.client.0.smithi115.stdout: PID TTY TIME CMD 2024-09-18T04:39:12.594 INFO:tasks.workunit.client.0.smithi115.stdout: 19203 ? 00:00:00 rbd-nbd 2024-09-18T04:39:12.595 INFO:tasks.workunit.client.0.smithi115.stderr:+ unmap_device testrbdnbd17342 19203 2024-09-18T04:39:12.595 INFO:tasks.workunit.client.0.smithi115.stderr:+ local args=testrbdnbd17342 2024-09-18T04:39:12.595 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pid=19203 2024-09-18T04:39:12.595 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo rbd device --device-type nbd unmap testrbdnbd17342 2024-09-18T04:39:12.595 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:39:12.595 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:39:12.597 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:39:12.597 INFO:tasks.workunit.client.0.smithi115.stderr:++ which rbd 2024-09-18T04:39:12.598 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/bin/rbd 2024-09-18T04:39:12.598 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:39:12.599 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap testrbdnbd17342 2024-09-18T04:39:12.656 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:12.656 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:12.666 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:39:12.678 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:12.673+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:12.709 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd device --device-type nbd list 2024-09-18T04:39:12.709 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false grep '^19203\b' 2024-09-18T04:39:12.709 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep '^19203\b' 2024-09-18T04:39:12.770 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:12.770 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:12.779 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:12.773+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:12.780 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:39:12.780 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:39:12.781 INFO:tasks.workunit.client.0.smithi115.stderr:+ ps -C rbd-nbd 2024-09-18T04:39:12.781 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false grep '^ *19203\b' 2024-09-18T04:39:12.781 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep '^ *19203\b' 2024-09-18T04:39:12.791 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:39:12.792 INFO:tasks.workunit.client.0.smithi115.stderr:+ sleep 0.5 2024-09-18T04:39:12.880 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:12.873+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:12.930 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:12.930 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:12.940 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:39:12.988 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:12.981+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:13.045 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:13.045 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:13.055 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:39:13.126 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:13.117+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:13.184 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:13.184 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:13.185 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:13.177+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:13.196 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:39:13.242 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T04:39:13.285 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:13.277+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:13.294 INFO:tasks.workunit.client.0.smithi115.stderr:+ DEV= 2024-09-18T04:39:13.294 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd config global set global rbd_default_pool new_default_pool17342 2024-09-18T04:39:13.319 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:13.320 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:13.330 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:39:13.355 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd create --size 10M testrbdnbd17342 2024-09-18T04:39:13.386 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:13.381+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:13.390 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:39:13.412 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:39:13.428 INFO:tasks.workunit.client.0.smithi115.stderr:++ _sudo rbd device --device-type nbd map testrbdnbd17342 2024-09-18T04:39:13.428 INFO:tasks.workunit.client.0.smithi115.stderr:++ local cmd 2024-09-18T04:39:13.429 INFO:tasks.workunit.client.0.smithi115.stderr:+++ id -u 2024-09-18T04:39:13.431 INFO:tasks.workunit.client.0.smithi115.stderr:++ '[' 1000 -eq 0 ']' 2024-09-18T04:39:13.431 INFO:tasks.workunit.client.0.smithi115.stderr:+++ which rbd 2024-09-18T04:39:13.434 INFO:tasks.workunit.client.0.smithi115.stderr:++ cmd=/usr/bin/rbd 2024-09-18T04:39:13.434 INFO:tasks.workunit.client.0.smithi115.stderr:++ shift 2024-09-18T04:39:13.434 INFO:tasks.workunit.client.0.smithi115.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map testrbdnbd17342 2024-09-18T04:39:13.436 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:39:13.442 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:13.442 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:13.442 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:39:13.442 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:39:13.442 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:13.452 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:39:13.459 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:39:13.482 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:39:13.486 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:13.481+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:13.504 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:39:13.506 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:39:13.506 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:39:13.506 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:39:13.506 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:39:13.506 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:39:13.506 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:39:13.570 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:13.570 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:39:13.570 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:39:13.571 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:39:13.571 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.571 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:37 2.5 e51/28 2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:head v 51'1, mlcod=0'0)", 2024-09-18T04:39:13.571 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:39:13.571 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 31.273634095999999, 2024-09-18T04:39:13.571 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012279973, 2024-09-18T04:39:13.571 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:13.571 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:13.571 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:13.571 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.572 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:13.572 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:39:13.572 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:13.572 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.572 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.572 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:13.573 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:39:13.573 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:13.573 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.573 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.573 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:13.573 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292468+0000", 2024-09-18T04:39:13.573 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:13.573 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.573 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.573 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:13.573 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295837+0000", 2024-09-18T04:39:13.573 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0033691699999999999 2024-09-18T04:39:13.574 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.574 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.574 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:13.574 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295860+0000", 2024-09-18T04:39:13.574 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2756999999999999e-05 2024-09-18T04:39:13.574 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.574 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.574 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:13.574 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295872+0000", 2024-09-18T04:39:13.574 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2442e-05 2024-09-18T04:39:13.574 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.574 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.574 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:13.575 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295934+0000", 2024-09-18T04:39:13.575 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1187999999999998e-05 2024-09-18T04:39:13.575 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.575 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.575 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:13.575 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295967+0000", 2024-09-18T04:39:13.575 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3185999999999997e-05 2024-09-18T04:39:13.575 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.575 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.576 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:13.576 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304697+0000", 2024-09-18T04:39:13.576 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0087298059999999997 2024-09-18T04:39:13.576 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.576 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.577 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:13.577 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304749+0000", 2024-09-18T04:39:13.577 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1922000000000002e-05 2024-09-18T04:39:13.577 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:13.577 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:13.577 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:13.577 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.578 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.578 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:49 2.5 e51/28 2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head v 51'2, mlcod=0'0)", 2024-09-18T04:39:13.578 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:39:13.578 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 31.240307258000001, 2024-09-18T04:39:13.578 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018033041999999999, 2024-09-18T04:39:13.578 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:13.578 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:13.578 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:13.578 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.578 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:13.578 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:39:13.578 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:13.578 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.579 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.579 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:13.579 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:39:13.579 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:13.579 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.579 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.579 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:13.579 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325793+0000", 2024-09-18T04:39:13.579 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:13.579 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.579 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.580 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:13.580 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329038+0000", 2024-09-18T04:39:13.580 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0032444010000000001 2024-09-18T04:39:13.580 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.580 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.580 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:13.580 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329065+0000", 2024-09-18T04:39:13.580 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7500000000000001e-05 2024-09-18T04:39:13.580 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.580 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.580 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:13.580 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329077+0000", 2024-09-18T04:39:13.580 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1919999999999999e-05 2024-09-18T04:39:13.580 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.581 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.581 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:13.581 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329149+0000", 2024-09-18T04:39:13.581 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2349999999999997e-05 2024-09-18T04:39:13.581 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.581 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.581 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:13.581 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329192+0000", 2024-09-18T04:39:13.581 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2744000000000003e-05 2024-09-18T04:39:13.581 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.581 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.581 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:13.581 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343773+0000", 2024-09-18T04:39:13.582 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014581246000000001 2024-09-18T04:39:13.582 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.582 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.582 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:13.582 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343829+0000", 2024-09-18T04:39:13.582 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5143999999999999e-05 2024-09-18T04:39:13.582 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:13.582 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:13.582 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:13.582 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.582 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.582 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:47 2.4 e51/17 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 51'3, mlcod=51'1)", 2024-09-18T04:39:13.582 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:39:13.582 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 31.239726993000001, 2024-09-18T04:39:13.583 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018960186, 2024-09-18T04:39:13.583 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:13.583 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:13.583 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:13.583 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.583 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:13.583 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:39:13.583 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:13.583 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.583 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.583 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:13.583 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:39:13.583 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:13.583 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.583 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.584 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:13.584 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326374+0000", 2024-09-18T04:39:13.584 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:13.584 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.584 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.584 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:13.584 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330883+0000", 2024-09-18T04:39:13.584 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0045084740000000002 2024-09-18T04:39:13.584 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.584 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.584 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:13.584 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330901+0000", 2024-09-18T04:39:13.584 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7890999999999999e-05 2024-09-18T04:39:13.584 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.585 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.585 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:13.585 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330912+0000", 2024-09-18T04:39:13.585 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1041e-05 2024-09-18T04:39:13.585 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.585 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.585 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:13.585 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.331003+0000", 2024-09-18T04:39:13.585 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1323999999999995e-05 2024-09-18T04:39:13.585 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.585 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.585 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:13.585 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.331030+0000", 2024-09-18T04:39:13.585 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7351999999999998e-05 2024-09-18T04:39:13.586 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.586 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.586 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:13.586 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345287+0000", 2024-09-18T04:39:13.586 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014256332 2024-09-18T04:39:13.586 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.586 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.586 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:13.586 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345336+0000", 2024-09-18T04:39:13.586 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9356e-05 2024-09-18T04:39:13.586 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:13.586 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:13.586 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:13.586 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.587 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.587 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:58 2.5 e51/28 2:b227e6b1:::rbd_data.125fc09df6d1.000000000000000e:head v 51'3, mlcod=51'1)", 2024-09-18T04:39:13.587 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:39:13.587 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 31.199300365999999, 2024-09-18T04:39:13.587 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016513678, 2024-09-18T04:39:13.587 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:13.587 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:13.587 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:13.587 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.587 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:13.587 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:39:13.587 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:13.587 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.588 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.588 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:13.588 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:39:13.588 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:13.588 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.588 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.588 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:13.588 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366800+0000", 2024-09-18T04:39:13.588 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:13.588 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.588 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.588 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:13.588 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368644+0000", 2024-09-18T04:39:13.588 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001844399 2024-09-18T04:39:13.589 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.589 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.589 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:13.589 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368681+0000", 2024-09-18T04:39:13.589 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6526999999999998e-05 2024-09-18T04:39:13.589 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.589 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.589 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:13.589 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368698+0000", 2024-09-18T04:39:13.589 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7272999999999999e-05 2024-09-18T04:39:13.589 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.589 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.589 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:13.590 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368773+0000", 2024-09-18T04:39:13.590 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4862000000000003e-05 2024-09-18T04:39:13.590 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.590 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.590 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:13.590 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368801+0000", 2024-09-18T04:39:13.590 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7925000000000001e-05 2024-09-18T04:39:13.590 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.590 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.590 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:13.590 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383261+0000", 2024-09-18T04:39:13.590 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014460377999999999 2024-09-18T04:39:13.590 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.590 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.591 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:13.591 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383316+0000", 2024-09-18T04:39:13.591 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4623999999999997e-05 2024-09-18T04:39:13.591 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:13.591 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:13.591 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:13.591 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.591 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.591 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:56 2.4 e51/17 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 51'4, mlcod=51'2)", 2024-09-18T04:39:13.591 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:39:13.591 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 31.197296340000001, 2024-09-18T04:39:13.591 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015105221, 2024-09-18T04:39:13.591 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:13.591 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:13.592 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:13.592 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.592 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:13.592 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:39:13.592 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:13.592 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.592 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.592 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:13.592 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:39:13.592 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:13.592 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.592 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.592 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:13.592 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368805+0000", 2024-09-18T04:39:13.593 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:13.593 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.593 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.593 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:13.593 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371603+0000", 2024-09-18T04:39:13.593 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027982660000000002 2024-09-18T04:39:13.593 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.593 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.593 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:13.593 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371633+0000", 2024-09-18T04:39:13.593 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0073000000000002e-05 2024-09-18T04:39:13.593 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.593 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.593 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:13.594 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371652+0000", 2024-09-18T04:39:13.594 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8597000000000002e-05 2024-09-18T04:39:13.594 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.594 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.594 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:13.594 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371752+0000", 2024-09-18T04:39:13.594 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9772999999999995e-05 2024-09-18T04:39:13.594 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.594 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.594 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:13.594 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371793+0000", 2024-09-18T04:39:13.594 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1173e-05 2024-09-18T04:39:13.594 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.595 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.595 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:13.595 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383875+0000", 2024-09-18T04:39:13.595 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012081852000000001 2024-09-18T04:39:13.595 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.595 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.595 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:13.595 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383912+0000", 2024-09-18T04:39:13.595 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7045000000000002e-05 2024-09-18T04:39:13.595 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:13.595 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:13.595 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:13.595 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.595 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.596 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:36 2.5 e51/28 2:bd5b8dd8:ns::rbd_data.133e813ab5bc.0000000000000002:head v 51'4, mlcod=51'2)", 2024-09-18T04:39:13.596 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:39:13.596 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 31.028577348999999, 2024-09-18T04:39:13.596 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020758161000000001, 2024-09-18T04:39:13.596 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:13.596 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:13.596 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:13.596 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.596 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:13.596 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:39:13.596 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:13.596 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.596 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.596 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:13.597 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:39:13.597 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:13.597 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.597 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.597 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:13.597 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537523+0000", 2024-09-18T04:39:13.597 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:13.597 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.597 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.597 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:13.597 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541578+0000", 2024-09-18T04:39:13.597 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0040549710000000001 2024-09-18T04:39:13.597 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.597 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.597 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:13.598 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541608+0000", 2024-09-18T04:39:13.598 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9349e-05 2024-09-18T04:39:13.598 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.598 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.598 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:13.598 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541621+0000", 2024-09-18T04:39:13.598 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2736e-05 2024-09-18T04:39:13.598 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.598 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.598 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:13.598 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541715+0000", 2024-09-18T04:39:13.598 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4197999999999995e-05 2024-09-18T04:39:13.598 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.598 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.599 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:13.599 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541755+0000", 2024-09-18T04:39:13.599 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9818999999999999e-05 2024-09-18T04:39:13.599 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.599 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.599 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:13.599 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558217+0000", 2024-09-18T04:39:13.599 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016462234999999999 2024-09-18T04:39:13.599 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.599 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.599 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:13.599 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558284+0000", 2024-09-18T04:39:13.599 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6774999999999997e-05 2024-09-18T04:39:13.599 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:13.600 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:13.600 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:13.600 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.600 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.600 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:39 2.4 e51/17 2:304364d1:ns::rbd_data.133e813ab5bc.0000000000000004:head v 51'5, mlcod=51'3)", 2024-09-18T04:39:13.600 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:39:13.600 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 31.020438221999999, 2024-09-18T04:39:13.600 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016777558000000001, 2024-09-18T04:39:13.600 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:13.600 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:13.600 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:13.600 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.600 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:13.600 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:39:13.601 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:13.601 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.601 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.601 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:13.601 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:39:13.601 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:13.601 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.601 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.601 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:13.601 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545663+0000", 2024-09-18T04:39:13.601 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:13.601 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.602 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.602 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:13.602 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548286+0000", 2024-09-18T04:39:13.602 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026233390000000001 2024-09-18T04:39:13.602 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.602 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.602 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:13.602 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548305+0000", 2024-09-18T04:39:13.602 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8425000000000001e-05 2024-09-18T04:39:13.602 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.602 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.602 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:13.602 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548316+0000", 2024-09-18T04:39:13.602 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1163e-05 2024-09-18T04:39:13.603 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.603 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.603 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:13.603 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548379+0000", 2024-09-18T04:39:13.603 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2780000000000005e-05 2024-09-18T04:39:13.603 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.603 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.603 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:13.603 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548399+0000", 2024-09-18T04:39:13.603 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0080000000000001e-05 2024-09-18T04:39:13.603 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.603 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.603 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:13.603 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562383+0000", 2024-09-18T04:39:13.604 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013984671000000001 2024-09-18T04:39:13.604 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.604 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.604 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:13.604 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562442+0000", 2024-09-18T04:39:13.604 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8700999999999999e-05 2024-09-18T04:39:13.604 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:13.604 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:13.604 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:13.604 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.604 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.604 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:37 2.5 e51/28 2:bf39b8e4:ns::rbd_data.133e813ab5bc.0000000000000003:head v 51'5, mlcod=51'2)", 2024-09-18T04:39:13.604 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:39:13.604 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 31.017569952999999, 2024-09-18T04:39:13.605 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014264492, 2024-09-18T04:39:13.605 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:13.605 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:13.605 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:13.605 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.605 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:13.605 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:39:13.605 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:13.605 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.605 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.605 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:13.605 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:39:13.605 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:13.605 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.606 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.606 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:13.606 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548531+0000", 2024-09-18T04:39:13.606 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:13.606 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.606 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.606 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:13.606 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551500+0000", 2024-09-18T04:39:13.606 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0029688129999999998 2024-09-18T04:39:13.606 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.606 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.606 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:13.606 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551516+0000", 2024-09-18T04:39:13.607 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5994e-05 2024-09-18T04:39:13.607 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.607 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.607 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:13.607 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551526+0000", 2024-09-18T04:39:13.607 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9299999999999998e-06 2024-09-18T04:39:13.607 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.607 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.607 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:13.607 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551577+0000", 2024-09-18T04:39:13.607 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0871000000000001e-05 2024-09-18T04:39:13.607 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.607 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.607 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:13.607 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551602+0000", 2024-09-18T04:39:13.608 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4813000000000001e-05 2024-09-18T04:39:13.608 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.608 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.608 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:13.608 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562741+0000", 2024-09-18T04:39:13.608 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011139428999999999 2024-09-18T04:39:13.608 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.608 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.608 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:13.608 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562797+0000", 2024-09-18T04:39:13.608 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6329000000000002e-05 2024-09-18T04:39:13.608 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:13.608 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:13.609 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:13.609 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.609 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.609 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:49 2.4 e51/17 2:20f0b0c9:ns::rbd_data.133e813ab5bc.0000000000000009:head v 51'6, mlcod=51'4)", 2024-09-18T04:39:13.609 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:39:13.609 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 30.986135395000002, 2024-09-18T04:39:13.609 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013849046, 2024-09-18T04:39:13.609 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:13.609 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:13.609 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:13.609 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.609 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:13.609 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:39:13.609 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:13.610 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.610 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.610 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:13.610 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:39:13.610 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:13.610 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.610 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.610 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:13.610 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579966+0000", 2024-09-18T04:39:13.610 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:13.610 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.610 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.610 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:13.611 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582324+0000", 2024-09-18T04:39:13.611 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023578890000000002 2024-09-18T04:39:13.611 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.611 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.611 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:13.611 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582355+0000", 2024-09-18T04:39:13.611 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1485000000000001e-05 2024-09-18T04:39:13.611 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.611 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.611 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:13.611 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582371+0000", 2024-09-18T04:39:13.611 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6147999999999999e-05 2024-09-18T04:39:13.611 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.611 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.612 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:13.612 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582467+0000", 2024-09-18T04:39:13.612 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6059999999999998e-05 2024-09-18T04:39:13.612 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.612 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.612 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:13.612 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582505+0000", 2024-09-18T04:39:13.612 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7679000000000003e-05 2024-09-18T04:39:13.612 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.612 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.612 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:13.612 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593743+0000", 2024-09-18T04:39:13.612 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011238356 2024-09-18T04:39:13.612 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.613 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.613 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:13.613 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593816+0000", 2024-09-18T04:39:13.613 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2935000000000006e-05 2024-09-18T04:39:13.613 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:13.613 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:13.613 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:13.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.613 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.613 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:53 2.4 e51/17 2:39f7f174:ns::rbd_data.133e813ab5bc.000000000000000b:head v 51'7, mlcod=51'4)", 2024-09-18T04:39:13.613 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:39:13.613 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 30.971434240000001, 2024-09-18T04:39:13.613 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0079914159999999995, 2024-09-18T04:39:13.614 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:13.614 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:13.614 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:13.614 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.614 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:13.614 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:39:13.614 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:13.614 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.614 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.614 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:13.614 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:39:13.614 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:13.614 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.614 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.615 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:13.615 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594667+0000", 2024-09-18T04:39:13.615 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:13.615 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.615 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.615 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:13.615 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596446+0000", 2024-09-18T04:39:13.615 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001779033 2024-09-18T04:39:13.615 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.615 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.615 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:13.615 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596472+0000", 2024-09-18T04:39:13.615 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6120000000000001e-05 2024-09-18T04:39:13.615 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.616 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.616 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:13.616 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596500+0000", 2024-09-18T04:39:13.616 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7518e-05 2024-09-18T04:39:13.616 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.616 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.616 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:13.616 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596564+0000", 2024-09-18T04:39:13.616 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3880999999999994e-05 2024-09-18T04:39:13.616 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.616 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.616 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:13.616 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596596+0000", 2024-09-18T04:39:13.617 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2638999999999997e-05 2024-09-18T04:39:13.617 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.617 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.617 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:13.617 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602605+0000", 2024-09-18T04:39:13.617 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0060080699999999999 2024-09-18T04:39:13.617 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.617 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.617 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:13.617 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602660+0000", 2024-09-18T04:39:13.617 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5396000000000003e-05 2024-09-18T04:39:13.617 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:13.617 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:13.617 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:13.618 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.618 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.618 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:60 2.5 e51/28 2:a06f51e4:ns::rbd_data.133e813ab5bc.000000000000000f:head v 51'6, mlcod=51'4)", 2024-09-18T04:39:13.618 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:39:13.618 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 30.956617255000001, 2024-09-18T04:39:13.618 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0093954989999999999, 2024-09-18T04:39:13.618 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:13.618 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:13.618 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:13.618 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.618 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:13.618 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:39:13.618 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:13.618 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.619 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.619 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:13.619 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:39:13.619 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:13.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.619 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.619 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:13.619 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609483+0000", 2024-09-18T04:39:13.619 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:13.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.620 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.620 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:13.620 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611767+0000", 2024-09-18T04:39:13.620 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002283724 2024-09-18T04:39:13.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.620 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.620 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:13.620 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611791+0000", 2024-09-18T04:39:13.620 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3547e-05 2024-09-18T04:39:13.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.620 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.620 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:13.620 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611800+0000", 2024-09-18T04:39:13.621 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4979999999999995e-06 2024-09-18T04:39:13.621 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.621 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.621 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:13.621 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611864+0000", 2024-09-18T04:39:13.621 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3974999999999996e-05 2024-09-18T04:39:13.621 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.621 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.621 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:13.621 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611887+0000", 2024-09-18T04:39:13.621 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2633000000000002e-05 2024-09-18T04:39:13.621 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.621 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.622 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:13.622 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.618777+0000", 2024-09-18T04:39:13.622 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0068897979999999999 2024-09-18T04:39:13.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.622 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.622 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:13.622 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.618881+0000", 2024-09-18T04:39:13.622 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010437699999999999 2024-09-18T04:39:13.622 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:13.622 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:13.622 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:13.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.624 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.624 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:663 2.4 e57/17 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 57'13, mlcod=57'11)", 2024-09-18T04:39:13.624 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.355300+0000", 2024-09-18T04:39:13.624 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.210802319999999, 2024-09-18T04:39:13.624 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0049192760000000002, 2024-09-18T04:39:13.625 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:13.625 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:13.625 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:13.625 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.625 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:13.625 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355300+0000", 2024-09-18T04:39:13.625 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:13.625 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.625 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.625 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:13.625 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355300+0000", 2024-09-18T04:39:13.625 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:13.625 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.626 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.626 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:13.626 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355299+0000", 2024-09-18T04:39:13.626 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:13.626 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.626 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.626 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:13.626 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355876+0000", 2024-09-18T04:39:13.626 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00057730900000000003 2024-09-18T04:39:13.626 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.626 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.626 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:13.626 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355897+0000", 2024-09-18T04:39:13.626 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0477000000000001e-05 2024-09-18T04:39:13.627 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.627 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.627 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:13.627 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355905+0000", 2024-09-18T04:39:13.627 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5739999999999996e-06 2024-09-18T04:39:13.627 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.627 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.627 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:13.627 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355989+0000", 2024-09-18T04:39:13.627 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4002e-05 2024-09-18T04:39:13.627 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.627 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.627 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:13.627 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.356041+0000", 2024-09-18T04:39:13.628 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1347000000000002e-05 2024-09-18T04:39:13.628 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.628 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.628 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:13.628 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.360172+0000", 2024-09-18T04:39:13.628 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0041315090000000002 2024-09-18T04:39:13.628 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.628 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.628 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:13.628 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.360220+0000", 2024-09-18T04:39:13.628 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7584999999999999e-05 2024-09-18T04:39:13.628 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:13.628 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:13.628 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:13.629 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.629 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.629 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:679 2.5 e57/28 2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head v 57'20, mlcod=57'18)", 2024-09-18T04:39:13.629 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.445316+0000", 2024-09-18T04:39:13.629 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.120786843000001, 2024-09-18T04:39:13.629 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0047016760000000001, 2024-09-18T04:39:13.629 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:13.629 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:13.629 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:13.629 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.629 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:13.629 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445316+0000", 2024-09-18T04:39:13.629 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:13.629 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.630 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.630 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:13.630 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445316+0000", 2024-09-18T04:39:13.630 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:13.630 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.630 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.630 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:13.630 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445314+0000", 2024-09-18T04:39:13.630 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:13.630 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.630 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.630 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:13.630 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445947+0000", 2024-09-18T04:39:13.630 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00063298900000000003 2024-09-18T04:39:13.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.631 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.631 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:13.631 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445968+0000", 2024-09-18T04:39:13.631 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.054e-05 2024-09-18T04:39:13.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.631 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.631 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:13.631 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445981+0000", 2024-09-18T04:39:13.631 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2697000000000001e-05 2024-09-18T04:39:13.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.631 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.631 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:13.631 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.446102+0000", 2024-09-18T04:39:13.632 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012189300000000001 2024-09-18T04:39:13.632 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.632 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.632 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:13.632 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.446151+0000", 2024-09-18T04:39:13.632 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8279000000000003e-05 2024-09-18T04:39:13.632 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.632 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.632 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:13.632 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.449953+0000", 2024-09-18T04:39:13.632 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003801979 2024-09-18T04:39:13.632 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.632 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.633 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:13.633 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.450018+0000", 2024-09-18T04:39:13.633 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4873e-05 2024-09-18T04:39:13.633 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:13.633 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:13.633 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:13.633 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.633 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.633 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:699 2.4 e57/17 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 57'17, mlcod=57'15)", 2024-09-18T04:39:13.633 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.566073+0000", 2024-09-18T04:39:13.633 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.00002933, 2024-09-18T04:39:13.633 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055398380000000001, 2024-09-18T04:39:13.633 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:13.633 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:13.634 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:13.634 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.634 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:13.634 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566073+0000", 2024-09-18T04:39:13.634 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:13.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.634 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.634 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:13.634 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566073+0000", 2024-09-18T04:39:13.634 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:13.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.634 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.634 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:13.634 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566071+0000", 2024-09-18T04:39:13.635 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:13.635 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.635 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.635 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:13.635 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566637+0000", 2024-09-18T04:39:13.635 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00056516699999999999 2024-09-18T04:39:13.635 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.635 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.635 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:13.635 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566668+0000", 2024-09-18T04:39:13.635 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1544999999999997e-05 2024-09-18T04:39:13.635 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.635 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.635 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:13.636 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566685+0000", 2024-09-18T04:39:13.636 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6857000000000001e-05 2024-09-18T04:39:13.636 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.636 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.636 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:13.636 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566816+0000", 2024-09-18T04:39:13.636 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013092799999999999 2024-09-18T04:39:13.636 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.636 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.636 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:13.636 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566875+0000", 2024-09-18T04:39:13.636 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9583999999999997e-05 2024-09-18T04:39:13.636 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.637 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.637 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:13.637 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.571567+0000", 2024-09-18T04:39:13.637 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.004691682 2024-09-18T04:39:13.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.637 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.637 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:13.637 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.571613+0000", 2024-09-18T04:39:13.637 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6125000000000003e-05 2024-09-18T04:39:13.637 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:13.637 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:13.637 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:13.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.637 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.638 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:98 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'27, mlcod=60'25)", 2024-09-18T04:39:13.638 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.847978+0000", 2024-09-18T04:39:13.638 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 12.718124770999999, 2024-09-18T04:39:13.638 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0044617709999999998, 2024-09-18T04:39:13.638 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:13.638 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:13.638 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:13.638 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.638 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:13.638 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.847978+0000", 2024-09-18T04:39:13.638 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:13.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.638 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.638 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:13.639 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.847978+0000", 2024-09-18T04:39:13.639 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:13.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.639 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.639 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:13.639 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.847977+0000", 2024-09-18T04:39:13.639 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:13.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.639 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.639 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:13.639 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849038+0000", 2024-09-18T04:39:13.639 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0010614820000000001 2024-09-18T04:39:13.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.640 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.640 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:13.640 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849065+0000", 2024-09-18T04:39:13.640 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7127999999999998e-05 2024-09-18T04:39:13.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.640 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.640 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:13.640 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849082+0000", 2024-09-18T04:39:13.640 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7170999999999999e-05 2024-09-18T04:39:13.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.640 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.640 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:13.640 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849188+0000", 2024-09-18T04:39:13.640 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010605800000000001 2024-09-18T04:39:13.641 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.641 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.641 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:13.641 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849233+0000", 2024-09-18T04:39:13.641 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4579000000000001e-05 2024-09-18T04:39:13.641 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.641 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.641 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:13.641 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.852370+0000", 2024-09-18T04:39:13.641 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00313751 2024-09-18T04:39:13.641 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.641 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.641 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:13.641 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.852440+0000", 2024-09-18T04:39:13.642 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9238000000000004e-05 2024-09-18T04:39:13.642 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:13.642 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:13.642 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:13.642 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.642 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.642 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:100 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'29, mlcod=64'27)", 2024-09-18T04:39:13.642 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.858766+0000", 2024-09-18T04:39:13.642 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 12.707336442000001, 2024-09-18T04:39:13.642 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0061964840000000004, 2024-09-18T04:39:13.642 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:13.642 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:13.642 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:13.643 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.643 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:13.643 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.858766+0000", 2024-09-18T04:39:13.643 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:13.643 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.643 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.643 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:13.643 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.858766+0000", 2024-09-18T04:39:13.643 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:13.643 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.643 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.643 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:13.643 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.858765+0000", 2024-09-18T04:39:13.643 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:13.644 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.644 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.644 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:13.644 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861612+0000", 2024-09-18T04:39:13.644 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028473169999999998 2024-09-18T04:39:13.644 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.644 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.644 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:13.644 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861643+0000", 2024-09-18T04:39:13.644 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1356000000000003e-05 2024-09-18T04:39:13.644 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.644 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.644 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:13.644 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861656+0000", 2024-09-18T04:39:13.645 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2899e-05 2024-09-18T04:39:13.645 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.645 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.645 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:13.645 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861773+0000", 2024-09-18T04:39:13.645 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011637999999999999 2024-09-18T04:39:13.645 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.645 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.645 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:13.645 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861808+0000", 2024-09-18T04:39:13.645 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5002999999999997e-05 2024-09-18T04:39:13.645 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.645 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.645 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:13.646 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.864876+0000", 2024-09-18T04:39:13.646 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030680719999999998 2024-09-18T04:39:13.646 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.646 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.646 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:13.646 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.864963+0000", 2024-09-18T04:39:13.646 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7114000000000007e-05 2024-09-18T04:39:13.646 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:13.646 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:13.646 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:13.646 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.646 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.646 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:102 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'31, mlcod=64'29)", 2024-09-18T04:39:13.646 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.872098+0000", 2024-09-18T04:39:13.647 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 12.694005185, 2024-09-18T04:39:13.647 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0064676450000000002, 2024-09-18T04:39:13.647 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:13.647 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:13.647 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:13.647 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.647 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:13.647 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.872098+0000", 2024-09-18T04:39:13.647 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:13.647 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.647 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.647 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:13.647 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.872098+0000", 2024-09-18T04:39:13.647 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:13.648 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.648 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.648 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:13.648 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.872096+0000", 2024-09-18T04:39:13.648 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:13.648 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.648 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.648 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:13.648 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875176+0000", 2024-09-18T04:39:13.648 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030799809999999999 2024-09-18T04:39:13.648 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.648 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.648 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:13.649 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875204+0000", 2024-09-18T04:39:13.649 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8025999999999999e-05 2024-09-18T04:39:13.649 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.649 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.649 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:13.649 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875216+0000", 2024-09-18T04:39:13.649 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2072999999999999e-05 2024-09-18T04:39:13.649 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.649 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.649 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:13.650 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875307+0000", 2024-09-18T04:39:13.650 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0373999999999999e-05 2024-09-18T04:39:13.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.650 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.650 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:13.650 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875349+0000", 2024-09-18T04:39:13.650 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2299000000000003e-05 2024-09-18T04:39:13.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.650 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.650 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:13.650 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.878496+0000", 2024-09-18T04:39:13.650 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0031468960000000002 2024-09-18T04:39:13.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.651 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.651 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:13.651 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.878565+0000", 2024-09-18T04:39:13.651 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9465e-05 2024-09-18T04:39:13.651 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:13.651 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:13.651 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:13.651 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.651 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.651 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:104 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'33, mlcod=64'31)", 2024-09-18T04:39:13.651 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.885100+0000", 2024-09-18T04:39:13.651 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 12.68100265, 2024-09-18T04:39:13.651 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0051888159999999997, 2024-09-18T04:39:13.651 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:13.652 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:13.652 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:13.652 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.652 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:13.652 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.885100+0000", 2024-09-18T04:39:13.652 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:13.652 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.652 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.652 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:13.652 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.885100+0000", 2024-09-18T04:39:13.652 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:13.652 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.652 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.652 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:13.653 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.885098+0000", 2024-09-18T04:39:13.653 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:13.653 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.653 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.653 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:13.653 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886809+0000", 2024-09-18T04:39:13.653 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017110459999999999 2024-09-18T04:39:13.653 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.653 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.653 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:13.653 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886842+0000", 2024-09-18T04:39:13.654 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2638999999999997e-05 2024-09-18T04:39:13.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.654 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.654 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:13.654 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886855+0000", 2024-09-18T04:39:13.654 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3380000000000001e-05 2024-09-18T04:39:13.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.654 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.654 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:13.654 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886939+0000", 2024-09-18T04:39:13.654 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3733999999999995e-05 2024-09-18T04:39:13.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.655 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.655 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:13.655 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886981+0000", 2024-09-18T04:39:13.655 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1956000000000003e-05 2024-09-18T04:39:13.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.655 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.655 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:13.655 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.890201+0000", 2024-09-18T04:39:13.655 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003219927 2024-09-18T04:39:13.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.655 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.655 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:13.655 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.890289+0000", 2024-09-18T04:39:13.655 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7997999999999993e-05 2024-09-18T04:39:13.656 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:13.656 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:13.656 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:13.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.656 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.656 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:145 2.4 e64/62 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 64'37, mlcod=64'35)", 2024-09-18T04:39:13.656 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:01.099398+0000", 2024-09-18T04:39:13.656 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 12.466705162, 2024-09-18T04:39:13.656 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0050078459999999998, 2024-09-18T04:39:13.656 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:13.656 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:13.656 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:13.656 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.656 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:13.657 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.099398+0000", 2024-09-18T04:39:13.657 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:13.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.657 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.657 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:13.657 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.099398+0000", 2024-09-18T04:39:13.657 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:13.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.657 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.657 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:13.657 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.099396+0000", 2024-09-18T04:39:13.657 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:13.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.657 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.658 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:13.658 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100199+0000", 2024-09-18T04:39:13.658 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00080338000000000002 2024-09-18T04:39:13.658 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.658 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.658 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:13.658 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100230+0000", 2024-09-18T04:39:13.658 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0716000000000002e-05 2024-09-18T04:39:13.658 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.658 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.658 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:13.658 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100243+0000", 2024-09-18T04:39:13.658 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3359e-05 2024-09-18T04:39:13.658 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.659 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.659 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:13.659 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100337+0000", 2024-09-18T04:39:13.659 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4387000000000003e-05 2024-09-18T04:39:13.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.659 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.659 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:13.659 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100396+0000", 2024-09-18T04:39:13.659 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8164000000000001e-05 2024-09-18T04:39:13.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.659 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.659 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:13.659 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.104309+0000", 2024-09-18T04:39:13.659 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0039137800000000004 2024-09-18T04:39:13.660 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.660 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.660 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:13.660 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.104405+0000", 2024-09-18T04:39:13.660 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6039999999999995e-05 2024-09-18T04:39:13.660 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:13.660 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:13.660 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:13.660 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.660 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.660 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:147 2.4 e64/62 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 64'39, mlcod=64'37)", 2024-09-18T04:39:13.660 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:01.110658+0000", 2024-09-18T04:39:13.660 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 12.455445073, 2024-09-18T04:39:13.660 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0046316910000000003, 2024-09-18T04:39:13.661 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:13.661 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:13.661 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:13.661 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.661 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:13.661 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.110658+0000", 2024-09-18T04:39:13.661 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:13.661 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.661 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.661 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:13.661 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.110658+0000", 2024-09-18T04:39:13.661 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:13.661 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.661 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.662 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:13.662 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.110656+0000", 2024-09-18T04:39:13.662 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:13.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.662 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.662 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:13.662 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111444+0000", 2024-09-18T04:39:13.662 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00078762400000000001 2024-09-18T04:39:13.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.662 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.662 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:13.662 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111474+0000", 2024-09-18T04:39:13.662 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0085000000000001e-05 2024-09-18T04:39:13.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.662 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.663 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:13.663 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111486+0000", 2024-09-18T04:39:13.663 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2645e-05 2024-09-18T04:39:13.663 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.663 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.663 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:13.663 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111581+0000", 2024-09-18T04:39:13.663 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4432000000000007e-05 2024-09-18T04:39:13.663 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.663 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.663 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:13.663 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111631+0000", 2024-09-18T04:39:13.663 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9901000000000002e-05 2024-09-18T04:39:13.664 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.664 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.664 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:13.664 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.115209+0000", 2024-09-18T04:39:13.664 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003577916 2024-09-18T04:39:13.664 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:13.664 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:13.664 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:13.664 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.115289+0000", 2024-09-18T04:39:13.664 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0704999999999994e-05 2024-09-18T04:39:13.664 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:13.664 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:13.664 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:13.664 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:13.664 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:13.665 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:13.666 INFO:tasks.workunit.client.0.smithi115.stderr:+ DEV=/dev/nbd0 2024-09-18T04:39:13.667 INFO:tasks.workunit.client.0.smithi115.stderr:+ get_pid new_default_pool17342 2024-09-18T04:39:13.667 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pool=new_default_pool17342 2024-09-18T04:39:13.667 INFO:tasks.workunit.client.0.smithi115.stderr:+ local ns= 2024-09-18T04:39:13.667 INFO:tasks.workunit.client.0.smithi115.stderr:++ rbd device --device-type nbd --format xml list 2024-09-18T04:39:13.667 INFO:tasks.workunit.client.0.smithi115.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''new_default_pool17342'\''][namespace='\'''\''][image='\''testrbdnbd17342'\''][device='\''/dev/nbd0'\'']/id' 2024-09-18T04:39:13.667 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:39:13.668 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:13.661+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:13.681 INFO:tasks.workunit.client.0.smithi115.stderr:+ PID=19331 2024-09-18T04:39:13.681 INFO:tasks.workunit.client.0.smithi115.stderr:+ test -n 19331 2024-09-18T04:39:13.681 INFO:tasks.workunit.client.0.smithi115.stderr:+ ps -p 19331 -C rbd-nbd 2024-09-18T04:39:13.691 INFO:tasks.workunit.client.0.smithi115.stdout: PID TTY TIME CMD 2024-09-18T04:39:13.691 INFO:tasks.workunit.client.0.smithi115.stdout: 19331 ? 00:00:00 rbd-nbd 2024-09-18T04:39:13.692 INFO:tasks.workunit.client.0.smithi115.stderr:+ unmap_device testrbdnbd17342 19331 2024-09-18T04:39:13.692 INFO:tasks.workunit.client.0.smithi115.stderr:+ local args=testrbdnbd17342 2024-09-18T04:39:13.692 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pid=19331 2024-09-18T04:39:13.692 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo rbd device --device-type nbd unmap testrbdnbd17342 2024-09-18T04:39:13.692 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:39:13.692 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:39:13.694 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:39:13.694 INFO:tasks.workunit.client.0.smithi115.stderr:++ which rbd 2024-09-18T04:39:13.696 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/bin/rbd 2024-09-18T04:39:13.696 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:39:13.696 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap testrbdnbd17342 2024-09-18T04:39:13.767 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:13.761+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:13.788 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:13.788 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:13.788 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:39:13.788 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:13.799 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:39:13.804 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd device --device-type nbd list 2024-09-18T04:39:13.804 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false grep '^19331\b' 2024-09-18T04:39:13.804 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep '^19331\b' 2024-09-18T04:39:13.868 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:13.861+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:13.870 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:39:13.871 INFO:tasks.workunit.client.0.smithi115.stderr:+ ps -C rbd-nbd 2024-09-18T04:39:13.871 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false grep '^ *19331\b' 2024-09-18T04:39:13.872 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep '^ *19331\b' 2024-09-18T04:39:13.881 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:39:13.882 INFO:tasks.workunit.client.0.smithi115.stderr:+ sleep 0.5 2024-09-18T04:39:13.904 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:13.904 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:13.904 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:39:13.905 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:39:13.905 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:13.914 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:39:13.969 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:13.961+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:39:14.070 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:14.065+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:14.074 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:14.074 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:39:14.074 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:39:14.074 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:39:14.074 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.074 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:33 2.1 e51/24 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head v 51'4, mlcod=51'2)", 2024-09-18T04:39:14.074 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:39:14.075 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 31.790251346000002, 2024-09-18T04:39:14.075 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018702106, 2024-09-18T04:39:14.075 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:14.075 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:14.075 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:14.075 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.075 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:14.075 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:39:14.075 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.075 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.075 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.075 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:14.075 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:39:14.075 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.076 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.076 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.076 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:14.076 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279197+0000", 2024-09-18T04:39:14.076 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:14.076 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.076 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.076 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:14.076 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282682+0000", 2024-09-18T04:39:14.076 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0034848689999999998 2024-09-18T04:39:14.076 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.076 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.076 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:14.076 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282709+0000", 2024-09-18T04:39:14.077 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7599000000000001e-05 2024-09-18T04:39:14.077 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.077 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.077 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:14.077 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282720+0000", 2024-09-18T04:39:14.077 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.092e-05 2024-09-18T04:39:14.077 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.077 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.077 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:14.077 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282791+0000", 2024-09-18T04:39:14.077 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1268999999999998e-05 2024-09-18T04:39:14.077 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.077 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.078 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:14.078 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282843+0000", 2024-09-18T04:39:14.078 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1771999999999998e-05 2024-09-18T04:39:14.078 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.078 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.078 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:14.078 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.297830+0000", 2024-09-18T04:39:14.078 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014987228999999999 2024-09-18T04:39:14.078 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.078 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.078 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:14.078 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.297901+0000", 2024-09-18T04:39:14.078 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0209999999999994e-05 2024-09-18T04:39:14.078 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.079 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:14.079 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.079 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.079 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.079 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:38 2.0 e51/17 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head v 51'10, mlcod=51'8)", 2024-09-18T04:39:14.079 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:39:14.079 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 31.788832916, 2024-09-18T04:39:14.079 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.023984472, 2024-09-18T04:39:14.079 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:14.079 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:14.080 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:14.080 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.080 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:14.080 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:39:14.080 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.080 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.080 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.080 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:14.080 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:39:14.080 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.080 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.080 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.081 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:14.081 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280614+0000", 2024-09-18T04:39:14.081 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:14.081 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.081 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.081 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:14.081 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293094+0000", 2024-09-18T04:39:14.081 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012479571 2024-09-18T04:39:14.081 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.081 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.081 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:14.081 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293111+0000", 2024-09-18T04:39:14.081 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7354999999999999e-05 2024-09-18T04:39:14.081 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.082 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.082 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:14.082 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293121+0000", 2024-09-18T04:39:14.082 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6979999999999993e-06 2024-09-18T04:39:14.082 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.082 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.082 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:14.082 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293180+0000", 2024-09-18T04:39:14.082 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9333000000000003e-05 2024-09-18T04:39:14.082 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.082 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.082 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:14.082 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293203+0000", 2024-09-18T04:39:14.082 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2633000000000002e-05 2024-09-18T04:39:14.083 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.083 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.083 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:14.083 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304534+0000", 2024-09-18T04:39:14.083 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011330761 2024-09-18T04:39:14.083 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.083 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.083 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:14.083 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304601+0000", 2024-09-18T04:39:14.083 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7657000000000006e-05 2024-09-18T04:39:14.083 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.083 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:14.083 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.084 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.084 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.084 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:46 2.6 2:7ba25245:::rbd_data.125fc09df6d1.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:14.084 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:39:14.084 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 31.761290116000001, 2024-09-18T04:39:14.084 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.035604595000000003, 2024-09-18T04:39:14.084 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:14.084 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:14.084 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:14.084 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:14.084 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:14.084 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 46 2024-09-18T04:39:14.084 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.084 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:14.085 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.085 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:14.085 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:39:14.085 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.085 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.085 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.085 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:14.085 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:39:14.085 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.085 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.085 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.085 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:14.085 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308158+0000", 2024-09-18T04:39:14.086 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:14.086 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.086 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.086 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:14.086 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313191+0000", 2024-09-18T04:39:14.086 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0050323929999999996 2024-09-18T04:39:14.086 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.086 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.086 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:14.086 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313210+0000", 2024-09-18T04:39:14.086 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9673e-05 2024-09-18T04:39:14.086 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.086 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.086 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:14.087 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313221+0000", 2024-09-18T04:39:14.087 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0444e-05 2024-09-18T04:39:14.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.087 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.087 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:14.087 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313294+0000", 2024-09-18T04:39:14.087 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2650000000000004e-05 2024-09-18T04:39:14.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.087 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.087 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:14.087 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313493+0000", 2024-09-18T04:39:14.087 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019968300000000001 2024-09-18T04:39:14.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.087 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.088 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:14.088 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313666+0000", 2024-09-18T04:39:14.088 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00017248599999999999 2024-09-18T04:39:14.088 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.088 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.088 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:14.088 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.323175+0000", 2024-09-18T04:39:14.088 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0095088819999999998 2024-09-18T04:39:14.088 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.088 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.088 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:14.088 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343651+0000", 2024-09-18T04:39:14.088 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020475892999999998 2024-09-18T04:39:14.088 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.089 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.089 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:14.089 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343715+0000", 2024-09-18T04:39:14.089 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4443999999999995e-05 2024-09-18T04:39:14.089 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.089 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.089 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:14.089 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343764+0000", 2024-09-18T04:39:14.089 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9280000000000003e-05 2024-09-18T04:39:14.089 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.089 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:14.089 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.089 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.089 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.090 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:47 2.4 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:14.090 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:39:14.090 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 31.756208079, 2024-09-18T04:39:14.090 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.032570693999999997, 2024-09-18T04:39:14.090 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:14.090 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:14.090 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:14.090 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:14.090 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:14.090 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 47 2024-09-18T04:39:14.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.090 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:14.090 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.091 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:14.091 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:39:14.091 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.091 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.091 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:14.091 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:39:14.091 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.091 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.091 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:14.091 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313240+0000", 2024-09-18T04:39:14.091 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:14.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.092 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.092 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:14.092 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325825+0000", 2024-09-18T04:39:14.092 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012584425999999999 2024-09-18T04:39:14.092 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.092 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.092 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:14.092 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325848+0000", 2024-09-18T04:39:14.092 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3025999999999999e-05 2024-09-18T04:39:14.092 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.092 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.092 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:14.092 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325863+0000", 2024-09-18T04:39:14.092 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5152e-05 2024-09-18T04:39:14.093 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.093 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.093 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:14.093 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325938+0000", 2024-09-18T04:39:14.093 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4982999999999997e-05 2024-09-18T04:39:14.093 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.093 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.093 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:14.093 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326090+0000", 2024-09-18T04:39:14.093 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015207900000000001 2024-09-18T04:39:14.093 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.093 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.093 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:14.094 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326242+0000", 2024-09-18T04:39:14.094 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015163500000000001 2024-09-18T04:39:14.094 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.094 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.094 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:14.094 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343015+0000", 2024-09-18T04:39:14.094 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016773414 2024-09-18T04:39:14.094 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.094 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.094 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:14.094 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345664+0000", 2024-09-18T04:39:14.094 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026487870000000001 2024-09-18T04:39:14.094 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.095 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.095 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:14.095 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345739+0000", 2024-09-18T04:39:14.095 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5018999999999994e-05 2024-09-18T04:39:14.095 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.095 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.095 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:14.095 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345812+0000", 2024-09-18T04:39:14.095 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3547000000000006e-05 2024-09-18T04:39:14.095 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.095 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:14.095 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.095 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.095 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.096 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:48 2.6 2:61db7a3a:::rbd_data.125fc09df6d1.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:14.096 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:39:14.096 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 31.743551406999998, 2024-09-18T04:39:14.096 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.025064388, 2024-09-18T04:39:14.096 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:14.096 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:14.096 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:14.096 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:14.096 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:14.096 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 48 2024-09-18T04:39:14.096 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.096 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:14.096 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.097 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:14.097 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:39:14.097 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.097 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.097 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.097 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:14.097 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:39:14.097 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.097 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.097 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.097 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:14.097 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325897+0000", 2024-09-18T04:39:14.097 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:14.097 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.098 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.098 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:14.098 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333697+0000", 2024-09-18T04:39:14.098 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0078002649999999998 2024-09-18T04:39:14.098 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.098 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.098 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:14.098 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333716+0000", 2024-09-18T04:39:14.098 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8910999999999999e-05 2024-09-18T04:39:14.098 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.098 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.098 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:14.098 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333728+0000", 2024-09-18T04:39:14.098 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1936000000000001e-05 2024-09-18T04:39:14.099 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.099 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.099 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:14.099 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333792+0000", 2024-09-18T04:39:14.099 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4010999999999994e-05 2024-09-18T04:39:14.099 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.099 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.099 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:14.099 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.334082+0000", 2024-09-18T04:39:14.099 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00028955799999999998 2024-09-18T04:39:14.099 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.099 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.099 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:14.099 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.334326+0000", 2024-09-18T04:39:14.100 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00024399099999999999 2024-09-18T04:39:14.100 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.100 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.100 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:14.100 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350745+0000", 2024-09-18T04:39:14.100 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016419474999999999 2024-09-18T04:39:14.100 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.100 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.100 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:14.100 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350808+0000", 2024-09-18T04:39:14.100 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3244999999999995e-05 2024-09-18T04:39:14.100 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.100 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.100 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:14.101 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350892+0000", 2024-09-18T04:39:14.101 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4074999999999997e-05 2024-09-18T04:39:14.101 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.101 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.101 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:14.101 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350963+0000", 2024-09-18T04:39:14.101 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0412000000000003e-05 2024-09-18T04:39:14.101 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.101 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:14.101 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.101 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.101 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.101 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:51 2.6 2:775d7b11:::rbd_data.125fc09df6d1.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:14.101 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:39:14.101 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 31.735699271000001, 2024-09-18T04:39:14.102 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.023824221, 2024-09-18T04:39:14.102 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:14.102 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:14.102 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:14.102 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:14.102 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:14.102 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 51 2024-09-18T04:39:14.102 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.102 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:14.102 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.102 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:14.102 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:39:14.102 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.102 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.103 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.103 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:14.103 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:39:14.103 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.103 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.103 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.103 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:14.103 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333749+0000", 2024-09-18T04:39:14.103 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:14.103 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.103 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.103 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:14.103 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340899+0000", 2024-09-18T04:39:14.103 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0071500340000000004 2024-09-18T04:39:14.104 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.104 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.104 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:14.104 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340929+0000", 2024-09-18T04:39:14.104 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0093000000000001e-05 2024-09-18T04:39:14.104 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.104 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.104 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:14.104 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340943+0000", 2024-09-18T04:39:14.104 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4019000000000001e-05 2024-09-18T04:39:14.104 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.104 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.104 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:14.105 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343834+0000", 2024-09-18T04:39:14.105 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028918120000000001 2024-09-18T04:39:14.105 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.105 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.105 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:14.105 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344020+0000", 2024-09-18T04:39:14.105 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000185892 2024-09-18T04:39:14.105 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.105 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.105 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:14.105 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344190+0000", 2024-09-18T04:39:14.105 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016946800000000001 2024-09-18T04:39:14.105 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.106 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.106 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:14.106 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.351291+0000", 2024-09-18T04:39:14.106 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.007100805 2024-09-18T04:39:14.106 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.106 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.106 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:14.106 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357470+0000", 2024-09-18T04:39:14.106 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0061797570000000001 2024-09-18T04:39:14.106 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.106 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.106 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:14.106 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357533+0000", 2024-09-18T04:39:14.106 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2906999999999999e-05 2024-09-18T04:39:14.107 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.107 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.107 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:14.107 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357575+0000", 2024-09-18T04:39:14.107 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1403999999999999e-05 2024-09-18T04:39:14.107 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.107 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:14.107 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.107 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.107 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.107 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:56 2.4 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:14.107 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:39:14.107 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 31.719654177999999, 2024-09-18T04:39:14.107 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.034295870999999999, 2024-09-18T04:39:14.107 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:14.108 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:14.108 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:14.108 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:14.108 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:14.108 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 56 2024-09-18T04:39:14.108 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.108 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:14.108 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.108 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:14.108 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:39:14.108 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.108 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.108 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.109 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:14.109 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:39:14.109 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.109 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.109 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.109 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:14.109 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349794+0000", 2024-09-18T04:39:14.109 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:14.109 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.109 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.109 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:14.109 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366632+0000", 2024-09-18T04:39:14.109 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01683755 2024-09-18T04:39:14.109 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.110 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.110 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:14.110 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366669+0000", 2024-09-18T04:39:14.110 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7351000000000002e-05 2024-09-18T04:39:14.110 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.110 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.110 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:14.110 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366686+0000", 2024-09-18T04:39:14.110 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6507999999999998e-05 2024-09-18T04:39:14.110 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.110 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.110 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:14.110 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366962+0000", 2024-09-18T04:39:14.110 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00027655900000000001 2024-09-18T04:39:14.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.111 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.111 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:14.111 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.367452+0000", 2024-09-18T04:39:14.111 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00049025299999999996 2024-09-18T04:39:14.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.111 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.111 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:14.111 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368113+0000", 2024-09-18T04:39:14.111 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00066093399999999996 2024-09-18T04:39:14.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.111 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.111 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:14.112 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383719+0000", 2024-09-18T04:39:14.112 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015606106999999999 2024-09-18T04:39:14.112 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.112 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.112 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:14.112 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384035+0000", 2024-09-18T04:39:14.112 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00031590399999999999 2024-09-18T04:39:14.112 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.112 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.112 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:14.112 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384067+0000", 2024-09-18T04:39:14.112 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.129e-05 2024-09-18T04:39:14.112 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.112 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.113 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:14.113 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384091+0000", 2024-09-18T04:39:14.113 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4788000000000001e-05 2024-09-18T04:39:14.113 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.113 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:14.113 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.113 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.113 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.113 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:57 2.0 e51/17 2:0cdd070e:::rbd_data.125fc09df6d1.000000000000000d:head v 51'21, mlcod=51'19)", 2024-09-18T04:39:14.113 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:39:14.113 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 31.711826690999999, 2024-09-18T04:39:14.113 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.025216311000000002, 2024-09-18T04:39:14.113 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:14.113 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:14.114 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:14.114 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.114 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:14.114 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:39:14.114 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.114 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.114 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.114 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:14.114 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:39:14.114 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.114 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.114 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.114 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:14.115 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:39:14.115 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:14.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.115 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.115 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:14.115 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363844+0000", 2024-09-18T04:39:14.115 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0062218450000000002 2024-09-18T04:39:14.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.115 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.115 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:14.115 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363879+0000", 2024-09-18T04:39:14.115 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4382999999999999e-05 2024-09-18T04:39:14.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.115 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.116 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:14.116 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363893+0000", 2024-09-18T04:39:14.116 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4348999999999999e-05 2024-09-18T04:39:14.116 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.116 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.116 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:14.116 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363977+0000", 2024-09-18T04:39:14.116 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3499999999999997e-05 2024-09-18T04:39:14.116 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.116 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.116 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:14.116 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364009+0000", 2024-09-18T04:39:14.116 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2292000000000002e-05 2024-09-18T04:39:14.117 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.117 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.117 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:14.117 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.382781+0000", 2024-09-18T04:39:14.117 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018771626 2024-09-18T04:39:14.117 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.117 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.117 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:14.117 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.382839+0000", 2024-09-18T04:39:14.117 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8758000000000002e-05 2024-09-18T04:39:14.117 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.117 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:14.117 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.117 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.117 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.118 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:59 2.0 e51/17 2:19c52cde:::rbd_data.125fc09df6d1.000000000000000f:head v 51'22, mlcod=51'19)", 2024-09-18T04:39:14.118 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:39:14.118 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 31.705166765000001, 2024-09-18T04:39:14.118 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.019497112, 2024-09-18T04:39:14.118 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:14.118 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:14.118 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:14.118 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.118 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:14.118 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:39:14.118 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.118 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.118 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.119 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:14.119 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:39:14.119 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.119 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.119 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:14.119 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364282+0000", 2024-09-18T04:39:14.119 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:14.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.119 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.119 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:14.119 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371823+0000", 2024-09-18T04:39:14.119 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0075411829999999999 2024-09-18T04:39:14.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.120 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.120 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:14.120 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371837+0000", 2024-09-18T04:39:14.120 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4336e-05 2024-09-18T04:39:14.120 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.120 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.120 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:14.120 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371846+0000", 2024-09-18T04:39:14.120 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9770000000000006e-06 2024-09-18T04:39:14.120 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.120 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.120 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:14.121 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.373566+0000", 2024-09-18T04:39:14.121 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017192539999999999 2024-09-18T04:39:14.121 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.121 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.121 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:14.121 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.373595+0000", 2024-09-18T04:39:14.121 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8975999999999998e-05 2024-09-18T04:39:14.121 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.121 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.121 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:14.122 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383740+0000", 2024-09-18T04:39:14.122 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01014525 2024-09-18T04:39:14.122 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.122 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.122 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:14.122 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383780+0000", 2024-09-18T04:39:14.122 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0275999999999999e-05 2024-09-18T04:39:14.122 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.122 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:14.122 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.122 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.122 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.122 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:34 2.6 2:66289171:ns::rbd_data.133e813ab5bc.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:14.123 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.524015+0000", 2024-09-18T04:39:14.123 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 31.545435156, 2024-09-18T04:39:14.123 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027582111999999999, 2024-09-18T04:39:14.123 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:14.123 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:14.123 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:14.123 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:14.123 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:14.123 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 34 2024-09-18T04:39:14.123 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.124 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:14.124 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.124 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:14.124 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524015+0000", 2024-09-18T04:39:14.124 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.124 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.124 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.124 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:14.124 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524015+0000", 2024-09-18T04:39:14.124 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.124 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.124 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.124 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:14.125 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524012+0000", 2024-09-18T04:39:14.125 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:14.125 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.125 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.125 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:14.125 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534956+0000", 2024-09-18T04:39:14.125 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010943403000000001 2024-09-18T04:39:14.125 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.125 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.125 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:14.125 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534978+0000", 2024-09-18T04:39:14.126 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1965999999999999e-05 2024-09-18T04:39:14.126 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.126 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.126 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:14.126 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534990+0000", 2024-09-18T04:39:14.126 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2065999999999999e-05 2024-09-18T04:39:14.126 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.126 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.126 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:14.126 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535065+0000", 2024-09-18T04:39:14.126 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5523000000000002e-05 2024-09-18T04:39:14.126 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.127 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.127 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:14.127 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535208+0000", 2024-09-18T04:39:14.127 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014309500000000001 2024-09-18T04:39:14.127 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.127 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.127 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:14.127 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535343+0000", 2024-09-18T04:39:14.127 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013437400000000001 2024-09-18T04:39:14.127 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.127 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.127 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:14.127 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.546576+0000", 2024-09-18T04:39:14.128 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011233425999999999 2024-09-18T04:39:14.128 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.128 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.128 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:14.128 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551498+0000", 2024-09-18T04:39:14.128 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0049219349999999997 2024-09-18T04:39:14.128 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.128 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.128 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:14.128 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551554+0000", 2024-09-18T04:39:14.128 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6400999999999998e-05 2024-09-18T04:39:14.128 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.128 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.128 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:14.129 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551597+0000", 2024-09-18T04:39:14.129 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2283000000000002e-05 2024-09-18T04:39:14.129 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.129 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:14.129 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.129 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.129 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.129 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:33 2.1 e51/24 2:8b40f1c5:ns::rbd_data.133e813ab5bc.0000000000000000:head v 51'7, mlcod=51'5)", 2024-09-18T04:39:14.129 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.531767+0000", 2024-09-18T04:39:14.129 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 31.537683239, 2024-09-18T04:39:14.129 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015370295000000001, 2024-09-18T04:39:14.129 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:14.129 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:14.129 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:14.130 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.130 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:14.130 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531767+0000", 2024-09-18T04:39:14.130 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.130 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.130 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.130 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:14.130 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531767+0000", 2024-09-18T04:39:14.130 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.130 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.130 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.130 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:14.130 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531765+0000", 2024-09-18T04:39:14.130 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:14.131 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.131 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.131 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:14.131 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534258+0000", 2024-09-18T04:39:14.131 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002493561 2024-09-18T04:39:14.131 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.131 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.131 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:14.131 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534288+0000", 2024-09-18T04:39:14.131 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.976e-05 2024-09-18T04:39:14.131 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.131 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.131 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:14.131 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534300+0000", 2024-09-18T04:39:14.132 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1928e-05 2024-09-18T04:39:14.132 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.132 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.132 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:14.132 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534367+0000", 2024-09-18T04:39:14.132 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7045999999999995e-05 2024-09-18T04:39:14.132 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.132 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.132 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:14.132 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534394+0000", 2024-09-18T04:39:14.132 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6903999999999999e-05 2024-09-18T04:39:14.132 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.133 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.133 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:14.133 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.547073+0000", 2024-09-18T04:39:14.133 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012679365 2024-09-18T04:39:14.133 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.133 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.133 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:14.133 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.547137+0000", 2024-09-18T04:39:14.133 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3691999999999999e-05 2024-09-18T04:39:14.133 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.133 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:14.133 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.133 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.133 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.134 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:39 2.4 2:304364d1:ns::rbd_data.133e813ab5bc.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:14.134 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.535013+0000", 2024-09-18T04:39:14.134 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 31.534436304, 2024-09-18T04:39:14.134 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027764487000000001, 2024-09-18T04:39:14.134 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:14.134 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:14.134 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:14.134 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:14.134 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:14.134 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 39 2024-09-18T04:39:14.134 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.134 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:14.134 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.134 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:14.135 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535013+0000", 2024-09-18T04:39:14.135 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.135 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.135 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.135 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:14.135 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535013+0000", 2024-09-18T04:39:14.135 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.135 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.135 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.135 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:14.135 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535012+0000", 2024-09-18T04:39:14.135 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:14.135 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.135 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.136 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:14.136 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545016+0000", 2024-09-18T04:39:14.136 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01000322 2024-09-18T04:39:14.136 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.136 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.136 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:14.136 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545035+0000", 2024-09-18T04:39:14.136 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9845999999999999e-05 2024-09-18T04:39:14.136 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.136 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.136 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:14.136 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545045+0000", 2024-09-18T04:39:14.136 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3640000000000005e-06 2024-09-18T04:39:14.136 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.137 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.137 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:14.137 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545152+0000", 2024-09-18T04:39:14.137 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000107025 2024-09-18T04:39:14.137 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.137 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.137 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:14.137 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545338+0000", 2024-09-18T04:39:14.137 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018643500000000001 2024-09-18T04:39:14.137 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.137 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.137 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:14.137 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545492+0000", 2024-09-18T04:39:14.137 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015393199999999999 2024-09-18T04:39:14.138 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.138 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.138 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:14.138 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.553549+0000", 2024-09-18T04:39:14.138 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0080569170000000002 2024-09-18T04:39:14.138 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.138 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.138 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:14.138 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562693+0000", 2024-09-18T04:39:14.138 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0091441609999999996 2024-09-18T04:39:14.138 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.138 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.138 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:14.138 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562748+0000", 2024-09-18T04:39:14.139 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5047000000000003e-05 2024-09-18T04:39:14.139 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.139 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.139 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:14.139 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562778+0000", 2024-09-18T04:39:14.139 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9652e-05 2024-09-18T04:39:14.139 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.139 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:14.139 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.139 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.139 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.139 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:40 2.6 2:784d6cd1:ns::rbd_data.133e813ab5bc.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:14.139 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.545062+0000", 2024-09-18T04:39:14.139 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 31.524387253, 2024-09-18T04:39:14.140 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.032771623, 2024-09-18T04:39:14.140 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:14.140 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:14.140 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:14.140 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:14.140 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:14.140 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 40 2024-09-18T04:39:14.140 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.140 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:14.140 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.140 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:14.140 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545062+0000", 2024-09-18T04:39:14.140 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.140 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.140 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.141 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:14.141 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545062+0000", 2024-09-18T04:39:14.141 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.141 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.141 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:14.141 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545061+0000", 2024-09-18T04:39:14.141 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:14.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.141 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.141 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:14.141 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552477+0000", 2024-09-18T04:39:14.141 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.007415914 2024-09-18T04:39:14.142 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.142 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.142 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:14.142 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552500+0000", 2024-09-18T04:39:14.142 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3546000000000001e-05 2024-09-18T04:39:14.142 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.142 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.142 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:14.142 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552515+0000", 2024-09-18T04:39:14.142 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4606999999999999e-05 2024-09-18T04:39:14.142 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.142 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.142 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:14.142 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552661+0000", 2024-09-18T04:39:14.143 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000146007 2024-09-18T04:39:14.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.143 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.143 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:14.143 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552876+0000", 2024-09-18T04:39:14.143 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000214618 2024-09-18T04:39:14.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.143 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.143 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:14.143 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.553061+0000", 2024-09-18T04:39:14.143 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018564100000000001 2024-09-18T04:39:14.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.144 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.144 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:14.144 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.565612+0000", 2024-09-18T04:39:14.144 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012550301999999999 2024-09-18T04:39:14.144 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.144 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.144 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:14.144 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577695+0000", 2024-09-18T04:39:14.144 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012083493000000001 2024-09-18T04:39:14.144 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.144 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.144 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:14.144 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577772+0000", 2024-09-18T04:39:14.145 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7158999999999998e-05 2024-09-18T04:39:14.145 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.145 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.145 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:14.145 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577834+0000", 2024-09-18T04:39:14.145 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1851000000000001e-05 2024-09-18T04:39:14.145 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.145 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:14.145 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.145 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.145 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.145 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:43 2.6 2:759f27b6:ns::rbd_data.133e813ab5bc.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:14.145 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.555868+0000", 2024-09-18T04:39:14.145 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 31.513581759000001, 2024-09-18T04:39:14.146 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.022332497999999999, 2024-09-18T04:39:14.146 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:14.146 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:14.146 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:14.146 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:14.146 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:14.146 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 43 2024-09-18T04:39:14.146 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.146 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:14.146 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.146 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:14.146 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555868+0000", 2024-09-18T04:39:14.146 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.146 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.147 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.147 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:14.147 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555868+0000", 2024-09-18T04:39:14.147 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.147 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.147 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.147 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:14.147 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555865+0000", 2024-09-18T04:39:14.147 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999971 2024-09-18T04:39:14.147 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.147 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.147 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:14.147 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561031+0000", 2024-09-18T04:39:14.147 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0051658770000000001 2024-09-18T04:39:14.148 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.148 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.148 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:14.148 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561048+0000", 2024-09-18T04:39:14.148 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6569e-05 2024-09-18T04:39:14.148 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.148 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.148 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:14.148 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561055+0000", 2024-09-18T04:39:14.148 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.328e-06 2024-09-18T04:39:14.148 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.148 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.149 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:14.149 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561114+0000", 2024-09-18T04:39:14.149 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9540000000000003e-05 2024-09-18T04:39:14.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.149 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.149 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:14.149 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561314+0000", 2024-09-18T04:39:14.149 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019965799999999999 2024-09-18T04:39:14.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.149 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.149 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:14.149 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561473+0000", 2024-09-18T04:39:14.149 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000158904 2024-09-18T04:39:14.150 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.150 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.150 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:14.150 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.567015+0000", 2024-09-18T04:39:14.150 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055420859999999999 2024-09-18T04:39:14.150 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.150 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.150 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:14.150 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577982+0000", 2024-09-18T04:39:14.150 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010966872000000001 2024-09-18T04:39:14.150 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.150 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.150 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:14.150 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578162+0000", 2024-09-18T04:39:14.151 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018050999999999999 2024-09-18T04:39:14.151 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.151 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.151 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:14.151 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578200+0000", 2024-09-18T04:39:14.151 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8000999999999998e-05 2024-09-18T04:39:14.151 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.151 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:14.151 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.151 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.151 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.151 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:48 2.6 2:7c225365:ns::rbd_data.133e813ab5bc.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:14.151 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:39:14.152 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 31.505782536000002, 2024-09-18T04:39:14.152 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.024576377, 2024-09-18T04:39:14.152 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:14.152 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:14.152 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:14.152 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:14.152 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:14.152 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 48 2024-09-18T04:39:14.152 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.152 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:14.152 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.152 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:14.152 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:39:14.152 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.153 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.153 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.153 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:14.153 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:39:14.153 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.153 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.153 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.153 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:14.153 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.563666+0000", 2024-09-18T04:39:14.153 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:14.153 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.153 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.153 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:14.153 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570331+0000", 2024-09-18T04:39:14.154 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0066653329999999998 2024-09-18T04:39:14.154 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.154 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.154 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:14.154 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570348+0000", 2024-09-18T04:39:14.154 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6507e-05 2024-09-18T04:39:14.154 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.154 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.154 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:14.154 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570355+0000", 2024-09-18T04:39:14.154 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2749999999999998e-06 2024-09-18T04:39:14.154 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.154 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.154 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:14.155 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570396+0000", 2024-09-18T04:39:14.155 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0809000000000002e-05 2024-09-18T04:39:14.155 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.155 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.155 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:14.155 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570571+0000", 2024-09-18T04:39:14.155 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000174823 2024-09-18T04:39:14.155 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.155 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.155 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:14.155 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570722+0000", 2024-09-18T04:39:14.155 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015154599999999999 2024-09-18T04:39:14.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.156 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.156 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:14.156 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579756+0000", 2024-09-18T04:39:14.156 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0090337000000000004 2024-09-18T04:39:14.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.156 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.156 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:14.156 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588093+0000", 2024-09-18T04:39:14.156 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.008336553 2024-09-18T04:39:14.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.157 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.157 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:14.157 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588173+0000", 2024-09-18T04:39:14.157 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0333999999999994e-05 2024-09-18T04:39:14.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.157 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.157 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:14.157 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588244+0000", 2024-09-18T04:39:14.157 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0573000000000003e-05 2024-09-18T04:39:14.157 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.157 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:14.157 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.157 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.158 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:49 2.4 2:20f0b0c9:ns::rbd_data.133e813ab5bc.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:14.158 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.570370+0000", 2024-09-18T04:39:14.158 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 31.499080208999999, 2024-09-18T04:39:14.158 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.029633981, 2024-09-18T04:39:14.158 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:14.158 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:14.158 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:14.158 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:14.158 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:14.158 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 49 2024-09-18T04:39:14.158 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.158 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:14.158 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.158 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:14.159 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570370+0000", 2024-09-18T04:39:14.159 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.159 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.159 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:14.159 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570370+0000", 2024-09-18T04:39:14.159 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.159 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.159 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:14.159 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570369+0000", 2024-09-18T04:39:14.159 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:14.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.160 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.160 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:14.160 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579090+0000", 2024-09-18T04:39:14.160 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0087212339999999996 2024-09-18T04:39:14.160 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.160 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.160 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:14.160 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579109+0000", 2024-09-18T04:39:14.160 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9120999999999999e-05 2024-09-18T04:39:14.160 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.160 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.160 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:14.160 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579123+0000", 2024-09-18T04:39:14.161 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4382000000000001e-05 2024-09-18T04:39:14.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.161 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.161 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:14.161 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579221+0000", 2024-09-18T04:39:14.161 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.7503999999999993e-05 2024-09-18T04:39:14.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.161 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.161 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:14.161 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579553+0000", 2024-09-18T04:39:14.161 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00033254200000000002 2024-09-18T04:39:14.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.161 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.162 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:14.162 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579817+0000", 2024-09-18T04:39:14.162 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00026317099999999998 2024-09-18T04:39:14.162 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.162 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.162 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:14.162 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.599905+0000", 2024-09-18T04:39:14.162 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020088575000000001 2024-09-18T04:39:14.162 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.162 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.162 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:14.162 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.599923+0000", 2024-09-18T04:39:14.162 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7425999999999999e-05 2024-09-18T04:39:14.163 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.163 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.163 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:14.163 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.599973+0000", 2024-09-18T04:39:14.163 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0649999999999998e-05 2024-09-18T04:39:14.163 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.163 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.163 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:14.163 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.600004+0000", 2024-09-18T04:39:14.163 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0361999999999998e-05 2024-09-18T04:39:14.163 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.163 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:14.163 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.163 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.164 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.164 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:50 2.1 e51/24 2:8c706374:ns::rbd_data.133e813ab5bc.000000000000000a:head v 51'8, mlcod=51'6)", 2024-09-18T04:39:14.164 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.577029+0000", 2024-09-18T04:39:14.164 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 31.492420838000001, 2024-09-18T04:39:14.164 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.017362855, 2024-09-18T04:39:14.164 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:14.164 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:14.164 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:14.164 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.164 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:14.164 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577029+0000", 2024-09-18T04:39:14.164 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.164 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.164 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.165 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:14.165 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577029+0000", 2024-09-18T04:39:14.165 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.165 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.165 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.165 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:14.165 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577027+0000", 2024-09-18T04:39:14.165 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:14.165 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.165 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.165 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:14.165 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578880+0000", 2024-09-18T04:39:14.166 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0018532080000000001 2024-09-18T04:39:14.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.166 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.166 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:14.166 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578906+0000", 2024-09-18T04:39:14.166 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5573999999999999e-05 2024-09-18T04:39:14.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.166 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.166 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:14.166 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578920+0000", 2024-09-18T04:39:14.166 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4483e-05 2024-09-18T04:39:14.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.166 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.166 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:14.166 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578983+0000", 2024-09-18T04:39:14.167 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2248000000000004e-05 2024-09-18T04:39:14.167 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.167 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.167 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:14.167 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579016+0000", 2024-09-18T04:39:14.167 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3364000000000001e-05 2024-09-18T04:39:14.167 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.167 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.167 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:14.167 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594342+0000", 2024-09-18T04:39:14.167 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01532564 2024-09-18T04:39:14.167 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.167 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.167 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:14.168 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594392+0000", 2024-09-18T04:39:14.168 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0068999999999998e-05 2024-09-18T04:39:14.168 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.168 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:14.168 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.168 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.168 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.168 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:45 2.0 e51/17 2:0fbfa3dc:ns::rbd_data.133e813ab5bc.0000000000000007:head v 51'24, mlcod=51'22)", 2024-09-18T04:39:14.168 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.581892+0000", 2024-09-18T04:39:14.168 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 31.487558165999999, 2024-09-18T04:39:14.168 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015646006, 2024-09-18T04:39:14.168 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:14.168 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:14.168 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:14.169 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.169 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:14.169 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581892+0000", 2024-09-18T04:39:14.169 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.169 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.169 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.169 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:14.169 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581892+0000", 2024-09-18T04:39:14.169 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.169 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.169 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.169 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:14.169 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581890+0000", 2024-09-18T04:39:14.169 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:14.170 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.170 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.170 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:14.170 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586188+0000", 2024-09-18T04:39:14.170 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0042985769999999996 2024-09-18T04:39:14.170 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.170 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.170 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:14.170 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586210+0000", 2024-09-18T04:39:14.170 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1069000000000001e-05 2024-09-18T04:39:14.170 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.170 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.170 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:14.170 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586221+0000", 2024-09-18T04:39:14.170 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1287000000000001e-05 2024-09-18T04:39:14.171 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.171 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.171 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:14.171 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586313+0000", 2024-09-18T04:39:14.171 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1935999999999995e-05 2024-09-18T04:39:14.171 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.171 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.171 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:14.171 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586368+0000", 2024-09-18T04:39:14.171 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5578000000000002e-05 2024-09-18T04:39:14.171 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.171 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.171 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:14.171 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.597473+0000", 2024-09-18T04:39:14.171 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011105139999999999 2024-09-18T04:39:14.172 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.172 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.172 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:14.172 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.597538+0000", 2024-09-18T04:39:14.172 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4133000000000004e-05 2024-09-18T04:39:14.172 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.172 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:14.172 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.172 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.172 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.172 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:53 2.4 2:39f7f174:ns::rbd_data.133e813ab5bc.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:14.172 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.584263+0000", 2024-09-18T04:39:14.173 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 31.485186292000002, 2024-09-18T04:39:14.173 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018701180000000001, 2024-09-18T04:39:14.173 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:14.173 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:14.173 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:14.173 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:14.173 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:14.173 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 53 2024-09-18T04:39:14.173 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.173 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:14.173 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.173 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:14.174 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584263+0000", 2024-09-18T04:39:14.174 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.174 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.174 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.174 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:14.174 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584263+0000", 2024-09-18T04:39:14.174 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.174 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.174 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.174 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:14.174 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584262+0000", 2024-09-18T04:39:14.174 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:14.174 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.175 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.175 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:14.175 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.591617+0000", 2024-09-18T04:39:14.175 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0073550600000000001 2024-09-18T04:39:14.175 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.175 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.175 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:14.175 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.591637+0000", 2024-09-18T04:39:14.175 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0432999999999999e-05 2024-09-18T04:39:14.175 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.175 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.175 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:14.175 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.591646+0000", 2024-09-18T04:39:14.175 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7849999999999997e-06 2024-09-18T04:39:14.176 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.176 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.176 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:14.176 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593478+0000", 2024-09-18T04:39:14.176 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001831495 2024-09-18T04:39:14.176 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.176 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.176 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:14.176 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593661+0000", 2024-09-18T04:39:14.176 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018364999999999999 2024-09-18T04:39:14.176 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.176 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.176 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:14.177 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593834+0000", 2024-09-18T04:39:14.177 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000173065 2024-09-18T04:39:14.177 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.177 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.177 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:14.177 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.600670+0000", 2024-09-18T04:39:14.177 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0068354970000000003 2024-09-18T04:39:14.177 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.177 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.177 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:14.177 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602851+0000", 2024-09-18T04:39:14.177 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002181653 2024-09-18T04:39:14.177 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.177 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.178 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:14.178 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602930+0000", 2024-09-18T04:39:14.178 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8045000000000002e-05 2024-09-18T04:39:14.178 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.178 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.178 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:14.178 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602965+0000", 2024-09-18T04:39:14.178 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5102999999999999e-05 2024-09-18T04:39:14.178 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.178 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:14.178 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.178 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.178 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.178 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:56 2.6 2:72453c3e:ns::rbd_data.133e813ab5bc.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:14.179 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.595988+0000", 2024-09-18T04:39:14.179 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 31.473461889999999, 2024-09-18T04:39:14.179 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.030296172, 2024-09-18T04:39:14.179 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:14.179 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:14.179 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:14.179 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:14.179 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:14.179 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 56 2024-09-18T04:39:14.179 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.179 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:14.179 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.179 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:14.180 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.595988+0000", 2024-09-18T04:39:14.180 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.180 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.180 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.180 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:14.180 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.595988+0000", 2024-09-18T04:39:14.180 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.180 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.180 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.180 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:14.180 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.595986+0000", 2024-09-18T04:39:14.180 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:14.180 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.181 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.181 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:14.181 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605262+0000", 2024-09-18T04:39:14.181 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0092767240000000001 2024-09-18T04:39:14.181 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.181 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.181 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:14.181 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605282+0000", 2024-09-18T04:39:14.181 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9307e-05 2024-09-18T04:39:14.181 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.181 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.181 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:14.181 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605292+0000", 2024-09-18T04:39:14.182 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9510000000000001e-06 2024-09-18T04:39:14.182 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.182 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.182 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:14.182 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605369+0000", 2024-09-18T04:39:14.182 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7846e-05 2024-09-18T04:39:14.182 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.182 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.182 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:14.182 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605668+0000", 2024-09-18T04:39:14.182 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00029896099999999999 2024-09-18T04:39:14.182 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.182 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.182 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:14.183 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605906+0000", 2024-09-18T04:39:14.183 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000237758 2024-09-18T04:39:14.183 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.183 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.183 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:14.183 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.613532+0000", 2024-09-18T04:39:14.183 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0076260590000000001 2024-09-18T04:39:14.183 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.183 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.183 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:14.183 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.626116+0000", 2024-09-18T04:39:14.183 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012584092999999999 2024-09-18T04:39:14.183 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.183 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.184 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:14.184 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.626216+0000", 2024-09-18T04:39:14.184 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9630999999999995e-05 2024-09-18T04:39:14.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.184 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.184 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:14.184 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.626284+0000", 2024-09-18T04:39:14.184 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8034999999999995e-05 2024-09-18T04:39:14.184 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.184 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:14.184 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.184 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.184 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:14.185 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:14.186 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:39:14.187 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:14.181+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:14.287 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:14.281+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:14.345 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:14.345 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:14.345 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:39:14.345 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:14.355 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:39:14.384 INFO:tasks.workunit.client.0.smithi115.stderr:+ DEV= 2024-09-18T04:39:14.384 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd config global rm global rbd_default_pool 2024-09-18T04:39:14.388 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:14.385+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:14.445 INFO:tasks.workunit.client.0.smithi115.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd17342 2024-09-18T04:39:14.446 INFO:tasks.workunit.client.0.smithi115.stderr:++ local cmd 2024-09-18T04:39:14.446 INFO:tasks.workunit.client.0.smithi115.stderr:+++ id -u 2024-09-18T04:39:14.447 INFO:tasks.workunit.client.0.smithi115.stderr:++ '[' 1000 -eq 0 ']' 2024-09-18T04:39:14.448 INFO:tasks.workunit.client.0.smithi115.stderr:+++ which rbd 2024-09-18T04:39:14.450 INFO:tasks.workunit.client.0.smithi115.stderr:++ cmd=/usr/bin/rbd 2024-09-18T04:39:14.450 INFO:tasks.workunit.client.0.smithi115.stderr:++ shift 2024-09-18T04:39:14.450 INFO:tasks.workunit.client.0.smithi115.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd17342 2024-09-18T04:39:14.469 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:14.469 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:14.469 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:39:14.469 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:39:14.470 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:14.479 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:39:14.489 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:14.481+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:14.549 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-18T04:39:14.591 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:14.585+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:14.612 INFO:tasks.workunit.client.0.smithi115.stderr:+ DEV=/dev/nbd0 2024-09-18T04:39:14.612 INFO:tasks.workunit.client.0.smithi115.stderr:+ get_pid rbd 2024-09-18T04:39:14.612 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pool=rbd 2024-09-18T04:39:14.612 INFO:tasks.workunit.client.0.smithi115.stderr:+ local ns= 2024-09-18T04:39:14.613 INFO:tasks.workunit.client.0.smithi115.stderr:++ rbd device --device-type nbd --format xml list 2024-09-18T04:39:14.613 INFO:tasks.workunit.client.0.smithi115.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd17342'\''][device='\''/dev/nbd0'\'']/id' 2024-09-18T04:39:14.615 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:14.615 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:39:14.615 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:39:14.616 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:39:14.616 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.616 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:33 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:14.616 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.261182+0000", 2024-09-18T04:39:14.616 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 32.351084444000001, 2024-09-18T04:39:14.616 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.037082208999999998, 2024-09-18T04:39:14.617 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:14.617 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:14.617 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:14.617 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:14.617 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:14.617 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 33 2024-09-18T04:39:14.618 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.618 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:14.618 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.618 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:14.618 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.261182+0000", 2024-09-18T04:39:14.619 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.619 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.619 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:14.619 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.261182+0000", 2024-09-18T04:39:14.619 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.619 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.620 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:14.620 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.261180+0000", 2024-09-18T04:39:14.620 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:14.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.620 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.620 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:14.620 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278669+0000", 2024-09-18T04:39:14.620 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.017488877999999999 2024-09-18T04:39:14.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.621 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.621 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:14.621 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278689+0000", 2024-09-18T04:39:14.621 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0047999999999998e-05 2024-09-18T04:39:14.621 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.621 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.621 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:14.621 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278699+0000", 2024-09-18T04:39:14.621 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6800000000000005e-06 2024-09-18T04:39:14.621 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.621 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.622 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:14.622 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278775+0000", 2024-09-18T04:39:14.622 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6087999999999994e-05 2024-09-18T04:39:14.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.622 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.622 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:14.622 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278935+0000", 2024-09-18T04:39:14.622 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000159834 2024-09-18T04:39:14.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.622 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.622 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:14.622 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279064+0000", 2024-09-18T04:39:14.622 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012954199999999999 2024-09-18T04:39:14.623 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.623 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.623 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:14.623 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295562+0000", 2024-09-18T04:39:14.623 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016497583999999999 2024-09-18T04:39:14.623 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.623 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.623 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:14.623 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.298147+0000", 2024-09-18T04:39:14.623 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002584982 2024-09-18T04:39:14.623 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.623 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.623 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:14.623 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.298217+0000", 2024-09-18T04:39:14.624 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0400000000000004e-05 2024-09-18T04:39:14.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.624 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.624 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:14.624 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.298265+0000", 2024-09-18T04:39:14.624 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7580000000000002e-05 2024-09-18T04:39:14.624 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.624 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:14.624 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.624 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.624 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:34 2.3 e51/24 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head v 51'3, mlcod=18'1)", 2024-09-18T04:39:14.624 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.273067+0000", 2024-09-18T04:39:14.625 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 32.339199653999998, 2024-09-18T04:39:14.625 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.021784103999999999, 2024-09-18T04:39:14.625 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:14.625 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:14.625 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:14.625 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.625 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:14.625 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.273067+0000", 2024-09-18T04:39:14.626 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.626 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.626 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.626 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:14.626 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.273067+0000", 2024-09-18T04:39:14.626 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.627 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.627 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.627 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:14.627 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.273060+0000", 2024-09-18T04:39:14.627 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999924 2024-09-18T04:39:14.627 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.627 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.627 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:14.628 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278596+0000", 2024-09-18T04:39:14.628 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055365839999999998 2024-09-18T04:39:14.628 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.628 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.628 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:14.628 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278637+0000", 2024-09-18T04:39:14.629 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0596000000000002e-05 2024-09-18T04:39:14.629 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.629 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.629 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:14.629 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278653+0000", 2024-09-18T04:39:14.630 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6626999999999999e-05 2024-09-18T04:39:14.630 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.630 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.630 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:14.630 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278748+0000", 2024-09-18T04:39:14.630 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4483999999999999e-05 2024-09-18T04:39:14.630 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.631 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.631 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:14.631 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278799+0000", 2024-09-18T04:39:14.631 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1484999999999999e-05 2024-09-18T04:39:14.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.631 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.631 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:14.631 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.294785+0000", 2024-09-18T04:39:14.631 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015985256999999999 2024-09-18T04:39:14.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.631 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.632 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:14.632 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.294851+0000", 2024-09-18T04:39:14.632 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6728000000000002e-05 2024-09-18T04:39:14.632 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.632 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:14.632 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.632 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.632 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.632 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:37 2.5 2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:14.632 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.278722+0000", 2024-09-18T04:39:14.633 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 32.333544605, 2024-09-18T04:39:14.633 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.026323801000000001, 2024-09-18T04:39:14.633 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:14.633 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:14.633 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:14.633 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:14.633 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:14.633 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 37 2024-09-18T04:39:14.633 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.633 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:14.634 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.634 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:14.634 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278722+0000", 2024-09-18T04:39:14.634 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.634 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.634 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:14.634 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278722+0000", 2024-09-18T04:39:14.634 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.635 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.635 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:14.635 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278720+0000", 2024-09-18T04:39:14.635 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:14.635 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.635 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.635 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:14.635 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.291875+0000", 2024-09-18T04:39:14.635 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013154414 2024-09-18T04:39:14.636 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.636 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.636 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:14.636 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.291901+0000", 2024-09-18T04:39:14.636 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6339999999999999e-05 2024-09-18T04:39:14.636 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.636 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.636 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:14.636 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.291914+0000", 2024-09-18T04:39:14.637 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3043e-05 2024-09-18T04:39:14.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.637 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.637 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:14.637 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.291975+0000", 2024-09-18T04:39:14.637 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0949999999999998e-05 2024-09-18T04:39:14.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.637 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.637 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:14.638 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292173+0000", 2024-09-18T04:39:14.638 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019825499999999999 2024-09-18T04:39:14.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.638 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.638 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:14.638 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292300+0000", 2024-09-18T04:39:14.638 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000126616 2024-09-18T04:39:14.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.638 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.638 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:14.638 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304334+0000", 2024-09-18T04:39:14.638 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012033934 2024-09-18T04:39:14.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.638 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.639 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:14.639 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304901+0000", 2024-09-18T04:39:14.639 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00056694099999999995 2024-09-18T04:39:14.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.639 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.639 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:14.639 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304981+0000", 2024-09-18T04:39:14.639 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9974000000000004e-05 2024-09-18T04:39:14.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.639 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.639 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:14.639 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.305046+0000", 2024-09-18T04:39:14.639 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5184000000000005e-05 2024-09-18T04:39:14.639 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.640 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:14.640 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.640 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.640 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:49 2.5 2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:14.640 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.314271+0000", 2024-09-18T04:39:14.640 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 32.297996003000002, 2024-09-18T04:39:14.640 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.029779953000000001, 2024-09-18T04:39:14.640 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:14.640 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:14.640 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:14.640 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:14.640 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:14.641 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 49 2024-09-18T04:39:14.641 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.641 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:14.641 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.641 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:14.641 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.314271+0000", 2024-09-18T04:39:14.641 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.641 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.641 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.641 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:14.641 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.314271+0000", 2024-09-18T04:39:14.641 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.641 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.642 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.642 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:14.642 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.314269+0000", 2024-09-18T04:39:14.642 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:14.642 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.642 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.642 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:14.642 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.324780+0000", 2024-09-18T04:39:14.642 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010511609999999999 2024-09-18T04:39:14.642 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.642 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.642 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:14.642 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.324821+0000", 2024-09-18T04:39:14.642 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0899000000000003e-05 2024-09-18T04:39:14.643 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.643 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.643 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:14.643 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.324840+0000", 2024-09-18T04:39:14.645 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8947e-05 2024-09-18T04:39:14.645 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.645 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.645 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:14.645 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.324936+0000", 2024-09-18T04:39:14.645 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6342999999999996e-05 2024-09-18T04:39:14.645 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.645 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.645 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:14.645 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325284+0000", 2024-09-18T04:39:14.645 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000347146 2024-09-18T04:39:14.645 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.645 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.646 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:14.646 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325586+0000", 2024-09-18T04:39:14.646 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00030199799999999999 2024-09-18T04:39:14.646 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.646 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.646 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:14.646 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343710+0000", 2024-09-18T04:39:14.646 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018124583999999999 2024-09-18T04:39:14.646 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.646 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.646 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:14.646 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343991+0000", 2024-09-18T04:39:14.646 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00028085600000000001 2024-09-18T04:39:14.646 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.647 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.647 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:14.647 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344028+0000", 2024-09-18T04:39:14.647 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6637000000000002e-05 2024-09-18T04:39:14.647 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.647 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.647 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:14.647 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344051+0000", 2024-09-18T04:39:14.647 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3260000000000001e-05 2024-09-18T04:39:14.647 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.647 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:14.647 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.648 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.648 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.648 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:50 2.3 e51/24 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head v 51'4, mlcod=51'2)", 2024-09-18T04:39:14.648 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.323066+0000", 2024-09-18T04:39:14.648 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 32.289200610000002, 2024-09-18T04:39:14.648 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.022135774, 2024-09-18T04:39:14.648 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:14.648 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:14.648 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:14.648 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.648 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:14.648 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.323066+0000", 2024-09-18T04:39:14.648 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.649 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.649 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.649 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:14.649 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.323066+0000", 2024-09-18T04:39:14.649 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.649 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.649 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.649 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:14.649 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.323065+0000", 2024-09-18T04:39:14.649 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:14.649 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.649 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.649 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:14.649 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.328664+0000", 2024-09-18T04:39:14.650 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055993600000000003 2024-09-18T04:39:14.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.650 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.650 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:14.650 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.328692+0000", 2024-09-18T04:39:14.650 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7654e-05 2024-09-18T04:39:14.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.650 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.650 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:14.650 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.328706+0000", 2024-09-18T04:39:14.650 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4097000000000001e-05 2024-09-18T04:39:14.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.650 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.650 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:14.651 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.328816+0000", 2024-09-18T04:39:14.651 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000110281 2024-09-18T04:39:14.651 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.651 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.651 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:14.651 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.328848+0000", 2024-09-18T04:39:14.651 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1837e-05 2024-09-18T04:39:14.651 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.651 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.651 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:14.651 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345160+0000", 2024-09-18T04:39:14.652 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016312099 2024-09-18T04:39:14.652 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.652 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.652 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:14.652 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345202+0000", 2024-09-18T04:39:14.652 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1857999999999998e-05 2024-09-18T04:39:14.652 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.652 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:14.652 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.652 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.653 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.653 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:58 2.5 2:b227e6b1:::rbd_data.125fc09df6d1.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:14.653 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.351564+0000", 2024-09-18T04:39:14.653 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 32.260703157999998, 2024-09-18T04:39:14.653 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.032125956999999997, 2024-09-18T04:39:14.653 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:14.653 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:14.653 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:14.653 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:14.654 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:14.654 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 58 2024-09-18T04:39:14.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.654 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:14.654 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.654 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:14.654 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.351564+0000", 2024-09-18T04:39:14.654 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.654 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.655 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:14.655 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.351564+0000", 2024-09-18T04:39:14.655 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.655 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.655 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:14.655 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.351561+0000", 2024-09-18T04:39:14.655 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:14.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.655 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.656 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:14.656 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364390+0000", 2024-09-18T04:39:14.656 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012829063999999999 2024-09-18T04:39:14.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.656 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.656 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:14.656 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364418+0000", 2024-09-18T04:39:14.656 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7307000000000001e-05 2024-09-18T04:39:14.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.656 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.657 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:14.657 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364428+0000", 2024-09-18T04:39:14.657 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0675e-05 2024-09-18T04:39:14.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.657 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.657 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:14.657 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364523+0000", 2024-09-18T04:39:14.657 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4698999999999996e-05 2024-09-18T04:39:14.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.658 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.658 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:14.658 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364785+0000", 2024-09-18T04:39:14.658 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00026170099999999998 2024-09-18T04:39:14.658 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.658 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.658 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:14.658 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.365039+0000", 2024-09-18T04:39:14.658 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00025462299999999999 2024-09-18T04:39:14.658 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.659 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.659 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:14.659 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383143+0000", 2024-09-18T04:39:14.659 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018103768999999999 2024-09-18T04:39:14.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.659 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.659 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:14.659 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383582+0000", 2024-09-18T04:39:14.659 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00043860600000000002 2024-09-18T04:39:14.660 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.660 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.660 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:14.660 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383644+0000", 2024-09-18T04:39:14.660 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2682999999999996e-05 2024-09-18T04:39:14.660 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.660 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.660 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:14.660 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383690+0000", 2024-09-18T04:39:14.661 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5247999999999998e-05 2024-09-18T04:39:14.661 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.661 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:14.661 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.661 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.661 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.661 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:33 2.1 2:8b40f1c5:ns::rbd_data.133e813ab5bc.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:14.661 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.523765+0000", 2024-09-18T04:39:14.661 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 32.088502118000001, 2024-09-18T04:39:14.662 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027472973000000001, 2024-09-18T04:39:14.662 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:14.662 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:14.662 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:14.662 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:14.662 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:14.663 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 33 2024-09-18T04:39:14.663 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.663 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:14.663 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.663 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:14.663 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.523765+0000", 2024-09-18T04:39:14.664 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.664 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.664 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.664 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:14.664 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.523765+0000", 2024-09-18T04:39:14.665 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.665 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.665 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.665 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:14.665 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.523762+0000", 2024-09-18T04:39:14.665 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:14.666 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.666 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.666 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:14.666 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531127+0000", 2024-09-18T04:39:14.666 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0073644560000000001 2024-09-18T04:39:14.667 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.667 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.667 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:14.667 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531189+0000", 2024-09-18T04:39:14.667 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2315000000000002e-05 2024-09-18T04:39:14.668 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.668 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.668 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:14.668 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531211+0000", 2024-09-18T04:39:14.668 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1542000000000001e-05 2024-09-18T04:39:14.669 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.669 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.669 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:14.669 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531314+0000", 2024-09-18T04:39:14.669 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000103582 2024-09-18T04:39:14.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.670 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.670 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:14.670 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531493+0000", 2024-09-18T04:39:14.670 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000178433 2024-09-18T04:39:14.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.671 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.671 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:14.671 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531653+0000", 2024-09-18T04:39:14.671 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000160388 2024-09-18T04:39:14.671 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.671 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.671 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:14.672 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.547345+0000", 2024-09-18T04:39:14.672 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015691742000000002 2024-09-18T04:39:14.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.672 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.672 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:14.672 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551076+0000", 2024-09-18T04:39:14.673 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003730957 2024-09-18T04:39:14.673 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.673 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.673 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:14.673 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551176+0000", 2024-09-18T04:39:14.673 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010002599999999999 2024-09-18T04:39:14.673 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.674 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.674 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:14.674 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551238+0000", 2024-09-18T04:39:14.674 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1945000000000004e-05 2024-09-18T04:39:14.674 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.674 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:14.674 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.675 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.675 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.675 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:36 2.5 2:bd5b8dd8:ns::rbd_data.133e813ab5bc.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:14.675 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.531253+0000", 2024-09-18T04:39:14.675 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 32.081013808999998, 2024-09-18T04:39:14.675 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027295212999999999, 2024-09-18T04:39:14.675 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:14.676 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:14.676 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:14.676 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:14.676 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:14.676 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 36 2024-09-18T04:39:14.676 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.677 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:14.677 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.677 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:14.677 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531253+0000", 2024-09-18T04:39:14.677 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.677 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.677 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.678 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:14.678 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531253+0000", 2024-09-18T04:39:14.678 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.678 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.678 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.678 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:14.678 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531251+0000", 2024-09-18T04:39:14.678 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:14.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.679 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.679 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:14.679 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536540+0000", 2024-09-18T04:39:14.679 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0052885839999999998 2024-09-18T04:39:14.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.679 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.680 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:14.680 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536604+0000", 2024-09-18T04:39:14.680 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4824000000000001e-05 2024-09-18T04:39:14.680 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.680 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.680 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:14.680 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536621+0000", 2024-09-18T04:39:14.681 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6215999999999999e-05 2024-09-18T04:39:14.681 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.681 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.681 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:14.681 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536768+0000", 2024-09-18T04:39:14.681 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014767900000000001 2024-09-18T04:39:14.681 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.682 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.682 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:14.682 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537084+0000", 2024-09-18T04:39:14.682 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00031553499999999998 2024-09-18T04:39:14.682 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.682 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.682 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:14.683 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537300+0000", 2024-09-18T04:39:14.683 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000216145 2024-09-18T04:39:14.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.683 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.683 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:14.683 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555326+0000", 2024-09-18T04:39:14.683 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018025861000000001 2024-09-18T04:39:14.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.684 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.684 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:14.684 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558449+0000", 2024-09-18T04:39:14.684 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0031229750000000001 2024-09-18T04:39:14.684 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.684 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.684 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:14.685 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558502+0000", 2024-09-18T04:39:14.685 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3572000000000001e-05 2024-09-18T04:39:14.685 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.685 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.685 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:14.685 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558548+0000", 2024-09-18T04:39:14.686 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5921999999999998e-05 2024-09-18T04:39:14.686 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.686 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:14.686 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.686 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.686 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.687 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:37 2.5 2:bf39b8e4:ns::rbd_data.133e813ab5bc.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:14.687 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.536664+0000", 2024-09-18T04:39:14.687 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 32.075602392999997, 2024-09-18T04:39:14.687 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.026331765, 2024-09-18T04:39:14.687 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:14.687 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:14.688 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:14.688 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:14.688 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:14.688 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 37 2024-09-18T04:39:14.688 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.688 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:14.688 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.688 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:14.688 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536664+0000", 2024-09-18T04:39:14.688 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.689 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.689 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:14.689 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536664+0000", 2024-09-18T04:39:14.689 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.689 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.690 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:14.690 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536662+0000", 2024-09-18T04:39:14.690 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:14.690 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.690 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.690 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:14.690 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.543699+0000", 2024-09-18T04:39:14.690 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0070367119999999997 2024-09-18T04:39:14.690 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.690 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.691 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:14.691 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.543749+0000", 2024-09-18T04:39:14.691 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0476999999999998e-05 2024-09-18T04:39:14.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.691 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.691 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:14.691 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.543800+0000", 2024-09-18T04:39:14.692 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0822999999999998e-05 2024-09-18T04:39:14.692 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.692 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.692 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:14.692 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.547868+0000", 2024-09-18T04:39:14.692 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0040679339999999996 2024-09-18T04:39:14.692 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.692 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.692 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:14.693 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548087+0000", 2024-09-18T04:39:14.693 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000218994 2024-09-18T04:39:14.693 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.693 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.693 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:14.693 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548383+0000", 2024-09-18T04:39:14.693 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000295588 2024-09-18T04:39:14.693 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.693 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.693 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:14.693 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562622+0000", 2024-09-18T04:39:14.694 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014239915000000001 2024-09-18T04:39:14.694 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.694 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.694 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:14.694 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562924+0000", 2024-09-18T04:39:14.694 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00030127600000000001 2024-09-18T04:39:14.694 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.694 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.694 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:14.694 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562957+0000", 2024-09-18T04:39:14.694 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3365999999999999e-05 2024-09-18T04:39:14.694 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.695 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.695 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:14.695 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562996+0000", 2024-09-18T04:39:14.695 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9153999999999998e-05 2024-09-18T04:39:14.695 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.695 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:14.695 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.695 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.695 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:50 2.1 2:8c706374:ns::rbd_data.133e813ab5bc.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:14.696 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.570513+0000", 2024-09-18T04:39:14.696 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 32.041754111000003, 2024-09-18T04:39:14.696 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.024182995999999998, 2024-09-18T04:39:14.696 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:14.696 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:14.696 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:14.696 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:14.696 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:14.696 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 50 2024-09-18T04:39:14.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.696 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:14.696 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.697 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:14.697 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570513+0000", 2024-09-18T04:39:14.697 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.697 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.697 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:14.697 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570513+0000", 2024-09-18T04:39:14.697 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.697 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.697 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:14.698 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570511+0000", 2024-09-18T04:39:14.698 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:14.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.698 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.698 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:14.698 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.576327+0000", 2024-09-18T04:39:14.698 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0058162530000000004 2024-09-18T04:39:14.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.699 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.699 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:14.699 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.576355+0000", 2024-09-18T04:39:14.699 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8030000000000001e-05 2024-09-18T04:39:14.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.699 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.699 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:14.699 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.576366+0000", 2024-09-18T04:39:14.700 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1139e-05 2024-09-18T04:39:14.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.700 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.700 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:14.700 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.576458+0000", 2024-09-18T04:39:14.700 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1780999999999994e-05 2024-09-18T04:39:14.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.700 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.700 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:14.701 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.576701+0000", 2024-09-18T04:39:14.701 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00024303899999999999 2024-09-18T04:39:14.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.701 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.701 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:14.701 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.576876+0000", 2024-09-18T04:39:14.701 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000174637 2024-09-18T04:39:14.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.701 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.701 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:14.702 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588432+0000", 2024-09-18T04:39:14.702 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011555943000000001 2024-09-18T04:39:14.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.702 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.702 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:14.702 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594573+0000", 2024-09-18T04:39:14.702 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0061408670000000004 2024-09-18T04:39:14.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.702 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.703 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:14.703 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594640+0000", 2024-09-18T04:39:14.703 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7404999999999996e-05 2024-09-18T04:39:14.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.703 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.703 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:14.703 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594696+0000", 2024-09-18T04:39:14.703 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5612999999999998e-05 2024-09-18T04:39:14.703 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.703 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:14.704 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.704 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.704 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.704 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:60 2.5 2:a06f51e4:ns::rbd_data.133e813ab5bc.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:14.704 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.604753+0000", 2024-09-18T04:39:14.704 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 32.007513992, 2024-09-18T04:39:14.704 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014400181, 2024-09-18T04:39:14.704 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:14.704 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:14.704 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:14.705 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:14.705 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:14.705 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 60 2024-09-18T04:39:14.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.705 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:14.705 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.706 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:14.706 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.604753+0000", 2024-09-18T04:39:14.706 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.706 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.706 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:14.707 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.604753+0000", 2024-09-18T04:39:14.707 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.707 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.707 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:14.707 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.604751+0000", 2024-09-18T04:39:14.707 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:14.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.707 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.707 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:14.707 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.608980+0000", 2024-09-18T04:39:14.708 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0042292420000000002 2024-09-18T04:39:14.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.708 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.708 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:14.708 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609014+0000", 2024-09-18T04:39:14.708 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3572999999999999e-05 2024-09-18T04:39:14.709 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.709 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.709 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:14.709 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609032+0000", 2024-09-18T04:39:14.709 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8150000000000001e-05 2024-09-18T04:39:14.709 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.709 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.709 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:14.709 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609090+0000", 2024-09-18T04:39:14.709 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8214999999999997e-05 2024-09-18T04:39:14.709 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.709 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.709 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:14.710 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609235+0000", 2024-09-18T04:39:14.710 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014437799999999999 2024-09-18T04:39:14.710 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.710 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.710 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:14.710 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609360+0000", 2024-09-18T04:39:14.710 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012526099999999999 2024-09-18T04:39:14.710 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.710 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.710 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:14.710 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.614952+0000", 2024-09-18T04:39:14.711 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055922300000000001 2024-09-18T04:39:14.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.711 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.711 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:14.711 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.619070+0000", 2024-09-18T04:39:14.711 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0041178250000000003 2024-09-18T04:39:14.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.711 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.711 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:14.711 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.619117+0000", 2024-09-18T04:39:14.711 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7275999999999999e-05 2024-09-18T04:39:14.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.711 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.711 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:14.712 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.619153+0000", 2024-09-18T04:39:14.712 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5970000000000003e-05 2024-09-18T04:39:14.712 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.712 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:14.712 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.712 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.712 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.712 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4805.0:592 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [write 393216~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:14.712 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.023386+0000", 2024-09-18T04:39:14.712 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 23.588881287, 2024-09-18T04:39:14.712 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0086813870000000005, 2024-09-18T04:39:14.712 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:14.712 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:14.712 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:14.713 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4805", 2024-09-18T04:39:14.713 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:14.713 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 592 2024-09-18T04:39:14.713 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.713 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:14.713 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.713 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:14.713 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023386+0000", 2024-09-18T04:39:14.713 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.713 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.713 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.713 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:14.713 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023386+0000", 2024-09-18T04:39:14.714 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.714 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.714 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:14.714 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023384+0000", 2024-09-18T04:39:14.714 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:14.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.714 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.714 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:14.714 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023616+0000", 2024-09-18T04:39:14.714 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00023165600000000001 2024-09-18T04:39:14.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.714 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.715 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:14.715 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023637+0000", 2024-09-18T04:39:14.715 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1163e-05 2024-09-18T04:39:14.715 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.715 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.715 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:14.715 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023652+0000", 2024-09-18T04:39:14.715 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4562000000000001e-05 2024-09-18T04:39:14.715 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.715 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.715 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:14.715 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.027732+0000", 2024-09-18T04:39:14.715 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0040803929999999999 2024-09-18T04:39:14.715 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.716 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.716 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:14.716 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.027841+0000", 2024-09-18T04:39:14.716 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000108978 2024-09-18T04:39:14.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.716 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.716 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:14.716 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.027990+0000", 2024-09-18T04:39:14.716 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014852999999999999 2024-09-18T04:39:14.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.716 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.716 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:14.716 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.031930+0000", 2024-09-18T04:39:14.716 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0039406299999999997 2024-09-18T04:39:14.717 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.717 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.717 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:14.717 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.031978+0000", 2024-09-18T04:39:14.717 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7817e-05 2024-09-18T04:39:14.717 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.717 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.717 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:14.717 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.032019+0000", 2024-09-18T04:39:14.717 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0531999999999998e-05 2024-09-18T04:39:14.717 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.717 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.717 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:14.718 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.032067+0000", 2024-09-18T04:39:14.718 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8371000000000002e-05 2024-09-18T04:39:14.718 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.718 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:14.718 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.718 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.718 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.718 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4805.0:593 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [write 524288~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:14.718 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.023665+0000", 2024-09-18T04:39:14.718 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 23.588601440000001, 2024-09-18T04:39:14.718 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0085518090000000005, 2024-09-18T04:39:14.719 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:14.719 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:14.719 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:14.719 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4805", 2024-09-18T04:39:14.719 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:14.719 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 593 2024-09-18T04:39:14.719 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.719 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:14.719 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.719 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:14.719 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023665+0000", 2024-09-18T04:39:14.720 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.720 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.720 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.720 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:14.720 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023665+0000", 2024-09-18T04:39:14.720 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.721 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.721 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.721 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:14.721 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023664+0000", 2024-09-18T04:39:14.721 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:14.721 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.722 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.722 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:14.722 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023875+0000", 2024-09-18T04:39:14.722 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000210862 2024-09-18T04:39:14.722 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.722 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.722 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:14.723 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023906+0000", 2024-09-18T04:39:14.723 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0660000000000001e-05 2024-09-18T04:39:14.723 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.723 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.723 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:14.723 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023922+0000", 2024-09-18T04:39:14.723 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5831000000000001e-05 2024-09-18T04:39:14.723 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.724 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.724 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:14.724 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.029603+0000", 2024-09-18T04:39:14.724 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0056816719999999996 2024-09-18T04:39:14.724 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.724 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.725 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:14.725 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.029766+0000", 2024-09-18T04:39:14.725 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000162903 2024-09-18T04:39:14.725 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.725 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.725 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:14.726 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.030042+0000", 2024-09-18T04:39:14.726 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00027561399999999998 2024-09-18T04:39:14.726 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.726 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.726 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:14.726 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.032127+0000", 2024-09-18T04:39:14.726 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020850920000000002 2024-09-18T04:39:14.726 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.727 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.727 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:14.727 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.032137+0000", 2024-09-18T04:39:14.727 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0312000000000001e-05 2024-09-18T04:39:14.727 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.727 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.727 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:14.727 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.032183+0000", 2024-09-18T04:39:14.727 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6020999999999999e-05 2024-09-18T04:39:14.728 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.728 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.728 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:14.728 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.032217+0000", 2024-09-18T04:39:14.728 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4109999999999997e-05 2024-09-18T04:39:14.728 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.728 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:14.728 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.729 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.729 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.729 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4805.0:594 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [write 655360~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:14.729 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.023941+0000", 2024-09-18T04:39:14.729 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 23.588325902000001, 2024-09-18T04:39:14.729 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010767616000000001, 2024-09-18T04:39:14.730 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:14.730 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:14.730 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:14.730 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4805", 2024-09-18T04:39:14.730 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:14.731 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 594 2024-09-18T04:39:14.731 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.731 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:14.731 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.731 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:14.732 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023941+0000", 2024-09-18T04:39:14.732 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.732 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.732 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.732 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:14.732 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023941+0000", 2024-09-18T04:39:14.733 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.733 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.733 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.733 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:14.733 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.023940+0000", 2024-09-18T04:39:14.733 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:14.733 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.734 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.734 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:14.734 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024163+0000", 2024-09-18T04:39:14.734 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000223017 2024-09-18T04:39:14.734 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.734 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.735 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:14.735 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024188+0000", 2024-09-18T04:39:14.735 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5174e-05 2024-09-18T04:39:14.735 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.735 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.735 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:14.736 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024214+0000", 2024-09-18T04:39:14.736 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5782000000000002e-05 2024-09-18T04:39:14.736 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.736 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.736 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:14.736 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.031116+0000", 2024-09-18T04:39:14.736 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.006902282 2024-09-18T04:39:14.736 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.736 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.736 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:14.737 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.031199+0000", 2024-09-18T04:39:14.737 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3203999999999998e-05 2024-09-18T04:39:14.737 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.737 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.737 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:14.737 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.031315+0000", 2024-09-18T04:39:14.737 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000115742 2024-09-18T04:39:14.737 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.737 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.737 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:14.737 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.033780+0000", 2024-09-18T04:39:14.737 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002465058 2024-09-18T04:39:14.737 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.737 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.738 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:14.738 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.034630+0000", 2024-09-18T04:39:14.738 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00084966800000000004 2024-09-18T04:39:14.738 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.738 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.738 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:14.738 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.034682+0000", 2024-09-18T04:39:14.738 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2562999999999998e-05 2024-09-18T04:39:14.738 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.738 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.738 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:14.738 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.034709+0000", 2024-09-18T04:39:14.738 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.62e-05 2024-09-18T04:39:14.739 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.739 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:14.739 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.739 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.739 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.739 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4805.0:595 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [write 786432~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:14.739 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.024233+0000", 2024-09-18T04:39:14.739 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 23.588033527, 2024-09-18T04:39:14.740 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010599558, 2024-09-18T04:39:14.740 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:14.740 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:14.740 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:14.740 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4805", 2024-09-18T04:39:14.740 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:14.740 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 595 2024-09-18T04:39:14.740 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.741 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:14.741 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.741 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:14.741 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024233+0000", 2024-09-18T04:39:14.741 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.741 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.741 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.741 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:14.742 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024233+0000", 2024-09-18T04:39:14.742 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.742 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.742 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.742 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:14.742 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024232+0000", 2024-09-18T04:39:14.742 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:14.743 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.743 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.743 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:14.743 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024449+0000", 2024-09-18T04:39:14.743 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000216868 2024-09-18T04:39:14.743 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.743 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.744 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:14.744 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024474+0000", 2024-09-18T04:39:14.744 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4598999999999999e-05 2024-09-18T04:39:14.744 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.744 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.745 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:14.745 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024487+0000", 2024-09-18T04:39:14.745 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3451999999999999e-05 2024-09-18T04:39:14.745 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.745 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.745 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:14.745 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.032272+0000", 2024-09-18T04:39:14.745 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0077845709999999997 2024-09-18T04:39:14.745 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.746 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.746 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:14.746 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.032332+0000", 2024-09-18T04:39:14.746 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0736999999999998e-05 2024-09-18T04:39:14.746 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.746 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.746 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:14.746 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.032421+0000", 2024-09-18T04:39:14.746 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8993000000000007e-05 2024-09-18T04:39:14.746 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.746 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.746 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:14.746 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.033786+0000", 2024-09-18T04:39:14.747 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001364451 2024-09-18T04:39:14.747 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.747 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.747 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:14.747 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.034752+0000", 2024-09-18T04:39:14.747 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00096642400000000002 2024-09-18T04:39:14.747 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.747 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.747 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:14.747 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.034789+0000", 2024-09-18T04:39:14.747 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6899e-05 2024-09-18T04:39:14.747 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.747 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.747 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:14.748 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.034833+0000", 2024-09-18T04:39:14.748 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3846e-05 2024-09-18T04:39:14.748 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.748 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:14.748 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.748 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.748 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.748 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4805.0:596 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [write 917504~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:14.748 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.024500+0000", 2024-09-18T04:39:14.748 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 23.587766741999999, 2024-09-18T04:39:14.748 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010770534, 2024-09-18T04:39:14.748 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:14.748 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:14.748 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:14.749 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4805", 2024-09-18T04:39:14.749 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:14.749 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 596 2024-09-18T04:39:14.749 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.749 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:14.749 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.749 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:14.749 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024500+0000", 2024-09-18T04:39:14.749 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.749 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.749 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.749 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:14.749 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024500+0000", 2024-09-18T04:39:14.750 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.750 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.750 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.750 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:14.750 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024499+0000", 2024-09-18T04:39:14.750 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:14.750 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.750 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.750 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:14.750 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024700+0000", 2024-09-18T04:39:14.750 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00020038599999999999 2024-09-18T04:39:14.750 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.750 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.751 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:14.751 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024729+0000", 2024-09-18T04:39:14.751 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9224999999999999e-05 2024-09-18T04:39:14.751 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.751 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.751 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:14.751 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.024776+0000", 2024-09-18T04:39:14.751 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7385000000000001e-05 2024-09-18T04:39:14.751 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.751 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.751 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:14.751 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.032998+0000", 2024-09-18T04:39:14.751 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0082216189999999995 2024-09-18T04:39:14.751 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.751 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.752 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:14.752 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.033076+0000", 2024-09-18T04:39:14.752 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8668e-05 2024-09-18T04:39:14.752 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.752 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.752 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:14.752 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.033167+0000", 2024-09-18T04:39:14.752 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0728000000000003e-05 2024-09-18T04:39:14.752 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.752 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.752 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:14.752 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.034399+0000", 2024-09-18T04:39:14.752 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00123234 2024-09-18T04:39:14.753 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.753 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.753 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:14.753 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.035206+0000", 2024-09-18T04:39:14.753 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00080701799999999995 2024-09-18T04:39:14.753 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.753 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.753 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:14.753 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.035244+0000", 2024-09-18T04:39:14.753 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7487999999999998e-05 2024-09-18T04:39:14.753 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.753 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.753 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:14.753 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.035271+0000", 2024-09-18T04:39:14.754 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6709000000000001e-05 2024-09-18T04:39:14.754 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.754 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:14.754 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.754 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.754 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.754 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4805.0:597 2.1 2:9370a598:::rbd_header.125fc09df6d1:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:14.754 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.025206+0000", 2024-09-18T04:39:14.754 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 23.587060641000001, 2024-09-18T04:39:14.754 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0091536900000000008, 2024-09-18T04:39:14.754 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:14.754 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:14.754 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:14.754 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4805", 2024-09-18T04:39:14.754 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:14.755 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 597 2024-09-18T04:39:14.755 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.755 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:14.755 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.755 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:14.755 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.025206+0000", 2024-09-18T04:39:14.755 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.755 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.755 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.755 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:14.755 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.025206+0000", 2024-09-18T04:39:14.755 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.755 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.756 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.756 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:14.756 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.025204+0000", 2024-09-18T04:39:14.756 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:14.756 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.756 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.756 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:14.756 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.025216+0000", 2024-09-18T04:39:14.756 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2150999999999999e-05 2024-09-18T04:39:14.756 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.756 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.756 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:14.756 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.025237+0000", 2024-09-18T04:39:14.756 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0740000000000001e-05 2024-09-18T04:39:14.757 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.757 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.757 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:14.757 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.025254+0000", 2024-09-18T04:39:14.757 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6731e-05 2024-09-18T04:39:14.757 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.757 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.757 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:14.757 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.034049+0000", 2024-09-18T04:39:14.757 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0087952810000000003 2024-09-18T04:39:14.757 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.757 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.757 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:14.757 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.034202+0000", 2024-09-18T04:39:14.758 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0001526 2024-09-18T04:39:14.758 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.758 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.758 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:14.758 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.034360+0000", 2024-09-18T04:39:14.758 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015809299999999999 2024-09-18T04:39:14.758 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.758 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:14.758 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.758 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.758 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.758 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4972.0:14 2.5 2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:head [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:14.758 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.949251+0000", 2024-09-18T04:39:14.758 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.663015694999999, 2024-09-18T04:39:14.759 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0098324480000000006, 2024-09-18T04:39:14.759 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:14.759 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:14.759 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:14.759 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4972", 2024-09-18T04:39:14.759 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2280251472", 2024-09-18T04:39:14.759 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 14 2024-09-18T04:39:14.759 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.759 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:14.759 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.759 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:14.759 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949251+0000", 2024-09-18T04:39:14.759 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.760 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.760 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.760 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:14.760 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949251+0000", 2024-09-18T04:39:14.760 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.760 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.760 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.760 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:14.760 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949250+0000", 2024-09-18T04:39:14.760 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:14.760 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.760 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.760 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:14.761 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949258+0000", 2024-09-18T04:39:14.761 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0590000000000003e-06 2024-09-18T04:39:14.761 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.761 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.761 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:14.761 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949269+0000", 2024-09-18T04:39:14.761 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0607e-05 2024-09-18T04:39:14.761 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.761 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.761 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:14.761 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949277+0000", 2024-09-18T04:39:14.761 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8269999999999998e-06 2024-09-18T04:39:14.761 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.761 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.761 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:14.762 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949394+0000", 2024-09-18T04:39:14.762 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000117113 2024-09-18T04:39:14.762 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.762 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.762 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:14.762 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949527+0000", 2024-09-18T04:39:14.762 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013288300000000001 2024-09-18T04:39:14.762 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.762 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.762 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:14.762 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.959084+0000", 2024-09-18T04:39:14.762 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0095569550000000007 2024-09-18T04:39:14.762 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.762 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:14.763 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.763 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.763 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.763 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4972.0:20 2.5 2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:14.763 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.949296+0000", 2024-09-18T04:39:14.763 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.662971388999999, 2024-09-18T04:39:14.763 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014709732, 2024-09-18T04:39:14.763 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:14.763 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:14.763 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:14.763 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4972", 2024-09-18T04:39:14.763 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2280251472", 2024-09-18T04:39:14.763 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 20 2024-09-18T04:39:14.764 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.764 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:14.764 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.764 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:14.764 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949296+0000", 2024-09-18T04:39:14.764 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.764 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.764 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.764 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:14.764 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949296+0000", 2024-09-18T04:39:14.764 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.764 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.764 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.764 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:14.765 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949295+0000", 2024-09-18T04:39:14.765 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:14.765 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.765 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.765 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:14.765 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949301+0000", 2024-09-18T04:39:14.765 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4829999999999996e-06 2024-09-18T04:39:14.765 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.765 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.765 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:14.765 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949317+0000", 2024-09-18T04:39:14.765 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5866999999999999e-05 2024-09-18T04:39:14.765 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.766 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.766 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:14.766 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949331+0000", 2024-09-18T04:39:14.766 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3485999999999999e-05 2024-09-18T04:39:14.766 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.766 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.766 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:14.766 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.959137+0000", 2024-09-18T04:39:14.766 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0098066520000000008 2024-09-18T04:39:14.766 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.766 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.766 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:14.766 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.959251+0000", 2024-09-18T04:39:14.766 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000113783 2024-09-18T04:39:14.767 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.767 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.767 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:14.767 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.964005+0000", 2024-09-18T04:39:14.767 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0047542080000000002 2024-09-18T04:39:14.767 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.767 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:14.767 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.767 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.767 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.767 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4970.0:37 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [write 1966080~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:14.767 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.516050+0000", 2024-09-18T04:39:14.767 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 14.096216532, 2024-09-18T04:39:14.767 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0083751499999999996, 2024-09-18T04:39:14.768 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:14.768 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:14.768 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:14.768 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4970", 2024-09-18T04:39:14.768 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:14.768 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 37 2024-09-18T04:39:14.768 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.768 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:14.768 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.768 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:14.768 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.516050+0000", 2024-09-18T04:39:14.768 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.768 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.769 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.769 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:14.769 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.516050+0000", 2024-09-18T04:39:14.769 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:14.769 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.769 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.769 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:14.769 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.516050+0000", 2024-09-18T04:39:14.769 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:14.769 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.769 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.769 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:14.769 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.516150+0000", 2024-09-18T04:39:14.769 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000100159 2024-09-18T04:39:14.770 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.770 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.770 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:14.770 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.516157+0000", 2024-09-18T04:39:14.770 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7710000000000001e-06 2024-09-18T04:39:14.770 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.770 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.770 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:14.770 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.516163+0000", 2024-09-18T04:39:14.770 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.596e-06 2024-09-18T04:39:14.770 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.770 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.770 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:14.770 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.522975+0000", 2024-09-18T04:39:14.771 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0068118140000000002 2024-09-18T04:39:14.771 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.771 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.771 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:14.771 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.523028+0000", 2024-09-18T04:39:14.771 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2998999999999999e-05 2024-09-18T04:39:14.771 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.771 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.771 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:14.771 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.523107+0000", 2024-09-18T04:39:14.771 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9357000000000006e-05 2024-09-18T04:39:14.771 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.771 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.771 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:14.772 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.524167+0000", 2024-09-18T04:39:14.772 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0010592329999999999 2024-09-18T04:39:14.772 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.772 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.772 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:14.772 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.524369+0000", 2024-09-18T04:39:14.772 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00020209000000000001 2024-09-18T04:39:14.772 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.772 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.772 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:14.772 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.524402+0000", 2024-09-18T04:39:14.772 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3685e-05 2024-09-18T04:39:14.772 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:14.773 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:14.773 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:14.773 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.524426+0000", 2024-09-18T04:39:14.773 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3178000000000001e-05 2024-09-18T04:39:14.773 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.773 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:14.773 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.773 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:14.773 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:14.773 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:14.774 INFO:tasks.workunit.client.0.smithi115.stderr:+ PID=19427 2024-09-18T04:39:14.774 INFO:tasks.workunit.client.0.smithi115.stderr:+ test -n 19427 2024-09-18T04:39:14.775 INFO:tasks.workunit.client.0.smithi115.stderr:+ ps -p 19427 -C rbd-nbd 2024-09-18T04:39:14.775 INFO:tasks.workunit.client.0.smithi115.stdout: PID TTY TIME CMD 2024-09-18T04:39:14.775 INFO:tasks.workunit.client.0.smithi115.stdout: 19427 ? 00:00:00 rbd-nbd 2024-09-18T04:39:14.776 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:39:14.777 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo kill 19427 2024-09-18T04:39:14.777 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:39:14.777 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:39:14.777 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:39:14.778 INFO:tasks.workunit.client.0.smithi115.stderr:++ which kill 2024-09-18T04:39:14.778 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/bin/kill 2024-09-18T04:39:14.778 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:39:14.778 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/bin/kill 19427 2024-09-18T04:39:14.778 INFO:tasks.workunit.client.0.smithi115.stderr:++ seq 10 2024-09-18T04:39:14.778 INFO:tasks.workunit.client.0.smithi115.stderr:+ for i in `seq 10` 2024-09-18T04:39:14.778 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd device --device-type nbd list 2024-09-18T04:39:14.778 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false grep '^19427 *rbd *testrbdnbd17342' 2024-09-18T04:39:14.778 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep '^19427 *rbd *testrbdnbd17342' 2024-09-18T04:39:14.779 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:14.769+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:14.823 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:39:14.824 INFO:tasks.workunit.client.0.smithi115.stderr:+ break 2024-09-18T04:39:14.824 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd device --device-type nbd list 2024-09-18T04:39:14.824 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false grep '^19427 *rbd *testrbdnbd17342' 2024-09-18T04:39:14.824 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep '^19427 *rbd *testrbdnbd17342' 2024-09-18T04:39:14.876 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:14.869+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:14.893 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:14.893 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:14.893 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:39:14.894 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:14.894 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:39:14.894 INFO:tasks.workunit.client.0.smithi115.stderr:+ QUIESCE_HOOK=/tmp/tmp.WJMNJxIdAN/quiesce.sh 2024-09-18T04:39:14.895 INFO:tasks.workunit.client.0.smithi115.stderr:++ _sudo rbd device --device-type nbd map --quiesce --quiesce-hook /tmp/tmp.WJMNJxIdAN/quiesce.sh rbd/testrbdnbd17342 2024-09-18T04:39:14.895 INFO:tasks.workunit.client.0.smithi115.stderr:++ local cmd 2024-09-18T04:39:14.895 INFO:tasks.workunit.client.0.smithi115.stderr:+++ id -u 2024-09-18T04:39:14.895 INFO:tasks.workunit.client.0.smithi115.stderr:++ '[' 1000 -eq 0 ']' 2024-09-18T04:39:14.896 INFO:tasks.workunit.client.0.smithi115.stderr:+++ which rbd 2024-09-18T04:39:14.898 INFO:tasks.workunit.client.0.smithi115.stderr:++ cmd=/usr/bin/rbd 2024-09-18T04:39:14.898 INFO:tasks.workunit.client.0.smithi115.stderr:++ shift 2024-09-18T04:39:14.898 INFO:tasks.workunit.client.0.smithi115.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map --quiesce --quiesce-hook /tmp/tmp.WJMNJxIdAN/quiesce.sh rbd/testrbdnbd17342 2024-09-18T04:39:14.903 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:39:14.977 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:14.969+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:39:15.009 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:15.010 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:15.010 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:39:15.010 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:39:15.010 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:15.020 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:39:15.061 INFO:tasks.workunit.client.0.smithi115.stderr:+ DEV=/dev/nbd0 2024-09-18T04:39:15.061 INFO:tasks.workunit.client.0.smithi115.stderr:+ get_pid rbd 2024-09-18T04:39:15.061 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pool=rbd 2024-09-18T04:39:15.061 INFO:tasks.workunit.client.0.smithi115.stderr:+ local ns= 2024-09-18T04:39:15.061 INFO:tasks.workunit.client.0.smithi115.stderr:++ rbd device --device-type nbd --format xml list 2024-09-18T04:39:15.062 INFO:tasks.workunit.client.0.smithi115.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd17342'\''][device='\''/dev/nbd0'\'']/id' 2024-09-18T04:39:15.078 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:15.073+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:39:15.127 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:15.127 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:39:15.127 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:39:15.127 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:39:15.127 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.127 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:39:15.127 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:39:15.127 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 107.334651776, 2024-09-18T04:39:15.127 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017399218000000001, 2024-09-18T04:39:15.127 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:15.128 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:15.128 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:15.128 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:39:15.128 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:39:15.128 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 114 2024-09-18T04:39:15.128 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.128 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:15.128 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.128 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:15.128 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:39:15.128 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:15.128 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.128 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.129 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:15.129 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:39:15.129 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:15.129 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.129 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.129 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:15.129 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786900+0000", 2024-09-18T04:39:15.129 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:15.129 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.129 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.129 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:15.129 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786908+0000", 2024-09-18T04:39:15.129 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8199999999999997e-06 2024-09-18T04:39:15.129 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.130 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.130 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:15.130 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786925+0000", 2024-09-18T04:39:15.130 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6849000000000001e-05 2024-09-18T04:39:15.130 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.130 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.130 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:15.130 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786934+0000", 2024-09-18T04:39:15.130 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6299999999999993e-06 2024-09-18T04:39:15.130 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.130 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.130 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:15.130 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800926+0000", 2024-09-18T04:39:15.130 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013991632 2024-09-18T04:39:15.131 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.131 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.131 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:15.131 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801060+0000", 2024-09-18T04:39:15.131 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013378199999999999 2024-09-18T04:39:15.131 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.131 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.131 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:15.131 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801166+0000", 2024-09-18T04:39:15.131 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010607099999999999 2024-09-18T04:39:15.131 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.131 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.131 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:15.131 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803026+0000", 2024-09-18T04:39:15.132 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001860469 2024-09-18T04:39:15.132 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.132 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.132 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:15.132 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804245+0000", 2024-09-18T04:39:15.132 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0012185900000000001 2024-09-18T04:39:15.132 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.132 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.132 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:15.132 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804278+0000", 2024-09-18T04:39:15.132 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3272000000000003e-05 2024-09-18T04:39:15.132 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.132 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.132 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:15.133 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804300+0000", 2024-09-18T04:39:15.133 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2266e-05 2024-09-18T04:39:15.133 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:15.133 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:15.133 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:15.133 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.133 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.133 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:39:15.133 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:39:15.133 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 107.33373887800001, 2024-09-18T04:39:15.133 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017521662, 2024-09-18T04:39:15.133 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:15.133 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:15.134 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:15.134 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:39:15.134 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:39:15.134 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 116 2024-09-18T04:39:15.134 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.134 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:15.134 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.134 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:15.134 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:39:15.134 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:15.134 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.134 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.134 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:15.134 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:39:15.135 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:15.135 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.135 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.135 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:15.135 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787813+0000", 2024-09-18T04:39:15.135 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:15.135 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.135 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.135 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:15.135 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787824+0000", 2024-09-18T04:39:15.135 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1157e-05 2024-09-18T04:39:15.135 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.135 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.135 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:15.136 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787840+0000", 2024-09-18T04:39:15.136 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6269e-05 2024-09-18T04:39:15.136 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.136 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.136 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:15.136 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787853+0000", 2024-09-18T04:39:15.136 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3156000000000001e-05 2024-09-18T04:39:15.136 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.136 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.136 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:15.136 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803110+0000", 2024-09-18T04:39:15.136 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015256736 2024-09-18T04:39:15.136 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.137 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.137 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:15.137 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803197+0000", 2024-09-18T04:39:15.137 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7195999999999997e-05 2024-09-18T04:39:15.137 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.137 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.137 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:15.137 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803308+0000", 2024-09-18T04:39:15.137 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111005 2024-09-18T04:39:15.137 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.137 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.137 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:15.137 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804316+0000", 2024-09-18T04:39:15.137 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010073949999999999 2024-09-18T04:39:15.138 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.138 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.138 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:15.138 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805232+0000", 2024-09-18T04:39:15.138 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091636300000000001 2024-09-18T04:39:15.138 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.138 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.138 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:15.138 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805277+0000", 2024-09-18T04:39:15.138 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4864000000000003e-05 2024-09-18T04:39:15.138 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.138 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.138 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:15.138 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805336+0000", 2024-09-18T04:39:15.139 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8968000000000002e-05 2024-09-18T04:39:15.139 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:15.139 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:15.139 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:15.139 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.139 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.139 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:39:15.139 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:39:15.139 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 107.33368682299999, 2024-09-18T04:39:15.139 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017983453, 2024-09-18T04:39:15.139 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:15.139 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:15.139 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:15.139 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:39:15.140 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:39:15.140 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 117 2024-09-18T04:39:15.140 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.140 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:15.140 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.140 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:15.140 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:39:15.140 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:15.140 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.140 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.140 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:15.140 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:39:15.140 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:15.140 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.141 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.141 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:15.141 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787865+0000", 2024-09-18T04:39:15.141 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:15.141 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.141 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.141 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:15.141 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787889+0000", 2024-09-18T04:39:15.141 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4244999999999999e-05 2024-09-18T04:39:15.141 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.141 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.141 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:15.141 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787905+0000", 2024-09-18T04:39:15.142 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5739999999999998e-05 2024-09-18T04:39:15.142 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.142 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.142 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:15.142 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787918+0000", 2024-09-18T04:39:15.142 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.252e-05 2024-09-18T04:39:15.142 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.142 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.142 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:15.142 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803693+0000", 2024-09-18T04:39:15.142 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015774872999999998 2024-09-18T04:39:15.142 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.142 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.142 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:15.143 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803775+0000", 2024-09-18T04:39:15.143 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2761000000000003e-05 2024-09-18T04:39:15.143 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.143 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.143 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:15.143 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803887+0000", 2024-09-18T04:39:15.143 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111708 2024-09-18T04:39:15.143 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.143 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.143 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:15.143 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804823+0000", 2024-09-18T04:39:15.143 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00093625100000000003 2024-09-18T04:39:15.143 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.143 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.143 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:15.144 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805804+0000", 2024-09-18T04:39:15.144 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00098114699999999992 2024-09-18T04:39:15.144 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.144 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.144 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:15.144 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805831+0000", 2024-09-18T04:39:15.144 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6401999999999999e-05 2024-09-18T04:39:15.144 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.144 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.144 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:15.144 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805850+0000", 2024-09-18T04:39:15.144 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8862e-05 2024-09-18T04:39:15.144 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:15.145 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:15.145 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:15.145 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.145 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.145 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:36 2.7 2:fe1b02ad:::rbd_data.125fc09df6d1.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:15.145 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:39:15.145 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 32.855830713000003, 2024-09-18T04:39:15.145 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.040387804999999999, 2024-09-18T04:39:15.145 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:15.145 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:15.145 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:15.145 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:15.145 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:15.146 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 36 2024-09-18T04:39:15.146 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.146 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:15.146 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.146 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:15.146 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:39:15.146 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:15.146 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.146 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.146 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:15.146 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:39:15.146 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:15.146 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.146 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.147 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:15.147 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265720+0000", 2024-09-18T04:39:15.147 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:15.147 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.147 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.147 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:15.147 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272590+0000", 2024-09-18T04:39:15.147 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.006870508 2024-09-18T04:39:15.147 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.147 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.147 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:15.147 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272623+0000", 2024-09-18T04:39:15.147 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2716000000000003e-05 2024-09-18T04:39:15.147 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.148 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.148 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:15.148 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272637+0000", 2024-09-18T04:39:15.148 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4175999999999999e-05 2024-09-18T04:39:15.148 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.148 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.148 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:15.148 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272779+0000", 2024-09-18T04:39:15.148 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000141392 2024-09-18T04:39:15.148 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.148 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.148 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:15.148 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273031+0000", 2024-09-18T04:39:15.148 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00025249099999999998 2024-09-18T04:39:15.149 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.149 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.149 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:39:15.149 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273219+0000", 2024-09-18T04:39:15.149 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000187531 2024-09-18T04:39:15.149 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.149 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.149 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:15.149 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.302806+0000", 2024-09-18T04:39:15.149 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.029587553999999999 2024-09-18T04:39:15.149 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.149 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.149 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:15.150 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306004+0000", 2024-09-18T04:39:15.150 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0031982629999999998 2024-09-18T04:39:15.150 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.150 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.150 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:15.150 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306071+0000", 2024-09-18T04:39:15.150 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6829999999999995e-05 2024-09-18T04:39:15.150 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.150 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.150 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:15.150 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306110+0000", 2024-09-18T04:39:15.150 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8874000000000001e-05 2024-09-18T04:39:15.150 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:15.150 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:15.151 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:15.151 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.151 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.151 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:38 2.0 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:15.151 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:39:15.151 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 32.848877176000002, 2024-09-18T04:39:15.151 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.033754348000000003, 2024-09-18T04:39:15.151 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:15.151 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:15.151 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:15.151 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:15.151 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:15.151 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 38 2024-09-18T04:39:15.151 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.152 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:15.152 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.152 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:15.152 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:39:15.152 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:15.152 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.152 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.152 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:15.152 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:39:15.152 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:15.152 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.152 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.152 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:15.152 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272673+0000", 2024-09-18T04:39:15.153 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:15.153 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.153 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.153 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:15.153 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277166+0000", 2024-09-18T04:39:15.153 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044922249999999999 2024-09-18T04:39:15.153 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.153 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.153 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:15.153 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277189+0000", 2024-09-18T04:39:15.153 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3283000000000001e-05 2024-09-18T04:39:15.153 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.153 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.153 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:15.154 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277205+0000", 2024-09-18T04:39:15.154 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6305000000000001e-05 2024-09-18T04:39:15.154 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.154 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.154 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:15.154 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277290+0000", 2024-09-18T04:39:15.154 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5135000000000004e-05 2024-09-18T04:39:15.154 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.154 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.154 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:15.154 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277531+0000", 2024-09-18T04:39:15.154 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00024014699999999999 2024-09-18T04:39:15.154 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.154 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.155 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:15.155 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277793+0000", 2024-09-18T04:39:15.155 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00026224499999999998 2024-09-18T04:39:15.155 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.155 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.155 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:15.155 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303990+0000", 2024-09-18T04:39:15.155 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.026197295999999998 2024-09-18T04:39:15.155 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.155 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.155 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:15.155 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306340+0000", 2024-09-18T04:39:15.155 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.002349524 2024-09-18T04:39:15.155 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.156 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.156 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:15.156 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306394+0000", 2024-09-18T04:39:15.156 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3934999999999998e-05 2024-09-18T04:39:15.156 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.156 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.156 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:15.156 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306430+0000", 2024-09-18T04:39:15.156 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6690000000000003e-05 2024-09-18T04:39:15.156 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:15.156 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:15.156 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:15.156 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.156 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.157 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:39 2.7 2:e9c909c5:::rbd_data.125fc09df6d1.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:15.157 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:39:15.157 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 32.844317570999998, 2024-09-18T04:39:15.157 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.029110673, 2024-09-18T04:39:15.157 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:15.157 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:15.157 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:15.157 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:15.157 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:15.157 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 39 2024-09-18T04:39:15.157 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.157 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:15.157 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.157 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:15.158 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:39:15.158 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:15.158 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.158 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.158 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:15.158 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:39:15.158 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:15.158 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.158 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.158 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:15.158 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277234+0000", 2024-09-18T04:39:15.158 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:15.158 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.158 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.159 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:15.159 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281351+0000", 2024-09-18T04:39:15.159 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0041170920000000001 2024-09-18T04:39:15.159 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.159 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.159 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:15.159 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281375+0000", 2024-09-18T04:39:15.159 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.446e-05 2024-09-18T04:39:15.159 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.159 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.159 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:15.159 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281390+0000", 2024-09-18T04:39:15.159 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4309e-05 2024-09-18T04:39:15.159 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.160 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.160 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:15.160 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.282306+0000", 2024-09-18T04:39:15.160 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091662000000000004 2024-09-18T04:39:15.160 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.160 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.160 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:15.160 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.282613+0000", 2024-09-18T04:39:15.160 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00030663799999999997 2024-09-18T04:39:15.160 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.160 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.160 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:39:15.160 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283001+0000", 2024-09-18T04:39:15.160 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00038801599999999998 2024-09-18T04:39:15.161 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.161 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.161 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:15.161 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306239+0000", 2024-09-18T04:39:15.161 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023238466999999999 2024-09-18T04:39:15.161 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.161 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.161 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:15.161 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306259+0000", 2024-09-18T04:39:15.161 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9837999999999999e-05 2024-09-18T04:39:15.161 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.161 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.161 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:15.161 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306299+0000", 2024-09-18T04:39:15.162 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0194000000000002e-05 2024-09-18T04:39:15.162 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.162 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.162 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:15.162 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306346+0000", 2024-09-18T04:39:15.162 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6845000000000003e-05 2024-09-18T04:39:15.162 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:15.162 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:15.162 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:15.162 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.162 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.162 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:46 2.6 e51/17 2:7ba25245:::rbd_data.125fc09df6d1.0000000000000006:head v 51'2, mlcod=0'0)", 2024-09-18T04:39:15.162 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:39:15.162 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 32.808396438999999, 2024-09-18T04:39:15.163 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019843098, 2024-09-18T04:39:15.163 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:15.163 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:15.163 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:15.163 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.163 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:15.163 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:39:15.163 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:15.163 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.163 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.163 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:15.163 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:39:15.163 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:15.163 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.164 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.164 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:15.164 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313156+0000", 2024-09-18T04:39:15.164 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:15.164 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.164 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.164 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:15.164 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320144+0000", 2024-09-18T04:39:15.164 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.006988345 2024-09-18T04:39:15.164 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.164 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.164 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:15.164 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320175+0000", 2024-09-18T04:39:15.164 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.0861999999999997e-05 2024-09-18T04:39:15.165 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.165 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.165 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:15.165 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320188+0000", 2024-09-18T04:39:15.165 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2714999999999999e-05 2024-09-18T04:39:15.165 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.165 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.165 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:15.165 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320267+0000", 2024-09-18T04:39:15.165 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9586999999999995e-05 2024-09-18T04:39:15.165 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.165 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.165 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:15.165 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320311+0000", 2024-09-18T04:39:15.166 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3529000000000003e-05 2024-09-18T04:39:15.166 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.166 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.166 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:15.166 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.332949+0000", 2024-09-18T04:39:15.166 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012638267999999999 2024-09-18T04:39:15.166 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.166 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.166 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:15.166 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.333000+0000", 2024-09-18T04:39:15.166 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0736999999999999e-05 2024-09-18T04:39:15.166 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:15.166 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:15.166 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:15.166 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.167 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.167 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:57 2.0 2:0cdd070e:::rbd_data.125fc09df6d1.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:15.167 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:39:15.167 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 32.772538863999998, 2024-09-18T04:39:15.167 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.033329974999999998, 2024-09-18T04:39:15.167 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:15.167 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:15.167 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:15.167 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:15.167 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:15.167 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 57 2024-09-18T04:39:15.167 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.167 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:15.167 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.168 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:15.168 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:39:15.168 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:15.168 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.168 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.168 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:15.168 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:39:15.168 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:15.168 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.168 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.168 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:15.168 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349013+0000", 2024-09-18T04:39:15.168 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:15.168 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.169 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.169 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:15.169 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356138+0000", 2024-09-18T04:39:15.169 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071258010000000002 2024-09-18T04:39:15.169 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.169 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.169 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:15.169 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356168+0000", 2024-09-18T04:39:15.169 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9679000000000002e-05 2024-09-18T04:39:15.169 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.169 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.169 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:15.169 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356181+0000", 2024-09-18T04:39:15.169 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3164e-05 2024-09-18T04:39:15.170 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.170 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.170 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:15.170 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356254+0000", 2024-09-18T04:39:15.170 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3182999999999994e-05 2024-09-18T04:39:15.170 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.170 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.170 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:15.170 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356453+0000", 2024-09-18T04:39:15.170 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019876599999999999 2024-09-18T04:39:15.170 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.170 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.170 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:15.170 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356627+0000", 2024-09-18T04:39:15.171 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017340199999999999 2024-09-18T04:39:15.171 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.171 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.171 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:15.171 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.369200+0000", 2024-09-18T04:39:15.171 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012573302 2024-09-18T04:39:15.171 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.171 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.171 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:15.171 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382262+0000", 2024-09-18T04:39:15.171 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013062549 2024-09-18T04:39:15.171 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.171 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.171 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:15.171 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382310+0000", 2024-09-18T04:39:15.172 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7366e-05 2024-09-18T04:39:15.172 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.172 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.172 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:15.172 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382344+0000", 2024-09-18T04:39:15.172 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4319999999999997e-05 2024-09-18T04:39:15.172 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:15.172 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:15.172 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:15.172 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.172 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.172 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:59 2.0 2:19c52cde:::rbd_data.125fc09df6d1.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:15.172 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:39:15.172 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 32.765347699000003, 2024-09-18T04:39:15.173 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.027137247999999999, 2024-09-18T04:39:15.173 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:15.173 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:15.173 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:15.173 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:15.173 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:15.173 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 59 2024-09-18T04:39:15.173 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.173 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:15.173 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.173 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:15.173 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:39:15.173 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:15.173 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.174 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.174 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:15.174 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:39:15.174 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:15.174 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.174 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.174 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:15.174 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356204+0000", 2024-09-18T04:39:15.174 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:15.174 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.174 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.174 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:15.174 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362661+0000", 2024-09-18T04:39:15.174 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0064565129999999997 2024-09-18T04:39:15.174 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.175 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.175 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:15.175 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362684+0000", 2024-09-18T04:39:15.175 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3688000000000001e-05 2024-09-18T04:39:15.175 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.175 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.175 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:15.175 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362699+0000", 2024-09-18T04:39:15.175 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4798e-05 2024-09-18T04:39:15.175 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.175 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.175 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:15.175 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363016+0000", 2024-09-18T04:39:15.175 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031737300000000003 2024-09-18T04:39:15.175 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.175 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.176 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:15.176 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363201+0000", 2024-09-18T04:39:15.176 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000184131 2024-09-18T04:39:15.176 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.176 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.176 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:15.176 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363343+0000", 2024-09-18T04:39:15.176 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014236699999999999 2024-09-18T04:39:15.176 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.176 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.176 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:15.176 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.372189+0000", 2024-09-18T04:39:15.176 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0088464819999999993 2024-09-18T04:39:15.176 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.177 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.177 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:15.177 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383269+0000", 2024-09-18T04:39:15.177 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011079594 2024-09-18T04:39:15.177 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.177 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.177 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:15.177 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383314+0000", 2024-09-18T04:39:15.177 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4932000000000003e-05 2024-09-18T04:39:15.177 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.177 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.177 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:15.177 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383343+0000", 2024-09-18T04:39:15.177 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8600999999999999e-05 2024-09-18T04:39:15.178 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:15.178 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:15.178 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:15.178 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.178 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.178 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:40 2.6 e51/17 2:784d6cd1:ns::rbd_data.133e813ab5bc.0000000000000005:head v 51'7, mlcod=51'5)", 2024-09-18T04:39:15.178 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:39:15.178 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 32.569097964999997, 2024-09-18T04:39:15.178 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020497496, 2024-09-18T04:39:15.178 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:15.178 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:15.178 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:15.178 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.178 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:15.178 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:39:15.179 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:15.179 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.179 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.179 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:15.179 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:39:15.179 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:15.179 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.179 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.179 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:15.179 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552454+0000", 2024-09-18T04:39:15.179 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:15.179 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.179 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.179 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:15.179 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556448+0000", 2024-09-18T04:39:15.180 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0039940990000000001 2024-09-18T04:39:15.180 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.180 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.180 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:15.180 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556481+0000", 2024-09-18T04:39:15.180 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3778000000000001e-05 2024-09-18T04:39:15.180 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.180 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.180 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:15.180 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556493+0000", 2024-09-18T04:39:15.180 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1622e-05 2024-09-18T04:39:15.180 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.180 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.181 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:15.181 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556556+0000", 2024-09-18T04:39:15.181 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2582999999999994e-05 2024-09-18T04:39:15.181 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.181 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.181 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:15.181 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556592+0000", 2024-09-18T04:39:15.181 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5880999999999997e-05 2024-09-18T04:39:15.181 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.181 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.181 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:15.181 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.572786+0000", 2024-09-18T04:39:15.181 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016194571000000001 2024-09-18T04:39:15.182 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.182 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.182 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:15.182 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.572953+0000", 2024-09-18T04:39:15.182 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000166436 2024-09-18T04:39:15.182 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:15.182 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:15.182 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:15.182 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.182 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.182 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:45 2.0 2:0fbfa3dc:ns::rbd_data.133e813ab5bc.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:15.182 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:39:15.182 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 32.557883666000002, 2024-09-18T04:39:15.182 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.036558168000000002, 2024-09-18T04:39:15.183 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:15.183 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:15.183 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:15.183 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:15.183 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:15.183 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 45 2024-09-18T04:39:15.183 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.183 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:15.183 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.183 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:15.183 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:39:15.183 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:15.183 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.183 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.184 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:15.184 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:39:15.184 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:15.184 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.184 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.184 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:15.184 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:39:15.184 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:15.184 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.184 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.184 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:15.184 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578594+0000", 2024-09-18T04:39:15.184 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014927147999999999 2024-09-18T04:39:15.184 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.185 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.185 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:15.185 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578612+0000", 2024-09-18T04:39:15.185 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7623999999999999e-05 2024-09-18T04:39:15.185 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.185 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.185 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:15.185 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578620+0000", 2024-09-18T04:39:15.185 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9070000000000007e-06 2024-09-18T04:39:15.185 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.185 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.185 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:15.185 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578702+0000", 2024-09-18T04:39:15.185 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2349999999999996e-05 2024-09-18T04:39:15.186 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.186 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.186 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:15.186 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.579028+0000", 2024-09-18T04:39:15.186 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032622899999999998 2024-09-18T04:39:15.186 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.186 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.186 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:15.186 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.579343+0000", 2024-09-18T04:39:15.186 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031448899999999999 2024-09-18T04:39:15.186 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.186 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.186 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:15.186 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.590078+0000", 2024-09-18T04:39:15.187 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010734946 2024-09-18T04:39:15.187 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.187 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.187 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:15.187 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600089+0000", 2024-09-18T04:39:15.187 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010011195000000001 2024-09-18T04:39:15.187 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.187 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.187 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:15.187 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600177+0000", 2024-09-18T04:39:15.187 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8425999999999997e-05 2024-09-18T04:39:15.187 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.187 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.188 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:15.188 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600228+0000", 2024-09-18T04:39:15.188 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0216999999999997e-05 2024-09-18T04:39:15.188 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:15.188 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:15.188 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:15.188 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.188 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.188 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:54 2.0 2:04f17ef6:ns::rbd_data.133e813ab5bc.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:15.188 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:39:15.188 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 32.536311828999999, 2024-09-18T04:39:15.188 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.024612999, 2024-09-18T04:39:15.188 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:15.188 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:15.189 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:15.189 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:15.189 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:15.189 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 54 2024-09-18T04:39:15.189 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.189 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:15.189 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.189 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:15.189 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:39:15.189 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:15.189 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.189 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.189 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:15.190 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:39:15.190 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:15.190 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.190 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.190 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:15.190 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585239+0000", 2024-09-18T04:39:15.190 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:15.190 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.190 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.190 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:15.190 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593702+0000", 2024-09-18T04:39:15.190 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0084632990000000005 2024-09-18T04:39:15.190 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.190 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.191 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:15.191 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593719+0000", 2024-09-18T04:39:15.191 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.664e-05 2024-09-18T04:39:15.191 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.191 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.191 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:15.191 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593727+0000", 2024-09-18T04:39:15.191 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6839999999999996e-06 2024-09-18T04:39:15.191 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.191 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.191 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:15.191 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593767+0000", 2024-09-18T04:39:15.191 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0330999999999997e-05 2024-09-18T04:39:15.191 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.191 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.192 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:15.192 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593905+0000", 2024-09-18T04:39:15.192 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013754800000000001 2024-09-18T04:39:15.192 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.192 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.192 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:15.192 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.594013+0000", 2024-09-18T04:39:15.192 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000108805 2024-09-18T04:39:15.192 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.192 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.192 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:15.192 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604678+0000", 2024-09-18T04:39:15.192 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010664391 2024-09-18T04:39:15.193 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.193 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.193 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:15.193 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609662+0000", 2024-09-18T04:39:15.193 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0049846100000000004 2024-09-18T04:39:15.193 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.193 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.193 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:15.193 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609788+0000", 2024-09-18T04:39:15.193 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012518800000000001 2024-09-18T04:39:15.193 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.193 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.193 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:15.193 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609854+0000", 2024-09-18T04:39:15.194 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6556000000000004e-05 2024-09-18T04:39:15.194 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:15.194 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:15.194 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:15.194 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.194 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.194 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:58 2.7 2:fc7c6cd3:ns::rbd_data.133e813ab5bc.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:15.194 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:39:15.194 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 32.523469400000003, 2024-09-18T04:39:15.194 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.027690566999999999, 2024-09-18T04:39:15.194 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:15.194 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:15.194 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:15.194 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:15.195 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:15.195 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 58 2024-09-18T04:39:15.195 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.195 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:15.195 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.195 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:15.195 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:39:15.195 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:15.195 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.195 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.195 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:15.195 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:39:15.195 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:15.195 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.195 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.196 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:15.196 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598082+0000", 2024-09-18T04:39:15.196 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:15.196 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.196 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.196 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:15.196 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604832+0000", 2024-09-18T04:39:15.196 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0067504510000000002 2024-09-18T04:39:15.196 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.196 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.196 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:15.196 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604852+0000", 2024-09-18T04:39:15.196 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0228999999999999e-05 2024-09-18T04:39:15.196 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.196 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.197 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:15.197 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604870+0000", 2024-09-18T04:39:15.197 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7952e-05 2024-09-18T04:39:15.197 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.197 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.197 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:15.197 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604936+0000", 2024-09-18T04:39:15.197 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5431000000000004e-05 2024-09-18T04:39:15.197 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.197 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.197 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:15.197 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605085+0000", 2024-09-18T04:39:15.197 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014910300000000001 2024-09-18T04:39:15.197 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.198 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.198 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:39:15.198 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605205+0000", 2024-09-18T04:39:15.198 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000119887 2024-09-18T04:39:15.198 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.198 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.198 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:15.198 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625586+0000", 2024-09-18T04:39:15.198 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020381399000000001 2024-09-18T04:39:15.198 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.198 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.198 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:15.198 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625628+0000", 2024-09-18T04:39:15.198 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1873999999999999e-05 2024-09-18T04:39:15.199 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.199 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.199 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:15.199 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625722+0000", 2024-09-18T04:39:15.199 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3787000000000002e-05 2024-09-18T04:39:15.199 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.199 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.199 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:15.199 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625774+0000", 2024-09-18T04:39:15.199 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2503999999999997e-05 2024-09-18T04:39:15.199 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:15.199 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:15.199 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:15.199 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.199 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.200 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:56 2.6 e51/17 2:72453c3e:ns::rbd_data.133e813ab5bc.000000000000000d:head v 51'10, mlcod=51'8)", 2024-09-18T04:39:15.200 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:39:15.200 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 32.516272291999996, 2024-09-18T04:39:15.200 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019766552999999999, 2024-09-18T04:39:15.200 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:15.200 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:15.200 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:15.200 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.200 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:15.200 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:39:15.200 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:15.200 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.200 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.201 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:15.201 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:39:15.201 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:15.201 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.201 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.201 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:15.201 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605279+0000", 2024-09-18T04:39:15.201 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:15.201 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.201 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.201 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:15.201 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609417+0000", 2024-09-18T04:39:15.201 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0041378059999999999 2024-09-18T04:39:15.201 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.202 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.202 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:15.202 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609443+0000", 2024-09-18T04:39:15.202 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6098e-05 2024-09-18T04:39:15.202 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.202 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.202 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:15.202 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609458+0000", 2024-09-18T04:39:15.202 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4671000000000001e-05 2024-09-18T04:39:15.202 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.202 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.202 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:15.202 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609553+0000", 2024-09-18T04:39:15.203 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5265000000000004e-05 2024-09-18T04:39:15.203 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.203 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.203 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:15.203 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609583+0000", 2024-09-18T04:39:15.203 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9839e-05 2024-09-18T04:39:15.203 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.203 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.203 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:15.203 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625004+0000", 2024-09-18T04:39:15.203 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015420639999999999 2024-09-18T04:39:15.203 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.203 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.203 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:15.204 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625047+0000", 2024-09-18T04:39:15.204 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3665999999999998e-05 2024-09-18T04:39:15.204 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:15.204 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:15.204 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:15.204 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.204 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.204 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:99 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:15.204 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.582718+0000", 2024-09-18T04:39:15.204 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 18.538835382999999, 2024-09-18T04:39:15.204 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019362575, 2024-09-18T04:39:15.204 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:15.204 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:15.204 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:15.204 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:15.205 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:15.205 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 99 2024-09-18T04:39:15.205 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.205 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:15.205 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.205 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:15.205 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582718+0000", 2024-09-18T04:39:15.205 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:15.205 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.205 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.205 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:15.205 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582718+0000", 2024-09-18T04:39:15.205 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:15.206 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.206 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.206 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:15.206 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582717+0000", 2024-09-18T04:39:15.206 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:15.206 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.206 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.206 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:15.206 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582720+0000", 2024-09-18T04:39:15.206 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2239999999999998e-06 2024-09-18T04:39:15.206 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.206 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.206 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:15.206 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582723+0000", 2024-09-18T04:39:15.207 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.258e-06 2024-09-18T04:39:15.207 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.207 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.207 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:15.207 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582726+0000", 2024-09-18T04:39:15.207 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8870000000000001e-06 2024-09-18T04:39:15.207 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.207 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.207 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:15.207 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.592018+0000", 2024-09-18T04:39:15.207 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0092926420000000003 2024-09-18T04:39:15.207 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.207 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.207 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:15.208 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.592214+0000", 2024-09-18T04:39:15.208 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000195267 2024-09-18T04:39:15.208 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.208 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.208 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:15.208 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.592913+0000", 2024-09-18T04:39:15.208 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00069924600000000005 2024-09-18T04:39:15.208 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.208 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.208 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:15.208 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597493+0000", 2024-09-18T04:39:15.208 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045797629999999997 2024-09-18T04:39:15.208 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.209 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.209 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:15.209 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.602009+0000", 2024-09-18T04:39:15.209 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045159559999999998 2024-09-18T04:39:15.209 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.209 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.209 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:15.209 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.602042+0000", 2024-09-18T04:39:15.209 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3084999999999999e-05 2024-09-18T04:39:15.209 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.209 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.209 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:15.209 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.602080+0000", 2024-09-18T04:39:15.210 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8600000000000003e-05 2024-09-18T04:39:15.210 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:15.210 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:15.210 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:15.210 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.210 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.210 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:104 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:15.210 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.583165+0000", 2024-09-18T04:39:15.210 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 18.538388085000001, 2024-09-18T04:39:15.210 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023679603, 2024-09-18T04:39:15.210 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:15.210 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:15.210 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:15.210 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:15.211 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:15.211 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 104 2024-09-18T04:39:15.211 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.211 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:15.211 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.211 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:15.211 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583165+0000", 2024-09-18T04:39:15.211 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:15.211 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.211 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.211 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:15.211 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583165+0000", 2024-09-18T04:39:15.211 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:15.211 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.212 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.212 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:15.212 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583164+0000", 2024-09-18T04:39:15.212 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:15.212 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.212 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.212 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:15.212 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583167+0000", 2024-09-18T04:39:15.212 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9210000000000001e-06 2024-09-18T04:39:15.212 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.212 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.212 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:15.212 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583171+0000", 2024-09-18T04:39:15.212 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9759999999999997e-06 2024-09-18T04:39:15.213 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.213 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.213 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:15.213 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583174+0000", 2024-09-18T04:39:15.213 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9960000000000002e-06 2024-09-18T04:39:15.213 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.213 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.213 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:15.213 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.594373+0000", 2024-09-18T04:39:15.213 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011198665999999999 2024-09-18T04:39:15.213 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.213 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.213 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:15.213 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.594434+0000", 2024-09-18T04:39:15.214 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.1446999999999996e-05 2024-09-18T04:39:15.214 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.214 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.214 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:15.214 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.594692+0000", 2024-09-18T04:39:15.214 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00025756599999999998 2024-09-18T04:39:15.214 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.214 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.214 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:15.214 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599302+0000", 2024-09-18T04:39:15.214 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0046099640000000002 2024-09-18T04:39:15.214 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.214 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.214 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:15.214 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606710+0000", 2024-09-18T04:39:15.215 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0074077739999999998 2024-09-18T04:39:15.215 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.215 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.215 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:15.215 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606798+0000", 2024-09-18T04:39:15.215 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8133999999999993e-05 2024-09-18T04:39:15.215 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.215 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.215 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:15.215 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606845+0000", 2024-09-18T04:39:15.215 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6653000000000002e-05 2024-09-18T04:39:15.215 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:15.215 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:15.215 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:15.216 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.216 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.216 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:105 2.0 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:15.216 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.583193+0000", 2024-09-18T04:39:15.216 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 18.538360466, 2024-09-18T04:39:15.216 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023713649, 2024-09-18T04:39:15.216 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:15.216 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:15.216 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:15.216 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:15.216 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:15.216 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 105 2024-09-18T04:39:15.216 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.216 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:15.216 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.217 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:15.217 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583193+0000", 2024-09-18T04:39:15.217 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:15.217 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.217 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.217 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:15.217 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583193+0000", 2024-09-18T04:39:15.217 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:15.217 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.217 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.217 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:15.217 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583192+0000", 2024-09-18T04:39:15.217 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:15.218 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.218 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.218 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:15.218 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583194+0000", 2024-09-18T04:39:15.218 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.305e-06 2024-09-18T04:39:15.218 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.218 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.218 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:15.218 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583198+0000", 2024-09-18T04:39:15.218 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9369999999999997e-06 2024-09-18T04:39:15.218 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.218 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.218 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:15.218 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583202+0000", 2024-09-18T04:39:15.219 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6480000000000001e-06 2024-09-18T04:39:15.219 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.219 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.219 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:15.219 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.595304+0000", 2024-09-18T04:39:15.219 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012102063999999999 2024-09-18T04:39:15.219 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.219 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.219 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:15.219 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.595390+0000", 2024-09-18T04:39:15.219 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5493000000000003e-05 2024-09-18T04:39:15.219 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.219 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.219 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:15.219 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.595509+0000", 2024-09-18T04:39:15.220 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011966199999999999 2024-09-18T04:39:15.220 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.220 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.220 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:15.220 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599358+0000", 2024-09-18T04:39:15.220 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0038486689999999999 2024-09-18T04:39:15.220 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.220 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.220 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:15.220 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606859+0000", 2024-09-18T04:39:15.220 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0075010789999999999 2024-09-18T04:39:15.220 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.220 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.221 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:15.221 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606887+0000", 2024-09-18T04:39:15.221 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7773e-05 2024-09-18T04:39:15.221 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.221 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.221 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:15.221 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606906+0000", 2024-09-18T04:39:15.221 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9417000000000001e-05 2024-09-18T04:39:15.221 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:15.221 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:15.221 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:15.221 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.221 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.222 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:106 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:15.222 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.584425+0000", 2024-09-18T04:39:15.222 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 18.537128169999999, 2024-09-18T04:39:15.222 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.022529293999999998, 2024-09-18T04:39:15.222 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:15.222 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:15.222 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:15.222 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:15.222 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:15.222 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 106 2024-09-18T04:39:15.222 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.222 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:15.222 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.222 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:15.222 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584425+0000", 2024-09-18T04:39:15.223 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:15.223 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.223 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.223 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:15.223 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584425+0000", 2024-09-18T04:39:15.223 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:15.223 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.223 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.223 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:15.223 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584424+0000", 2024-09-18T04:39:15.223 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:15.223 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.223 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.223 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:15.224 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584430+0000", 2024-09-18T04:39:15.224 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6389999999999997e-06 2024-09-18T04:39:15.224 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.224 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.224 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:15.224 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584456+0000", 2024-09-18T04:39:15.224 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6188999999999999e-05 2024-09-18T04:39:15.224 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.224 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.224 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:15.224 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584463+0000", 2024-09-18T04:39:15.224 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7340000000000002e-06 2024-09-18T04:39:15.224 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.224 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.225 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:15.225 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597537+0000", 2024-09-18T04:39:15.225 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013074161000000001 2024-09-18T04:39:15.225 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.225 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.225 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:15.225 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597598+0000", 2024-09-18T04:39:15.225 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0930000000000001e-05 2024-09-18T04:39:15.225 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.225 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.225 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:15.225 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597922+0000", 2024-09-18T04:39:15.225 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032473000000000002 2024-09-18T04:39:15.226 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.226 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.226 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:15.226 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.601103+0000", 2024-09-18T04:39:15.226 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0031809849999999999 2024-09-18T04:39:15.226 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.226 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.226 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:15.226 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606912+0000", 2024-09-18T04:39:15.226 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058083550000000003 2024-09-18T04:39:15.226 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.226 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.226 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:15.226 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606935+0000", 2024-09-18T04:39:15.226 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2759999999999999e-05 2024-09-18T04:39:15.227 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.227 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.227 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:15.227 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606954+0000", 2024-09-18T04:39:15.227 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9602999999999998e-05 2024-09-18T04:39:15.227 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:15.227 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:15.227 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:15.227 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.227 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.227 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:109 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:15.227 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.585231+0000", 2024-09-18T04:39:15.227 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 18.53632155, 2024-09-18T04:39:15.227 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.021767496000000001, 2024-09-18T04:39:15.228 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:15.228 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:15.228 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:15.228 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:15.228 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:15.228 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 109 2024-09-18T04:39:15.228 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.228 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:15.228 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.228 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:15.228 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585231+0000", 2024-09-18T04:39:15.228 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:15.228 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.228 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.229 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:15.229 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585231+0000", 2024-09-18T04:39:15.229 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:15.229 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.229 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.229 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:15.229 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585230+0000", 2024-09-18T04:39:15.229 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:15.229 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.229 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.229 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:15.229 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585237+0000", 2024-09-18T04:39:15.229 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.618e-06 2024-09-18T04:39:15.229 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.229 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.230 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:15.230 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585256+0000", 2024-09-18T04:39:15.230 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9048000000000001e-05 2024-09-18T04:39:15.230 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.230 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.230 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:15.230 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585274+0000", 2024-09-18T04:39:15.230 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8153000000000001e-05 2024-09-18T04:39:15.230 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.230 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.230 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:15.230 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.598284+0000", 2024-09-18T04:39:15.230 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013009682999999999 2024-09-18T04:39:15.230 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.231 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.231 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:15.231 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.598383+0000", 2024-09-18T04:39:15.231 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.8722999999999997e-05 2024-09-18T04:39:15.231 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.231 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.231 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:15.231 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.598829+0000", 2024-09-18T04:39:15.231 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00044623999999999998 2024-09-18T04:39:15.231 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.231 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.231 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:15.231 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.601149+0000", 2024-09-18T04:39:15.231 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0023204150000000002 2024-09-18T04:39:15.232 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.232 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.232 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:15.232 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606960+0000", 2024-09-18T04:39:15.232 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058106390000000003 2024-09-18T04:39:15.232 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.232 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.232 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:15.232 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606982+0000", 2024-09-18T04:39:15.232 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2252e-05 2024-09-18T04:39:15.232 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.232 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.232 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:15.232 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606999+0000", 2024-09-18T04:39:15.233 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6871000000000002e-05 2024-09-18T04:39:15.233 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:15.233 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:15.233 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:15.233 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.233 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.233 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:110 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:15.233 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.586640+0000", 2024-09-18T04:39:15.233 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 18.534913231000001, 2024-09-18T04:39:15.233 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020400911000000001, 2024-09-18T04:39:15.233 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:15.233 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:15.233 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:15.233 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:15.234 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:15.234 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 110 2024-09-18T04:39:15.234 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.234 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:15.234 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.234 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:15.234 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586640+0000", 2024-09-18T04:39:15.234 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:15.234 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.234 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.234 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:15.234 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586640+0000", 2024-09-18T04:39:15.234 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:15.234 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.235 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.235 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:15.235 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586639+0000", 2024-09-18T04:39:15.235 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:15.235 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.235 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.235 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:15.235 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586652+0000", 2024-09-18T04:39:15.235 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2482e-05 2024-09-18T04:39:15.235 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.235 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.235 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:15.235 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586660+0000", 2024-09-18T04:39:15.235 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1659999999999994e-06 2024-09-18T04:39:15.236 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.236 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.236 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:15.236 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586666+0000", 2024-09-18T04:39:15.236 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6900000000000003e-06 2024-09-18T04:39:15.236 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.236 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.236 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:15.236 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599381+0000", 2024-09-18T04:39:15.236 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012714328 2024-09-18T04:39:15.236 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.236 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.236 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:15.237 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599436+0000", 2024-09-18T04:39:15.237 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4802999999999997e-05 2024-09-18T04:39:15.237 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.237 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.237 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:15.237 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599629+0000", 2024-09-18T04:39:15.237 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000193921 2024-09-18T04:39:15.237 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.237 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.237 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:15.237 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.601174+0000", 2024-09-18T04:39:15.237 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001544857 2024-09-18T04:39:15.237 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.237 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.238 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:15.238 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.607004+0000", 2024-09-18T04:39:15.238 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058300820000000003 2024-09-18T04:39:15.238 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.238 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.238 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:15.238 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.607024+0000", 2024-09-18T04:39:15.238 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9888e-05 2024-09-18T04:39:15.238 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.238 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.238 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:15.238 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.607041+0000", 2024-09-18T04:39:15.238 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6450999999999999e-05 2024-09-18T04:39:15.238 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:15.238 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:15.239 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:15.239 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:15.239 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:15.239 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:15.240 INFO:tasks.workunit.client.0.smithi115.stderr:+ PID=19491 2024-09-18T04:39:15.240 INFO:tasks.workunit.client.0.smithi115.stderr:+ test -n 19491 2024-09-18T04:39:15.240 INFO:tasks.workunit.client.0.smithi115.stderr:+ ps -p 19491 -C rbd-nbd 2024-09-18T04:39:15.241 INFO:tasks.workunit.client.0.smithi115.stdout: PID TTY TIME CMD 2024-09-18T04:39:15.241 INFO:tasks.workunit.client.0.smithi115.stdout: 19491 ? 00:00:00 rbd-nbd 2024-09-18T04:39:15.241 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:39:15.242 INFO:tasks.workunit.client.0.smithi115.stderr:+ test '!' -e /tmp/tmp.WJMNJxIdAN/quiesce.sh 2024-09-18T04:39:15.242 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false rbd snap create rbd/testrbdnbd17342@quiesce1 2024-09-18T04:39:15.243 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd snap create rbd/testrbdnbd17342@quiesce1 2024-09-18T04:39:15.243 INFO:tasks.workunit.client.0.smithi115.stderr: Creating snap: 10% complete...2024-09-18T04:39:15.223+0000 7fb33d609700 -1 librbd::ImageWatcher: 0x7fb3240076e0 failed to notify quiesce: (1) Operation not permitted 2024-09-18T04:39:15.243 INFO:tasks.workunit.client.0.smithi115.stderr:2024-09-18T04:39:15.223+0000 7fb33d609700 -1 librbd::SnapshotCreateRequest: failed to notify quiesce: (1) Operation not permitted 2024-09-18T04:39:15.243 INFO:tasks.workunit.client.0.smithi115.stderr: Creating snap: 10% complete...failed. 2024-09-18T04:39:15.243 INFO:tasks.workunit.client.0.smithi115.stderr:rbd: failed to create snapshot: (1) Operation not permitted 2024-09-18T04:39:15.243 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:15.233+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:39:15.249 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:39:15.249 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo dd if=/tmp/tmp.WJMNJxIdAN/data of=/dev/nbd0 bs=1M count=1 oflag=direct 2024-09-18T04:39:15.249 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:39:15.249 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:39:15.251 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:39:15.252 INFO:tasks.workunit.client.0.smithi115.stderr:++ which dd 2024-09-18T04:39:15.257 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/bin/dd 2024-09-18T04:39:15.257 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:39:15.257 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.WJMNJxIdAN/data of=/dev/nbd0 bs=1M count=1 oflag=direct 2024-09-18T04:39:15.296 INFO:tasks.workunit.client.0.smithi115.stderr:1+0 records in 2024-09-18T04:39:15.297 INFO:tasks.workunit.client.0.smithi115.stderr:1+0 records out 2024-09-18T04:39:15.297 INFO:tasks.workunit.client.0.smithi115.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323473 s, 32.4 MB/s 2024-09-18T04:39:15.298 INFO:tasks.workunit.client.0.smithi115.stderr:+ touch /tmp/tmp.WJMNJxIdAN/quiesce.sh 2024-09-18T04:39:15.300 INFO:tasks.workunit.client.0.smithi115.stderr:+ chmod +x /tmp/tmp.WJMNJxIdAN/quiesce.sh 2024-09-18T04:39:15.301 INFO:tasks.workunit.client.0.smithi115.stderr:+ cat 2024-09-18T04:39:15.304 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd snap create rbd/testrbdnbd17342@quiesce1 2024-09-18T04:39:15.341 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:15.333+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:15.361 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:15.361 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:15.361 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:39:15.361 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:15.371 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:39:15.442 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:15.437+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:15.518 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T04:39:15.523 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:15.523 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:15.523 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:39:15.523 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:39:15.523 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:15.533 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:39:15.555 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-09-18T04:39:15.556 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:15.549+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:15.563 INFO:tasks.workunit.client.0.smithi115.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-18T04:39:15.572 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo dd if=/tmp/tmp.WJMNJxIdAN/data of=/dev/nbd0 bs=1M count=1 oflag=direct 2024-09-18T04:39:15.572 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:39:15.573 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:39:15.575 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:39:15.575 INFO:tasks.workunit.client.0.smithi115.stderr:++ which dd 2024-09-18T04:39:15.577 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/bin/dd 2024-09-18T04:39:15.577 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:39:15.578 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.WJMNJxIdAN/data of=/dev/nbd0 bs=1M count=1 oflag=direct 2024-09-18T04:39:15.609 INFO:tasks.workunit.client.0.smithi115.stderr:1+0 records in 2024-09-18T04:39:15.609 INFO:tasks.workunit.client.0.smithi115.stderr:1+0 records out 2024-09-18T04:39:15.609 INFO:tasks.workunit.client.0.smithi115.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222345 s, 47.2 MB/s 2024-09-18T04:39:15.611 INFO:tasks.workunit.client.0.smithi115.stderr:++ cat /tmp/tmp.WJMNJxIdAN/quiesce 2024-09-18T04:39:15.613 INFO:tasks.workunit.client.0.smithi115.stderr:+ test /dev/nbd0 = /dev/nbd0 2024-09-18T04:39:15.613 INFO:tasks.workunit.client.0.smithi115.stderr:++ cat /tmp/tmp.WJMNJxIdAN/unquiesce 2024-09-18T04:39:15.615 INFO:tasks.workunit.client.0.smithi115.stderr:+ test /dev/nbd0 = /dev/nbd0 2024-09-18T04:39:15.615 INFO:tasks.workunit.client.0.smithi115.stderr:+ touch /tmp/tmp.WJMNJxIdAN/quiesce.sh 2024-09-18T04:39:15.617 INFO:tasks.workunit.client.0.smithi115.stderr:+ chmod +x /tmp/tmp.WJMNJxIdAN/quiesce.sh 2024-09-18T04:39:15.618 INFO:tasks.workunit.client.0.smithi115.stderr:+ cat 2024-09-18T04:39:15.620 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false rbd snap create rbd/testrbdnbd17342@quiesce2 2024-09-18T04:39:15.621 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd snap create rbd/testrbdnbd17342@quiesce2 2024-09-18T04:39:15.640 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:15.640 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:39:15.640 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:39:15.640 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:39:15.640 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.640 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.5431.0:1 3.3 3:d80af3cb:::rbd_id.testrbdnbd17342:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-18T04:39:15.640 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:13.542818+0000", 2024-09-18T04:39:15.640 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 2.0950242939999999, 2024-09-18T04:39:15.640 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048493400000000002, 2024-09-18T04:39:15.640 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:15.641 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:39:15.641 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:15.641 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.5431", 2024-09-18T04:39:15.641 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3147136083", 2024-09-18T04:39:15.641 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 1 2024-09-18T04:39:15.641 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.641 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:15.641 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.641 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:15.641 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542818+0000", 2024-09-18T04:39:15.641 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:15.641 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.641 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.641 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:15.642 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542818+0000", 2024-09-18T04:39:15.642 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:15.642 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.642 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.642 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:15.642 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542816+0000", 2024-09-18T04:39:15.642 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:15.642 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.642 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.642 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:15.642 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542828+0000", 2024-09-18T04:39:15.642 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2699e-05 2024-09-18T04:39:15.642 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.642 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.643 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:15.643 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542852+0000", 2024-09-18T04:39:15.643 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3176e-05 2024-09-18T04:39:15.643 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.643 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.643 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:15.643 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542874+0000", 2024-09-18T04:39:15.643 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1967000000000002e-05 2024-09-18T04:39:15.643 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.643 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.643 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:15.643 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542984+0000", 2024-09-18T04:39:15.643 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011026099999999999 2024-09-18T04:39:15.643 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.643 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.644 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:15.644 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543161+0000", 2024-09-18T04:39:15.644 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017720299999999999 2024-09-18T04:39:15.644 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.644 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.644 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:15.644 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543303+0000", 2024-09-18T04:39:15.644 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014174700000000001 2024-09-18T04:39:15.644 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:15.644 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:15.644 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:15.644 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.644 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.645 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.5431.0:2 3.3 3:d80af3cb:::rbd_id.testrbdnbd17342:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-18T04:39:15.645 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:13.543564+0000", 2024-09-18T04:39:15.645 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 2.0942778080000002, 2024-09-18T04:39:15.645 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010399929999999999, 2024-09-18T04:39:15.645 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:15.645 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:39:15.645 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:15.645 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.5431", 2024-09-18T04:39:15.645 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3147136083", 2024-09-18T04:39:15.645 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 2 2024-09-18T04:39:15.645 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.645 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:15.645 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.645 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:15.646 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543564+0000", 2024-09-18T04:39:15.646 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:15.646 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.646 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.646 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:15.646 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543564+0000", 2024-09-18T04:39:15.646 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:15.646 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.646 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.646 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:15.646 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543563+0000", 2024-09-18T04:39:15.646 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:15.646 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.647 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.647 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:15.647 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543571+0000", 2024-09-18T04:39:15.647 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4270000000000008e-06 2024-09-18T04:39:15.647 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.647 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.647 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:15.647 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543591+0000", 2024-09-18T04:39:15.647 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.929e-05 2024-09-18T04:39:15.647 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.647 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.647 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:15.648 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543600+0000", 2024-09-18T04:39:15.648 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.1390000000000001e-06 2024-09-18T04:39:15.648 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.648 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.648 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:15.648 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543724+0000", 2024-09-18T04:39:15.648 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012458400000000001 2024-09-18T04:39:15.648 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.648 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.648 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:15.648 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543968+0000", 2024-09-18T04:39:15.648 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00024323199999999999 2024-09-18T04:39:15.649 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:15.649 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:15.649 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:15.649 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.544604+0000", 2024-09-18T04:39:15.649 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00063675799999999996 2024-09-18T04:39:15.649 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:15.649 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:15.649 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:15.649 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:15.649 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:15.649 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:15.651 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:39:15.651 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:15.645+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:15.678 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:39:15.699 INFO:tasks.workunit.client.0.smithi115.stderr: Creating snap: 10% complete... Creating snap: 10% complete...failed. 2024-09-18T04:39:15.699 INFO:tasks.workunit.client.0.smithi115.stderr:rbd: failed to create snapshot: (22) Invalid argument 2024-09-18T04:39:15.705 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:39:15.710 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:39:15.710 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo dd if=/tmp/tmp.WJMNJxIdAN/data of=/dev/nbd0 bs=1M count=1 oflag=direct 2024-09-18T04:39:15.710 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:39:15.711 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:39:15.713 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:39:15.713 INFO:tasks.workunit.client.0.smithi115.stderr:++ which dd 2024-09-18T04:39:15.715 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/bin/dd 2024-09-18T04:39:15.715 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:39:15.715 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.WJMNJxIdAN/data of=/dev/nbd0 bs=1M count=1 oflag=direct 2024-09-18T04:39:15.725 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:39:15.737 INFO:tasks.workunit.client.0.smithi115.stderr:1+0 records in 2024-09-18T04:39:15.737 INFO:tasks.workunit.client.0.smithi115.stderr:1+0 records out 2024-09-18T04:39:15.737 INFO:tasks.workunit.client.0.smithi115.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124703 s, 84.1 MB/s 2024-09-18T04:39:15.738 INFO:tasks.workunit.client.0.smithi115.stderr:+ cat 2024-09-18T04:39:15.740 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd snap create rbd/testrbdnbd17342@quiesce2 2024-09-18T04:39:15.751 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:15.745+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:15.752 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:39:15.759 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:15.759 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:15.760 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:39:15.760 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:15.770 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:39:15.779 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:39:15.803 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:39:15.805 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:39:15.805 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:39:15.806 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:39:15.806 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:39:15.806 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:39:15.806 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:39:15.852 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:15.845+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:15.870 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:15.870 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:15.880 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:39:15.953 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:15.949+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:39:15.984 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:15.984 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:15.994 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:39:16.054 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:16.049+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:39:16.114 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:16.114 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:16.124 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:39:16.155 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:16.149+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:16.233 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:16.234 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:16.243 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:39:16.256 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:16.253+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:16.352 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:16.353 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:16.356 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:16.349+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:16.363 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:39:16.457 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:16.449+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:16.472 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:16.473 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:16.483 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:39:16.558 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:16.549+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:16.586 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:16.587 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:16.597 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:39:16.658 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:16.653+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:16.703 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:16.703 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:16.703 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-18T04:39:16.703 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T04:39:16.703 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 1, 3, 4, 5, 2] 2024-09-18T04:39:16.704 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 4 2024-09-18T04:39:16.713 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:39:16.759 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:16.753+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:16.825 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:16.825 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:16.836 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:39:16.878 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:16.873+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:16.960 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:16.953+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:16.983 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:16.983 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:16.992 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:39:17.014 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-09-18T04:39:17.009+0000 7f8dc00db700 -1 mon.a@0(leader).osd e70 definitely_dead 0 2024-09-18T04:39:17.061 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:17.057+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:17.097 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:17.098 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:17.107 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:39:17.162 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:17.157+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:17.225 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:17.225 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:17.235 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:39:17.263 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:17.261+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:17.346 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:17.346 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:17.355 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:39:17.364 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:17.357+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:17.458 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:17.458 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:17.465 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:17.461+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:17.468 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:39:17.566 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:17.561+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:39:17.569 INFO:teuthology.orchestra.run.smithi028.stderr:marked out osd.4. 2024-09-18T04:39:17.575 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-18T04:39:17.576 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:17.576 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:17.585 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:39:17.666 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:17.661+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:17.687 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:17.687 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:17.697 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:39:17.767 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:17.761+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:17.804 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:17.804 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:17.814 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:39:17.820 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T04:39:17.868 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:17.861+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:17.922 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:17.923 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:17.932 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:39:17.969 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:39:17.969 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:17.961+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:17.990 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:39:18.012 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:39:18.036 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:39:18.046 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:18.046 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:18.056 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:39:18.063 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:39:18.070 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:18.065+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:18.084 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:39:18.086 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:39:18.087 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:39:18.087 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:39:18.087 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:39:18.087 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:39:18.087 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:39:18.158 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:18.159 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:39:18.159 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:39:18.159 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:39:18.159 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.159 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-18T04:39:18.159 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:39:18.159 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 103.68322997, 2024-09-18T04:39:18.159 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0095781189999999995, 2024-09-18T04:39:18.159 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:18.159 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:18.159 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:18.160 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:39:18.160 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:39:18.160 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 1 2024-09-18T04:39:18.160 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.160 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:18.160 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.160 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:18.160 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:39:18.160 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:18.160 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.160 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.160 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:18.160 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:39:18.160 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:18.161 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.161 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.161 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:18.161 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469862+0000", 2024-09-18T04:39:18.161 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:18.161 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.161 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.161 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:18.161 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469879+0000", 2024-09-18T04:39:18.161 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7431999999999998e-05 2024-09-18T04:39:18.161 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.161 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.161 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:18.161 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469896+0000", 2024-09-18T04:39:18.161 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6742999999999999e-05 2024-09-18T04:39:18.162 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.162 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.162 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:18.162 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469915+0000", 2024-09-18T04:39:18.162 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8991000000000002e-05 2024-09-18T04:39:18.162 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.162 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.162 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:18.162 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470397+0000", 2024-09-18T04:39:18.162 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048167799999999998 2024-09-18T04:39:18.162 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.162 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.162 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-09-18T04:39:18.162 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470412+0000", 2024-09-18T04:39:18.163 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5526e-05 2024-09-18T04:39:18.163 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.163 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.163 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:18.163 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473279+0000", 2024-09-18T04:39:18.163 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0028667469999999998 2024-09-18T04:39:18.163 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.163 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.163 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:18.163 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473441+0000", 2024-09-18T04:39:18.163 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000162082 2024-09-18T04:39:18.163 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.163 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.163 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:18.164 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473573+0000", 2024-09-18T04:39:18.164 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000132032 2024-09-18T04:39:18.164 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.164 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.164 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:18.164 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474826+0000", 2024-09-18T04:39:18.164 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001252717 2024-09-18T04:39:18.164 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.164 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.164 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:18.164 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479307+0000", 2024-09-18T04:39:18.164 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044813220000000003 2024-09-18T04:39:18.164 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.164 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.165 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:18.165 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479366+0000", 2024-09-18T04:39:18.165 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9024999999999998e-05 2024-09-18T04:39:18.165 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.165 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.165 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:18.165 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479443+0000", 2024-09-18T04:39:18.165 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6339000000000002e-05 2024-09-18T04:39:18.165 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:18.165 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:18.165 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:18.165 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.165 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.165 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-18T04:39:18.165 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:39:18.166 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 101.67022154999999, 2024-09-18T04:39:18.166 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082699560000000002, 2024-09-18T04:39:18.166 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:18.166 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:18.166 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:18.166 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:39:18.166 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:39:18.166 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 7 2024-09-18T04:39:18.166 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.166 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:18.166 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.166 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:18.166 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:39:18.166 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:18.166 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.167 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.167 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:18.167 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:39:18.167 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:18.167 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.167 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.167 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:18.167 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482870+0000", 2024-09-18T04:39:18.167 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999967 2024-09-18T04:39:18.167 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.167 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.167 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:18.167 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482889+0000", 2024-09-18T04:39:18.167 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.965e-05 2024-09-18T04:39:18.168 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.168 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.168 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:18.168 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482938+0000", 2024-09-18T04:39:18.168 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8548999999999999e-05 2024-09-18T04:39:18.168 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.168 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.168 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:18.168 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482963+0000", 2024-09-18T04:39:18.168 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5075999999999998e-05 2024-09-18T04:39:18.168 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.168 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.168 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:18.168 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483121+0000", 2024-09-18T04:39:18.169 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015819400000000001 2024-09-18T04:39:18.169 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.169 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.169 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op must wait for map", 2024-09-18T04:39:18.169 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483138+0000", 2024-09-18T04:39:18.169 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7068999999999999e-05 2024-09-18T04:39:18.169 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.169 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.169 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:18.169 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488666+0000", 2024-09-18T04:39:18.169 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0055282049999999996 2024-09-18T04:39:18.169 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.169 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.169 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:18.169 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488783+0000", 2024-09-18T04:39:18.170 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000116737 2024-09-18T04:39:18.170 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.170 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.170 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:18.170 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488991+0000", 2024-09-18T04:39:18.170 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020781400000000001 2024-09-18T04:39:18.170 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.170 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.170 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:18.170 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490554+0000", 2024-09-18T04:39:18.170 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0015628370000000001 2024-09-18T04:39:18.170 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.170 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.170 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:18.171 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491019+0000", 2024-09-18T04:39:18.171 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000465608 2024-09-18T04:39:18.171 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.171 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.171 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:18.171 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491087+0000", 2024-09-18T04:39:18.171 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7988e-05 2024-09-18T04:39:18.171 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.171 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.171 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:18.171 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491143+0000", 2024-09-18T04:39:18.171 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5538999999999998e-05 2024-09-18T04:39:18.171 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:18.171 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:18.172 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:18.172 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.172 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.172 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:34 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:18.172 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.261758+0000", 2024-09-18T04:39:18.172 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 35.891336398999997, 2024-09-18T04:39:18.172 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.042024191000000002, 2024-09-18T04:39:18.172 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:18.172 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:18.172 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:18.172 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:18.172 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:18.172 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 34 2024-09-18T04:39:18.172 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.173 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:18.173 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.173 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:18.173 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.261758+0000", 2024-09-18T04:39:18.173 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:18.173 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.173 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.173 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:18.173 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.261758+0000", 2024-09-18T04:39:18.173 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:18.173 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.173 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.173 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:18.173 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.261756+0000", 2024-09-18T04:39:18.173 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:18.174 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.174 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.174 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:18.174 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.270785+0000", 2024-09-18T04:39:18.174 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0090292340000000006 2024-09-18T04:39:18.174 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.174 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.174 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:18.174 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.270890+0000", 2024-09-18T04:39:18.174 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000104541 2024-09-18T04:39:18.174 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.174 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.174 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:18.174 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.270991+0000", 2024-09-18T04:39:18.175 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000100778 2024-09-18T04:39:18.175 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.175 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.175 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:18.175 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.271157+0000", 2024-09-18T04:39:18.175 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016653999999999999 2024-09-18T04:39:18.175 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.175 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.175 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:18.175 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.271591+0000", 2024-09-18T04:39:18.175 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00043409500000000001 2024-09-18T04:39:18.175 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.175 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.175 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:18.175 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272017+0000", 2024-09-18T04:39:18.176 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000425769 2024-09-18T04:39:18.176 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.176 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.176 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:18.176 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.294246+0000", 2024-09-18T04:39:18.176 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.022229237999999998 2024-09-18T04:39:18.176 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.176 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.176 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:18.176 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303571+0000", 2024-09-18T04:39:18.176 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0093250239999999995 2024-09-18T04:39:18.176 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.176 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.176 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:18.177 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303705+0000", 2024-09-18T04:39:18.177 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000133831 2024-09-18T04:39:18.177 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.177 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.177 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:18.177 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303782+0000", 2024-09-18T04:39:18.177 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7156000000000005e-05 2024-09-18T04:39:18.177 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:18.177 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:18.177 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:18.177 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.177 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.177 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:36 2.7 e51/32 2:fe1b02ad:::rbd_data.125fc09df6d1.0000000000000002:head v 51'1, mlcod=0'0)", 2024-09-18T04:39:18.177 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.273441+0000", 2024-09-18T04:39:18.177 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 35.879653417999997, 2024-09-18T04:39:18.178 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.032400768000000003, 2024-09-18T04:39:18.178 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:18.178 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:18.178 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:18.178 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.178 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:18.178 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273441+0000", 2024-09-18T04:39:18.178 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:18.178 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.178 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.178 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:18.178 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273441+0000", 2024-09-18T04:39:18.178 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:18.178 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.179 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.179 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:18.179 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273440+0000", 2024-09-18T04:39:18.179 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:18.179 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.179 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.179 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:18.179 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278723+0000", 2024-09-18T04:39:18.179 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0052829909999999999 2024-09-18T04:39:18.179 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.179 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.179 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:18.179 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278752+0000", 2024-09-18T04:39:18.179 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9178000000000001e-05 2024-09-18T04:39:18.180 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.180 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.180 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:18.180 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278761+0000", 2024-09-18T04:39:18.180 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7900000000000005e-06 2024-09-18T04:39:18.180 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.180 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.180 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:18.180 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278864+0000", 2024-09-18T04:39:18.180 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000102699 2024-09-18T04:39:18.180 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.180 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.180 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:18.180 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278900+0000", 2024-09-18T04:39:18.181 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6001999999999999e-05 2024-09-18T04:39:18.181 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.181 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.181 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:18.181 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.305793+0000", 2024-09-18T04:39:18.181 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.026893344 2024-09-18T04:39:18.181 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.181 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.181 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:18.181 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.305842+0000", 2024-09-18T04:39:18.181 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8720000000000001e-05 2024-09-18T04:39:18.181 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:18.181 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:18.181 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:18.181 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.182 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.182 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:39 2.7 e51/32 2:e9c909c5:::rbd_data.125fc09df6d1.0000000000000005:head v 51'2, mlcod=0'0)", 2024-09-18T04:39:18.182 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.283175+0000", 2024-09-18T04:39:18.182 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 35.869919729000003, 2024-09-18T04:39:18.182 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.022864102000000001, 2024-09-18T04:39:18.182 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:18.182 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:18.182 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:18.182 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.182 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:18.182 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283175+0000", 2024-09-18T04:39:18.182 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:18.182 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.182 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.183 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:18.183 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283175+0000", 2024-09-18T04:39:18.183 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:18.183 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.183 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.183 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:18.183 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283173+0000", 2024-09-18T04:39:18.183 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:18.183 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.183 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.183 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:18.183 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.285951+0000", 2024-09-18T04:39:18.183 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0027784540000000001 2024-09-18T04:39:18.183 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.183 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.184 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:18.184 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.285970+0000", 2024-09-18T04:39:18.184 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8855000000000002e-05 2024-09-18T04:39:18.184 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.184 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.184 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:18.184 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.285984+0000", 2024-09-18T04:39:18.184 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4252e-05 2024-09-18T04:39:18.184 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.184 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.184 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:18.184 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.286041+0000", 2024-09-18T04:39:18.184 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6431999999999998e-05 2024-09-18T04:39:18.184 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.185 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.185 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:18.185 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.286086+0000", 2024-09-18T04:39:18.185 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5423000000000002e-05 2024-09-18T04:39:18.185 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.185 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.185 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:18.185 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.305998+0000", 2024-09-18T04:39:18.185 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019911697999999999 2024-09-18T04:39:18.185 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.185 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.185 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:18.185 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306039+0000", 2024-09-18T04:39:18.185 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0973999999999998e-05 2024-09-18T04:39:18.186 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:18.186 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:18.186 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:18.186 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.186 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.186 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:50 2.3 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:18.186 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.312563+0000", 2024-09-18T04:39:18.186 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 35.840531702, 2024-09-18T04:39:18.186 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.032636001999999997, 2024-09-18T04:39:18.186 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:18.186 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:18.186 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:18.186 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:18.186 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:18.186 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 50 2024-09-18T04:39:18.187 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.187 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:18.187 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.187 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:18.187 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.312563+0000", 2024-09-18T04:39:18.187 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:18.187 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.187 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.187 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:18.187 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.312563+0000", 2024-09-18T04:39:18.187 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:18.187 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.187 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.187 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:18.188 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.312561+0000", 2024-09-18T04:39:18.188 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:18.188 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.188 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.188 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:18.188 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321709+0000", 2024-09-18T04:39:18.188 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0091486650000000003 2024-09-18T04:39:18.188 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.188 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.188 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:18.188 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321739+0000", 2024-09-18T04:39:18.188 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9666e-05 2024-09-18T04:39:18.188 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.188 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.189 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:18.189 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321751+0000", 2024-09-18T04:39:18.189 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2086000000000001e-05 2024-09-18T04:39:18.189 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.189 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.189 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:18.189 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321833+0000", 2024-09-18T04:39:18.189 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1842999999999996e-05 2024-09-18T04:39:18.189 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.189 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.189 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:18.189 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321981+0000", 2024-09-18T04:39:18.189 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000147441 2024-09-18T04:39:18.189 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.189 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.190 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:18.190 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.322109+0000", 2024-09-18T04:39:18.190 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012851200000000001 2024-09-18T04:39:18.190 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.190 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.190 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:18.190 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.332829+0000", 2024-09-18T04:39:18.190 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010720225999999999 2024-09-18T04:39:18.190 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.190 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.190 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:18.190 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.345039+0000", 2024-09-18T04:39:18.190 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012209831000000001 2024-09-18T04:39:18.190 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.191 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.191 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:18.191 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.345120+0000", 2024-09-18T04:39:18.191 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1145999999999999e-05 2024-09-18T04:39:18.191 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.191 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.191 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:18.191 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.345199+0000", 2024-09-18T04:39:18.191 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8546000000000003e-05 2024-09-18T04:39:18.191 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:18.191 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:18.191 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:18.191 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.191 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.192 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:30 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:18.192 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.516122+0000", 2024-09-18T04:39:18.192 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 35.636972540000002, 2024-09-18T04:39:18.192 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0091865030000000004, 2024-09-18T04:39:18.192 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:18.192 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:18.192 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:18.192 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:18.192 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:18.192 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 30 2024-09-18T04:39:18.192 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.192 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:18.192 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.192 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:18.193 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516122+0000", 2024-09-18T04:39:18.193 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:18.193 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.193 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.193 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:18.193 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516122+0000", 2024-09-18T04:39:18.193 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:18.193 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.193 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.193 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:18.193 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516120+0000", 2024-09-18T04:39:18.193 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:18.193 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.194 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.194 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:18.194 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516129+0000", 2024-09-18T04:39:18.194 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.85e-06 2024-09-18T04:39:18.194 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.194 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.194 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:18.194 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516155+0000", 2024-09-18T04:39:18.194 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5882000000000001e-05 2024-09-18T04:39:18.194 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.194 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.194 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:18.194 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516175+0000", 2024-09-18T04:39:18.194 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.022e-05 2024-09-18T04:39:18.194 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.195 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.195 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:18.195 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519780+0000", 2024-09-18T04:39:18.195 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003604187 2024-09-18T04:39:18.195 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.195 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.195 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:18.195 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519964+0000", 2024-09-18T04:39:18.195 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00018391999999999999 2024-09-18T04:39:18.195 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.195 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.195 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:18.195 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.520794+0000", 2024-09-18T04:39:18.195 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00083078100000000005 2024-09-18T04:39:18.195 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.196 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.196 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:18.196 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.524129+0000", 2024-09-18T04:39:18.196 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033342419999999998 2024-09-18T04:39:18.196 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.196 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.196 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:18.196 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525239+0000", 2024-09-18T04:39:18.196 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001110778 2024-09-18T04:39:18.196 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.196 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.196 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:18.196 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525277+0000", 2024-09-18T04:39:18.196 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7596999999999999e-05 2024-09-18T04:39:18.197 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.197 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.197 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:18.197 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525308+0000", 2024-09-18T04:39:18.197 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.1535999999999998e-05 2024-09-18T04:39:18.197 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:18.197 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:18.197 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:18.197 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.197 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.197 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:32 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:18.197 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.519217+0000", 2024-09-18T04:39:18.197 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 35.633877220000002, 2024-09-18T04:39:18.197 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082266370000000002, 2024-09-18T04:39:18.198 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:18.198 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:18.198 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:18.198 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:18.198 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:18.198 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 32 2024-09-18T04:39:18.198 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.198 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:18.198 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.198 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:18.198 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519217+0000", 2024-09-18T04:39:18.198 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:18.198 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.198 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.198 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:18.199 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519217+0000", 2024-09-18T04:39:18.199 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:18.199 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.199 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.199 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:18.199 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519215+0000", 2024-09-18T04:39:18.199 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:18.199 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.199 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.199 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:18.199 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519229+0000", 2024-09-18T04:39:18.199 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3329999999999999e-05 2024-09-18T04:39:18.199 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.199 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.200 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:18.200 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519268+0000", 2024-09-18T04:39:18.200 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9533000000000002e-05 2024-09-18T04:39:18.200 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.200 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.200 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:18.200 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519285+0000", 2024-09-18T04:39:18.200 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6446000000000001e-05 2024-09-18T04:39:18.200 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.200 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.200 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:18.200 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.522358+0000", 2024-09-18T04:39:18.200 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030734310000000002 2024-09-18T04:39:18.200 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.201 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.201 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:18.201 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.522551+0000", 2024-09-18T04:39:18.201 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019317600000000001 2024-09-18T04:39:18.201 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.201 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.201 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:18.201 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.523262+0000", 2024-09-18T04:39:18.201 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00071069999999999998 2024-09-18T04:39:18.201 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.201 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.201 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:18.201 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525318+0000", 2024-09-18T04:39:18.201 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0020561469999999999 2024-09-18T04:39:18.201 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.202 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.202 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:18.202 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.527376+0000", 2024-09-18T04:39:18.202 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0020584969999999998 2024-09-18T04:39:18.202 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.202 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.202 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:18.202 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.527416+0000", 2024-09-18T04:39:18.202 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9054000000000003e-05 2024-09-18T04:39:18.202 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.202 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.202 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:18.202 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.527444+0000", 2024-09-18T04:39:18.202 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8347000000000002e-05 2024-09-18T04:39:18.203 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:18.203 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:18.203 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:18.203 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.203 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.203 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:42 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:18.203 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.551160+0000", 2024-09-18T04:39:18.203 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 35.601934827999997, 2024-09-18T04:39:18.203 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010692757000000001, 2024-09-18T04:39:18.203 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:18.203 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:18.203 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:18.203 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:18.203 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:18.203 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 42 2024-09-18T04:39:18.204 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.204 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:18.204 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.204 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:18.204 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551160+0000", 2024-09-18T04:39:18.204 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:18.204 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.204 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.204 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:18.204 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551160+0000", 2024-09-18T04:39:18.204 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:18.204 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.204 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.204 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:18.205 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551159+0000", 2024-09-18T04:39:18.205 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:18.205 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.205 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.205 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:18.205 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551166+0000", 2024-09-18T04:39:18.205 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1029999999999996e-06 2024-09-18T04:39:18.205 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.205 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.205 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:18.205 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551177+0000", 2024-09-18T04:39:18.205 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1384e-05 2024-09-18T04:39:18.205 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.205 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.206 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:18.206 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551185+0000", 2024-09-18T04:39:18.206 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1990000000000006e-06 2024-09-18T04:39:18.206 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.206 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.206 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:18.206 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552021+0000", 2024-09-18T04:39:18.206 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00083574400000000003 2024-09-18T04:39:18.206 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.206 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.206 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:18.206 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552339+0000", 2024-09-18T04:39:18.206 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031771399999999997 2024-09-18T04:39:18.206 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.206 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.207 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:18.207 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.553140+0000", 2024-09-18T04:39:18.207 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00080088100000000003 2024-09-18T04:39:18.207 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.207 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.207 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:18.207 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.554585+0000", 2024-09-18T04:39:18.207 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001445801 2024-09-18T04:39:18.207 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.207 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.207 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:18.207 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.561711+0000", 2024-09-18T04:39:18.207 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071260109999999998 2024-09-18T04:39:18.207 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.207 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.208 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:18.208 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.561790+0000", 2024-09-18T04:39:18.208 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8967000000000005e-05 2024-09-18T04:39:18.208 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.208 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.208 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:18.208 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.561852+0000", 2024-09-18T04:39:18.208 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2045999999999995e-05 2024-09-18T04:39:18.208 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:18.208 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:18.208 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:18.208 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.208 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.208 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:58 2.7 e51/32 2:fc7c6cd3:ns::rbd_data.133e813ab5bc.000000000000000e:head v 51'7, mlcod=51'5)", 2024-09-18T04:39:18.209 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.605333+0000", 2024-09-18T04:39:18.209 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 35.547761623, 2024-09-18T04:39:18.209 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020085902999999999, 2024-09-18T04:39:18.209 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:18.209 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:18.209 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:18.209 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.209 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:18.209 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605333+0000", 2024-09-18T04:39:18.209 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:18.209 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.209 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.209 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:18.209 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605333+0000", 2024-09-18T04:39:18.209 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:18.210 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.210 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.210 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:18.210 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605332+0000", 2024-09-18T04:39:18.210 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:18.210 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.210 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.210 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:18.210 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608667+0000", 2024-09-18T04:39:18.210 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033350300000000001 2024-09-18T04:39:18.210 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.210 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.210 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:18.210 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608706+0000", 2024-09-18T04:39:18.210 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8951e-05 2024-09-18T04:39:18.211 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.211 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.211 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:18.211 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608725+0000", 2024-09-18T04:39:18.211 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9675999999999999e-05 2024-09-18T04:39:18.211 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.211 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.211 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:18.211 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608798+0000", 2024-09-18T04:39:18.211 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3135999999999999e-05 2024-09-18T04:39:18.211 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.211 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.211 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:18.211 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608824+0000", 2024-09-18T04:39:18.211 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5284000000000001e-05 2024-09-18T04:39:18.212 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.212 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.212 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:18.212 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625370+0000", 2024-09-18T04:39:18.212 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016545975000000001 2024-09-18T04:39:18.212 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.212 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.212 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:18.212 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625419+0000", 2024-09-18T04:39:18.212 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9076999999999998e-05 2024-09-18T04:39:18.212 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:18.212 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:18.212 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:18.213 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.213 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.213 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:611 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:18.213 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.069561+0000", 2024-09-18T04:39:18.213 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 27.083533069000001, 2024-09-18T04:39:18.213 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0072330299999999997, 2024-09-18T04:39:18.213 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:18.213 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:18.213 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:18.213 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:39:18.213 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:18.213 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 611 2024-09-18T04:39:18.213 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.213 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:18.214 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.214 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:18.214 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.069561+0000", 2024-09-18T04:39:18.214 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:18.214 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.214 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.214 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:18.214 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.069561+0000", 2024-09-18T04:39:18.214 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:18.214 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.214 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.214 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:18.214 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.069559+0000", 2024-09-18T04:39:18.214 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:18.214 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.215 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.215 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:18.215 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070801+0000", 2024-09-18T04:39:18.215 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0012423359999999999 2024-09-18T04:39:18.215 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.215 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.215 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:18.215 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070828+0000", 2024-09-18T04:39:18.215 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6387e-05 2024-09-18T04:39:18.215 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.215 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.215 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:18.215 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070840+0000", 2024-09-18T04:39:18.215 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2689e-05 2024-09-18T04:39:18.216 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.216 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.216 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:18.216 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070990+0000", 2024-09-18T04:39:18.216 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014946800000000001 2024-09-18T04:39:18.216 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.216 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.216 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:18.216 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.071128+0000", 2024-09-18T04:39:18.216 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013796699999999999 2024-09-18T04:39:18.216 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.216 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.216 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:18.216 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.071318+0000", 2024-09-18T04:39:18.216 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000190106 2024-09-18T04:39:18.217 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.217 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.217 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:18.217 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.075711+0000", 2024-09-18T04:39:18.217 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.004393002 2024-09-18T04:39:18.217 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.217 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.217 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:18.217 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.076665+0000", 2024-09-18T04:39:18.217 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00095390399999999999 2024-09-18T04:39:18.217 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.217 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.217 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:18.217 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.076742+0000", 2024-09-18T04:39:18.217 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7112000000000003e-05 2024-09-18T04:39:18.218 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.218 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.218 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:18.218 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.076794+0000", 2024-09-18T04:39:18.218 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2475999999999997e-05 2024-09-18T04:39:18.218 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:18.218 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:18.218 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:18.218 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.218 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.218 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:687 2.3 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:18.218 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.487994+0000", 2024-09-18T04:39:18.218 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 26.665100085999999, 2024-09-18T04:39:18.218 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0074167690000000001, 2024-09-18T04:39:18.219 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:18.219 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:18.219 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:18.219 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:39:18.219 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:18.219 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 687 2024-09-18T04:39:18.219 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.219 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:18.219 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.219 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:18.219 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.487994+0000", 2024-09-18T04:39:18.219 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:18.219 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.219 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.220 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:18.220 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.487994+0000", 2024-09-18T04:39:18.220 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:18.220 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.220 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.220 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:18.220 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.487993+0000", 2024-09-18T04:39:18.220 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:18.220 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.220 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.220 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:18.220 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.488979+0000", 2024-09-18T04:39:18.220 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00098620700000000001 2024-09-18T04:39:18.220 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.220 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.221 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:18.221 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489002+0000", 2024-09-18T04:39:18.221 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2500000000000001e-05 2024-09-18T04:39:18.221 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.221 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.221 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:18.221 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489020+0000", 2024-09-18T04:39:18.221 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7691000000000001e-05 2024-09-18T04:39:18.221 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.221 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.221 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:18.221 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489091+0000", 2024-09-18T04:39:18.221 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1307e-05 2024-09-18T04:39:18.221 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.222 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.222 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:18.222 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489219+0000", 2024-09-18T04:39:18.222 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012766000000000001 2024-09-18T04:39:18.222 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.222 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.222 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:18.222 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489387+0000", 2024-09-18T04:39:18.222 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016855400000000001 2024-09-18T04:39:18.222 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.222 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.222 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:18.222 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495243+0000", 2024-09-18T04:39:18.222 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058563189999999996 2024-09-18T04:39:18.223 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.223 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.223 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:18.223 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495266+0000", 2024-09-18T04:39:18.223 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2810999999999999e-05 2024-09-18T04:39:18.223 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.223 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.223 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:18.223 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495332+0000", 2024-09-18T04:39:18.223 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5469999999999995e-05 2024-09-18T04:39:18.223 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.223 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.223 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:18.223 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495411+0000", 2024-09-18T04:39:18.223 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9395999999999997e-05 2024-09-18T04:39:18.224 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:18.224 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:18.224 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:18.224 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.224 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.224 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:689 2.3 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:18.224 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.500070+0000", 2024-09-18T04:39:18.224 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 26.65302427, 2024-09-18T04:39:18.224 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071096650000000003, 2024-09-18T04:39:18.224 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:18.224 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:18.224 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:18.224 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:39:18.224 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:18.225 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 689 2024-09-18T04:39:18.225 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.225 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:18.225 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.225 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:18.225 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.500070+0000", 2024-09-18T04:39:18.225 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:18.225 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.225 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.225 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:18.225 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.500070+0000", 2024-09-18T04:39:18.225 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:18.225 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.225 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.226 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:18.226 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.500069+0000", 2024-09-18T04:39:18.226 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:18.226 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.226 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.226 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:18.226 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501082+0000", 2024-09-18T04:39:18.226 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010136570000000001 2024-09-18T04:39:18.226 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.226 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.226 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:18.226 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501104+0000", 2024-09-18T04:39:18.226 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1883e-05 2024-09-18T04:39:18.226 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.227 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.227 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:18.227 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501115+0000", 2024-09-18T04:39:18.227 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0414000000000001e-05 2024-09-18T04:39:18.227 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.227 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.227 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:18.227 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501178+0000", 2024-09-18T04:39:18.227 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3128999999999998e-05 2024-09-18T04:39:18.227 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.227 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.227 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:18.227 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501257+0000", 2024-09-18T04:39:18.227 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9592999999999995e-05 2024-09-18T04:39:18.227 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.228 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.228 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:18.228 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501366+0000", 2024-09-18T04:39:18.228 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000108643 2024-09-18T04:39:18.228 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.228 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.228 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:18.228 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.504041+0000", 2024-09-18T04:39:18.228 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0026750889999999999 2024-09-18T04:39:18.228 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.228 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.228 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:18.228 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.507097+0000", 2024-09-18T04:39:18.228 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030556070000000001 2024-09-18T04:39:18.229 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.229 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.229 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:18.229 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.507140+0000", 2024-09-18T04:39:18.229 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3745000000000002e-05 2024-09-18T04:39:18.229 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.229 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.229 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:18.229 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.507180+0000", 2024-09-18T04:39:18.229 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9475999999999999e-05 2024-09-18T04:39:18.229 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:18.229 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:18.229 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:18.229 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.230 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.230 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4972.0:13 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:18.230 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.948263+0000", 2024-09-18T04:39:18.230 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 26.204831478999999, 2024-09-18T04:39:18.230 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0091358970000000005, 2024-09-18T04:39:18.230 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:18.230 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:39:18.230 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:18.230 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4972", 2024-09-18T04:39:18.230 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2280251472", 2024-09-18T04:39:18.230 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 13 2024-09-18T04:39:18.230 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.230 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:18.230 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.230 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:18.231 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948263+0000", 2024-09-18T04:39:18.231 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:18.231 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.231 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.231 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:18.231 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948263+0000", 2024-09-18T04:39:18.231 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:18.231 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.231 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.231 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:18.231 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948261+0000", 2024-09-18T04:39:18.231 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:18.231 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.231 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.231 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:18.232 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948273+0000", 2024-09-18T04:39:18.232 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1576999999999999e-05 2024-09-18T04:39:18.232 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.232 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.232 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:18.232 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948305+0000", 2024-09-18T04:39:18.232 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2209000000000003e-05 2024-09-18T04:39:18.232 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.232 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.232 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:18.232 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948325+0000", 2024-09-18T04:39:18.232 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.022e-05 2024-09-18T04:39:18.232 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.232 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.233 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:18.233 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948455+0000", 2024-09-18T04:39:18.233 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000129624 2024-09-18T04:39:18.233 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.233 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.233 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:18.233 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948598+0000", 2024-09-18T04:39:18.233 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014376699999999999 2024-09-18T04:39:18.233 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.233 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.233 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:18.233 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.957399+0000", 2024-09-18T04:39:18.233 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0088004299999999997 2024-09-18T04:39:18.233 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:18.233 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:18.234 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:18.234 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.234 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.234 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:101 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:18.234 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.576367+0000", 2024-09-18T04:39:18.234 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 21.576727771000002, 2024-09-18T04:39:18.234 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0078671020000000008, 2024-09-18T04:39:18.234 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:18.234 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:18.234 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:18.234 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:18.234 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:18.234 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 101 2024-09-18T04:39:18.234 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.234 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:18.235 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.235 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:18.235 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576367+0000", 2024-09-18T04:39:18.235 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:18.235 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.235 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.235 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:18.235 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576367+0000", 2024-09-18T04:39:18.235 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:18.235 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.235 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.235 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:18.235 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576364+0000", 2024-09-18T04:39:18.235 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999971 2024-09-18T04:39:18.236 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.236 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.236 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:18.236 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576378+0000", 2024-09-18T04:39:18.236 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3879999999999999e-05 2024-09-18T04:39:18.236 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.236 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.236 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:18.236 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576410+0000", 2024-09-18T04:39:18.236 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2286000000000002e-05 2024-09-18T04:39:18.236 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.236 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.236 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:18.236 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576425+0000", 2024-09-18T04:39:18.237 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5290000000000001e-05 2024-09-18T04:39:18.237 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.237 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.237 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:18.237 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576551+0000", 2024-09-18T04:39:18.237 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012555199999999999 2024-09-18T04:39:18.237 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.237 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.237 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:18.237 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576719+0000", 2024-09-18T04:39:18.237 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016787400000000001 2024-09-18T04:39:18.237 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.237 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.237 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:18.238 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576951+0000", 2024-09-18T04:39:18.238 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00023252400000000001 2024-09-18T04:39:18.238 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.238 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.238 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:18.238 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.581459+0000", 2024-09-18T04:39:18.238 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045074410000000001 2024-09-18T04:39:18.238 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.238 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.238 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:18.238 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584076+0000", 2024-09-18T04:39:18.238 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0026173860000000002 2024-09-18T04:39:18.238 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.238 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.239 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:18.239 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584160+0000", 2024-09-18T04:39:18.239 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4191999999999996e-05 2024-09-18T04:39:18.239 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.239 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.239 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:18.239 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584234+0000", 2024-09-18T04:39:18.239 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.339e-05 2024-09-18T04:39:18.239 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:18.239 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:18.239 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:18.239 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.239 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.239 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:44 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:18.240 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.550448+0000", 2024-09-18T04:39:18.240 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 17.602646581999998, 2024-09-18T04:39:18.240 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082117590000000008, 2024-09-18T04:39:18.240 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:18.240 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:18.240 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:18.240 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:39:18.240 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:18.240 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 44 2024-09-18T04:39:18.240 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.240 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:18.240 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.240 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:18.240 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.550448+0000", 2024-09-18T04:39:18.240 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:18.241 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.241 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.241 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:18.241 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.550448+0000", 2024-09-18T04:39:18.241 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:18.241 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.241 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.241 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:18.241 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.550445+0000", 2024-09-18T04:39:18.241 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999971 2024-09-18T04:39:18.241 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.241 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.241 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:18.242 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.551904+0000", 2024-09-18T04:39:18.242 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001458322 2024-09-18T04:39:18.242 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.242 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.242 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:18.242 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.551938+0000", 2024-09-18T04:39:18.242 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4359000000000001e-05 2024-09-18T04:39:18.242 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.242 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.242 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:18.242 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.551959+0000", 2024-09-18T04:39:18.242 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1403000000000002e-05 2024-09-18T04:39:18.242 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.242 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.243 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:18.243 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.552049+0000", 2024-09-18T04:39:18.243 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9970999999999996e-05 2024-09-18T04:39:18.243 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.243 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.243 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:18.243 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.552190+0000", 2024-09-18T04:39:18.243 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000140271 2024-09-18T04:39:18.243 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.243 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.243 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:18.243 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.552385+0000", 2024-09-18T04:39:18.243 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019498700000000001 2024-09-18T04:39:18.243 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.244 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.244 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:18.244 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.556308+0000", 2024-09-18T04:39:18.244 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00392333 2024-09-18T04:39:18.244 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.244 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.244 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:18.244 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.558479+0000", 2024-09-18T04:39:18.244 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0021715889999999998 2024-09-18T04:39:18.244 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.244 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.244 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:18.244 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.558616+0000", 2024-09-18T04:39:18.244 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013613399999999999 2024-09-18T04:39:18.245 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.245 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.245 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:18.245 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.558660+0000", 2024-09-18T04:39:18.245 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4053e-05 2024-09-18T04:39:18.245 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:18.245 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:18.245 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:18.245 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.245 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.245 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:46 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:18.245 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.564089+0000", 2024-09-18T04:39:18.245 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 17.589005452999999, 2024-09-18T04:39:18.245 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0069437090000000002, 2024-09-18T04:39:18.246 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:18.246 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:18.246 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:18.246 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:39:18.246 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:18.246 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 46 2024-09-18T04:39:18.246 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.246 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:18.246 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.246 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:18.246 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.564089+0000", 2024-09-18T04:39:18.246 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:18.246 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.246 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.247 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:18.247 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.564089+0000", 2024-09-18T04:39:18.247 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:18.247 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.247 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.247 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:18.247 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.564087+0000", 2024-09-18T04:39:18.247 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:18.247 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.247 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.247 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:18.247 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565126+0000", 2024-09-18T04:39:18.247 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001038902 2024-09-18T04:39:18.247 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.247 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.248 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:18.248 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565148+0000", 2024-09-18T04:39:18.248 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1911000000000001e-05 2024-09-18T04:39:18.248 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.248 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.248 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:18.248 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565157+0000", 2024-09-18T04:39:18.248 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6220000000000004e-06 2024-09-18T04:39:18.248 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.248 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.248 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:18.248 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565261+0000", 2024-09-18T04:39:18.248 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000103063 2024-09-18T04:39:18.248 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.249 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.249 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:18.249 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565394+0000", 2024-09-18T04:39:18.249 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000133032 2024-09-18T04:39:18.249 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.249 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.249 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:18.249 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565564+0000", 2024-09-18T04:39:18.249 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017068000000000001 2024-09-18T04:39:18.249 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.249 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.249 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:18.249 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.570556+0000", 2024-09-18T04:39:18.249 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0049920620000000002 2024-09-18T04:39:18.250 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.250 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.250 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:18.250 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.570885+0000", 2024-09-18T04:39:18.250 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032827900000000001 2024-09-18T04:39:18.250 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.250 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.250 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:18.250 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.570959+0000", 2024-09-18T04:39:18.250 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3955999999999995e-05 2024-09-18T04:39:18.250 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.250 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.250 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:18.250 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.571033+0000", 2024-09-18T04:39:18.250 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4191999999999997e-05 2024-09-18T04:39:18.251 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:18.251 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:18.251 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:18.251 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.251 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.251 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:100 2.4 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:18.251 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.856389+0000", 2024-09-18T04:39:18.251 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 17.296705340999999, 2024-09-18T04:39:18.251 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0081480800000000003, 2024-09-18T04:39:18.251 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:18.251 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:18.251 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:18.251 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:39:18.251 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:18.252 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 100 2024-09-18T04:39:18.252 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.252 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:18.252 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.252 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:18.252 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.856389+0000", 2024-09-18T04:39:18.252 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:18.252 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.252 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.252 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:18.252 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.856389+0000", 2024-09-18T04:39:18.252 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:18.252 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.252 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.253 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:18.253 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.856388+0000", 2024-09-18T04:39:18.253 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:18.253 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.253 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.253 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:18.253 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857295+0000", 2024-09-18T04:39:18.253 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00090725599999999995 2024-09-18T04:39:18.253 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.253 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.253 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:18.253 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857311+0000", 2024-09-18T04:39:18.253 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6092000000000001e-05 2024-09-18T04:39:18.254 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.254 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.254 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:18.254 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857319+0000", 2024-09-18T04:39:18.254 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9200000000000004e-06 2024-09-18T04:39:18.254 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.254 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.254 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:18.254 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857388+0000", 2024-09-18T04:39:18.254 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8591999999999997e-05 2024-09-18T04:39:18.254 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.254 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.254 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:18.254 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857535+0000", 2024-09-18T04:39:18.255 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014684299999999999 2024-09-18T04:39:18.255 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.255 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.255 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:18.255 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857747+0000", 2024-09-18T04:39:18.255 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000212361 2024-09-18T04:39:18.255 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.255 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.255 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:18.255 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.860831+0000", 2024-09-18T04:39:18.255 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030841789999999998 2024-09-18T04:39:18.255 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.255 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.255 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:18.256 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.864361+0000", 2024-09-18T04:39:18.256 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0035298780000000002 2024-09-18T04:39:18.256 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.256 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.256 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:18.256 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.864456+0000", 2024-09-18T04:39:18.256 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4557999999999998e-05 2024-09-18T04:39:18.256 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.256 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.256 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:18.256 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.864537+0000", 2024-09-18T04:39:18.256 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1577999999999997e-05 2024-09-18T04:39:18.256 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:18.256 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:18.257 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:18.257 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.257 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.257 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:102 2.4 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:18.257 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.869773+0000", 2024-09-18T04:39:18.257 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 17.283321684000001, 2024-09-18T04:39:18.257 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0084430359999999992, 2024-09-18T04:39:18.257 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:18.257 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:18.257 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:18.257 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:39:18.257 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:18.258 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 102 2024-09-18T04:39:18.258 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.258 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:18.258 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.258 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:18.258 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.869773+0000", 2024-09-18T04:39:18.258 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:18.258 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.258 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.258 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:18.258 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.869773+0000", 2024-09-18T04:39:18.258 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:18.258 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.259 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.259 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:18.259 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.869771+0000", 2024-09-18T04:39:18.259 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:18.259 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.259 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.259 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:18.259 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870629+0000", 2024-09-18T04:39:18.259 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00085760999999999997 2024-09-18T04:39:18.259 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.259 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.259 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:18.259 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870647+0000", 2024-09-18T04:39:18.259 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8363999999999999e-05 2024-09-18T04:39:18.259 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.260 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.260 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:18.260 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870655+0000", 2024-09-18T04:39:18.260 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0099999999999995e-06 2024-09-18T04:39:18.260 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.260 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.260 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:18.260 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870803+0000", 2024-09-18T04:39:18.260 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014798 2024-09-18T04:39:18.260 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.260 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.260 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:18.260 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870951+0000", 2024-09-18T04:39:18.260 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014715999999999999 2024-09-18T04:39:18.261 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.261 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.261 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:18.261 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.871145+0000", 2024-09-18T04:39:18.261 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000194253 2024-09-18T04:39:18.261 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.261 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.261 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:18.261 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.877033+0000", 2024-09-18T04:39:18.261 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058883099999999999 2024-09-18T04:39:18.261 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.261 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.261 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:18.261 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.878078+0000", 2024-09-18T04:39:18.261 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001044747 2024-09-18T04:39:18.262 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.262 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.262 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:18.262 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.878160+0000", 2024-09-18T04:39:18.262 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1904999999999996e-05 2024-09-18T04:39:18.262 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.262 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.262 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:18.262 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.878216+0000", 2024-09-18T04:39:18.262 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6082000000000003e-05 2024-09-18T04:39:18.262 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:18.262 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:18.262 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:18.262 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.263 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.263 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:145 2.4 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:18.263 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:01.096935+0000", 2024-09-18T04:39:18.263 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 17.056159904000001, 2024-09-18T04:39:18.263 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0069937869999999996, 2024-09-18T04:39:18.263 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:18.263 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:18.263 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:18.263 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:39:18.263 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:18.263 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 145 2024-09-18T04:39:18.263 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.263 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:18.263 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.263 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:18.264 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.096935+0000", 2024-09-18T04:39:18.264 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:18.264 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.264 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.264 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:18.264 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.096935+0000", 2024-09-18T04:39:18.264 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:18.264 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.264 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.264 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:18.264 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.096932+0000", 2024-09-18T04:39:18.264 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:18.264 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.264 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.265 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:18.265 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097815+0000", 2024-09-18T04:39:18.265 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00088276000000000001 2024-09-18T04:39:18.265 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.265 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.265 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:18.265 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097842+0000", 2024-09-18T04:39:18.265 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6806999999999999e-05 2024-09-18T04:39:18.265 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.265 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.265 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:18.265 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097852+0000", 2024-09-18T04:39:18.265 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0548e-05 2024-09-18T04:39:18.265 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.266 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.266 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:18.266 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097931+0000", 2024-09-18T04:39:18.266 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8417000000000005e-05 2024-09-18T04:39:18.266 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.266 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.266 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:18.266 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.098047+0000", 2024-09-18T04:39:18.266 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011611 2024-09-18T04:39:18.266 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.266 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.266 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:18.266 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.098172+0000", 2024-09-18T04:39:18.266 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000125482 2024-09-18T04:39:18.266 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.267 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.267 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:18.267 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.100888+0000", 2024-09-18T04:39:18.267 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0027154430000000001 2024-09-18T04:39:18.267 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.267 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.267 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:18.267 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.103814+0000", 2024-09-18T04:39:18.267 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0029259619999999998 2024-09-18T04:39:18.267 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.267 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.267 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:18.267 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.103891+0000", 2024-09-18T04:39:18.267 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7619000000000003e-05 2024-09-18T04:39:18.267 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:18.268 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:18.268 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:18.268 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.103928+0000", 2024-09-18T04:39:18.268 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6900999999999998e-05 2024-09-18T04:39:18.268 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:18.268 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:18.268 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:18.268 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:18.268 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:18.268 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:18.270 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:18.261+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:18.270 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:39:18.371 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:18.365+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:18.390 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:18.390 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:18.390 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:39:18.390 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:18.400 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:39:18.471 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:18.465+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:18.512 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:18.513 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:18.513 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:39:18.513 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:39:18.513 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:18.522 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:39:18.572 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:18.565+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:18.582 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-09-18T04:39:18.635 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:18.635 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:39:18.635 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:39:18.635 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:39:18.636 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.636 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:37 2.5 e51/28 2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:head v 51'1, mlcod=0'0)", 2024-09-18T04:39:18.636 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:39:18.636 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 36.338708339, 2024-09-18T04:39:18.636 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012279973, 2024-09-18T04:39:18.636 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:18.636 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:18.636 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:18.636 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.636 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:18.636 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:39:18.636 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:18.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.637 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.637 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:18.637 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:39:18.637 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:18.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.637 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.637 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:18.637 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292468+0000", 2024-09-18T04:39:18.637 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:18.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.637 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.637 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:18.637 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295837+0000", 2024-09-18T04:39:18.638 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0033691699999999999 2024-09-18T04:39:18.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.638 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.638 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:18.638 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295860+0000", 2024-09-18T04:39:18.638 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2756999999999999e-05 2024-09-18T04:39:18.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.638 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.638 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:18.638 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295872+0000", 2024-09-18T04:39:18.638 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2442e-05 2024-09-18T04:39:18.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.638 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.638 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:18.639 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295934+0000", 2024-09-18T04:39:18.639 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1187999999999998e-05 2024-09-18T04:39:18.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.639 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.639 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:18.639 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295967+0000", 2024-09-18T04:39:18.639 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3185999999999997e-05 2024-09-18T04:39:18.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.639 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.639 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:18.639 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304697+0000", 2024-09-18T04:39:18.639 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0087298059999999997 2024-09-18T04:39:18.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.639 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.640 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:18.640 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304749+0000", 2024-09-18T04:39:18.640 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1922000000000002e-05 2024-09-18T04:39:18.640 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:18.640 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:18.640 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:18.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.640 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.640 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:49 2.5 e51/28 2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head v 51'2, mlcod=0'0)", 2024-09-18T04:39:18.641 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:39:18.641 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 36.305381500999999, 2024-09-18T04:39:18.641 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018033041999999999, 2024-09-18T04:39:18.641 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:18.641 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:18.641 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:18.641 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.641 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:18.641 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:39:18.641 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:18.641 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.641 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.641 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:18.641 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:39:18.642 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:18.642 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.642 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.642 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:18.642 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325793+0000", 2024-09-18T04:39:18.642 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:18.642 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.642 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.642 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:18.642 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329038+0000", 2024-09-18T04:39:18.642 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0032444010000000001 2024-09-18T04:39:18.642 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.642 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.643 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:18.643 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329065+0000", 2024-09-18T04:39:18.643 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7500000000000001e-05 2024-09-18T04:39:18.643 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.643 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.643 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:18.643 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329077+0000", 2024-09-18T04:39:18.643 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1919999999999999e-05 2024-09-18T04:39:18.643 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.643 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.643 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:18.643 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329149+0000", 2024-09-18T04:39:18.643 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2349999999999997e-05 2024-09-18T04:39:18.643 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.644 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.644 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:18.644 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329192+0000", 2024-09-18T04:39:18.644 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2744000000000003e-05 2024-09-18T04:39:18.644 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.644 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.644 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:18.644 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343773+0000", 2024-09-18T04:39:18.644 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014581246000000001 2024-09-18T04:39:18.644 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.644 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.644 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:18.644 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343829+0000", 2024-09-18T04:39:18.644 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5143999999999999e-05 2024-09-18T04:39:18.645 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:18.645 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:18.645 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:18.645 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.645 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.645 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:47 2.4 e51/17 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 51'3, mlcod=51'1)", 2024-09-18T04:39:18.645 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:39:18.645 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 36.304801236000003, 2024-09-18T04:39:18.645 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018960186, 2024-09-18T04:39:18.645 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:18.645 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:18.645 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:18.645 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.646 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:18.646 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:39:18.646 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:18.646 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.646 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.646 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:18.646 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:39:18.646 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:18.646 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.646 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.646 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:18.646 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326374+0000", 2024-09-18T04:39:18.646 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:18.646 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.646 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.647 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:18.647 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330883+0000", 2024-09-18T04:39:18.647 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0045084740000000002 2024-09-18T04:39:18.647 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.647 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.647 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:18.647 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330901+0000", 2024-09-18T04:39:18.647 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7890999999999999e-05 2024-09-18T04:39:18.647 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.647 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.647 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:18.647 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330912+0000", 2024-09-18T04:39:18.647 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1041e-05 2024-09-18T04:39:18.648 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.648 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.648 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:18.648 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.331003+0000", 2024-09-18T04:39:18.648 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1323999999999995e-05 2024-09-18T04:39:18.648 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.648 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.648 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:18.648 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.331030+0000", 2024-09-18T04:39:18.648 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7351999999999998e-05 2024-09-18T04:39:18.648 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.648 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.648 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:18.648 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345287+0000", 2024-09-18T04:39:18.649 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014256332 2024-09-18T04:39:18.649 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.649 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.649 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:18.649 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345336+0000", 2024-09-18T04:39:18.649 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9356e-05 2024-09-18T04:39:18.649 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:18.649 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:18.649 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:18.649 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.649 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.649 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:58 2.5 e51/28 2:b227e6b1:::rbd_data.125fc09df6d1.000000000000000e:head v 51'3, mlcod=51'1)", 2024-09-18T04:39:18.649 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:39:18.649 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 36.264374609000001, 2024-09-18T04:39:18.649 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016513678, 2024-09-18T04:39:18.650 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:18.650 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:18.650 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:18.650 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.650 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:18.650 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:39:18.650 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:18.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.650 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.650 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:18.650 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:39:18.650 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:18.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.651 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.651 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:18.651 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366800+0000", 2024-09-18T04:39:18.651 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:18.651 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.651 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.651 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:18.651 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368644+0000", 2024-09-18T04:39:18.651 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001844399 2024-09-18T04:39:18.651 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.651 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.651 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:18.651 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368681+0000", 2024-09-18T04:39:18.651 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6526999999999998e-05 2024-09-18T04:39:18.651 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.652 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.652 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:18.652 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368698+0000", 2024-09-18T04:39:18.652 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7272999999999999e-05 2024-09-18T04:39:18.652 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.652 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.652 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:18.652 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368773+0000", 2024-09-18T04:39:18.652 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4862000000000003e-05 2024-09-18T04:39:18.652 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.652 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.652 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:18.652 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368801+0000", 2024-09-18T04:39:18.652 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7925000000000001e-05 2024-09-18T04:39:18.653 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.653 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.653 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:18.653 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383261+0000", 2024-09-18T04:39:18.653 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014460377999999999 2024-09-18T04:39:18.653 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.653 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.653 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:18.653 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383316+0000", 2024-09-18T04:39:18.653 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4623999999999997e-05 2024-09-18T04:39:18.653 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:18.653 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:18.653 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:18.653 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.654 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.654 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:56 2.4 e51/17 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 51'4, mlcod=51'2)", 2024-09-18T04:39:18.654 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:39:18.654 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 36.262370582999999, 2024-09-18T04:39:18.654 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015105221, 2024-09-18T04:39:18.654 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:18.654 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:18.654 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:18.654 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.654 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:18.654 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:39:18.654 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:18.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.654 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.655 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:18.655 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:39:18.655 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:18.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.655 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.655 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:18.655 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368805+0000", 2024-09-18T04:39:18.655 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:18.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.655 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.655 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:18.655 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371603+0000", 2024-09-18T04:39:18.655 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027982660000000002 2024-09-18T04:39:18.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.656 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.656 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:18.656 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371633+0000", 2024-09-18T04:39:18.656 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0073000000000002e-05 2024-09-18T04:39:18.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.656 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.656 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:18.656 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371652+0000", 2024-09-18T04:39:18.656 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8597000000000002e-05 2024-09-18T04:39:18.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.656 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.656 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:18.656 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371752+0000", 2024-09-18T04:39:18.656 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9772999999999995e-05 2024-09-18T04:39:18.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.657 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.657 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:18.657 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371793+0000", 2024-09-18T04:39:18.657 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1173e-05 2024-09-18T04:39:18.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.657 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.657 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:18.657 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383875+0000", 2024-09-18T04:39:18.657 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012081852000000001 2024-09-18T04:39:18.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.657 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.657 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:18.657 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383912+0000", 2024-09-18T04:39:18.658 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7045000000000002e-05 2024-09-18T04:39:18.658 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:18.658 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:18.658 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:18.658 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.658 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.658 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:36 2.5 e51/28 2:bd5b8dd8:ns::rbd_data.133e813ab5bc.0000000000000002:head v 51'4, mlcod=51'2)", 2024-09-18T04:39:18.658 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:39:18.658 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 36.093651592000001, 2024-09-18T04:39:18.658 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020758161000000001, 2024-09-18T04:39:18.658 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:18.658 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:18.658 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:18.658 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.659 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:18.659 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:39:18.659 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:18.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.659 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.659 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:18.659 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:39:18.659 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:18.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.659 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.659 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:18.659 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537523+0000", 2024-09-18T04:39:18.659 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:18.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.660 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.660 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:18.660 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541578+0000", 2024-09-18T04:39:18.660 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0040549710000000001 2024-09-18T04:39:18.660 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.660 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.660 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:18.660 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541608+0000", 2024-09-18T04:39:18.660 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9349e-05 2024-09-18T04:39:18.660 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.660 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.660 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:18.660 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541621+0000", 2024-09-18T04:39:18.660 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2736e-05 2024-09-18T04:39:18.661 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.661 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.661 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:18.661 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541715+0000", 2024-09-18T04:39:18.661 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4197999999999995e-05 2024-09-18T04:39:18.661 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.661 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.661 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:18.661 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541755+0000", 2024-09-18T04:39:18.661 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9818999999999999e-05 2024-09-18T04:39:18.661 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.661 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.661 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:18.661 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558217+0000", 2024-09-18T04:39:18.661 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016462234999999999 2024-09-18T04:39:18.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.662 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.662 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:18.662 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558284+0000", 2024-09-18T04:39:18.662 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6774999999999997e-05 2024-09-18T04:39:18.662 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:18.662 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:18.662 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:18.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.662 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.662 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:39 2.4 e51/17 2:304364d1:ns::rbd_data.133e813ab5bc.0000000000000004:head v 51'5, mlcod=51'3)", 2024-09-18T04:39:18.662 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:39:18.662 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 36.085512465000001, 2024-09-18T04:39:18.662 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016777558000000001, 2024-09-18T04:39:18.663 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:18.663 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:18.663 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:18.663 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.663 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:18.663 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:39:18.663 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:18.663 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.663 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.663 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:18.663 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:39:18.663 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:18.663 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.663 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.663 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:18.664 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545663+0000", 2024-09-18T04:39:18.664 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:18.664 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.664 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.664 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:18.664 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548286+0000", 2024-09-18T04:39:18.664 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026233390000000001 2024-09-18T04:39:18.664 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.664 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.664 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:18.664 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548305+0000", 2024-09-18T04:39:18.664 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8425000000000001e-05 2024-09-18T04:39:18.664 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.665 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.665 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:18.665 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548316+0000", 2024-09-18T04:39:18.665 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1163e-05 2024-09-18T04:39:18.665 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.665 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.665 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:18.665 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548379+0000", 2024-09-18T04:39:18.665 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2780000000000005e-05 2024-09-18T04:39:18.665 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.665 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.665 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:18.665 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548399+0000", 2024-09-18T04:39:18.666 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0080000000000001e-05 2024-09-18T04:39:18.666 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.666 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.666 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:18.666 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562383+0000", 2024-09-18T04:39:18.666 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013984671000000001 2024-09-18T04:39:18.666 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.666 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.666 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:18.666 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562442+0000", 2024-09-18T04:39:18.666 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8700999999999999e-05 2024-09-18T04:39:18.666 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:18.666 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:18.666 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:18.667 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.667 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.667 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:37 2.5 e51/28 2:bf39b8e4:ns::rbd_data.133e813ab5bc.0000000000000003:head v 51'5, mlcod=51'2)", 2024-09-18T04:39:18.667 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:39:18.667 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 36.082644195999997, 2024-09-18T04:39:18.667 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014264492, 2024-09-18T04:39:18.667 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:18.667 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:18.667 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:18.667 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.667 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:18.667 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:39:18.667 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:18.667 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.667 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.668 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:18.668 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:39:18.668 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:18.668 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.668 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.668 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:18.668 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548531+0000", 2024-09-18T04:39:18.668 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:18.668 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.668 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.668 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:18.668 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551500+0000", 2024-09-18T04:39:18.668 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0029688129999999998 2024-09-18T04:39:18.668 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.669 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.669 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:18.669 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551516+0000", 2024-09-18T04:39:18.669 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5994e-05 2024-09-18T04:39:18.669 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.669 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.669 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:18.669 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551526+0000", 2024-09-18T04:39:18.669 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9299999999999998e-06 2024-09-18T04:39:18.669 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.669 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.669 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:18.669 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551577+0000", 2024-09-18T04:39:18.669 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0871000000000001e-05 2024-09-18T04:39:18.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.670 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.670 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:18.670 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551602+0000", 2024-09-18T04:39:18.670 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4813000000000001e-05 2024-09-18T04:39:18.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.670 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.670 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:18.670 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562741+0000", 2024-09-18T04:39:18.670 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011139428999999999 2024-09-18T04:39:18.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.670 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.670 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:18.670 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562797+0000", 2024-09-18T04:39:18.670 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6329000000000002e-05 2024-09-18T04:39:18.671 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:18.671 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:18.671 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:18.671 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.671 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.671 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:49 2.4 e51/17 2:20f0b0c9:ns::rbd_data.133e813ab5bc.0000000000000009:head v 51'6, mlcod=51'4)", 2024-09-18T04:39:18.671 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:39:18.671 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 36.051209638000003, 2024-09-18T04:39:18.671 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013849046, 2024-09-18T04:39:18.671 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:18.671 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:18.671 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:18.671 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.671 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:18.672 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:39:18.672 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:18.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.672 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.672 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:18.672 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:39:18.672 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:18.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.672 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.672 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:18.672 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579966+0000", 2024-09-18T04:39:18.672 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:18.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.672 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.673 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:18.673 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582324+0000", 2024-09-18T04:39:18.673 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023578890000000002 2024-09-18T04:39:18.673 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.673 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.673 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:18.673 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582355+0000", 2024-09-18T04:39:18.673 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1485000000000001e-05 2024-09-18T04:39:18.673 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.673 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.673 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:18.673 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582371+0000", 2024-09-18T04:39:18.673 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6147999999999999e-05 2024-09-18T04:39:18.673 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.674 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.674 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:18.674 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582467+0000", 2024-09-18T04:39:18.674 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6059999999999998e-05 2024-09-18T04:39:18.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.674 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.674 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:18.674 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582505+0000", 2024-09-18T04:39:18.674 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7679000000000003e-05 2024-09-18T04:39:18.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.674 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.674 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:18.674 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593743+0000", 2024-09-18T04:39:18.674 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011238356 2024-09-18T04:39:18.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.675 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.675 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:18.675 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593816+0000", 2024-09-18T04:39:18.675 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2935000000000006e-05 2024-09-18T04:39:18.675 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:18.675 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:18.675 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:18.675 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.675 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.675 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:53 2.4 e51/17 2:39f7f174:ns::rbd_data.133e813ab5bc.000000000000000b:head v 51'7, mlcod=51'4)", 2024-09-18T04:39:18.675 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:39:18.675 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 36.036508482999999, 2024-09-18T04:39:18.675 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0079914159999999995, 2024-09-18T04:39:18.675 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:18.675 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:18.676 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:18.676 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.676 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:18.676 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:39:18.676 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:18.676 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.676 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.676 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:18.676 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:39:18.676 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:18.676 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.676 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.676 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:18.677 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594667+0000", 2024-09-18T04:39:18.677 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:18.677 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.677 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.677 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:18.677 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596446+0000", 2024-09-18T04:39:18.677 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001779033 2024-09-18T04:39:18.677 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.677 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.677 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:18.677 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596472+0000", 2024-09-18T04:39:18.677 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6120000000000001e-05 2024-09-18T04:39:18.677 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.677 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.678 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:18.678 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596500+0000", 2024-09-18T04:39:18.678 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7518e-05 2024-09-18T04:39:18.678 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.678 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.678 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:18.678 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596564+0000", 2024-09-18T04:39:18.678 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3880999999999994e-05 2024-09-18T04:39:18.678 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.678 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.678 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:18.678 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596596+0000", 2024-09-18T04:39:18.678 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2638999999999997e-05 2024-09-18T04:39:18.678 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.679 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.679 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:18.679 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602605+0000", 2024-09-18T04:39:18.679 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0060080699999999999 2024-09-18T04:39:18.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.679 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.679 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:18.679 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602660+0000", 2024-09-18T04:39:18.679 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5396000000000003e-05 2024-09-18T04:39:18.679 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:18.679 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:18.679 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:18.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.679 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.680 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:60 2.5 e51/28 2:a06f51e4:ns::rbd_data.133e813ab5bc.000000000000000f:head v 51'6, mlcod=51'4)", 2024-09-18T04:39:18.680 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:39:18.680 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 36.021691498000003, 2024-09-18T04:39:18.680 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0093954989999999999, 2024-09-18T04:39:18.680 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:18.680 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:18.680 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:18.680 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.680 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:18.680 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:39:18.680 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:18.680 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.680 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.680 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:18.680 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:39:18.681 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:18.681 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.681 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.681 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:18.681 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609483+0000", 2024-09-18T04:39:18.681 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:18.681 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.681 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.681 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:18.681 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611767+0000", 2024-09-18T04:39:18.681 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002283724 2024-09-18T04:39:18.681 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.681 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.681 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:18.682 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611791+0000", 2024-09-18T04:39:18.682 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3547e-05 2024-09-18T04:39:18.682 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.682 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.682 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:18.682 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611800+0000", 2024-09-18T04:39:18.682 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4979999999999995e-06 2024-09-18T04:39:18.682 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.682 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.682 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:18.682 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611864+0000", 2024-09-18T04:39:18.682 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3974999999999996e-05 2024-09-18T04:39:18.682 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.682 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.682 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:18.683 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611887+0000", 2024-09-18T04:39:18.683 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2633000000000002e-05 2024-09-18T04:39:18.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.683 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.683 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:18.683 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.618777+0000", 2024-09-18T04:39:18.683 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0068897979999999999 2024-09-18T04:39:18.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.683 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.683 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:18.683 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.618881+0000", 2024-09-18T04:39:18.683 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010437699999999999 2024-09-18T04:39:18.683 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:18.683 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:18.684 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:18.684 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.684 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.684 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:663 2.4 e57/17 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 57'13, mlcod=57'11)", 2024-09-18T04:39:18.684 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.355300+0000", 2024-09-18T04:39:18.684 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 27.275876563000001, 2024-09-18T04:39:18.684 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0049192760000000002, 2024-09-18T04:39:18.684 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:18.684 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:18.684 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:18.684 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.684 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:18.684 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355300+0000", 2024-09-18T04:39:18.685 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:18.685 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.685 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.685 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:18.685 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355300+0000", 2024-09-18T04:39:18.685 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:18.685 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.685 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.685 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:18.685 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355299+0000", 2024-09-18T04:39:18.685 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:18.685 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.685 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.685 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:18.685 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355876+0000", 2024-09-18T04:39:18.686 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00057730900000000003 2024-09-18T04:39:18.686 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.686 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.686 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:18.686 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355897+0000", 2024-09-18T04:39:18.686 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0477000000000001e-05 2024-09-18T04:39:18.686 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.686 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.686 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:18.686 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355905+0000", 2024-09-18T04:39:18.686 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5739999999999996e-06 2024-09-18T04:39:18.686 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.686 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.686 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:18.687 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355989+0000", 2024-09-18T04:39:18.687 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4002e-05 2024-09-18T04:39:18.687 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.687 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.687 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:18.687 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.356041+0000", 2024-09-18T04:39:18.687 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1347000000000002e-05 2024-09-18T04:39:18.687 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.687 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.687 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:18.687 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.360172+0000", 2024-09-18T04:39:18.687 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0041315090000000002 2024-09-18T04:39:18.687 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.687 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.688 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:18.688 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.360220+0000", 2024-09-18T04:39:18.688 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7584999999999999e-05 2024-09-18T04:39:18.688 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:18.688 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:18.688 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:18.688 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.688 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.688 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:679 2.5 e57/28 2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head v 57'20, mlcod=57'18)", 2024-09-18T04:39:18.688 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.445316+0000", 2024-09-18T04:39:18.688 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 27.185861085999999, 2024-09-18T04:39:18.688 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0047016760000000001, 2024-09-18T04:39:18.688 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:18.688 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:18.688 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:18.689 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.689 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:18.689 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445316+0000", 2024-09-18T04:39:18.689 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:18.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.689 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.689 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:18.689 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445316+0000", 2024-09-18T04:39:18.689 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:18.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.689 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.689 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:18.689 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445314+0000", 2024-09-18T04:39:18.689 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:18.690 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.690 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.690 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:18.690 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445947+0000", 2024-09-18T04:39:18.690 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00063298900000000003 2024-09-18T04:39:18.690 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.690 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.690 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:18.690 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445968+0000", 2024-09-18T04:39:18.690 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.054e-05 2024-09-18T04:39:18.690 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.690 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.690 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:18.690 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445981+0000", 2024-09-18T04:39:18.690 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2697000000000001e-05 2024-09-18T04:39:18.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.691 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.691 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:18.691 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.446102+0000", 2024-09-18T04:39:18.691 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012189300000000001 2024-09-18T04:39:18.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.691 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.691 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:18.691 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.446151+0000", 2024-09-18T04:39:18.691 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8279000000000003e-05 2024-09-18T04:39:18.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.691 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.691 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:18.692 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.449953+0000", 2024-09-18T04:39:18.692 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003801979 2024-09-18T04:39:18.692 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.692 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.692 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:18.692 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.450018+0000", 2024-09-18T04:39:18.692 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4873e-05 2024-09-18T04:39:18.692 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:18.692 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:18.692 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:18.692 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.692 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.692 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:699 2.4 e57/17 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 57'17, mlcod=57'15)", 2024-09-18T04:39:18.692 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.566073+0000", 2024-09-18T04:39:18.692 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 27.065103572999998, 2024-09-18T04:39:18.693 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055398380000000001, 2024-09-18T04:39:18.693 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:18.693 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:18.693 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:18.693 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.693 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:18.693 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566073+0000", 2024-09-18T04:39:18.693 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:18.693 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.693 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.693 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:18.693 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566073+0000", 2024-09-18T04:39:18.693 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:18.694 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.694 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.694 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:18.694 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566071+0000", 2024-09-18T04:39:18.694 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:18.694 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.694 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.694 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:18.694 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566637+0000", 2024-09-18T04:39:18.694 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00056516699999999999 2024-09-18T04:39:18.694 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.694 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.694 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:18.694 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566668+0000", 2024-09-18T04:39:18.695 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1544999999999997e-05 2024-09-18T04:39:18.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.695 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.695 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:18.695 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566685+0000", 2024-09-18T04:39:18.695 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6857000000000001e-05 2024-09-18T04:39:18.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.695 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.695 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:18.695 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566816+0000", 2024-09-18T04:39:18.695 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013092799999999999 2024-09-18T04:39:18.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.695 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.695 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:18.696 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566875+0000", 2024-09-18T04:39:18.696 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9583999999999997e-05 2024-09-18T04:39:18.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.696 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.696 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:18.696 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.571567+0000", 2024-09-18T04:39:18.696 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.004691682 2024-09-18T04:39:18.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.696 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.696 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:18.696 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.571613+0000", 2024-09-18T04:39:18.696 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6125000000000003e-05 2024-09-18T04:39:18.696 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:18.697 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:18.697 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:18.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.697 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.697 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:98 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'27, mlcod=60'25)", 2024-09-18T04:39:18.697 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.847978+0000", 2024-09-18T04:39:18.697 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.783199014000001, 2024-09-18T04:39:18.697 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0044617709999999998, 2024-09-18T04:39:18.697 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:18.697 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:18.697 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:18.697 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.697 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:18.697 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.847978+0000", 2024-09-18T04:39:18.697 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:18.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.698 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.698 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:18.698 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.847978+0000", 2024-09-18T04:39:18.698 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:18.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.698 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.698 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:18.698 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.847977+0000", 2024-09-18T04:39:18.698 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:18.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.698 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.698 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:18.698 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849038+0000", 2024-09-18T04:39:18.699 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0010614820000000001 2024-09-18T04:39:18.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.699 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.699 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:18.699 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849065+0000", 2024-09-18T04:39:18.699 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7127999999999998e-05 2024-09-18T04:39:18.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.699 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.699 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:18.699 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849082+0000", 2024-09-18T04:39:18.699 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7170999999999999e-05 2024-09-18T04:39:18.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.699 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.699 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:18.700 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849188+0000", 2024-09-18T04:39:18.700 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010605800000000001 2024-09-18T04:39:18.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.700 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.700 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:18.700 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849233+0000", 2024-09-18T04:39:18.700 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4579000000000001e-05 2024-09-18T04:39:18.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.700 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.700 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:18.700 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.852370+0000", 2024-09-18T04:39:18.700 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00313751 2024-09-18T04:39:18.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.700 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.701 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:18.701 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.852440+0000", 2024-09-18T04:39:18.701 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9238000000000004e-05 2024-09-18T04:39:18.701 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:18.701 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:18.701 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:18.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.701 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.701 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:100 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'29, mlcod=64'27)", 2024-09-18T04:39:18.701 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.858766+0000", 2024-09-18T04:39:18.701 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.772410685000001, 2024-09-18T04:39:18.701 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0061964840000000004, 2024-09-18T04:39:18.701 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:18.701 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:18.702 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:18.702 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.702 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:18.702 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.858766+0000", 2024-09-18T04:39:18.702 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:18.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.702 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.702 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:18.702 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.858766+0000", 2024-09-18T04:39:18.702 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:18.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.702 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.702 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:18.702 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.858765+0000", 2024-09-18T04:39:18.703 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:18.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.703 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.703 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:18.703 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861612+0000", 2024-09-18T04:39:18.703 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028473169999999998 2024-09-18T04:39:18.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.703 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.703 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:18.703 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861643+0000", 2024-09-18T04:39:18.703 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1356000000000003e-05 2024-09-18T04:39:18.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.703 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.703 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:18.703 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861656+0000", 2024-09-18T04:39:18.704 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2899e-05 2024-09-18T04:39:18.704 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.704 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.704 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:18.704 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861773+0000", 2024-09-18T04:39:18.704 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011637999999999999 2024-09-18T04:39:18.704 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.704 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.704 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:18.704 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861808+0000", 2024-09-18T04:39:18.704 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5002999999999997e-05 2024-09-18T04:39:18.704 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.704 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.704 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:18.705 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.864876+0000", 2024-09-18T04:39:18.705 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030680719999999998 2024-09-18T04:39:18.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.705 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.705 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:18.705 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.864963+0000", 2024-09-18T04:39:18.705 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7114000000000007e-05 2024-09-18T04:39:18.705 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:18.705 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:18.705 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:18.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.705 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.705 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:102 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'31, mlcod=64'29)", 2024-09-18T04:39:18.705 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.872098+0000", 2024-09-18T04:39:18.706 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.759079428, 2024-09-18T04:39:18.706 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0064676450000000002, 2024-09-18T04:39:18.706 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:18.706 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:18.706 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:18.706 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.706 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:18.706 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.872098+0000", 2024-09-18T04:39:18.706 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:18.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.706 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.706 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:18.706 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.872098+0000", 2024-09-18T04:39:18.706 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:18.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.707 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.707 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:18.707 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.872096+0000", 2024-09-18T04:39:18.707 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:18.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.707 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.707 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:18.707 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875176+0000", 2024-09-18T04:39:18.707 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030799809999999999 2024-09-18T04:39:18.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.707 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.707 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:18.707 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875204+0000", 2024-09-18T04:39:18.708 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8025999999999999e-05 2024-09-18T04:39:18.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.708 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.708 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:18.708 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875216+0000", 2024-09-18T04:39:18.708 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2072999999999999e-05 2024-09-18T04:39:18.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.708 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.708 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:18.708 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875307+0000", 2024-09-18T04:39:18.708 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0373999999999999e-05 2024-09-18T04:39:18.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.708 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.708 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:18.709 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875349+0000", 2024-09-18T04:39:18.709 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2299000000000003e-05 2024-09-18T04:39:18.709 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.709 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.709 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:18.709 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.878496+0000", 2024-09-18T04:39:18.709 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0031468960000000002 2024-09-18T04:39:18.709 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.709 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.709 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:18.709 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.878565+0000", 2024-09-18T04:39:18.709 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9465e-05 2024-09-18T04:39:18.709 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:18.709 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:18.710 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:18.710 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.710 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.710 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:104 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'33, mlcod=64'31)", 2024-09-18T04:39:18.710 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.885100+0000", 2024-09-18T04:39:18.710 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.746076893000001, 2024-09-18T04:39:18.710 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0051888159999999997, 2024-09-18T04:39:18.710 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:18.710 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:18.710 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:18.710 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.710 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:18.710 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.885100+0000", 2024-09-18T04:39:18.710 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:18.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.711 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.711 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:18.711 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.885100+0000", 2024-09-18T04:39:18.711 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:18.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.711 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.711 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:18.711 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.885098+0000", 2024-09-18T04:39:18.711 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:18.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.711 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.711 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:18.711 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886809+0000", 2024-09-18T04:39:18.712 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017110459999999999 2024-09-18T04:39:18.712 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.712 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.712 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:18.712 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886842+0000", 2024-09-18T04:39:18.712 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2638999999999997e-05 2024-09-18T04:39:18.712 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.712 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.712 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:18.712 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886855+0000", 2024-09-18T04:39:18.712 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3380000000000001e-05 2024-09-18T04:39:18.712 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.712 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.712 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:18.713 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886939+0000", 2024-09-18T04:39:18.713 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3733999999999995e-05 2024-09-18T04:39:18.713 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.713 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.713 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:18.713 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886981+0000", 2024-09-18T04:39:18.713 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1956000000000003e-05 2024-09-18T04:39:18.713 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.713 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.713 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:18.713 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.890201+0000", 2024-09-18T04:39:18.713 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003219927 2024-09-18T04:39:18.713 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.713 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.714 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:18.714 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.890289+0000", 2024-09-18T04:39:18.714 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7997999999999993e-05 2024-09-18T04:39:18.714 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:18.714 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:18.714 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:18.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.714 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.714 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:145 2.4 e64/62 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 64'37, mlcod=64'35)", 2024-09-18T04:39:18.714 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:01.099398+0000", 2024-09-18T04:39:18.714 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.531779405000002, 2024-09-18T04:39:18.714 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0050078459999999998, 2024-09-18T04:39:18.714 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:18.714 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:18.714 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:18.715 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.715 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:18.715 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.099398+0000", 2024-09-18T04:39:18.715 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:18.715 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.715 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.715 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:18.715 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.099398+0000", 2024-09-18T04:39:18.715 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:18.715 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.715 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.715 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:18.715 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.099396+0000", 2024-09-18T04:39:18.715 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:18.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.716 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.716 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:18.716 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100199+0000", 2024-09-18T04:39:18.716 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00080338000000000002 2024-09-18T04:39:18.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.716 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.716 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:18.716 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100230+0000", 2024-09-18T04:39:18.716 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0716000000000002e-05 2024-09-18T04:39:18.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.716 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.716 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:18.716 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100243+0000", 2024-09-18T04:39:18.717 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3359e-05 2024-09-18T04:39:18.717 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.717 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.717 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:18.717 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100337+0000", 2024-09-18T04:39:18.717 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4387000000000003e-05 2024-09-18T04:39:18.717 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.717 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.717 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:18.717 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100396+0000", 2024-09-18T04:39:18.717 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8164000000000001e-05 2024-09-18T04:39:18.717 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.717 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.717 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:18.718 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.104309+0000", 2024-09-18T04:39:18.718 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0039137800000000004 2024-09-18T04:39:18.718 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.718 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.718 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:18.718 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.104405+0000", 2024-09-18T04:39:18.718 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6039999999999995e-05 2024-09-18T04:39:18.718 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:18.718 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:18.718 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:18.718 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.718 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.718 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:147 2.4 e64/62 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 64'39, mlcod=64'37)", 2024-09-18T04:39:18.718 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:01.110658+0000", 2024-09-18T04:39:18.719 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.520519316000001, 2024-09-18T04:39:18.719 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0046316910000000003, 2024-09-18T04:39:18.719 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:18.719 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:18.719 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:18.719 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.719 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:18.719 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.110658+0000", 2024-09-18T04:39:18.719 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:18.719 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.719 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.719 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:18.719 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.110658+0000", 2024-09-18T04:39:18.719 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:18.720 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.720 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.720 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:18.720 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.110656+0000", 2024-09-18T04:39:18.720 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:18.720 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.720 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.720 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:18.720 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111444+0000", 2024-09-18T04:39:18.720 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00078762400000000001 2024-09-18T04:39:18.720 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.720 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.720 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:18.720 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111474+0000", 2024-09-18T04:39:18.721 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0085000000000001e-05 2024-09-18T04:39:18.721 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.721 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.721 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:18.721 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111486+0000", 2024-09-18T04:39:18.721 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2645e-05 2024-09-18T04:39:18.721 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.721 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.721 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:18.721 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111581+0000", 2024-09-18T04:39:18.721 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4432000000000007e-05 2024-09-18T04:39:18.721 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.721 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.721 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:18.722 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111631+0000", 2024-09-18T04:39:18.722 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9901000000000002e-05 2024-09-18T04:39:18.722 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.722 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.722 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:18.722 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.115209+0000", 2024-09-18T04:39:18.722 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003577916 2024-09-18T04:39:18.722 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:18.722 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:18.722 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:18.722 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.115289+0000", 2024-09-18T04:39:18.722 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0704999999999994e-05 2024-09-18T04:39:18.722 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:18.722 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:18.722 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:18.723 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:18.723 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:18.723 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:18.724 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:39:18.725 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:18.721+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:18.825 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:18.817+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:18.841 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:18.841 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:18.841 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:39:18.841 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:18.851 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:39:18.926 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:18.921+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:18.961 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:18.961 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:18.961 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:39:18.961 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:39:18.961 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:18.971 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:39:19.026 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:19.021+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:19.083 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:19.084 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:39:19.084 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:39:19.084 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:39:19.084 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.084 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:33 2.1 e51/24 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head v 51'4, mlcod=51'2)", 2024-09-18T04:39:19.084 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:39:19.084 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 36.799643254000003, 2024-09-18T04:39:19.084 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018702106, 2024-09-18T04:39:19.084 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:19.084 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:19.084 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:19.085 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.085 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:19.085 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:39:19.085 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.085 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.085 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.085 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:19.085 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:39:19.085 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.085 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.085 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.085 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:19.085 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279197+0000", 2024-09-18T04:39:19.085 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:19.086 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.086 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.086 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:19.086 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282682+0000", 2024-09-18T04:39:19.086 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0034848689999999998 2024-09-18T04:39:19.086 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.086 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.086 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:19.086 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282709+0000", 2024-09-18T04:39:19.086 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7599000000000001e-05 2024-09-18T04:39:19.086 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.086 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.086 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:19.086 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282720+0000", 2024-09-18T04:39:19.087 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.092e-05 2024-09-18T04:39:19.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.087 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.087 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:19.087 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282791+0000", 2024-09-18T04:39:19.087 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1268999999999998e-05 2024-09-18T04:39:19.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.087 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.087 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:19.087 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282843+0000", 2024-09-18T04:39:19.087 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1771999999999998e-05 2024-09-18T04:39:19.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.087 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.088 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:19.088 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.297830+0000", 2024-09-18T04:39:19.088 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014987228999999999 2024-09-18T04:39:19.088 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.088 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.088 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:19.088 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.297901+0000", 2024-09-18T04:39:19.088 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0209999999999994e-05 2024-09-18T04:39:19.088 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.088 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:19.088 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.088 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.088 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.089 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:38 2.0 e51/17 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head v 51'10, mlcod=51'8)", 2024-09-18T04:39:19.089 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:39:19.089 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 36.798224824000002, 2024-09-18T04:39:19.089 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.023984472, 2024-09-18T04:39:19.089 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:19.089 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:19.089 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:19.089 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.089 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:19.089 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:39:19.089 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.089 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.089 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.089 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:19.090 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:39:19.090 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.090 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.090 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:19.090 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280614+0000", 2024-09-18T04:39:19.090 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:19.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.090 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.090 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:19.090 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293094+0000", 2024-09-18T04:39:19.090 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012479571 2024-09-18T04:39:19.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.090 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.091 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:19.091 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293111+0000", 2024-09-18T04:39:19.091 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7354999999999999e-05 2024-09-18T04:39:19.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.091 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.091 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:19.091 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293121+0000", 2024-09-18T04:39:19.091 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6979999999999993e-06 2024-09-18T04:39:19.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.091 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.091 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:19.091 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293180+0000", 2024-09-18T04:39:19.091 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9333000000000003e-05 2024-09-18T04:39:19.092 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.092 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.092 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:19.092 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293203+0000", 2024-09-18T04:39:19.092 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2633000000000002e-05 2024-09-18T04:39:19.092 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.092 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.092 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:19.092 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304534+0000", 2024-09-18T04:39:19.092 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011330761 2024-09-18T04:39:19.092 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.092 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.092 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:19.092 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304601+0000", 2024-09-18T04:39:19.093 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7657000000000006e-05 2024-09-18T04:39:19.093 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.093 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:19.093 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.093 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.093 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.093 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:46 2.6 2:7ba25245:::rbd_data.125fc09df6d1.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:19.093 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:39:19.093 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 36.770682024000003, 2024-09-18T04:39:19.093 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.035604595000000003, 2024-09-18T04:39:19.093 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:19.093 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:19.093 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:19.093 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:19.094 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:19.094 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 46 2024-09-18T04:39:19.094 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.094 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:19.094 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.094 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:19.094 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:39:19.094 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.094 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.094 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.094 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:19.094 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:39:19.094 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.094 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.095 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.095 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:19.095 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308158+0000", 2024-09-18T04:39:19.095 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:19.095 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.095 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.095 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:19.095 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313191+0000", 2024-09-18T04:39:19.095 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0050323929999999996 2024-09-18T04:39:19.095 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.095 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.095 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:19.095 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313210+0000", 2024-09-18T04:39:19.096 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9673e-05 2024-09-18T04:39:19.096 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.096 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.096 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:19.096 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313221+0000", 2024-09-18T04:39:19.096 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0444e-05 2024-09-18T04:39:19.096 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.096 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.096 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:19.096 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313294+0000", 2024-09-18T04:39:19.096 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2650000000000004e-05 2024-09-18T04:39:19.096 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.096 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.096 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:19.097 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313493+0000", 2024-09-18T04:39:19.097 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019968300000000001 2024-09-18T04:39:19.097 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.097 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.097 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:19.097 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313666+0000", 2024-09-18T04:39:19.097 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00017248599999999999 2024-09-18T04:39:19.097 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.097 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.097 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:19.097 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.323175+0000", 2024-09-18T04:39:19.097 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0095088819999999998 2024-09-18T04:39:19.097 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.097 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.098 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:19.098 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343651+0000", 2024-09-18T04:39:19.098 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020475892999999998 2024-09-18T04:39:19.098 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.098 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.098 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:19.098 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343715+0000", 2024-09-18T04:39:19.098 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4443999999999995e-05 2024-09-18T04:39:19.098 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.098 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.098 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:19.098 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343764+0000", 2024-09-18T04:39:19.098 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9280000000000003e-05 2024-09-18T04:39:19.098 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.099 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:19.099 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.099 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.099 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.099 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:47 2.4 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:19.099 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:39:19.099 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 36.765599987000002, 2024-09-18T04:39:19.099 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.032570693999999997, 2024-09-18T04:39:19.099 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:19.099 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:19.099 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:19.099 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:19.099 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:19.099 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 47 2024-09-18T04:39:19.100 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.100 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:19.100 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.100 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:19.100 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:39:19.100 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.100 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.100 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.100 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:19.100 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:39:19.100 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.100 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.100 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.100 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:19.101 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313240+0000", 2024-09-18T04:39:19.101 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:19.101 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.101 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.101 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:19.101 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325825+0000", 2024-09-18T04:39:19.101 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012584425999999999 2024-09-18T04:39:19.101 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.101 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.101 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:19.101 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325848+0000", 2024-09-18T04:39:19.101 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3025999999999999e-05 2024-09-18T04:39:19.101 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.101 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.102 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:19.102 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325863+0000", 2024-09-18T04:39:19.102 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5152e-05 2024-09-18T04:39:19.102 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.102 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.102 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:19.102 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325938+0000", 2024-09-18T04:39:19.102 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4982999999999997e-05 2024-09-18T04:39:19.102 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.102 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.102 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:19.102 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326090+0000", 2024-09-18T04:39:19.102 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015207900000000001 2024-09-18T04:39:19.102 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.103 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.103 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:19.103 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326242+0000", 2024-09-18T04:39:19.103 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015163500000000001 2024-09-18T04:39:19.103 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.103 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.103 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:19.103 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343015+0000", 2024-09-18T04:39:19.103 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016773414 2024-09-18T04:39:19.103 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.103 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.103 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:19.103 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345664+0000", 2024-09-18T04:39:19.103 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026487870000000001 2024-09-18T04:39:19.104 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.104 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.104 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:19.104 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345739+0000", 2024-09-18T04:39:19.104 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5018999999999994e-05 2024-09-18T04:39:19.104 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.104 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.104 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:19.104 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345812+0000", 2024-09-18T04:39:19.104 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3547000000000006e-05 2024-09-18T04:39:19.104 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.104 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:19.104 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.104 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.105 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.105 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:48 2.6 2:61db7a3a:::rbd_data.125fc09df6d1.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:19.105 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:39:19.105 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 36.752943315000003, 2024-09-18T04:39:19.105 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.025064388, 2024-09-18T04:39:19.105 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:19.105 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:19.105 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:19.105 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:19.105 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:19.105 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 48 2024-09-18T04:39:19.105 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.105 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:19.105 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.106 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:19.106 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:39:19.106 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.106 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.106 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.106 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:19.106 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:39:19.106 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.106 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.106 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.106 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:19.106 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325897+0000", 2024-09-18T04:39:19.106 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:19.106 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.107 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.107 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:19.107 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333697+0000", 2024-09-18T04:39:19.107 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0078002649999999998 2024-09-18T04:39:19.107 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.107 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.107 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:19.107 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333716+0000", 2024-09-18T04:39:19.107 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8910999999999999e-05 2024-09-18T04:39:19.107 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.107 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.107 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:19.107 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333728+0000", 2024-09-18T04:39:19.107 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1936000000000001e-05 2024-09-18T04:39:19.108 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.108 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.108 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:19.108 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333792+0000", 2024-09-18T04:39:19.108 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4010999999999994e-05 2024-09-18T04:39:19.108 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.108 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.108 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:19.108 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.334082+0000", 2024-09-18T04:39:19.108 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00028955799999999998 2024-09-18T04:39:19.108 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.108 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.108 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:19.109 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.334326+0000", 2024-09-18T04:39:19.109 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00024399099999999999 2024-09-18T04:39:19.109 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.109 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.109 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:19.109 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350745+0000", 2024-09-18T04:39:19.109 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016419474999999999 2024-09-18T04:39:19.109 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.109 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.109 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:19.109 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350808+0000", 2024-09-18T04:39:19.109 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3244999999999995e-05 2024-09-18T04:39:19.109 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.109 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.110 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:19.110 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350892+0000", 2024-09-18T04:39:19.110 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4074999999999997e-05 2024-09-18T04:39:19.110 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.110 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.110 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:19.110 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350963+0000", 2024-09-18T04:39:19.110 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0412000000000003e-05 2024-09-18T04:39:19.110 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.110 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:19.110 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.110 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.110 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.110 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:51 2.6 2:775d7b11:::rbd_data.125fc09df6d1.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:19.111 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:39:19.111 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 36.745091178999999, 2024-09-18T04:39:19.111 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.023824221, 2024-09-18T04:39:19.111 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:19.111 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:19.111 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:19.111 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:19.111 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:19.111 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 51 2024-09-18T04:39:19.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.111 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:19.111 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.112 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:19.112 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:39:19.112 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.112 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.112 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.112 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:19.112 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:39:19.113 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.113 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.113 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.113 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:19.113 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333749+0000", 2024-09-18T04:39:19.113 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:19.114 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.114 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.114 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:19.114 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340899+0000", 2024-09-18T04:39:19.114 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0071500340000000004 2024-09-18T04:39:19.114 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.115 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.115 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:19.115 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340929+0000", 2024-09-18T04:39:19.115 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0093000000000001e-05 2024-09-18T04:39:19.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.115 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.115 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:19.115 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340943+0000", 2024-09-18T04:39:19.115 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4019000000000001e-05 2024-09-18T04:39:19.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.115 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.115 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:19.116 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343834+0000", 2024-09-18T04:39:19.116 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028918120000000001 2024-09-18T04:39:19.116 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.116 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.116 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:19.116 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344020+0000", 2024-09-18T04:39:19.116 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000185892 2024-09-18T04:39:19.116 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.116 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.116 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:19.116 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344190+0000", 2024-09-18T04:39:19.116 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016946800000000001 2024-09-18T04:39:19.116 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.116 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.117 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:19.117 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.351291+0000", 2024-09-18T04:39:19.117 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.007100805 2024-09-18T04:39:19.117 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.117 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.117 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:19.117 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357470+0000", 2024-09-18T04:39:19.117 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0061797570000000001 2024-09-18T04:39:19.117 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.117 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.117 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:19.117 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357533+0000", 2024-09-18T04:39:19.117 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2906999999999999e-05 2024-09-18T04:39:19.117 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.118 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.118 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:19.118 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357575+0000", 2024-09-18T04:39:19.118 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1403999999999999e-05 2024-09-18T04:39:19.118 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.118 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:19.118 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.118 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.118 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.118 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:56 2.4 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:19.118 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:39:19.118 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 36.729046085999997, 2024-09-18T04:39:19.118 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.034295870999999999, 2024-09-18T04:39:19.118 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:19.119 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:19.119 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:19.119 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:19.119 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:19.119 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 56 2024-09-18T04:39:19.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.119 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:19.119 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.119 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:19.120 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:39:19.120 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.120 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.120 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.120 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:19.121 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:39:19.121 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.121 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.121 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.121 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:19.121 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349794+0000", 2024-09-18T04:39:19.122 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:19.122 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.122 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.122 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:19.122 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366632+0000", 2024-09-18T04:39:19.122 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01683755 2024-09-18T04:39:19.123 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.123 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.123 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:19.123 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366669+0000", 2024-09-18T04:39:19.123 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7351000000000002e-05 2024-09-18T04:39:19.123 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.123 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.123 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:19.124 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366686+0000", 2024-09-18T04:39:19.124 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6507999999999998e-05 2024-09-18T04:39:19.124 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.124 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.124 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:19.124 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366962+0000", 2024-09-18T04:39:19.124 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00027655900000000001 2024-09-18T04:39:19.124 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.124 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.124 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:19.124 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.367452+0000", 2024-09-18T04:39:19.124 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00049025299999999996 2024-09-18T04:39:19.124 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.124 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.125 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:19.125 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368113+0000", 2024-09-18T04:39:19.125 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00066093399999999996 2024-09-18T04:39:19.125 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.125 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.125 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:19.125 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383719+0000", 2024-09-18T04:39:19.125 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015606106999999999 2024-09-18T04:39:19.125 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.125 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.125 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:19.125 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384035+0000", 2024-09-18T04:39:19.125 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00031590399999999999 2024-09-18T04:39:19.125 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.126 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.126 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:19.126 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384067+0000", 2024-09-18T04:39:19.126 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.129e-05 2024-09-18T04:39:19.126 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.126 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.126 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:19.126 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384091+0000", 2024-09-18T04:39:19.126 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4788000000000001e-05 2024-09-18T04:39:19.126 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.126 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:19.126 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.126 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.126 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.127 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:57 2.0 e51/17 2:0cdd070e:::rbd_data.125fc09df6d1.000000000000000d:head v 51'21, mlcod=51'19)", 2024-09-18T04:39:19.127 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:39:19.127 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 36.721218598999997, 2024-09-18T04:39:19.127 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.025216311000000002, 2024-09-18T04:39:19.127 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:19.127 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:19.127 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:19.127 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.127 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:19.127 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:39:19.127 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.127 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.127 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.127 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:19.128 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:39:19.128 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.128 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.128 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.128 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:19.128 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:39:19.128 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:19.128 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.128 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.128 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:19.128 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363844+0000", 2024-09-18T04:39:19.128 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0062218450000000002 2024-09-18T04:39:19.128 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.129 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.129 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:19.129 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363879+0000", 2024-09-18T04:39:19.129 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4382999999999999e-05 2024-09-18T04:39:19.129 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.129 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.129 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:19.129 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363893+0000", 2024-09-18T04:39:19.129 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4348999999999999e-05 2024-09-18T04:39:19.129 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.129 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.129 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:19.129 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363977+0000", 2024-09-18T04:39:19.130 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3499999999999997e-05 2024-09-18T04:39:19.130 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.130 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.130 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:19.130 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364009+0000", 2024-09-18T04:39:19.130 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2292000000000002e-05 2024-09-18T04:39:19.130 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.130 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.130 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:19.130 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.382781+0000", 2024-09-18T04:39:19.130 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018771626 2024-09-18T04:39:19.130 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.130 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.131 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:19.131 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.382839+0000", 2024-09-18T04:39:19.131 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8758000000000002e-05 2024-09-18T04:39:19.131 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.131 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:19.131 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.131 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.131 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.131 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:59 2.0 e51/17 2:19c52cde:::rbd_data.125fc09df6d1.000000000000000f:head v 51'22, mlcod=51'19)", 2024-09-18T04:39:19.131 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:39:19.131 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 36.714558672999999, 2024-09-18T04:39:19.131 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.019497112, 2024-09-18T04:39:19.131 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:19.131 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:19.132 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:19.132 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.132 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:19.132 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:39:19.132 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.132 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.132 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.132 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:19.132 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:39:19.132 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.132 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.132 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.132 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:19.132 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364282+0000", 2024-09-18T04:39:19.133 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:19.133 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.133 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.133 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:19.133 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371823+0000", 2024-09-18T04:39:19.133 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0075411829999999999 2024-09-18T04:39:19.133 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.133 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.133 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:19.133 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371837+0000", 2024-09-18T04:39:19.133 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4336e-05 2024-09-18T04:39:19.133 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.133 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.134 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:19.134 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371846+0000", 2024-09-18T04:39:19.134 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9770000000000006e-06 2024-09-18T04:39:19.134 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.134 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.134 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:19.134 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.373566+0000", 2024-09-18T04:39:19.134 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017192539999999999 2024-09-18T04:39:19.134 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.134 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.134 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:19.134 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.373595+0000", 2024-09-18T04:39:19.134 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8975999999999998e-05 2024-09-18T04:39:19.134 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.135 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.135 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:19.135 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383740+0000", 2024-09-18T04:39:19.135 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01014525 2024-09-18T04:39:19.135 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.135 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.135 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:19.135 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383780+0000", 2024-09-18T04:39:19.135 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0275999999999999e-05 2024-09-18T04:39:19.135 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.135 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:19.135 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.135 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.136 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.136 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:34 2.6 2:66289171:ns::rbd_data.133e813ab5bc.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:19.136 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.524015+0000", 2024-09-18T04:39:19.136 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 36.554827064000001, 2024-09-18T04:39:19.136 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027582111999999999, 2024-09-18T04:39:19.136 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:19.136 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:19.136 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:19.136 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:19.136 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:19.136 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 34 2024-09-18T04:39:19.136 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.136 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:19.136 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.137 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:19.137 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524015+0000", 2024-09-18T04:39:19.137 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.137 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.137 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.137 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:19.137 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524015+0000", 2024-09-18T04:39:19.137 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.137 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.137 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.137 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:19.137 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524012+0000", 2024-09-18T04:39:19.137 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:19.138 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.138 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.138 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:19.138 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534956+0000", 2024-09-18T04:39:19.138 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010943403000000001 2024-09-18T04:39:19.138 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.138 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.138 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:19.138 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534978+0000", 2024-09-18T04:39:19.138 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1965999999999999e-05 2024-09-18T04:39:19.138 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.138 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.138 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:19.138 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534990+0000", 2024-09-18T04:39:19.139 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2065999999999999e-05 2024-09-18T04:39:19.139 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.139 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.139 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:19.139 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535065+0000", 2024-09-18T04:39:19.139 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5523000000000002e-05 2024-09-18T04:39:19.139 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.139 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.139 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:19.139 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535208+0000", 2024-09-18T04:39:19.139 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014309500000000001 2024-09-18T04:39:19.139 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.139 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.139 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:19.140 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535343+0000", 2024-09-18T04:39:19.140 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013437400000000001 2024-09-18T04:39:19.140 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.140 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.140 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:19.140 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.546576+0000", 2024-09-18T04:39:19.140 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011233425999999999 2024-09-18T04:39:19.140 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.140 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.140 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:19.140 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551498+0000", 2024-09-18T04:39:19.140 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0049219349999999997 2024-09-18T04:39:19.140 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.140 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.141 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:19.141 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551554+0000", 2024-09-18T04:39:19.141 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6400999999999998e-05 2024-09-18T04:39:19.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.141 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.141 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:19.141 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551597+0000", 2024-09-18T04:39:19.141 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2283000000000002e-05 2024-09-18T04:39:19.141 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.141 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:19.141 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.141 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.141 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:33 2.1 e51/24 2:8b40f1c5:ns::rbd_data.133e813ab5bc.0000000000000000:head v 51'7, mlcod=51'5)", 2024-09-18T04:39:19.142 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.531767+0000", 2024-09-18T04:39:19.142 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 36.547075147000001, 2024-09-18T04:39:19.142 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015370295000000001, 2024-09-18T04:39:19.142 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:19.142 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:19.142 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:19.142 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.142 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:19.142 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531767+0000", 2024-09-18T04:39:19.142 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.142 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.142 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.142 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:19.142 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531767+0000", 2024-09-18T04:39:19.143 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.143 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.143 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:19.143 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531765+0000", 2024-09-18T04:39:19.143 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:19.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.143 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.143 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:19.143 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534258+0000", 2024-09-18T04:39:19.143 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002493561 2024-09-18T04:39:19.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.143 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.144 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:19.144 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534288+0000", 2024-09-18T04:39:19.144 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.976e-05 2024-09-18T04:39:19.144 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.144 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.144 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:19.144 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534300+0000", 2024-09-18T04:39:19.144 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1928e-05 2024-09-18T04:39:19.144 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.144 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.144 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:19.144 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534367+0000", 2024-09-18T04:39:19.144 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7045999999999995e-05 2024-09-18T04:39:19.144 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.145 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.145 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:19.145 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534394+0000", 2024-09-18T04:39:19.145 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6903999999999999e-05 2024-09-18T04:39:19.145 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.145 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.145 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:19.145 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.547073+0000", 2024-09-18T04:39:19.145 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012679365 2024-09-18T04:39:19.145 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.145 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.145 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:19.145 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.547137+0000", 2024-09-18T04:39:19.145 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3691999999999999e-05 2024-09-18T04:39:19.146 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.146 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:19.146 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.146 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.146 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.146 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:39 2.4 2:304364d1:ns::rbd_data.133e813ab5bc.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:19.146 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.535013+0000", 2024-09-18T04:39:19.146 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 36.543828212000001, 2024-09-18T04:39:19.146 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027764487000000001, 2024-09-18T04:39:19.146 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:19.146 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:19.146 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:19.146 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:19.146 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:19.147 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 39 2024-09-18T04:39:19.147 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.147 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:19.147 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.147 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:19.147 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535013+0000", 2024-09-18T04:39:19.147 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.147 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.147 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.147 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:19.147 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535013+0000", 2024-09-18T04:39:19.147 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.147 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.147 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.148 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:19.148 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535012+0000", 2024-09-18T04:39:19.148 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:19.148 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.148 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.148 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:19.148 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545016+0000", 2024-09-18T04:39:19.148 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01000322 2024-09-18T04:39:19.148 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.148 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.148 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:19.148 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545035+0000", 2024-09-18T04:39:19.148 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9845999999999999e-05 2024-09-18T04:39:19.148 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.149 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.149 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:19.149 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545045+0000", 2024-09-18T04:39:19.149 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3640000000000005e-06 2024-09-18T04:39:19.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.149 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.149 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:19.149 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545152+0000", 2024-09-18T04:39:19.149 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000107025 2024-09-18T04:39:19.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.149 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.149 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:19.149 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545338+0000", 2024-09-18T04:39:19.150 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018643500000000001 2024-09-18T04:39:19.150 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.150 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.150 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:19.150 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545492+0000", 2024-09-18T04:39:19.150 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015393199999999999 2024-09-18T04:39:19.150 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.150 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.150 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:19.150 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.553549+0000", 2024-09-18T04:39:19.150 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0080569170000000002 2024-09-18T04:39:19.150 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.151 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.151 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:19.151 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562693+0000", 2024-09-18T04:39:19.151 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0091441609999999996 2024-09-18T04:39:19.151 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.151 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.151 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:19.151 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562748+0000", 2024-09-18T04:39:19.151 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5047000000000003e-05 2024-09-18T04:39:19.151 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.151 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.151 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:19.151 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562778+0000", 2024-09-18T04:39:19.152 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9652e-05 2024-09-18T04:39:19.152 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.152 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:19.152 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.152 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.152 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.152 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:40 2.6 2:784d6cd1:ns::rbd_data.133e813ab5bc.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:19.152 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.545062+0000", 2024-09-18T04:39:19.152 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 36.533779160999998, 2024-09-18T04:39:19.152 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.032771623, 2024-09-18T04:39:19.152 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:19.152 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:19.152 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:19.152 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:19.153 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:19.153 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 40 2024-09-18T04:39:19.153 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.153 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:19.153 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.153 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:19.153 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545062+0000", 2024-09-18T04:39:19.153 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.153 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.153 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.153 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:19.153 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545062+0000", 2024-09-18T04:39:19.153 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.153 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.154 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.154 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:19.154 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545061+0000", 2024-09-18T04:39:19.154 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:19.154 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.154 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.154 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:19.154 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552477+0000", 2024-09-18T04:39:19.154 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.007415914 2024-09-18T04:39:19.154 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.154 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.154 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:19.154 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552500+0000", 2024-09-18T04:39:19.154 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3546000000000001e-05 2024-09-18T04:39:19.155 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.155 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.155 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:19.155 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552515+0000", 2024-09-18T04:39:19.155 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4606999999999999e-05 2024-09-18T04:39:19.155 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.155 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.155 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:19.155 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552661+0000", 2024-09-18T04:39:19.155 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000146007 2024-09-18T04:39:19.155 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.155 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.155 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:19.155 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552876+0000", 2024-09-18T04:39:19.156 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000214618 2024-09-18T04:39:19.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.156 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.156 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:19.156 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.553061+0000", 2024-09-18T04:39:19.156 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018564100000000001 2024-09-18T04:39:19.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.156 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.156 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:19.156 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.565612+0000", 2024-09-18T04:39:19.156 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012550301999999999 2024-09-18T04:39:19.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.156 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.156 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:19.157 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577695+0000", 2024-09-18T04:39:19.157 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012083493000000001 2024-09-18T04:39:19.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.157 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.157 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:19.157 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577772+0000", 2024-09-18T04:39:19.157 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7158999999999998e-05 2024-09-18T04:39:19.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.157 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.157 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:19.157 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577834+0000", 2024-09-18T04:39:19.157 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1851000000000001e-05 2024-09-18T04:39:19.157 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.158 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:19.158 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.158 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.158 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.158 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:43 2.6 2:759f27b6:ns::rbd_data.133e813ab5bc.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:19.158 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.555868+0000", 2024-09-18T04:39:19.158 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 36.522973667000002, 2024-09-18T04:39:19.158 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.022332497999999999, 2024-09-18T04:39:19.158 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:19.158 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:19.158 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:19.158 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:19.158 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:19.158 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 43 2024-09-18T04:39:19.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.159 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:19.159 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.159 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:19.159 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555868+0000", 2024-09-18T04:39:19.159 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.159 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.159 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:19.159 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555868+0000", 2024-09-18T04:39:19.159 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.160 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.160 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.160 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:19.160 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555865+0000", 2024-09-18T04:39:19.160 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999971 2024-09-18T04:39:19.160 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.160 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.160 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:19.160 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561031+0000", 2024-09-18T04:39:19.160 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0051658770000000001 2024-09-18T04:39:19.160 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.160 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.160 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:19.160 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561048+0000", 2024-09-18T04:39:19.161 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6569e-05 2024-09-18T04:39:19.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.161 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.161 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:19.161 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561055+0000", 2024-09-18T04:39:19.161 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.328e-06 2024-09-18T04:39:19.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.161 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.161 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:19.161 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561114+0000", 2024-09-18T04:39:19.161 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9540000000000003e-05 2024-09-18T04:39:19.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.161 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.162 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:19.162 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561314+0000", 2024-09-18T04:39:19.162 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019965799999999999 2024-09-18T04:39:19.162 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.162 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.162 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:19.162 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561473+0000", 2024-09-18T04:39:19.162 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000158904 2024-09-18T04:39:19.162 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.162 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.162 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:19.162 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.567015+0000", 2024-09-18T04:39:19.162 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055420859999999999 2024-09-18T04:39:19.162 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.163 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.163 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:19.163 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577982+0000", 2024-09-18T04:39:19.163 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010966872000000001 2024-09-18T04:39:19.163 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.163 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.163 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:19.163 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578162+0000", 2024-09-18T04:39:19.163 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018050999999999999 2024-09-18T04:39:19.163 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.163 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.163 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:19.163 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578200+0000", 2024-09-18T04:39:19.163 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8000999999999998e-05 2024-09-18T04:39:19.164 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.164 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:19.164 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.164 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.164 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.164 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:48 2.6 2:7c225365:ns::rbd_data.133e813ab5bc.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:19.164 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:39:19.164 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 36.515174444000003, 2024-09-18T04:39:19.164 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.024576377, 2024-09-18T04:39:19.164 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:19.164 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:19.164 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:19.164 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:19.164 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:19.165 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 48 2024-09-18T04:39:19.165 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.165 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:19.165 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.165 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:19.165 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:39:19.165 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.165 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.165 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.165 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:19.165 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:39:19.165 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.165 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.165 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.166 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:19.166 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.563666+0000", 2024-09-18T04:39:19.166 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:19.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.166 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.166 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:19.166 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570331+0000", 2024-09-18T04:39:19.166 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0066653329999999998 2024-09-18T04:39:19.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.166 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.166 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:19.166 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570348+0000", 2024-09-18T04:39:19.166 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6507e-05 2024-09-18T04:39:19.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.167 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.167 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:19.167 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570355+0000", 2024-09-18T04:39:19.167 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2749999999999998e-06 2024-09-18T04:39:19.167 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.167 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.167 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:19.167 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570396+0000", 2024-09-18T04:39:19.167 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0809000000000002e-05 2024-09-18T04:39:19.167 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.167 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.167 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:19.167 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570571+0000", 2024-09-18T04:39:19.167 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000174823 2024-09-18T04:39:19.168 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.168 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.168 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:19.168 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570722+0000", 2024-09-18T04:39:19.168 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015154599999999999 2024-09-18T04:39:19.168 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.168 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.168 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:19.168 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579756+0000", 2024-09-18T04:39:19.168 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0090337000000000004 2024-09-18T04:39:19.168 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.168 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.168 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:19.169 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588093+0000", 2024-09-18T04:39:19.169 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.008336553 2024-09-18T04:39:19.169 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.169 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.169 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:19.169 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588173+0000", 2024-09-18T04:39:19.169 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0333999999999994e-05 2024-09-18T04:39:19.169 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.169 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.169 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:19.169 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588244+0000", 2024-09-18T04:39:19.169 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0573000000000003e-05 2024-09-18T04:39:19.169 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.169 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:19.170 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.170 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.170 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.170 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:49 2.4 2:20f0b0c9:ns::rbd_data.133e813ab5bc.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:19.170 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.570370+0000", 2024-09-18T04:39:19.170 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 36.508472116999997, 2024-09-18T04:39:19.170 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.029633981, 2024-09-18T04:39:19.170 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:19.170 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:19.170 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:19.170 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:19.170 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:19.170 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 49 2024-09-18T04:39:19.170 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.171 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:19.171 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.171 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:19.171 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570370+0000", 2024-09-18T04:39:19.171 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.171 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.171 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.171 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:19.171 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570370+0000", 2024-09-18T04:39:19.171 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.171 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.171 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.171 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:19.171 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570369+0000", 2024-09-18T04:39:19.172 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:19.172 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.172 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.172 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:19.172 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579090+0000", 2024-09-18T04:39:19.172 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0087212339999999996 2024-09-18T04:39:19.172 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.172 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.172 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:19.172 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579109+0000", 2024-09-18T04:39:19.172 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9120999999999999e-05 2024-09-18T04:39:19.172 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.172 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.172 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:19.173 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579123+0000", 2024-09-18T04:39:19.173 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4382000000000001e-05 2024-09-18T04:39:19.173 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.173 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.173 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:19.173 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579221+0000", 2024-09-18T04:39:19.173 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.7503999999999993e-05 2024-09-18T04:39:19.173 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.173 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.173 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:19.173 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579553+0000", 2024-09-18T04:39:19.173 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00033254200000000002 2024-09-18T04:39:19.173 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.174 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.174 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:19.174 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579817+0000", 2024-09-18T04:39:19.174 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00026317099999999998 2024-09-18T04:39:19.174 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.174 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.174 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:19.174 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.599905+0000", 2024-09-18T04:39:19.174 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020088575000000001 2024-09-18T04:39:19.175 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.175 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.175 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:19.175 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.599923+0000", 2024-09-18T04:39:19.175 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7425999999999999e-05 2024-09-18T04:39:19.175 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.175 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.175 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:19.175 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.599973+0000", 2024-09-18T04:39:19.175 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0649999999999998e-05 2024-09-18T04:39:19.175 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.175 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.175 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:19.175 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.600004+0000", 2024-09-18T04:39:19.176 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0361999999999998e-05 2024-09-18T04:39:19.176 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.176 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:19.176 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.176 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.176 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.176 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:50 2.1 e51/24 2:8c706374:ns::rbd_data.133e813ab5bc.000000000000000a:head v 51'8, mlcod=51'6)", 2024-09-18T04:39:19.176 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.577029+0000", 2024-09-18T04:39:19.176 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 36.501812745999999, 2024-09-18T04:39:19.176 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.017362855, 2024-09-18T04:39:19.176 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:19.176 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:19.176 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:19.176 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.177 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:19.177 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577029+0000", 2024-09-18T04:39:19.177 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.177 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.177 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.177 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:19.177 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577029+0000", 2024-09-18T04:39:19.177 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.177 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.177 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.177 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:19.177 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577027+0000", 2024-09-18T04:39:19.177 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:19.177 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.178 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.178 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:19.178 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578880+0000", 2024-09-18T04:39:19.178 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0018532080000000001 2024-09-18T04:39:19.178 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.178 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.178 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:19.178 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578906+0000", 2024-09-18T04:39:19.178 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5573999999999999e-05 2024-09-18T04:39:19.178 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.178 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.178 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:19.178 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578920+0000", 2024-09-18T04:39:19.178 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4483e-05 2024-09-18T04:39:19.179 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.179 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.179 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:19.179 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578983+0000", 2024-09-18T04:39:19.179 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2248000000000004e-05 2024-09-18T04:39:19.179 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.179 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.179 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:19.179 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579016+0000", 2024-09-18T04:39:19.179 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3364000000000001e-05 2024-09-18T04:39:19.179 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.179 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.179 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:19.179 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594342+0000", 2024-09-18T04:39:19.179 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01532564 2024-09-18T04:39:19.180 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.180 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.180 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:19.180 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594392+0000", 2024-09-18T04:39:19.180 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0068999999999998e-05 2024-09-18T04:39:19.180 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.180 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:19.180 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.180 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.180 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.180 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:45 2.0 e51/17 2:0fbfa3dc:ns::rbd_data.133e813ab5bc.0000000000000007:head v 51'24, mlcod=51'22)", 2024-09-18T04:39:19.180 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.581892+0000", 2024-09-18T04:39:19.180 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 36.496950073999997, 2024-09-18T04:39:19.180 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015646006, 2024-09-18T04:39:19.181 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:19.181 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:19.181 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:19.181 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.181 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:19.181 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581892+0000", 2024-09-18T04:39:19.181 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.181 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.181 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.181 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:19.181 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581892+0000", 2024-09-18T04:39:19.181 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.181 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.181 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.182 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:19.182 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581890+0000", 2024-09-18T04:39:19.182 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:19.182 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.182 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.182 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:19.182 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586188+0000", 2024-09-18T04:39:19.182 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0042985769999999996 2024-09-18T04:39:19.182 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.182 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.182 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:19.182 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586210+0000", 2024-09-18T04:39:19.182 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1069000000000001e-05 2024-09-18T04:39:19.182 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.183 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.183 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:19.183 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586221+0000", 2024-09-18T04:39:19.183 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1287000000000001e-05 2024-09-18T04:39:19.183 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.183 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.183 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:19.183 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586313+0000", 2024-09-18T04:39:19.183 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1935999999999995e-05 2024-09-18T04:39:19.183 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.183 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.183 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:19.183 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586368+0000", 2024-09-18T04:39:19.183 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5578000000000002e-05 2024-09-18T04:39:19.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.184 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.184 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:19.184 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.597473+0000", 2024-09-18T04:39:19.184 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011105139999999999 2024-09-18T04:39:19.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.184 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.184 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:19.184 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.597538+0000", 2024-09-18T04:39:19.184 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4133000000000004e-05 2024-09-18T04:39:19.184 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.184 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:19.184 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.184 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.185 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:53 2.4 2:39f7f174:ns::rbd_data.133e813ab5bc.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:19.185 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.584263+0000", 2024-09-18T04:39:19.185 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 36.494578199999999, 2024-09-18T04:39:19.185 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018701180000000001, 2024-09-18T04:39:19.185 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:19.185 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:19.185 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:19.185 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:19.185 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:19.185 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 53 2024-09-18T04:39:19.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.185 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:19.185 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.186 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:19.186 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584263+0000", 2024-09-18T04:39:19.186 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.186 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.186 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:19.186 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584263+0000", 2024-09-18T04:39:19.186 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.186 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.186 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:19.186 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584262+0000", 2024-09-18T04:39:19.186 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:19.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.187 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.187 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:19.187 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.591617+0000", 2024-09-18T04:39:19.187 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0073550600000000001 2024-09-18T04:39:19.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.187 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.187 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:19.187 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.591637+0000", 2024-09-18T04:39:19.187 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0432999999999999e-05 2024-09-18T04:39:19.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.187 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.187 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:19.187 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.591646+0000", 2024-09-18T04:39:19.187 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7849999999999997e-06 2024-09-18T04:39:19.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.188 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.188 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:19.188 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593478+0000", 2024-09-18T04:39:19.188 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001831495 2024-09-18T04:39:19.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.188 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.188 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:19.188 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593661+0000", 2024-09-18T04:39:19.188 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018364999999999999 2024-09-18T04:39:19.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.188 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.188 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:19.189 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593834+0000", 2024-09-18T04:39:19.189 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000173065 2024-09-18T04:39:19.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.189 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.189 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:19.189 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.600670+0000", 2024-09-18T04:39:19.189 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0068354970000000003 2024-09-18T04:39:19.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.189 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.189 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:19.189 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602851+0000", 2024-09-18T04:39:19.189 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002181653 2024-09-18T04:39:19.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.189 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.189 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:19.190 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602930+0000", 2024-09-18T04:39:19.190 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8045000000000002e-05 2024-09-18T04:39:19.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.190 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.190 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:19.190 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602965+0000", 2024-09-18T04:39:19.190 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5102999999999999e-05 2024-09-18T04:39:19.190 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.190 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:19.190 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.190 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.190 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:56 2.6 2:72453c3e:ns::rbd_data.133e813ab5bc.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:19.191 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.595988+0000", 2024-09-18T04:39:19.191 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 36.482853798000001, 2024-09-18T04:39:19.191 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.030296172, 2024-09-18T04:39:19.191 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:19.191 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:19.191 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:19.191 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:19.191 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:19.191 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 56 2024-09-18T04:39:19.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.191 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:19.191 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.191 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:19.191 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.595988+0000", 2024-09-18T04:39:19.192 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.192 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.192 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:19.192 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.595988+0000", 2024-09-18T04:39:19.192 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.192 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.192 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:19.192 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.595986+0000", 2024-09-18T04:39:19.192 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:19.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.192 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.192 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:19.193 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605262+0000", 2024-09-18T04:39:19.193 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0092767240000000001 2024-09-18T04:39:19.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.193 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.193 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:19.193 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605282+0000", 2024-09-18T04:39:19.193 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9307e-05 2024-09-18T04:39:19.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.193 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.193 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:19.193 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605292+0000", 2024-09-18T04:39:19.193 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9510000000000001e-06 2024-09-18T04:39:19.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.193 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.194 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:19.194 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605369+0000", 2024-09-18T04:39:19.194 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7846e-05 2024-09-18T04:39:19.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.194 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.194 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:19.194 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605668+0000", 2024-09-18T04:39:19.194 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00029896099999999999 2024-09-18T04:39:19.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.194 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.194 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:19.194 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605906+0000", 2024-09-18T04:39:19.194 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000237758 2024-09-18T04:39:19.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.195 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.195 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:19.195 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.613532+0000", 2024-09-18T04:39:19.195 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0076260590000000001 2024-09-18T04:39:19.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.195 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.195 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:19.195 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.626116+0000", 2024-09-18T04:39:19.195 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012584092999999999 2024-09-18T04:39:19.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.195 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.195 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:19.195 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.626216+0000", 2024-09-18T04:39:19.195 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9630999999999995e-05 2024-09-18T04:39:19.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.196 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.196 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:19.196 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.626284+0000", 2024-09-18T04:39:19.196 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8034999999999995e-05 2024-09-18T04:39:19.196 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.196 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:19.196 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.196 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.196 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:19.196 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:19.198 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:39:19.198 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:19.193+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:19.298 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:19.293+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:19.314 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:19.314 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:19.314 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:39:19.315 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:19.324 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:39:19.400 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:19.393+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:19.477 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:19.477 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:19.477 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:39:19.477 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:39:19.477 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:19.487 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:39:19.500 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:19.493+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:39:19.601 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:19.593+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:19.604 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:19.604 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:39:19.604 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:39:19.604 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:39:19.604 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.604 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:33 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:19.604 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.261182+0000", 2024-09-18T04:39:19.605 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 37.338646850000003, 2024-09-18T04:39:19.605 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.037082208999999998, 2024-09-18T04:39:19.605 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:19.605 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:19.605 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:19.605 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:19.605 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:19.605 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 33 2024-09-18T04:39:19.605 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.605 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:19.605 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.605 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:19.605 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.261182+0000", 2024-09-18T04:39:19.606 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.606 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.606 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.606 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:19.606 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.261182+0000", 2024-09-18T04:39:19.606 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.606 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.606 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.606 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:19.606 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.261180+0000", 2024-09-18T04:39:19.606 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:19.606 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.606 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.606 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:19.607 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278669+0000", 2024-09-18T04:39:19.607 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.017488877999999999 2024-09-18T04:39:19.607 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.607 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.607 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:19.607 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278689+0000", 2024-09-18T04:39:19.607 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0047999999999998e-05 2024-09-18T04:39:19.607 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.607 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.607 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:19.607 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278699+0000", 2024-09-18T04:39:19.607 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6800000000000005e-06 2024-09-18T04:39:19.607 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.607 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.608 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:19.608 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278775+0000", 2024-09-18T04:39:19.608 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6087999999999994e-05 2024-09-18T04:39:19.608 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.608 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.608 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:19.608 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278935+0000", 2024-09-18T04:39:19.608 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000159834 2024-09-18T04:39:19.608 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.608 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.608 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:19.608 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279064+0000", 2024-09-18T04:39:19.608 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012954199999999999 2024-09-18T04:39:19.608 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.609 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.609 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:19.609 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295562+0000", 2024-09-18T04:39:19.609 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016497583999999999 2024-09-18T04:39:19.609 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.609 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.609 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:19.609 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.298147+0000", 2024-09-18T04:39:19.609 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002584982 2024-09-18T04:39:19.609 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.609 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.609 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:19.609 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.298217+0000", 2024-09-18T04:39:19.609 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0400000000000004e-05 2024-09-18T04:39:19.610 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.610 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.610 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:19.610 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.298265+0000", 2024-09-18T04:39:19.610 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7580000000000002e-05 2024-09-18T04:39:19.610 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.610 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:19.610 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.610 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.610 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.610 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:34 2.3 e51/24 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head v 51'3, mlcod=18'1)", 2024-09-18T04:39:19.610 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.273067+0000", 2024-09-18T04:39:19.610 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 37.32676206, 2024-09-18T04:39:19.610 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.021784103999999999, 2024-09-18T04:39:19.610 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:19.611 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:19.611 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:19.611 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.611 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:19.611 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.273067+0000", 2024-09-18T04:39:19.611 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.611 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.611 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.611 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:19.611 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.273067+0000", 2024-09-18T04:39:19.611 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.611 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.611 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.611 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:19.612 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.273060+0000", 2024-09-18T04:39:19.612 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999924 2024-09-18T04:39:19.612 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.612 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.612 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:19.612 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278596+0000", 2024-09-18T04:39:19.612 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055365839999999998 2024-09-18T04:39:19.612 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.612 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.612 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:19.612 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278637+0000", 2024-09-18T04:39:19.612 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0596000000000002e-05 2024-09-18T04:39:19.612 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.612 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.613 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:19.613 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278653+0000", 2024-09-18T04:39:19.613 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6626999999999999e-05 2024-09-18T04:39:19.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.613 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.613 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:19.613 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278748+0000", 2024-09-18T04:39:19.613 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4483999999999999e-05 2024-09-18T04:39:19.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.613 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.613 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:19.613 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278799+0000", 2024-09-18T04:39:19.613 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1484999999999999e-05 2024-09-18T04:39:19.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.614 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.614 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:19.614 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.294785+0000", 2024-09-18T04:39:19.614 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015985256999999999 2024-09-18T04:39:19.614 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.614 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.614 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:19.614 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.294851+0000", 2024-09-18T04:39:19.614 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6728000000000002e-05 2024-09-18T04:39:19.614 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.614 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:19.614 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.614 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.614 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.615 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:37 2.5 2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:19.615 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.278722+0000", 2024-09-18T04:39:19.615 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 37.321107011000002, 2024-09-18T04:39:19.615 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.026323801000000001, 2024-09-18T04:39:19.615 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:19.615 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:19.615 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:19.615 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:19.615 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:19.615 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 37 2024-09-18T04:39:19.615 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.615 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:19.615 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.615 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:19.616 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278722+0000", 2024-09-18T04:39:19.616 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.616 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.616 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.616 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:19.616 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278722+0000", 2024-09-18T04:39:19.616 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.616 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.616 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.616 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:19.616 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278720+0000", 2024-09-18T04:39:19.616 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:19.616 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.616 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.616 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:19.617 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.291875+0000", 2024-09-18T04:39:19.617 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013154414 2024-09-18T04:39:19.617 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.617 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.617 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:19.617 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.291901+0000", 2024-09-18T04:39:19.617 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6339999999999999e-05 2024-09-18T04:39:19.617 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.617 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.617 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:19.617 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.291914+0000", 2024-09-18T04:39:19.617 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3043e-05 2024-09-18T04:39:19.617 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.618 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.618 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:19.618 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.291975+0000", 2024-09-18T04:39:19.618 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0949999999999998e-05 2024-09-18T04:39:19.618 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.618 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.618 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:19.618 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292173+0000", 2024-09-18T04:39:19.618 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019825499999999999 2024-09-18T04:39:19.618 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.618 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.618 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:19.618 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292300+0000", 2024-09-18T04:39:19.618 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000126616 2024-09-18T04:39:19.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.619 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.619 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:19.619 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304334+0000", 2024-09-18T04:39:19.619 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012033934 2024-09-18T04:39:19.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.619 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.619 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:19.619 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304901+0000", 2024-09-18T04:39:19.619 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00056694099999999995 2024-09-18T04:39:19.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.619 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.619 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:19.620 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304981+0000", 2024-09-18T04:39:19.620 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9974000000000004e-05 2024-09-18T04:39:19.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.620 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.620 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:19.620 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.305046+0000", 2024-09-18T04:39:19.620 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5184000000000005e-05 2024-09-18T04:39:19.620 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.620 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:19.620 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.620 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.620 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:49 2.5 2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:19.620 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.314271+0000", 2024-09-18T04:39:19.621 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 37.285558408999997, 2024-09-18T04:39:19.621 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.029779953000000001, 2024-09-18T04:39:19.621 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:19.621 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:19.621 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:19.621 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:19.621 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:19.621 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 49 2024-09-18T04:39:19.621 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.621 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:19.621 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.621 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:19.621 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.314271+0000", 2024-09-18T04:39:19.621 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.622 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.622 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:19.622 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.314271+0000", 2024-09-18T04:39:19.622 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.622 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.622 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:19.622 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.314269+0000", 2024-09-18T04:39:19.622 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:19.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.622 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.622 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:19.622 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.324780+0000", 2024-09-18T04:39:19.623 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010511609999999999 2024-09-18T04:39:19.623 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.623 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.623 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:19.623 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.324821+0000", 2024-09-18T04:39:19.623 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0899000000000003e-05 2024-09-18T04:39:19.623 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.623 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.623 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:19.623 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.324840+0000", 2024-09-18T04:39:19.623 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8947e-05 2024-09-18T04:39:19.623 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.623 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.623 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:19.624 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.324936+0000", 2024-09-18T04:39:19.624 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6342999999999996e-05 2024-09-18T04:39:19.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.624 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.624 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:19.624 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325284+0000", 2024-09-18T04:39:19.624 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000347146 2024-09-18T04:39:19.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.624 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.624 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:19.624 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325586+0000", 2024-09-18T04:39:19.624 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00030199799999999999 2024-09-18T04:39:19.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.624 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.625 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:19.625 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343710+0000", 2024-09-18T04:39:19.625 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018124583999999999 2024-09-18T04:39:19.625 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.625 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.625 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:19.625 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343991+0000", 2024-09-18T04:39:19.625 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00028085600000000001 2024-09-18T04:39:19.625 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.625 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.625 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:19.625 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344028+0000", 2024-09-18T04:39:19.625 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6637000000000002e-05 2024-09-18T04:39:19.625 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.626 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.626 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:19.626 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344051+0000", 2024-09-18T04:39:19.626 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3260000000000001e-05 2024-09-18T04:39:19.626 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.626 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:19.626 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.626 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.626 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.626 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:50 2.3 e51/24 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head v 51'4, mlcod=51'2)", 2024-09-18T04:39:19.626 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.323066+0000", 2024-09-18T04:39:19.626 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 37.276763015999997, 2024-09-18T04:39:19.626 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.022135774, 2024-09-18T04:39:19.626 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:19.626 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:19.627 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:19.627 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.627 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:19.627 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.323066+0000", 2024-09-18T04:39:19.627 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.627 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.627 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.627 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:19.627 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.323066+0000", 2024-09-18T04:39:19.627 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.627 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.627 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.627 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:19.628 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.323065+0000", 2024-09-18T04:39:19.628 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:19.628 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.628 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.628 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:19.628 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.328664+0000", 2024-09-18T04:39:19.628 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055993600000000003 2024-09-18T04:39:19.628 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.628 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.628 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:19.628 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.328692+0000", 2024-09-18T04:39:19.628 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7654e-05 2024-09-18T04:39:19.628 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.628 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.629 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:19.629 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.328706+0000", 2024-09-18T04:39:19.629 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4097000000000001e-05 2024-09-18T04:39:19.629 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.629 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.629 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:19.629 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.328816+0000", 2024-09-18T04:39:19.629 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000110281 2024-09-18T04:39:19.629 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.629 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.629 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:19.629 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.328848+0000", 2024-09-18T04:39:19.629 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1837e-05 2024-09-18T04:39:19.629 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.629 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.630 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:19.630 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345160+0000", 2024-09-18T04:39:19.630 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016312099 2024-09-18T04:39:19.630 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.630 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.630 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:19.630 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345202+0000", 2024-09-18T04:39:19.630 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1857999999999998e-05 2024-09-18T04:39:19.630 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.630 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:19.630 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.630 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.630 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.630 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:58 2.5 2:b227e6b1:::rbd_data.125fc09df6d1.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:19.630 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.351564+0000", 2024-09-18T04:39:19.631 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 37.248265564, 2024-09-18T04:39:19.631 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.032125956999999997, 2024-09-18T04:39:19.631 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:19.631 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:19.631 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:19.631 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:19.631 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:19.631 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 58 2024-09-18T04:39:19.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.631 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:19.631 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.631 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:19.631 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.351564+0000", 2024-09-18T04:39:19.632 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.632 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.632 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.632 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:19.632 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.351564+0000", 2024-09-18T04:39:19.632 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.632 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.632 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.632 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:19.632 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.351561+0000", 2024-09-18T04:39:19.632 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:19.632 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.632 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.633 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:19.633 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364390+0000", 2024-09-18T04:39:19.633 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012829063999999999 2024-09-18T04:39:19.633 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.633 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.633 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:19.633 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364418+0000", 2024-09-18T04:39:19.633 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7307000000000001e-05 2024-09-18T04:39:19.633 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.633 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.633 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:19.633 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364428+0000", 2024-09-18T04:39:19.633 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0675e-05 2024-09-18T04:39:19.633 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.634 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.634 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:19.634 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364523+0000", 2024-09-18T04:39:19.634 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4698999999999996e-05 2024-09-18T04:39:19.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.634 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.634 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:19.634 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364785+0000", 2024-09-18T04:39:19.634 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00026170099999999998 2024-09-18T04:39:19.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.634 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.634 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:19.634 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.365039+0000", 2024-09-18T04:39:19.635 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00025462299999999999 2024-09-18T04:39:19.635 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.635 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.635 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:19.635 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383143+0000", 2024-09-18T04:39:19.635 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018103768999999999 2024-09-18T04:39:19.635 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.635 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.635 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:19.635 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383582+0000", 2024-09-18T04:39:19.635 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00043860600000000002 2024-09-18T04:39:19.635 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.635 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.635 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:19.636 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383644+0000", 2024-09-18T04:39:19.636 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2682999999999996e-05 2024-09-18T04:39:19.636 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.636 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.636 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:19.636 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383690+0000", 2024-09-18T04:39:19.636 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5247999999999998e-05 2024-09-18T04:39:19.636 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.636 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:19.636 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.636 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.636 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.636 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:33 2.1 2:8b40f1c5:ns::rbd_data.133e813ab5bc.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:19.636 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.523765+0000", 2024-09-18T04:39:19.637 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 37.076064524000003, 2024-09-18T04:39:19.637 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027472973000000001, 2024-09-18T04:39:19.637 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:19.637 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:19.637 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:19.637 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:19.637 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:19.637 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 33 2024-09-18T04:39:19.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.637 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:19.637 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.637 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:19.637 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.523765+0000", 2024-09-18T04:39:19.637 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.638 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.638 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:19.638 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.523765+0000", 2024-09-18T04:39:19.638 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.638 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.638 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:19.638 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.523762+0000", 2024-09-18T04:39:19.638 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:19.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.638 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.638 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:19.639 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531127+0000", 2024-09-18T04:39:19.639 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0073644560000000001 2024-09-18T04:39:19.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.639 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.639 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:19.639 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531189+0000", 2024-09-18T04:39:19.639 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2315000000000002e-05 2024-09-18T04:39:19.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.639 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.639 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:19.639 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531211+0000", 2024-09-18T04:39:19.639 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1542000000000001e-05 2024-09-18T04:39:19.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.639 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.640 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:19.640 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531314+0000", 2024-09-18T04:39:19.640 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000103582 2024-09-18T04:39:19.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.640 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.640 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:19.640 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531493+0000", 2024-09-18T04:39:19.640 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000178433 2024-09-18T04:39:19.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.640 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.640 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:19.640 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531653+0000", 2024-09-18T04:39:19.640 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000160388 2024-09-18T04:39:19.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.641 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.641 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:19.641 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.547345+0000", 2024-09-18T04:39:19.641 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015691742000000002 2024-09-18T04:39:19.641 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.641 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.641 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:19.641 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551076+0000", 2024-09-18T04:39:19.641 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003730957 2024-09-18T04:39:19.641 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.641 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.641 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:19.641 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551176+0000", 2024-09-18T04:39:19.641 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010002599999999999 2024-09-18T04:39:19.642 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.642 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.642 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:19.642 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551238+0000", 2024-09-18T04:39:19.642 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1945000000000004e-05 2024-09-18T04:39:19.642 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.642 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:19.642 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.642 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.642 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.642 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:36 2.5 2:bd5b8dd8:ns::rbd_data.133e813ab5bc.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:19.642 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.531253+0000", 2024-09-18T04:39:19.642 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 37.068576215, 2024-09-18T04:39:19.642 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027295212999999999, 2024-09-18T04:39:19.643 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:19.643 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:19.643 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:19.643 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:19.643 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:19.643 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 36 2024-09-18T04:39:19.643 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.643 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:19.643 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.643 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:19.643 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531253+0000", 2024-09-18T04:39:19.643 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.643 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.643 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.644 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:19.644 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531253+0000", 2024-09-18T04:39:19.644 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.644 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.644 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.644 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:19.644 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531251+0000", 2024-09-18T04:39:19.644 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:19.644 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.644 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.644 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:19.644 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536540+0000", 2024-09-18T04:39:19.644 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0052885839999999998 2024-09-18T04:39:19.644 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.645 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.645 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:19.645 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536604+0000", 2024-09-18T04:39:19.645 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4824000000000001e-05 2024-09-18T04:39:19.645 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.645 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.645 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:19.645 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536621+0000", 2024-09-18T04:39:19.645 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6215999999999999e-05 2024-09-18T04:39:19.645 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.645 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.645 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:19.645 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536768+0000", 2024-09-18T04:39:19.645 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014767900000000001 2024-09-18T04:39:19.646 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.646 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.646 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:19.646 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537084+0000", 2024-09-18T04:39:19.646 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00031553499999999998 2024-09-18T04:39:19.646 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.646 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.646 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:19.646 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537300+0000", 2024-09-18T04:39:19.646 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000216145 2024-09-18T04:39:19.646 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.646 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.646 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:19.646 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555326+0000", 2024-09-18T04:39:19.647 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018025861000000001 2024-09-18T04:39:19.647 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.647 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.647 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:19.647 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558449+0000", 2024-09-18T04:39:19.647 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0031229750000000001 2024-09-18T04:39:19.647 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.647 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.647 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:19.647 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558502+0000", 2024-09-18T04:39:19.647 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3572000000000001e-05 2024-09-18T04:39:19.647 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.647 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.647 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:19.647 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558548+0000", 2024-09-18T04:39:19.648 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5921999999999998e-05 2024-09-18T04:39:19.648 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.648 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:19.648 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.648 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.648 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.648 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:37 2.5 2:bf39b8e4:ns::rbd_data.133e813ab5bc.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:19.648 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.536664+0000", 2024-09-18T04:39:19.648 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 37.063164798999999, 2024-09-18T04:39:19.648 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.026331765, 2024-09-18T04:39:19.648 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:19.648 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:19.648 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:19.649 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:19.649 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:19.649 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 37 2024-09-18T04:39:19.649 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.649 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:19.649 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.649 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:19.649 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536664+0000", 2024-09-18T04:39:19.649 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.649 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.649 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.649 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:19.649 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536664+0000", 2024-09-18T04:39:19.649 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.650 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.650 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:19.650 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536662+0000", 2024-09-18T04:39:19.650 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:19.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.650 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.650 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:19.650 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.543699+0000", 2024-09-18T04:39:19.650 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0070367119999999997 2024-09-18T04:39:19.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.650 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.650 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:19.650 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.543749+0000", 2024-09-18T04:39:19.651 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0476999999999998e-05 2024-09-18T04:39:19.651 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.651 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.651 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:19.651 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.543800+0000", 2024-09-18T04:39:19.651 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0822999999999998e-05 2024-09-18T04:39:19.651 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.651 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.651 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:19.651 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.547868+0000", 2024-09-18T04:39:19.651 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0040679339999999996 2024-09-18T04:39:19.651 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.651 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.651 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:19.652 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548087+0000", 2024-09-18T04:39:19.652 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000218994 2024-09-18T04:39:19.652 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.652 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.652 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:19.652 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548383+0000", 2024-09-18T04:39:19.652 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000295588 2024-09-18T04:39:19.652 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.652 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.652 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:19.652 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562622+0000", 2024-09-18T04:39:19.652 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014239915000000001 2024-09-18T04:39:19.652 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.652 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.653 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:19.653 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562924+0000", 2024-09-18T04:39:19.653 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00030127600000000001 2024-09-18T04:39:19.653 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.653 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.653 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:19.653 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562957+0000", 2024-09-18T04:39:19.653 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3365999999999999e-05 2024-09-18T04:39:19.653 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.653 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.653 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:19.653 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562996+0000", 2024-09-18T04:39:19.653 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9153999999999998e-05 2024-09-18T04:39:19.654 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.654 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:19.654 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.654 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.654 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:50 2.1 2:8c706374:ns::rbd_data.133e813ab5bc.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:19.654 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.570513+0000", 2024-09-18T04:39:19.654 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 37.029316516999998, 2024-09-18T04:39:19.654 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.024182995999999998, 2024-09-18T04:39:19.654 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:19.654 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:19.654 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:19.654 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:19.654 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:19.654 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 50 2024-09-18T04:39:19.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.655 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:19.655 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.655 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:19.655 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570513+0000", 2024-09-18T04:39:19.655 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.655 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.655 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:19.655 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570513+0000", 2024-09-18T04:39:19.655 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.655 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.655 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:19.656 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570511+0000", 2024-09-18T04:39:19.656 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:19.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.656 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.656 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:19.656 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.576327+0000", 2024-09-18T04:39:19.656 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0058162530000000004 2024-09-18T04:39:19.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.656 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.656 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:19.656 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.576355+0000", 2024-09-18T04:39:19.656 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8030000000000001e-05 2024-09-18T04:39:19.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.656 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.657 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:19.657 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.576366+0000", 2024-09-18T04:39:19.657 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1139e-05 2024-09-18T04:39:19.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.657 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.657 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:19.657 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.576458+0000", 2024-09-18T04:39:19.657 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1780999999999994e-05 2024-09-18T04:39:19.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.657 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.657 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:19.657 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.576701+0000", 2024-09-18T04:39:19.657 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00024303899999999999 2024-09-18T04:39:19.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.658 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.658 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:19.658 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.576876+0000", 2024-09-18T04:39:19.658 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000174637 2024-09-18T04:39:19.658 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.658 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.658 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:19.658 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588432+0000", 2024-09-18T04:39:19.658 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011555943000000001 2024-09-18T04:39:19.658 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.658 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.658 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:19.658 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594573+0000", 2024-09-18T04:39:19.658 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0061408670000000004 2024-09-18T04:39:19.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.659 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.659 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:19.659 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594640+0000", 2024-09-18T04:39:19.659 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7404999999999996e-05 2024-09-18T04:39:19.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.659 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.659 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:19.659 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594696+0000", 2024-09-18T04:39:19.659 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5612999999999998e-05 2024-09-18T04:39:19.659 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.659 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:19.659 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.660 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.660 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:60 2.5 2:a06f51e4:ns::rbd_data.133e813ab5bc.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:19.660 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.604753+0000", 2024-09-18T04:39:19.660 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 36.995076398000002, 2024-09-18T04:39:19.660 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014400181, 2024-09-18T04:39:19.660 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:19.660 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:19.660 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:19.660 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:19.660 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:19.660 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 60 2024-09-18T04:39:19.660 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.660 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:19.660 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.661 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:19.661 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.604753+0000", 2024-09-18T04:39:19.661 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.661 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.661 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.661 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:19.661 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.604753+0000", 2024-09-18T04:39:19.661 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.661 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.661 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.661 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:19.661 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.604751+0000", 2024-09-18T04:39:19.661 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:19.661 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.662 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.662 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:19.662 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.608980+0000", 2024-09-18T04:39:19.662 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0042292420000000002 2024-09-18T04:39:19.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.662 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.662 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:19.662 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609014+0000", 2024-09-18T04:39:19.662 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3572999999999999e-05 2024-09-18T04:39:19.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.662 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.662 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:19.662 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609032+0000", 2024-09-18T04:39:19.662 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8150000000000001e-05 2024-09-18T04:39:19.663 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.663 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.663 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:19.663 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609090+0000", 2024-09-18T04:39:19.663 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8214999999999997e-05 2024-09-18T04:39:19.663 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.663 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.663 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:19.663 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609235+0000", 2024-09-18T04:39:19.663 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014437799999999999 2024-09-18T04:39:19.663 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.663 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.663 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:19.663 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609360+0000", 2024-09-18T04:39:19.664 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012526099999999999 2024-09-18T04:39:19.664 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.664 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.664 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:19.664 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.614952+0000", 2024-09-18T04:39:19.664 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055922300000000001 2024-09-18T04:39:19.664 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.664 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.664 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:19.664 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.619070+0000", 2024-09-18T04:39:19.664 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0041178250000000003 2024-09-18T04:39:19.664 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.664 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.664 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:19.664 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.619117+0000", 2024-09-18T04:39:19.665 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7275999999999999e-05 2024-09-18T04:39:19.665 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.665 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.665 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:19.665 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.619153+0000", 2024-09-18T04:39:19.665 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5970000000000003e-05 2024-09-18T04:39:19.665 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.665 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:19.665 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.665 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.665 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.665 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4972.0:20 2.5 2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:19.665 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.949296+0000", 2024-09-18T04:39:19.666 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 27.650533795000001, 2024-09-18T04:39:19.666 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014709732, 2024-09-18T04:39:19.666 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:19.666 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:19.666 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:19.666 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4972", 2024-09-18T04:39:19.666 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2280251472", 2024-09-18T04:39:19.666 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 20 2024-09-18T04:39:19.666 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.666 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:19.666 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.666 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:19.666 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949296+0000", 2024-09-18T04:39:19.666 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.666 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.667 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.667 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:19.667 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949296+0000", 2024-09-18T04:39:19.667 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.667 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.667 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.667 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:19.667 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949295+0000", 2024-09-18T04:39:19.667 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:19.667 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.667 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.667 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:19.667 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949301+0000", 2024-09-18T04:39:19.667 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4829999999999996e-06 2024-09-18T04:39:19.667 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.668 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.668 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:19.668 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949317+0000", 2024-09-18T04:39:19.668 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5866999999999999e-05 2024-09-18T04:39:19.668 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.668 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.668 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:19.668 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949331+0000", 2024-09-18T04:39:19.668 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3485999999999999e-05 2024-09-18T04:39:19.668 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.668 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.668 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:19.668 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.959137+0000", 2024-09-18T04:39:19.669 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0098066520000000008 2024-09-18T04:39:19.669 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.669 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.669 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:19.669 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.959251+0000", 2024-09-18T04:39:19.669 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000113783 2024-09-18T04:39:19.669 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.669 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.669 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:19.669 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.964005+0000", 2024-09-18T04:39:19.669 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0047542080000000002 2024-09-18T04:39:19.669 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.669 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:19.669 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.669 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.670 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.670 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.5246.0:96 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [write 524288~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-18T04:39:19.670 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:15.279086+0000", 2024-09-18T04:39:19.670 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.3207434439999997, 2024-09-18T04:39:19.670 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015391526000000001, 2024-09-18T04:39:19.670 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:19.670 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:19.670 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:19.670 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.5246", 2024-09-18T04:39:19.670 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2208214885", 2024-09-18T04:39:19.670 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 96 2024-09-18T04:39:19.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.670 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:19.670 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.671 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:19.671 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.279086+0000", 2024-09-18T04:39:19.671 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.671 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.671 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.671 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:19.671 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.279086+0000", 2024-09-18T04:39:19.671 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.671 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.671 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.671 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:19.671 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.279085+0000", 2024-09-18T04:39:19.671 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:19.671 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.672 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.672 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:19.672 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.279233+0000", 2024-09-18T04:39:19.672 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014805100000000001 2024-09-18T04:39:19.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.672 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.672 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:19.672 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.279254+0000", 2024-09-18T04:39:19.672 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1645e-05 2024-09-18T04:39:19.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.672 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.672 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:19.672 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.279270+0000", 2024-09-18T04:39:19.672 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5909e-05 2024-09-18T04:39:19.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.673 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.673 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:19.673 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.289235+0000", 2024-09-18T04:39:19.673 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0099644550000000005 2024-09-18T04:39:19.673 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.673 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.673 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:19.673 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.289280+0000", 2024-09-18T04:39:19.673 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5772999999999997e-05 2024-09-18T04:39:19.673 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.673 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.673 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:19.673 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.289355+0000", 2024-09-18T04:39:19.673 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4821999999999996e-05 2024-09-18T04:39:19.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.674 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.674 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:19.674 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.291704+0000", 2024-09-18T04:39:19.674 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023489460000000002 2024-09-18T04:39:19.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.674 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.674 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:19.674 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.294362+0000", 2024-09-18T04:39:19.674 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026582160000000001 2024-09-18T04:39:19.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.674 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.674 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:19.674 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.294428+0000", 2024-09-18T04:39:19.675 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5203999999999994e-05 2024-09-18T04:39:19.675 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.675 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.675 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:19.675 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.294477+0000", 2024-09-18T04:39:19.675 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9805000000000002e-05 2024-09-18T04:39:19.675 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.675 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:19.675 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.675 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.676 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.676 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.5246.0:97 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [write 655360~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-18T04:39:19.676 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:15.279285+0000", 2024-09-18T04:39:19.676 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.3205444660000003, 2024-09-18T04:39:19.676 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015374402000000001, 2024-09-18T04:39:19.676 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:19.676 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:19.676 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:19.676 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.5246", 2024-09-18T04:39:19.676 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2208214885", 2024-09-18T04:39:19.676 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 97 2024-09-18T04:39:19.676 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.676 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:19.677 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.677 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:19.677 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.279285+0000", 2024-09-18T04:39:19.677 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.677 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.677 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.677 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:19.677 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.279285+0000", 2024-09-18T04:39:19.677 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.677 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.677 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.677 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:19.677 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.279284+0000", 2024-09-18T04:39:19.677 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:19.678 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.678 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.678 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:19.678 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.279407+0000", 2024-09-18T04:39:19.678 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000123812 2024-09-18T04:39:19.678 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.678 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.678 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:19.678 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.279432+0000", 2024-09-18T04:39:19.678 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4212000000000001e-05 2024-09-18T04:39:19.678 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.678 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.678 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:19.678 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.279447+0000", 2024-09-18T04:39:19.678 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5753e-05 2024-09-18T04:39:19.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.679 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.679 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:19.679 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.290145+0000", 2024-09-18T04:39:19.679 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010697312 2024-09-18T04:39:19.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.679 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.679 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:19.679 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.290213+0000", 2024-09-18T04:39:19.679 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8598999999999998e-05 2024-09-18T04:39:19.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.679 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.679 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:19.680 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.290308+0000", 2024-09-18T04:39:19.680 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4449000000000003e-05 2024-09-18T04:39:19.680 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.680 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.680 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:19.680 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.294275+0000", 2024-09-18T04:39:19.680 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0039673260000000002 2024-09-18T04:39:19.680 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.680 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.680 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:19.680 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.294546+0000", 2024-09-18T04:39:19.680 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00027124199999999999 2024-09-18T04:39:19.680 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.680 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.681 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:19.681 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.294615+0000", 2024-09-18T04:39:19.681 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8595000000000003e-05 2024-09-18T04:39:19.681 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.681 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.681 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:19.681 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.294659+0000", 2024-09-18T04:39:19.681 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4314999999999998e-05 2024-09-18T04:39:19.681 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.681 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:19.681 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.681 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.681 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.681 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.5246.0:98 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [write 786432~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-18T04:39:19.682 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:15.279462+0000", 2024-09-18T04:39:19.682 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.3203671129999996, 2024-09-18T04:39:19.682 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015380944000000001, 2024-09-18T04:39:19.682 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:19.682 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:19.682 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:19.682 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.5246", 2024-09-18T04:39:19.682 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2208214885", 2024-09-18T04:39:19.682 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 98 2024-09-18T04:39:19.682 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.682 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:19.682 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.682 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:19.682 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.279462+0000", 2024-09-18T04:39:19.682 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.683 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.683 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:19.683 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.279462+0000", 2024-09-18T04:39:19.683 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.683 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.683 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:19.683 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.279461+0000", 2024-09-18T04:39:19.683 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:19.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.683 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.683 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:19.683 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.279712+0000", 2024-09-18T04:39:19.683 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00025065999999999999 2024-09-18T04:39:19.684 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.684 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.684 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:19.684 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.279751+0000", 2024-09-18T04:39:19.684 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9113999999999999e-05 2024-09-18T04:39:19.684 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.684 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.684 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:19.684 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.279768+0000", 2024-09-18T04:39:19.684 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7212999999999999e-05 2024-09-18T04:39:19.684 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.684 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.684 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:19.684 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.291737+0000", 2024-09-18T04:39:19.685 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011968673000000001 2024-09-18T04:39:19.685 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.685 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.685 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:19.685 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.291862+0000", 2024-09-18T04:39:19.685 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000125282 2024-09-18T04:39:19.685 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.685 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.685 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:19.685 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.291947+0000", 2024-09-18T04:39:19.685 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5424000000000001e-05 2024-09-18T04:39:19.685 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.685 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.686 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:19.686 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.294683+0000", 2024-09-18T04:39:19.686 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027360589999999999 2024-09-18T04:39:19.686 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.686 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.686 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:19.686 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.294720+0000", 2024-09-18T04:39:19.686 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6191e-05 2024-09-18T04:39:19.686 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.686 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.686 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:19.686 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.294807+0000", 2024-09-18T04:39:19.686 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7317000000000004e-05 2024-09-18T04:39:19.686 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.687 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.687 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:19.687 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.294843+0000", 2024-09-18T04:39:19.687 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6180999999999998e-05 2024-09-18T04:39:19.687 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.687 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:19.687 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.687 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.687 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.687 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.5246.0:99 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [write 917504~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-18T04:39:19.687 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:15.279786+0000", 2024-09-18T04:39:19.687 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.320043096, 2024-09-18T04:39:19.687 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015395890000000001, 2024-09-18T04:39:19.687 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:19.687 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:19.688 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:19.688 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.5246", 2024-09-18T04:39:19.688 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2208214885", 2024-09-18T04:39:19.688 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 99 2024-09-18T04:39:19.688 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.688 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:19.688 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.688 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:19.688 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.279786+0000", 2024-09-18T04:39:19.688 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.688 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.688 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.688 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:19.688 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.279786+0000", 2024-09-18T04:39:19.689 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.689 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.689 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:19.689 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.279785+0000", 2024-09-18T04:39:19.689 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:19.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.689 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.689 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:19.689 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.280030+0000", 2024-09-18T04:39:19.689 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00024536600000000001 2024-09-18T04:39:19.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.689 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.689 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:19.690 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.280066+0000", 2024-09-18T04:39:19.690 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5762000000000001e-05 2024-09-18T04:39:19.690 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.690 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.690 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:19.690 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.280084+0000", 2024-09-18T04:39:19.690 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8122e-05 2024-09-18T04:39:19.690 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.690 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.690 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:19.690 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.292557+0000", 2024-09-18T04:39:19.690 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012472584 2024-09-18T04:39:19.690 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.690 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.690 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:19.691 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.292742+0000", 2024-09-18T04:39:19.691 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000185262 2024-09-18T04:39:19.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.691 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.691 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:19.691 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.293177+0000", 2024-09-18T04:39:19.691 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000434852 2024-09-18T04:39:19.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.691 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.691 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:19.691 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.295110+0000", 2024-09-18T04:39:19.691 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001933216 2024-09-18T04:39:19.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.691 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.692 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:19.692 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.295116+0000", 2024-09-18T04:39:19.692 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.806e-06 2024-09-18T04:39:19.692 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.692 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.692 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:19.692 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.295149+0000", 2024-09-18T04:39:19.692 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3544999999999998e-05 2024-09-18T04:39:19.692 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.692 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.692 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:19.692 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.295182+0000", 2024-09-18T04:39:19.692 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2765000000000002e-05 2024-09-18T04:39:19.693 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.693 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:19.693 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.693 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.693 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.693 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.5246.0:100 2.1 2:9370a598:::rbd_header.125fc09df6d1:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-18T04:39:19.693 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:15.280425+0000", 2024-09-18T04:39:19.693 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.3194045269999997, 2024-09-18T04:39:19.693 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014635731000000001, 2024-09-18T04:39:19.693 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:19.693 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:19.693 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:19.693 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.5246", 2024-09-18T04:39:19.693 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2208214885", 2024-09-18T04:39:19.694 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 100 2024-09-18T04:39:19.694 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.694 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:19.694 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.694 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:19.694 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.280425+0000", 2024-09-18T04:39:19.694 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.694 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.694 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.694 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:19.694 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.280425+0000", 2024-09-18T04:39:19.694 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.694 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.694 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.695 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:19.695 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.280423+0000", 2024-09-18T04:39:19.695 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:19.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.695 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.695 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:19.695 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.280437+0000", 2024-09-18T04:39:19.695 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3956e-05 2024-09-18T04:39:19.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.695 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.695 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:19.695 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.280473+0000", 2024-09-18T04:39:19.695 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6028000000000001e-05 2024-09-18T04:39:19.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.696 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.696 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:19.696 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.280491+0000", 2024-09-18T04:39:19.696 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.859e-05 2024-09-18T04:39:19.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.696 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.696 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:19.696 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.294895+0000", 2024-09-18T04:39:19.696 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014403368 2024-09-18T04:39:19.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.696 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.696 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:19.696 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.294963+0000", 2024-09-18T04:39:19.696 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8802999999999998e-05 2024-09-18T04:39:19.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.697 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.697 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:19.697 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.295061+0000", 2024-09-18T04:39:19.697 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.7082000000000003e-05 2024-09-18T04:39:19.697 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.697 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:19.697 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.697 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.697 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.5246.0:195 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [write 262144~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-18T04:39:19.697 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:15.591182+0000", 2024-09-18T04:39:19.697 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.0086470390000004, 2024-09-18T04:39:19.698 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014381026999999999, 2024-09-18T04:39:19.698 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:19.698 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:19.698 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:19.698 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.5246", 2024-09-18T04:39:19.698 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2208214885", 2024-09-18T04:39:19.698 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 195 2024-09-18T04:39:19.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.698 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:19.698 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.698 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:19.698 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.591182+0000", 2024-09-18T04:39:19.698 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.699 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.699 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:19.699 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.591182+0000", 2024-09-18T04:39:19.699 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.699 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.699 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:19.699 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.591181+0000", 2024-09-18T04:39:19.699 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:19.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.699 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.699 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:19.699 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.591300+0000", 2024-09-18T04:39:19.699 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000119091 2024-09-18T04:39:19.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.700 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.700 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:19.700 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.591323+0000", 2024-09-18T04:39:19.700 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2442e-05 2024-09-18T04:39:19.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.700 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.700 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:19.700 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.591341+0000", 2024-09-18T04:39:19.700 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8093000000000001e-05 2024-09-18T04:39:19.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.700 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.700 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:19.700 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.598814+0000", 2024-09-18T04:39:19.701 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0074728829999999996 2024-09-18T04:39:19.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.701 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.701 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:19.701 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.599015+0000", 2024-09-18T04:39:19.701 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000201466 2024-09-18T04:39:19.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.701 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.701 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:19.701 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.599129+0000", 2024-09-18T04:39:19.701 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000114305 2024-09-18T04:39:19.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.701 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.701 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:19.702 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.603276+0000", 2024-09-18T04:39:19.702 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0041468959999999997 2024-09-18T04:39:19.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.702 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.702 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:19.702 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.605494+0000", 2024-09-18T04:39:19.702 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0022182180000000001 2024-09-18T04:39:19.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.702 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.702 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:19.702 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.605539+0000", 2024-09-18T04:39:19.702 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4435999999999999e-05 2024-09-18T04:39:19.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.702 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.703 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:19.703 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.605563+0000", 2024-09-18T04:39:19.703 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4425000000000001e-05 2024-09-18T04:39:19.703 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.703 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:19.703 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.703 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.703 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.5246.0:199 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [write 786432~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-18T04:39:19.703 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:15.591924+0000", 2024-09-18T04:39:19.703 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.0079055300000004, 2024-09-18T04:39:19.703 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014640029000000001, 2024-09-18T04:39:19.703 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:19.703 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:19.703 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:19.704 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.5246", 2024-09-18T04:39:19.704 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2208214885", 2024-09-18T04:39:19.704 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 199 2024-09-18T04:39:19.704 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.704 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:19.704 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.704 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:19.704 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.591924+0000", 2024-09-18T04:39:19.704 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.704 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.704 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.704 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:19.704 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.591924+0000", 2024-09-18T04:39:19.704 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.705 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.705 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:19.705 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.591923+0000", 2024-09-18T04:39:19.705 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:19.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.705 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.705 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:19.705 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.592050+0000", 2024-09-18T04:39:19.705 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012711999999999999 2024-09-18T04:39:19.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.705 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.705 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:19.705 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.592072+0000", 2024-09-18T04:39:19.705 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2044000000000001e-05 2024-09-18T04:39:19.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.706 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.706 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:19.706 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.592104+0000", 2024-09-18T04:39:19.706 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2258000000000002e-05 2024-09-18T04:39:19.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.706 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.706 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:19.706 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.604535+0000", 2024-09-18T04:39:19.706 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012430462999999999 2024-09-18T04:39:19.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.706 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.706 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:19.706 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.604580+0000", 2024-09-18T04:39:19.707 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5158999999999999e-05 2024-09-18T04:39:19.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.707 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.707 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:19.707 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.604648+0000", 2024-09-18T04:39:19.707 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8516000000000006e-05 2024-09-18T04:39:19.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.707 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.707 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:19.707 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.606487+0000", 2024-09-18T04:39:19.707 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0018382839999999999 2024-09-18T04:39:19.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.707 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.707 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:19.707 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.606497+0000", 2024-09-18T04:39:19.708 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9979999999999998e-06 2024-09-18T04:39:19.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.708 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.708 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:19.708 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.606538+0000", 2024-09-18T04:39:19.708 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1600999999999997e-05 2024-09-18T04:39:19.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.708 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.708 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:19.708 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.606564+0000", 2024-09-18T04:39:19.708 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5630999999999999e-05 2024-09-18T04:39:19.708 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.708 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:19.708 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.709 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.709 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.709 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.5246.0:200 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [write 917504~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-18T04:39:19.709 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:15.592117+0000", 2024-09-18T04:39:19.709 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.007712078, 2024-09-18T04:39:19.709 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015242980999999999, 2024-09-18T04:39:19.709 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:19.709 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:19.709 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:19.709 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.5246", 2024-09-18T04:39:19.709 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2208214885", 2024-09-18T04:39:19.709 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 200 2024-09-18T04:39:19.709 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.709 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:19.710 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.710 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:19.710 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.592117+0000", 2024-09-18T04:39:19.710 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.710 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.710 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.710 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:19.710 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.592117+0000", 2024-09-18T04:39:19.710 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:19.710 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.710 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.710 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:19.710 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.592116+0000", 2024-09-18T04:39:19.710 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:19.710 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.711 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.711 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:19.711 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.592207+0000", 2024-09-18T04:39:19.711 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0478999999999999e-05 2024-09-18T04:39:19.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.711 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.711 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:19.711 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.592235+0000", 2024-09-18T04:39:19.711 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8759e-05 2024-09-18T04:39:19.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.711 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.711 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:19.711 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.592250+0000", 2024-09-18T04:39:19.711 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4073000000000001e-05 2024-09-18T04:39:19.712 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.712 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.712 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:19.712 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.605911+0000", 2024-09-18T04:39:19.712 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013661227 2024-09-18T04:39:19.712 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.712 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.712 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:19.712 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.605972+0000", 2024-09-18T04:39:19.712 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0999000000000003e-05 2024-09-18T04:39:19.712 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.712 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.712 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:19.712 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.606069+0000", 2024-09-18T04:39:19.712 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6751999999999998e-05 2024-09-18T04:39:19.713 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.713 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.713 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:19.713 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.606786+0000", 2024-09-18T04:39:19.713 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00071704199999999998 2024-09-18T04:39:19.713 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.713 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.713 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:19.713 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.607315+0000", 2024-09-18T04:39:19.713 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00052962400000000002 2024-09-18T04:39:19.713 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.713 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.713 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:19.713 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.607342+0000", 2024-09-18T04:39:19.713 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6658000000000001e-05 2024-09-18T04:39:19.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:19.714 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:19.714 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:19.714 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.607360+0000", 2024-09-18T04:39:19.714 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8331000000000001e-05 2024-09-18T04:39:19.714 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.714 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:19.714 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.714 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:19.714 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:19.714 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:19.716 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:39:19.716 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:19.709+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:39:19.816 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:19.813+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:39:19.834 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:19.834 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:19.834 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:39:19.834 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:19.844 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:39:19.917 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:19.909+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:19.946 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:19.946 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:19.946 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:39:19.946 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:39:19.947 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:19.956 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:39:20.018 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:20.013+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:20.063 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:20.063 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:39:20.063 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:39:20.063 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:39:20.064 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.064 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:39:20.064 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:39:20.064 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 112.270805928, 2024-09-18T04:39:20.064 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017399218000000001, 2024-09-18T04:39:20.064 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:20.064 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:20.064 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:20.064 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:39:20.064 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:39:20.064 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 114 2024-09-18T04:39:20.064 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.065 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:20.065 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.065 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:20.065 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:39:20.065 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:20.065 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.065 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.065 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:20.065 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:39:20.065 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:20.065 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.065 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.065 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:20.065 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786900+0000", 2024-09-18T04:39:20.066 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:20.066 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.066 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.066 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:20.066 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786908+0000", 2024-09-18T04:39:20.066 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8199999999999997e-06 2024-09-18T04:39:20.066 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.066 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.066 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:20.066 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786925+0000", 2024-09-18T04:39:20.066 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6849000000000001e-05 2024-09-18T04:39:20.066 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.066 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.066 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:20.067 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786934+0000", 2024-09-18T04:39:20.067 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6299999999999993e-06 2024-09-18T04:39:20.067 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.067 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.067 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:20.067 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800926+0000", 2024-09-18T04:39:20.067 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013991632 2024-09-18T04:39:20.067 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.067 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.067 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:20.067 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801060+0000", 2024-09-18T04:39:20.067 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013378199999999999 2024-09-18T04:39:20.068 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.068 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.068 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:20.068 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801166+0000", 2024-09-18T04:39:20.068 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010607099999999999 2024-09-18T04:39:20.068 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.068 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.068 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:20.068 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803026+0000", 2024-09-18T04:39:20.068 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001860469 2024-09-18T04:39:20.068 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.068 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.068 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:20.069 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804245+0000", 2024-09-18T04:39:20.069 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0012185900000000001 2024-09-18T04:39:20.069 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.069 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.069 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:20.069 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804278+0000", 2024-09-18T04:39:20.069 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3272000000000003e-05 2024-09-18T04:39:20.069 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.069 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.069 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:20.069 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804300+0000", 2024-09-18T04:39:20.069 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2266e-05 2024-09-18T04:39:20.069 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:20.069 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:20.070 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:20.070 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.070 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.070 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:39:20.070 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:39:20.070 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 112.26989303000001, 2024-09-18T04:39:20.070 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017521662, 2024-09-18T04:39:20.070 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:20.070 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:20.071 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:20.071 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:39:20.071 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:39:20.071 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 116 2024-09-18T04:39:20.071 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.071 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:20.071 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.071 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:20.071 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:39:20.071 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:20.071 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.071 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.071 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:20.072 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:39:20.072 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:20.072 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.072 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.072 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:20.072 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787813+0000", 2024-09-18T04:39:20.072 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:20.072 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.072 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.072 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:20.072 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787824+0000", 2024-09-18T04:39:20.072 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1157e-05 2024-09-18T04:39:20.072 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.072 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.073 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:20.073 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787840+0000", 2024-09-18T04:39:20.073 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6269e-05 2024-09-18T04:39:20.073 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.073 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.073 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:20.073 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787853+0000", 2024-09-18T04:39:20.073 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3156000000000001e-05 2024-09-18T04:39:20.073 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.073 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.073 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:20.073 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803110+0000", 2024-09-18T04:39:20.073 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015256736 2024-09-18T04:39:20.074 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.074 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.074 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:20.074 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803197+0000", 2024-09-18T04:39:20.074 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7195999999999997e-05 2024-09-18T04:39:20.074 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.074 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.074 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:20.074 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803308+0000", 2024-09-18T04:39:20.074 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111005 2024-09-18T04:39:20.074 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.074 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.074 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:20.074 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804316+0000", 2024-09-18T04:39:20.075 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010073949999999999 2024-09-18T04:39:20.075 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.075 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.075 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:20.075 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805232+0000", 2024-09-18T04:39:20.075 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091636300000000001 2024-09-18T04:39:20.075 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.075 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.075 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:20.075 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805277+0000", 2024-09-18T04:39:20.075 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4864000000000003e-05 2024-09-18T04:39:20.075 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.075 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.075 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:20.076 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805336+0000", 2024-09-18T04:39:20.076 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8968000000000002e-05 2024-09-18T04:39:20.076 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:20.076 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:20.076 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:20.076 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.076 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.076 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:39:20.076 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:39:20.076 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 112.26984097499999, 2024-09-18T04:39:20.076 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017983453, 2024-09-18T04:39:20.076 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:20.076 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:20.077 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:20.077 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:39:20.077 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:39:20.077 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 117 2024-09-18T04:39:20.077 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.077 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:20.077 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.077 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:20.077 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:39:20.077 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:20.077 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.077 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.077 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:20.078 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:39:20.078 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:20.078 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.078 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.078 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:20.078 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787865+0000", 2024-09-18T04:39:20.078 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:20.078 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.078 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.078 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:20.078 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787889+0000", 2024-09-18T04:39:20.078 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4244999999999999e-05 2024-09-18T04:39:20.078 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.079 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.079 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:20.079 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787905+0000", 2024-09-18T04:39:20.079 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5739999999999998e-05 2024-09-18T04:39:20.079 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.079 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.079 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:20.079 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787918+0000", 2024-09-18T04:39:20.079 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.252e-05 2024-09-18T04:39:20.079 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.079 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.079 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:20.079 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803693+0000", 2024-09-18T04:39:20.079 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015774872999999998 2024-09-18T04:39:20.080 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.080 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.080 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:20.080 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803775+0000", 2024-09-18T04:39:20.080 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2761000000000003e-05 2024-09-18T04:39:20.080 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.080 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.080 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:20.080 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803887+0000", 2024-09-18T04:39:20.080 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111708 2024-09-18T04:39:20.080 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.080 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.081 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:20.081 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804823+0000", 2024-09-18T04:39:20.081 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00093625100000000003 2024-09-18T04:39:20.081 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.081 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.081 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:20.081 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805804+0000", 2024-09-18T04:39:20.081 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00098114699999999992 2024-09-18T04:39:20.081 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.081 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.081 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:20.082 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805831+0000", 2024-09-18T04:39:20.082 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6401999999999999e-05 2024-09-18T04:39:20.082 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.082 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.082 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:20.082 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805850+0000", 2024-09-18T04:39:20.082 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8862e-05 2024-09-18T04:39:20.082 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:20.082 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:20.082 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:20.082 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.082 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.082 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:36 2.7 2:fe1b02ad:::rbd_data.125fc09df6d1.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:20.083 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:39:20.083 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 37.791984865000003, 2024-09-18T04:39:20.083 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.040387804999999999, 2024-09-18T04:39:20.083 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:20.083 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:20.083 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:20.083 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:20.083 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:20.083 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 36 2024-09-18T04:39:20.083 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.083 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:20.083 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.083 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:20.083 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:39:20.084 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:20.084 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.084 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.084 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:20.084 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:39:20.084 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:20.084 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.084 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.084 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:20.084 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265720+0000", 2024-09-18T04:39:20.084 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:20.084 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.084 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.084 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:20.085 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272590+0000", 2024-09-18T04:39:20.085 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.006870508 2024-09-18T04:39:20.085 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.085 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.085 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:20.085 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272623+0000", 2024-09-18T04:39:20.085 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2716000000000003e-05 2024-09-18T04:39:20.085 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.085 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.085 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:20.085 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272637+0000", 2024-09-18T04:39:20.085 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4175999999999999e-05 2024-09-18T04:39:20.085 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.085 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.086 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:20.086 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272779+0000", 2024-09-18T04:39:20.086 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000141392 2024-09-18T04:39:20.086 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.086 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.086 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:20.086 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273031+0000", 2024-09-18T04:39:20.086 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00025249099999999998 2024-09-18T04:39:20.086 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.086 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.086 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:39:20.086 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273219+0000", 2024-09-18T04:39:20.086 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000187531 2024-09-18T04:39:20.087 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.087 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.087 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:20.087 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.302806+0000", 2024-09-18T04:39:20.087 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.029587553999999999 2024-09-18T04:39:20.087 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.087 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.087 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:20.087 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306004+0000", 2024-09-18T04:39:20.087 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0031982629999999998 2024-09-18T04:39:20.087 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.087 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.087 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:20.088 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306071+0000", 2024-09-18T04:39:20.088 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6829999999999995e-05 2024-09-18T04:39:20.088 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.088 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.088 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:20.088 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306110+0000", 2024-09-18T04:39:20.088 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8874000000000001e-05 2024-09-18T04:39:20.088 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:20.088 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:20.088 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:20.088 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.088 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.088 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:38 2.0 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:20.088 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:39:20.089 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 37.785031328000002, 2024-09-18T04:39:20.089 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.033754348000000003, 2024-09-18T04:39:20.089 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:20.089 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:20.089 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:20.089 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:20.089 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:20.089 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 38 2024-09-18T04:39:20.089 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.089 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:20.089 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.089 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:20.089 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:39:20.089 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:20.089 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.090 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.090 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:20.090 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:39:20.090 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:20.090 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.090 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.090 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:20.090 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272673+0000", 2024-09-18T04:39:20.090 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:20.090 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.090 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.091 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:20.091 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277166+0000", 2024-09-18T04:39:20.091 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044922249999999999 2024-09-18T04:39:20.091 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.091 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.091 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:20.091 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277189+0000", 2024-09-18T04:39:20.091 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3283000000000001e-05 2024-09-18T04:39:20.091 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.091 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.091 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:20.091 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277205+0000", 2024-09-18T04:39:20.091 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6305000000000001e-05 2024-09-18T04:39:20.091 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.092 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.092 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:20.092 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277290+0000", 2024-09-18T04:39:20.092 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5135000000000004e-05 2024-09-18T04:39:20.092 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.092 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.092 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:20.092 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277531+0000", 2024-09-18T04:39:20.092 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00024014699999999999 2024-09-18T04:39:20.092 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.092 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.092 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:20.092 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277793+0000", 2024-09-18T04:39:20.093 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00026224499999999998 2024-09-18T04:39:20.093 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.093 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.093 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:20.093 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303990+0000", 2024-09-18T04:39:20.093 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.026197295999999998 2024-09-18T04:39:20.093 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.093 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.093 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:20.093 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306340+0000", 2024-09-18T04:39:20.093 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.002349524 2024-09-18T04:39:20.093 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.093 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.094 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:20.094 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306394+0000", 2024-09-18T04:39:20.094 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3934999999999998e-05 2024-09-18T04:39:20.094 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.094 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.094 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:20.094 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306430+0000", 2024-09-18T04:39:20.094 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6690000000000003e-05 2024-09-18T04:39:20.094 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:20.094 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:20.094 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:20.094 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.094 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.094 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:39 2.7 2:e9c909c5:::rbd_data.125fc09df6d1.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:20.095 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:39:20.095 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 37.780471722999998, 2024-09-18T04:39:20.095 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.029110673, 2024-09-18T04:39:20.095 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:20.095 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:20.095 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:20.095 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:20.095 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:20.095 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 39 2024-09-18T04:39:20.095 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.095 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:20.095 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.095 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:20.096 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:39:20.096 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:20.096 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.096 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.096 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:20.096 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:39:20.096 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:20.096 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.096 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.096 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:20.096 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277234+0000", 2024-09-18T04:39:20.096 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:20.096 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.096 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.097 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:20.097 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281351+0000", 2024-09-18T04:39:20.097 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0041170920000000001 2024-09-18T04:39:20.097 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.097 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.097 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:20.097 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281375+0000", 2024-09-18T04:39:20.097 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.446e-05 2024-09-18T04:39:20.097 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.097 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.097 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:20.097 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281390+0000", 2024-09-18T04:39:20.097 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4309e-05 2024-09-18T04:39:20.097 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.098 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.098 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:20.098 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.282306+0000", 2024-09-18T04:39:20.098 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091662000000000004 2024-09-18T04:39:20.098 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.098 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.098 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:20.098 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.282613+0000", 2024-09-18T04:39:20.098 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00030663799999999997 2024-09-18T04:39:20.098 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.098 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.098 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:39:20.098 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283001+0000", 2024-09-18T04:39:20.098 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00038801599999999998 2024-09-18T04:39:20.099 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.099 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.099 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:20.099 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306239+0000", 2024-09-18T04:39:20.099 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023238466999999999 2024-09-18T04:39:20.099 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.099 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.099 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:20.099 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306259+0000", 2024-09-18T04:39:20.099 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9837999999999999e-05 2024-09-18T04:39:20.099 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.099 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.099 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:20.099 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306299+0000", 2024-09-18T04:39:20.100 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0194000000000002e-05 2024-09-18T04:39:20.100 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.100 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.100 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:20.100 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306346+0000", 2024-09-18T04:39:20.100 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6845000000000003e-05 2024-09-18T04:39:20.100 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:20.100 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:20.100 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:20.100 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.100 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.100 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:46 2.6 e51/17 2:7ba25245:::rbd_data.125fc09df6d1.0000000000000006:head v 51'2, mlcod=0'0)", 2024-09-18T04:39:20.100 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:39:20.101 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 37.744550590999999, 2024-09-18T04:39:20.101 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019843098, 2024-09-18T04:39:20.101 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:20.101 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:20.101 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:20.101 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.101 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:20.101 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:39:20.101 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:20.101 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.101 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.101 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:20.101 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:39:20.101 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:20.102 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.102 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.102 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:20.102 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313156+0000", 2024-09-18T04:39:20.102 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:20.102 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.102 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.102 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:20.102 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320144+0000", 2024-09-18T04:39:20.102 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.006988345 2024-09-18T04:39:20.102 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.102 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.102 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:20.103 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320175+0000", 2024-09-18T04:39:20.103 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.0861999999999997e-05 2024-09-18T04:39:20.103 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.103 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.103 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:20.103 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320188+0000", 2024-09-18T04:39:20.103 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2714999999999999e-05 2024-09-18T04:39:20.103 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.103 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.103 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:20.103 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320267+0000", 2024-09-18T04:39:20.103 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9586999999999995e-05 2024-09-18T04:39:20.103 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.103 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.104 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:20.104 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320311+0000", 2024-09-18T04:39:20.104 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3529000000000003e-05 2024-09-18T04:39:20.104 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.104 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.104 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:20.104 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.332949+0000", 2024-09-18T04:39:20.104 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012638267999999999 2024-09-18T04:39:20.104 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.104 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.104 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:20.104 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.333000+0000", 2024-09-18T04:39:20.104 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0736999999999999e-05 2024-09-18T04:39:20.104 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:20.105 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:20.105 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:20.105 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.105 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.105 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:57 2.0 2:0cdd070e:::rbd_data.125fc09df6d1.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:20.105 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:39:20.105 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 37.708693015999998, 2024-09-18T04:39:20.105 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.033329974999999998, 2024-09-18T04:39:20.105 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:20.105 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:20.105 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:20.105 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:20.105 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:20.106 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 57 2024-09-18T04:39:20.106 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.106 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:20.106 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.106 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:20.106 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:39:20.106 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:20.106 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.106 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.106 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:20.106 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:39:20.106 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:20.106 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.106 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.107 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:20.107 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349013+0000", 2024-09-18T04:39:20.107 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:20.107 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.107 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.107 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:20.107 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356138+0000", 2024-09-18T04:39:20.107 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071258010000000002 2024-09-18T04:39:20.107 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.107 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.107 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:20.107 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356168+0000", 2024-09-18T04:39:20.107 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9679000000000002e-05 2024-09-18T04:39:20.108 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.108 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.108 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:20.108 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356181+0000", 2024-09-18T04:39:20.108 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3164e-05 2024-09-18T04:39:20.108 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.108 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.108 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:20.108 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356254+0000", 2024-09-18T04:39:20.108 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3182999999999994e-05 2024-09-18T04:39:20.108 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.108 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.108 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:20.109 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356453+0000", 2024-09-18T04:39:20.109 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019876599999999999 2024-09-18T04:39:20.109 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.109 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.109 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:20.109 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356627+0000", 2024-09-18T04:39:20.109 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017340199999999999 2024-09-18T04:39:20.109 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.109 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.109 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:20.109 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.369200+0000", 2024-09-18T04:39:20.109 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012573302 2024-09-18T04:39:20.109 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.110 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.110 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:20.110 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382262+0000", 2024-09-18T04:39:20.110 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013062549 2024-09-18T04:39:20.110 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.110 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.110 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:20.110 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382310+0000", 2024-09-18T04:39:20.110 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7366e-05 2024-09-18T04:39:20.110 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.110 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.110 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:20.110 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382344+0000", 2024-09-18T04:39:20.110 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4319999999999997e-05 2024-09-18T04:39:20.111 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:20.111 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:20.111 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:20.111 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.111 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.111 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:59 2.0 2:19c52cde:::rbd_data.125fc09df6d1.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:20.111 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:39:20.111 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 37.701501851000003, 2024-09-18T04:39:20.111 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.027137247999999999, 2024-09-18T04:39:20.111 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:20.111 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:20.111 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:20.111 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:20.112 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:20.112 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 59 2024-09-18T04:39:20.112 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.112 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:20.112 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.112 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:20.112 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:39:20.112 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:20.112 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.112 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.112 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:20.112 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:39:20.112 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:20.112 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.113 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.113 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:20.113 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356204+0000", 2024-09-18T04:39:20.113 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:20.113 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.113 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.113 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:20.113 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362661+0000", 2024-09-18T04:39:20.113 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0064565129999999997 2024-09-18T04:39:20.113 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.113 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.113 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:20.113 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362684+0000", 2024-09-18T04:39:20.114 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3688000000000001e-05 2024-09-18T04:39:20.114 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.114 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.114 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:20.114 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362699+0000", 2024-09-18T04:39:20.114 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4798e-05 2024-09-18T04:39:20.114 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.114 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.114 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:20.114 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363016+0000", 2024-09-18T04:39:20.114 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031737300000000003 2024-09-18T04:39:20.114 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.114 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.115 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:20.115 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363201+0000", 2024-09-18T04:39:20.115 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000184131 2024-09-18T04:39:20.115 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.115 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.115 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:20.115 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363343+0000", 2024-09-18T04:39:20.115 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014236699999999999 2024-09-18T04:39:20.115 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.115 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.115 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:20.115 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.372189+0000", 2024-09-18T04:39:20.115 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0088464819999999993 2024-09-18T04:39:20.115 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.116 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.116 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:20.116 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383269+0000", 2024-09-18T04:39:20.116 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011079594 2024-09-18T04:39:20.116 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.116 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.116 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:20.116 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383314+0000", 2024-09-18T04:39:20.116 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4932000000000003e-05 2024-09-18T04:39:20.116 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.116 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.116 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:20.116 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383343+0000", 2024-09-18T04:39:20.116 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8600999999999999e-05 2024-09-18T04:39:20.116 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:20.117 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:20.117 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:20.117 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.117 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.117 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:40 2.6 e51/17 2:784d6cd1:ns::rbd_data.133e813ab5bc.0000000000000005:head v 51'7, mlcod=51'5)", 2024-09-18T04:39:20.117 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:39:20.117 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 37.505252116999998, 2024-09-18T04:39:20.117 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020497496, 2024-09-18T04:39:20.117 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:20.117 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:20.117 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:20.117 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.117 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:20.118 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:39:20.118 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:20.118 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.118 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.118 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:20.118 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:39:20.118 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:20.118 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.118 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.118 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:20.118 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552454+0000", 2024-09-18T04:39:20.118 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:20.118 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.118 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.119 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:20.119 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556448+0000", 2024-09-18T04:39:20.119 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0039940990000000001 2024-09-18T04:39:20.119 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.119 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.119 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:20.119 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556481+0000", 2024-09-18T04:39:20.119 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3778000000000001e-05 2024-09-18T04:39:20.119 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.119 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.119 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:20.119 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556493+0000", 2024-09-18T04:39:20.119 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1622e-05 2024-09-18T04:39:20.120 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.120 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.120 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:20.120 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556556+0000", 2024-09-18T04:39:20.120 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2582999999999994e-05 2024-09-18T04:39:20.120 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.120 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.120 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:20.120 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556592+0000", 2024-09-18T04:39:20.120 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5880999999999997e-05 2024-09-18T04:39:20.120 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.120 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.120 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:20.121 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.572786+0000", 2024-09-18T04:39:20.121 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016194571000000001 2024-09-18T04:39:20.121 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.121 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.121 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:20.121 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.572953+0000", 2024-09-18T04:39:20.121 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000166436 2024-09-18T04:39:20.121 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:20.121 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:20.121 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:20.121 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.121 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.121 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:45 2.0 2:0fbfa3dc:ns::rbd_data.133e813ab5bc.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:20.121 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:39:20.122 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 37.494037818000002, 2024-09-18T04:39:20.122 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.036558168000000002, 2024-09-18T04:39:20.122 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:20.122 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:20.122 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:20.122 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:20.122 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:20.122 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 45 2024-09-18T04:39:20.122 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.122 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:20.122 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.122 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:20.122 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:39:20.122 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:20.123 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.123 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.123 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:20.123 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:39:20.123 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:20.123 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.123 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.123 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:20.123 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:39:20.123 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:20.123 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.123 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.124 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:20.124 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578594+0000", 2024-09-18T04:39:20.124 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014927147999999999 2024-09-18T04:39:20.124 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.124 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.124 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:20.124 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578612+0000", 2024-09-18T04:39:20.124 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7623999999999999e-05 2024-09-18T04:39:20.124 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.124 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.124 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:20.124 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578620+0000", 2024-09-18T04:39:20.125 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9070000000000007e-06 2024-09-18T04:39:20.125 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.125 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.125 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:20.125 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578702+0000", 2024-09-18T04:39:20.125 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2349999999999996e-05 2024-09-18T04:39:20.125 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.125 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.125 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:20.125 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.579028+0000", 2024-09-18T04:39:20.125 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032622899999999998 2024-09-18T04:39:20.125 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.125 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.125 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:20.126 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.579343+0000", 2024-09-18T04:39:20.126 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031448899999999999 2024-09-18T04:39:20.126 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.126 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.126 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:20.126 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.590078+0000", 2024-09-18T04:39:20.126 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010734946 2024-09-18T04:39:20.126 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.126 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.126 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:20.126 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600089+0000", 2024-09-18T04:39:20.126 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010011195000000001 2024-09-18T04:39:20.126 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.127 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.127 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:20.127 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600177+0000", 2024-09-18T04:39:20.127 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8425999999999997e-05 2024-09-18T04:39:20.127 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.127 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.127 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:20.127 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600228+0000", 2024-09-18T04:39:20.127 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0216999999999997e-05 2024-09-18T04:39:20.127 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:20.127 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:20.127 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:20.127 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.128 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.128 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:54 2.0 2:04f17ef6:ns::rbd_data.133e813ab5bc.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:20.128 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:39:20.128 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 37.472465980999999, 2024-09-18T04:39:20.128 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.024612999, 2024-09-18T04:39:20.128 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:20.128 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:20.128 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:20.128 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:20.128 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:20.128 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 54 2024-09-18T04:39:20.128 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.128 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:20.128 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.129 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:20.129 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:39:20.129 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:20.129 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.129 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.129 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:20.129 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:39:20.129 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:20.129 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.129 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.129 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:20.129 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585239+0000", 2024-09-18T04:39:20.129 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:20.129 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.130 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.130 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:20.130 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593702+0000", 2024-09-18T04:39:20.130 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0084632990000000005 2024-09-18T04:39:20.130 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.130 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.130 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:20.130 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593719+0000", 2024-09-18T04:39:20.130 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.664e-05 2024-09-18T04:39:20.130 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.130 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.130 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:20.130 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593727+0000", 2024-09-18T04:39:20.130 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6839999999999996e-06 2024-09-18T04:39:20.130 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.131 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.131 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:20.131 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593767+0000", 2024-09-18T04:39:20.131 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0330999999999997e-05 2024-09-18T04:39:20.131 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.131 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.131 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:20.131 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593905+0000", 2024-09-18T04:39:20.131 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013754800000000001 2024-09-18T04:39:20.131 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.131 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.131 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:20.131 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.594013+0000", 2024-09-18T04:39:20.131 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000108805 2024-09-18T04:39:20.132 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.132 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.132 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:20.132 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604678+0000", 2024-09-18T04:39:20.132 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010664391 2024-09-18T04:39:20.132 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.132 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.132 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:20.132 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609662+0000", 2024-09-18T04:39:20.132 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0049846100000000004 2024-09-18T04:39:20.132 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.132 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.132 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:20.132 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609788+0000", 2024-09-18T04:39:20.133 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012518800000000001 2024-09-18T04:39:20.133 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.133 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.133 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:20.133 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609854+0000", 2024-09-18T04:39:20.133 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6556000000000004e-05 2024-09-18T04:39:20.133 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:20.133 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:20.133 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:20.133 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.133 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.133 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:58 2.7 2:fc7c6cd3:ns::rbd_data.133e813ab5bc.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:20.133 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:39:20.134 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 37.459623551999996, 2024-09-18T04:39:20.134 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.027690566999999999, 2024-09-18T04:39:20.134 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:20.134 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:20.134 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:20.134 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:20.134 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:20.134 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 58 2024-09-18T04:39:20.134 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.134 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:20.134 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.134 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:20.134 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:39:20.134 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:20.135 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.135 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.135 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:20.135 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:39:20.135 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:20.135 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.135 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.135 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:20.135 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598082+0000", 2024-09-18T04:39:20.135 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:20.135 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.135 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.135 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:20.135 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604832+0000", 2024-09-18T04:39:20.136 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0067504510000000002 2024-09-18T04:39:20.136 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.136 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.136 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:20.136 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604852+0000", 2024-09-18T04:39:20.136 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0228999999999999e-05 2024-09-18T04:39:20.136 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.136 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.136 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:20.136 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604870+0000", 2024-09-18T04:39:20.136 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7952e-05 2024-09-18T04:39:20.136 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.136 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.136 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:20.137 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604936+0000", 2024-09-18T04:39:20.137 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5431000000000004e-05 2024-09-18T04:39:20.137 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.137 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.137 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:20.137 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605085+0000", 2024-09-18T04:39:20.137 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014910300000000001 2024-09-18T04:39:20.137 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.137 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.137 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:39:20.137 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605205+0000", 2024-09-18T04:39:20.137 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000119887 2024-09-18T04:39:20.137 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.138 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.138 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:20.138 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625586+0000", 2024-09-18T04:39:20.138 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020381399000000001 2024-09-18T04:39:20.138 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.138 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.138 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:20.138 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625628+0000", 2024-09-18T04:39:20.138 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1873999999999999e-05 2024-09-18T04:39:20.138 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.138 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.138 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:20.138 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625722+0000", 2024-09-18T04:39:20.138 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3787000000000002e-05 2024-09-18T04:39:20.139 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.139 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.139 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:20.139 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625774+0000", 2024-09-18T04:39:20.139 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2503999999999997e-05 2024-09-18T04:39:20.139 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:20.139 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:20.139 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:20.139 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.139 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.139 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:56 2.6 e51/17 2:72453c3e:ns::rbd_data.133e813ab5bc.000000000000000d:head v 51'10, mlcod=51'8)", 2024-09-18T04:39:20.140 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:39:20.140 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 37.452426443999997, 2024-09-18T04:39:20.140 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019766552999999999, 2024-09-18T04:39:20.140 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:20.140 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:20.140 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:20.140 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.140 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:20.140 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:39:20.140 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:20.140 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.140 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.140 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:20.141 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:39:20.141 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:20.141 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.141 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.141 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:20.141 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605279+0000", 2024-09-18T04:39:20.141 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:20.141 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.141 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.141 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:20.141 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609417+0000", 2024-09-18T04:39:20.141 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0041378059999999999 2024-09-18T04:39:20.141 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.142 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.142 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:20.142 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609443+0000", 2024-09-18T04:39:20.142 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6098e-05 2024-09-18T04:39:20.142 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.142 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.142 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:20.142 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609458+0000", 2024-09-18T04:39:20.142 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4671000000000001e-05 2024-09-18T04:39:20.142 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.142 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.142 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:20.143 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609553+0000", 2024-09-18T04:39:20.143 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5265000000000004e-05 2024-09-18T04:39:20.143 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.143 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.143 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:20.143 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609583+0000", 2024-09-18T04:39:20.143 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9839e-05 2024-09-18T04:39:20.143 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.144 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.144 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:20.144 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625004+0000", 2024-09-18T04:39:20.144 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015420639999999999 2024-09-18T04:39:20.144 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.144 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.144 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:20.144 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625047+0000", 2024-09-18T04:39:20.144 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3665999999999998e-05 2024-09-18T04:39:20.144 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:20.144 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:20.144 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:20.144 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.145 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.145 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:99 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:20.145 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.582718+0000", 2024-09-18T04:39:20.145 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 23.474989534999999, 2024-09-18T04:39:20.145 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019362575, 2024-09-18T04:39:20.145 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:20.145 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:20.145 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:20.145 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:20.145 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:20.145 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 99 2024-09-18T04:39:20.145 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.145 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:20.146 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.146 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:20.146 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582718+0000", 2024-09-18T04:39:20.146 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:20.146 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.146 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.146 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:20.146 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582718+0000", 2024-09-18T04:39:20.146 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:20.146 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.146 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.146 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:20.146 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582717+0000", 2024-09-18T04:39:20.146 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:20.147 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.147 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.147 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:20.147 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582720+0000", 2024-09-18T04:39:20.147 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2239999999999998e-06 2024-09-18T04:39:20.147 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.147 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.147 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:20.147 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582723+0000", 2024-09-18T04:39:20.147 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.258e-06 2024-09-18T04:39:20.147 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.147 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.147 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:20.147 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582726+0000", 2024-09-18T04:39:20.148 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8870000000000001e-06 2024-09-18T04:39:20.148 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.148 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.148 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:20.148 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.592018+0000", 2024-09-18T04:39:20.148 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0092926420000000003 2024-09-18T04:39:20.148 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.148 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.148 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:20.148 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.592214+0000", 2024-09-18T04:39:20.148 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000195267 2024-09-18T04:39:20.148 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.148 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.148 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:20.149 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.592913+0000", 2024-09-18T04:39:20.149 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00069924600000000005 2024-09-18T04:39:20.149 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.149 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.149 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:20.149 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597493+0000", 2024-09-18T04:39:20.149 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045797629999999997 2024-09-18T04:39:20.152 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.152 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.152 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:20.152 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.602009+0000", 2024-09-18T04:39:20.153 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045159559999999998 2024-09-18T04:39:20.153 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.153 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.153 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:20.153 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.602042+0000", 2024-09-18T04:39:20.153 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3084999999999999e-05 2024-09-18T04:39:20.153 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.153 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.153 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:20.153 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.602080+0000", 2024-09-18T04:39:20.153 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8600000000000003e-05 2024-09-18T04:39:20.153 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:20.153 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:20.154 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:20.154 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.154 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.154 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:104 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:20.154 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.583165+0000", 2024-09-18T04:39:20.154 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 23.474542237000001, 2024-09-18T04:39:20.154 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023679603, 2024-09-18T04:39:20.154 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:20.154 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:20.154 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:20.154 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:20.154 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:20.154 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 104 2024-09-18T04:39:20.155 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.155 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:20.155 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.155 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:20.155 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583165+0000", 2024-09-18T04:39:20.155 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:20.155 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.155 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.155 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:20.155 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583165+0000", 2024-09-18T04:39:20.155 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:20.155 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.155 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.156 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:20.156 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583164+0000", 2024-09-18T04:39:20.156 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:20.156 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.156 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.156 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:20.156 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583167+0000", 2024-09-18T04:39:20.156 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9210000000000001e-06 2024-09-18T04:39:20.156 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.156 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.156 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:20.156 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583171+0000", 2024-09-18T04:39:20.156 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9759999999999997e-06 2024-09-18T04:39:20.157 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.157 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.157 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:20.157 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583174+0000", 2024-09-18T04:39:20.157 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9960000000000002e-06 2024-09-18T04:39:20.157 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.157 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.157 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:20.157 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.594373+0000", 2024-09-18T04:39:20.157 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011198665999999999 2024-09-18T04:39:20.157 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.157 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.157 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:20.158 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.594434+0000", 2024-09-18T04:39:20.158 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.1446999999999996e-05 2024-09-18T04:39:20.158 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.158 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.158 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:20.158 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.594692+0000", 2024-09-18T04:39:20.158 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00025756599999999998 2024-09-18T04:39:20.158 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.158 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.158 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:20.158 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599302+0000", 2024-09-18T04:39:20.158 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0046099640000000002 2024-09-18T04:39:20.158 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.158 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.159 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:20.159 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606710+0000", 2024-09-18T04:39:20.159 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0074077739999999998 2024-09-18T04:39:20.159 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.159 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.159 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:20.159 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606798+0000", 2024-09-18T04:39:20.159 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8133999999999993e-05 2024-09-18T04:39:20.159 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.159 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.159 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:20.159 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606845+0000", 2024-09-18T04:39:20.159 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6653000000000002e-05 2024-09-18T04:39:20.159 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:20.160 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:20.160 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:20.160 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.160 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.160 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:105 2.0 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:20.160 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.583193+0000", 2024-09-18T04:39:20.160 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 23.474514618000001, 2024-09-18T04:39:20.160 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023713649, 2024-09-18T04:39:20.160 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:20.160 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:20.160 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:20.160 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:20.160 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:20.160 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 105 2024-09-18T04:39:20.161 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.161 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:20.161 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.161 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:20.161 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583193+0000", 2024-09-18T04:39:20.161 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:20.161 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.161 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.161 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:20.161 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583193+0000", 2024-09-18T04:39:20.161 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:20.161 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.161 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.161 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:20.162 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583192+0000", 2024-09-18T04:39:20.162 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:20.162 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.162 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.162 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:20.162 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583194+0000", 2024-09-18T04:39:20.162 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.305e-06 2024-09-18T04:39:20.162 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.162 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.162 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:20.162 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583198+0000", 2024-09-18T04:39:20.162 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9369999999999997e-06 2024-09-18T04:39:20.162 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.162 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.163 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:20.163 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583202+0000", 2024-09-18T04:39:20.163 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6480000000000001e-06 2024-09-18T04:39:20.163 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.163 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.163 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:20.163 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.595304+0000", 2024-09-18T04:39:20.163 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012102063999999999 2024-09-18T04:39:20.163 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.163 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.163 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:20.163 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.595390+0000", 2024-09-18T04:39:20.163 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5493000000000003e-05 2024-09-18T04:39:20.163 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.164 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.164 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:20.164 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.595509+0000", 2024-09-18T04:39:20.164 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011966199999999999 2024-09-18T04:39:20.164 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.164 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.164 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:20.164 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599358+0000", 2024-09-18T04:39:20.164 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0038486689999999999 2024-09-18T04:39:20.164 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.164 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.164 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:20.164 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606859+0000", 2024-09-18T04:39:20.164 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0075010789999999999 2024-09-18T04:39:20.165 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.165 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.165 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:20.165 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606887+0000", 2024-09-18T04:39:20.165 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7773e-05 2024-09-18T04:39:20.165 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.165 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.165 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:20.165 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606906+0000", 2024-09-18T04:39:20.165 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9417000000000001e-05 2024-09-18T04:39:20.165 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:20.165 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:20.165 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:20.166 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.166 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.166 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:106 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:20.166 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.584425+0000", 2024-09-18T04:39:20.166 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 23.473282321999999, 2024-09-18T04:39:20.166 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.022529293999999998, 2024-09-18T04:39:20.166 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:20.166 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:20.166 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:20.166 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:20.166 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:20.166 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 106 2024-09-18T04:39:20.166 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.166 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:20.167 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.167 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:20.167 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584425+0000", 2024-09-18T04:39:20.167 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:20.167 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.167 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.167 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:20.167 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584425+0000", 2024-09-18T04:39:20.167 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:20.167 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.167 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.167 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:20.167 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584424+0000", 2024-09-18T04:39:20.167 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:20.168 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.168 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.168 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:20.168 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584430+0000", 2024-09-18T04:39:20.168 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6389999999999997e-06 2024-09-18T04:39:20.168 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.168 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.168 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:20.168 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584456+0000", 2024-09-18T04:39:20.168 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6188999999999999e-05 2024-09-18T04:39:20.168 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.168 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.168 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:20.169 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584463+0000", 2024-09-18T04:39:20.169 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7340000000000002e-06 2024-09-18T04:39:20.169 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.169 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.169 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:20.169 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597537+0000", 2024-09-18T04:39:20.169 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013074161000000001 2024-09-18T04:39:20.169 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.169 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.169 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:20.169 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597598+0000", 2024-09-18T04:39:20.169 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0930000000000001e-05 2024-09-18T04:39:20.169 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.169 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.170 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:20.170 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597922+0000", 2024-09-18T04:39:20.170 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032473000000000002 2024-09-18T04:39:20.170 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.170 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.170 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:20.170 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.601103+0000", 2024-09-18T04:39:20.170 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0031809849999999999 2024-09-18T04:39:20.170 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.170 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.170 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:20.170 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606912+0000", 2024-09-18T04:39:20.170 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058083550000000003 2024-09-18T04:39:20.171 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.171 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.171 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:20.171 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606935+0000", 2024-09-18T04:39:20.171 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2759999999999999e-05 2024-09-18T04:39:20.171 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.171 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.171 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:20.171 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606954+0000", 2024-09-18T04:39:20.171 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9602999999999998e-05 2024-09-18T04:39:20.171 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:20.171 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:20.171 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:20.172 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.172 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.172 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:109 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:20.172 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.585231+0000", 2024-09-18T04:39:20.172 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 23.472475702000001, 2024-09-18T04:39:20.172 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.021767496000000001, 2024-09-18T04:39:20.172 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:20.172 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:20.172 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:20.172 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:20.172 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:20.172 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 109 2024-09-18T04:39:20.172 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.173 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:20.173 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.173 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:20.173 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585231+0000", 2024-09-18T04:39:20.173 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:20.173 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.173 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.173 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:20.173 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585231+0000", 2024-09-18T04:39:20.173 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:20.173 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.173 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.173 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:20.173 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585230+0000", 2024-09-18T04:39:20.174 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:20.174 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.174 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.174 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:20.174 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585237+0000", 2024-09-18T04:39:20.174 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.618e-06 2024-09-18T04:39:20.174 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.174 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.174 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:20.174 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585256+0000", 2024-09-18T04:39:20.174 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9048000000000001e-05 2024-09-18T04:39:20.174 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.174 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.174 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:20.174 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585274+0000", 2024-09-18T04:39:20.175 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8153000000000001e-05 2024-09-18T04:39:20.175 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.175 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.175 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:20.175 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.598284+0000", 2024-09-18T04:39:20.175 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013009682999999999 2024-09-18T04:39:20.175 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.175 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.175 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:20.175 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.598383+0000", 2024-09-18T04:39:20.175 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.8722999999999997e-05 2024-09-18T04:39:20.175 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.175 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.175 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:20.176 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.598829+0000", 2024-09-18T04:39:20.176 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00044623999999999998 2024-09-18T04:39:20.176 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.176 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.176 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:20.176 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.601149+0000", 2024-09-18T04:39:20.176 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0023204150000000002 2024-09-18T04:39:20.176 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.176 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.176 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:20.176 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606960+0000", 2024-09-18T04:39:20.176 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058106390000000003 2024-09-18T04:39:20.176 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.176 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.177 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:20.177 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606982+0000", 2024-09-18T04:39:20.177 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2252e-05 2024-09-18T04:39:20.177 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.177 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.177 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:20.177 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606999+0000", 2024-09-18T04:39:20.177 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6871000000000002e-05 2024-09-18T04:39:20.177 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:20.177 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:20.177 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:20.177 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.177 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.177 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:110 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:20.178 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.586640+0000", 2024-09-18T04:39:20.178 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 23.471067383000001, 2024-09-18T04:39:20.178 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020400911000000001, 2024-09-18T04:39:20.178 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:20.178 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:20.178 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:20.178 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:20.178 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:20.178 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 110 2024-09-18T04:39:20.178 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.178 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:20.178 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.178 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:20.178 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586640+0000", 2024-09-18T04:39:20.179 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:20.179 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.179 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.179 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:20.179 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586640+0000", 2024-09-18T04:39:20.179 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:20.179 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.179 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.179 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:20.179 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586639+0000", 2024-09-18T04:39:20.179 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:20.179 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.179 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.179 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:20.180 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586652+0000", 2024-09-18T04:39:20.180 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2482e-05 2024-09-18T04:39:20.180 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.180 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.180 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:20.180 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586660+0000", 2024-09-18T04:39:20.180 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1659999999999994e-06 2024-09-18T04:39:20.180 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.180 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.180 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:20.180 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586666+0000", 2024-09-18T04:39:20.180 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6900000000000003e-06 2024-09-18T04:39:20.180 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.180 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.181 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:20.181 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599381+0000", 2024-09-18T04:39:20.181 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012714328 2024-09-18T04:39:20.181 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.181 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.181 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:20.181 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599436+0000", 2024-09-18T04:39:20.181 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4802999999999997e-05 2024-09-18T04:39:20.181 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.182 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.182 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:20.182 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599629+0000", 2024-09-18T04:39:20.182 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000193921 2024-09-18T04:39:20.182 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.182 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.182 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:20.182 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.601174+0000", 2024-09-18T04:39:20.182 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001544857 2024-09-18T04:39:20.182 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.182 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.182 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:20.182 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.607004+0000", 2024-09-18T04:39:20.182 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058300820000000003 2024-09-18T04:39:20.183 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.183 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.183 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:20.183 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.607024+0000", 2024-09-18T04:39:20.183 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9888e-05 2024-09-18T04:39:20.183 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:20.183 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:20.183 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:20.183 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.607041+0000", 2024-09-18T04:39:20.183 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6450999999999999e-05 2024-09-18T04:39:20.183 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:20.183 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:20.183 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:20.183 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:20.184 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:20.184 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:20.184 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T04:39:20.186 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:39:20.186 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:20.177+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:20.286 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:20.281+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:20.299 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:20.299 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:20.300 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:39:20.300 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:20.309 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:39:20.337 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:39:20.357 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:39:20.378 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:39:20.387 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:20.381+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:20.401 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:39:20.420 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:20.420 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:20.420 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:39:20.420 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:39:20.420 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:20.423 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:39:20.430 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:39:20.446 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:39:20.448 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:39:20.449 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:39:20.449 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:39:20.449 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:39:20.449 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:39:20.449 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:39:20.487 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:20.481+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:20.533 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:20.533 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:20.543 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:39:20.589 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:20.581+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:20.591 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-18T04:39:20.646 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:20.647 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:20.656 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:39:20.698 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:20.693+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:20.791 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:20.785+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:20.807 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:20.807 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:20.817 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:39:20.891 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:20.885+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:20.922 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:20.922 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:20.932 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:39:20.992 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:20.985+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:39:21.040 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:21.041 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:21.051 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:39:21.094 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:21.085+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:21.194 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:21.189+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:21.206 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:21.207 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:21.217 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:39:21.295 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:21.289+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:21.323 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:21.323 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:21.333 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:39:21.396 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:21.393+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:21.483 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:21.483 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:21.492 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:39:21.497 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:21.493+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:21.592 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-09-18T04:39:21.598 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:21.589+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:39:21.612 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:21.612 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:21.621 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:39:21.699 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:21.697+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:21.775 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:21.775 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:21.785 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:39:21.830 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:21.829+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:21.897 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:21.897 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:21.900 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:21.897+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:21.907 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:39:22.002 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:21.993+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:39:22.019 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:22.019 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:22.029 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:39:22.102 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:22.097+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:39:22.138 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:22.138 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:22.148 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:39:22.204 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:22.200+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:39:22.265 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:22.265 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:22.275 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:39:22.305 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:22.300+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:39:22.385 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:22.385 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:22.398 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:39:22.405 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:22.397+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:22.461 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T04:39:22.505 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:22.506 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:22.506 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:22.501+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:22.515 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:39:22.585 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 2] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-18T04:39:22.585 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T04:39:22.585 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-09-18T04:39:22.585 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-18T04:39:22.585 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-18T04:39:22.586 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config set filestore_inject_stall 3 2024-09-18T04:39:22.606 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:22.604+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:22.613 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:39:22.633 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:39:22.634 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:22.634 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:22.643 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:39:22.652 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:39:22.673 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:39:22.687 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:22.687 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-18T04:39:22.687 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:22.695 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:39:22.707 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:22.701+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:22.716 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:39:22.718 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:39:22.718 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:39:22.718 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:39:22.718 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:39:22.718 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:39:22.718 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:39:22.750 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:22.750 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:22.750 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:39:22.750 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:39:22.751 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:22.760 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:39:22.808 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:22.804+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:22.908 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:22.904+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:22.913 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:22.913 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:39:22.914 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:39:22.914 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:39:22.914 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:22.914 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.5431.0:1 3.3 3:d80af3cb:::rbd_id.testrbdnbd17342:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-18T04:39:22.914 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:13.542818+0000", 2024-09-18T04:39:22.914 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 9.3687073729999994, 2024-09-18T04:39:22.914 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048493400000000002, 2024-09-18T04:39:22.914 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:22.914 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:39:22.914 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:22.914 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.5431", 2024-09-18T04:39:22.914 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3147136083", 2024-09-18T04:39:22.914 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 1 2024-09-18T04:39:22.914 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:22.915 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:22.915 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:22.915 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:22.915 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542818+0000", 2024-09-18T04:39:22.915 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:22.915 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:22.915 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:22.915 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:22.915 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542818+0000", 2024-09-18T04:39:22.915 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:22.915 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:22.915 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:22.915 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:22.915 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542816+0000", 2024-09-18T04:39:22.915 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:22.916 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:22.916 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:22.916 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:22.916 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542828+0000", 2024-09-18T04:39:22.916 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2699e-05 2024-09-18T04:39:22.916 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:22.916 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:22.916 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:22.916 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542852+0000", 2024-09-18T04:39:22.916 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3176e-05 2024-09-18T04:39:22.916 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:22.916 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:22.916 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:22.917 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542874+0000", 2024-09-18T04:39:22.917 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1967000000000002e-05 2024-09-18T04:39:22.917 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:22.917 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:22.917 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:22.917 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542984+0000", 2024-09-18T04:39:22.917 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011026099999999999 2024-09-18T04:39:22.917 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:22.917 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:22.917 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:22.917 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543161+0000", 2024-09-18T04:39:22.917 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017720299999999999 2024-09-18T04:39:22.917 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:22.918 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:22.918 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:22.918 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543303+0000", 2024-09-18T04:39:22.918 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014174700000000001 2024-09-18T04:39:22.918 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:22.918 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:22.918 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:22.918 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:22.918 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:22.918 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.5431.0:2 3.3 3:d80af3cb:::rbd_id.testrbdnbd17342:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-18T04:39:22.918 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:13.543564+0000", 2024-09-18T04:39:22.918 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 9.3679608870000006, 2024-09-18T04:39:22.918 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010399929999999999, 2024-09-18T04:39:22.918 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:22.918 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:39:22.919 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:22.919 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.5431", 2024-09-18T04:39:22.919 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3147136083", 2024-09-18T04:39:22.919 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 2 2024-09-18T04:39:22.919 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:22.919 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:22.919 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:22.919 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:22.919 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543564+0000", 2024-09-18T04:39:22.919 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:22.919 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:22.919 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:22.919 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:22.920 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543564+0000", 2024-09-18T04:39:22.920 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:22.920 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:22.920 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:22.920 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:22.920 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543563+0000", 2024-09-18T04:39:22.920 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:22.920 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:22.920 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:22.920 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:22.920 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543571+0000", 2024-09-18T04:39:22.920 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4270000000000008e-06 2024-09-18T04:39:22.920 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:22.920 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:22.921 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:22.921 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543591+0000", 2024-09-18T04:39:22.921 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.929e-05 2024-09-18T04:39:22.921 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:22.921 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:22.921 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:22.921 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543600+0000", 2024-09-18T04:39:22.921 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.1390000000000001e-06 2024-09-18T04:39:22.921 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:22.921 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:22.921 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:22.921 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543724+0000", 2024-09-18T04:39:22.921 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012458400000000001 2024-09-18T04:39:22.922 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:22.922 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:22.922 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:22.922 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543968+0000", 2024-09-18T04:39:22.922 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00024323199999999999 2024-09-18T04:39:22.922 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:22.922 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:22.922 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:22.922 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.544604+0000", 2024-09-18T04:39:22.922 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00063675799999999996 2024-09-18T04:39:22.922 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:22.922 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:22.922 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:22.922 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:22.923 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:22.923 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:22.924 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:39:23.009 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:23.004+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:23.032 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:23.032 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:23.032 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:39:23.032 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:23.042 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:39:23.111 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:23.108+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:23.147 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:23.147 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:23.147 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:39:23.148 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:39:23.148 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:23.157 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:39:23.212 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:23.208+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:23.266 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:23.266 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:39:23.266 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:39:23.266 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:39:23.266 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.267 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-18T04:39:23.267 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:39:23.267 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 108.789420581, 2024-09-18T04:39:23.267 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0095781189999999995, 2024-09-18T04:39:23.267 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:23.268 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:23.268 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:23.268 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:39:23.268 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:39:23.268 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 1 2024-09-18T04:39:23.269 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.269 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:23.269 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.269 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:23.269 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:39:23.270 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:23.270 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.270 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.270 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:23.271 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:39:23.271 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:23.271 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.271 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.271 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:23.272 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469862+0000", 2024-09-18T04:39:23.272 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:23.272 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.272 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.273 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:23.273 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469879+0000", 2024-09-18T04:39:23.273 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7431999999999998e-05 2024-09-18T04:39:23.273 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.273 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.274 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:23.274 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469896+0000", 2024-09-18T04:39:23.274 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6742999999999999e-05 2024-09-18T04:39:23.274 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.274 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.275 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:23.275 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469915+0000", 2024-09-18T04:39:23.275 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8991000000000002e-05 2024-09-18T04:39:23.275 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.275 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.276 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:23.276 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470397+0000", 2024-09-18T04:39:23.276 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048167799999999998 2024-09-18T04:39:23.276 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.277 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.277 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-09-18T04:39:23.277 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470412+0000", 2024-09-18T04:39:23.277 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5526e-05 2024-09-18T04:39:23.277 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.278 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.278 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:23.278 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473279+0000", 2024-09-18T04:39:23.278 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0028667469999999998 2024-09-18T04:39:23.278 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.279 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.279 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:23.279 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473441+0000", 2024-09-18T04:39:23.279 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000162082 2024-09-18T04:39:23.280 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.280 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.280 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:23.280 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473573+0000", 2024-09-18T04:39:23.280 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000132032 2024-09-18T04:39:23.281 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.281 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.281 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:23.281 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474826+0000", 2024-09-18T04:39:23.281 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001252717 2024-09-18T04:39:23.282 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.282 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.282 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:23.282 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479307+0000", 2024-09-18T04:39:23.282 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044813220000000003 2024-09-18T04:39:23.283 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.283 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.283 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:23.283 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479366+0000", 2024-09-18T04:39:23.283 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9024999999999998e-05 2024-09-18T04:39:23.284 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.284 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.284 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:23.284 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479443+0000", 2024-09-18T04:39:23.285 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6339000000000002e-05 2024-09-18T04:39:23.285 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:23.285 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:23.285 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:23.285 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.286 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.286 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-18T04:39:23.286 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:39:23.286 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 106.776412161, 2024-09-18T04:39:23.286 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082699560000000002, 2024-09-18T04:39:23.286 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:23.287 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:23.287 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:23.287 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:39:23.287 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:39:23.287 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 7 2024-09-18T04:39:23.287 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.287 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:23.288 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.288 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:23.288 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:39:23.288 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:23.288 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.289 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.289 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:23.289 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:39:23.289 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:23.289 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.289 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.290 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:23.290 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482870+0000", 2024-09-18T04:39:23.290 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999967 2024-09-18T04:39:23.290 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.290 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.290 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:23.291 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482889+0000", 2024-09-18T04:39:23.291 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.965e-05 2024-09-18T04:39:23.291 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.291 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.291 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:23.291 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482938+0000", 2024-09-18T04:39:23.292 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8548999999999999e-05 2024-09-18T04:39:23.292 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.292 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.292 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:23.292 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482963+0000", 2024-09-18T04:39:23.292 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5075999999999998e-05 2024-09-18T04:39:23.292 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.293 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.293 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:23.293 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483121+0000", 2024-09-18T04:39:23.293 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015819400000000001 2024-09-18T04:39:23.293 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.293 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.294 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op must wait for map", 2024-09-18T04:39:23.294 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483138+0000", 2024-09-18T04:39:23.294 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7068999999999999e-05 2024-09-18T04:39:23.294 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.294 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.294 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:23.295 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488666+0000", 2024-09-18T04:39:23.295 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0055282049999999996 2024-09-18T04:39:23.295 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.295 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.295 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:23.295 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488783+0000", 2024-09-18T04:39:23.296 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000116737 2024-09-18T04:39:23.296 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.296 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.296 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:23.297 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488991+0000", 2024-09-18T04:39:23.297 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020781400000000001 2024-09-18T04:39:23.297 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.297 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.297 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:23.297 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490554+0000", 2024-09-18T04:39:23.298 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0015628370000000001 2024-09-18T04:39:23.298 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.298 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.298 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:23.298 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491019+0000", 2024-09-18T04:39:23.298 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000465608 2024-09-18T04:39:23.299 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.299 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.299 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:23.299 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491087+0000", 2024-09-18T04:39:23.299 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7988e-05 2024-09-18T04:39:23.299 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.299 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.300 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:23.300 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491143+0000", 2024-09-18T04:39:23.300 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5538999999999998e-05 2024-09-18T04:39:23.300 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:23.300 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:23.300 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:23.301 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.301 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.301 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:34 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:23.301 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.261758+0000", 2024-09-18T04:39:23.301 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 40.997527009999999, 2024-09-18T04:39:23.301 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.042024191000000002, 2024-09-18T04:39:23.302 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:23.302 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:23.302 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:23.302 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:23.302 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:23.302 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 34 2024-09-18T04:39:23.302 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.303 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:23.303 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.303 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:23.303 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.261758+0000", 2024-09-18T04:39:23.303 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:23.303 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.303 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.304 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:23.304 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.261758+0000", 2024-09-18T04:39:23.304 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:23.304 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.304 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.304 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:23.305 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.261756+0000", 2024-09-18T04:39:23.305 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:23.305 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.305 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.305 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:23.305 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.270785+0000", 2024-09-18T04:39:23.305 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0090292340000000006 2024-09-18T04:39:23.306 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.306 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.306 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:23.306 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.270890+0000", 2024-09-18T04:39:23.306 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000104541 2024-09-18T04:39:23.306 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.307 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.307 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:23.307 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.270991+0000", 2024-09-18T04:39:23.307 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000100778 2024-09-18T04:39:23.307 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.307 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.307 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:23.308 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.271157+0000", 2024-09-18T04:39:23.308 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016653999999999999 2024-09-18T04:39:23.308 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.308 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.308 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:23.308 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.271591+0000", 2024-09-18T04:39:23.309 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00043409500000000001 2024-09-18T04:39:23.309 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.309 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.309 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:23.309 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272017+0000", 2024-09-18T04:39:23.309 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000425769 2024-09-18T04:39:23.310 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.310 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.310 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:23.310 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.294246+0000", 2024-09-18T04:39:23.310 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.022229237999999998 2024-09-18T04:39:23.310 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.311 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.311 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:23.311 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303571+0000", 2024-09-18T04:39:23.311 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0093250239999999995 2024-09-18T04:39:23.311 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.311 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.311 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:23.312 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303705+0000", 2024-09-18T04:39:23.312 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000133831 2024-09-18T04:39:23.312 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.312 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.312 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:23.312 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303782+0000", 2024-09-18T04:39:23.313 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7156000000000005e-05 2024-09-18T04:39:23.313 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:23.313 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:23.313 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:23.313 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.314 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.314 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:36 2.7 e51/32 2:fe1b02ad:::rbd_data.125fc09df6d1.0000000000000002:head v 51'1, mlcod=0'0)", 2024-09-18T04:39:23.314 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.273441+0000", 2024-09-18T04:39:23.314 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 40.985844028999999, 2024-09-18T04:39:23.315 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.032400768000000003, 2024-09-18T04:39:23.315 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:23.315 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:23.315 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:23.315 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.316 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:23.316 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273441+0000", 2024-09-18T04:39:23.316 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:23.316 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.317 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.317 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:23.317 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273441+0000", 2024-09-18T04:39:23.317 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:23.318 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.318 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.318 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:23.318 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273440+0000", 2024-09-18T04:39:23.318 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:23.319 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.319 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.319 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:23.319 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278723+0000", 2024-09-18T04:39:23.320 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0052829909999999999 2024-09-18T04:39:23.320 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.320 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.320 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:23.321 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278752+0000", 2024-09-18T04:39:23.321 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9178000000000001e-05 2024-09-18T04:39:23.321 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.321 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.321 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:23.322 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278761+0000", 2024-09-18T04:39:23.322 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7900000000000005e-06 2024-09-18T04:39:23.322 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.322 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.323 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:23.323 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278864+0000", 2024-09-18T04:39:23.323 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000102699 2024-09-18T04:39:23.323 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.323 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.324 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:23.324 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278900+0000", 2024-09-18T04:39:23.324 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6001999999999999e-05 2024-09-18T04:39:23.324 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.325 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.325 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:23.325 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.305793+0000", 2024-09-18T04:39:23.325 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.026893344 2024-09-18T04:39:23.325 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.325 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.325 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:23.326 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.305842+0000", 2024-09-18T04:39:23.326 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8720000000000001e-05 2024-09-18T04:39:23.326 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:23.326 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:23.326 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:23.326 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.326 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.326 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:39 2.7 e51/32 2:e9c909c5:::rbd_data.125fc09df6d1.0000000000000005:head v 51'2, mlcod=0'0)", 2024-09-18T04:39:23.326 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.283175+0000", 2024-09-18T04:39:23.326 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 40.976110339999998, 2024-09-18T04:39:23.326 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.022864102000000001, 2024-09-18T04:39:23.326 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:23.326 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:23.327 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:23.327 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.327 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:23.327 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283175+0000", 2024-09-18T04:39:23.327 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:23.327 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.327 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.327 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:23.327 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283175+0000", 2024-09-18T04:39:23.327 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:23.327 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.327 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.327 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:23.327 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283173+0000", 2024-09-18T04:39:23.327 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:23.328 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.328 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.328 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:23.328 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.285951+0000", 2024-09-18T04:39:23.328 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0027784540000000001 2024-09-18T04:39:23.328 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.328 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.328 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:23.328 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.285970+0000", 2024-09-18T04:39:23.328 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8855000000000002e-05 2024-09-18T04:39:23.328 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.328 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.328 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:23.329 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.285984+0000", 2024-09-18T04:39:23.329 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4252e-05 2024-09-18T04:39:23.329 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.329 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.329 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:23.329 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.286041+0000", 2024-09-18T04:39:23.329 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6431999999999998e-05 2024-09-18T04:39:23.329 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.329 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.329 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:23.329 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.286086+0000", 2024-09-18T04:39:23.329 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5423000000000002e-05 2024-09-18T04:39:23.329 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.329 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.330 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:23.330 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.305998+0000", 2024-09-18T04:39:23.330 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019911697999999999 2024-09-18T04:39:23.330 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.330 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.330 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:23.330 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306039+0000", 2024-09-18T04:39:23.330 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0973999999999998e-05 2024-09-18T04:39:23.330 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:23.330 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:23.330 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:23.330 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.330 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.331 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:50 2.3 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:23.331 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.312563+0000", 2024-09-18T04:39:23.331 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 40.946722313000002, 2024-09-18T04:39:23.331 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.032636001999999997, 2024-09-18T04:39:23.331 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:23.331 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:23.331 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:23.331 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:23.331 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:23.331 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 50 2024-09-18T04:39:23.331 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.331 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:23.331 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.332 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:23.332 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.312563+0000", 2024-09-18T04:39:23.332 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:23.332 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.332 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.332 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:23.332 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.312563+0000", 2024-09-18T04:39:23.332 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:23.332 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.332 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.332 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:23.332 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.312561+0000", 2024-09-18T04:39:23.333 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:23.333 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.333 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.333 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:23.333 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321709+0000", 2024-09-18T04:39:23.333 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0091486650000000003 2024-09-18T04:39:23.333 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.333 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.333 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:23.333 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321739+0000", 2024-09-18T04:39:23.333 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9666e-05 2024-09-18T04:39:23.333 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.333 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.333 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:23.334 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321751+0000", 2024-09-18T04:39:23.334 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2086000000000001e-05 2024-09-18T04:39:23.334 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.334 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.334 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:23.334 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321833+0000", 2024-09-18T04:39:23.334 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1842999999999996e-05 2024-09-18T04:39:23.334 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.334 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.334 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:23.334 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321981+0000", 2024-09-18T04:39:23.334 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000147441 2024-09-18T04:39:23.335 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.335 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.335 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:23.335 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.322109+0000", 2024-09-18T04:39:23.335 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012851200000000001 2024-09-18T04:39:23.335 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.335 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.335 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:23.335 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.332829+0000", 2024-09-18T04:39:23.335 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010720225999999999 2024-09-18T04:39:23.335 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.335 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.335 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:23.336 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.345039+0000", 2024-09-18T04:39:23.336 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012209831000000001 2024-09-18T04:39:23.336 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.336 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.336 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:23.336 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.345120+0000", 2024-09-18T04:39:23.336 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1145999999999999e-05 2024-09-18T04:39:23.336 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.336 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.336 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:23.336 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.345199+0000", 2024-09-18T04:39:23.336 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8546000000000003e-05 2024-09-18T04:39:23.336 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:23.336 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:23.337 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:23.337 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.337 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.337 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:30 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:23.337 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.516122+0000", 2024-09-18T04:39:23.337 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 40.743163150999997, 2024-09-18T04:39:23.337 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0091865030000000004, 2024-09-18T04:39:23.337 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:23.337 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:23.337 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:23.337 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:23.337 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:23.337 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 30 2024-09-18T04:39:23.338 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.338 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:23.338 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.338 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:23.338 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516122+0000", 2024-09-18T04:39:23.338 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:23.338 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.338 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.338 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:23.338 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516122+0000", 2024-09-18T04:39:23.338 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:23.338 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.338 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.338 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:23.339 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516120+0000", 2024-09-18T04:39:23.339 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:23.339 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.339 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.339 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:23.339 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516129+0000", 2024-09-18T04:39:23.339 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.85e-06 2024-09-18T04:39:23.339 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.339 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.339 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:23.339 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516155+0000", 2024-09-18T04:39:23.339 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5882000000000001e-05 2024-09-18T04:39:23.339 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.340 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.340 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:23.340 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516175+0000", 2024-09-18T04:39:23.340 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.022e-05 2024-09-18T04:39:23.340 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.340 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.340 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:23.340 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519780+0000", 2024-09-18T04:39:23.340 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003604187 2024-09-18T04:39:23.340 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.340 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.340 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:23.340 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519964+0000", 2024-09-18T04:39:23.340 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00018391999999999999 2024-09-18T04:39:23.340 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.341 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.341 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:23.341 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.520794+0000", 2024-09-18T04:39:23.341 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00083078100000000005 2024-09-18T04:39:23.341 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.341 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.341 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:23.341 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.524129+0000", 2024-09-18T04:39:23.341 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033342419999999998 2024-09-18T04:39:23.341 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.341 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.341 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:23.341 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525239+0000", 2024-09-18T04:39:23.341 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001110778 2024-09-18T04:39:23.342 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.342 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.342 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:23.342 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525277+0000", 2024-09-18T04:39:23.342 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7596999999999999e-05 2024-09-18T04:39:23.342 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.342 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.342 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:23.342 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525308+0000", 2024-09-18T04:39:23.342 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.1535999999999998e-05 2024-09-18T04:39:23.342 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:23.342 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:23.342 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:23.342 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.343 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.343 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:32 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:23.343 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.519217+0000", 2024-09-18T04:39:23.343 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 40.740067830999998, 2024-09-18T04:39:23.343 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082266370000000002, 2024-09-18T04:39:23.343 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:23.343 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:23.343 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:23.343 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:23.343 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:23.343 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 32 2024-09-18T04:39:23.343 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.343 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:23.343 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.344 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:23.344 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519217+0000", 2024-09-18T04:39:23.344 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:23.344 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.344 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.344 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:23.344 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519217+0000", 2024-09-18T04:39:23.344 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:23.344 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.344 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.344 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:23.344 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519215+0000", 2024-09-18T04:39:23.345 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:23.345 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.345 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.345 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:23.345 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519229+0000", 2024-09-18T04:39:23.345 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3329999999999999e-05 2024-09-18T04:39:23.345 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.345 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.345 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:23.345 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519268+0000", 2024-09-18T04:39:23.345 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9533000000000002e-05 2024-09-18T04:39:23.345 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.345 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.345 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:23.346 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519285+0000", 2024-09-18T04:39:23.346 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6446000000000001e-05 2024-09-18T04:39:23.346 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.346 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.346 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:23.346 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.522358+0000", 2024-09-18T04:39:23.346 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030734310000000002 2024-09-18T04:39:23.346 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.346 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.346 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:23.346 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.522551+0000", 2024-09-18T04:39:23.346 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019317600000000001 2024-09-18T04:39:23.346 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.346 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.347 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:23.347 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.523262+0000", 2024-09-18T04:39:23.347 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00071069999999999998 2024-09-18T04:39:23.347 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.347 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.347 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:23.347 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525318+0000", 2024-09-18T04:39:23.347 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0020561469999999999 2024-09-18T04:39:23.347 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.347 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.347 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:23.347 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.527376+0000", 2024-09-18T04:39:23.347 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0020584969999999998 2024-09-18T04:39:23.347 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.348 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.348 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:23.348 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.527416+0000", 2024-09-18T04:39:23.348 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9054000000000003e-05 2024-09-18T04:39:23.348 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.348 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.348 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:23.348 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.527444+0000", 2024-09-18T04:39:23.348 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8347000000000002e-05 2024-09-18T04:39:23.348 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:23.348 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:23.348 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:23.348 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.349 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.349 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:42 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:23.349 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.551160+0000", 2024-09-18T04:39:23.349 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 40.708125439, 2024-09-18T04:39:23.349 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010692757000000001, 2024-09-18T04:39:23.349 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:23.349 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:23.349 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:23.349 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:23.349 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:23.349 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 42 2024-09-18T04:39:23.349 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.349 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:23.349 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.350 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:23.350 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551160+0000", 2024-09-18T04:39:23.350 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:23.350 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.350 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.350 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:23.350 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551160+0000", 2024-09-18T04:39:23.350 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:23.350 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.350 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.350 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:23.350 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551159+0000", 2024-09-18T04:39:23.350 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:23.350 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.351 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.351 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:23.351 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551166+0000", 2024-09-18T04:39:23.351 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1029999999999996e-06 2024-09-18T04:39:23.351 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.351 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.351 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:23.351 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551177+0000", 2024-09-18T04:39:23.351 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1384e-05 2024-09-18T04:39:23.351 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.351 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.351 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:23.351 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551185+0000", 2024-09-18T04:39:23.352 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1990000000000006e-06 2024-09-18T04:39:23.352 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.352 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.352 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:23.352 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552021+0000", 2024-09-18T04:39:23.352 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00083574400000000003 2024-09-18T04:39:23.352 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.352 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.352 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:23.352 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552339+0000", 2024-09-18T04:39:23.352 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031771399999999997 2024-09-18T04:39:23.352 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.352 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.353 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:23.353 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.553140+0000", 2024-09-18T04:39:23.353 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00080088100000000003 2024-09-18T04:39:23.353 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.353 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.353 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:23.353 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.554585+0000", 2024-09-18T04:39:23.353 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001445801 2024-09-18T04:39:23.353 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.353 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.353 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:23.353 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.561711+0000", 2024-09-18T04:39:23.353 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071260109999999998 2024-09-18T04:39:23.354 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.354 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.354 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:23.354 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.561790+0000", 2024-09-18T04:39:23.354 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8967000000000005e-05 2024-09-18T04:39:23.354 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.354 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.354 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:23.354 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.561852+0000", 2024-09-18T04:39:23.354 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2045999999999995e-05 2024-09-18T04:39:23.354 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:23.354 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:23.354 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:23.354 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.354 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.355 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:58 2.7 e51/32 2:fc7c6cd3:ns::rbd_data.133e813ab5bc.000000000000000e:head v 51'7, mlcod=51'5)", 2024-09-18T04:39:23.355 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.605333+0000", 2024-09-18T04:39:23.355 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 40.653952234000002, 2024-09-18T04:39:23.355 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020085902999999999, 2024-09-18T04:39:23.355 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:23.355 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:23.355 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:23.355 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.355 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:23.355 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605333+0000", 2024-09-18T04:39:23.355 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:23.355 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.355 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.355 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:23.356 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605333+0000", 2024-09-18T04:39:23.356 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:23.356 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.356 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.356 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:23.356 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605332+0000", 2024-09-18T04:39:23.356 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:23.356 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.356 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.356 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:23.356 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608667+0000", 2024-09-18T04:39:23.356 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033350300000000001 2024-09-18T04:39:23.356 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.357 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.357 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:23.357 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608706+0000", 2024-09-18T04:39:23.357 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8951e-05 2024-09-18T04:39:23.357 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.357 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.357 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:23.357 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608725+0000", 2024-09-18T04:39:23.357 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9675999999999999e-05 2024-09-18T04:39:23.357 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.357 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.357 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:23.357 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608798+0000", 2024-09-18T04:39:23.357 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3135999999999999e-05 2024-09-18T04:39:23.357 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.358 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.358 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:23.358 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608824+0000", 2024-09-18T04:39:23.358 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5284000000000001e-05 2024-09-18T04:39:23.358 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.358 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.358 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:23.358 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625370+0000", 2024-09-18T04:39:23.358 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016545975000000001 2024-09-18T04:39:23.358 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.358 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.358 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:23.359 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625419+0000", 2024-09-18T04:39:23.359 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9076999999999998e-05 2024-09-18T04:39:23.359 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:23.359 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:23.359 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:23.359 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.359 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.359 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:611 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:23.359 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.069561+0000", 2024-09-18T04:39:23.359 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 32.18972368, 2024-09-18T04:39:23.359 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0072330299999999997, 2024-09-18T04:39:23.359 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:23.359 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:23.359 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:23.360 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:39:23.360 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:23.360 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 611 2024-09-18T04:39:23.360 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.360 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:23.360 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.360 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:23.360 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.069561+0000", 2024-09-18T04:39:23.360 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:23.360 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.360 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.360 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:23.360 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.069561+0000", 2024-09-18T04:39:23.361 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:23.361 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.361 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.361 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:23.361 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.069559+0000", 2024-09-18T04:39:23.361 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:23.361 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.361 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.361 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:23.361 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070801+0000", 2024-09-18T04:39:23.361 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0012423359999999999 2024-09-18T04:39:23.361 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.361 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.361 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:23.362 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070828+0000", 2024-09-18T04:39:23.362 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6387e-05 2024-09-18T04:39:23.362 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.362 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.362 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:23.362 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070840+0000", 2024-09-18T04:39:23.362 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2689e-05 2024-09-18T04:39:23.362 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.362 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.362 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:23.362 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070990+0000", 2024-09-18T04:39:23.362 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014946800000000001 2024-09-18T04:39:23.362 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.362 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.363 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:23.363 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.071128+0000", 2024-09-18T04:39:23.363 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013796699999999999 2024-09-18T04:39:23.363 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.363 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.363 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:23.363 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.071318+0000", 2024-09-18T04:39:23.363 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000190106 2024-09-18T04:39:23.363 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.363 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.363 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:23.363 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.075711+0000", 2024-09-18T04:39:23.363 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.004393002 2024-09-18T04:39:23.363 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.364 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.364 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:23.364 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.076665+0000", 2024-09-18T04:39:23.364 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00095390399999999999 2024-09-18T04:39:23.364 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.364 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.364 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:23.364 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.076742+0000", 2024-09-18T04:39:23.364 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7112000000000003e-05 2024-09-18T04:39:23.364 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.364 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.364 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:23.364 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.076794+0000", 2024-09-18T04:39:23.364 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2475999999999997e-05 2024-09-18T04:39:23.365 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:23.365 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:23.365 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:23.365 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.365 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.365 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:687 2.3 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:23.365 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.487994+0000", 2024-09-18T04:39:23.365 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 31.771290697000001, 2024-09-18T04:39:23.365 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0074167690000000001, 2024-09-18T04:39:23.365 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:23.365 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:23.365 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:23.365 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:39:23.366 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:23.366 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 687 2024-09-18T04:39:23.366 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.366 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:23.366 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.366 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:23.366 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.487994+0000", 2024-09-18T04:39:23.366 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:23.366 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.366 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.366 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:23.366 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.487994+0000", 2024-09-18T04:39:23.366 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:23.366 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.367 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.367 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:23.367 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.487993+0000", 2024-09-18T04:39:23.367 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:23.367 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.367 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.367 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:23.367 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.488979+0000", 2024-09-18T04:39:23.367 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00098620700000000001 2024-09-18T04:39:23.367 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.367 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.367 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:23.367 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489002+0000", 2024-09-18T04:39:23.367 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2500000000000001e-05 2024-09-18T04:39:23.367 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.368 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.368 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:23.368 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489020+0000", 2024-09-18T04:39:23.368 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7691000000000001e-05 2024-09-18T04:39:23.368 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.368 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.368 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:23.368 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489091+0000", 2024-09-18T04:39:23.368 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1307e-05 2024-09-18T04:39:23.368 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.368 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.368 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:23.368 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489219+0000", 2024-09-18T04:39:23.368 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012766000000000001 2024-09-18T04:39:23.369 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.369 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.369 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:23.369 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489387+0000", 2024-09-18T04:39:23.369 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016855400000000001 2024-09-18T04:39:23.369 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.369 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.369 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:23.369 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495243+0000", 2024-09-18T04:39:23.369 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058563189999999996 2024-09-18T04:39:23.369 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.369 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.369 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:23.370 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495266+0000", 2024-09-18T04:39:23.370 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2810999999999999e-05 2024-09-18T04:39:23.370 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.370 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.370 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:23.370 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495332+0000", 2024-09-18T04:39:23.370 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5469999999999995e-05 2024-09-18T04:39:23.370 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.370 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.370 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:23.370 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495411+0000", 2024-09-18T04:39:23.370 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9395999999999997e-05 2024-09-18T04:39:23.370 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:23.370 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:23.371 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:23.371 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.371 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.371 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:689 2.3 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:23.371 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.500070+0000", 2024-09-18T04:39:23.371 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 31.759214880999998, 2024-09-18T04:39:23.371 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071096650000000003, 2024-09-18T04:39:23.371 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:23.371 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:23.371 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:23.371 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:39:23.371 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:23.371 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 689 2024-09-18T04:39:23.371 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.371 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:23.372 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.372 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:23.372 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.500070+0000", 2024-09-18T04:39:23.372 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:23.372 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.372 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.372 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:23.372 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.500070+0000", 2024-09-18T04:39:23.372 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:23.372 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.372 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.372 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:23.372 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.500069+0000", 2024-09-18T04:39:23.373 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:23.373 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.373 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.373 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:23.373 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501082+0000", 2024-09-18T04:39:23.373 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010136570000000001 2024-09-18T04:39:23.373 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.373 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.373 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:23.373 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501104+0000", 2024-09-18T04:39:23.373 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1883e-05 2024-09-18T04:39:23.373 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.373 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.373 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:23.374 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501115+0000", 2024-09-18T04:39:23.374 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0414000000000001e-05 2024-09-18T04:39:23.374 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.374 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.374 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:23.374 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501178+0000", 2024-09-18T04:39:23.374 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3128999999999998e-05 2024-09-18T04:39:23.374 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.374 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.374 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:23.374 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501257+0000", 2024-09-18T04:39:23.374 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9592999999999995e-05 2024-09-18T04:39:23.374 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.374 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.375 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:23.375 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501366+0000", 2024-09-18T04:39:23.375 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000108643 2024-09-18T04:39:23.375 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.375 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.375 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:23.375 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.504041+0000", 2024-09-18T04:39:23.375 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0026750889999999999 2024-09-18T04:39:23.375 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.375 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.375 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:23.375 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.507097+0000", 2024-09-18T04:39:23.375 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030556070000000001 2024-09-18T04:39:23.375 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.376 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.376 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:23.376 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.507140+0000", 2024-09-18T04:39:23.376 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3745000000000002e-05 2024-09-18T04:39:23.376 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.376 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.376 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:23.376 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.507180+0000", 2024-09-18T04:39:23.376 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9475999999999999e-05 2024-09-18T04:39:23.376 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:23.376 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:23.376 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:23.376 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.377 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.377 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4972.0:13 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:23.377 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.948263+0000", 2024-09-18T04:39:23.377 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 31.311022090000002, 2024-09-18T04:39:23.377 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0091358970000000005, 2024-09-18T04:39:23.377 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:23.377 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:39:23.377 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:23.377 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4972", 2024-09-18T04:39:23.377 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2280251472", 2024-09-18T04:39:23.377 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 13 2024-09-18T04:39:23.377 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.377 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:23.377 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.378 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:23.378 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948263+0000", 2024-09-18T04:39:23.378 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:23.378 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.378 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.378 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:23.378 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948263+0000", 2024-09-18T04:39:23.378 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:23.378 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.378 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.378 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:23.378 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948261+0000", 2024-09-18T04:39:23.378 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:23.378 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.379 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.379 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:23.379 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948273+0000", 2024-09-18T04:39:23.379 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1576999999999999e-05 2024-09-18T04:39:23.379 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.379 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.379 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:23.379 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948305+0000", 2024-09-18T04:39:23.379 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2209000000000003e-05 2024-09-18T04:39:23.379 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.379 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.379 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:23.379 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948325+0000", 2024-09-18T04:39:23.380 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.022e-05 2024-09-18T04:39:23.380 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.380 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.380 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:23.380 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948455+0000", 2024-09-18T04:39:23.380 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000129624 2024-09-18T04:39:23.380 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.380 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.380 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:23.380 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948598+0000", 2024-09-18T04:39:23.380 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014376699999999999 2024-09-18T04:39:23.380 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.380 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.380 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:23.381 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.957399+0000", 2024-09-18T04:39:23.381 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0088004299999999997 2024-09-18T04:39:23.381 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:23.381 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:23.381 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:23.381 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.381 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.381 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:101 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:23.381 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.576367+0000", 2024-09-18T04:39:23.381 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 26.682918382, 2024-09-18T04:39:23.381 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0078671020000000008, 2024-09-18T04:39:23.381 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:23.381 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:23.381 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:23.382 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:23.382 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:23.382 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 101 2024-09-18T04:39:23.382 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.382 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:23.382 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.382 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:23.382 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576367+0000", 2024-09-18T04:39:23.382 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:23.382 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.382 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.382 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:23.382 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576367+0000", 2024-09-18T04:39:23.382 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:23.383 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.383 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.383 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:23.383 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576364+0000", 2024-09-18T04:39:23.383 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999971 2024-09-18T04:39:23.383 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.383 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.383 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:23.383 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576378+0000", 2024-09-18T04:39:23.383 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3879999999999999e-05 2024-09-18T04:39:23.383 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.383 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.383 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:23.383 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576410+0000", 2024-09-18T04:39:23.384 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2286000000000002e-05 2024-09-18T04:39:23.384 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.384 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.384 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:23.384 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576425+0000", 2024-09-18T04:39:23.384 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5290000000000001e-05 2024-09-18T04:39:23.384 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.384 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.384 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:23.384 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576551+0000", 2024-09-18T04:39:23.384 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012555199999999999 2024-09-18T04:39:23.384 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.384 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.384 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:23.385 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576719+0000", 2024-09-18T04:39:23.385 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016787400000000001 2024-09-18T04:39:23.385 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.385 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.385 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:23.385 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576951+0000", 2024-09-18T04:39:23.385 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00023252400000000001 2024-09-18T04:39:23.385 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.385 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.385 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:23.385 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.581459+0000", 2024-09-18T04:39:23.385 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045074410000000001 2024-09-18T04:39:23.385 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.386 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.386 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:23.386 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584076+0000", 2024-09-18T04:39:23.386 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0026173860000000002 2024-09-18T04:39:23.386 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.386 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.386 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:23.386 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584160+0000", 2024-09-18T04:39:23.386 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4191999999999996e-05 2024-09-18T04:39:23.386 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.386 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.386 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:23.386 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584234+0000", 2024-09-18T04:39:23.387 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.339e-05 2024-09-18T04:39:23.387 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:23.387 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:23.387 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:23.387 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.387 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.387 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:44 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:23.387 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.550448+0000", 2024-09-18T04:39:23.387 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 22.708837193000001, 2024-09-18T04:39:23.387 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082117590000000008, 2024-09-18T04:39:23.387 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:23.387 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:23.387 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:23.387 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:39:23.388 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:23.388 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 44 2024-09-18T04:39:23.388 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.388 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:23.388 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.388 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:23.388 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.550448+0000", 2024-09-18T04:39:23.388 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:23.388 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.388 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.388 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:23.388 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.550448+0000", 2024-09-18T04:39:23.388 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:23.388 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.389 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.389 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:23.389 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.550445+0000", 2024-09-18T04:39:23.389 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999971 2024-09-18T04:39:23.389 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.389 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.389 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:23.389 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.551904+0000", 2024-09-18T04:39:23.389 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001458322 2024-09-18T04:39:23.389 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.389 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.389 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:23.389 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.551938+0000", 2024-09-18T04:39:23.389 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4359000000000001e-05 2024-09-18T04:39:23.390 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.390 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.390 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:23.390 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.551959+0000", 2024-09-18T04:39:23.390 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1403000000000002e-05 2024-09-18T04:39:23.390 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.390 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.390 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:23.390 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.552049+0000", 2024-09-18T04:39:23.390 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9970999999999996e-05 2024-09-18T04:39:23.390 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.390 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.390 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:23.390 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.552190+0000", 2024-09-18T04:39:23.391 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000140271 2024-09-18T04:39:23.391 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.391 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.391 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:23.391 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.552385+0000", 2024-09-18T04:39:23.391 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019498700000000001 2024-09-18T04:39:23.391 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.391 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.391 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:23.391 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.556308+0000", 2024-09-18T04:39:23.391 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00392333 2024-09-18T04:39:23.391 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.391 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.391 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:23.392 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.558479+0000", 2024-09-18T04:39:23.392 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0021715889999999998 2024-09-18T04:39:23.392 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.392 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.392 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:23.392 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.558616+0000", 2024-09-18T04:39:23.392 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013613399999999999 2024-09-18T04:39:23.392 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.392 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.392 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:23.392 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.558660+0000", 2024-09-18T04:39:23.392 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4053e-05 2024-09-18T04:39:23.392 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:23.393 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:23.393 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:23.393 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.393 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.393 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:46 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:23.393 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.564089+0000", 2024-09-18T04:39:23.393 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 22.695196064000001, 2024-09-18T04:39:23.393 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0069437090000000002, 2024-09-18T04:39:23.393 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:23.393 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:23.393 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:23.393 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:39:23.394 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:23.394 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 46 2024-09-18T04:39:23.394 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.394 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:23.394 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.394 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:23.394 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.564089+0000", 2024-09-18T04:39:23.394 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:23.394 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.394 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.394 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:23.394 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.564089+0000", 2024-09-18T04:39:23.394 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:23.394 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.395 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.395 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:23.395 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.564087+0000", 2024-09-18T04:39:23.395 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:23.395 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.395 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.395 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:23.395 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565126+0000", 2024-09-18T04:39:23.395 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001038902 2024-09-18T04:39:23.395 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.395 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.395 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:23.395 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565148+0000", 2024-09-18T04:39:23.396 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1911000000000001e-05 2024-09-18T04:39:23.396 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.396 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.396 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:23.396 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565157+0000", 2024-09-18T04:39:23.396 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6220000000000004e-06 2024-09-18T04:39:23.396 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.396 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.396 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:23.396 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565261+0000", 2024-09-18T04:39:23.396 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000103063 2024-09-18T04:39:23.396 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.396 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.397 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:23.397 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565394+0000", 2024-09-18T04:39:23.397 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000133032 2024-09-18T04:39:23.397 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.397 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.397 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:23.397 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565564+0000", 2024-09-18T04:39:23.397 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017068000000000001 2024-09-18T04:39:23.397 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.397 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.397 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:23.397 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.570556+0000", 2024-09-18T04:39:23.397 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0049920620000000002 2024-09-18T04:39:23.397 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.398 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.398 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:23.398 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.570885+0000", 2024-09-18T04:39:23.398 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032827900000000001 2024-09-18T04:39:23.398 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.398 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.398 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:23.398 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.570959+0000", 2024-09-18T04:39:23.398 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3955999999999995e-05 2024-09-18T04:39:23.398 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.398 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.398 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:23.398 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.571033+0000", 2024-09-18T04:39:23.398 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4191999999999997e-05 2024-09-18T04:39:23.398 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:23.399 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:23.399 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:23.399 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.399 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.399 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:100 2.4 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:23.399 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.856389+0000", 2024-09-18T04:39:23.399 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 22.402895952000002, 2024-09-18T04:39:23.399 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0081480800000000003, 2024-09-18T04:39:23.399 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:23.399 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:23.399 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:23.399 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:39:23.399 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:23.400 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 100 2024-09-18T04:39:23.400 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.400 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:23.400 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.400 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:23.400 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.856389+0000", 2024-09-18T04:39:23.400 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:23.400 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.400 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.400 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:23.400 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.856389+0000", 2024-09-18T04:39:23.400 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:23.400 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.400 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.401 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:23.401 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.856388+0000", 2024-09-18T04:39:23.401 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:23.401 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.401 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.401 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:23.401 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857295+0000", 2024-09-18T04:39:23.401 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00090725599999999995 2024-09-18T04:39:23.401 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.401 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.401 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:23.401 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857311+0000", 2024-09-18T04:39:23.401 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6092000000000001e-05 2024-09-18T04:39:23.401 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.402 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.402 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:23.402 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857319+0000", 2024-09-18T04:39:23.402 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9200000000000004e-06 2024-09-18T04:39:23.402 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.402 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.402 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:23.402 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857388+0000", 2024-09-18T04:39:23.402 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8591999999999997e-05 2024-09-18T04:39:23.402 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.402 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.402 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:23.402 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857535+0000", 2024-09-18T04:39:23.402 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014684299999999999 2024-09-18T04:39:23.403 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.403 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.403 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:23.403 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857747+0000", 2024-09-18T04:39:23.403 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000212361 2024-09-18T04:39:23.403 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.403 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.403 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:23.403 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.860831+0000", 2024-09-18T04:39:23.403 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030841789999999998 2024-09-18T04:39:23.403 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.403 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.403 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:23.403 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.864361+0000", 2024-09-18T04:39:23.404 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0035298780000000002 2024-09-18T04:39:23.404 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.404 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.404 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:23.404 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.864456+0000", 2024-09-18T04:39:23.404 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4557999999999998e-05 2024-09-18T04:39:23.404 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.404 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.404 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:23.404 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.864537+0000", 2024-09-18T04:39:23.404 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1577999999999997e-05 2024-09-18T04:39:23.404 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:23.404 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:23.404 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:23.405 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.405 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.405 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:102 2.4 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:23.405 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.869773+0000", 2024-09-18T04:39:23.405 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 22.389512294999999, 2024-09-18T04:39:23.405 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0084430359999999992, 2024-09-18T04:39:23.405 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:23.405 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:23.405 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:23.405 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:39:23.405 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:23.405 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 102 2024-09-18T04:39:23.405 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.405 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:23.406 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.406 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:23.406 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.869773+0000", 2024-09-18T04:39:23.406 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:23.406 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.406 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.406 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:23.406 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.869773+0000", 2024-09-18T04:39:23.406 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:23.406 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.406 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.406 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:23.406 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.869771+0000", 2024-09-18T04:39:23.407 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:23.407 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.407 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.407 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:23.407 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870629+0000", 2024-09-18T04:39:23.407 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00085760999999999997 2024-09-18T04:39:23.407 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.407 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.407 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:23.407 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870647+0000", 2024-09-18T04:39:23.407 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8363999999999999e-05 2024-09-18T04:39:23.407 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.407 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.408 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:23.408 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870655+0000", 2024-09-18T04:39:23.408 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0099999999999995e-06 2024-09-18T04:39:23.408 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.408 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.408 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:23.408 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870803+0000", 2024-09-18T04:39:23.408 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014798 2024-09-18T04:39:23.408 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.408 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.408 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:23.408 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870951+0000", 2024-09-18T04:39:23.408 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014715999999999999 2024-09-18T04:39:23.409 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.409 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.409 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:23.409 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.871145+0000", 2024-09-18T04:39:23.409 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000194253 2024-09-18T04:39:23.409 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.409 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.409 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:23.409 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.877033+0000", 2024-09-18T04:39:23.409 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058883099999999999 2024-09-18T04:39:23.409 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.409 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.409 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:23.410 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.878078+0000", 2024-09-18T04:39:23.410 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001044747 2024-09-18T04:39:23.410 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.410 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.410 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:23.410 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.878160+0000", 2024-09-18T04:39:23.410 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1904999999999996e-05 2024-09-18T04:39:23.410 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.410 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.410 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:23.410 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.878216+0000", 2024-09-18T04:39:23.410 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6082000000000003e-05 2024-09-18T04:39:23.410 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:23.411 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:23.411 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:23.411 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.411 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.411 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:145 2.4 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:23.411 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:01.096935+0000", 2024-09-18T04:39:23.411 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 22.162350515, 2024-09-18T04:39:23.411 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0069937869999999996, 2024-09-18T04:39:23.411 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:23.411 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:23.411 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:23.411 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:39:23.411 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:23.411 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 145 2024-09-18T04:39:23.412 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.412 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:23.412 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.412 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:23.412 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.096935+0000", 2024-09-18T04:39:23.412 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:23.412 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.412 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.412 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:23.412 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.096935+0000", 2024-09-18T04:39:23.412 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:23.413 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.413 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.413 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:23.413 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.096932+0000", 2024-09-18T04:39:23.413 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:23.413 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.413 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.413 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:23.413 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097815+0000", 2024-09-18T04:39:23.413 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00088276000000000001 2024-09-18T04:39:23.413 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.413 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.413 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:23.414 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097842+0000", 2024-09-18T04:39:23.414 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6806999999999999e-05 2024-09-18T04:39:23.414 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.414 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.414 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:23.414 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097852+0000", 2024-09-18T04:39:23.414 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0548e-05 2024-09-18T04:39:23.414 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.414 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.414 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:23.414 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097931+0000", 2024-09-18T04:39:23.414 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8417000000000005e-05 2024-09-18T04:39:23.414 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.414 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.415 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:23.415 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.098047+0000", 2024-09-18T04:39:23.415 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011611 2024-09-18T04:39:23.415 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.415 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.415 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:23.415 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.098172+0000", 2024-09-18T04:39:23.415 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000125482 2024-09-18T04:39:23.415 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.415 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.415 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:23.415 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.100888+0000", 2024-09-18T04:39:23.415 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0027154430000000001 2024-09-18T04:39:23.416 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.416 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.416 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:23.416 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.103814+0000", 2024-09-18T04:39:23.416 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0029259619999999998 2024-09-18T04:39:23.416 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.416 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.416 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:23.416 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.103891+0000", 2024-09-18T04:39:23.416 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7619000000000003e-05 2024-09-18T04:39:23.416 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:23.416 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:23.416 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:23.416 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.103928+0000", 2024-09-18T04:39:23.417 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6900999999999998e-05 2024-09-18T04:39:23.417 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:23.417 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:23.417 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:23.417 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:23.417 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:23.417 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:23.418 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:39:23.419 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:23.416+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:23.519 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:23.516+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:23.535 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:23.536 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:23.536 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:39:23.536 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:23.546 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:39:23.605 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-18T04:39:23.616 INFO:tasks.workunit.client.0.smithi115.stderr: Creating snap: 10% complete... Creating snap: 20% complete... Creating snap: 100% complete...done. 2024-09-18T04:39:23.628 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo dd if=/tmp/tmp.WJMNJxIdAN/data of=/dev/nbd0 bs=1M count=1 oflag=direct 2024-09-18T04:39:23.628 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:39:23.628 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:39:23.630 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:39:23.631 INFO:tasks.workunit.client.0.smithi115.stderr:++ which dd 2024-09-18T04:39:23.633 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/bin/dd 2024-09-18T04:39:23.633 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:39:23.633 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.WJMNJxIdAN/data of=/dev/nbd0 bs=1M count=1 oflag=direct 2024-09-18T04:39:23.650 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:23.648+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:23.660 INFO:tasks.workunit.client.0.smithi115.stderr:1+0 records in 2024-09-18T04:39:23.661 INFO:tasks.workunit.client.0.smithi115.stderr:1+0 records out 2024-09-18T04:39:23.661 INFO:tasks.workunit.client.0.smithi115.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188816 s, 55.5 MB/s 2024-09-18T04:39:23.662 INFO:tasks.workunit.client.0.smithi115.stderr:+ unmap_device /dev/nbd0 19491 2024-09-18T04:39:23.662 INFO:tasks.workunit.client.0.smithi115.stderr:+ local args=/dev/nbd0 2024-09-18T04:39:23.662 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pid=19491 2024-09-18T04:39:23.662 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-18T04:39:23.662 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:39:23.663 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:23.663 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:23.663 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:39:23.663 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:39:23.663 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:23.663 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:39:23.664 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:39:23.665 INFO:tasks.workunit.client.0.smithi115.stderr:++ which rbd 2024-09-18T04:39:23.667 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/bin/rbd 2024-09-18T04:39:23.668 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:39:23.668 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-18T04:39:23.672 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:39:23.722 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:23.713+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:23.790 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:23.790 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:39:23.790 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:39:23.790 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:39:23.790 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.790 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:37 2.5 e51/28 2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:head v 51'1, mlcod=0'0)", 2024-09-18T04:39:23.790 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:39:23.790 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 41.494406269999999, 2024-09-18T04:39:23.790 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012279973, 2024-09-18T04:39:23.790 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:23.790 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:23.790 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:23.791 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.791 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:23.791 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:39:23.791 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:23.791 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.791 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.791 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:23.791 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:39:23.791 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:23.791 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.791 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.791 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:23.791 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292468+0000", 2024-09-18T04:39:23.792 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:23.792 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.792 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.792 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:23.792 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295837+0000", 2024-09-18T04:39:23.792 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0033691699999999999 2024-09-18T04:39:23.792 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.792 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.792 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:23.792 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295860+0000", 2024-09-18T04:39:23.792 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2756999999999999e-05 2024-09-18T04:39:23.792 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.792 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.792 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:23.793 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295872+0000", 2024-09-18T04:39:23.793 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2442e-05 2024-09-18T04:39:23.793 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.793 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.793 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:23.793 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295934+0000", 2024-09-18T04:39:23.793 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1187999999999998e-05 2024-09-18T04:39:23.793 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.793 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.793 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:23.793 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295967+0000", 2024-09-18T04:39:23.793 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3185999999999997e-05 2024-09-18T04:39:23.793 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.794 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.794 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:23.794 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304697+0000", 2024-09-18T04:39:23.794 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0087298059999999997 2024-09-18T04:39:23.794 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.794 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.794 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:23.794 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304749+0000", 2024-09-18T04:39:23.794 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1922000000000002e-05 2024-09-18T04:39:23.795 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:23.795 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:23.795 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:23.795 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.795 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.795 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:49 2.5 e51/28 2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head v 51'2, mlcod=0'0)", 2024-09-18T04:39:23.795 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:39:23.795 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 41.461079431999998, 2024-09-18T04:39:23.795 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018033041999999999, 2024-09-18T04:39:23.795 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:23.795 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:23.795 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:23.795 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.795 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:23.796 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:39:23.796 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:23.796 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.796 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.796 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:23.796 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:39:23.796 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:23.796 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.796 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.797 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:23.797 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325793+0000", 2024-09-18T04:39:23.797 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:23.797 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.797 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.797 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:23.797 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329038+0000", 2024-09-18T04:39:23.797 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0032444010000000001 2024-09-18T04:39:23.797 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.797 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.797 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:23.797 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329065+0000", 2024-09-18T04:39:23.797 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7500000000000001e-05 2024-09-18T04:39:23.798 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.798 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.798 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:23.798 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329077+0000", 2024-09-18T04:39:23.798 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1919999999999999e-05 2024-09-18T04:39:23.798 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.798 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.798 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:23.798 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329149+0000", 2024-09-18T04:39:23.798 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2349999999999997e-05 2024-09-18T04:39:23.798 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.798 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.798 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:23.798 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329192+0000", 2024-09-18T04:39:23.798 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2744000000000003e-05 2024-09-18T04:39:23.799 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.799 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.799 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:23.799 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343773+0000", 2024-09-18T04:39:23.799 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014581246000000001 2024-09-18T04:39:23.799 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.799 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.799 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:23.799 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343829+0000", 2024-09-18T04:39:23.799 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5143999999999999e-05 2024-09-18T04:39:23.799 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:23.799 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:23.799 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:23.799 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.800 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.800 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:47 2.4 e51/17 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 51'3, mlcod=51'1)", 2024-09-18T04:39:23.800 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:39:23.800 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 41.460499167000002, 2024-09-18T04:39:23.800 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018960186, 2024-09-18T04:39:23.800 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:23.800 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:23.800 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:23.800 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.800 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:23.800 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:39:23.800 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:23.800 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.801 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.801 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:23.801 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:39:23.801 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:23.801 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.801 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.801 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:23.801 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326374+0000", 2024-09-18T04:39:23.801 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:23.801 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.801 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.801 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:23.801 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330883+0000", 2024-09-18T04:39:23.801 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0045084740000000002 2024-09-18T04:39:23.802 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.802 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.802 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:23.802 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330901+0000", 2024-09-18T04:39:23.802 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7890999999999999e-05 2024-09-18T04:39:23.802 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.802 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.802 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:23.802 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330912+0000", 2024-09-18T04:39:23.802 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1041e-05 2024-09-18T04:39:23.802 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.802 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.802 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:23.802 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.331003+0000", 2024-09-18T04:39:23.803 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1323999999999995e-05 2024-09-18T04:39:23.803 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.803 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.803 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:23.803 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.331030+0000", 2024-09-18T04:39:23.803 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7351999999999998e-05 2024-09-18T04:39:23.803 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.803 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.803 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:23.803 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345287+0000", 2024-09-18T04:39:23.803 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014256332 2024-09-18T04:39:23.803 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.803 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.803 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:23.804 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345336+0000", 2024-09-18T04:39:23.804 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9356e-05 2024-09-18T04:39:23.804 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:23.804 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:23.804 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:23.804 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.804 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.804 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:58 2.5 e51/28 2:b227e6b1:::rbd_data.125fc09df6d1.000000000000000e:head v 51'3, mlcod=51'1)", 2024-09-18T04:39:23.804 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:39:23.804 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 41.42007254, 2024-09-18T04:39:23.804 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016513678, 2024-09-18T04:39:23.804 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:23.804 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:23.804 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:23.805 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.805 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:23.805 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:39:23.805 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:23.805 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.805 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.805 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:23.805 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:39:23.805 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:23.805 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.805 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.805 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:23.805 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366800+0000", 2024-09-18T04:39:23.806 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:23.806 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.806 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.806 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:23.806 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368644+0000", 2024-09-18T04:39:23.806 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001844399 2024-09-18T04:39:23.806 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.806 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.806 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:23.806 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368681+0000", 2024-09-18T04:39:23.806 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6526999999999998e-05 2024-09-18T04:39:23.806 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.806 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.806 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:23.807 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368698+0000", 2024-09-18T04:39:23.807 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7272999999999999e-05 2024-09-18T04:39:23.807 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.807 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.807 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:23.807 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368773+0000", 2024-09-18T04:39:23.807 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4862000000000003e-05 2024-09-18T04:39:23.807 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.807 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.807 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:23.807 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368801+0000", 2024-09-18T04:39:23.807 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7925000000000001e-05 2024-09-18T04:39:23.807 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.808 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.808 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:23.808 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383261+0000", 2024-09-18T04:39:23.808 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014460377999999999 2024-09-18T04:39:23.808 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.808 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.808 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:23.808 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383316+0000", 2024-09-18T04:39:23.808 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4623999999999997e-05 2024-09-18T04:39:23.808 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:23.808 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:23.808 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:23.809 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.809 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.809 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:56 2.4 e51/17 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 51'4, mlcod=51'2)", 2024-09-18T04:39:23.809 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:39:23.809 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 41.418068513999998, 2024-09-18T04:39:23.809 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015105221, 2024-09-18T04:39:23.809 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:23.809 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:23.809 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:23.809 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.809 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:23.809 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:39:23.809 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:23.809 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.810 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.810 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:23.810 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:39:23.810 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:23.810 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.810 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.810 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:23.810 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368805+0000", 2024-09-18T04:39:23.810 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:23.810 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.810 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.810 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:23.810 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371603+0000", 2024-09-18T04:39:23.810 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027982660000000002 2024-09-18T04:39:23.811 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.811 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.811 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:23.811 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371633+0000", 2024-09-18T04:39:23.811 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0073000000000002e-05 2024-09-18T04:39:23.811 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.811 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.811 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:23.811 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371652+0000", 2024-09-18T04:39:23.811 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8597000000000002e-05 2024-09-18T04:39:23.811 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.811 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.811 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:23.811 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371752+0000", 2024-09-18T04:39:23.812 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9772999999999995e-05 2024-09-18T04:39:23.812 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.812 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.812 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:23.812 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371793+0000", 2024-09-18T04:39:23.812 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1173e-05 2024-09-18T04:39:23.812 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.812 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.812 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:23.812 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383875+0000", 2024-09-18T04:39:23.812 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012081852000000001 2024-09-18T04:39:23.812 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.812 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.812 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:23.813 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383912+0000", 2024-09-18T04:39:23.813 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7045000000000002e-05 2024-09-18T04:39:23.813 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:23.813 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:23.813 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:23.813 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.813 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.813 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:36 2.5 e51/28 2:bd5b8dd8:ns::rbd_data.133e813ab5bc.0000000000000002:head v 51'4, mlcod=51'2)", 2024-09-18T04:39:23.813 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:39:23.813 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 41.249349522999999, 2024-09-18T04:39:23.813 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020758161000000001, 2024-09-18T04:39:23.813 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:23.813 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:23.813 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:23.814 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.814 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:23.814 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:39:23.814 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:23.814 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.814 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.814 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:23.814 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:39:23.814 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:23.814 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.814 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.814 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:23.814 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537523+0000", 2024-09-18T04:39:23.814 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:23.814 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.815 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.815 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:23.815 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541578+0000", 2024-09-18T04:39:23.815 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0040549710000000001 2024-09-18T04:39:23.815 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.815 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.815 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:23.815 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541608+0000", 2024-09-18T04:39:23.815 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9349e-05 2024-09-18T04:39:23.815 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.815 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.815 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:23.815 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541621+0000", 2024-09-18T04:39:23.815 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2736e-05 2024-09-18T04:39:23.816 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.816 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.816 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:23.816 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541715+0000", 2024-09-18T04:39:23.816 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4197999999999995e-05 2024-09-18T04:39:23.816 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.816 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.816 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:23.816 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541755+0000", 2024-09-18T04:39:23.816 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9818999999999999e-05 2024-09-18T04:39:23.816 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.816 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.816 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:23.817 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558217+0000", 2024-09-18T04:39:23.817 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016462234999999999 2024-09-18T04:39:23.817 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.817 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.817 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:23.817 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558284+0000", 2024-09-18T04:39:23.817 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6774999999999997e-05 2024-09-18T04:39:23.817 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:23.817 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:23.817 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:23.817 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.817 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.817 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:39 2.4 e51/17 2:304364d1:ns::rbd_data.133e813ab5bc.0000000000000004:head v 51'5, mlcod=51'3)", 2024-09-18T04:39:23.817 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:39:23.818 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 41.241210396, 2024-09-18T04:39:23.818 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016777558000000001, 2024-09-18T04:39:23.818 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:23.818 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:23.818 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:23.818 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.818 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:23.818 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:39:23.818 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:23.818 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.818 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.818 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:23.818 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:39:23.818 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:23.819 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.819 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.819 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:23.819 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545663+0000", 2024-09-18T04:39:23.819 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:23.819 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.819 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.819 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:23.819 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548286+0000", 2024-09-18T04:39:23.819 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026233390000000001 2024-09-18T04:39:23.819 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.819 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.819 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:23.819 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548305+0000", 2024-09-18T04:39:23.820 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8425000000000001e-05 2024-09-18T04:39:23.820 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.820 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.820 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:23.820 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548316+0000", 2024-09-18T04:39:23.820 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1163e-05 2024-09-18T04:39:23.820 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.820 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.820 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:23.820 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548379+0000", 2024-09-18T04:39:23.820 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2780000000000005e-05 2024-09-18T04:39:23.820 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.820 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.821 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:23.821 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548399+0000", 2024-09-18T04:39:23.821 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0080000000000001e-05 2024-09-18T04:39:23.821 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.821 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.821 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:23.821 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562383+0000", 2024-09-18T04:39:23.821 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013984671000000001 2024-09-18T04:39:23.821 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.821 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.821 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:23.821 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562442+0000", 2024-09-18T04:39:23.821 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8700999999999999e-05 2024-09-18T04:39:23.821 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:23.822 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:23.822 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:23.822 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.822 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.822 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:37 2.5 e51/28 2:bf39b8e4:ns::rbd_data.133e813ab5bc.0000000000000003:head v 51'5, mlcod=51'2)", 2024-09-18T04:39:23.822 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:39:23.822 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 41.238342127000003, 2024-09-18T04:39:23.822 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014264492, 2024-09-18T04:39:23.822 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:23.822 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:23.822 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:23.822 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.822 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:23.822 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:39:23.823 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:23.823 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.823 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.823 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:23.823 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:39:23.823 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:23.823 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.823 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.823 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:23.823 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548531+0000", 2024-09-18T04:39:23.823 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:23.823 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.823 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.823 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:23.824 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551500+0000", 2024-09-18T04:39:23.824 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0029688129999999998 2024-09-18T04:39:23.824 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.824 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.824 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:23.824 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551516+0000", 2024-09-18T04:39:23.824 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5994e-05 2024-09-18T04:39:23.824 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.824 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.824 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:23.824 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551526+0000", 2024-09-18T04:39:23.824 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9299999999999998e-06 2024-09-18T04:39:23.825 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.825 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.825 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:23.825 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551577+0000", 2024-09-18T04:39:23.825 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0871000000000001e-05 2024-09-18T04:39:23.825 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.825 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.825 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:23.825 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551602+0000", 2024-09-18T04:39:23.825 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4813000000000001e-05 2024-09-18T04:39:23.825 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.825 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.825 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:23.825 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562741+0000", 2024-09-18T04:39:23.826 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011139428999999999 2024-09-18T04:39:23.826 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.826 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.826 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:23.826 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562797+0000", 2024-09-18T04:39:23.826 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6329000000000002e-05 2024-09-18T04:39:23.826 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:23.826 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:23.826 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:23.826 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.826 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.826 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:49 2.4 e51/17 2:20f0b0c9:ns::rbd_data.133e813ab5bc.0000000000000009:head v 51'6, mlcod=51'4)", 2024-09-18T04:39:23.826 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:39:23.826 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 41.206907569000002, 2024-09-18T04:39:23.827 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013849046, 2024-09-18T04:39:23.827 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:23.827 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:23.827 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:23.827 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.827 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:23.827 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:39:23.827 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:23.827 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.827 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.827 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:23.827 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:39:23.827 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:23.827 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.828 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.828 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:23.828 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579966+0000", 2024-09-18T04:39:23.828 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:23.828 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.828 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.828 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:23.828 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582324+0000", 2024-09-18T04:39:23.828 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023578890000000002 2024-09-18T04:39:23.828 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.828 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.828 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:23.829 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582355+0000", 2024-09-18T04:39:23.829 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1485000000000001e-05 2024-09-18T04:39:23.829 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.829 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.829 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:23.829 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582371+0000", 2024-09-18T04:39:23.829 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6147999999999999e-05 2024-09-18T04:39:23.829 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.829 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.829 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:23.829 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582467+0000", 2024-09-18T04:39:23.829 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6059999999999998e-05 2024-09-18T04:39:23.829 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.829 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.830 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:23.830 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582505+0000", 2024-09-18T04:39:23.830 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7679000000000003e-05 2024-09-18T04:39:23.830 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.830 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.830 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:23.830 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593743+0000", 2024-09-18T04:39:23.830 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011238356 2024-09-18T04:39:23.830 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.830 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.830 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:23.830 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593816+0000", 2024-09-18T04:39:23.830 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2935000000000006e-05 2024-09-18T04:39:23.831 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:23.831 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:23.831 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:23.831 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.831 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.831 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:53 2.4 e51/17 2:39f7f174:ns::rbd_data.133e813ab5bc.000000000000000b:head v 51'7, mlcod=51'4)", 2024-09-18T04:39:23.831 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:39:23.831 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 41.192206413999997, 2024-09-18T04:39:23.831 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0079914159999999995, 2024-09-18T04:39:23.831 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:23.831 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:23.831 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:23.831 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.832 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:23.832 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:39:23.832 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:23.832 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.832 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.832 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:23.832 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:39:23.832 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:23.832 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.832 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.832 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:23.832 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594667+0000", 2024-09-18T04:39:23.832 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:23.833 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.833 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.833 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:23.833 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596446+0000", 2024-09-18T04:39:23.833 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001779033 2024-09-18T04:39:23.833 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.833 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.833 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:23.833 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596472+0000", 2024-09-18T04:39:23.833 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6120000000000001e-05 2024-09-18T04:39:23.833 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.833 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.834 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:23.834 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596500+0000", 2024-09-18T04:39:23.834 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7518e-05 2024-09-18T04:39:23.834 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.834 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.834 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:23.834 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596564+0000", 2024-09-18T04:39:23.834 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3880999999999994e-05 2024-09-18T04:39:23.834 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.834 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.834 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:23.834 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596596+0000", 2024-09-18T04:39:23.834 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2638999999999997e-05 2024-09-18T04:39:23.834 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.835 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.835 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:23.835 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602605+0000", 2024-09-18T04:39:23.835 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0060080699999999999 2024-09-18T04:39:23.835 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.835 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.835 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:23.835 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602660+0000", 2024-09-18T04:39:23.835 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5396000000000003e-05 2024-09-18T04:39:23.835 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:23.835 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:23.835 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:23.835 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.835 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.836 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:60 2.5 e51/28 2:a06f51e4:ns::rbd_data.133e813ab5bc.000000000000000f:head v 51'6, mlcod=51'4)", 2024-09-18T04:39:23.836 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:39:23.836 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 41.177389429000002, 2024-09-18T04:39:23.836 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0093954989999999999, 2024-09-18T04:39:23.836 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:23.836 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:23.836 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:23.836 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.836 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:23.836 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:39:23.836 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:23.836 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.836 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.837 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:23.837 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:39:23.837 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:23.837 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.837 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.837 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:23.837 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609483+0000", 2024-09-18T04:39:23.837 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:23.837 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.837 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.837 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:23.837 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611767+0000", 2024-09-18T04:39:23.837 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002283724 2024-09-18T04:39:23.837 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.838 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.838 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:23.838 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611791+0000", 2024-09-18T04:39:23.838 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3547e-05 2024-09-18T04:39:23.838 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.838 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.838 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:23.838 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611800+0000", 2024-09-18T04:39:23.838 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4979999999999995e-06 2024-09-18T04:39:23.838 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.838 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.838 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:23.838 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611864+0000", 2024-09-18T04:39:23.838 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3974999999999996e-05 2024-09-18T04:39:23.839 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.839 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.839 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:23.839 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611887+0000", 2024-09-18T04:39:23.839 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2633000000000002e-05 2024-09-18T04:39:23.839 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.839 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.839 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:23.839 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.618777+0000", 2024-09-18T04:39:23.839 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0068897979999999999 2024-09-18T04:39:23.839 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.839 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.839 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:23.839 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.618881+0000", 2024-09-18T04:39:23.840 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010437699999999999 2024-09-18T04:39:23.840 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:23.840 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:23.840 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:23.840 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.840 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.840 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:663 2.4 e57/17 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 57'13, mlcod=57'11)", 2024-09-18T04:39:23.840 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.355300+0000", 2024-09-18T04:39:23.840 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 32.431574494000003, 2024-09-18T04:39:23.840 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0049192760000000002, 2024-09-18T04:39:23.840 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:23.840 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:23.840 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:23.840 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.841 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:23.841 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355300+0000", 2024-09-18T04:39:23.841 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:23.841 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.841 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.841 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:23.841 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355300+0000", 2024-09-18T04:39:23.841 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:23.841 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.841 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.841 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:23.841 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355299+0000", 2024-09-18T04:39:23.841 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:23.841 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.842 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.842 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:23.842 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355876+0000", 2024-09-18T04:39:23.842 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00057730900000000003 2024-09-18T04:39:23.842 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.842 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.842 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:23.842 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355897+0000", 2024-09-18T04:39:23.842 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0477000000000001e-05 2024-09-18T04:39:23.842 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.842 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.842 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:23.842 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355905+0000", 2024-09-18T04:39:23.843 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5739999999999996e-06 2024-09-18T04:39:23.843 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.843 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.843 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:23.843 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355989+0000", 2024-09-18T04:39:23.843 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4002e-05 2024-09-18T04:39:23.843 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.843 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.843 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:23.843 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.356041+0000", 2024-09-18T04:39:23.843 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1347000000000002e-05 2024-09-18T04:39:23.843 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.843 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.843 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:23.844 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.360172+0000", 2024-09-18T04:39:23.844 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0041315090000000002 2024-09-18T04:39:23.844 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.844 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.844 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:23.844 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.360220+0000", 2024-09-18T04:39:23.844 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7584999999999999e-05 2024-09-18T04:39:23.844 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:23.844 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:23.844 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:23.844 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.844 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.844 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:679 2.5 e57/28 2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head v 57'20, mlcod=57'18)", 2024-09-18T04:39:23.845 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.445316+0000", 2024-09-18T04:39:23.845 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 32.341559017000002, 2024-09-18T04:39:23.845 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0047016760000000001, 2024-09-18T04:39:23.845 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:23.845 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:23.845 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:23.845 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.845 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:23.845 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445316+0000", 2024-09-18T04:39:23.845 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:23.845 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.845 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.845 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:23.845 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445316+0000", 2024-09-18T04:39:23.846 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:23.846 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.846 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.846 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:23.846 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445314+0000", 2024-09-18T04:39:23.846 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:23.846 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.846 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.846 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:23.846 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445947+0000", 2024-09-18T04:39:23.846 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00063298900000000003 2024-09-18T04:39:23.846 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.846 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.846 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:23.847 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445968+0000", 2024-09-18T04:39:23.847 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.054e-05 2024-09-18T04:39:23.847 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.847 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.847 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:23.847 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445981+0000", 2024-09-18T04:39:23.847 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2697000000000001e-05 2024-09-18T04:39:23.847 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.847 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.847 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:23.847 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.446102+0000", 2024-09-18T04:39:23.847 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012189300000000001 2024-09-18T04:39:23.847 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.848 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.848 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:23.848 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.446151+0000", 2024-09-18T04:39:23.848 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8279000000000003e-05 2024-09-18T04:39:23.848 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.848 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.848 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:23.848 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.449953+0000", 2024-09-18T04:39:23.848 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003801979 2024-09-18T04:39:23.848 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.848 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.848 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:23.848 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.450018+0000", 2024-09-18T04:39:23.849 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4873e-05 2024-09-18T04:39:23.849 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:23.849 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:23.849 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:23.849 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.849 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.849 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:699 2.4 e57/17 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 57'17, mlcod=57'15)", 2024-09-18T04:39:23.849 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.566073+0000", 2024-09-18T04:39:23.849 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 32.220801504000001, 2024-09-18T04:39:23.849 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055398380000000001, 2024-09-18T04:39:23.849 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:23.849 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:23.850 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:23.850 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.850 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:23.850 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566073+0000", 2024-09-18T04:39:23.850 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:23.850 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.850 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.850 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:23.850 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566073+0000", 2024-09-18T04:39:23.850 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:23.850 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.850 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.850 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:23.850 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566071+0000", 2024-09-18T04:39:23.851 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:23.851 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.851 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.851 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:23.851 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566637+0000", 2024-09-18T04:39:23.851 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00056516699999999999 2024-09-18T04:39:23.851 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.851 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.851 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:23.851 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566668+0000", 2024-09-18T04:39:23.851 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1544999999999997e-05 2024-09-18T04:39:23.851 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.852 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.852 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:23.852 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566685+0000", 2024-09-18T04:39:23.852 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6857000000000001e-05 2024-09-18T04:39:23.852 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.852 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.852 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:23.852 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566816+0000", 2024-09-18T04:39:23.852 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013092799999999999 2024-09-18T04:39:23.852 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.852 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.852 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:23.852 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566875+0000", 2024-09-18T04:39:23.853 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9583999999999997e-05 2024-09-18T04:39:23.853 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.853 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.853 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:23.853 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.571567+0000", 2024-09-18T04:39:23.853 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.004691682 2024-09-18T04:39:23.853 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.853 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.853 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:23.853 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.571613+0000", 2024-09-18T04:39:23.853 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6125000000000003e-05 2024-09-18T04:39:23.853 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:23.853 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:23.854 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:23.854 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.854 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.854 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:98 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'27, mlcod=60'25)", 2024-09-18T04:39:23.854 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.847978+0000", 2024-09-18T04:39:23.854 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.938896945, 2024-09-18T04:39:23.854 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0044617709999999998, 2024-09-18T04:39:23.854 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:23.854 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:23.854 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:23.854 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.854 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:23.855 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.847978+0000", 2024-09-18T04:39:23.855 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:23.855 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.855 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.855 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:23.855 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.847978+0000", 2024-09-18T04:39:23.855 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:23.855 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.855 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.855 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:23.855 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.847977+0000", 2024-09-18T04:39:23.855 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:23.855 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.855 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.856 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:23.856 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849038+0000", 2024-09-18T04:39:23.856 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0010614820000000001 2024-09-18T04:39:23.856 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.856 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.856 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:23.856 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849065+0000", 2024-09-18T04:39:23.856 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7127999999999998e-05 2024-09-18T04:39:23.856 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.856 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.856 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:23.856 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849082+0000", 2024-09-18T04:39:23.856 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7170999999999999e-05 2024-09-18T04:39:23.856 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.857 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.857 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:23.857 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849188+0000", 2024-09-18T04:39:23.857 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010605800000000001 2024-09-18T04:39:23.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.857 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.857 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:23.857 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849233+0000", 2024-09-18T04:39:23.857 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4579000000000001e-05 2024-09-18T04:39:23.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.857 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.857 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:23.857 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.852370+0000", 2024-09-18T04:39:23.857 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00313751 2024-09-18T04:39:23.858 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.858 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.858 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:23.858 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.852440+0000", 2024-09-18T04:39:23.858 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9238000000000004e-05 2024-09-18T04:39:23.858 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:23.858 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:23.858 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:23.858 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.858 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.858 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:100 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'29, mlcod=64'27)", 2024-09-18T04:39:23.858 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.858766+0000", 2024-09-18T04:39:23.858 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.928108615999999, 2024-09-18T04:39:23.858 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0061964840000000004, 2024-09-18T04:39:23.859 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:23.859 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:23.859 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:23.859 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.859 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:23.859 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.858766+0000", 2024-09-18T04:39:23.859 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:23.859 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.859 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.859 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:23.859 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.858766+0000", 2024-09-18T04:39:23.859 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:23.859 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.859 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.860 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:23.860 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.858765+0000", 2024-09-18T04:39:23.860 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:23.860 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.860 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.860 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:23.860 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861612+0000", 2024-09-18T04:39:23.860 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028473169999999998 2024-09-18T04:39:23.860 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.860 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.860 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:23.860 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861643+0000", 2024-09-18T04:39:23.860 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1356000000000003e-05 2024-09-18T04:39:23.860 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.861 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.861 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:23.861 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861656+0000", 2024-09-18T04:39:23.861 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2899e-05 2024-09-18T04:39:23.861 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.861 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.861 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:23.861 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861773+0000", 2024-09-18T04:39:23.861 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011637999999999999 2024-09-18T04:39:23.861 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.861 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.861 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:23.861 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861808+0000", 2024-09-18T04:39:23.861 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5002999999999997e-05 2024-09-18T04:39:23.862 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.862 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.862 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:23.862 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.864876+0000", 2024-09-18T04:39:23.862 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030680719999999998 2024-09-18T04:39:23.862 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.862 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.862 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:23.862 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.864963+0000", 2024-09-18T04:39:23.862 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7114000000000007e-05 2024-09-18T04:39:23.862 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:23.862 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:23.862 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:23.863 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.863 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.863 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:102 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'31, mlcod=64'29)", 2024-09-18T04:39:23.863 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.872098+0000", 2024-09-18T04:39:23.863 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.914777358999999, 2024-09-18T04:39:23.863 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0064676450000000002, 2024-09-18T04:39:23.863 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:23.863 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:23.863 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:23.863 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.863 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:23.863 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.872098+0000", 2024-09-18T04:39:23.863 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:23.863 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.864 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.864 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:23.864 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.872098+0000", 2024-09-18T04:39:23.864 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:23.864 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.864 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.864 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:23.864 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.872096+0000", 2024-09-18T04:39:23.864 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:23.864 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.864 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.864 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:23.864 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875176+0000", 2024-09-18T04:39:23.865 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030799809999999999 2024-09-18T04:39:23.865 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.865 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.865 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:23.865 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875204+0000", 2024-09-18T04:39:23.865 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8025999999999999e-05 2024-09-18T04:39:23.865 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.865 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.865 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:23.865 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875216+0000", 2024-09-18T04:39:23.865 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2072999999999999e-05 2024-09-18T04:39:23.865 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.865 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.866 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:23.866 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875307+0000", 2024-09-18T04:39:23.866 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0373999999999999e-05 2024-09-18T04:39:23.866 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.866 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.866 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:23.866 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875349+0000", 2024-09-18T04:39:23.866 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2299000000000003e-05 2024-09-18T04:39:23.866 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.866 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.866 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:23.866 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.878496+0000", 2024-09-18T04:39:23.866 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0031468960000000002 2024-09-18T04:39:23.866 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.867 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.867 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:23.867 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.878565+0000", 2024-09-18T04:39:23.867 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9465e-05 2024-09-18T04:39:23.867 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:23.867 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:23.867 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:23.867 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.867 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.867 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:104 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'33, mlcod=64'31)", 2024-09-18T04:39:23.867 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.885100+0000", 2024-09-18T04:39:23.867 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.901774824, 2024-09-18T04:39:23.867 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0051888159999999997, 2024-09-18T04:39:23.867 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:23.867 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:23.868 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:23.868 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.868 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:23.868 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.885100+0000", 2024-09-18T04:39:23.868 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:23.868 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.868 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.868 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:23.868 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.885100+0000", 2024-09-18T04:39:23.868 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:23.868 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.868 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.868 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:23.869 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.885098+0000", 2024-09-18T04:39:23.869 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:23.869 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.869 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.869 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:23.869 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886809+0000", 2024-09-18T04:39:23.869 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017110459999999999 2024-09-18T04:39:23.869 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.869 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.869 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:23.869 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886842+0000", 2024-09-18T04:39:23.869 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2638999999999997e-05 2024-09-18T04:39:23.869 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.869 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.869 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:23.870 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886855+0000", 2024-09-18T04:39:23.870 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3380000000000001e-05 2024-09-18T04:39:23.870 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.870 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.870 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:23.870 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886939+0000", 2024-09-18T04:39:23.870 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3733999999999995e-05 2024-09-18T04:39:23.870 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.870 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.870 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:23.870 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886981+0000", 2024-09-18T04:39:23.870 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1956000000000003e-05 2024-09-18T04:39:23.870 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.870 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.871 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:23.871 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.890201+0000", 2024-09-18T04:39:23.871 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003219927 2024-09-18T04:39:23.871 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.871 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.871 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:23.871 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.890289+0000", 2024-09-18T04:39:23.871 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7997999999999993e-05 2024-09-18T04:39:23.871 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:23.871 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:23.871 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:23.871 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.871 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.871 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:145 2.4 e64/62 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 64'37, mlcod=64'35)", 2024-09-18T04:39:23.872 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:01.099398+0000", 2024-09-18T04:39:23.872 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.687477336000001, 2024-09-18T04:39:23.872 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0050078459999999998, 2024-09-18T04:39:23.872 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:23.872 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:23.872 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:23.872 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.872 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:23.872 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.099398+0000", 2024-09-18T04:39:23.872 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:23.872 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.872 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.872 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:23.873 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.099398+0000", 2024-09-18T04:39:23.873 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:23.873 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.873 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.873 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:23.873 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.099396+0000", 2024-09-18T04:39:23.873 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:23.873 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.873 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.873 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:23.873 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100199+0000", 2024-09-18T04:39:23.873 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00080338000000000002 2024-09-18T04:39:23.873 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.874 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.874 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:23.874 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100230+0000", 2024-09-18T04:39:23.874 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0716000000000002e-05 2024-09-18T04:39:23.874 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.874 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.874 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:23.874 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100243+0000", 2024-09-18T04:39:23.874 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3359e-05 2024-09-18T04:39:23.874 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.874 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.874 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:23.875 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100337+0000", 2024-09-18T04:39:23.875 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4387000000000003e-05 2024-09-18T04:39:23.875 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.875 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.875 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:23.875 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100396+0000", 2024-09-18T04:39:23.875 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8164000000000001e-05 2024-09-18T04:39:23.875 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.875 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.875 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:23.875 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.104309+0000", 2024-09-18T04:39:23.875 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0039137800000000004 2024-09-18T04:39:23.875 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.875 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.875 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:23.876 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.104405+0000", 2024-09-18T04:39:23.876 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6039999999999995e-05 2024-09-18T04:39:23.876 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:23.876 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:23.876 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:23.876 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.876 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.876 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:147 2.4 e64/62 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 64'39, mlcod=64'37)", 2024-09-18T04:39:23.876 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:01.110658+0000", 2024-09-18T04:39:23.876 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.676217247, 2024-09-18T04:39:23.876 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0046316910000000003, 2024-09-18T04:39:23.876 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:23.876 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:23.877 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:23.877 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.877 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:23.877 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.110658+0000", 2024-09-18T04:39:23.877 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:23.877 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.877 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.877 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:23.877 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.110658+0000", 2024-09-18T04:39:23.877 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:23.877 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.877 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.877 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:23.877 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.110656+0000", 2024-09-18T04:39:23.878 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:23.878 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.878 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.878 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:23.878 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111444+0000", 2024-09-18T04:39:23.878 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00078762400000000001 2024-09-18T04:39:23.878 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.878 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.878 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:23.878 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111474+0000", 2024-09-18T04:39:23.878 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0085000000000001e-05 2024-09-18T04:39:23.878 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.878 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.878 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:23.879 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111486+0000", 2024-09-18T04:39:23.879 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2645e-05 2024-09-18T04:39:23.879 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.879 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.879 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:23.879 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111581+0000", 2024-09-18T04:39:23.879 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4432000000000007e-05 2024-09-18T04:39:23.879 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.879 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.879 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:23.879 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111631+0000", 2024-09-18T04:39:23.879 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9901000000000002e-05 2024-09-18T04:39:23.879 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.879 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.880 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:23.880 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.115209+0000", 2024-09-18T04:39:23.880 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003577916 2024-09-18T04:39:23.880 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:23.880 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:23.880 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:23.880 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.115289+0000", 2024-09-18T04:39:23.880 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0704999999999994e-05 2024-09-18T04:39:23.880 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:23.880 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:23.880 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:23.880 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:23.880 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:23.881 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:23.882 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:39:23.883 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:23.877+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:23.983 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:23.981+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:23.989 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:23.989 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:23.989 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:39:23.989 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:23.999 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:39:24.084 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:24.077+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:24.102 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:24.103 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:24.103 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:39:24.103 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:39:24.103 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:24.112 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:39:24.185 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:24.180+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:39:24.222 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:24.222 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:39:24.222 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:39:24.222 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:39:24.222 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.222 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:33 2.1 e51/24 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head v 51'4, mlcod=51'2)", 2024-09-18T04:39:24.222 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:39:24.222 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 41.937662050999997, 2024-09-18T04:39:24.222 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018702106, 2024-09-18T04:39:24.222 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:24.222 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:24.222 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:24.223 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.223 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:24.223 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:39:24.223 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.223 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.223 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.223 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:24.223 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:39:24.223 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.223 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.223 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.223 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:24.223 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279197+0000", 2024-09-18T04:39:24.224 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:24.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.224 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.224 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:24.224 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282682+0000", 2024-09-18T04:39:24.224 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0034848689999999998 2024-09-18T04:39:24.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.224 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.224 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:24.224 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282709+0000", 2024-09-18T04:39:24.224 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7599000000000001e-05 2024-09-18T04:39:24.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.224 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.224 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:24.225 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282720+0000", 2024-09-18T04:39:24.225 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.092e-05 2024-09-18T04:39:24.225 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.225 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.225 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:24.225 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282791+0000", 2024-09-18T04:39:24.225 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1268999999999998e-05 2024-09-18T04:39:24.225 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.225 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.225 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:24.225 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282843+0000", 2024-09-18T04:39:24.225 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1771999999999998e-05 2024-09-18T04:39:24.225 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.226 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.226 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:24.226 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.297830+0000", 2024-09-18T04:39:24.226 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014987228999999999 2024-09-18T04:39:24.226 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.226 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.226 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:24.226 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.297901+0000", 2024-09-18T04:39:24.226 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0209999999999994e-05 2024-09-18T04:39:24.226 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.226 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:24.226 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.226 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.226 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.227 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:38 2.0 e51/17 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head v 51'10, mlcod=51'8)", 2024-09-18T04:39:24.227 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:39:24.227 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 41.936243621000003, 2024-09-18T04:39:24.227 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.023984472, 2024-09-18T04:39:24.227 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:24.227 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:24.227 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:24.227 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.227 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:24.227 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:39:24.227 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.227 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.227 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.228 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:24.228 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:39:24.228 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.228 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.228 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.228 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:24.228 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280614+0000", 2024-09-18T04:39:24.228 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:24.228 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.228 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.228 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:24.228 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293094+0000", 2024-09-18T04:39:24.228 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012479571 2024-09-18T04:39:24.229 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.229 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.229 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:24.229 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293111+0000", 2024-09-18T04:39:24.229 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7354999999999999e-05 2024-09-18T04:39:24.229 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.229 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.229 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:24.229 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293121+0000", 2024-09-18T04:39:24.229 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6979999999999993e-06 2024-09-18T04:39:24.229 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.229 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.229 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:24.229 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293180+0000", 2024-09-18T04:39:24.230 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9333000000000003e-05 2024-09-18T04:39:24.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.230 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.230 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:24.230 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293203+0000", 2024-09-18T04:39:24.230 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2633000000000002e-05 2024-09-18T04:39:24.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.230 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.230 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:24.230 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304534+0000", 2024-09-18T04:39:24.230 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011330761 2024-09-18T04:39:24.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.230 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.230 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:24.231 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304601+0000", 2024-09-18T04:39:24.231 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7657000000000006e-05 2024-09-18T04:39:24.231 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.231 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:24.231 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.231 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.231 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.231 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:46 2.6 2:7ba25245:::rbd_data.125fc09df6d1.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:24.231 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:39:24.231 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 41.908700820999996, 2024-09-18T04:39:24.231 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.035604595000000003, 2024-09-18T04:39:24.231 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:24.231 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:24.231 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:24.232 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:24.232 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:24.232 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 46 2024-09-18T04:39:24.232 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.232 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:24.232 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.232 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:24.232 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:39:24.232 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.232 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.232 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.232 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:24.232 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:39:24.232 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.233 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.233 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.233 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:24.233 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308158+0000", 2024-09-18T04:39:24.233 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:24.233 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.233 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.233 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:24.233 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313191+0000", 2024-09-18T04:39:24.233 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0050323929999999996 2024-09-18T04:39:24.233 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.233 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.233 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:24.233 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313210+0000", 2024-09-18T04:39:24.234 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9673e-05 2024-09-18T04:39:24.234 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.234 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.234 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:24.234 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313221+0000", 2024-09-18T04:39:24.234 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0444e-05 2024-09-18T04:39:24.234 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.234 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.234 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:24.234 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313294+0000", 2024-09-18T04:39:24.234 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2650000000000004e-05 2024-09-18T04:39:24.235 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.235 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.235 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:24.235 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313493+0000", 2024-09-18T04:39:24.235 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019968300000000001 2024-09-18T04:39:24.235 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.235 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.235 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:24.235 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313666+0000", 2024-09-18T04:39:24.235 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00017248599999999999 2024-09-18T04:39:24.235 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.235 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.235 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:24.235 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.323175+0000", 2024-09-18T04:39:24.236 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0095088819999999998 2024-09-18T04:39:24.236 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.236 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.236 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:24.236 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343651+0000", 2024-09-18T04:39:24.236 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020475892999999998 2024-09-18T04:39:24.236 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.236 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.236 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:24.236 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343715+0000", 2024-09-18T04:39:24.236 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4443999999999995e-05 2024-09-18T04:39:24.236 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.236 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.237 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:24.237 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343764+0000", 2024-09-18T04:39:24.237 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9280000000000003e-05 2024-09-18T04:39:24.237 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.237 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:24.237 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.237 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.237 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.237 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:47 2.4 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:24.237 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:39:24.237 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 41.903618784000003, 2024-09-18T04:39:24.237 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.032570693999999997, 2024-09-18T04:39:24.237 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:24.237 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:24.238 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:24.238 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:24.238 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:24.238 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 47 2024-09-18T04:39:24.238 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.238 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:24.238 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.238 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:24.238 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:39:24.238 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.238 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.238 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.238 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:24.238 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:39:24.239 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.239 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.239 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.239 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:24.239 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313240+0000", 2024-09-18T04:39:24.239 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:24.239 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.239 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.239 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:24.239 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325825+0000", 2024-09-18T04:39:24.239 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012584425999999999 2024-09-18T04:39:24.239 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.239 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.240 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:24.240 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325848+0000", 2024-09-18T04:39:24.240 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3025999999999999e-05 2024-09-18T04:39:24.240 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.240 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.240 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:24.240 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325863+0000", 2024-09-18T04:39:24.240 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5152e-05 2024-09-18T04:39:24.240 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.240 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.240 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:24.240 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325938+0000", 2024-09-18T04:39:24.240 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4982999999999997e-05 2024-09-18T04:39:24.241 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.241 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.241 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:24.241 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326090+0000", 2024-09-18T04:39:24.241 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015207900000000001 2024-09-18T04:39:24.241 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.241 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.241 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:24.241 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326242+0000", 2024-09-18T04:39:24.241 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015163500000000001 2024-09-18T04:39:24.241 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.241 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.241 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:24.242 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343015+0000", 2024-09-18T04:39:24.242 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016773414 2024-09-18T04:39:24.242 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.242 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.242 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:24.242 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345664+0000", 2024-09-18T04:39:24.242 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026487870000000001 2024-09-18T04:39:24.242 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.242 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.242 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:24.242 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345739+0000", 2024-09-18T04:39:24.242 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5018999999999994e-05 2024-09-18T04:39:24.242 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.242 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.243 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:24.243 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345812+0000", 2024-09-18T04:39:24.243 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3547000000000006e-05 2024-09-18T04:39:24.243 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.243 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:24.243 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.243 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.243 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.243 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:48 2.6 2:61db7a3a:::rbd_data.125fc09df6d1.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:24.243 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:39:24.243 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 41.890962111999997, 2024-09-18T04:39:24.243 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.025064388, 2024-09-18T04:39:24.244 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:24.244 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:24.244 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:24.244 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:24.244 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:24.244 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 48 2024-09-18T04:39:24.244 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.244 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:24.244 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.244 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:24.244 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:39:24.244 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.244 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.244 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.244 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:24.245 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:39:24.245 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.245 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.245 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.245 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:24.245 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325897+0000", 2024-09-18T04:39:24.245 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:24.245 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.245 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.245 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:24.245 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333697+0000", 2024-09-18T04:39:24.245 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0078002649999999998 2024-09-18T04:39:24.246 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.246 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.246 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:24.246 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333716+0000", 2024-09-18T04:39:24.246 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8910999999999999e-05 2024-09-18T04:39:24.246 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.246 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.246 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:24.246 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333728+0000", 2024-09-18T04:39:24.246 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1936000000000001e-05 2024-09-18T04:39:24.246 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.246 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.246 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:24.246 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333792+0000", 2024-09-18T04:39:24.246 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4010999999999994e-05 2024-09-18T04:39:24.247 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.247 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.247 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:24.247 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.334082+0000", 2024-09-18T04:39:24.247 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00028955799999999998 2024-09-18T04:39:24.247 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.247 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.247 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:24.247 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.334326+0000", 2024-09-18T04:39:24.247 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00024399099999999999 2024-09-18T04:39:24.247 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.247 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.247 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:24.247 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350745+0000", 2024-09-18T04:39:24.247 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016419474999999999 2024-09-18T04:39:24.248 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.248 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.248 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:24.248 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350808+0000", 2024-09-18T04:39:24.248 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3244999999999995e-05 2024-09-18T04:39:24.248 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.248 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.248 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:24.248 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350892+0000", 2024-09-18T04:39:24.248 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4074999999999997e-05 2024-09-18T04:39:24.248 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.248 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.248 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:24.249 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350963+0000", 2024-09-18T04:39:24.249 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0412000000000003e-05 2024-09-18T04:39:24.249 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.249 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:24.249 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.249 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.249 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.249 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:51 2.6 2:775d7b11:::rbd_data.125fc09df6d1.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:24.249 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:39:24.249 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 41.883109976, 2024-09-18T04:39:24.249 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.023824221, 2024-09-18T04:39:24.249 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:24.249 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:24.250 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:24.250 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:24.250 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:24.250 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 51 2024-09-18T04:39:24.250 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.250 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:24.250 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.250 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:24.250 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:39:24.250 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.250 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.250 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.250 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:24.251 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:39:24.251 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.251 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.251 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.251 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:24.251 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333749+0000", 2024-09-18T04:39:24.251 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:24.251 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.251 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.251 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:24.251 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340899+0000", 2024-09-18T04:39:24.251 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0071500340000000004 2024-09-18T04:39:24.251 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.251 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.252 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:24.252 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340929+0000", 2024-09-18T04:39:24.252 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0093000000000001e-05 2024-09-18T04:39:24.252 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.252 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.252 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:24.252 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340943+0000", 2024-09-18T04:39:24.252 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4019000000000001e-05 2024-09-18T04:39:24.252 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.252 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.252 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:24.252 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343834+0000", 2024-09-18T04:39:24.253 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028918120000000001 2024-09-18T04:39:24.253 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.253 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.253 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:24.253 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344020+0000", 2024-09-18T04:39:24.253 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000185892 2024-09-18T04:39:24.253 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.253 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.253 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:24.253 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344190+0000", 2024-09-18T04:39:24.253 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016946800000000001 2024-09-18T04:39:24.253 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.253 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.253 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:24.254 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.351291+0000", 2024-09-18T04:39:24.254 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.007100805 2024-09-18T04:39:24.254 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.254 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.254 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:24.254 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357470+0000", 2024-09-18T04:39:24.254 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0061797570000000001 2024-09-18T04:39:24.254 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.254 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.254 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:24.254 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357533+0000", 2024-09-18T04:39:24.254 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2906999999999999e-05 2024-09-18T04:39:24.254 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.254 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.254 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:24.255 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357575+0000", 2024-09-18T04:39:24.255 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1403999999999999e-05 2024-09-18T04:39:24.255 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.255 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:24.255 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.255 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.255 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.255 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:56 2.4 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:24.255 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:39:24.255 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 41.867064882999998, 2024-09-18T04:39:24.255 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.034295870999999999, 2024-09-18T04:39:24.255 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:24.255 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:24.255 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:24.256 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:24.256 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:24.256 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 56 2024-09-18T04:39:24.256 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.256 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:24.256 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.256 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:24.256 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:39:24.256 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.256 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.256 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.256 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:24.256 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:39:24.256 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.257 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.257 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.257 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:24.257 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349794+0000", 2024-09-18T04:39:24.257 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:24.257 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.257 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.257 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:24.257 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366632+0000", 2024-09-18T04:39:24.257 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01683755 2024-09-18T04:39:24.257 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.257 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.257 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:24.257 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366669+0000", 2024-09-18T04:39:24.258 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7351000000000002e-05 2024-09-18T04:39:24.258 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.258 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.258 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:24.258 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366686+0000", 2024-09-18T04:39:24.258 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6507999999999998e-05 2024-09-18T04:39:24.258 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.258 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.258 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:24.258 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366962+0000", 2024-09-18T04:39:24.258 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00027655900000000001 2024-09-18T04:39:24.258 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.258 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.258 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:24.259 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.367452+0000", 2024-09-18T04:39:24.259 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00049025299999999996 2024-09-18T04:39:24.259 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.259 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.259 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:24.259 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368113+0000", 2024-09-18T04:39:24.259 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00066093399999999996 2024-09-18T04:39:24.259 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.259 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.259 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:24.259 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383719+0000", 2024-09-18T04:39:24.259 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015606106999999999 2024-09-18T04:39:24.259 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.259 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.260 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:24.260 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384035+0000", 2024-09-18T04:39:24.260 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00031590399999999999 2024-09-18T04:39:24.260 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.260 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.260 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:24.260 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384067+0000", 2024-09-18T04:39:24.260 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.129e-05 2024-09-18T04:39:24.260 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.260 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.260 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:24.260 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384091+0000", 2024-09-18T04:39:24.260 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4788000000000001e-05 2024-09-18T04:39:24.261 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.261 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:24.261 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.261 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.261 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.261 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:57 2.0 e51/17 2:0cdd070e:::rbd_data.125fc09df6d1.000000000000000d:head v 51'21, mlcod=51'19)", 2024-09-18T04:39:24.261 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:39:24.261 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 41.859237395999997, 2024-09-18T04:39:24.261 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.025216311000000002, 2024-09-18T04:39:24.261 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:24.261 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:24.261 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:24.261 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.261 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:24.261 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:39:24.262 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.262 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.262 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.262 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:24.262 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:39:24.262 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.262 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.262 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.262 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:24.262 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:39:24.262 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:24.262 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.262 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.262 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:24.263 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363844+0000", 2024-09-18T04:39:24.263 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0062218450000000002 2024-09-18T04:39:24.263 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.263 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.263 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:24.263 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363879+0000", 2024-09-18T04:39:24.263 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4382999999999999e-05 2024-09-18T04:39:24.263 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.263 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.263 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:24.263 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363893+0000", 2024-09-18T04:39:24.263 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4348999999999999e-05 2024-09-18T04:39:24.263 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.263 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.264 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:24.264 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363977+0000", 2024-09-18T04:39:24.264 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3499999999999997e-05 2024-09-18T04:39:24.264 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.264 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.264 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:24.264 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364009+0000", 2024-09-18T04:39:24.264 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2292000000000002e-05 2024-09-18T04:39:24.264 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.264 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.264 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:24.265 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.382781+0000", 2024-09-18T04:39:24.265 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018771626 2024-09-18T04:39:24.265 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.265 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.265 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:24.265 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.382839+0000", 2024-09-18T04:39:24.265 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8758000000000002e-05 2024-09-18T04:39:24.265 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.265 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:24.265 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.265 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.265 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.265 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:59 2.0 e51/17 2:19c52cde:::rbd_data.125fc09df6d1.000000000000000f:head v 51'22, mlcod=51'19)", 2024-09-18T04:39:24.265 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:39:24.266 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 41.85257747, 2024-09-18T04:39:24.266 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.019497112, 2024-09-18T04:39:24.266 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:24.266 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:24.266 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:24.266 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.266 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:24.266 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:39:24.266 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.266 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.266 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.266 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:24.266 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:39:24.266 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.266 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.267 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.267 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:24.267 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364282+0000", 2024-09-18T04:39:24.267 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:24.267 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.267 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.267 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:24.267 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371823+0000", 2024-09-18T04:39:24.267 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0075411829999999999 2024-09-18T04:39:24.267 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.267 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.267 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:24.267 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371837+0000", 2024-09-18T04:39:24.267 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4336e-05 2024-09-18T04:39:24.268 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.268 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.268 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:24.268 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371846+0000", 2024-09-18T04:39:24.268 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9770000000000006e-06 2024-09-18T04:39:24.268 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.268 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.268 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:24.268 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.373566+0000", 2024-09-18T04:39:24.268 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017192539999999999 2024-09-18T04:39:24.268 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.268 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.268 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:24.269 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.373595+0000", 2024-09-18T04:39:24.269 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8975999999999998e-05 2024-09-18T04:39:24.269 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.269 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.269 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:24.269 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383740+0000", 2024-09-18T04:39:24.269 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01014525 2024-09-18T04:39:24.269 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.269 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.269 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:24.269 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383780+0000", 2024-09-18T04:39:24.269 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0275999999999999e-05 2024-09-18T04:39:24.269 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.269 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:24.270 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.270 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.270 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.270 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:34 2.6 2:66289171:ns::rbd_data.133e813ab5bc.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:24.270 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.524015+0000", 2024-09-18T04:39:24.270 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 41.692845861000002, 2024-09-18T04:39:24.270 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027582111999999999, 2024-09-18T04:39:24.270 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:24.270 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:24.270 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:24.270 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:24.270 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:24.270 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 34 2024-09-18T04:39:24.270 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.271 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:24.271 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.271 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:24.271 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524015+0000", 2024-09-18T04:39:24.271 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.271 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.271 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.271 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:24.271 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524015+0000", 2024-09-18T04:39:24.271 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.271 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.271 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.271 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:24.272 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524012+0000", 2024-09-18T04:39:24.272 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:24.272 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.272 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.272 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:24.272 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534956+0000", 2024-09-18T04:39:24.272 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010943403000000001 2024-09-18T04:39:24.272 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.272 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.272 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:24.272 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534978+0000", 2024-09-18T04:39:24.272 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1965999999999999e-05 2024-09-18T04:39:24.272 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.272 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.273 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:24.273 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534990+0000", 2024-09-18T04:39:24.273 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2065999999999999e-05 2024-09-18T04:39:24.273 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.273 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.273 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:24.273 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535065+0000", 2024-09-18T04:39:24.273 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5523000000000002e-05 2024-09-18T04:39:24.273 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.273 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.273 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:24.273 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535208+0000", 2024-09-18T04:39:24.273 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014309500000000001 2024-09-18T04:39:24.273 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.274 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.274 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:24.274 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535343+0000", 2024-09-18T04:39:24.274 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013437400000000001 2024-09-18T04:39:24.274 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.274 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.274 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:24.274 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.546576+0000", 2024-09-18T04:39:24.274 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011233425999999999 2024-09-18T04:39:24.274 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.274 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.274 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:24.274 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551498+0000", 2024-09-18T04:39:24.274 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0049219349999999997 2024-09-18T04:39:24.275 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.275 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.275 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:24.275 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551554+0000", 2024-09-18T04:39:24.275 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6400999999999998e-05 2024-09-18T04:39:24.275 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.275 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.275 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:24.275 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551597+0000", 2024-09-18T04:39:24.275 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2283000000000002e-05 2024-09-18T04:39:24.275 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.275 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:24.275 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.276 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.276 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.276 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:33 2.1 e51/24 2:8b40f1c5:ns::rbd_data.133e813ab5bc.0000000000000000:head v 51'7, mlcod=51'5)", 2024-09-18T04:39:24.276 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.531767+0000", 2024-09-18T04:39:24.276 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 41.685093944000002, 2024-09-18T04:39:24.276 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015370295000000001, 2024-09-18T04:39:24.276 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:24.276 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:24.276 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:24.276 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.276 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:24.276 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531767+0000", 2024-09-18T04:39:24.276 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.276 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.277 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.277 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:24.277 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531767+0000", 2024-09-18T04:39:24.277 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.277 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.277 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.277 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:24.277 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531765+0000", 2024-09-18T04:39:24.277 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:24.277 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.277 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.277 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:24.277 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534258+0000", 2024-09-18T04:39:24.277 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002493561 2024-09-18T04:39:24.278 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.278 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.278 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:24.278 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534288+0000", 2024-09-18T04:39:24.278 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.976e-05 2024-09-18T04:39:24.278 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.278 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.278 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:24.278 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534300+0000", 2024-09-18T04:39:24.278 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1928e-05 2024-09-18T04:39:24.278 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.278 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.278 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:24.278 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534367+0000", 2024-09-18T04:39:24.279 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7045999999999995e-05 2024-09-18T04:39:24.279 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.279 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.279 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:24.279 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534394+0000", 2024-09-18T04:39:24.279 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6903999999999999e-05 2024-09-18T04:39:24.279 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.279 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.279 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:24.279 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.547073+0000", 2024-09-18T04:39:24.279 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012679365 2024-09-18T04:39:24.279 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.279 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.280 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:24.280 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.547137+0000", 2024-09-18T04:39:24.280 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3691999999999999e-05 2024-09-18T04:39:24.280 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.280 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:24.280 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.280 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.280 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.280 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:39 2.4 2:304364d1:ns::rbd_data.133e813ab5bc.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:24.280 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.535013+0000", 2024-09-18T04:39:24.280 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 41.681847009000002, 2024-09-18T04:39:24.280 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027764487000000001, 2024-09-18T04:39:24.280 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:24.281 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:24.281 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:24.281 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:24.281 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:24.281 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 39 2024-09-18T04:39:24.281 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.281 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:24.281 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.281 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:24.281 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535013+0000", 2024-09-18T04:39:24.281 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.281 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.281 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.282 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:24.282 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535013+0000", 2024-09-18T04:39:24.282 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.282 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.282 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.282 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:24.282 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535012+0000", 2024-09-18T04:39:24.282 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:24.282 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.282 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.282 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:24.282 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545016+0000", 2024-09-18T04:39:24.282 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01000322 2024-09-18T04:39:24.283 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.283 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.283 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:24.283 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545035+0000", 2024-09-18T04:39:24.283 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9845999999999999e-05 2024-09-18T04:39:24.283 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.283 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.283 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:24.283 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545045+0000", 2024-09-18T04:39:24.283 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3640000000000005e-06 2024-09-18T04:39:24.283 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.283 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.283 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:24.283 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545152+0000", 2024-09-18T04:39:24.284 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000107025 2024-09-18T04:39:24.284 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.284 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.284 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:24.284 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545338+0000", 2024-09-18T04:39:24.284 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018643500000000001 2024-09-18T04:39:24.284 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.284 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.284 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:24.284 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545492+0000", 2024-09-18T04:39:24.284 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015393199999999999 2024-09-18T04:39:24.284 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.284 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.285 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:24.285 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.553549+0000", 2024-09-18T04:39:24.285 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0080569170000000002 2024-09-18T04:39:24.285 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.285 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.285 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:24.285 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562693+0000", 2024-09-18T04:39:24.285 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0091441609999999996 2024-09-18T04:39:24.285 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.285 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.285 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:24.285 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562748+0000", 2024-09-18T04:39:24.285 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5047000000000003e-05 2024-09-18T04:39:24.286 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.286 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.286 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:24.286 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562778+0000", 2024-09-18T04:39:24.286 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9652e-05 2024-09-18T04:39:24.286 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.286 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:24.286 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.286 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.286 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.286 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:40 2.6 2:784d6cd1:ns::rbd_data.133e813ab5bc.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:24.286 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.545062+0000", 2024-09-18T04:39:24.286 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 41.671797957999999, 2024-09-18T04:39:24.286 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.032771623, 2024-09-18T04:39:24.287 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:24.287 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:24.287 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:24.287 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:24.287 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:24.287 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 40 2024-09-18T04:39:24.287 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.287 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:24.287 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.287 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:24.287 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545062+0000", 2024-09-18T04:39:24.287 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.287 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.288 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.288 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:24.288 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545062+0000", 2024-09-18T04:39:24.288 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.288 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.288 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.288 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:24.288 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545061+0000", 2024-09-18T04:39:24.288 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:24.288 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.288 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.288 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:24.288 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552477+0000", 2024-09-18T04:39:24.289 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.007415914 2024-09-18T04:39:24.289 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.289 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.289 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:24.289 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552500+0000", 2024-09-18T04:39:24.289 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3546000000000001e-05 2024-09-18T04:39:24.289 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.289 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.289 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:24.289 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552515+0000", 2024-09-18T04:39:24.289 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4606999999999999e-05 2024-09-18T04:39:24.289 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.289 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.289 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:24.290 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552661+0000", 2024-09-18T04:39:24.290 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000146007 2024-09-18T04:39:24.290 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.290 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.290 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:24.290 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552876+0000", 2024-09-18T04:39:24.290 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000214618 2024-09-18T04:39:24.290 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.290 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.290 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:24.290 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.553061+0000", 2024-09-18T04:39:24.290 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018564100000000001 2024-09-18T04:39:24.290 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.290 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.291 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:24.291 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.565612+0000", 2024-09-18T04:39:24.291 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012550301999999999 2024-09-18T04:39:24.291 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.291 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.291 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:24.291 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577695+0000", 2024-09-18T04:39:24.291 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012083493000000001 2024-09-18T04:39:24.291 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.291 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.291 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:24.291 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577772+0000", 2024-09-18T04:39:24.292 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7158999999999998e-05 2024-09-18T04:39:24.292 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.292 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.292 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:24.292 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577834+0000", 2024-09-18T04:39:24.292 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1851000000000001e-05 2024-09-18T04:39:24.292 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.292 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:24.292 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.292 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.292 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.292 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:43 2.6 2:759f27b6:ns::rbd_data.133e813ab5bc.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:24.292 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.555868+0000", 2024-09-18T04:39:24.293 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 41.660992464000003, 2024-09-18T04:39:24.293 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.022332497999999999, 2024-09-18T04:39:24.293 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:24.293 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:24.293 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:24.293 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:24.293 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:24.293 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 43 2024-09-18T04:39:24.293 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.293 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:24.293 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.293 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:24.293 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555868+0000", 2024-09-18T04:39:24.294 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.294 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.294 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.294 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:24.294 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555868+0000", 2024-09-18T04:39:24.294 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.294 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.294 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.294 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:24.294 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555865+0000", 2024-09-18T04:39:24.294 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999971 2024-09-18T04:39:24.294 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.294 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.294 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:24.295 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561031+0000", 2024-09-18T04:39:24.295 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0051658770000000001 2024-09-18T04:39:24.295 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.295 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.295 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:24.295 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561048+0000", 2024-09-18T04:39:24.295 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6569e-05 2024-09-18T04:39:24.295 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.295 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.295 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:24.295 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561055+0000", 2024-09-18T04:39:24.295 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.328e-06 2024-09-18T04:39:24.295 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.295 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.296 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:24.296 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561114+0000", 2024-09-18T04:39:24.296 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9540000000000003e-05 2024-09-18T04:39:24.296 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.296 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.296 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:24.296 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561314+0000", 2024-09-18T04:39:24.296 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019965799999999999 2024-09-18T04:39:24.296 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.296 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.296 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:24.296 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561473+0000", 2024-09-18T04:39:24.296 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000158904 2024-09-18T04:39:24.297 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.297 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.297 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:24.297 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.567015+0000", 2024-09-18T04:39:24.297 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055420859999999999 2024-09-18T04:39:24.297 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.297 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.297 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:24.297 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577982+0000", 2024-09-18T04:39:24.297 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010966872000000001 2024-09-18T04:39:24.297 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.297 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.297 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:24.297 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578162+0000", 2024-09-18T04:39:24.298 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018050999999999999 2024-09-18T04:39:24.298 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.298 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.298 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:24.298 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578200+0000", 2024-09-18T04:39:24.298 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8000999999999998e-05 2024-09-18T04:39:24.298 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.298 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:24.298 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.298 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.298 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.298 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:48 2.6 2:7c225365:ns::rbd_data.133e813ab5bc.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:24.298 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:39:24.298 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 41.653193240999997, 2024-09-18T04:39:24.299 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.024576377, 2024-09-18T04:39:24.299 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:24.299 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:24.299 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:24.299 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:24.299 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:24.299 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 48 2024-09-18T04:39:24.299 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.299 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:24.299 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.299 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:24.299 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:39:24.299 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.299 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.300 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.300 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:24.300 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:39:24.300 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.300 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.300 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.300 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:24.300 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.563666+0000", 2024-09-18T04:39:24.300 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:24.300 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.300 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.300 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:24.300 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570331+0000", 2024-09-18T04:39:24.300 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0066653329999999998 2024-09-18T04:39:24.301 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.301 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.301 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:24.301 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570348+0000", 2024-09-18T04:39:24.301 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6507e-05 2024-09-18T04:39:24.301 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.301 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.301 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:24.301 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570355+0000", 2024-09-18T04:39:24.301 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2749999999999998e-06 2024-09-18T04:39:24.301 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.301 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.301 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:24.302 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570396+0000", 2024-09-18T04:39:24.302 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0809000000000002e-05 2024-09-18T04:39:24.302 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.302 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.302 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:24.302 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570571+0000", 2024-09-18T04:39:24.302 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000174823 2024-09-18T04:39:24.302 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.302 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.302 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:24.302 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570722+0000", 2024-09-18T04:39:24.302 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015154599999999999 2024-09-18T04:39:24.302 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.303 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.303 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:24.303 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579756+0000", 2024-09-18T04:39:24.303 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0090337000000000004 2024-09-18T04:39:24.303 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.303 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.303 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:24.303 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588093+0000", 2024-09-18T04:39:24.303 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.008336553 2024-09-18T04:39:24.303 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.303 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.303 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:24.303 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588173+0000", 2024-09-18T04:39:24.303 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0333999999999994e-05 2024-09-18T04:39:24.304 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.304 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.304 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:24.304 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588244+0000", 2024-09-18T04:39:24.304 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0573000000000003e-05 2024-09-18T04:39:24.304 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.304 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:24.304 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.304 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.304 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.304 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:49 2.4 2:20f0b0c9:ns::rbd_data.133e813ab5bc.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:24.304 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.570370+0000", 2024-09-18T04:39:24.305 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 41.646490913999997, 2024-09-18T04:39:24.305 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.029633981, 2024-09-18T04:39:24.305 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:24.305 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:24.305 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:24.305 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:24.305 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:24.305 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 49 2024-09-18T04:39:24.305 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.305 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:24.305 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.305 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:24.305 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570370+0000", 2024-09-18T04:39:24.305 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.306 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.306 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.306 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:24.306 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570370+0000", 2024-09-18T04:39:24.306 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.306 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.306 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.306 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:24.306 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570369+0000", 2024-09-18T04:39:24.306 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:24.306 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.306 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.306 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:24.306 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579090+0000", 2024-09-18T04:39:24.307 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0087212339999999996 2024-09-18T04:39:24.307 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.307 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.307 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:24.307 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579109+0000", 2024-09-18T04:39:24.307 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9120999999999999e-05 2024-09-18T04:39:24.307 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.307 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.307 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:24.307 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579123+0000", 2024-09-18T04:39:24.307 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4382000000000001e-05 2024-09-18T04:39:24.307 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.307 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.307 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:24.308 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579221+0000", 2024-09-18T04:39:24.308 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.7503999999999993e-05 2024-09-18T04:39:24.308 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.308 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.308 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:24.308 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579553+0000", 2024-09-18T04:39:24.308 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00033254200000000002 2024-09-18T04:39:24.308 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.308 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.308 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:24.308 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579817+0000", 2024-09-18T04:39:24.308 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00026317099999999998 2024-09-18T04:39:24.308 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.309 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.309 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:24.309 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.599905+0000", 2024-09-18T04:39:24.309 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020088575000000001 2024-09-18T04:39:24.309 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.309 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.309 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:24.309 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.599923+0000", 2024-09-18T04:39:24.309 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7425999999999999e-05 2024-09-18T04:39:24.309 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.309 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.309 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:24.309 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.599973+0000", 2024-09-18T04:39:24.310 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0649999999999998e-05 2024-09-18T04:39:24.310 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.310 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.310 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:24.310 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.600004+0000", 2024-09-18T04:39:24.310 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0361999999999998e-05 2024-09-18T04:39:24.310 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.310 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:24.310 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.310 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.310 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.310 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:50 2.1 e51/24 2:8c706374:ns::rbd_data.133e813ab5bc.000000000000000a:head v 51'8, mlcod=51'6)", 2024-09-18T04:39:24.310 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.577029+0000", 2024-09-18T04:39:24.311 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 41.639831543, 2024-09-18T04:39:24.311 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.017362855, 2024-09-18T04:39:24.311 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:24.311 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:24.311 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:24.311 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.311 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:24.311 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577029+0000", 2024-09-18T04:39:24.311 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.311 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.311 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.311 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:24.311 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577029+0000", 2024-09-18T04:39:24.311 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.312 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.312 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.312 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:24.312 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577027+0000", 2024-09-18T04:39:24.312 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:24.312 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.312 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.312 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:24.312 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578880+0000", 2024-09-18T04:39:24.312 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0018532080000000001 2024-09-18T04:39:24.312 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.312 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.312 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:24.312 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578906+0000", 2024-09-18T04:39:24.313 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5573999999999999e-05 2024-09-18T04:39:24.313 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.313 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.313 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:24.313 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578920+0000", 2024-09-18T04:39:24.313 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4483e-05 2024-09-18T04:39:24.313 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.313 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.313 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:24.313 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578983+0000", 2024-09-18T04:39:24.313 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2248000000000004e-05 2024-09-18T04:39:24.313 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.313 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.314 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:24.314 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579016+0000", 2024-09-18T04:39:24.314 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3364000000000001e-05 2024-09-18T04:39:24.314 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.314 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.314 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:24.314 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594342+0000", 2024-09-18T04:39:24.314 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01532564 2024-09-18T04:39:24.314 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.314 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.314 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:24.314 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594392+0000", 2024-09-18T04:39:24.314 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0068999999999998e-05 2024-09-18T04:39:24.314 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.315 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:24.315 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.315 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.315 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.315 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:45 2.0 e51/17 2:0fbfa3dc:ns::rbd_data.133e813ab5bc.0000000000000007:head v 51'24, mlcod=51'22)", 2024-09-18T04:39:24.315 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.581892+0000", 2024-09-18T04:39:24.315 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 41.634968870999998, 2024-09-18T04:39:24.315 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015646006, 2024-09-18T04:39:24.315 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:24.315 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:24.315 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:24.315 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.315 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:24.315 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581892+0000", 2024-09-18T04:39:24.316 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.316 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.316 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.316 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:24.316 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581892+0000", 2024-09-18T04:39:24.316 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.316 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.316 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.316 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:24.316 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581890+0000", 2024-09-18T04:39:24.316 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:24.316 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.316 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.317 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:24.317 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586188+0000", 2024-09-18T04:39:24.317 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0042985769999999996 2024-09-18T04:39:24.317 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.317 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.317 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:24.317 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586210+0000", 2024-09-18T04:39:24.317 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1069000000000001e-05 2024-09-18T04:39:24.317 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.317 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.317 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:24.317 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586221+0000", 2024-09-18T04:39:24.317 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1287000000000001e-05 2024-09-18T04:39:24.317 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.318 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.318 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:24.318 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586313+0000", 2024-09-18T04:39:24.318 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1935999999999995e-05 2024-09-18T04:39:24.318 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.318 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.318 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:24.318 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586368+0000", 2024-09-18T04:39:24.318 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5578000000000002e-05 2024-09-18T04:39:24.318 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.318 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.318 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:24.318 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.597473+0000", 2024-09-18T04:39:24.318 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011105139999999999 2024-09-18T04:39:24.319 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.319 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.319 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:24.319 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.597538+0000", 2024-09-18T04:39:24.319 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4133000000000004e-05 2024-09-18T04:39:24.319 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.319 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:24.319 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.319 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.319 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.319 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:53 2.4 2:39f7f174:ns::rbd_data.133e813ab5bc.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:24.319 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.584263+0000", 2024-09-18T04:39:24.319 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 41.632596997, 2024-09-18T04:39:24.320 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018701180000000001, 2024-09-18T04:39:24.320 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:24.320 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:24.320 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:24.320 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:24.320 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:24.320 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 53 2024-09-18T04:39:24.320 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.320 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:24.320 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.320 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:24.320 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584263+0000", 2024-09-18T04:39:24.320 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.321 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.321 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.321 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:24.321 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584263+0000", 2024-09-18T04:39:24.321 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.321 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.321 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.321 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:24.321 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584262+0000", 2024-09-18T04:39:24.321 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:24.321 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.321 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.321 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:24.322 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.591617+0000", 2024-09-18T04:39:24.322 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0073550600000000001 2024-09-18T04:39:24.322 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.322 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.322 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:24.322 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.591637+0000", 2024-09-18T04:39:24.322 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0432999999999999e-05 2024-09-18T04:39:24.322 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.322 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.322 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:24.322 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.591646+0000", 2024-09-18T04:39:24.322 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7849999999999997e-06 2024-09-18T04:39:24.322 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.322 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.323 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:24.323 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593478+0000", 2024-09-18T04:39:24.323 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001831495 2024-09-18T04:39:24.323 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.323 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.323 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:24.323 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593661+0000", 2024-09-18T04:39:24.323 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018364999999999999 2024-09-18T04:39:24.323 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.323 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.323 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:24.323 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593834+0000", 2024-09-18T04:39:24.323 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000173065 2024-09-18T04:39:24.324 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.324 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.324 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:24.324 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.600670+0000", 2024-09-18T04:39:24.324 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0068354970000000003 2024-09-18T04:39:24.324 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.324 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.324 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:24.324 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602851+0000", 2024-09-18T04:39:24.324 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002181653 2024-09-18T04:39:24.324 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.324 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.325 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:24.325 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602930+0000", 2024-09-18T04:39:24.325 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8045000000000002e-05 2024-09-18T04:39:24.325 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.325 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.325 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:24.325 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602965+0000", 2024-09-18T04:39:24.325 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5102999999999999e-05 2024-09-18T04:39:24.325 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.325 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:24.325 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.325 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.325 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.326 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:56 2.6 2:72453c3e:ns::rbd_data.133e813ab5bc.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:24.326 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.595988+0000", 2024-09-18T04:39:24.326 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 41.620872595000002, 2024-09-18T04:39:24.326 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.030296172, 2024-09-18T04:39:24.326 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:24.326 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:24.326 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:24.326 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:24.326 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:24.326 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 56 2024-09-18T04:39:24.326 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.326 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:24.326 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.326 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:24.326 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.595988+0000", 2024-09-18T04:39:24.327 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.327 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.327 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.327 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:24.327 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.595988+0000", 2024-09-18T04:39:24.327 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.327 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.327 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.327 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:24.327 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.595986+0000", 2024-09-18T04:39:24.327 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:24.327 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.327 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.328 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:24.328 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605262+0000", 2024-09-18T04:39:24.328 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0092767240000000001 2024-09-18T04:39:24.328 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.328 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.328 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:24.328 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605282+0000", 2024-09-18T04:39:24.328 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9307e-05 2024-09-18T04:39:24.328 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.328 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.328 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:24.328 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605292+0000", 2024-09-18T04:39:24.328 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9510000000000001e-06 2024-09-18T04:39:24.328 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.329 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.329 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:24.329 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605369+0000", 2024-09-18T04:39:24.329 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7846e-05 2024-09-18T04:39:24.329 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.329 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.329 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:24.329 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605668+0000", 2024-09-18T04:39:24.329 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00029896099999999999 2024-09-18T04:39:24.329 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.329 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.329 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:24.329 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605906+0000", 2024-09-18T04:39:24.329 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000237758 2024-09-18T04:39:24.330 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.330 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.330 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:24.330 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.613532+0000", 2024-09-18T04:39:24.330 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0076260590000000001 2024-09-18T04:39:24.330 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.330 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.330 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:24.330 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.626116+0000", 2024-09-18T04:39:24.330 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012584092999999999 2024-09-18T04:39:24.330 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.330 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.330 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:24.330 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.626216+0000", 2024-09-18T04:39:24.331 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9630999999999995e-05 2024-09-18T04:39:24.331 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.331 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.331 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:24.331 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.626284+0000", 2024-09-18T04:39:24.331 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8034999999999995e-05 2024-09-18T04:39:24.331 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.331 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:24.331 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.331 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.331 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:24.331 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:24.333 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:39:24.333 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:24.328+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:39:24.434 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:24.429+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:24.493 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:24.493 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:24.493 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:39:24.493 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:24.503 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:39:24.535 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:24.532+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:24.609 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-09-18T04:39:24.610 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:24.610 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:24.610 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:39:24.610 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:39:24.610 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:24.620 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:39:24.636 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:24.629+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:24.726 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:24.727 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:39:24.727 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:39:24.727 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:39:24.727 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.727 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:33 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:24.727 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.261182+0000", 2024-09-18T04:39:24.727 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 42.460100156999999, 2024-09-18T04:39:24.727 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.037082208999999998, 2024-09-18T04:39:24.727 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:24.728 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:24.728 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:24.728 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:24.728 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:24.728 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 33 2024-09-18T04:39:24.728 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.728 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:24.728 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.728 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:24.728 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.261182+0000", 2024-09-18T04:39:24.728 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.728 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.729 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.729 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:24.729 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.261182+0000", 2024-09-18T04:39:24.729 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.729 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.729 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.729 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:24.729 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.261180+0000", 2024-09-18T04:39:24.729 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:24.729 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.729 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.729 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:24.729 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278669+0000", 2024-09-18T04:39:24.730 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.017488877999999999 2024-09-18T04:39:24.730 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.730 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.730 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:24.730 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278689+0000", 2024-09-18T04:39:24.730 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0047999999999998e-05 2024-09-18T04:39:24.730 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.730 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.730 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:24.730 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278699+0000", 2024-09-18T04:39:24.730 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6800000000000005e-06 2024-09-18T04:39:24.730 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.730 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.730 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:24.731 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278775+0000", 2024-09-18T04:39:24.731 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6087999999999994e-05 2024-09-18T04:39:24.731 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.731 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.731 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:24.731 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278935+0000", 2024-09-18T04:39:24.731 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000159834 2024-09-18T04:39:24.731 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.731 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.731 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:24.731 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279064+0000", 2024-09-18T04:39:24.731 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012954199999999999 2024-09-18T04:39:24.731 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.732 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.732 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:24.732 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295562+0000", 2024-09-18T04:39:24.732 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016497583999999999 2024-09-18T04:39:24.732 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.732 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.732 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:24.732 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.298147+0000", 2024-09-18T04:39:24.732 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002584982 2024-09-18T04:39:24.732 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.732 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.732 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:24.732 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.298217+0000", 2024-09-18T04:39:24.733 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0400000000000004e-05 2024-09-18T04:39:24.733 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.733 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.733 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:24.733 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.298265+0000", 2024-09-18T04:39:24.733 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7580000000000002e-05 2024-09-18T04:39:24.733 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.733 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:24.733 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.733 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.733 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.733 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:34 2.3 e51/24 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head v 51'3, mlcod=18'1)", 2024-09-18T04:39:24.733 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.273067+0000", 2024-09-18T04:39:24.733 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 42.448215367000003, 2024-09-18T04:39:24.734 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.021784103999999999, 2024-09-18T04:39:24.734 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:24.734 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:24.734 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:24.734 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.734 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:24.734 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.273067+0000", 2024-09-18T04:39:24.734 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.734 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.734 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.734 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:24.734 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.273067+0000", 2024-09-18T04:39:24.734 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.735 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.735 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.735 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:24.735 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.273060+0000", 2024-09-18T04:39:24.735 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999924 2024-09-18T04:39:24.735 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.735 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.735 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:24.735 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278596+0000", 2024-09-18T04:39:24.735 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055365839999999998 2024-09-18T04:39:24.735 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.735 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.735 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:24.735 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278637+0000", 2024-09-18T04:39:24.736 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0596000000000002e-05 2024-09-18T04:39:24.736 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.736 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.736 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:24.736 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278653+0000", 2024-09-18T04:39:24.736 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6626999999999999e-05 2024-09-18T04:39:24.736 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.736 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.736 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:24.736 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278748+0000", 2024-09-18T04:39:24.737 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4483999999999999e-05 2024-09-18T04:39:24.737 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.737 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.737 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:24.737 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278799+0000", 2024-09-18T04:39:24.737 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1484999999999999e-05 2024-09-18T04:39:24.737 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.737 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.737 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:24.737 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.294785+0000", 2024-09-18T04:39:24.737 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015985256999999999 2024-09-18T04:39:24.737 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.737 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.737 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:24.738 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.294851+0000", 2024-09-18T04:39:24.738 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6728000000000002e-05 2024-09-18T04:39:24.738 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.738 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:24.738 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.738 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.738 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.738 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:37 2.5 2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:24.738 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.278722+0000", 2024-09-18T04:39:24.738 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 42.442560317999998, 2024-09-18T04:39:24.738 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.026323801000000001, 2024-09-18T04:39:24.738 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:24.738 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:24.738 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:24.739 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:24.739 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:24.739 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 37 2024-09-18T04:39:24.739 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.739 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:24.739 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.739 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:24.739 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278722+0000", 2024-09-18T04:39:24.739 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.739 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.739 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.739 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:24.739 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278722+0000", 2024-09-18T04:39:24.739 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.740 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.740 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.740 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:24.740 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.278720+0000", 2024-09-18T04:39:24.740 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:24.740 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.740 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.740 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:24.740 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.291875+0000", 2024-09-18T04:39:24.740 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013154414 2024-09-18T04:39:24.740 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.740 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.741 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:24.741 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.291901+0000", 2024-09-18T04:39:24.741 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6339999999999999e-05 2024-09-18T04:39:24.741 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.741 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.741 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:24.741 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.291914+0000", 2024-09-18T04:39:24.741 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3043e-05 2024-09-18T04:39:24.741 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.741 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.741 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:24.741 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.291975+0000", 2024-09-18T04:39:24.741 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0949999999999998e-05 2024-09-18T04:39:24.741 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.742 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.742 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:24.742 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292173+0000", 2024-09-18T04:39:24.742 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019825499999999999 2024-09-18T04:39:24.742 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.742 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.742 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:24.742 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292300+0000", 2024-09-18T04:39:24.742 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000126616 2024-09-18T04:39:24.742 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.742 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.742 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:24.742 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304334+0000", 2024-09-18T04:39:24.742 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012033934 2024-09-18T04:39:24.743 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.743 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.743 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:24.743 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304901+0000", 2024-09-18T04:39:24.743 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00056694099999999995 2024-09-18T04:39:24.743 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.743 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.743 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:24.743 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304981+0000", 2024-09-18T04:39:24.743 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9974000000000004e-05 2024-09-18T04:39:24.743 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.743 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.743 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:24.743 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.305046+0000", 2024-09-18T04:39:24.744 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5184000000000005e-05 2024-09-18T04:39:24.744 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.744 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:24.744 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.744 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.744 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.744 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:49 2.5 2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:24.744 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.314271+0000", 2024-09-18T04:39:24.744 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 42.407011716, 2024-09-18T04:39:24.744 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.029779953000000001, 2024-09-18T04:39:24.744 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:24.744 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:24.744 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:24.744 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:24.745 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:24.745 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 49 2024-09-18T04:39:24.745 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.745 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:24.745 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.745 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:24.745 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.314271+0000", 2024-09-18T04:39:24.745 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.745 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.745 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.745 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:24.745 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.314271+0000", 2024-09-18T04:39:24.745 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.746 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.746 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.746 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:24.746 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.314269+0000", 2024-09-18T04:39:24.746 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:24.746 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.746 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.746 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:24.746 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.324780+0000", 2024-09-18T04:39:24.746 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010511609999999999 2024-09-18T04:39:24.746 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.746 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.746 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:24.746 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.324821+0000", 2024-09-18T04:39:24.747 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0899000000000003e-05 2024-09-18T04:39:24.747 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.747 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.747 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:24.747 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.324840+0000", 2024-09-18T04:39:24.747 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8947e-05 2024-09-18T04:39:24.747 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.747 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.747 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:24.747 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.324936+0000", 2024-09-18T04:39:24.747 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6342999999999996e-05 2024-09-18T04:39:24.747 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.747 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.747 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:24.748 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325284+0000", 2024-09-18T04:39:24.748 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000347146 2024-09-18T04:39:24.748 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.748 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.748 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:24.748 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325586+0000", 2024-09-18T04:39:24.748 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00030199799999999999 2024-09-18T04:39:24.748 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.748 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.748 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:24.748 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343710+0000", 2024-09-18T04:39:24.748 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018124583999999999 2024-09-18T04:39:24.748 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.749 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.749 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:24.749 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343991+0000", 2024-09-18T04:39:24.749 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00028085600000000001 2024-09-18T04:39:24.749 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.749 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.749 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:24.749 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344028+0000", 2024-09-18T04:39:24.749 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6637000000000002e-05 2024-09-18T04:39:24.749 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.749 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.749 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:24.749 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344051+0000", 2024-09-18T04:39:24.750 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3260000000000001e-05 2024-09-18T04:39:24.750 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.750 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:24.750 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.750 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.750 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.750 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:50 2.3 e51/24 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head v 51'4, mlcod=51'2)", 2024-09-18T04:39:24.750 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.323066+0000", 2024-09-18T04:39:24.750 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 42.398216323, 2024-09-18T04:39:24.750 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.022135774, 2024-09-18T04:39:24.750 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:24.750 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:24.750 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:24.750 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.751 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:24.751 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.323066+0000", 2024-09-18T04:39:24.751 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.751 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.751 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.751 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:24.751 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.323066+0000", 2024-09-18T04:39:24.751 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.751 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.751 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.751 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:24.752 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.323065+0000", 2024-09-18T04:39:24.752 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:24.752 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.752 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.752 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:24.752 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.328664+0000", 2024-09-18T04:39:24.752 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055993600000000003 2024-09-18T04:39:24.752 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.752 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.752 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:24.753 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.328692+0000", 2024-09-18T04:39:24.753 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7654e-05 2024-09-18T04:39:24.753 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.753 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.753 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:24.753 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.328706+0000", 2024-09-18T04:39:24.753 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4097000000000001e-05 2024-09-18T04:39:24.753 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.753 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.753 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:24.753 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.328816+0000", 2024-09-18T04:39:24.753 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000110281 2024-09-18T04:39:24.753 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.753 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.754 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:24.754 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.328848+0000", 2024-09-18T04:39:24.754 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1837e-05 2024-09-18T04:39:24.754 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.754 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.754 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:24.754 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345160+0000", 2024-09-18T04:39:24.755 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016312099 2024-09-18T04:39:24.755 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.755 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.755 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:24.755 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345202+0000", 2024-09-18T04:39:24.755 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1857999999999998e-05 2024-09-18T04:39:24.755 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.755 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:24.755 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.755 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.756 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.756 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:58 2.5 2:b227e6b1:::rbd_data.125fc09df6d1.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:24.756 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.351564+0000", 2024-09-18T04:39:24.756 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 42.369718871000003, 2024-09-18T04:39:24.756 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.032125956999999997, 2024-09-18T04:39:24.756 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:24.756 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:24.756 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:24.756 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:24.756 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:24.757 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 58 2024-09-18T04:39:24.757 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.757 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:24.757 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.757 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:24.757 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.351564+0000", 2024-09-18T04:39:24.757 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.758 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.758 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.758 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:24.758 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.351564+0000", 2024-09-18T04:39:24.758 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.758 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.758 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.758 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:24.758 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.351561+0000", 2024-09-18T04:39:24.758 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:24.758 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.759 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.759 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:24.759 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364390+0000", 2024-09-18T04:39:24.759 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012829063999999999 2024-09-18T04:39:24.759 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.759 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.759 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:24.760 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364418+0000", 2024-09-18T04:39:24.760 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7307000000000001e-05 2024-09-18T04:39:24.760 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.760 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.760 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:24.761 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364428+0000", 2024-09-18T04:39:24.761 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0675e-05 2024-09-18T04:39:24.761 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.761 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.762 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:24.762 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364523+0000", 2024-09-18T04:39:24.762 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4698999999999996e-05 2024-09-18T04:39:24.762 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.762 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.762 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:24.762 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364785+0000", 2024-09-18T04:39:24.762 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00026170099999999998 2024-09-18T04:39:24.762 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.762 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.762 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:24.763 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.365039+0000", 2024-09-18T04:39:24.763 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00025462299999999999 2024-09-18T04:39:24.763 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.763 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.763 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:24.763 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383143+0000", 2024-09-18T04:39:24.763 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018103768999999999 2024-09-18T04:39:24.763 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.763 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.763 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:24.763 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383582+0000", 2024-09-18T04:39:24.763 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00043860600000000002 2024-09-18T04:39:24.763 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.764 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.764 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:24.764 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383644+0000", 2024-09-18T04:39:24.764 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2682999999999996e-05 2024-09-18T04:39:24.764 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.764 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.764 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:24.764 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383690+0000", 2024-09-18T04:39:24.764 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5247999999999998e-05 2024-09-18T04:39:24.764 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.764 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:24.764 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.764 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.765 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.765 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:33 2.1 2:8b40f1c5:ns::rbd_data.133e813ab5bc.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:24.765 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.523765+0000", 2024-09-18T04:39:24.765 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 42.197517830999999, 2024-09-18T04:39:24.765 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027472973000000001, 2024-09-18T04:39:24.765 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:24.765 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:24.765 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:24.765 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:24.765 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:24.765 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 33 2024-09-18T04:39:24.765 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.765 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:24.766 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.766 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:24.766 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.523765+0000", 2024-09-18T04:39:24.766 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.766 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.766 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.766 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:24.766 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.523765+0000", 2024-09-18T04:39:24.766 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.766 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.766 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.766 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:24.766 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.523762+0000", 2024-09-18T04:39:24.766 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:24.767 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.767 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.767 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:24.767 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531127+0000", 2024-09-18T04:39:24.767 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0073644560000000001 2024-09-18T04:39:24.767 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.767 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.767 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:24.767 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531189+0000", 2024-09-18T04:39:24.767 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2315000000000002e-05 2024-09-18T04:39:24.767 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.767 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.767 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:24.767 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531211+0000", 2024-09-18T04:39:24.768 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1542000000000001e-05 2024-09-18T04:39:24.768 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.768 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.768 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:24.768 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531314+0000", 2024-09-18T04:39:24.768 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000103582 2024-09-18T04:39:24.768 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.768 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.768 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:24.768 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531493+0000", 2024-09-18T04:39:24.768 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000178433 2024-09-18T04:39:24.768 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.768 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.768 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:24.769 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531653+0000", 2024-09-18T04:39:24.769 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000160388 2024-09-18T04:39:24.769 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.769 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.769 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:24.769 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.547345+0000", 2024-09-18T04:39:24.769 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015691742000000002 2024-09-18T04:39:24.769 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.769 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.769 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:24.769 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551076+0000", 2024-09-18T04:39:24.769 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003730957 2024-09-18T04:39:24.769 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.769 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.770 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:24.770 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551176+0000", 2024-09-18T04:39:24.770 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010002599999999999 2024-09-18T04:39:24.770 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.770 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.770 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:24.770 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551238+0000", 2024-09-18T04:39:24.770 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1945000000000004e-05 2024-09-18T04:39:24.770 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.770 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:24.770 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.771 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.771 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.771 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:36 2.5 2:bd5b8dd8:ns::rbd_data.133e813ab5bc.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:24.771 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.531253+0000", 2024-09-18T04:39:24.771 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 42.190029522000003, 2024-09-18T04:39:24.771 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027295212999999999, 2024-09-18T04:39:24.771 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:24.771 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:24.771 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:24.771 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:24.771 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:24.771 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 36 2024-09-18T04:39:24.771 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.771 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:24.772 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.772 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:24.772 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531253+0000", 2024-09-18T04:39:24.772 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.772 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.772 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.772 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:24.772 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531253+0000", 2024-09-18T04:39:24.772 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.772 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.772 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.772 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:24.772 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531251+0000", 2024-09-18T04:39:24.772 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:24.773 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.773 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.773 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:24.773 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536540+0000", 2024-09-18T04:39:24.773 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0052885839999999998 2024-09-18T04:39:24.773 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.773 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.773 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:24.773 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536604+0000", 2024-09-18T04:39:24.773 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4824000000000001e-05 2024-09-18T04:39:24.773 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.773 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.773 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:24.774 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536621+0000", 2024-09-18T04:39:24.774 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6215999999999999e-05 2024-09-18T04:39:24.774 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.774 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.774 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:24.774 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536768+0000", 2024-09-18T04:39:24.774 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014767900000000001 2024-09-18T04:39:24.774 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.774 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.774 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:24.774 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537084+0000", 2024-09-18T04:39:24.774 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00031553499999999998 2024-09-18T04:39:24.774 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.775 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.775 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:24.775 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537300+0000", 2024-09-18T04:39:24.775 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000216145 2024-09-18T04:39:24.775 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.775 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.775 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:24.775 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555326+0000", 2024-09-18T04:39:24.775 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018025861000000001 2024-09-18T04:39:24.775 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.775 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.775 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:24.775 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558449+0000", 2024-09-18T04:39:24.776 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0031229750000000001 2024-09-18T04:39:24.776 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.776 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.776 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:24.776 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558502+0000", 2024-09-18T04:39:24.776 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3572000000000001e-05 2024-09-18T04:39:24.776 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.776 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.776 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:24.776 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558548+0000", 2024-09-18T04:39:24.776 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5921999999999998e-05 2024-09-18T04:39:24.776 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.776 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:24.777 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.777 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.777 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.777 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:37 2.5 2:bf39b8e4:ns::rbd_data.133e813ab5bc.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:24.777 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.536664+0000", 2024-09-18T04:39:24.777 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 42.184618106000002, 2024-09-18T04:39:24.777 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.026331765, 2024-09-18T04:39:24.777 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:24.777 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:24.777 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:24.777 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:24.777 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:24.777 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 37 2024-09-18T04:39:24.777 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.778 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:24.778 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.778 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:24.778 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536664+0000", 2024-09-18T04:39:24.778 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.778 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.778 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.778 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:24.778 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536664+0000", 2024-09-18T04:39:24.778 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.778 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.778 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.778 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:24.779 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.536662+0000", 2024-09-18T04:39:24.779 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:24.779 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.779 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.779 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:24.779 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.543699+0000", 2024-09-18T04:39:24.779 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0070367119999999997 2024-09-18T04:39:24.779 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.779 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.779 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:24.779 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.543749+0000", 2024-09-18T04:39:24.779 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0476999999999998e-05 2024-09-18T04:39:24.779 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.779 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.780 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:24.780 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.543800+0000", 2024-09-18T04:39:24.780 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0822999999999998e-05 2024-09-18T04:39:24.780 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.780 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.780 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:24.780 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.547868+0000", 2024-09-18T04:39:24.780 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0040679339999999996 2024-09-18T04:39:24.780 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.780 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.780 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:24.780 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548087+0000", 2024-09-18T04:39:24.780 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000218994 2024-09-18T04:39:24.780 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.781 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.781 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:24.781 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548383+0000", 2024-09-18T04:39:24.781 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000295588 2024-09-18T04:39:24.781 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.781 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.781 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:24.781 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562622+0000", 2024-09-18T04:39:24.781 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014239915000000001 2024-09-18T04:39:24.781 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.781 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.781 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:24.781 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562924+0000", 2024-09-18T04:39:24.781 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00030127600000000001 2024-09-18T04:39:24.782 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.782 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.782 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:24.782 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562957+0000", 2024-09-18T04:39:24.782 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3365999999999999e-05 2024-09-18T04:39:24.782 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.782 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.782 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:24.782 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562996+0000", 2024-09-18T04:39:24.782 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9153999999999998e-05 2024-09-18T04:39:24.782 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.782 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:24.782 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.782 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.783 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.783 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:50 2.1 2:8c706374:ns::rbd_data.133e813ab5bc.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:24.783 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.570513+0000", 2024-09-18T04:39:24.783 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 42.150769824000001, 2024-09-18T04:39:24.783 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.024182995999999998, 2024-09-18T04:39:24.783 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:24.783 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:24.783 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:24.783 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:24.783 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:24.783 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 50 2024-09-18T04:39:24.783 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.783 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:24.783 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.784 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:24.784 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570513+0000", 2024-09-18T04:39:24.784 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.784 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.784 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.784 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:24.784 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570513+0000", 2024-09-18T04:39:24.785 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.785 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.785 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.785 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:24.785 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570511+0000", 2024-09-18T04:39:24.785 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:24.785 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.785 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.785 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:24.785 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.576327+0000", 2024-09-18T04:39:24.785 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0058162530000000004 2024-09-18T04:39:24.785 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.785 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.785 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:24.786 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.576355+0000", 2024-09-18T04:39:24.786 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8030000000000001e-05 2024-09-18T04:39:24.786 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.786 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.786 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:24.786 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.576366+0000", 2024-09-18T04:39:24.786 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1139e-05 2024-09-18T04:39:24.786 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.786 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.786 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:24.786 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.576458+0000", 2024-09-18T04:39:24.787 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1780999999999994e-05 2024-09-18T04:39:24.787 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.787 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.787 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:24.787 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.576701+0000", 2024-09-18T04:39:24.787 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00024303899999999999 2024-09-18T04:39:24.787 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.787 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.787 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:24.788 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.576876+0000", 2024-09-18T04:39:24.788 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000174637 2024-09-18T04:39:24.788 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.788 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.788 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:24.788 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588432+0000", 2024-09-18T04:39:24.788 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011555943000000001 2024-09-18T04:39:24.788 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.788 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.788 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:24.788 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594573+0000", 2024-09-18T04:39:24.788 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0061408670000000004 2024-09-18T04:39:24.789 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.789 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.789 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:24.789 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594640+0000", 2024-09-18T04:39:24.789 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7404999999999996e-05 2024-09-18T04:39:24.789 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.789 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.789 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:24.790 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594696+0000", 2024-09-18T04:39:24.790 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5612999999999998e-05 2024-09-18T04:39:24.790 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.790 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:24.790 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.790 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.790 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.790 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:60 2.5 2:a06f51e4:ns::rbd_data.133e813ab5bc.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:24.790 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.604753+0000", 2024-09-18T04:39:24.791 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 42.116529704999998, 2024-09-18T04:39:24.791 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014400181, 2024-09-18T04:39:24.791 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:24.791 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:24.791 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:24.791 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:24.791 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:24.791 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 60 2024-09-18T04:39:24.791 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.791 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:24.791 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.792 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:24.792 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.604753+0000", 2024-09-18T04:39:24.792 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.792 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.792 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.792 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:24.792 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.604753+0000", 2024-09-18T04:39:24.793 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.793 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.793 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.793 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:24.793 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.604751+0000", 2024-09-18T04:39:24.793 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:24.793 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.793 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.793 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:24.794 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.608980+0000", 2024-09-18T04:39:24.794 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0042292420000000002 2024-09-18T04:39:24.794 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.794 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.794 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:24.794 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609014+0000", 2024-09-18T04:39:24.794 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3572999999999999e-05 2024-09-18T04:39:24.794 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.794 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.795 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:24.795 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609032+0000", 2024-09-18T04:39:24.795 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8150000000000001e-05 2024-09-18T04:39:24.795 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.795 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.795 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:24.796 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609090+0000", 2024-09-18T04:39:24.796 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8214999999999997e-05 2024-09-18T04:39:24.796 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.796 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.796 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:24.796 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609235+0000", 2024-09-18T04:39:24.796 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014437799999999999 2024-09-18T04:39:24.797 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.797 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.797 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:24.797 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609360+0000", 2024-09-18T04:39:24.797 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012526099999999999 2024-09-18T04:39:24.797 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.797 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.797 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:24.797 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.614952+0000", 2024-09-18T04:39:24.797 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055922300000000001 2024-09-18T04:39:24.797 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.797 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.797 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:24.798 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.619070+0000", 2024-09-18T04:39:24.798 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0041178250000000003 2024-09-18T04:39:24.798 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.798 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.798 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:24.798 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.619117+0000", 2024-09-18T04:39:24.798 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7275999999999999e-05 2024-09-18T04:39:24.798 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.798 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.798 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:24.798 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.619153+0000", 2024-09-18T04:39:24.798 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5970000000000003e-05 2024-09-18T04:39:24.798 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.798 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:24.799 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.799 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.799 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.799 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4972.0:20 2.5 2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:24.799 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.949296+0000", 2024-09-18T04:39:24.799 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 32.771987101999997, 2024-09-18T04:39:24.799 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014709732, 2024-09-18T04:39:24.799 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:24.799 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:24.799 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:24.799 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4972", 2024-09-18T04:39:24.799 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2280251472", 2024-09-18T04:39:24.799 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 20 2024-09-18T04:39:24.799 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.800 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:24.800 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.800 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:24.800 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949296+0000", 2024-09-18T04:39:24.800 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.800 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.800 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.800 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:24.800 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949296+0000", 2024-09-18T04:39:24.800 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.800 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.800 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.800 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:24.801 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949295+0000", 2024-09-18T04:39:24.801 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:24.801 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.801 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.801 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:24.801 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949301+0000", 2024-09-18T04:39:24.801 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4829999999999996e-06 2024-09-18T04:39:24.801 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.801 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.801 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:24.801 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949317+0000", 2024-09-18T04:39:24.801 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5866999999999999e-05 2024-09-18T04:39:24.802 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.802 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.802 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:24.802 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.949331+0000", 2024-09-18T04:39:24.802 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3485999999999999e-05 2024-09-18T04:39:24.802 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.802 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.802 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:24.802 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.959137+0000", 2024-09-18T04:39:24.802 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0098066520000000008 2024-09-18T04:39:24.802 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.802 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.803 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:24.803 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.959251+0000", 2024-09-18T04:39:24.803 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000113783 2024-09-18T04:39:24.803 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.803 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.803 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:24.803 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.964005+0000", 2024-09-18T04:39:24.803 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0047542080000000002 2024-09-18T04:39:24.803 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.803 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:24.803 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.803 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.803 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.804 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.5246.0:96 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [write 524288~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-18T04:39:24.804 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:15.279086+0000", 2024-09-18T04:39:24.804 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 9.4421967510000009, 2024-09-18T04:39:24.804 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015391526000000001, 2024-09-18T04:39:24.804 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:24.804 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:24.804 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:24.804 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.5246", 2024-09-18T04:39:24.804 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2208214885", 2024-09-18T04:39:24.804 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 96 2024-09-18T04:39:24.804 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.804 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:24.804 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.805 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:24.805 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.279086+0000", 2024-09-18T04:39:24.805 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.805 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.805 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.805 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:24.805 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.279086+0000", 2024-09-18T04:39:24.805 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.805 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.805 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.805 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:24.805 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.279085+0000", 2024-09-18T04:39:24.806 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:24.806 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.806 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.806 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:24.806 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.279233+0000", 2024-09-18T04:39:24.806 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014805100000000001 2024-09-18T04:39:24.806 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.806 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.806 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:24.806 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.279254+0000", 2024-09-18T04:39:24.806 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1645e-05 2024-09-18T04:39:24.806 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.806 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.807 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:24.807 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.279270+0000", 2024-09-18T04:39:24.807 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5909e-05 2024-09-18T04:39:24.807 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.807 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.807 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:24.807 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.289235+0000", 2024-09-18T04:39:24.807 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0099644550000000005 2024-09-18T04:39:24.807 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.807 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.807 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:24.807 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.289280+0000", 2024-09-18T04:39:24.807 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5772999999999997e-05 2024-09-18T04:39:24.808 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.808 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.808 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:24.808 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.289355+0000", 2024-09-18T04:39:24.808 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4821999999999996e-05 2024-09-18T04:39:24.808 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.808 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.808 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:24.808 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.291704+0000", 2024-09-18T04:39:24.808 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023489460000000002 2024-09-18T04:39:24.808 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.808 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.808 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:24.809 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.294362+0000", 2024-09-18T04:39:24.809 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026582160000000001 2024-09-18T04:39:24.809 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.809 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.809 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:24.809 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.294428+0000", 2024-09-18T04:39:24.809 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5203999999999994e-05 2024-09-18T04:39:24.809 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.809 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.809 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:24.809 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.294477+0000", 2024-09-18T04:39:24.809 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9805000000000002e-05 2024-09-18T04:39:24.809 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.809 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:24.810 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.810 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.810 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.810 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.5246.0:97 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [write 655360~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-18T04:39:24.810 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:15.279285+0000", 2024-09-18T04:39:24.810 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 9.4419977730000006, 2024-09-18T04:39:24.810 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015374402000000001, 2024-09-18T04:39:24.810 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:24.810 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:24.810 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:24.810 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.5246", 2024-09-18T04:39:24.810 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2208214885", 2024-09-18T04:39:24.810 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 97 2024-09-18T04:39:24.810 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.811 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:24.811 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.811 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:24.811 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.279285+0000", 2024-09-18T04:39:24.811 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.811 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.811 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.811 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:24.811 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.279285+0000", 2024-09-18T04:39:24.811 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.811 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.811 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.811 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:24.811 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.279284+0000", 2024-09-18T04:39:24.812 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:24.812 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.812 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.812 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:24.812 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.279407+0000", 2024-09-18T04:39:24.812 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000123812 2024-09-18T04:39:24.812 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.812 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.812 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:24.812 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.279432+0000", 2024-09-18T04:39:24.812 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4212000000000001e-05 2024-09-18T04:39:24.812 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.812 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.812 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:24.813 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.279447+0000", 2024-09-18T04:39:24.813 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5753e-05 2024-09-18T04:39:24.813 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.813 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.813 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:24.813 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.290145+0000", 2024-09-18T04:39:24.813 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010697312 2024-09-18T04:39:24.813 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.813 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.813 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:24.813 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.290213+0000", 2024-09-18T04:39:24.813 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8598999999999998e-05 2024-09-18T04:39:24.813 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.813 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.814 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:24.814 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.290308+0000", 2024-09-18T04:39:24.814 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4449000000000003e-05 2024-09-18T04:39:24.814 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.814 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.814 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:24.814 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.294275+0000", 2024-09-18T04:39:24.814 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0039673260000000002 2024-09-18T04:39:24.814 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.814 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.814 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:24.814 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.294546+0000", 2024-09-18T04:39:24.814 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00027124199999999999 2024-09-18T04:39:24.815 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.815 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.815 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:24.815 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.294615+0000", 2024-09-18T04:39:24.815 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8595000000000003e-05 2024-09-18T04:39:24.815 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.815 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.815 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:24.815 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.294659+0000", 2024-09-18T04:39:24.815 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4314999999999998e-05 2024-09-18T04:39:24.815 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.815 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:24.815 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.816 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.816 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.816 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.5246.0:98 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [write 786432~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-18T04:39:24.816 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:15.279462+0000", 2024-09-18T04:39:24.816 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 9.4418204199999991, 2024-09-18T04:39:24.816 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015380944000000001, 2024-09-18T04:39:24.816 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:24.816 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:24.816 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:24.816 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.5246", 2024-09-18T04:39:24.816 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2208214885", 2024-09-18T04:39:24.816 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 98 2024-09-18T04:39:24.816 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.816 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:24.817 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.817 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:24.817 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.279462+0000", 2024-09-18T04:39:24.817 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.817 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.817 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.817 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:24.817 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.279462+0000", 2024-09-18T04:39:24.817 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.817 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.817 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.817 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:24.817 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.279461+0000", 2024-09-18T04:39:24.817 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:24.818 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.818 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.818 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:24.818 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.279712+0000", 2024-09-18T04:39:24.818 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00025065999999999999 2024-09-18T04:39:24.818 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.818 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.818 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:24.818 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.279751+0000", 2024-09-18T04:39:24.818 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9113999999999999e-05 2024-09-18T04:39:24.818 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.818 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.818 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:24.818 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.279768+0000", 2024-09-18T04:39:24.819 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7212999999999999e-05 2024-09-18T04:39:24.819 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.819 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.819 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:24.819 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.291737+0000", 2024-09-18T04:39:24.819 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011968673000000001 2024-09-18T04:39:24.819 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.819 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.819 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:24.820 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.291862+0000", 2024-09-18T04:39:24.820 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000125282 2024-09-18T04:39:24.820 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.820 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.820 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:24.820 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.291947+0000", 2024-09-18T04:39:24.820 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5424000000000001e-05 2024-09-18T04:39:24.820 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.820 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.820 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:24.820 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.294683+0000", 2024-09-18T04:39:24.820 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027360589999999999 2024-09-18T04:39:24.820 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.821 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.821 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:24.821 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.294720+0000", 2024-09-18T04:39:24.821 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6191e-05 2024-09-18T04:39:24.821 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.821 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.821 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:24.821 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.294807+0000", 2024-09-18T04:39:24.821 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7317000000000004e-05 2024-09-18T04:39:24.821 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.821 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.821 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:24.821 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.294843+0000", 2024-09-18T04:39:24.822 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6180999999999998e-05 2024-09-18T04:39:24.822 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.822 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:24.822 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.822 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.822 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.822 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.5246.0:99 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [write 917504~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-18T04:39:24.822 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:15.279786+0000", 2024-09-18T04:39:24.822 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 9.4414964030000004, 2024-09-18T04:39:24.822 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015395890000000001, 2024-09-18T04:39:24.822 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:24.822 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:24.822 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:24.823 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.5246", 2024-09-18T04:39:24.823 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2208214885", 2024-09-18T04:39:24.823 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 99 2024-09-18T04:39:24.823 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.823 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:24.823 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.823 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:24.823 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.279786+0000", 2024-09-18T04:39:24.823 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.823 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.823 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.823 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:24.823 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.279786+0000", 2024-09-18T04:39:24.823 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.824 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.824 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.824 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:24.824 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.279785+0000", 2024-09-18T04:39:24.824 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:24.824 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.824 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.824 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:24.824 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.280030+0000", 2024-09-18T04:39:24.824 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00024536600000000001 2024-09-18T04:39:24.824 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.824 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.824 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:24.824 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.280066+0000", 2024-09-18T04:39:24.825 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5762000000000001e-05 2024-09-18T04:39:24.825 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.825 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.825 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:24.825 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.280084+0000", 2024-09-18T04:39:24.825 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8122e-05 2024-09-18T04:39:24.825 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.825 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.825 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:24.825 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.292557+0000", 2024-09-18T04:39:24.826 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012472584 2024-09-18T04:39:24.826 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.826 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.826 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:24.826 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.292742+0000", 2024-09-18T04:39:24.826 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000185262 2024-09-18T04:39:24.826 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.826 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.826 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:24.826 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.293177+0000", 2024-09-18T04:39:24.826 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000434852 2024-09-18T04:39:24.826 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.827 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.827 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:24.827 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.295110+0000", 2024-09-18T04:39:24.827 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001933216 2024-09-18T04:39:24.827 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.827 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.827 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:24.827 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.295116+0000", 2024-09-18T04:39:24.827 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.806e-06 2024-09-18T04:39:24.827 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.827 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.828 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:24.828 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.295149+0000", 2024-09-18T04:39:24.828 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3544999999999998e-05 2024-09-18T04:39:24.828 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.828 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.828 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:24.828 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.295182+0000", 2024-09-18T04:39:24.828 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2765000000000002e-05 2024-09-18T04:39:24.828 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.828 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:24.828 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.829 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.829 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.829 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.5246.0:100 2.1 2:9370a598:::rbd_header.125fc09df6d1:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-18T04:39:24.829 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:15.280425+0000", 2024-09-18T04:39:24.829 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 9.4408578339999991, 2024-09-18T04:39:24.829 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014635731000000001, 2024-09-18T04:39:24.829 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:24.829 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:24.829 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:24.829 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.5246", 2024-09-18T04:39:24.830 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2208214885", 2024-09-18T04:39:24.830 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 100 2024-09-18T04:39:24.830 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.830 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:24.830 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.830 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:24.830 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.280425+0000", 2024-09-18T04:39:24.830 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.830 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.830 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.830 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:24.830 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.280425+0000", 2024-09-18T04:39:24.831 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.831 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.831 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.831 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:24.831 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.280423+0000", 2024-09-18T04:39:24.831 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:24.831 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.831 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.831 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:24.832 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.280437+0000", 2024-09-18T04:39:24.832 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3956e-05 2024-09-18T04:39:24.832 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.832 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.832 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:24.832 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.280473+0000", 2024-09-18T04:39:24.832 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6028000000000001e-05 2024-09-18T04:39:24.832 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.832 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.832 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:24.832 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.280491+0000", 2024-09-18T04:39:24.833 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.859e-05 2024-09-18T04:39:24.833 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.833 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.833 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:24.833 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.294895+0000", 2024-09-18T04:39:24.833 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014403368 2024-09-18T04:39:24.833 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.833 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.833 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:24.833 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.294963+0000", 2024-09-18T04:39:24.834 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8802999999999998e-05 2024-09-18T04:39:24.834 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.834 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.834 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:24.834 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.295061+0000", 2024-09-18T04:39:24.834 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.7082000000000003e-05 2024-09-18T04:39:24.834 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.834 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:24.835 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.835 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.835 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.835 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.5246.0:195 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [write 262144~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-18T04:39:24.835 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:15.591182+0000", 2024-09-18T04:39:24.835 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 9.1301003460000008, 2024-09-18T04:39:24.835 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014381026999999999, 2024-09-18T04:39:24.835 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:24.835 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:24.836 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:24.836 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.5246", 2024-09-18T04:39:24.836 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2208214885", 2024-09-18T04:39:24.836 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 195 2024-09-18T04:39:24.836 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.836 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:24.836 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.836 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:24.836 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.591182+0000", 2024-09-18T04:39:24.836 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.836 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.837 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.837 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:24.837 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.591182+0000", 2024-09-18T04:39:24.837 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.837 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.837 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.837 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:24.837 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.591181+0000", 2024-09-18T04:39:24.837 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:24.837 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.837 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.837 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:24.837 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.591300+0000", 2024-09-18T04:39:24.838 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000119091 2024-09-18T04:39:24.838 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.838 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.838 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:24.838 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.591323+0000", 2024-09-18T04:39:24.838 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2442e-05 2024-09-18T04:39:24.838 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.838 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.838 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:24.838 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.591341+0000", 2024-09-18T04:39:24.838 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8093000000000001e-05 2024-09-18T04:39:24.838 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.838 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.839 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:24.839 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.598814+0000", 2024-09-18T04:39:24.839 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0074728829999999996 2024-09-18T04:39:24.839 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.839 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.839 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:24.839 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.599015+0000", 2024-09-18T04:39:24.839 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000201466 2024-09-18T04:39:24.839 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.839 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.839 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:24.839 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.599129+0000", 2024-09-18T04:39:24.839 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000114305 2024-09-18T04:39:24.839 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.840 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.840 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:24.840 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.603276+0000", 2024-09-18T04:39:24.840 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0041468959999999997 2024-09-18T04:39:24.840 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.840 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.840 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:24.840 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.605494+0000", 2024-09-18T04:39:24.840 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0022182180000000001 2024-09-18T04:39:24.840 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.840 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.840 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:24.841 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.605539+0000", 2024-09-18T04:39:24.841 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4435999999999999e-05 2024-09-18T04:39:24.841 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.844 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.845 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:24.845 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.605563+0000", 2024-09-18T04:39:24.845 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4425000000000001e-05 2024-09-18T04:39:24.845 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.845 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:24.845 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.845 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.845 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.845 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.5246.0:199 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [write 786432~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-18T04:39:24.845 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:15.591924+0000", 2024-09-18T04:39:24.845 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 9.1293588369999998, 2024-09-18T04:39:24.845 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014640029000000001, 2024-09-18T04:39:24.846 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:24.846 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:24.846 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:24.846 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.5246", 2024-09-18T04:39:24.846 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2208214885", 2024-09-18T04:39:24.846 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 199 2024-09-18T04:39:24.846 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.846 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:24.846 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.846 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:24.846 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.591924+0000", 2024-09-18T04:39:24.846 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.846 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.846 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.847 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:24.847 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.591924+0000", 2024-09-18T04:39:24.847 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.847 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.847 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.847 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:24.847 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.591923+0000", 2024-09-18T04:39:24.847 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:24.847 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.847 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.847 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:24.847 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.592050+0000", 2024-09-18T04:39:24.847 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012711999999999999 2024-09-18T04:39:24.848 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.848 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.848 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:24.848 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.592072+0000", 2024-09-18T04:39:24.848 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2044000000000001e-05 2024-09-18T04:39:24.848 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.848 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.848 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:24.848 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.592104+0000", 2024-09-18T04:39:24.848 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2258000000000002e-05 2024-09-18T04:39:24.848 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.848 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.848 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:24.848 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.604535+0000", 2024-09-18T04:39:24.849 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012430462999999999 2024-09-18T04:39:24.849 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.849 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.849 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:24.849 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.604580+0000", 2024-09-18T04:39:24.849 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5158999999999999e-05 2024-09-18T04:39:24.849 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.849 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.849 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:24.849 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.604648+0000", 2024-09-18T04:39:24.849 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8516000000000006e-05 2024-09-18T04:39:24.849 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.849 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.849 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:24.850 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.606487+0000", 2024-09-18T04:39:24.850 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0018382839999999999 2024-09-18T04:39:24.850 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.850 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.850 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:24.850 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.606497+0000", 2024-09-18T04:39:24.850 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9979999999999998e-06 2024-09-18T04:39:24.850 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.850 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.850 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:24.850 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.606538+0000", 2024-09-18T04:39:24.850 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1600999999999997e-05 2024-09-18T04:39:24.850 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.851 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.851 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:24.851 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.606564+0000", 2024-09-18T04:39:24.851 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5630999999999999e-05 2024-09-18T04:39:24.851 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.851 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:24.851 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.851 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.851 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.851 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.5246.0:200 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [write 917504~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-18T04:39:24.851 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:15.592117+0000", 2024-09-18T04:39:24.851 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 9.1291653850000003, 2024-09-18T04:39:24.852 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015242980999999999, 2024-09-18T04:39:24.852 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:24.852 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:24.852 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:24.852 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.5246", 2024-09-18T04:39:24.852 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2208214885", 2024-09-18T04:39:24.852 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 200 2024-09-18T04:39:24.852 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.852 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:24.852 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.852 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:24.852 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.592117+0000", 2024-09-18T04:39:24.852 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.853 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.853 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.853 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:24.853 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.592117+0000", 2024-09-18T04:39:24.853 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:24.853 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.853 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.853 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:24.853 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.592116+0000", 2024-09-18T04:39:24.854 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:24.854 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.854 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.854 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:24.854 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.592207+0000", 2024-09-18T04:39:24.854 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0478999999999999e-05 2024-09-18T04:39:24.854 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.854 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.854 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:24.854 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.592235+0000", 2024-09-18T04:39:24.855 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8759e-05 2024-09-18T04:39:24.855 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.855 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.855 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:24.855 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.592250+0000", 2024-09-18T04:39:24.855 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4073000000000001e-05 2024-09-18T04:39:24.855 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.855 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.855 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:24.855 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.605911+0000", 2024-09-18T04:39:24.855 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013661227 2024-09-18T04:39:24.856 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.856 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.856 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:24.856 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.605972+0000", 2024-09-18T04:39:24.856 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0999000000000003e-05 2024-09-18T04:39:24.856 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.856 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.856 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:24.856 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.606069+0000", 2024-09-18T04:39:24.857 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6751999999999998e-05 2024-09-18T04:39:24.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.857 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.857 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:24.857 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.606786+0000", 2024-09-18T04:39:24.857 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00071704199999999998 2024-09-18T04:39:24.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.857 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.857 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:24.857 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.607315+0000", 2024-09-18T04:39:24.857 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00052962400000000002 2024-09-18T04:39:24.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.857 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.858 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:24.858 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.607342+0000", 2024-09-18T04:39:24.858 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6658000000000001e-05 2024-09-18T04:39:24.858 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:24.858 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:24.858 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:24.858 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:15.607360+0000", 2024-09-18T04:39:24.858 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8331000000000001e-05 2024-09-18T04:39:24.858 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.858 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:24.858 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.858 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:24.858 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:24.858 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:24.859 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T04:39:24.861 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:39:24.861 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:24.856+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:39:24.961 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:24.953+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:24.981 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:24.981 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:24.981 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:39:24.981 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:24.991 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:39:25.055 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:39:25.062 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:25.057+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:25.074 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:39:25.093 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:39:25.104 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:25.104 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:25.104 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:39:25.104 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:39:25.104 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:25.114 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:39:25.116 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:39:25.139 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:39:25.162 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:39:25.163 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:25.160+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:39:25.164 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:39:25.165 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:39:25.165 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:39:25.165 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:39:25.165 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:39:25.165 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:39:25.219 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:25.219 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:25.229 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:39:25.264 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:25.257+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:25.338 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:25.339 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:25.349 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:39:25.365 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:25.360+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:39:25.456 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:25.457 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:25.465 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:39:25.466 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:25.461+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:25.567 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:25.561+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:25.574 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:25.574 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:25.583 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:39:25.668 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:25.664+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:25.687 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:25.687 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:25.697 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:39:25.769 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:25.761+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:25.802 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:25.802 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:25.812 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:39:25.870 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:25.865+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:25.967 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:25.967 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:25.970 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:25.968+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:25.975 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:39:26.072 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:26.065+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:26.094 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:26.094 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:26.105 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:39:26.173 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:26.165+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:26.217 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:26.218 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:26.228 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:39:26.273 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:26.268+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:26.340 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:26.341 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:26.350 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:39:26.374 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:26.365+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:26.477 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:26.473+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:26.508 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:26.508 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:26.518 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:39:26.578 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:26.576+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:26.622 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-18T04:39:26.628 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:26.628 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:26.639 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:39:26.682 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:26.677+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:26.782 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:26.780+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:39:26.797 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:26.797 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:26.808 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:39:26.885 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:26.880+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:26.916 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:26.917 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:26.926 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:39:26.984 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:26.980+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:27.080 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:27.080 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:27.089 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:27.077+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:27.091 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:39:27.176 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T04:39:27.186 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:27.181+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:27.206 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:27.206 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:27.220 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:39:27.287 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:27.281+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:27.317 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:39:27.335 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:39:27.337 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:27.337 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:27.348 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:39:27.354 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:39:27.374 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:39:27.388 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:27.384+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:39:27.397 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:39:27.418 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:39:27.420 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:39:27.420 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:39:27.420 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:39:27.420 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:39:27.421 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:39:27.421 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:39:27.457 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:27.457 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:27.458 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:39:27.458 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:39:27.458 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:27.467 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:39:27.489 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:27.481+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:27.584 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:27.584 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:39:27.584 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:39:27.584 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:39:27.584 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.584 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:39:27.585 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:39:27.585 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 119.791515813, 2024-09-18T04:39:27.585 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017399218000000001, 2024-09-18T04:39:27.585 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:27.585 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:27.585 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:27.585 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:39:27.585 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:39:27.585 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 114 2024-09-18T04:39:27.585 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.585 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:27.585 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.585 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:27.586 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:39:27.586 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:27.586 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.586 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.586 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:27.586 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:39:27.586 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:27.586 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.586 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.586 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:27.586 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786900+0000", 2024-09-18T04:39:27.586 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:27.586 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.586 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.587 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:27.587 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786908+0000", 2024-09-18T04:39:27.587 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8199999999999997e-06 2024-09-18T04:39:27.587 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.587 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.587 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:27.587 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786925+0000", 2024-09-18T04:39:27.587 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6849000000000001e-05 2024-09-18T04:39:27.587 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.587 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.587 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:27.587 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786934+0000", 2024-09-18T04:39:27.587 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6299999999999993e-06 2024-09-18T04:39:27.588 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.588 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.588 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:27.588 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800926+0000", 2024-09-18T04:39:27.588 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013991632 2024-09-18T04:39:27.588 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.588 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.588 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:27.588 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801060+0000", 2024-09-18T04:39:27.588 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013378199999999999 2024-09-18T04:39:27.588 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.588 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.588 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:27.589 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801166+0000", 2024-09-18T04:39:27.589 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010607099999999999 2024-09-18T04:39:27.589 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.589 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.589 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:27.589 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803026+0000", 2024-09-18T04:39:27.589 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001860469 2024-09-18T04:39:27.589 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.589 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.589 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:27.589 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804245+0000", 2024-09-18T04:39:27.589 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0012185900000000001 2024-09-18T04:39:27.589 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.589 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.590 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:27.590 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804278+0000", 2024-09-18T04:39:27.590 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3272000000000003e-05 2024-09-18T04:39:27.590 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.590 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.590 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:27.590 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804300+0000", 2024-09-18T04:39:27.590 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2266e-05 2024-09-18T04:39:27.590 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:27.590 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:27.590 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:27.590 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.590 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.590 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:39:27.591 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:39:27.591 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 119.79060291499999, 2024-09-18T04:39:27.591 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017521662, 2024-09-18T04:39:27.591 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:27.591 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:27.591 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:27.591 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:39:27.591 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:39:27.591 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 116 2024-09-18T04:39:27.591 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.591 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:27.591 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.591 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:27.591 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:39:27.592 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:27.592 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.592 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.592 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:27.592 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:39:27.592 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:27.592 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.592 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.592 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:27.592 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787813+0000", 2024-09-18T04:39:27.592 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:27.592 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.592 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.592 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:27.593 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787824+0000", 2024-09-18T04:39:27.593 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1157e-05 2024-09-18T04:39:27.593 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.593 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.593 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:27.593 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787840+0000", 2024-09-18T04:39:27.593 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6269e-05 2024-09-18T04:39:27.593 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.593 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.593 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:27.593 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787853+0000", 2024-09-18T04:39:27.593 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3156000000000001e-05 2024-09-18T04:39:27.593 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.594 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.594 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:27.594 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803110+0000", 2024-09-18T04:39:27.594 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015256736 2024-09-18T04:39:27.594 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.594 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.594 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:27.594 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803197+0000", 2024-09-18T04:39:27.594 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7195999999999997e-05 2024-09-18T04:39:27.594 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.594 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.594 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:27.594 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803308+0000", 2024-09-18T04:39:27.594 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111005 2024-09-18T04:39:27.595 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.595 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.595 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:27.595 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804316+0000", 2024-09-18T04:39:27.595 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010073949999999999 2024-09-18T04:39:27.595 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.595 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.595 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:27.595 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805232+0000", 2024-09-18T04:39:27.595 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091636300000000001 2024-09-18T04:39:27.595 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.595 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.595 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:27.595 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805277+0000", 2024-09-18T04:39:27.596 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4864000000000003e-05 2024-09-18T04:39:27.596 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.596 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.596 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:27.596 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805336+0000", 2024-09-18T04:39:27.596 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8968000000000002e-05 2024-09-18T04:39:27.596 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:27.596 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:27.596 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:27.596 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.596 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.596 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:39:27.596 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:39:27.596 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 119.79055086, 2024-09-18T04:39:27.597 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017983453, 2024-09-18T04:39:27.597 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:27.597 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:27.597 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:27.597 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:39:27.597 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:39:27.597 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 117 2024-09-18T04:39:27.597 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.597 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:27.597 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.597 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:27.597 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:39:27.597 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:27.597 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.598 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.598 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:27.598 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:39:27.598 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:27.598 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.598 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.598 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:27.598 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787865+0000", 2024-09-18T04:39:27.598 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:27.598 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.598 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.598 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:27.598 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787889+0000", 2024-09-18T04:39:27.599 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4244999999999999e-05 2024-09-18T04:39:27.599 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.599 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.599 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:27.599 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787905+0000", 2024-09-18T04:39:27.599 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5739999999999998e-05 2024-09-18T04:39:27.599 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.599 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.599 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:27.599 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787918+0000", 2024-09-18T04:39:27.599 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.252e-05 2024-09-18T04:39:27.599 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.599 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.599 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:27.600 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803693+0000", 2024-09-18T04:39:27.600 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015774872999999998 2024-09-18T04:39:27.600 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.600 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.600 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:27.600 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803775+0000", 2024-09-18T04:39:27.600 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2761000000000003e-05 2024-09-18T04:39:27.600 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.600 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.600 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:27.600 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803887+0000", 2024-09-18T04:39:27.600 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111708 2024-09-18T04:39:27.600 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.601 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.601 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:27.601 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804823+0000", 2024-09-18T04:39:27.601 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00093625100000000003 2024-09-18T04:39:27.601 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.601 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.601 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:27.601 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805804+0000", 2024-09-18T04:39:27.601 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00098114699999999992 2024-09-18T04:39:27.601 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.601 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.601 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:27.601 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805831+0000", 2024-09-18T04:39:27.601 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6401999999999999e-05 2024-09-18T04:39:27.602 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.602 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.602 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:27.602 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805850+0000", 2024-09-18T04:39:27.602 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8862e-05 2024-09-18T04:39:27.602 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:27.602 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:27.602 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:27.602 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.602 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.602 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:36 2.7 2:fe1b02ad:::rbd_data.125fc09df6d1.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:27.602 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:39:27.602 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 45.312694749999999, 2024-09-18T04:39:27.602 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.040387804999999999, 2024-09-18T04:39:27.603 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:27.603 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:27.603 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:27.603 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:27.603 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:27.603 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 36 2024-09-18T04:39:27.603 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.603 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:27.603 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.603 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:27.603 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:39:27.603 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:27.603 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.603 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.604 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:27.604 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:39:27.604 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:27.604 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.604 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.604 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:27.604 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265720+0000", 2024-09-18T04:39:27.604 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:27.604 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.604 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.604 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:27.604 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272590+0000", 2024-09-18T04:39:27.604 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.006870508 2024-09-18T04:39:27.604 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.605 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.605 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:27.605 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272623+0000", 2024-09-18T04:39:27.605 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2716000000000003e-05 2024-09-18T04:39:27.605 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.605 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.605 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:27.605 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272637+0000", 2024-09-18T04:39:27.605 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4175999999999999e-05 2024-09-18T04:39:27.606 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.606 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.606 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:27.606 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272779+0000", 2024-09-18T04:39:27.606 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000141392 2024-09-18T04:39:27.606 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.607 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.607 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:27.607 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273031+0000", 2024-09-18T04:39:27.607 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00025249099999999998 2024-09-18T04:39:27.607 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.607 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.607 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:39:27.607 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273219+0000", 2024-09-18T04:39:27.607 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000187531 2024-09-18T04:39:27.607 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.607 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.607 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:27.607 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.302806+0000", 2024-09-18T04:39:27.608 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.029587553999999999 2024-09-18T04:39:27.608 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.608 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.608 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:27.608 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306004+0000", 2024-09-18T04:39:27.608 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0031982629999999998 2024-09-18T04:39:27.608 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.608 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.608 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:27.608 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306071+0000", 2024-09-18T04:39:27.608 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6829999999999995e-05 2024-09-18T04:39:27.608 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.608 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.608 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:27.609 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306110+0000", 2024-09-18T04:39:27.609 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8874000000000001e-05 2024-09-18T04:39:27.609 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:27.609 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:27.609 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:27.609 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.609 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.609 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:38 2.0 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:27.609 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:39:27.609 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 45.305741212999997, 2024-09-18T04:39:27.609 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.033754348000000003, 2024-09-18T04:39:27.609 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:27.609 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:27.609 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:27.610 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:27.610 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:27.610 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 38 2024-09-18T04:39:27.610 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.610 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:27.610 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.610 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:27.610 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:39:27.610 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:27.610 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.610 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.610 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:27.610 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:39:27.610 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:27.611 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.611 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.611 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:27.611 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272673+0000", 2024-09-18T04:39:27.611 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:27.611 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.611 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.611 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:27.611 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277166+0000", 2024-09-18T04:39:27.611 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044922249999999999 2024-09-18T04:39:27.611 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.611 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.611 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:27.611 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277189+0000", 2024-09-18T04:39:27.612 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3283000000000001e-05 2024-09-18T04:39:27.612 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.612 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.612 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:27.612 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277205+0000", 2024-09-18T04:39:27.612 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6305000000000001e-05 2024-09-18T04:39:27.612 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.612 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.612 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:27.612 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277290+0000", 2024-09-18T04:39:27.612 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5135000000000004e-05 2024-09-18T04:39:27.612 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.612 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.612 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:27.613 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277531+0000", 2024-09-18T04:39:27.613 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00024014699999999999 2024-09-18T04:39:27.613 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.613 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.613 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:27.613 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277793+0000", 2024-09-18T04:39:27.613 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00026224499999999998 2024-09-18T04:39:27.613 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.613 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.613 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:27.613 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303990+0000", 2024-09-18T04:39:27.613 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.026197295999999998 2024-09-18T04:39:27.614 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.614 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.614 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:27.614 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306340+0000", 2024-09-18T04:39:27.614 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.002349524 2024-09-18T04:39:27.614 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.614 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.614 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:27.614 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306394+0000", 2024-09-18T04:39:27.614 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3934999999999998e-05 2024-09-18T04:39:27.614 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.614 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.614 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:27.614 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306430+0000", 2024-09-18T04:39:27.615 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6690000000000003e-05 2024-09-18T04:39:27.615 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:27.615 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:27.615 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:27.615 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.615 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.615 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:39 2.7 2:e9c909c5:::rbd_data.125fc09df6d1.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:27.615 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:39:27.615 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 45.301181608, 2024-09-18T04:39:27.615 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.029110673, 2024-09-18T04:39:27.615 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:27.615 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:27.615 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:27.615 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:27.616 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:27.616 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 39 2024-09-18T04:39:27.616 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.616 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:27.616 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.616 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:27.616 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:39:27.616 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:27.616 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.616 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.616 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:27.616 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:39:27.616 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:27.617 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.617 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.617 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:27.617 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277234+0000", 2024-09-18T04:39:27.617 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:27.617 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.617 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.617 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:27.617 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281351+0000", 2024-09-18T04:39:27.617 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0041170920000000001 2024-09-18T04:39:27.617 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.617 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.617 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:27.617 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281375+0000", 2024-09-18T04:39:27.618 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.446e-05 2024-09-18T04:39:27.618 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.618 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.618 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:27.618 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281390+0000", 2024-09-18T04:39:27.618 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4309e-05 2024-09-18T04:39:27.618 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.618 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.618 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:27.618 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.282306+0000", 2024-09-18T04:39:27.618 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091662000000000004 2024-09-18T04:39:27.618 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.618 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.618 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:27.619 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.282613+0000", 2024-09-18T04:39:27.619 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00030663799999999997 2024-09-18T04:39:27.619 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.619 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.619 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:39:27.619 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283001+0000", 2024-09-18T04:39:27.619 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00038801599999999998 2024-09-18T04:39:27.619 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.619 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.619 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:27.619 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306239+0000", 2024-09-18T04:39:27.619 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023238466999999999 2024-09-18T04:39:27.619 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.619 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.620 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:27.620 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306259+0000", 2024-09-18T04:39:27.620 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9837999999999999e-05 2024-09-18T04:39:27.620 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.620 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.620 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:27.620 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306299+0000", 2024-09-18T04:39:27.620 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0194000000000002e-05 2024-09-18T04:39:27.620 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.620 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.620 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:27.620 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306346+0000", 2024-09-18T04:39:27.620 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6845000000000003e-05 2024-09-18T04:39:27.621 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:27.621 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:27.621 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:27.621 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.621 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.621 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:46 2.6 e51/17 2:7ba25245:::rbd_data.125fc09df6d1.0000000000000006:head v 51'2, mlcod=0'0)", 2024-09-18T04:39:27.621 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:39:27.621 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 45.265260476000002, 2024-09-18T04:39:27.621 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019843098, 2024-09-18T04:39:27.621 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:27.621 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:27.621 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:27.621 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.621 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:27.622 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:39:27.622 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:27.622 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.622 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.622 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:27.622 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:39:27.622 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:27.622 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.622 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.622 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:27.622 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313156+0000", 2024-09-18T04:39:27.622 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:27.622 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.622 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.623 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:27.623 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320144+0000", 2024-09-18T04:39:27.623 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.006988345 2024-09-18T04:39:27.623 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.623 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.623 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:27.623 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320175+0000", 2024-09-18T04:39:27.623 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.0861999999999997e-05 2024-09-18T04:39:27.623 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.623 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.623 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:27.623 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320188+0000", 2024-09-18T04:39:27.623 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2714999999999999e-05 2024-09-18T04:39:27.623 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.624 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.624 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:27.624 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320267+0000", 2024-09-18T04:39:27.624 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9586999999999995e-05 2024-09-18T04:39:27.624 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.624 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.624 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:27.624 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320311+0000", 2024-09-18T04:39:27.624 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3529000000000003e-05 2024-09-18T04:39:27.624 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.624 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.624 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:27.624 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.332949+0000", 2024-09-18T04:39:27.624 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012638267999999999 2024-09-18T04:39:27.625 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.625 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.625 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:27.625 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.333000+0000", 2024-09-18T04:39:27.625 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0736999999999999e-05 2024-09-18T04:39:27.625 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:27.625 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:27.625 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:27.625 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.625 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.625 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:57 2.0 2:0cdd070e:::rbd_data.125fc09df6d1.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:27.625 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:39:27.625 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 45.229402901, 2024-09-18T04:39:27.626 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.033329974999999998, 2024-09-18T04:39:27.626 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:27.626 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:27.626 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:27.626 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:27.626 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:27.626 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 57 2024-09-18T04:39:27.626 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.626 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:27.626 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.626 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:27.626 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:39:27.626 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:27.626 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.627 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.627 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:27.627 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:39:27.627 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:27.627 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.627 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.627 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:27.627 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349013+0000", 2024-09-18T04:39:27.627 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:27.627 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.627 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.627 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:27.627 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356138+0000", 2024-09-18T04:39:27.627 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071258010000000002 2024-09-18T04:39:27.628 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.628 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.628 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:27.628 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356168+0000", 2024-09-18T04:39:27.628 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9679000000000002e-05 2024-09-18T04:39:27.628 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.628 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.628 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:27.628 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356181+0000", 2024-09-18T04:39:27.628 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3164e-05 2024-09-18T04:39:27.628 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.628 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.628 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:27.628 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356254+0000", 2024-09-18T04:39:27.629 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3182999999999994e-05 2024-09-18T04:39:27.629 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.629 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.629 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:27.629 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356453+0000", 2024-09-18T04:39:27.629 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019876599999999999 2024-09-18T04:39:27.629 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.629 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.629 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:27.629 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356627+0000", 2024-09-18T04:39:27.629 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017340199999999999 2024-09-18T04:39:27.629 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.629 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.629 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:27.630 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.369200+0000", 2024-09-18T04:39:27.630 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012573302 2024-09-18T04:39:27.630 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.630 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.630 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:27.630 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382262+0000", 2024-09-18T04:39:27.630 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013062549 2024-09-18T04:39:27.630 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.630 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.630 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:27.630 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382310+0000", 2024-09-18T04:39:27.630 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7366e-05 2024-09-18T04:39:27.630 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.630 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.631 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:27.631 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382344+0000", 2024-09-18T04:39:27.631 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4319999999999997e-05 2024-09-18T04:39:27.631 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:27.631 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:27.631 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:27.631 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.631 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.631 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:59 2.0 2:19c52cde:::rbd_data.125fc09df6d1.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:27.631 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:39:27.631 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 45.222211735999998, 2024-09-18T04:39:27.631 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.027137247999999999, 2024-09-18T04:39:27.631 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:27.632 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:27.632 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:27.632 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:27.632 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:27.632 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 59 2024-09-18T04:39:27.632 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.632 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:27.632 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.632 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:27.632 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:39:27.632 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:27.632 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.632 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.632 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:27.633 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:39:27.633 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:27.633 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.633 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.633 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:27.633 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356204+0000", 2024-09-18T04:39:27.633 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:27.633 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.633 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.633 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:27.633 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362661+0000", 2024-09-18T04:39:27.633 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0064565129999999997 2024-09-18T04:39:27.633 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.634 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.634 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:27.634 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362684+0000", 2024-09-18T04:39:27.634 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3688000000000001e-05 2024-09-18T04:39:27.634 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.634 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.634 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:27.634 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362699+0000", 2024-09-18T04:39:27.634 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4798e-05 2024-09-18T04:39:27.634 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.634 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.634 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:27.634 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363016+0000", 2024-09-18T04:39:27.634 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031737300000000003 2024-09-18T04:39:27.635 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.635 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.635 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:27.635 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363201+0000", 2024-09-18T04:39:27.635 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000184131 2024-09-18T04:39:27.635 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.635 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.635 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:27.635 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363343+0000", 2024-09-18T04:39:27.635 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014236699999999999 2024-09-18T04:39:27.635 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.635 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.635 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:27.636 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.372189+0000", 2024-09-18T04:39:27.636 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0088464819999999993 2024-09-18T04:39:27.636 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.636 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.636 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:27.636 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383269+0000", 2024-09-18T04:39:27.636 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011079594 2024-09-18T04:39:27.636 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.636 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.636 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:27.636 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383314+0000", 2024-09-18T04:39:27.636 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4932000000000003e-05 2024-09-18T04:39:27.636 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.637 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.637 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:27.637 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383343+0000", 2024-09-18T04:39:27.637 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8600999999999999e-05 2024-09-18T04:39:27.637 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:27.637 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:27.637 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:27.637 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.637 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.637 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:40 2.6 e51/17 2:784d6cd1:ns::rbd_data.133e813ab5bc.0000000000000005:head v 51'7, mlcod=51'5)", 2024-09-18T04:39:27.637 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:39:27.637 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 45.025962002, 2024-09-18T04:39:27.637 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020497496, 2024-09-18T04:39:27.637 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:27.638 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:27.638 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:27.638 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.638 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:27.638 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:39:27.638 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:27.638 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.638 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.638 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:27.638 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:39:27.638 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:27.638 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.638 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.639 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:27.639 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552454+0000", 2024-09-18T04:39:27.639 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:27.639 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.639 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.639 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:27.639 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556448+0000", 2024-09-18T04:39:27.639 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0039940990000000001 2024-09-18T04:39:27.639 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.639 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.639 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:27.639 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556481+0000", 2024-09-18T04:39:27.639 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3778000000000001e-05 2024-09-18T04:39:27.639 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.640 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.640 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:27.640 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556493+0000", 2024-09-18T04:39:27.640 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1622e-05 2024-09-18T04:39:27.640 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.640 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.640 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:27.640 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556556+0000", 2024-09-18T04:39:27.640 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2582999999999994e-05 2024-09-18T04:39:27.640 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.640 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.640 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:27.640 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556592+0000", 2024-09-18T04:39:27.640 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5880999999999997e-05 2024-09-18T04:39:27.641 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.641 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.641 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:27.641 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.572786+0000", 2024-09-18T04:39:27.641 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016194571000000001 2024-09-18T04:39:27.641 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.641 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.641 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:27.641 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.572953+0000", 2024-09-18T04:39:27.641 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000166436 2024-09-18T04:39:27.641 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:27.641 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:27.641 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:27.641 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.642 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.642 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:45 2.0 2:0fbfa3dc:ns::rbd_data.133e813ab5bc.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:27.642 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:39:27.642 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 45.014747702999998, 2024-09-18T04:39:27.642 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.036558168000000002, 2024-09-18T04:39:27.642 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:27.642 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:27.642 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:27.642 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:27.642 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:27.642 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 45 2024-09-18T04:39:27.642 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.642 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:27.643 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.643 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:27.643 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:39:27.643 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:27.643 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.643 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.643 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:27.643 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:39:27.643 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:27.643 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.643 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.643 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:27.643 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:39:27.643 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:27.644 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.644 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.644 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:27.644 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578594+0000", 2024-09-18T04:39:27.644 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014927147999999999 2024-09-18T04:39:27.644 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.644 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.644 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:27.644 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578612+0000", 2024-09-18T04:39:27.644 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7623999999999999e-05 2024-09-18T04:39:27.644 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.644 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.644 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:27.644 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578620+0000", 2024-09-18T04:39:27.645 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9070000000000007e-06 2024-09-18T04:39:27.645 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.645 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.645 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:27.645 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578702+0000", 2024-09-18T04:39:27.645 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2349999999999996e-05 2024-09-18T04:39:27.645 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.645 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.645 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:27.645 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.579028+0000", 2024-09-18T04:39:27.645 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032622899999999998 2024-09-18T04:39:27.645 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.645 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.645 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:27.646 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.579343+0000", 2024-09-18T04:39:27.646 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031448899999999999 2024-09-18T04:39:27.646 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.646 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.646 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:27.646 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.590078+0000", 2024-09-18T04:39:27.646 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010734946 2024-09-18T04:39:27.646 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.646 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.646 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:27.646 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600089+0000", 2024-09-18T04:39:27.646 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010011195000000001 2024-09-18T04:39:27.646 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.647 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.647 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:27.647 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600177+0000", 2024-09-18T04:39:27.647 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8425999999999997e-05 2024-09-18T04:39:27.647 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.647 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.647 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:27.647 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600228+0000", 2024-09-18T04:39:27.647 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0216999999999997e-05 2024-09-18T04:39:27.647 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:27.647 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:27.647 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:27.647 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.647 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.648 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:54 2.0 2:04f17ef6:ns::rbd_data.133e813ab5bc.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:27.648 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:39:27.648 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 44.993175866000001, 2024-09-18T04:39:27.648 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.024612999, 2024-09-18T04:39:27.648 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:27.648 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:27.648 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:27.648 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:27.648 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:27.648 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 54 2024-09-18T04:39:27.648 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.648 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:27.648 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.649 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:27.649 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:39:27.649 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:27.649 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.649 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.649 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:27.649 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:39:27.649 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:27.649 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.649 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.649 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:27.649 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585239+0000", 2024-09-18T04:39:27.649 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:27.649 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.650 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.650 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:27.650 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593702+0000", 2024-09-18T04:39:27.650 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0084632990000000005 2024-09-18T04:39:27.650 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.650 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.650 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:27.650 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593719+0000", 2024-09-18T04:39:27.650 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.664e-05 2024-09-18T04:39:27.650 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.650 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.650 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:27.650 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593727+0000", 2024-09-18T04:39:27.650 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6839999999999996e-06 2024-09-18T04:39:27.651 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.651 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.651 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:27.651 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593767+0000", 2024-09-18T04:39:27.651 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0330999999999997e-05 2024-09-18T04:39:27.651 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.651 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.651 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:27.651 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593905+0000", 2024-09-18T04:39:27.651 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013754800000000001 2024-09-18T04:39:27.651 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.651 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.651 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:27.651 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.594013+0000", 2024-09-18T04:39:27.652 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000108805 2024-09-18T04:39:27.652 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.652 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.652 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:27.652 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604678+0000", 2024-09-18T04:39:27.652 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010664391 2024-09-18T04:39:27.652 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.652 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.652 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:27.652 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609662+0000", 2024-09-18T04:39:27.652 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0049846100000000004 2024-09-18T04:39:27.652 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.652 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.652 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:27.653 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609788+0000", 2024-09-18T04:39:27.653 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012518800000000001 2024-09-18T04:39:27.653 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.653 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.653 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:27.653 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609854+0000", 2024-09-18T04:39:27.653 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6556000000000004e-05 2024-09-18T04:39:27.653 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:27.653 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:27.653 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:27.653 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.653 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.653 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:58 2.7 2:fc7c6cd3:ns::rbd_data.133e813ab5bc.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:27.653 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:39:27.654 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 44.980333436999999, 2024-09-18T04:39:27.654 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.027690566999999999, 2024-09-18T04:39:27.654 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:27.654 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:27.654 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:27.654 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:27.654 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:27.654 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 58 2024-09-18T04:39:27.654 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.654 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:27.654 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.654 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:27.654 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:39:27.654 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:27.655 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.655 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.655 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:27.655 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:39:27.655 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:27.655 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.655 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.655 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:27.655 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598082+0000", 2024-09-18T04:39:27.655 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:27.655 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.655 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.655 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:27.655 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604832+0000", 2024-09-18T04:39:27.656 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0067504510000000002 2024-09-18T04:39:27.656 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.656 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.656 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:27.656 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604852+0000", 2024-09-18T04:39:27.656 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0228999999999999e-05 2024-09-18T04:39:27.656 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.656 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.656 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:27.656 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604870+0000", 2024-09-18T04:39:27.656 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7952e-05 2024-09-18T04:39:27.656 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.656 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.656 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:27.657 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604936+0000", 2024-09-18T04:39:27.657 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5431000000000004e-05 2024-09-18T04:39:27.657 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.657 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.657 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:27.657 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605085+0000", 2024-09-18T04:39:27.657 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014910300000000001 2024-09-18T04:39:27.657 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.657 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.657 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:39:27.657 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605205+0000", 2024-09-18T04:39:27.657 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000119887 2024-09-18T04:39:27.657 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.658 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.658 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:27.658 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625586+0000", 2024-09-18T04:39:27.658 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020381399000000001 2024-09-18T04:39:27.658 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.658 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.658 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:27.658 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625628+0000", 2024-09-18T04:39:27.658 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1873999999999999e-05 2024-09-18T04:39:27.658 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.658 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.658 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:27.658 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625722+0000", 2024-09-18T04:39:27.658 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3787000000000002e-05 2024-09-18T04:39:27.658 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.659 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.659 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:27.659 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625774+0000", 2024-09-18T04:39:27.659 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2503999999999997e-05 2024-09-18T04:39:27.659 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:27.659 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:27.659 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:27.659 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.659 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.659 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:56 2.6 e51/17 2:72453c3e:ns::rbd_data.133e813ab5bc.000000000000000d:head v 51'10, mlcod=51'8)", 2024-09-18T04:39:27.659 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:39:27.659 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 44.973136328999999, 2024-09-18T04:39:27.659 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019766552999999999, 2024-09-18T04:39:27.660 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:27.660 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:27.660 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:27.660 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.660 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:27.660 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:39:27.660 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:27.660 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.660 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.660 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:27.660 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:39:27.660 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:27.660 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.660 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.661 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:27.661 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605279+0000", 2024-09-18T04:39:27.661 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:27.661 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.661 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.661 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:27.661 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609417+0000", 2024-09-18T04:39:27.661 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0041378059999999999 2024-09-18T04:39:27.661 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.661 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.661 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:27.661 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609443+0000", 2024-09-18T04:39:27.661 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6098e-05 2024-09-18T04:39:27.661 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.662 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.662 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:27.662 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609458+0000", 2024-09-18T04:39:27.662 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4671000000000001e-05 2024-09-18T04:39:27.662 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.662 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.662 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:27.662 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609553+0000", 2024-09-18T04:39:27.662 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5265000000000004e-05 2024-09-18T04:39:27.662 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.662 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.662 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:27.662 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609583+0000", 2024-09-18T04:39:27.662 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9839e-05 2024-09-18T04:39:27.663 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.663 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.663 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:27.663 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625004+0000", 2024-09-18T04:39:27.663 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015420639999999999 2024-09-18T04:39:27.663 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.663 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.663 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:27.663 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625047+0000", 2024-09-18T04:39:27.663 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3665999999999998e-05 2024-09-18T04:39:27.663 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:27.663 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:27.663 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:27.663 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.664 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.664 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:99 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:27.664 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.582718+0000", 2024-09-18T04:39:27.664 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 30.995699420000001, 2024-09-18T04:39:27.664 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019362575, 2024-09-18T04:39:27.664 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:27.664 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:27.664 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:27.664 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:27.664 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:27.664 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 99 2024-09-18T04:39:27.664 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.664 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:27.665 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.665 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:27.665 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582718+0000", 2024-09-18T04:39:27.665 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:27.665 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.665 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.665 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:27.665 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582718+0000", 2024-09-18T04:39:27.665 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:27.665 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.665 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.665 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:27.665 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582717+0000", 2024-09-18T04:39:27.665 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:27.666 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.666 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.666 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:27.666 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582720+0000", 2024-09-18T04:39:27.666 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2239999999999998e-06 2024-09-18T04:39:27.666 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.666 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.666 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:27.666 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582723+0000", 2024-09-18T04:39:27.666 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.258e-06 2024-09-18T04:39:27.666 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.666 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.666 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:27.666 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582726+0000", 2024-09-18T04:39:27.667 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8870000000000001e-06 2024-09-18T04:39:27.667 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.667 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.667 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:27.667 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.592018+0000", 2024-09-18T04:39:27.667 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0092926420000000003 2024-09-18T04:39:27.667 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.667 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.667 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:27.667 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.592214+0000", 2024-09-18T04:39:27.667 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000195267 2024-09-18T04:39:27.667 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.667 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.667 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:27.668 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.592913+0000", 2024-09-18T04:39:27.668 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00069924600000000005 2024-09-18T04:39:27.668 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.668 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.668 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:27.668 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597493+0000", 2024-09-18T04:39:27.668 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045797629999999997 2024-09-18T04:39:27.668 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.668 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.668 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:27.668 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.602009+0000", 2024-09-18T04:39:27.668 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045159559999999998 2024-09-18T04:39:27.668 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.668 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.669 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:27.669 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.602042+0000", 2024-09-18T04:39:27.669 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3084999999999999e-05 2024-09-18T04:39:27.669 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.669 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.669 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:27.669 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.602080+0000", 2024-09-18T04:39:27.669 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8600000000000003e-05 2024-09-18T04:39:27.669 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:27.669 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:27.669 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:27.669 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.669 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.670 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:104 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:27.670 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.583165+0000", 2024-09-18T04:39:27.670 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 30.995252122, 2024-09-18T04:39:27.670 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023679603, 2024-09-18T04:39:27.670 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:27.670 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:27.670 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:27.670 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:27.670 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:27.670 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 104 2024-09-18T04:39:27.670 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.670 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:27.670 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.670 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:27.671 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583165+0000", 2024-09-18T04:39:27.671 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:27.671 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.671 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.671 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:27.671 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583165+0000", 2024-09-18T04:39:27.671 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:27.671 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.671 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.671 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:27.671 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583164+0000", 2024-09-18T04:39:27.672 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:27.672 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.672 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.672 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:27.672 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583167+0000", 2024-09-18T04:39:27.672 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9210000000000001e-06 2024-09-18T04:39:27.672 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.672 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.672 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:27.672 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583171+0000", 2024-09-18T04:39:27.672 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9759999999999997e-06 2024-09-18T04:39:27.672 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.672 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.672 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:27.673 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583174+0000", 2024-09-18T04:39:27.673 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9960000000000002e-06 2024-09-18T04:39:27.673 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.673 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.673 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:27.673 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.594373+0000", 2024-09-18T04:39:27.673 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011198665999999999 2024-09-18T04:39:27.673 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.673 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.673 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:27.673 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.594434+0000", 2024-09-18T04:39:27.673 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.1446999999999996e-05 2024-09-18T04:39:27.673 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.673 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.674 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:27.674 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.594692+0000", 2024-09-18T04:39:27.674 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00025756599999999998 2024-09-18T04:39:27.674 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.674 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.674 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:27.674 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599302+0000", 2024-09-18T04:39:27.674 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0046099640000000002 2024-09-18T04:39:27.674 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.674 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.674 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:27.674 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606710+0000", 2024-09-18T04:39:27.674 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0074077739999999998 2024-09-18T04:39:27.674 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.675 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.675 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:27.675 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606798+0000", 2024-09-18T04:39:27.675 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8133999999999993e-05 2024-09-18T04:39:27.675 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.675 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.675 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:27.675 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606845+0000", 2024-09-18T04:39:27.675 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6653000000000002e-05 2024-09-18T04:39:27.675 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:27.675 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:27.675 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:27.675 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.675 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.676 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:105 2.0 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:27.676 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.583193+0000", 2024-09-18T04:39:27.676 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 30.995224502999999, 2024-09-18T04:39:27.676 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023713649, 2024-09-18T04:39:27.676 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:27.676 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:27.676 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:27.676 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:27.676 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:27.676 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 105 2024-09-18T04:39:27.676 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.676 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:27.676 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.677 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:27.677 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583193+0000", 2024-09-18T04:39:27.677 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:27.677 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.677 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.677 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:27.677 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583193+0000", 2024-09-18T04:39:27.677 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:27.677 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.677 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.677 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:27.677 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583192+0000", 2024-09-18T04:39:27.677 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:27.677 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.678 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.678 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:27.678 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583194+0000", 2024-09-18T04:39:27.678 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.305e-06 2024-09-18T04:39:27.678 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.678 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.678 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:27.678 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583198+0000", 2024-09-18T04:39:27.678 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9369999999999997e-06 2024-09-18T04:39:27.678 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.678 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.678 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:27.678 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583202+0000", 2024-09-18T04:39:27.678 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6480000000000001e-06 2024-09-18T04:39:27.679 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.679 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.679 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:27.679 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.595304+0000", 2024-09-18T04:39:27.679 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012102063999999999 2024-09-18T04:39:27.679 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.679 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.679 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:27.679 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.595390+0000", 2024-09-18T04:39:27.679 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5493000000000003e-05 2024-09-18T04:39:27.679 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.679 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.679 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:27.679 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.595509+0000", 2024-09-18T04:39:27.680 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011966199999999999 2024-09-18T04:39:27.680 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.680 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.680 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:27.680 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599358+0000", 2024-09-18T04:39:27.680 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0038486689999999999 2024-09-18T04:39:27.680 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.680 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.680 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:27.680 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606859+0000", 2024-09-18T04:39:27.680 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0075010789999999999 2024-09-18T04:39:27.680 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.680 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.680 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:27.681 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606887+0000", 2024-09-18T04:39:27.681 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7773e-05 2024-09-18T04:39:27.681 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.681 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.681 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:27.681 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606906+0000", 2024-09-18T04:39:27.681 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9417000000000001e-05 2024-09-18T04:39:27.681 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:27.681 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:27.681 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:27.681 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.681 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.681 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:106 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:27.681 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.584425+0000", 2024-09-18T04:39:27.682 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 30.993992207000002, 2024-09-18T04:39:27.682 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.022529293999999998, 2024-09-18T04:39:27.682 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:27.682 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:27.682 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:27.682 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:27.682 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:27.682 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 106 2024-09-18T04:39:27.682 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.682 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:27.682 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.682 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:27.682 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584425+0000", 2024-09-18T04:39:27.682 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:27.683 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.683 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.683 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:27.683 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584425+0000", 2024-09-18T04:39:27.683 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:27.683 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.683 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.683 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:27.683 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584424+0000", 2024-09-18T04:39:27.683 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:27.683 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.683 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.683 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:27.683 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584430+0000", 2024-09-18T04:39:27.684 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6389999999999997e-06 2024-09-18T04:39:27.684 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.684 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.684 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:27.684 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584456+0000", 2024-09-18T04:39:27.684 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6188999999999999e-05 2024-09-18T04:39:27.684 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.684 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.684 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:27.684 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584463+0000", 2024-09-18T04:39:27.684 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7340000000000002e-06 2024-09-18T04:39:27.684 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.684 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.685 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:27.685 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597537+0000", 2024-09-18T04:39:27.685 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013074161000000001 2024-09-18T04:39:27.685 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.685 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.685 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:27.685 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597598+0000", 2024-09-18T04:39:27.685 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0930000000000001e-05 2024-09-18T04:39:27.685 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.685 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.685 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:27.685 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597922+0000", 2024-09-18T04:39:27.685 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032473000000000002 2024-09-18T04:39:27.685 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.686 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.686 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:27.686 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.601103+0000", 2024-09-18T04:39:27.686 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0031809849999999999 2024-09-18T04:39:27.686 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.686 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.686 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:27.686 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606912+0000", 2024-09-18T04:39:27.686 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058083550000000003 2024-09-18T04:39:27.686 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.686 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.686 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:27.686 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606935+0000", 2024-09-18T04:39:27.686 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2759999999999999e-05 2024-09-18T04:39:27.687 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.687 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.687 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:27.687 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606954+0000", 2024-09-18T04:39:27.687 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9602999999999998e-05 2024-09-18T04:39:27.687 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:27.687 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:27.687 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:27.687 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.687 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.687 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:109 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:27.687 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.585231+0000", 2024-09-18T04:39:27.687 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 30.993185586999999, 2024-09-18T04:39:27.687 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.021767496000000001, 2024-09-18T04:39:27.688 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:27.688 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:27.688 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:27.688 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:27.688 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:27.688 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 109 2024-09-18T04:39:27.688 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.688 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:27.688 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.688 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:27.688 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585231+0000", 2024-09-18T04:39:27.688 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:27.688 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.688 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.689 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:27.689 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585231+0000", 2024-09-18T04:39:27.689 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:27.689 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.689 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.689 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:27.689 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585230+0000", 2024-09-18T04:39:27.689 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:27.689 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.689 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.689 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:27.689 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585237+0000", 2024-09-18T04:39:27.689 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.618e-06 2024-09-18T04:39:27.689 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.690 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.690 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:27.690 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585256+0000", 2024-09-18T04:39:27.690 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9048000000000001e-05 2024-09-18T04:39:27.690 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.690 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.690 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:27.690 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585274+0000", 2024-09-18T04:39:27.690 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8153000000000001e-05 2024-09-18T04:39:27.690 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.690 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.690 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:27.690 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.598284+0000", 2024-09-18T04:39:27.690 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013009682999999999 2024-09-18T04:39:27.691 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.691 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.691 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:27.691 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.598383+0000", 2024-09-18T04:39:27.691 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.8722999999999997e-05 2024-09-18T04:39:27.691 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.691 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.691 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:27.691 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.598829+0000", 2024-09-18T04:39:27.691 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00044623999999999998 2024-09-18T04:39:27.691 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.691 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.691 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:27.691 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.601149+0000", 2024-09-18T04:39:27.692 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0023204150000000002 2024-09-18T04:39:27.692 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.692 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.692 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:27.692 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606960+0000", 2024-09-18T04:39:27.692 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058106390000000003 2024-09-18T04:39:27.692 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.692 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.692 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:27.692 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606982+0000", 2024-09-18T04:39:27.692 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2252e-05 2024-09-18T04:39:27.692 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.692 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.692 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:27.693 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606999+0000", 2024-09-18T04:39:27.693 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6871000000000002e-05 2024-09-18T04:39:27.693 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:27.693 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:27.693 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:27.693 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.693 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.693 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:110 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:27.693 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.586640+0000", 2024-09-18T04:39:27.693 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 30.991777268, 2024-09-18T04:39:27.693 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020400911000000001, 2024-09-18T04:39:27.693 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:27.693 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:27.693 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:27.694 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:27.694 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:27.694 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 110 2024-09-18T04:39:27.694 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.694 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:27.694 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.694 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:27.694 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586640+0000", 2024-09-18T04:39:27.694 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:27.694 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.694 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.694 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:27.694 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586640+0000", 2024-09-18T04:39:27.694 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:27.695 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.695 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.695 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:27.695 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586639+0000", 2024-09-18T04:39:27.695 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:27.695 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.695 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.695 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:27.695 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586652+0000", 2024-09-18T04:39:27.695 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2482e-05 2024-09-18T04:39:27.695 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.695 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.695 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:27.695 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586660+0000", 2024-09-18T04:39:27.696 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1659999999999994e-06 2024-09-18T04:39:27.696 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.696 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.696 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:27.696 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586666+0000", 2024-09-18T04:39:27.696 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6900000000000003e-06 2024-09-18T04:39:27.696 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.696 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.696 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:27.696 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599381+0000", 2024-09-18T04:39:27.696 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012714328 2024-09-18T04:39:27.696 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.696 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.696 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:27.697 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599436+0000", 2024-09-18T04:39:27.697 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4802999999999997e-05 2024-09-18T04:39:27.697 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.697 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.697 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:27.697 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599629+0000", 2024-09-18T04:39:27.697 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000193921 2024-09-18T04:39:27.697 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.697 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.697 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:27.697 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.601174+0000", 2024-09-18T04:39:27.697 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001544857 2024-09-18T04:39:27.697 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.697 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.698 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:27.698 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.607004+0000", 2024-09-18T04:39:27.698 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058300820000000003 2024-09-18T04:39:27.698 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.698 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.698 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:27.698 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.607024+0000", 2024-09-18T04:39:27.698 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9888e-05 2024-09-18T04:39:27.698 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:27.698 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:27.698 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:27.698 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.607041+0000", 2024-09-18T04:39:27.698 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6450999999999999e-05 2024-09-18T04:39:27.698 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:27.699 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:27.699 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:27.699 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:27.699 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:27.699 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:27.699 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 2] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-18T04:39:27.699 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T04:39:27.700 INFO:tasks.thrashosds.thrasher:Killing osd 2, live_osds are [0, 1, 2, 3, 4, 5] 2024-09-18T04:39:27.700 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-18T04:39:27.700 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T04:39:27.702 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-09-18T04:39:27.702 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:39:27.702 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:27.696+0000 7fad7390c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20962) UID: 0 2024-09-18T04:39:27.783 INFO:tasks.ceph.osd.2:Stopped 2024-09-18T04:39:27.783 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.2 2024-09-18T04:39:27.783 INFO:tasks.thrashosds.thrasher:remote for osd 2 is ubuntu@smithi028.front.sepia.ceph.com 2024-09-18T04:39:27.783 DEBUG:teuthology.orchestra.run.smithi028:> type ceph-objectstore-tool 2024-09-18T04:39:27.788 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-18T04:39:27.789 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --op list-pgs 2024-09-18T04:39:27.801 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:27.796+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:27.859 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:27.860 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:27.860 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:39:27.860 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:27.869 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:39:27.902 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:27.897+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:27.972 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:27.972 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:27.973 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:39:27.973 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:39:27.973 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:27.988 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:39:28.002 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:28.000+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:28.104 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:28.097+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:28.143 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:28.143 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:39:28.143 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:39:28.143 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:39:28.143 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.143 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-18T04:39:28.144 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:39:28.144 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 113.66784860999999, 2024-09-18T04:39:28.144 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0095781189999999995, 2024-09-18T04:39:28.144 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:28.144 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:28.144 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:28.144 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:39:28.144 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:39:28.144 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 1 2024-09-18T04:39:28.144 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.144 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:28.144 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.144 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:28.145 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:39:28.145 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:28.145 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.145 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.145 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:28.145 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:39:28.145 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:28.145 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.145 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.145 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:28.145 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469862+0000", 2024-09-18T04:39:28.145 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:28.145 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.145 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.146 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:28.146 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469879+0000", 2024-09-18T04:39:28.146 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7431999999999998e-05 2024-09-18T04:39:28.146 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.146 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.146 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:28.146 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469896+0000", 2024-09-18T04:39:28.146 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6742999999999999e-05 2024-09-18T04:39:28.146 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.146 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.146 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:28.146 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469915+0000", 2024-09-18T04:39:28.146 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8991000000000002e-05 2024-09-18T04:39:28.146 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.147 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.147 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:28.147 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470397+0000", 2024-09-18T04:39:28.147 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048167799999999998 2024-09-18T04:39:28.147 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.147 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.147 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-09-18T04:39:28.147 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470412+0000", 2024-09-18T04:39:28.147 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5526e-05 2024-09-18T04:39:28.147 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.147 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.147 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:28.147 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473279+0000", 2024-09-18T04:39:28.147 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0028667469999999998 2024-09-18T04:39:28.148 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.148 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.148 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:28.148 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473441+0000", 2024-09-18T04:39:28.148 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000162082 2024-09-18T04:39:28.148 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.148 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.148 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:28.148 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473573+0000", 2024-09-18T04:39:28.148 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000132032 2024-09-18T04:39:28.148 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.148 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.148 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:28.149 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474826+0000", 2024-09-18T04:39:28.149 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001252717 2024-09-18T04:39:28.149 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.149 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.149 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:28.149 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479307+0000", 2024-09-18T04:39:28.149 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044813220000000003 2024-09-18T04:39:28.149 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.149 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.149 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:28.149 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479366+0000", 2024-09-18T04:39:28.149 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9024999999999998e-05 2024-09-18T04:39:28.149 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.149 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.150 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:28.150 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479443+0000", 2024-09-18T04:39:28.150 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6339000000000002e-05 2024-09-18T04:39:28.150 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:28.150 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:28.150 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:28.150 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.150 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.150 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-18T04:39:28.150 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:39:28.150 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 111.65484019, 2024-09-18T04:39:28.150 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082699560000000002, 2024-09-18T04:39:28.150 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:28.150 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:28.150 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:28.151 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:39:28.151 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:39:28.151 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 7 2024-09-18T04:39:28.151 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.151 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:28.151 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.151 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:28.151 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:39:28.151 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:28.151 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.151 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.151 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:28.151 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:39:28.151 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:28.152 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.152 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.152 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:28.152 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482870+0000", 2024-09-18T04:39:28.152 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999967 2024-09-18T04:39:28.152 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.152 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.152 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:28.152 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482889+0000", 2024-09-18T04:39:28.152 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.965e-05 2024-09-18T04:39:28.152 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.152 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.152 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:28.152 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482938+0000", 2024-09-18T04:39:28.153 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8548999999999999e-05 2024-09-18T04:39:28.153 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.153 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.153 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:28.153 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482963+0000", 2024-09-18T04:39:28.153 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5075999999999998e-05 2024-09-18T04:39:28.153 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.153 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.153 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:28.153 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483121+0000", 2024-09-18T04:39:28.153 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015819400000000001 2024-09-18T04:39:28.153 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.153 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.153 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op must wait for map", 2024-09-18T04:39:28.154 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483138+0000", 2024-09-18T04:39:28.154 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7068999999999999e-05 2024-09-18T04:39:28.154 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.154 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.154 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:28.154 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488666+0000", 2024-09-18T04:39:28.154 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0055282049999999996 2024-09-18T04:39:28.154 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.154 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.154 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:28.154 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488783+0000", 2024-09-18T04:39:28.154 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000116737 2024-09-18T04:39:28.154 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.154 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.155 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:28.155 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488991+0000", 2024-09-18T04:39:28.155 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020781400000000001 2024-09-18T04:39:28.155 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.155 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.155 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:28.155 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490554+0000", 2024-09-18T04:39:28.155 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0015628370000000001 2024-09-18T04:39:28.155 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.155 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.155 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:28.156 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491019+0000", 2024-09-18T04:39:28.156 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000465608 2024-09-18T04:39:28.156 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.156 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.156 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:28.156 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491087+0000", 2024-09-18T04:39:28.156 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7988e-05 2024-09-18T04:39:28.156 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.156 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.156 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:28.156 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491143+0000", 2024-09-18T04:39:28.156 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5538999999999998e-05 2024-09-18T04:39:28.156 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:28.157 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:28.157 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:28.157 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.157 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.157 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:34 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:28.157 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.261758+0000", 2024-09-18T04:39:28.157 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 45.875955038999997, 2024-09-18T04:39:28.157 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.042024191000000002, 2024-09-18T04:39:28.157 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:28.157 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:28.157 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:28.157 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:28.157 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:28.157 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 34 2024-09-18T04:39:28.158 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.158 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:28.158 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.158 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:28.158 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.261758+0000", 2024-09-18T04:39:28.158 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:28.158 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.158 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.158 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:28.158 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.261758+0000", 2024-09-18T04:39:28.158 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:28.158 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.158 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.158 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:28.159 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.261756+0000", 2024-09-18T04:39:28.159 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:28.159 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.159 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.159 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:28.159 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.270785+0000", 2024-09-18T04:39:28.159 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0090292340000000006 2024-09-18T04:39:28.159 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.159 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.159 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:28.159 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.270890+0000", 2024-09-18T04:39:28.159 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000104541 2024-09-18T04:39:28.159 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.159 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.160 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:28.160 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.270991+0000", 2024-09-18T04:39:28.160 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000100778 2024-09-18T04:39:28.160 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.160 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.160 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:28.160 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.271157+0000", 2024-09-18T04:39:28.160 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016653999999999999 2024-09-18T04:39:28.160 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.160 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.160 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:28.160 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.271591+0000", 2024-09-18T04:39:28.160 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00043409500000000001 2024-09-18T04:39:28.160 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.161 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.161 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:28.161 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272017+0000", 2024-09-18T04:39:28.161 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000425769 2024-09-18T04:39:28.161 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.161 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.161 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:28.161 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.294246+0000", 2024-09-18T04:39:28.161 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.022229237999999998 2024-09-18T04:39:28.161 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.161 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.161 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:28.161 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303571+0000", 2024-09-18T04:39:28.161 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0093250239999999995 2024-09-18T04:39:28.162 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.162 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.162 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:28.162 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303705+0000", 2024-09-18T04:39:28.162 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000133831 2024-09-18T04:39:28.162 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.162 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.162 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:28.162 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303782+0000", 2024-09-18T04:39:28.162 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7156000000000005e-05 2024-09-18T04:39:28.162 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:28.162 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:28.162 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:28.162 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.163 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.163 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:36 2.7 e51/32 2:fe1b02ad:::rbd_data.125fc09df6d1.0000000000000002:head v 51'1, mlcod=0'0)", 2024-09-18T04:39:28.163 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.273441+0000", 2024-09-18T04:39:28.163 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 45.864272057999997, 2024-09-18T04:39:28.163 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.032400768000000003, 2024-09-18T04:39:28.163 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:28.163 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:28.163 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:28.163 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.163 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:28.163 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273441+0000", 2024-09-18T04:39:28.163 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:28.163 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.163 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.164 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:28.164 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273441+0000", 2024-09-18T04:39:28.164 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:28.164 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.164 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.164 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:28.164 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273440+0000", 2024-09-18T04:39:28.164 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:28.164 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.164 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.164 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:28.164 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278723+0000", 2024-09-18T04:39:28.164 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0052829909999999999 2024-09-18T04:39:28.164 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.165 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.165 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:28.165 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278752+0000", 2024-09-18T04:39:28.165 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9178000000000001e-05 2024-09-18T04:39:28.165 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.165 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.165 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:28.165 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278761+0000", 2024-09-18T04:39:28.165 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7900000000000005e-06 2024-09-18T04:39:28.165 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.165 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.165 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:28.165 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278864+0000", 2024-09-18T04:39:28.165 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000102699 2024-09-18T04:39:28.166 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.166 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.166 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:28.166 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278900+0000", 2024-09-18T04:39:28.166 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6001999999999999e-05 2024-09-18T04:39:28.166 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.166 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.166 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:28.166 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.305793+0000", 2024-09-18T04:39:28.166 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.026893344 2024-09-18T04:39:28.166 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.166 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.166 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:28.166 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.305842+0000", 2024-09-18T04:39:28.167 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8720000000000001e-05 2024-09-18T04:39:28.167 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:28.167 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:28.167 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:28.167 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.167 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.167 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:39 2.7 e51/32 2:e9c909c5:::rbd_data.125fc09df6d1.0000000000000005:head v 51'2, mlcod=0'0)", 2024-09-18T04:39:28.167 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.283175+0000", 2024-09-18T04:39:28.167 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 45.854538368999997, 2024-09-18T04:39:28.167 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.022864102000000001, 2024-09-18T04:39:28.167 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:28.167 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:28.167 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:28.167 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.168 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:28.168 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283175+0000", 2024-09-18T04:39:28.168 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:28.168 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.168 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.168 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:28.168 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283175+0000", 2024-09-18T04:39:28.168 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:28.168 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.168 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.168 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:28.168 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283173+0000", 2024-09-18T04:39:28.168 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:28.168 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.169 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.169 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:28.169 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.285951+0000", 2024-09-18T04:39:28.169 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0027784540000000001 2024-09-18T04:39:28.169 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.169 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.169 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:28.169 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.285970+0000", 2024-09-18T04:39:28.169 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8855000000000002e-05 2024-09-18T04:39:28.169 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.169 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.169 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:28.169 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.285984+0000", 2024-09-18T04:39:28.169 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4252e-05 2024-09-18T04:39:28.170 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.170 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.170 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:28.170 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.286041+0000", 2024-09-18T04:39:28.170 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6431999999999998e-05 2024-09-18T04:39:28.170 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.170 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.170 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:28.170 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.286086+0000", 2024-09-18T04:39:28.170 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5423000000000002e-05 2024-09-18T04:39:28.170 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.170 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.170 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:28.170 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.305998+0000", 2024-09-18T04:39:28.171 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019911697999999999 2024-09-18T04:39:28.171 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.171 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.171 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:28.171 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306039+0000", 2024-09-18T04:39:28.171 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0973999999999998e-05 2024-09-18T04:39:28.171 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:28.171 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:28.171 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:28.171 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.171 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.171 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:50 2.3 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:28.171 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.312563+0000", 2024-09-18T04:39:28.171 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 45.825150342000001, 2024-09-18T04:39:28.172 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.032636001999999997, 2024-09-18T04:39:28.172 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:28.172 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:28.172 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:28.172 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:28.172 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:28.172 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 50 2024-09-18T04:39:28.172 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.172 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:28.172 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.172 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:28.172 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.312563+0000", 2024-09-18T04:39:28.172 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:28.172 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.173 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.173 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:28.173 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.312563+0000", 2024-09-18T04:39:28.173 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:28.173 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.173 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.173 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:28.173 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.312561+0000", 2024-09-18T04:39:28.173 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:28.173 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.173 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.173 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:28.173 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321709+0000", 2024-09-18T04:39:28.173 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0091486650000000003 2024-09-18T04:39:28.174 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.174 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.174 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:28.174 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321739+0000", 2024-09-18T04:39:28.174 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9666e-05 2024-09-18T04:39:28.174 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.174 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.174 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:28.174 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321751+0000", 2024-09-18T04:39:28.174 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2086000000000001e-05 2024-09-18T04:39:28.174 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.174 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.174 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:28.174 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321833+0000", 2024-09-18T04:39:28.175 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1842999999999996e-05 2024-09-18T04:39:28.175 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.175 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.175 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:28.175 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321981+0000", 2024-09-18T04:39:28.175 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000147441 2024-09-18T04:39:28.175 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.175 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.175 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:28.175 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.322109+0000", 2024-09-18T04:39:28.175 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012851200000000001 2024-09-18T04:39:28.175 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.175 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.175 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:28.176 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.332829+0000", 2024-09-18T04:39:28.176 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010720225999999999 2024-09-18T04:39:28.176 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.176 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.176 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:28.176 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.345039+0000", 2024-09-18T04:39:28.176 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012209831000000001 2024-09-18T04:39:28.176 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.176 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.176 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:28.176 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.345120+0000", 2024-09-18T04:39:28.176 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1145999999999999e-05 2024-09-18T04:39:28.176 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.176 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.177 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:28.177 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.345199+0000", 2024-09-18T04:39:28.177 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8546000000000003e-05 2024-09-18T04:39:28.177 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:28.177 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:28.177 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:28.177 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.177 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.177 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:30 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:28.177 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.516122+0000", 2024-09-18T04:39:28.177 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 45.621591180000003, 2024-09-18T04:39:28.177 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0091865030000000004, 2024-09-18T04:39:28.177 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:28.177 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:28.178 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:28.178 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:28.178 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:28.178 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 30 2024-09-18T04:39:28.178 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.178 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:28.178 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.178 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:28.178 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516122+0000", 2024-09-18T04:39:28.178 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:28.178 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.178 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.178 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:28.178 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516122+0000", 2024-09-18T04:39:28.179 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:28.179 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.179 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.179 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:28.179 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516120+0000", 2024-09-18T04:39:28.179 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:28.179 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.179 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.179 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:28.179 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516129+0000", 2024-09-18T04:39:28.179 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.85e-06 2024-09-18T04:39:28.179 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.179 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.179 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:28.179 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516155+0000", 2024-09-18T04:39:28.180 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5882000000000001e-05 2024-09-18T04:39:28.180 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.180 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.180 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:28.180 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516175+0000", 2024-09-18T04:39:28.180 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.022e-05 2024-09-18T04:39:28.180 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.180 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.180 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:28.180 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519780+0000", 2024-09-18T04:39:28.180 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003604187 2024-09-18T04:39:28.180 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.180 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.181 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:28.181 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519964+0000", 2024-09-18T04:39:28.181 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00018391999999999999 2024-09-18T04:39:28.181 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.181 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.181 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:28.181 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.520794+0000", 2024-09-18T04:39:28.181 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00083078100000000005 2024-09-18T04:39:28.181 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.181 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.181 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:28.181 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.524129+0000", 2024-09-18T04:39:28.181 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033342419999999998 2024-09-18T04:39:28.181 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.181 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.182 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:28.182 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525239+0000", 2024-09-18T04:39:28.182 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001110778 2024-09-18T04:39:28.182 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.182 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.182 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:28.182 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525277+0000", 2024-09-18T04:39:28.182 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7596999999999999e-05 2024-09-18T04:39:28.182 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.182 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.182 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:28.182 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525308+0000", 2024-09-18T04:39:28.182 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.1535999999999998e-05 2024-09-18T04:39:28.182 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:28.183 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:28.183 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:28.183 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.183 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.183 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:32 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:28.183 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.519217+0000", 2024-09-18T04:39:28.183 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 45.618495860000003, 2024-09-18T04:39:28.183 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082266370000000002, 2024-09-18T04:39:28.183 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:28.183 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:28.183 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:28.183 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:28.183 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:28.183 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 32 2024-09-18T04:39:28.184 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.184 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:28.184 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.184 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:28.184 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519217+0000", 2024-09-18T04:39:28.184 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:28.184 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.184 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.184 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:28.184 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519217+0000", 2024-09-18T04:39:28.184 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:28.184 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.184 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.185 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:28.185 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519215+0000", 2024-09-18T04:39:28.185 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:28.185 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.185 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.185 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:28.185 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519229+0000", 2024-09-18T04:39:28.185 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3329999999999999e-05 2024-09-18T04:39:28.185 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.185 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.185 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:28.185 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519268+0000", 2024-09-18T04:39:28.185 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9533000000000002e-05 2024-09-18T04:39:28.185 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.186 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.186 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:28.186 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519285+0000", 2024-09-18T04:39:28.186 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6446000000000001e-05 2024-09-18T04:39:28.186 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.186 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.186 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:28.186 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.522358+0000", 2024-09-18T04:39:28.186 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030734310000000002 2024-09-18T04:39:28.186 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.186 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.186 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:28.186 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.522551+0000", 2024-09-18T04:39:28.186 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019317600000000001 2024-09-18T04:39:28.186 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.187 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.187 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:28.187 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.523262+0000", 2024-09-18T04:39:28.187 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00071069999999999998 2024-09-18T04:39:28.187 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.187 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.187 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:28.187 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525318+0000", 2024-09-18T04:39:28.187 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0020561469999999999 2024-09-18T04:39:28.187 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.187 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.187 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:28.187 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.527376+0000", 2024-09-18T04:39:28.187 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0020584969999999998 2024-09-18T04:39:28.188 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.188 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.188 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:28.188 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.527416+0000", 2024-09-18T04:39:28.188 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9054000000000003e-05 2024-09-18T04:39:28.188 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.188 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.188 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:28.188 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.527444+0000", 2024-09-18T04:39:28.188 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8347000000000002e-05 2024-09-18T04:39:28.188 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:28.188 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:28.188 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:28.188 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.189 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.189 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:42 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:28.189 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.551160+0000", 2024-09-18T04:39:28.189 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 45.586553467999998, 2024-09-18T04:39:28.189 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010692757000000001, 2024-09-18T04:39:28.189 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:28.189 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:28.189 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:28.189 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:28.189 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:28.189 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 42 2024-09-18T04:39:28.189 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.189 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:28.190 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.190 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:28.190 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551160+0000", 2024-09-18T04:39:28.190 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:28.190 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.190 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.190 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:28.190 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551160+0000", 2024-09-18T04:39:28.190 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:28.190 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.190 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.190 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:28.190 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551159+0000", 2024-09-18T04:39:28.190 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:28.190 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.191 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.191 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:28.191 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551166+0000", 2024-09-18T04:39:28.191 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1029999999999996e-06 2024-09-18T04:39:28.191 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.191 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.191 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:28.191 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551177+0000", 2024-09-18T04:39:28.191 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1384e-05 2024-09-18T04:39:28.191 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.191 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.191 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:28.191 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551185+0000", 2024-09-18T04:39:28.191 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1990000000000006e-06 2024-09-18T04:39:28.192 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.192 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.192 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:28.192 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552021+0000", 2024-09-18T04:39:28.192 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00083574400000000003 2024-09-18T04:39:28.192 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.192 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.192 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:28.192 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552339+0000", 2024-09-18T04:39:28.192 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031771399999999997 2024-09-18T04:39:28.192 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.192 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.192 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:28.192 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.553140+0000", 2024-09-18T04:39:28.193 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00080088100000000003 2024-09-18T04:39:28.193 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.193 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.193 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:28.193 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.554585+0000", 2024-09-18T04:39:28.193 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001445801 2024-09-18T04:39:28.193 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.193 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.193 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:28.193 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.561711+0000", 2024-09-18T04:39:28.193 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071260109999999998 2024-09-18T04:39:28.193 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.193 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.193 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:28.194 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.561790+0000", 2024-09-18T04:39:28.194 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8967000000000005e-05 2024-09-18T04:39:28.194 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.194 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.194 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:28.194 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.561852+0000", 2024-09-18T04:39:28.194 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2045999999999995e-05 2024-09-18T04:39:28.194 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:28.194 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:28.194 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:28.194 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.194 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.194 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:58 2.7 e51/32 2:fc7c6cd3:ns::rbd_data.133e813ab5bc.000000000000000e:head v 51'7, mlcod=51'5)", 2024-09-18T04:39:28.194 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.605333+0000", 2024-09-18T04:39:28.195 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 45.532380263, 2024-09-18T04:39:28.195 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020085902999999999, 2024-09-18T04:39:28.195 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:28.195 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:28.195 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:28.195 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.195 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:28.195 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605333+0000", 2024-09-18T04:39:28.195 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:28.195 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.195 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.195 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:28.195 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605333+0000", 2024-09-18T04:39:28.195 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:28.195 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.196 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.196 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:28.196 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605332+0000", 2024-09-18T04:39:28.196 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:28.196 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.196 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.196 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:28.196 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608667+0000", 2024-09-18T04:39:28.196 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033350300000000001 2024-09-18T04:39:28.196 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.196 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.196 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:28.196 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608706+0000", 2024-09-18T04:39:28.196 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8951e-05 2024-09-18T04:39:28.197 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.197 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.197 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:28.197 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608725+0000", 2024-09-18T04:39:28.197 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9675999999999999e-05 2024-09-18T04:39:28.197 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.197 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.197 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:28.197 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608798+0000", 2024-09-18T04:39:28.197 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3135999999999999e-05 2024-09-18T04:39:28.197 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.197 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.197 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:28.197 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608824+0000", 2024-09-18T04:39:28.198 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5284000000000001e-05 2024-09-18T04:39:28.198 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.198 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.198 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:28.198 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625370+0000", 2024-09-18T04:39:28.198 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016545975000000001 2024-09-18T04:39:28.198 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.198 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.198 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:28.198 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625419+0000", 2024-09-18T04:39:28.198 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9076999999999998e-05 2024-09-18T04:39:28.198 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:28.198 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:28.198 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:28.199 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.199 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.199 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:611 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:28.199 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.069561+0000", 2024-09-18T04:39:28.199 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 37.068151708999999, 2024-09-18T04:39:28.199 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0072330299999999997, 2024-09-18T04:39:28.199 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:28.199 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:28.199 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:28.199 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:39:28.199 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:28.199 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 611 2024-09-18T04:39:28.199 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.199 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:28.200 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.200 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:28.200 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.069561+0000", 2024-09-18T04:39:28.200 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:28.200 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.200 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.200 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:28.200 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.069561+0000", 2024-09-18T04:39:28.200 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:28.200 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.200 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.200 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:28.200 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.069559+0000", 2024-09-18T04:39:28.200 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:28.201 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.201 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.201 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:28.201 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070801+0000", 2024-09-18T04:39:28.201 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0012423359999999999 2024-09-18T04:39:28.201 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.201 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.201 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:28.201 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070828+0000", 2024-09-18T04:39:28.201 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6387e-05 2024-09-18T04:39:28.201 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.201 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.201 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:28.201 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070840+0000", 2024-09-18T04:39:28.202 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2689e-05 2024-09-18T04:39:28.202 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.202 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.202 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:28.202 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070990+0000", 2024-09-18T04:39:28.202 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014946800000000001 2024-09-18T04:39:28.202 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.202 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.202 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:28.202 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.071128+0000", 2024-09-18T04:39:28.202 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013796699999999999 2024-09-18T04:39:28.202 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.202 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.202 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:28.203 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.071318+0000", 2024-09-18T04:39:28.203 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000190106 2024-09-18T04:39:28.203 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.203 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.203 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:28.203 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.075711+0000", 2024-09-18T04:39:28.203 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.004393002 2024-09-18T04:39:28.203 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.203 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.203 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:28.203 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.076665+0000", 2024-09-18T04:39:28.203 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00095390399999999999 2024-09-18T04:39:28.203 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.203 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.203 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:28.204 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.076742+0000", 2024-09-18T04:39:28.204 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7112000000000003e-05 2024-09-18T04:39:28.204 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.204 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.204 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:28.204 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.076794+0000", 2024-09-18T04:39:28.204 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2475999999999997e-05 2024-09-18T04:39:28.204 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:28.204 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:28.204 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:28.204 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.204 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.204 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:687 2.3 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:28.204 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.487994+0000", 2024-09-18T04:39:28.205 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 36.649718726000003, 2024-09-18T04:39:28.205 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0074167690000000001, 2024-09-18T04:39:28.205 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:28.205 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:28.205 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:28.205 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:39:28.205 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:28.205 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 687 2024-09-18T04:39:28.205 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.205 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:28.205 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.205 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:28.205 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.487994+0000", 2024-09-18T04:39:28.205 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:28.206 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.206 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.206 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:28.206 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.487994+0000", 2024-09-18T04:39:28.206 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:28.206 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.206 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.206 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:28.206 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.487993+0000", 2024-09-18T04:39:28.206 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:28.206 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.206 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.206 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:28.206 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.488979+0000", 2024-09-18T04:39:28.207 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00098620700000000001 2024-09-18T04:39:28.207 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.207 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.207 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:28.207 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489002+0000", 2024-09-18T04:39:28.207 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2500000000000001e-05 2024-09-18T04:39:28.207 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.207 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.207 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:28.207 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489020+0000", 2024-09-18T04:39:28.207 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7691000000000001e-05 2024-09-18T04:39:28.207 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.207 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.207 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:28.208 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489091+0000", 2024-09-18T04:39:28.208 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1307e-05 2024-09-18T04:39:28.208 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.208 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.208 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:28.208 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489219+0000", 2024-09-18T04:39:28.208 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012766000000000001 2024-09-18T04:39:28.208 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.208 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.208 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:28.208 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489387+0000", 2024-09-18T04:39:28.208 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016855400000000001 2024-09-18T04:39:28.208 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.208 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.209 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:28.209 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495243+0000", 2024-09-18T04:39:28.209 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058563189999999996 2024-09-18T04:39:28.209 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.209 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.209 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:28.209 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495266+0000", 2024-09-18T04:39:28.209 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2810999999999999e-05 2024-09-18T04:39:28.209 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.209 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.209 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:28.209 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495332+0000", 2024-09-18T04:39:28.209 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5469999999999995e-05 2024-09-18T04:39:28.210 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.210 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.210 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:28.210 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495411+0000", 2024-09-18T04:39:28.210 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9395999999999997e-05 2024-09-18T04:39:28.210 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:28.210 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:28.210 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:28.210 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.210 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.210 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:689 2.3 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:28.210 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.500070+0000", 2024-09-18T04:39:28.210 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 36.637642909999997, 2024-09-18T04:39:28.210 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071096650000000003, 2024-09-18T04:39:28.211 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:28.211 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:28.211 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:28.211 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:39:28.211 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:28.211 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 689 2024-09-18T04:39:28.211 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.211 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:28.211 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.211 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:28.211 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.500070+0000", 2024-09-18T04:39:28.211 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:28.211 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.211 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.212 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:28.212 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.500070+0000", 2024-09-18T04:39:28.212 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:28.212 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.212 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.212 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:28.212 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.500069+0000", 2024-09-18T04:39:28.212 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:28.212 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.212 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.212 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:28.212 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501082+0000", 2024-09-18T04:39:28.212 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010136570000000001 2024-09-18T04:39:28.213 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.213 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.213 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:28.213 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501104+0000", 2024-09-18T04:39:28.213 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1883e-05 2024-09-18T04:39:28.213 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.213 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.213 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:28.213 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501115+0000", 2024-09-18T04:39:28.213 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0414000000000001e-05 2024-09-18T04:39:28.213 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.213 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.213 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:28.213 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501178+0000", 2024-09-18T04:39:28.213 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3128999999999998e-05 2024-09-18T04:39:28.214 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.214 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.214 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:28.214 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501257+0000", 2024-09-18T04:39:28.214 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9592999999999995e-05 2024-09-18T04:39:28.214 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.214 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.214 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:28.214 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501366+0000", 2024-09-18T04:39:28.214 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000108643 2024-09-18T04:39:28.214 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.214 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.214 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:28.214 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.504041+0000", 2024-09-18T04:39:28.215 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0026750889999999999 2024-09-18T04:39:28.215 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.215 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.215 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:28.215 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.507097+0000", 2024-09-18T04:39:28.215 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030556070000000001 2024-09-18T04:39:28.215 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.215 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.215 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:28.215 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.507140+0000", 2024-09-18T04:39:28.215 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3745000000000002e-05 2024-09-18T04:39:28.215 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.215 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.215 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:28.216 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.507180+0000", 2024-09-18T04:39:28.216 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9475999999999999e-05 2024-09-18T04:39:28.216 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:28.216 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:28.216 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:28.216 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.216 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.216 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4972.0:13 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:28.216 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.948263+0000", 2024-09-18T04:39:28.216 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 36.189450119, 2024-09-18T04:39:28.216 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0091358970000000005, 2024-09-18T04:39:28.216 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:28.216 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:39:28.216 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:28.217 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4972", 2024-09-18T04:39:28.217 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2280251472", 2024-09-18T04:39:28.217 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 13 2024-09-18T04:39:28.217 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.217 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:28.217 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.217 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:28.217 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948263+0000", 2024-09-18T04:39:28.217 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:28.217 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.217 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.217 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:28.217 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948263+0000", 2024-09-18T04:39:28.217 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:28.218 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.218 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.218 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:28.218 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948261+0000", 2024-09-18T04:39:28.218 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:28.218 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.218 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.218 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:28.218 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948273+0000", 2024-09-18T04:39:28.218 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1576999999999999e-05 2024-09-18T04:39:28.218 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.218 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.218 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:28.218 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948305+0000", 2024-09-18T04:39:28.219 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2209000000000003e-05 2024-09-18T04:39:28.219 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.219 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.219 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:28.219 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948325+0000", 2024-09-18T04:39:28.219 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.022e-05 2024-09-18T04:39:28.219 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.219 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.219 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:28.219 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948455+0000", 2024-09-18T04:39:28.219 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000129624 2024-09-18T04:39:28.219 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.219 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.219 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:28.219 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948598+0000", 2024-09-18T04:39:28.220 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014376699999999999 2024-09-18T04:39:28.220 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.220 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.220 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:28.220 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.957399+0000", 2024-09-18T04:39:28.220 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0088004299999999997 2024-09-18T04:39:28.220 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:28.220 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:28.220 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:28.220 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.220 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.220 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:101 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:28.220 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.576367+0000", 2024-09-18T04:39:28.220 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 31.561346410999999, 2024-09-18T04:39:28.221 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0078671020000000008, 2024-09-18T04:39:28.221 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:28.221 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:28.221 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:28.221 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:28.221 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:28.221 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 101 2024-09-18T04:39:28.221 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.221 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:28.221 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.221 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:28.221 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576367+0000", 2024-09-18T04:39:28.221 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:28.221 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.222 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.222 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:28.222 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576367+0000", 2024-09-18T04:39:28.222 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:28.222 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.222 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.222 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:28.222 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576364+0000", 2024-09-18T04:39:28.222 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999971 2024-09-18T04:39:28.222 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.222 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.222 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:28.222 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576378+0000", 2024-09-18T04:39:28.222 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3879999999999999e-05 2024-09-18T04:39:28.223 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.223 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.223 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:28.223 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576410+0000", 2024-09-18T04:39:28.223 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2286000000000002e-05 2024-09-18T04:39:28.223 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.223 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.223 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:28.223 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576425+0000", 2024-09-18T04:39:28.223 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5290000000000001e-05 2024-09-18T04:39:28.223 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.223 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.223 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:28.223 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576551+0000", 2024-09-18T04:39:28.224 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012555199999999999 2024-09-18T04:39:28.224 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.224 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.224 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:28.224 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576719+0000", 2024-09-18T04:39:28.224 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016787400000000001 2024-09-18T04:39:28.224 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.224 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.224 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:28.224 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576951+0000", 2024-09-18T04:39:28.224 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00023252400000000001 2024-09-18T04:39:28.224 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.224 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.224 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:28.224 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.581459+0000", 2024-09-18T04:39:28.225 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045074410000000001 2024-09-18T04:39:28.225 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.225 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.225 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:28.225 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584076+0000", 2024-09-18T04:39:28.225 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0026173860000000002 2024-09-18T04:39:28.225 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.225 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.225 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:28.225 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584160+0000", 2024-09-18T04:39:28.225 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4191999999999996e-05 2024-09-18T04:39:28.225 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.225 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.226 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:28.226 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584234+0000", 2024-09-18T04:39:28.226 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.339e-05 2024-09-18T04:39:28.226 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:28.226 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:28.226 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:28.226 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.226 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.226 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:44 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:28.226 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.550448+0000", 2024-09-18T04:39:28.226 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 27.587265221999999, 2024-09-18T04:39:28.226 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082117590000000008, 2024-09-18T04:39:28.226 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:28.226 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:28.227 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:28.227 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:39:28.227 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:28.227 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 44 2024-09-18T04:39:28.227 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.227 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:28.227 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.227 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:28.227 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.550448+0000", 2024-09-18T04:39:28.227 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:28.227 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.227 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.227 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:28.227 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.550448+0000", 2024-09-18T04:39:28.227 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:28.228 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.228 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.228 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:28.228 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.550445+0000", 2024-09-18T04:39:28.228 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999971 2024-09-18T04:39:28.228 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.228 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.228 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:28.228 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.551904+0000", 2024-09-18T04:39:28.228 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001458322 2024-09-18T04:39:28.228 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.228 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.228 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:28.228 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.551938+0000", 2024-09-18T04:39:28.229 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4359000000000001e-05 2024-09-18T04:39:28.229 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.229 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.229 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:28.229 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.551959+0000", 2024-09-18T04:39:28.229 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1403000000000002e-05 2024-09-18T04:39:28.229 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.229 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.229 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:28.229 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.552049+0000", 2024-09-18T04:39:28.229 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9970999999999996e-05 2024-09-18T04:39:28.229 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.229 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.229 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:28.230 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.552190+0000", 2024-09-18T04:39:28.230 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000140271 2024-09-18T04:39:28.230 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.230 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.230 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:28.230 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.552385+0000", 2024-09-18T04:39:28.230 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019498700000000001 2024-09-18T04:39:28.230 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.230 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.230 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:28.230 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.556308+0000", 2024-09-18T04:39:28.230 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00392333 2024-09-18T04:39:28.230 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.230 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.231 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:28.231 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.558479+0000", 2024-09-18T04:39:28.231 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0021715889999999998 2024-09-18T04:39:28.231 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.231 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.231 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:28.231 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.558616+0000", 2024-09-18T04:39:28.231 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013613399999999999 2024-09-18T04:39:28.231 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.231 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.231 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:28.231 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.558660+0000", 2024-09-18T04:39:28.231 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4053e-05 2024-09-18T04:39:28.231 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:28.232 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:28.232 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:28.232 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.232 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.232 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:46 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:28.232 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.564089+0000", 2024-09-18T04:39:28.232 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 27.573624092999999, 2024-09-18T04:39:28.232 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0069437090000000002, 2024-09-18T04:39:28.232 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:28.232 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:28.232 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:28.232 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:39:28.232 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:28.232 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 46 2024-09-18T04:39:28.233 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.233 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:28.233 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.233 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:28.233 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.564089+0000", 2024-09-18T04:39:28.233 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:28.233 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.233 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.233 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:28.233 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.564089+0000", 2024-09-18T04:39:28.233 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:28.233 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.233 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.233 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:28.234 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.564087+0000", 2024-09-18T04:39:28.234 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:28.234 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.234 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.234 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:28.234 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565126+0000", 2024-09-18T04:39:28.234 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001038902 2024-09-18T04:39:28.234 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.234 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.234 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:28.234 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565148+0000", 2024-09-18T04:39:28.234 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1911000000000001e-05 2024-09-18T04:39:28.234 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.234 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.235 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:28.235 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565157+0000", 2024-09-18T04:39:28.235 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6220000000000004e-06 2024-09-18T04:39:28.235 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.235 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.235 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:28.235 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565261+0000", 2024-09-18T04:39:28.235 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000103063 2024-09-18T04:39:28.235 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.235 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.235 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:28.235 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565394+0000", 2024-09-18T04:39:28.235 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000133032 2024-09-18T04:39:28.235 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.235 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.236 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:28.236 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565564+0000", 2024-09-18T04:39:28.236 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017068000000000001 2024-09-18T04:39:28.236 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.236 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.236 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:28.236 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.570556+0000", 2024-09-18T04:39:28.236 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0049920620000000002 2024-09-18T04:39:28.236 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.236 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.236 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:28.236 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.570885+0000", 2024-09-18T04:39:28.236 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032827900000000001 2024-09-18T04:39:28.236 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.237 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.237 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:28.237 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.570959+0000", 2024-09-18T04:39:28.237 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3955999999999995e-05 2024-09-18T04:39:28.237 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.237 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.237 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:28.237 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.571033+0000", 2024-09-18T04:39:28.237 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4191999999999997e-05 2024-09-18T04:39:28.237 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:28.237 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:28.237 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:28.238 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.238 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.238 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:100 2.4 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:28.238 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.856389+0000", 2024-09-18T04:39:28.238 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 27.281323981, 2024-09-18T04:39:28.238 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0081480800000000003, 2024-09-18T04:39:28.238 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:28.238 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:28.238 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:28.238 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:39:28.238 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:28.238 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 100 2024-09-18T04:39:28.239 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.239 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:28.239 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.239 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:28.239 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.856389+0000", 2024-09-18T04:39:28.239 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:28.239 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.239 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.239 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:28.239 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.856389+0000", 2024-09-18T04:39:28.239 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:28.239 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.239 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.240 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:28.240 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.856388+0000", 2024-09-18T04:39:28.240 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:28.240 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.240 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.240 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:28.240 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857295+0000", 2024-09-18T04:39:28.240 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00090725599999999995 2024-09-18T04:39:28.240 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.240 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.240 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:28.240 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857311+0000", 2024-09-18T04:39:28.240 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6092000000000001e-05 2024-09-18T04:39:28.240 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.241 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.241 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:28.241 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857319+0000", 2024-09-18T04:39:28.241 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9200000000000004e-06 2024-09-18T04:39:28.241 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.241 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.241 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:28.241 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857388+0000", 2024-09-18T04:39:28.241 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8591999999999997e-05 2024-09-18T04:39:28.241 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.241 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.241 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:28.241 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857535+0000", 2024-09-18T04:39:28.241 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014684299999999999 2024-09-18T04:39:28.242 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.242 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.242 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:28.242 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857747+0000", 2024-09-18T04:39:28.242 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000212361 2024-09-18T04:39:28.242 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.242 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.242 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:28.242 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.860831+0000", 2024-09-18T04:39:28.242 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030841789999999998 2024-09-18T04:39:28.242 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.242 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.242 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:28.242 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.864361+0000", 2024-09-18T04:39:28.243 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0035298780000000002 2024-09-18T04:39:28.243 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.243 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.243 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:28.243 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.864456+0000", 2024-09-18T04:39:28.243 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4557999999999998e-05 2024-09-18T04:39:28.243 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.243 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.243 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:28.243 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.864537+0000", 2024-09-18T04:39:28.243 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1577999999999997e-05 2024-09-18T04:39:28.243 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:28.243 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:28.243 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:28.243 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.244 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.244 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:102 2.4 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:28.244 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.869773+0000", 2024-09-18T04:39:28.244 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 27.267940324000001, 2024-09-18T04:39:28.244 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0084430359999999992, 2024-09-18T04:39:28.244 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:28.244 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:28.244 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:28.244 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:39:28.244 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:28.244 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 102 2024-09-18T04:39:28.244 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.244 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:28.244 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.245 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:28.245 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.869773+0000", 2024-09-18T04:39:28.245 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:28.245 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.245 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.245 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:28.245 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.869773+0000", 2024-09-18T04:39:28.245 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:28.245 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.245 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.245 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:28.245 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.869771+0000", 2024-09-18T04:39:28.245 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:28.245 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.246 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.246 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:28.246 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870629+0000", 2024-09-18T04:39:28.246 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00085760999999999997 2024-09-18T04:39:28.246 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.246 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.246 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:28.246 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870647+0000", 2024-09-18T04:39:28.246 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8363999999999999e-05 2024-09-18T04:39:28.246 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.246 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.246 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:28.246 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870655+0000", 2024-09-18T04:39:28.246 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0099999999999995e-06 2024-09-18T04:39:28.247 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.247 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.247 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:28.247 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870803+0000", 2024-09-18T04:39:28.247 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014798 2024-09-18T04:39:28.247 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.247 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.247 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:28.247 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870951+0000", 2024-09-18T04:39:28.247 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014715999999999999 2024-09-18T04:39:28.247 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.247 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.247 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:28.247 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.871145+0000", 2024-09-18T04:39:28.248 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000194253 2024-09-18T04:39:28.248 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.248 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.248 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:28.248 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.877033+0000", 2024-09-18T04:39:28.248 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058883099999999999 2024-09-18T04:39:28.248 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.248 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.248 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:28.248 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.878078+0000", 2024-09-18T04:39:28.248 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001044747 2024-09-18T04:39:28.248 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.248 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.248 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:28.248 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.878160+0000", 2024-09-18T04:39:28.249 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1904999999999996e-05 2024-09-18T04:39:28.249 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.249 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.249 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:28.249 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.878216+0000", 2024-09-18T04:39:28.249 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6082000000000003e-05 2024-09-18T04:39:28.249 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:28.249 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:28.249 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:28.249 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.249 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.249 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:145 2.4 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:28.249 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:01.096935+0000", 2024-09-18T04:39:28.249 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 27.040778543999998, 2024-09-18T04:39:28.250 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0069937869999999996, 2024-09-18T04:39:28.250 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:28.250 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:28.250 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:28.250 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:39:28.250 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:28.250 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 145 2024-09-18T04:39:28.250 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.250 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:28.250 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.250 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:28.250 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.096935+0000", 2024-09-18T04:39:28.250 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:28.250 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.251 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.251 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:28.251 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.096935+0000", 2024-09-18T04:39:28.251 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:28.251 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.251 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.251 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:28.251 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.096932+0000", 2024-09-18T04:39:28.251 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:28.251 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.251 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.251 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:28.251 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097815+0000", 2024-09-18T04:39:28.251 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00088276000000000001 2024-09-18T04:39:28.252 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.252 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.252 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:28.252 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097842+0000", 2024-09-18T04:39:28.252 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6806999999999999e-05 2024-09-18T04:39:28.252 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.252 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.252 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:28.252 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097852+0000", 2024-09-18T04:39:28.252 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0548e-05 2024-09-18T04:39:28.252 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.252 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.252 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:28.253 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097931+0000", 2024-09-18T04:39:28.253 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8417000000000005e-05 2024-09-18T04:39:28.253 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.253 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.253 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:28.253 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.098047+0000", 2024-09-18T04:39:28.253 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011611 2024-09-18T04:39:28.253 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.253 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.253 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:28.253 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.098172+0000", 2024-09-18T04:39:28.253 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000125482 2024-09-18T04:39:28.253 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.253 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.254 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:28.254 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.100888+0000", 2024-09-18T04:39:28.254 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0027154430000000001 2024-09-18T04:39:28.254 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.254 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.254 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:28.254 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.103814+0000", 2024-09-18T04:39:28.254 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0029259619999999998 2024-09-18T04:39:28.254 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.254 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.254 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:28.254 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.103891+0000", 2024-09-18T04:39:28.254 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7619000000000003e-05 2024-09-18T04:39:28.254 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:28.254 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:28.255 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:28.255 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.103928+0000", 2024-09-18T04:39:28.255 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6900999999999998e-05 2024-09-18T04:39:28.255 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:28.255 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:28.255 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:28.255 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:28.255 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:28.255 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:28.257 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:39:28.257 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:28.252+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:28.357 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:28.353+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:28.370 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:28.370 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:28.370 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:39:28.370 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:28.381 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:39:28.458 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:28.453+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:28.480 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:28.480 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:28.480 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:39:28.480 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:39:28.480 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:28.489 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:39:28.558 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:28.553+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:28.596 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:28.596 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:39:28.596 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:39:28.596 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:39:28.596 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.596 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:37 2.5 e51/28 2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:head v 51'1, mlcod=0'0)", 2024-09-18T04:39:28.596 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:39:28.597 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 46.300310592000002, 2024-09-18T04:39:28.597 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012279973, 2024-09-18T04:39:28.597 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:28.597 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:28.597 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:28.597 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.597 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:28.597 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:39:28.597 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:28.597 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.597 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.597 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:28.597 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:39:28.597 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:28.598 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.598 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.598 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:28.598 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292468+0000", 2024-09-18T04:39:28.598 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:28.598 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.598 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.598 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:28.598 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295837+0000", 2024-09-18T04:39:28.598 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0033691699999999999 2024-09-18T04:39:28.598 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.598 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.598 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:28.598 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295860+0000", 2024-09-18T04:39:28.598 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2756999999999999e-05 2024-09-18T04:39:28.599 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.599 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.599 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:28.599 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295872+0000", 2024-09-18T04:39:28.599 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2442e-05 2024-09-18T04:39:28.599 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.599 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.599 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:28.599 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295934+0000", 2024-09-18T04:39:28.599 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1187999999999998e-05 2024-09-18T04:39:28.599 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.599 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.599 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:28.599 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295967+0000", 2024-09-18T04:39:28.600 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3185999999999997e-05 2024-09-18T04:39:28.600 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.600 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.600 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:28.600 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304697+0000", 2024-09-18T04:39:28.600 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0087298059999999997 2024-09-18T04:39:28.600 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.600 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.600 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:28.600 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304749+0000", 2024-09-18T04:39:28.600 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1922000000000002e-05 2024-09-18T04:39:28.600 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:28.600 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:28.600 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:28.601 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.601 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.601 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:49 2.5 e51/28 2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head v 51'2, mlcod=0'0)", 2024-09-18T04:39:28.601 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:39:28.601 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 46.266983754000002, 2024-09-18T04:39:28.601 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018033041999999999, 2024-09-18T04:39:28.601 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:28.601 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:28.601 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:28.601 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.601 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:28.601 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:39:28.601 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:28.601 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.601 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.602 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:28.602 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:39:28.602 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:28.602 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.602 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.602 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:28.602 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325793+0000", 2024-09-18T04:39:28.602 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:28.602 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.602 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.602 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:28.603 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329038+0000", 2024-09-18T04:39:28.603 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0032444010000000001 2024-09-18T04:39:28.603 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.603 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.603 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:28.603 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329065+0000", 2024-09-18T04:39:28.603 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7500000000000001e-05 2024-09-18T04:39:28.603 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.603 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.603 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:28.603 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329077+0000", 2024-09-18T04:39:28.603 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1919999999999999e-05 2024-09-18T04:39:28.603 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.603 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.604 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:28.604 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329149+0000", 2024-09-18T04:39:28.604 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2349999999999997e-05 2024-09-18T04:39:28.604 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.604 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.604 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:28.604 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329192+0000", 2024-09-18T04:39:28.604 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2744000000000003e-05 2024-09-18T04:39:28.604 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.604 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.604 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:28.604 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343773+0000", 2024-09-18T04:39:28.604 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014581246000000001 2024-09-18T04:39:28.604 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.605 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.605 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:28.605 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343829+0000", 2024-09-18T04:39:28.605 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5143999999999999e-05 2024-09-18T04:39:28.605 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:28.605 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:28.605 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:28.605 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.605 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.605 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:47 2.4 e51/17 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 51'3, mlcod=51'1)", 2024-09-18T04:39:28.605 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:39:28.605 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 46.266403488999998, 2024-09-18T04:39:28.605 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018960186, 2024-09-18T04:39:28.605 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:28.605 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:28.606 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:28.606 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.606 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:28.606 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:39:28.606 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:28.606 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.606 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.606 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:28.606 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:39:28.606 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:28.606 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.606 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.606 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:28.606 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326374+0000", 2024-09-18T04:39:28.607 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:28.607 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.607 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.607 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:28.607 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330883+0000", 2024-09-18T04:39:28.607 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0045084740000000002 2024-09-18T04:39:28.607 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.607 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.607 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:28.607 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330901+0000", 2024-09-18T04:39:28.607 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7890999999999999e-05 2024-09-18T04:39:28.607 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.607 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.607 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:28.607 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330912+0000", 2024-09-18T04:39:28.607 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1041e-05 2024-09-18T04:39:28.608 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.608 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.608 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:28.608 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.331003+0000", 2024-09-18T04:39:28.608 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1323999999999995e-05 2024-09-18T04:39:28.608 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.608 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.608 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:28.608 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.331030+0000", 2024-09-18T04:39:28.608 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7351999999999998e-05 2024-09-18T04:39:28.608 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.608 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.608 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:28.608 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345287+0000", 2024-09-18T04:39:28.609 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014256332 2024-09-18T04:39:28.609 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.609 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.609 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:28.609 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345336+0000", 2024-09-18T04:39:28.609 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9356e-05 2024-09-18T04:39:28.609 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:28.609 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:28.609 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:28.609 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.609 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.609 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:58 2.5 e51/28 2:b227e6b1:::rbd_data.125fc09df6d1.000000000000000e:head v 51'3, mlcod=51'1)", 2024-09-18T04:39:28.609 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:39:28.609 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 46.225976862000003, 2024-09-18T04:39:28.609 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016513678, 2024-09-18T04:39:28.610 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:28.610 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:28.610 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:28.610 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.610 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:28.610 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:39:28.610 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:28.610 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.610 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.610 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:28.610 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:39:28.610 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:28.610 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.610 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.611 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:28.611 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366800+0000", 2024-09-18T04:39:28.611 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:28.611 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.611 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.611 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:28.611 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368644+0000", 2024-09-18T04:39:28.611 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001844399 2024-09-18T04:39:28.611 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.611 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.611 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:28.611 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368681+0000", 2024-09-18T04:39:28.611 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6526999999999998e-05 2024-09-18T04:39:28.611 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.612 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.612 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:28.612 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368698+0000", 2024-09-18T04:39:28.612 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7272999999999999e-05 2024-09-18T04:39:28.612 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.612 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.612 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:28.612 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368773+0000", 2024-09-18T04:39:28.612 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4862000000000003e-05 2024-09-18T04:39:28.612 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.612 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.612 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:28.612 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368801+0000", 2024-09-18T04:39:28.612 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7925000000000001e-05 2024-09-18T04:39:28.612 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.613 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.613 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:28.613 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383261+0000", 2024-09-18T04:39:28.613 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014460377999999999 2024-09-18T04:39:28.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.613 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.613 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:28.613 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383316+0000", 2024-09-18T04:39:28.613 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4623999999999997e-05 2024-09-18T04:39:28.613 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:28.613 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:28.613 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:28.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.613 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.614 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:56 2.4 e51/17 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 51'4, mlcod=51'2)", 2024-09-18T04:39:28.614 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:39:28.614 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 46.223972836000002, 2024-09-18T04:39:28.614 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015105221, 2024-09-18T04:39:28.614 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:28.614 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:28.614 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:28.614 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.614 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:28.614 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:39:28.614 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:28.614 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.614 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.614 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:28.614 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:39:28.615 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:28.615 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.615 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.615 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:28.615 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368805+0000", 2024-09-18T04:39:28.615 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:28.615 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.615 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.615 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:28.615 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371603+0000", 2024-09-18T04:39:28.615 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027982660000000002 2024-09-18T04:39:28.615 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.615 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.615 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:28.616 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371633+0000", 2024-09-18T04:39:28.616 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0073000000000002e-05 2024-09-18T04:39:28.616 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.616 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.616 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:28.616 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371652+0000", 2024-09-18T04:39:28.616 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8597000000000002e-05 2024-09-18T04:39:28.616 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.616 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.616 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:28.616 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371752+0000", 2024-09-18T04:39:28.616 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9772999999999995e-05 2024-09-18T04:39:28.616 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.616 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.617 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:28.617 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371793+0000", 2024-09-18T04:39:28.617 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1173e-05 2024-09-18T04:39:28.617 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.617 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.617 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:28.617 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383875+0000", 2024-09-18T04:39:28.617 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012081852000000001 2024-09-18T04:39:28.617 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.617 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.617 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:28.617 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383912+0000", 2024-09-18T04:39:28.617 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7045000000000002e-05 2024-09-18T04:39:28.617 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:28.617 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:28.618 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:28.618 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.618 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.618 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:36 2.5 e51/28 2:bd5b8dd8:ns::rbd_data.133e813ab5bc.0000000000000002:head v 51'4, mlcod=51'2)", 2024-09-18T04:39:28.618 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:39:28.618 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 46.055253845000003, 2024-09-18T04:39:28.618 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020758161000000001, 2024-09-18T04:39:28.618 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:28.618 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:28.618 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:28.618 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.618 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:28.618 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:39:28.618 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:28.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.619 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.619 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:28.619 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:39:28.619 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:28.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.619 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.619 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:28.619 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537523+0000", 2024-09-18T04:39:28.619 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:28.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.619 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.619 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:28.619 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541578+0000", 2024-09-18T04:39:28.619 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0040549710000000001 2024-09-18T04:39:28.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.620 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.620 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:28.620 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541608+0000", 2024-09-18T04:39:28.620 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9349e-05 2024-09-18T04:39:28.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.620 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.620 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:28.620 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541621+0000", 2024-09-18T04:39:28.620 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2736e-05 2024-09-18T04:39:28.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.620 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.620 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:28.620 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541715+0000", 2024-09-18T04:39:28.620 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4197999999999995e-05 2024-09-18T04:39:28.621 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.621 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.621 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:28.621 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541755+0000", 2024-09-18T04:39:28.621 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9818999999999999e-05 2024-09-18T04:39:28.621 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.621 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.621 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:28.621 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558217+0000", 2024-09-18T04:39:28.621 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016462234999999999 2024-09-18T04:39:28.621 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.621 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.621 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:28.621 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558284+0000", 2024-09-18T04:39:28.622 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6774999999999997e-05 2024-09-18T04:39:28.622 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:28.622 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:28.622 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:28.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.622 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.622 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:39 2.4 e51/17 2:304364d1:ns::rbd_data.133e813ab5bc.0000000000000004:head v 51'5, mlcod=51'3)", 2024-09-18T04:39:28.622 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:39:28.622 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 46.047114718000003, 2024-09-18T04:39:28.622 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016777558000000001, 2024-09-18T04:39:28.622 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:28.622 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:28.622 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:28.622 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.622 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:28.623 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:39:28.623 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:28.623 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.623 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.623 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:28.623 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:39:28.623 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:28.623 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.623 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.623 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:28.623 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545663+0000", 2024-09-18T04:39:28.623 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:28.623 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.623 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.623 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:28.624 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548286+0000", 2024-09-18T04:39:28.624 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026233390000000001 2024-09-18T04:39:28.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.624 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.624 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:28.624 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548305+0000", 2024-09-18T04:39:28.624 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8425000000000001e-05 2024-09-18T04:39:28.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.624 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.624 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:28.624 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548316+0000", 2024-09-18T04:39:28.624 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1163e-05 2024-09-18T04:39:28.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.624 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.625 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:28.625 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548379+0000", 2024-09-18T04:39:28.625 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2780000000000005e-05 2024-09-18T04:39:28.625 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.625 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.625 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:28.625 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548399+0000", 2024-09-18T04:39:28.625 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0080000000000001e-05 2024-09-18T04:39:28.625 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.625 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.625 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:28.625 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562383+0000", 2024-09-18T04:39:28.625 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013984671000000001 2024-09-18T04:39:28.625 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.625 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.626 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:28.626 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562442+0000", 2024-09-18T04:39:28.626 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8700999999999999e-05 2024-09-18T04:39:28.626 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:28.626 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:28.626 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:28.626 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.626 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.626 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:37 2.5 e51/28 2:bf39b8e4:ns::rbd_data.133e813ab5bc.0000000000000003:head v 51'5, mlcod=51'2)", 2024-09-18T04:39:28.626 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:39:28.626 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 46.044246448999999, 2024-09-18T04:39:28.626 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014264492, 2024-09-18T04:39:28.626 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:28.626 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:28.627 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:28.627 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.627 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:28.627 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:39:28.627 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:28.627 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.627 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.627 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:28.627 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:39:28.627 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:28.627 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.627 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.627 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:28.627 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548531+0000", 2024-09-18T04:39:28.627 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:28.628 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.628 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.628 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:28.628 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551500+0000", 2024-09-18T04:39:28.628 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0029688129999999998 2024-09-18T04:39:28.628 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.628 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.628 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:28.628 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551516+0000", 2024-09-18T04:39:28.628 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5994e-05 2024-09-18T04:39:28.628 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.628 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.628 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:28.628 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551526+0000", 2024-09-18T04:39:28.629 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9299999999999998e-06 2024-09-18T04:39:28.629 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.629 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.629 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:28.629 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551577+0000", 2024-09-18T04:39:28.629 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0871000000000001e-05 2024-09-18T04:39:28.629 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.629 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.629 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:28.629 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551602+0000", 2024-09-18T04:39:28.629 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4813000000000001e-05 2024-09-18T04:39:28.629 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.629 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.629 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:28.629 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562741+0000", 2024-09-18T04:39:28.630 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011139428999999999 2024-09-18T04:39:28.630 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.630 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.630 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:28.630 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562797+0000", 2024-09-18T04:39:28.630 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6329000000000002e-05 2024-09-18T04:39:28.630 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:28.630 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:28.630 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:28.630 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.630 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.630 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:49 2.4 e51/17 2:20f0b0c9:ns::rbd_data.133e813ab5bc.0000000000000009:head v 51'6, mlcod=51'4)", 2024-09-18T04:39:28.630 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:39:28.630 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 46.012811890999998, 2024-09-18T04:39:28.631 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013849046, 2024-09-18T04:39:28.631 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:28.631 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:28.631 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:28.631 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.631 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:28.631 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:39:28.631 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:28.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.631 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.631 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:28.631 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:39:28.631 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:28.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.631 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.632 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:28.632 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579966+0000", 2024-09-18T04:39:28.632 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:28.632 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.632 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.632 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:28.632 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582324+0000", 2024-09-18T04:39:28.632 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023578890000000002 2024-09-18T04:39:28.632 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.632 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.632 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:28.632 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582355+0000", 2024-09-18T04:39:28.632 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1485000000000001e-05 2024-09-18T04:39:28.632 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.632 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.633 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:28.633 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582371+0000", 2024-09-18T04:39:28.633 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6147999999999999e-05 2024-09-18T04:39:28.633 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.633 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.633 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:28.633 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582467+0000", 2024-09-18T04:39:28.633 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6059999999999998e-05 2024-09-18T04:39:28.633 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.633 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.633 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:28.633 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582505+0000", 2024-09-18T04:39:28.633 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7679000000000003e-05 2024-09-18T04:39:28.633 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.634 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.634 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:28.634 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593743+0000", 2024-09-18T04:39:28.634 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011238356 2024-09-18T04:39:28.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.634 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.634 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:28.634 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593816+0000", 2024-09-18T04:39:28.634 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2935000000000006e-05 2024-09-18T04:39:28.634 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:28.634 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:28.634 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:28.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.634 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.634 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:53 2.4 e51/17 2:39f7f174:ns::rbd_data.133e813ab5bc.000000000000000b:head v 51'7, mlcod=51'4)", 2024-09-18T04:39:28.635 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:39:28.635 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 45.998110736000001, 2024-09-18T04:39:28.635 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0079914159999999995, 2024-09-18T04:39:28.635 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:28.635 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:28.635 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:28.635 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.635 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:28.635 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:39:28.635 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:28.635 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.635 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.635 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:28.635 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:39:28.635 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:28.636 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.636 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.636 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:28.636 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594667+0000", 2024-09-18T04:39:28.636 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:28.636 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.636 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.636 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:28.636 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596446+0000", 2024-09-18T04:39:28.636 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001779033 2024-09-18T04:39:28.636 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.636 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.636 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:28.636 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596472+0000", 2024-09-18T04:39:28.637 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6120000000000001e-05 2024-09-18T04:39:28.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.637 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.637 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:28.637 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596500+0000", 2024-09-18T04:39:28.637 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7518e-05 2024-09-18T04:39:28.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.637 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.637 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:28.637 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596564+0000", 2024-09-18T04:39:28.637 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3880999999999994e-05 2024-09-18T04:39:28.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.637 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.637 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:28.637 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596596+0000", 2024-09-18T04:39:28.638 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2638999999999997e-05 2024-09-18T04:39:28.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.638 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.638 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:28.638 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602605+0000", 2024-09-18T04:39:28.638 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0060080699999999999 2024-09-18T04:39:28.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.638 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.638 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:28.638 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602660+0000", 2024-09-18T04:39:28.638 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5396000000000003e-05 2024-09-18T04:39:28.638 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:28.638 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:28.638 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:28.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.639 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.639 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:60 2.5 e51/28 2:a06f51e4:ns::rbd_data.133e813ab5bc.000000000000000f:head v 51'6, mlcod=51'4)", 2024-09-18T04:39:28.639 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:39:28.639 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 45.983293750999998, 2024-09-18T04:39:28.639 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0093954989999999999, 2024-09-18T04:39:28.639 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:28.639 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:28.639 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:28.639 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.639 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:28.640 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:39:28.640 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:28.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.640 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.640 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:28.640 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:39:28.640 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:28.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.640 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.640 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:28.640 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609483+0000", 2024-09-18T04:39:28.640 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:28.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.640 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.641 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:28.641 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611767+0000", 2024-09-18T04:39:28.641 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002283724 2024-09-18T04:39:28.641 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.641 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.641 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:28.641 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611791+0000", 2024-09-18T04:39:28.641 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3547e-05 2024-09-18T04:39:28.641 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.641 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.641 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:28.641 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611800+0000", 2024-09-18T04:39:28.641 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4979999999999995e-06 2024-09-18T04:39:28.641 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.642 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.642 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:28.642 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611864+0000", 2024-09-18T04:39:28.642 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3974999999999996e-05 2024-09-18T04:39:28.642 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.642 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.642 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:28.642 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611887+0000", 2024-09-18T04:39:28.642 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2633000000000002e-05 2024-09-18T04:39:28.642 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.642 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.642 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:28.642 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.618777+0000", 2024-09-18T04:39:28.642 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0068897979999999999 2024-09-18T04:39:28.643 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.643 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.643 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:28.643 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.618881+0000", 2024-09-18T04:39:28.643 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010437699999999999 2024-09-18T04:39:28.643 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:28.643 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:28.643 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:28.643 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.643 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.643 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:663 2.4 e57/17 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 57'13, mlcod=57'11)", 2024-09-18T04:39:28.643 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.355300+0000", 2024-09-18T04:39:28.643 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 37.237478815999999, 2024-09-18T04:39:28.643 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0049192760000000002, 2024-09-18T04:39:28.643 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:28.644 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:28.644 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:28.644 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.644 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:28.644 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355300+0000", 2024-09-18T04:39:28.644 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:28.644 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.644 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.644 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:28.644 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355300+0000", 2024-09-18T04:39:28.644 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:28.644 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.645 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.645 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:28.645 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355299+0000", 2024-09-18T04:39:28.645 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:28.645 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.645 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.645 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:28.645 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355876+0000", 2024-09-18T04:39:28.645 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00057730900000000003 2024-09-18T04:39:28.645 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.645 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.645 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:28.645 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355897+0000", 2024-09-18T04:39:28.645 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0477000000000001e-05 2024-09-18T04:39:28.645 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.646 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.646 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:28.646 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355905+0000", 2024-09-18T04:39:28.646 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5739999999999996e-06 2024-09-18T04:39:28.646 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.646 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.646 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:28.646 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355989+0000", 2024-09-18T04:39:28.646 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4002e-05 2024-09-18T04:39:28.646 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.646 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.646 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:28.646 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.356041+0000", 2024-09-18T04:39:28.646 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1347000000000002e-05 2024-09-18T04:39:28.646 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.647 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.647 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:28.647 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.360172+0000", 2024-09-18T04:39:28.647 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0041315090000000002 2024-09-18T04:39:28.647 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.647 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.647 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:28.647 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.360220+0000", 2024-09-18T04:39:28.647 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7584999999999999e-05 2024-09-18T04:39:28.647 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:28.647 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:28.647 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:28.647 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.647 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.647 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:679 2.5 e57/28 2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head v 57'20, mlcod=57'18)", 2024-09-18T04:39:28.648 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.445316+0000", 2024-09-18T04:39:28.648 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 37.147463338999998, 2024-09-18T04:39:28.648 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0047016760000000001, 2024-09-18T04:39:28.648 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:28.648 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:28.648 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:28.648 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.648 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:28.648 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445316+0000", 2024-09-18T04:39:28.648 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:28.648 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.648 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.648 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:28.648 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445316+0000", 2024-09-18T04:39:28.649 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:28.649 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.649 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.649 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:28.649 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445314+0000", 2024-09-18T04:39:28.649 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:28.649 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.649 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.649 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:28.649 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445947+0000", 2024-09-18T04:39:28.649 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00063298900000000003 2024-09-18T04:39:28.649 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.649 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.649 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:28.649 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445968+0000", 2024-09-18T04:39:28.650 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.054e-05 2024-09-18T04:39:28.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.650 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.650 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:28.650 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445981+0000", 2024-09-18T04:39:28.650 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2697000000000001e-05 2024-09-18T04:39:28.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.650 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.650 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:28.650 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.446102+0000", 2024-09-18T04:39:28.650 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012189300000000001 2024-09-18T04:39:28.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.650 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.650 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:28.651 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.446151+0000", 2024-09-18T04:39:28.651 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8279000000000003e-05 2024-09-18T04:39:28.651 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.651 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.651 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:28.651 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.449953+0000", 2024-09-18T04:39:28.651 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003801979 2024-09-18T04:39:28.651 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.651 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.651 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:28.651 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.450018+0000", 2024-09-18T04:39:28.651 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4873e-05 2024-09-18T04:39:28.651 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:28.651 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:28.651 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:28.652 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.652 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.652 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:699 2.4 e57/17 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 57'17, mlcod=57'15)", 2024-09-18T04:39:28.652 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.566073+0000", 2024-09-18T04:39:28.652 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 37.026705825999997, 2024-09-18T04:39:28.652 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055398380000000001, 2024-09-18T04:39:28.652 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:28.652 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:28.652 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:28.652 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.652 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:28.652 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566073+0000", 2024-09-18T04:39:28.652 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:28.652 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.653 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.653 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:28.653 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566073+0000", 2024-09-18T04:39:28.653 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:28.653 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.653 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.653 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:28.653 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566071+0000", 2024-09-18T04:39:28.653 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:28.653 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.653 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.653 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:28.653 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566637+0000", 2024-09-18T04:39:28.653 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00056516699999999999 2024-09-18T04:39:28.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.654 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.654 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:28.654 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566668+0000", 2024-09-18T04:39:28.654 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1544999999999997e-05 2024-09-18T04:39:28.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.654 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.654 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:28.654 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566685+0000", 2024-09-18T04:39:28.654 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6857000000000001e-05 2024-09-18T04:39:28.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.654 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.654 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:28.654 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566816+0000", 2024-09-18T04:39:28.654 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013092799999999999 2024-09-18T04:39:28.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.655 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.655 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:28.655 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566875+0000", 2024-09-18T04:39:28.655 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9583999999999997e-05 2024-09-18T04:39:28.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.655 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.655 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:28.655 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.571567+0000", 2024-09-18T04:39:28.655 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.004691682 2024-09-18T04:39:28.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.655 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.655 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:28.655 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.571613+0000", 2024-09-18T04:39:28.656 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6125000000000003e-05 2024-09-18T04:39:28.656 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:28.656 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:28.656 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:28.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.656 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.656 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:98 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'27, mlcod=60'25)", 2024-09-18T04:39:28.656 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.847978+0000", 2024-09-18T04:39:28.656 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 27.744801267, 2024-09-18T04:39:28.656 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0044617709999999998, 2024-09-18T04:39:28.656 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:28.656 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:28.656 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:28.656 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.656 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:28.657 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.847978+0000", 2024-09-18T04:39:28.657 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:28.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.657 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.657 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:28.657 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.847978+0000", 2024-09-18T04:39:28.657 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:28.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.657 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.657 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:28.657 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.847977+0000", 2024-09-18T04:39:28.657 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:28.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.657 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.658 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:28.658 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849038+0000", 2024-09-18T04:39:28.658 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0010614820000000001 2024-09-18T04:39:28.658 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.658 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.658 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:28.658 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849065+0000", 2024-09-18T04:39:28.658 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7127999999999998e-05 2024-09-18T04:39:28.658 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.658 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.658 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:28.658 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849082+0000", 2024-09-18T04:39:28.658 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7170999999999999e-05 2024-09-18T04:39:28.658 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.658 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.659 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:28.659 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849188+0000", 2024-09-18T04:39:28.659 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010605800000000001 2024-09-18T04:39:28.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.659 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.659 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:28.659 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849233+0000", 2024-09-18T04:39:28.659 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4579000000000001e-05 2024-09-18T04:39:28.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.659 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.659 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:28.659 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.852370+0000", 2024-09-18T04:39:28.659 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00313751 2024-09-18T04:39:28.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.659 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.660 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:28.660 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.852440+0000", 2024-09-18T04:39:28.660 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9238000000000004e-05 2024-09-18T04:39:28.660 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:28.660 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:28.660 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:28.660 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.660 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.660 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:100 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'29, mlcod=64'27)", 2024-09-18T04:39:28.660 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.858766+0000", 2024-09-18T04:39:28.660 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 27.734012937999999, 2024-09-18T04:39:28.660 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0061964840000000004, 2024-09-18T04:39:28.660 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:28.660 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:28.661 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:28.661 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.661 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:28.661 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.858766+0000", 2024-09-18T04:39:28.661 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:28.661 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.661 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.661 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:28.661 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.858766+0000", 2024-09-18T04:39:28.661 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:28.661 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.661 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.661 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:28.661 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.858765+0000", 2024-09-18T04:39:28.662 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:28.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.662 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.662 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:28.662 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861612+0000", 2024-09-18T04:39:28.662 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028473169999999998 2024-09-18T04:39:28.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.662 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.662 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:28.662 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861643+0000", 2024-09-18T04:39:28.662 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1356000000000003e-05 2024-09-18T04:39:28.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.662 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.662 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:28.662 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861656+0000", 2024-09-18T04:39:28.663 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2899e-05 2024-09-18T04:39:28.663 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.663 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.663 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:28.663 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861773+0000", 2024-09-18T04:39:28.663 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011637999999999999 2024-09-18T04:39:28.663 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.663 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.663 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:28.663 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861808+0000", 2024-09-18T04:39:28.663 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5002999999999997e-05 2024-09-18T04:39:28.663 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.663 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.663 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:28.663 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.864876+0000", 2024-09-18T04:39:28.664 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030680719999999998 2024-09-18T04:39:28.664 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.664 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.664 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:28.664 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.864963+0000", 2024-09-18T04:39:28.664 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7114000000000007e-05 2024-09-18T04:39:28.664 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:28.664 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:28.664 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:28.664 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.664 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.664 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:102 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'31, mlcod=64'29)", 2024-09-18T04:39:28.664 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.872098+0000", 2024-09-18T04:39:28.665 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 27.720681680999999, 2024-09-18T04:39:28.665 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0064676450000000002, 2024-09-18T04:39:28.665 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:28.665 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:28.665 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:28.665 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.665 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:28.665 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.872098+0000", 2024-09-18T04:39:28.665 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:28.665 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.665 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.665 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:28.665 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.872098+0000", 2024-09-18T04:39:28.665 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:28.665 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.666 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.666 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:28.666 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.872096+0000", 2024-09-18T04:39:28.666 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:28.666 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.666 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.666 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:28.666 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875176+0000", 2024-09-18T04:39:28.666 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030799809999999999 2024-09-18T04:39:28.666 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.666 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.666 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:28.666 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875204+0000", 2024-09-18T04:39:28.666 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8025999999999999e-05 2024-09-18T04:39:28.666 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.667 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.667 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:28.667 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875216+0000", 2024-09-18T04:39:28.667 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2072999999999999e-05 2024-09-18T04:39:28.667 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.667 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.667 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:28.667 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875307+0000", 2024-09-18T04:39:28.667 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0373999999999999e-05 2024-09-18T04:39:28.667 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.667 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.667 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:28.667 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875349+0000", 2024-09-18T04:39:28.667 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2299000000000003e-05 2024-09-18T04:39:28.668 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.668 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.668 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:28.668 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.878496+0000", 2024-09-18T04:39:28.668 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0031468960000000002 2024-09-18T04:39:28.668 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.668 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.668 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:28.668 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.878565+0000", 2024-09-18T04:39:28.668 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9465e-05 2024-09-18T04:39:28.668 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:28.668 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:28.668 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:28.668 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.669 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.669 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:104 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'33, mlcod=64'31)", 2024-09-18T04:39:28.669 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.885100+0000", 2024-09-18T04:39:28.669 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 27.707679146, 2024-09-18T04:39:28.669 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0051888159999999997, 2024-09-18T04:39:28.669 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:28.669 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:28.669 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:28.669 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.669 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:28.669 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.885100+0000", 2024-09-18T04:39:28.669 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:28.669 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.669 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.670 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:28.670 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.885100+0000", 2024-09-18T04:39:28.670 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:28.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.670 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.670 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:28.670 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.885098+0000", 2024-09-18T04:39:28.670 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:28.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.670 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.670 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:28.670 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886809+0000", 2024-09-18T04:39:28.670 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017110459999999999 2024-09-18T04:39:28.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.670 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.671 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:28.671 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886842+0000", 2024-09-18T04:39:28.671 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2638999999999997e-05 2024-09-18T04:39:28.671 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.671 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.671 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:28.671 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886855+0000", 2024-09-18T04:39:28.671 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3380000000000001e-05 2024-09-18T04:39:28.671 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.671 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.671 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:28.671 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886939+0000", 2024-09-18T04:39:28.671 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3733999999999995e-05 2024-09-18T04:39:28.671 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.671 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.672 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:28.672 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886981+0000", 2024-09-18T04:39:28.672 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1956000000000003e-05 2024-09-18T04:39:28.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.672 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.672 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:28.672 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.890201+0000", 2024-09-18T04:39:28.672 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003219927 2024-09-18T04:39:28.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.672 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.672 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:28.672 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.890289+0000", 2024-09-18T04:39:28.672 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7997999999999993e-05 2024-09-18T04:39:28.672 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:28.673 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:28.673 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:28.673 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.673 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.673 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:145 2.4 e64/62 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 64'37, mlcod=64'35)", 2024-09-18T04:39:28.673 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:01.099398+0000", 2024-09-18T04:39:28.673 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 27.493381658000001, 2024-09-18T04:39:28.673 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0050078459999999998, 2024-09-18T04:39:28.673 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:28.673 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:28.673 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:28.673 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.673 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:28.673 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.099398+0000", 2024-09-18T04:39:28.673 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:28.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.674 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.674 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:28.674 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.099398+0000", 2024-09-18T04:39:28.674 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:28.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.674 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.674 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:28.674 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.099396+0000", 2024-09-18T04:39:28.674 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:28.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.674 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.674 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:28.674 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100199+0000", 2024-09-18T04:39:28.674 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00080338000000000002 2024-09-18T04:39:28.675 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.675 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.675 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:28.675 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100230+0000", 2024-09-18T04:39:28.675 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0716000000000002e-05 2024-09-18T04:39:28.675 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.675 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.675 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:28.675 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100243+0000", 2024-09-18T04:39:28.675 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3359e-05 2024-09-18T04:39:28.675 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.675 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.675 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:28.675 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100337+0000", 2024-09-18T04:39:28.675 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4387000000000003e-05 2024-09-18T04:39:28.676 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.676 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.676 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:28.676 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100396+0000", 2024-09-18T04:39:28.676 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8164000000000001e-05 2024-09-18T04:39:28.676 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.676 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.676 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:28.676 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.104309+0000", 2024-09-18T04:39:28.676 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0039137800000000004 2024-09-18T04:39:28.676 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.676 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.676 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:28.676 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.104405+0000", 2024-09-18T04:39:28.677 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6039999999999995e-05 2024-09-18T04:39:28.677 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:28.677 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:28.677 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:28.677 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.677 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.677 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:147 2.4 e64/62 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 64'39, mlcod=64'37)", 2024-09-18T04:39:28.677 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:01.110658+0000", 2024-09-18T04:39:28.677 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 27.482121569, 2024-09-18T04:39:28.677 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0046316910000000003, 2024-09-18T04:39:28.677 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:28.677 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:28.677 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:28.677 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.677 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:28.678 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.110658+0000", 2024-09-18T04:39:28.678 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:28.678 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.678 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.678 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:28.678 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.110658+0000", 2024-09-18T04:39:28.678 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:28.678 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.678 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.678 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:28.678 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.110656+0000", 2024-09-18T04:39:28.678 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:28.678 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.678 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.678 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:28.679 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111444+0000", 2024-09-18T04:39:28.679 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00078762400000000001 2024-09-18T04:39:28.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.679 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.679 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:28.679 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111474+0000", 2024-09-18T04:39:28.679 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0085000000000001e-05 2024-09-18T04:39:28.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.679 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.679 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:28.679 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111486+0000", 2024-09-18T04:39:28.679 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2645e-05 2024-09-18T04:39:28.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.679 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.680 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:28.680 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111581+0000", 2024-09-18T04:39:28.680 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4432000000000007e-05 2024-09-18T04:39:28.680 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.680 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.680 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:28.680 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111631+0000", 2024-09-18T04:39:28.680 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9901000000000002e-05 2024-09-18T04:39:28.680 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.680 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.680 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:28.680 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.115209+0000", 2024-09-18T04:39:28.680 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003577916 2024-09-18T04:39:28.680 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:28.680 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:28.681 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:28.681 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.115289+0000", 2024-09-18T04:39:28.681 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0704999999999994e-05 2024-09-18T04:39:28.681 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:28.681 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:28.681 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:28.681 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:28.681 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:28.681 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:28.682 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:39:28.683 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:28.677+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:28.784 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:28.780+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:28.816 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:28.817 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:28.817 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:39:28.817 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:28.818 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:39:28.888 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:28.881+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:28.937 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:28.938 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:28.938 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:39:28.938 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:39:28.938 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:28.946 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:39:28.995 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:28.977+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:29.067 INFO:teuthology.orchestra.run.smithi028.stdout:3.7 2024-09-18T04:39:29.067 INFO:teuthology.orchestra.run.smithi028.stdout:3.4 2024-09-18T04:39:29.067 INFO:teuthology.orchestra.run.smithi028.stdout:3.1 2024-09-18T04:39:29.067 INFO:teuthology.orchestra.run.smithi028.stdout:2.5 2024-09-18T04:39:29.067 INFO:teuthology.orchestra.run.smithi028.stdout:3.5 2024-09-18T04:39:29.067 INFO:teuthology.orchestra.run.smithi028.stdout:2.3 2024-09-18T04:39:29.068 INFO:teuthology.orchestra.run.smithi028.stdout:3.0 2024-09-18T04:39:29.068 INFO:teuthology.orchestra.run.smithi028.stdout:2.1 2024-09-18T04:39:29.085 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:29.085 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:39:29.085 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:39:29.085 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:39:29.085 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.085 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:33 2.1 e51/24 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head v 51'4, mlcod=51'2)", 2024-09-18T04:39:29.085 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:39:29.085 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 46.800500216000003, 2024-09-18T04:39:29.086 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018702106, 2024-09-18T04:39:29.086 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:29.086 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:29.086 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:29.086 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.086 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:29.086 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:39:29.086 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:29.086 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.086 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.086 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:29.086 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:39:29.086 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:29.086 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.087 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.087 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:29.087 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279197+0000", 2024-09-18T04:39:29.087 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:29.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.087 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.088 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:29.088 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282682+0000", 2024-09-18T04:39:29.088 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0034848689999999998 2024-09-18T04:39:29.088 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.088 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.088 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:29.088 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282709+0000", 2024-09-18T04:39:29.088 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7599000000000001e-05 2024-09-18T04:39:29.088 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.089 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.089 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:29.089 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282720+0000", 2024-09-18T04:39:29.089 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.092e-05 2024-09-18T04:39:29.089 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.089 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.089 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:29.089 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282791+0000", 2024-09-18T04:39:29.089 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1268999999999998e-05 2024-09-18T04:39:29.089 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.089 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.090 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:29.090 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282843+0000", 2024-09-18T04:39:29.090 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1771999999999998e-05 2024-09-18T04:39:29.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.090 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.090 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:29.090 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.297830+0000", 2024-09-18T04:39:29.090 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014987228999999999 2024-09-18T04:39:29.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.090 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.090 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:29.091 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.297901+0000", 2024-09-18T04:39:29.091 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0209999999999994e-05 2024-09-18T04:39:29.091 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:29.091 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:29.091 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:29.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.091 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.091 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:38 2.0 e51/17 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head v 51'10, mlcod=51'8)", 2024-09-18T04:39:29.091 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:39:29.091 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 46.799081786000002, 2024-09-18T04:39:29.091 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.023984472, 2024-09-18T04:39:29.092 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:29.092 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:29.092 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:29.092 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.092 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:29.092 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:39:29.092 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:29.092 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.092 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.092 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:29.093 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:39:29.093 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:29.093 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.093 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.093 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:29.093 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280614+0000", 2024-09-18T04:39:29.093 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:29.093 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.093 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.093 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:29.093 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293094+0000", 2024-09-18T04:39:29.094 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012479571 2024-09-18T04:39:29.094 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.094 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.094 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:29.094 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293111+0000", 2024-09-18T04:39:29.094 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7354999999999999e-05 2024-09-18T04:39:29.094 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.094 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.094 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:29.094 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293121+0000", 2024-09-18T04:39:29.094 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6979999999999993e-06 2024-09-18T04:39:29.095 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.095 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.095 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:29.095 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293180+0000", 2024-09-18T04:39:29.095 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9333000000000003e-05 2024-09-18T04:39:29.095 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.095 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.095 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:29.095 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293203+0000", 2024-09-18T04:39:29.096 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2633000000000002e-05 2024-09-18T04:39:29.096 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.096 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.096 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:29.096 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304534+0000", 2024-09-18T04:39:29.096 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011330761 2024-09-18T04:39:29.096 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.096 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.096 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:29.096 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304601+0000", 2024-09-18T04:39:29.096 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7657000000000006e-05 2024-09-18T04:39:29.097 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:29.097 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:29.098 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:29.098 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.098 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.098 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:46 2.6 2:7ba25245:::rbd_data.125fc09df6d1.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:29.099 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:39:29.099 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 46.771538986000003, 2024-09-18T04:39:29.099 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.035604595000000003, 2024-09-18T04:39:29.099 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:29.099 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:29.099 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:29.099 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:29.099 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:29.099 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 46 2024-09-18T04:39:29.099 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.099 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:29.099 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.099 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:29.100 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:39:29.100 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:29.100 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.100 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.100 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:29.100 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:39:29.101 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:29.101 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.101 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.101 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:29.101 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308158+0000", 2024-09-18T04:39:29.101 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:29.101 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.101 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.101 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:29.102 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313191+0000", 2024-09-18T04:39:29.102 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0050323929999999996 2024-09-18T04:39:29.102 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.102 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.102 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:29.102 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313210+0000", 2024-09-18T04:39:29.102 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9673e-05 2024-09-18T04:39:29.102 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.102 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.102 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:29.102 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313221+0000", 2024-09-18T04:39:29.102 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0444e-05 2024-09-18T04:39:29.102 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.102 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.103 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:29.103 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313294+0000", 2024-09-18T04:39:29.103 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2650000000000004e-05 2024-09-18T04:39:29.103 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.103 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.103 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:29.103 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313493+0000", 2024-09-18T04:39:29.103 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019968300000000001 2024-09-18T04:39:29.103 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.103 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.103 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:29.103 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313666+0000", 2024-09-18T04:39:29.104 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00017248599999999999 2024-09-18T04:39:29.104 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.104 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.104 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:29.104 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.323175+0000", 2024-09-18T04:39:29.104 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0095088819999999998 2024-09-18T04:39:29.104 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.104 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.104 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:29.104 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343651+0000", 2024-09-18T04:39:29.104 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020475892999999998 2024-09-18T04:39:29.104 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.104 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.105 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:29.105 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343715+0000", 2024-09-18T04:39:29.105 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4443999999999995e-05 2024-09-18T04:39:29.105 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.105 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.105 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:29.105 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343764+0000", 2024-09-18T04:39:29.105 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9280000000000003e-05 2024-09-18T04:39:29.105 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:29.105 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:29.105 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:29.105 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.105 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.105 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:47 2.4 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:29.106 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:39:29.106 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 46.766456949000002, 2024-09-18T04:39:29.106 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.032570693999999997, 2024-09-18T04:39:29.106 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:29.106 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:29.106 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:29.106 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:29.106 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:29.106 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 47 2024-09-18T04:39:29.106 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.106 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:29.106 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.106 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:29.106 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:39:29.107 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:29.107 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.107 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.107 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:29.107 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:39:29.107 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:29.107 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.107 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.107 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:29.107 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313240+0000", 2024-09-18T04:39:29.107 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:29.107 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.107 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.107 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:29.108 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325825+0000", 2024-09-18T04:39:29.108 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012584425999999999 2024-09-18T04:39:29.108 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.108 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.108 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:29.108 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325848+0000", 2024-09-18T04:39:29.108 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3025999999999999e-05 2024-09-18T04:39:29.108 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.108 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.108 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:29.108 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325863+0000", 2024-09-18T04:39:29.108 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5152e-05 2024-09-18T04:39:29.108 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.108 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.109 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:29.109 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325938+0000", 2024-09-18T04:39:29.109 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4982999999999997e-05 2024-09-18T04:39:29.109 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.109 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.109 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:29.109 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326090+0000", 2024-09-18T04:39:29.109 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015207900000000001 2024-09-18T04:39:29.109 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.109 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.109 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:29.109 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326242+0000", 2024-09-18T04:39:29.109 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015163500000000001 2024-09-18T04:39:29.110 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.110 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.110 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:29.110 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343015+0000", 2024-09-18T04:39:29.110 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016773414 2024-09-18T04:39:29.110 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.110 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.110 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:29.110 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345664+0000", 2024-09-18T04:39:29.110 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026487870000000001 2024-09-18T04:39:29.110 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.110 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.110 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:29.111 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345739+0000", 2024-09-18T04:39:29.111 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5018999999999994e-05 2024-09-18T04:39:29.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.111 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.111 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:29.111 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345812+0000", 2024-09-18T04:39:29.111 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3547000000000006e-05 2024-09-18T04:39:29.111 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:29.111 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:29.111 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:29.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.111 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.111 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:48 2.6 2:61db7a3a:::rbd_data.125fc09df6d1.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:29.111 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:39:29.112 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 46.753800277000003, 2024-09-18T04:39:29.112 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.025064388, 2024-09-18T04:39:29.112 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:29.112 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:29.112 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:29.112 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:29.112 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:29.112 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 48 2024-09-18T04:39:29.112 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.112 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:29.112 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.112 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:29.112 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:39:29.113 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:29.113 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.113 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.113 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:29.113 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:39:29.113 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:29.113 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.113 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.113 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:29.114 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325897+0000", 2024-09-18T04:39:29.114 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:29.114 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.114 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.114 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:29.114 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333697+0000", 2024-09-18T04:39:29.114 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0078002649999999998 2024-09-18T04:39:29.114 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.114 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.114 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:29.114 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333716+0000", 2024-09-18T04:39:29.114 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8910999999999999e-05 2024-09-18T04:39:29.114 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.114 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.115 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:29.115 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333728+0000", 2024-09-18T04:39:29.115 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1936000000000001e-05 2024-09-18T04:39:29.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.115 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.115 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:29.115 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333792+0000", 2024-09-18T04:39:29.115 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4010999999999994e-05 2024-09-18T04:39:29.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.115 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.115 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:29.115 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.334082+0000", 2024-09-18T04:39:29.115 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00028955799999999998 2024-09-18T04:39:29.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.116 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.116 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:29.116 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.334326+0000", 2024-09-18T04:39:29.116 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00024399099999999999 2024-09-18T04:39:29.116 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.116 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.116 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:29.116 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350745+0000", 2024-09-18T04:39:29.116 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016419474999999999 2024-09-18T04:39:29.116 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.116 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.116 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:29.116 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350808+0000", 2024-09-18T04:39:29.116 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3244999999999995e-05 2024-09-18T04:39:29.117 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.117 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.117 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:29.117 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350892+0000", 2024-09-18T04:39:29.117 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4074999999999997e-05 2024-09-18T04:39:29.117 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.117 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.117 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:29.117 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350963+0000", 2024-09-18T04:39:29.117 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0412000000000003e-05 2024-09-18T04:39:29.117 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:29.117 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:29.117 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:29.117 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.118 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.118 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:51 2.6 2:775d7b11:::rbd_data.125fc09df6d1.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:29.118 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:39:29.118 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 46.745948141, 2024-09-18T04:39:29.118 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.023824221, 2024-09-18T04:39:29.118 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:29.118 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:29.118 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:29.118 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:29.118 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:29.118 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 51 2024-09-18T04:39:29.118 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.118 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:29.119 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.119 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:29.119 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:39:29.119 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:29.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.119 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.119 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:29.119 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:39:29.119 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:29.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.119 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.119 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:29.119 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333749+0000", 2024-09-18T04:39:29.119 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:29.120 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.120 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.120 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:29.120 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340899+0000", 2024-09-18T04:39:29.120 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0071500340000000004 2024-09-18T04:39:29.120 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.120 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.120 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:29.120 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340929+0000", 2024-09-18T04:39:29.121 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0093000000000001e-05 2024-09-18T04:39:29.121 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.121 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.121 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:29.121 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340943+0000", 2024-09-18T04:39:29.121 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4019000000000001e-05 2024-09-18T04:39:29.121 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.121 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.121 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:29.121 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343834+0000", 2024-09-18T04:39:29.121 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028918120000000001 2024-09-18T04:39:29.121 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.121 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.121 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:29.122 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344020+0000", 2024-09-18T04:39:29.122 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000185892 2024-09-18T04:39:29.122 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.122 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.122 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:29.122 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344190+0000", 2024-09-18T04:39:29.122 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016946800000000001 2024-09-18T04:39:29.122 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.122 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.122 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:29.122 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.351291+0000", 2024-09-18T04:39:29.122 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.007100805 2024-09-18T04:39:29.122 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.122 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.123 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:29.123 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357470+0000", 2024-09-18T04:39:29.123 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0061797570000000001 2024-09-18T04:39:29.123 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.123 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.123 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:29.123 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357533+0000", 2024-09-18T04:39:29.123 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2906999999999999e-05 2024-09-18T04:39:29.123 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.123 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.123 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:29.123 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357575+0000", 2024-09-18T04:39:29.124 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1403999999999999e-05 2024-09-18T04:39:29.124 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:29.124 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:29.124 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:29.124 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.124 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.124 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:56 2.4 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:29.124 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:39:29.124 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 46.729903047999997, 2024-09-18T04:39:29.124 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.034295870999999999, 2024-09-18T04:39:29.124 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:29.124 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:29.124 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:29.125 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:29.125 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:29.125 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 56 2024-09-18T04:39:29.125 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.129 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:29.130 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.130 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:29.130 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:39:29.130 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:29.130 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.130 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.130 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:29.130 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:39:29.131 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:29.131 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.131 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.131 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:29.131 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349794+0000", 2024-09-18T04:39:29.131 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:29.131 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.131 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.131 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:29.131 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366632+0000", 2024-09-18T04:39:29.131 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01683755 2024-09-18T04:39:29.131 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.131 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.131 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:29.132 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366669+0000", 2024-09-18T04:39:29.132 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7351000000000002e-05 2024-09-18T04:39:29.132 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.132 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.132 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:29.132 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366686+0000", 2024-09-18T04:39:29.132 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6507999999999998e-05 2024-09-18T04:39:29.132 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.132 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.132 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:29.132 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366962+0000", 2024-09-18T04:39:29.132 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00027655900000000001 2024-09-18T04:39:29.132 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.133 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.133 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:29.133 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.367452+0000", 2024-09-18T04:39:29.133 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00049025299999999996 2024-09-18T04:39:29.133 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.133 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.133 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:29.133 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368113+0000", 2024-09-18T04:39:29.133 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00066093399999999996 2024-09-18T04:39:29.133 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.133 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.133 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:29.133 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383719+0000", 2024-09-18T04:39:29.133 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015606106999999999 2024-09-18T04:39:29.134 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.134 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.134 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:29.134 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384035+0000", 2024-09-18T04:39:29.134 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00031590399999999999 2024-09-18T04:39:29.134 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.134 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.134 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:29.134 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384067+0000", 2024-09-18T04:39:29.134 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.129e-05 2024-09-18T04:39:29.134 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.134 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.134 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:29.134 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384091+0000", 2024-09-18T04:39:29.135 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4788000000000001e-05 2024-09-18T04:39:29.135 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:29.135 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:29.135 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:29.135 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.135 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.135 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:57 2.0 e51/17 2:0cdd070e:::rbd_data.125fc09df6d1.000000000000000d:head v 51'21, mlcod=51'19)", 2024-09-18T04:39:29.135 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:39:29.135 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 46.722075560999997, 2024-09-18T04:39:29.135 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.025216311000000002, 2024-09-18T04:39:29.135 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:29.135 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:29.135 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:29.135 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.135 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:29.136 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:39:29.136 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:29.136 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.136 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.136 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:29.136 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:39:29.136 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:29.136 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.136 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.136 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:29.136 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:39:29.136 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:29.136 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.137 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.137 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:29.137 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363844+0000", 2024-09-18T04:39:29.137 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0062218450000000002 2024-09-18T04:39:29.137 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.137 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.137 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:29.137 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363879+0000", 2024-09-18T04:39:29.137 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4382999999999999e-05 2024-09-18T04:39:29.137 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.137 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.137 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:29.137 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363893+0000", 2024-09-18T04:39:29.137 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4348999999999999e-05 2024-09-18T04:39:29.138 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.138 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.138 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:29.138 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363977+0000", 2024-09-18T04:39:29.138 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3499999999999997e-05 2024-09-18T04:39:29.138 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.138 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.138 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:29.138 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364009+0000", 2024-09-18T04:39:29.138 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2292000000000002e-05 2024-09-18T04:39:29.138 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.138 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.138 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:29.138 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.382781+0000", 2024-09-18T04:39:29.139 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018771626 2024-09-18T04:39:29.139 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.139 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.139 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:29.139 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.382839+0000", 2024-09-18T04:39:29.139 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8758000000000002e-05 2024-09-18T04:39:29.139 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:29.139 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:29.139 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:29.139 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.139 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.139 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:59 2.0 e51/17 2:19c52cde:::rbd_data.125fc09df6d1.000000000000000f:head v 51'22, mlcod=51'19)", 2024-09-18T04:39:29.139 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:39:29.139 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 46.715415634999999, 2024-09-18T04:39:29.140 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.019497112, 2024-09-18T04:39:29.140 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:29.140 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:29.140 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:29.140 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.140 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:29.140 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:39:29.140 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:29.140 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.140 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.140 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:29.140 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:39:29.140 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:29.140 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.141 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.141 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:29.141 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364282+0000", 2024-09-18T04:39:29.141 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:29.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.141 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.141 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:29.141 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371823+0000", 2024-09-18T04:39:29.141 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0075411829999999999 2024-09-18T04:39:29.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.141 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.141 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:29.141 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371837+0000", 2024-09-18T04:39:29.141 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4336e-05 2024-09-18T04:39:29.142 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.142 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.142 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:29.142 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371846+0000", 2024-09-18T04:39:29.142 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9770000000000006e-06 2024-09-18T04:39:29.142 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.142 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.142 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:29.142 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.373566+0000", 2024-09-18T04:39:29.142 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017192539999999999 2024-09-18T04:39:29.142 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.142 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.142 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:29.143 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.373595+0000", 2024-09-18T04:39:29.143 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8975999999999998e-05 2024-09-18T04:39:29.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.143 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.143 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:29.143 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383740+0000", 2024-09-18T04:39:29.143 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01014525 2024-09-18T04:39:29.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.143 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.143 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:29.143 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383780+0000", 2024-09-18T04:39:29.143 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0275999999999999e-05 2024-09-18T04:39:29.143 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:29.143 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:29.143 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:29.144 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.144 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.144 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:34 2.6 2:66289171:ns::rbd_data.133e813ab5bc.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:29.144 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.524015+0000", 2024-09-18T04:39:29.144 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 46.555684026000002, 2024-09-18T04:39:29.144 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027582111999999999, 2024-09-18T04:39:29.144 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:29.144 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:29.144 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:29.144 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:29.144 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:29.144 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 34 2024-09-18T04:39:29.144 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.145 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:29.145 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.145 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:29.145 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524015+0000", 2024-09-18T04:39:29.145 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:29.145 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.145 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.145 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:29.145 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524015+0000", 2024-09-18T04:39:29.145 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:29.145 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.145 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.145 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:29.145 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524012+0000", 2024-09-18T04:39:29.146 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:29.146 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.146 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.146 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:29.146 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534956+0000", 2024-09-18T04:39:29.146 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010943403000000001 2024-09-18T04:39:29.146 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.146 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.146 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:29.146 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534978+0000", 2024-09-18T04:39:29.146 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1965999999999999e-05 2024-09-18T04:39:29.146 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.146 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.146 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:29.146 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534990+0000", 2024-09-18T04:39:29.147 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2065999999999999e-05 2024-09-18T04:39:29.147 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.147 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.147 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:29.147 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535065+0000", 2024-09-18T04:39:29.147 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5523000000000002e-05 2024-09-18T04:39:29.147 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.147 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.147 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:29.147 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535208+0000", 2024-09-18T04:39:29.147 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014309500000000001 2024-09-18T04:39:29.147 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.147 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.147 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:29.148 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535343+0000", 2024-09-18T04:39:29.148 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013437400000000001 2024-09-18T04:39:29.148 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.148 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.148 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:29.148 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.546576+0000", 2024-09-18T04:39:29.148 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011233425999999999 2024-09-18T04:39:29.148 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.148 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.148 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:29.148 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551498+0000", 2024-09-18T04:39:29.148 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0049219349999999997 2024-09-18T04:39:29.148 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.148 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.148 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:29.149 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551554+0000", 2024-09-18T04:39:29.149 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6400999999999998e-05 2024-09-18T04:39:29.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.149 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.149 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:29.149 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551597+0000", 2024-09-18T04:39:29.149 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2283000000000002e-05 2024-09-18T04:39:29.149 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:29.149 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:29.149 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:29.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.149 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.149 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:33 2.1 e51/24 2:8b40f1c5:ns::rbd_data.133e813ab5bc.0000000000000000:head v 51'7, mlcod=51'5)", 2024-09-18T04:39:29.149 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.531767+0000", 2024-09-18T04:39:29.150 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 46.547932109000001, 2024-09-18T04:39:29.150 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015370295000000001, 2024-09-18T04:39:29.150 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:29.150 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:29.150 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:29.150 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.150 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:29.150 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531767+0000", 2024-09-18T04:39:29.150 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:29.150 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.150 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.150 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:29.150 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531767+0000", 2024-09-18T04:39:29.150 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:29.151 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.151 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.151 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:29.151 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531765+0000", 2024-09-18T04:39:29.151 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:29.151 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.151 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.153 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:29.153 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534258+0000", 2024-09-18T04:39:29.153 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002493561 2024-09-18T04:39:29.153 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.153 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.153 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:29.154 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534288+0000", 2024-09-18T04:39:29.154 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.976e-05 2024-09-18T04:39:29.154 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.154 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.154 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:29.154 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534300+0000", 2024-09-18T04:39:29.154 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1928e-05 2024-09-18T04:39:29.154 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.154 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.154 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:29.154 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534367+0000", 2024-09-18T04:39:29.154 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7045999999999995e-05 2024-09-18T04:39:29.154 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.154 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.155 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:29.155 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534394+0000", 2024-09-18T04:39:29.155 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6903999999999999e-05 2024-09-18T04:39:29.155 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.155 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.155 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:29.155 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.547073+0000", 2024-09-18T04:39:29.155 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012679365 2024-09-18T04:39:29.155 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.155 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.155 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:29.155 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.547137+0000", 2024-09-18T04:39:29.155 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3691999999999999e-05 2024-09-18T04:39:29.155 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:29.155 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:29.156 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:29.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.156 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.156 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:39 2.4 2:304364d1:ns::rbd_data.133e813ab5bc.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:29.156 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.535013+0000", 2024-09-18T04:39:29.156 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 46.544685174000001, 2024-09-18T04:39:29.156 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027764487000000001, 2024-09-18T04:39:29.156 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:29.156 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:29.156 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:29.156 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:29.156 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:29.156 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 39 2024-09-18T04:39:29.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.157 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:29.157 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.157 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:29.157 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535013+0000", 2024-09-18T04:39:29.157 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:29.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.157 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.157 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:29.157 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535013+0000", 2024-09-18T04:39:29.157 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:29.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.157 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.157 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:29.157 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535012+0000", 2024-09-18T04:39:29.158 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:29.158 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.158 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.158 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:29.158 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545016+0000", 2024-09-18T04:39:29.158 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01000322 2024-09-18T04:39:29.158 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.158 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.158 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:29.158 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545035+0000", 2024-09-18T04:39:29.158 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9845999999999999e-05 2024-09-18T04:39:29.158 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.158 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.159 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:29.159 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545045+0000", 2024-09-18T04:39:29.159 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3640000000000005e-06 2024-09-18T04:39:29.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.159 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.159 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:29.159 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545152+0000", 2024-09-18T04:39:29.159 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000107025 2024-09-18T04:39:29.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.159 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.159 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:29.159 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545338+0000", 2024-09-18T04:39:29.159 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018643500000000001 2024-09-18T04:39:29.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.159 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.160 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:29.160 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545492+0000", 2024-09-18T04:39:29.160 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015393199999999999 2024-09-18T04:39:29.160 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.160 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.160 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:29.160 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.553549+0000", 2024-09-18T04:39:29.160 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0080569170000000002 2024-09-18T04:39:29.160 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.160 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.160 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:29.160 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562693+0000", 2024-09-18T04:39:29.160 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0091441609999999996 2024-09-18T04:39:29.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.161 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.161 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:29.161 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562748+0000", 2024-09-18T04:39:29.161 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5047000000000003e-05 2024-09-18T04:39:29.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.161 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.161 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:29.161 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562778+0000", 2024-09-18T04:39:29.161 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9652e-05 2024-09-18T04:39:29.161 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:29.161 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:29.161 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:29.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.162 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.162 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:40 2.6 2:784d6cd1:ns::rbd_data.133e813ab5bc.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:29.162 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.545062+0000", 2024-09-18T04:39:29.162 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 46.534636122999999, 2024-09-18T04:39:29.162 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.032771623, 2024-09-18T04:39:29.162 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:29.162 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:29.162 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:29.162 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:29.162 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:29.162 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 40 2024-09-18T04:39:29.162 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.162 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:29.162 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.162 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:29.163 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545062+0000", 2024-09-18T04:39:29.163 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:29.163 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.163 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.163 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:29.163 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545062+0000", 2024-09-18T04:39:29.163 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:29.163 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.163 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.163 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:29.163 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545061+0000", 2024-09-18T04:39:29.163 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:29.163 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.163 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.164 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:29.164 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552477+0000", 2024-09-18T04:39:29.164 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.007415914 2024-09-18T04:39:29.164 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.164 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.164 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:29.164 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552500+0000", 2024-09-18T04:39:29.164 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3546000000000001e-05 2024-09-18T04:39:29.164 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.164 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.164 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:29.164 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552515+0000", 2024-09-18T04:39:29.164 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4606999999999999e-05 2024-09-18T04:39:29.165 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.165 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.165 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:29.165 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552661+0000", 2024-09-18T04:39:29.165 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000146007 2024-09-18T04:39:29.165 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.165 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.165 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:29.165 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552876+0000", 2024-09-18T04:39:29.165 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000214618 2024-09-18T04:39:29.165 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.165 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.165 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:29.165 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.553061+0000", 2024-09-18T04:39:29.166 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018564100000000001 2024-09-18T04:39:29.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.166 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.166 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:29.166 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.565612+0000", 2024-09-18T04:39:29.166 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012550301999999999 2024-09-18T04:39:29.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.166 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.166 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:29.166 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577695+0000", 2024-09-18T04:39:29.166 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012083493000000001 2024-09-18T04:39:29.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.166 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.166 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:29.167 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577772+0000", 2024-09-18T04:39:29.167 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7158999999999998e-05 2024-09-18T04:39:29.167 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.167 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.167 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:29.167 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577834+0000", 2024-09-18T04:39:29.167 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1851000000000001e-05 2024-09-18T04:39:29.167 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:29.167 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:29.167 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:29.167 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.167 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.167 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:43 2.6 2:759f27b6:ns::rbd_data.133e813ab5bc.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:29.168 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.555868+0000", 2024-09-18T04:39:29.168 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 46.523830629000003, 2024-09-18T04:39:29.168 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.022332497999999999, 2024-09-18T04:39:29.168 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:29.168 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:29.168 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:29.168 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:29.168 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:29.168 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 43 2024-09-18T04:39:29.168 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.168 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:29.168 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.168 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:29.168 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555868+0000", 2024-09-18T04:39:29.169 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:29.169 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.169 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.169 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:29.169 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555868+0000", 2024-09-18T04:39:29.169 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:29.169 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.169 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.169 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:29.169 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555865+0000", 2024-09-18T04:39:29.169 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999971 2024-09-18T04:39:29.169 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.169 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.170 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:29.170 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561031+0000", 2024-09-18T04:39:29.170 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0051658770000000001 2024-09-18T04:39:29.170 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.170 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.170 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:29.170 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561048+0000", 2024-09-18T04:39:29.170 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6569e-05 2024-09-18T04:39:29.170 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.170 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.170 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:29.170 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561055+0000", 2024-09-18T04:39:29.170 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.328e-06 2024-09-18T04:39:29.170 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.171 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.171 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:29.171 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561114+0000", 2024-09-18T04:39:29.171 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9540000000000003e-05 2024-09-18T04:39:29.171 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.171 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.171 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:29.171 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561314+0000", 2024-09-18T04:39:29.171 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019965799999999999 2024-09-18T04:39:29.171 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.171 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.171 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:29.171 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561473+0000", 2024-09-18T04:39:29.171 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000158904 2024-09-18T04:39:29.171 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.172 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.172 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:29.172 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.567015+0000", 2024-09-18T04:39:29.172 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055420859999999999 2024-09-18T04:39:29.172 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.172 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.172 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:29.172 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577982+0000", 2024-09-18T04:39:29.172 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010966872000000001 2024-09-18T04:39:29.172 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.172 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.172 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:29.173 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578162+0000", 2024-09-18T04:39:29.173 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018050999999999999 2024-09-18T04:39:29.173 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.173 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.173 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:29.173 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578200+0000", 2024-09-18T04:39:29.173 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8000999999999998e-05 2024-09-18T04:39:29.173 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:29.173 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:29.173 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:29.173 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.173 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.173 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:48 2.6 2:7c225365:ns::rbd_data.133e813ab5bc.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:29.173 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:39:29.174 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 46.516031406000003, 2024-09-18T04:39:29.174 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.024576377, 2024-09-18T04:39:29.174 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:29.174 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:29.174 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:29.174 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:29.174 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:29.174 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 48 2024-09-18T04:39:29.174 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.174 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:29.174 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.174 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:29.175 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:39:29.175 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:29.175 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.175 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.175 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:29.175 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:39:29.175 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:29.175 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.175 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.175 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:29.175 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.563666+0000", 2024-09-18T04:39:29.175 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:29.175 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.175 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.176 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:29.176 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570331+0000", 2024-09-18T04:39:29.176 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0066653329999999998 2024-09-18T04:39:29.176 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.176 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.176 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:29.176 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570348+0000", 2024-09-18T04:39:29.176 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6507e-05 2024-09-18T04:39:29.176 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.176 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.176 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:29.176 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570355+0000", 2024-09-18T04:39:29.176 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2749999999999998e-06 2024-09-18T04:39:29.176 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.177 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.177 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:29.177 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570396+0000", 2024-09-18T04:39:29.177 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0809000000000002e-05 2024-09-18T04:39:29.177 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.177 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.177 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:29.177 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570571+0000", 2024-09-18T04:39:29.177 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000174823 2024-09-18T04:39:29.177 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.177 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.177 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:29.177 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570722+0000", 2024-09-18T04:39:29.177 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015154599999999999 2024-09-18T04:39:29.178 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.178 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.178 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:29.178 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579756+0000", 2024-09-18T04:39:29.178 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0090337000000000004 2024-09-18T04:39:29.178 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.178 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.178 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:29.178 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588093+0000", 2024-09-18T04:39:29.178 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.008336553 2024-09-18T04:39:29.178 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.178 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.178 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:29.179 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588173+0000", 2024-09-18T04:39:29.179 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0333999999999994e-05 2024-09-18T04:39:29.179 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.179 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.179 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:29.179 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588244+0000", 2024-09-18T04:39:29.179 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0573000000000003e-05 2024-09-18T04:39:29.179 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:29.179 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:29.179 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:29.179 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.179 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.179 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:49 2.4 2:20f0b0c9:ns::rbd_data.133e813ab5bc.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:29.179 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.570370+0000", 2024-09-18T04:39:29.179 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 46.509329078999997, 2024-09-18T04:39:29.180 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.029633981, 2024-09-18T04:39:29.180 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:29.180 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:29.180 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:29.180 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:29.180 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:29.180 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 49 2024-09-18T04:39:29.180 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.180 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:29.180 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.180 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:29.180 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570370+0000", 2024-09-18T04:39:29.180 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:29.181 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.181 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.181 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:29.181 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570370+0000", 2024-09-18T04:39:29.181 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:29.181 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.181 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.181 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:29.181 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570369+0000", 2024-09-18T04:39:29.181 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:29.181 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.181 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.181 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:29.181 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579090+0000", 2024-09-18T04:39:29.181 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0087212339999999996 2024-09-18T04:39:29.182 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.182 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.182 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:29.182 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579109+0000", 2024-09-18T04:39:29.182 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9120999999999999e-05 2024-09-18T04:39:29.182 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.182 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.182 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:29.182 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579123+0000", 2024-09-18T04:39:29.182 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4382000000000001e-05 2024-09-18T04:39:29.182 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.182 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.182 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:29.182 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579221+0000", 2024-09-18T04:39:29.183 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.7503999999999993e-05 2024-09-18T04:39:29.183 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.183 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.183 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:29.183 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579553+0000", 2024-09-18T04:39:29.183 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00033254200000000002 2024-09-18T04:39:29.183 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.183 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.183 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:29.183 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579817+0000", 2024-09-18T04:39:29.183 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00026317099999999998 2024-09-18T04:39:29.183 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.183 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.184 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:29.184 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.599905+0000", 2024-09-18T04:39:29.184 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020088575000000001 2024-09-18T04:39:29.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.184 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.184 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:29.184 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.599923+0000", 2024-09-18T04:39:29.184 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7425999999999999e-05 2024-09-18T04:39:29.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.184 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.184 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:29.184 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.599973+0000", 2024-09-18T04:39:29.184 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0649999999999998e-05 2024-09-18T04:39:29.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.185 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.185 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:29.185 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.600004+0000", 2024-09-18T04:39:29.185 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0361999999999998e-05 2024-09-18T04:39:29.185 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:29.185 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:29.185 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:29.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.185 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.185 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:50 2.1 e51/24 2:8c706374:ns::rbd_data.133e813ab5bc.000000000000000a:head v 51'8, mlcod=51'6)", 2024-09-18T04:39:29.185 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.577029+0000", 2024-09-18T04:39:29.185 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 46.502669707999999, 2024-09-18T04:39:29.186 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.017362855, 2024-09-18T04:39:29.186 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:29.186 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:29.186 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:29.186 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.186 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:29.186 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577029+0000", 2024-09-18T04:39:29.186 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:29.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.186 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.186 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:29.186 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577029+0000", 2024-09-18T04:39:29.186 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:29.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.187 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.187 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:29.187 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577027+0000", 2024-09-18T04:39:29.187 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:29.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.187 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.187 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:29.187 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578880+0000", 2024-09-18T04:39:29.187 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0018532080000000001 2024-09-18T04:39:29.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.187 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.187 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:29.187 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578906+0000", 2024-09-18T04:39:29.188 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5573999999999999e-05 2024-09-18T04:39:29.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.188 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.188 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:29.188 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578920+0000", 2024-09-18T04:39:29.188 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4483e-05 2024-09-18T04:39:29.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.188 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.188 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:29.188 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578983+0000", 2024-09-18T04:39:29.188 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2248000000000004e-05 2024-09-18T04:39:29.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.188 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.188 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:29.189 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579016+0000", 2024-09-18T04:39:29.189 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3364000000000001e-05 2024-09-18T04:39:29.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.189 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.189 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:29.189 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594342+0000", 2024-09-18T04:39:29.189 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01532564 2024-09-18T04:39:29.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.189 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.189 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:29.189 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594392+0000", 2024-09-18T04:39:29.189 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0068999999999998e-05 2024-09-18T04:39:29.189 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:29.189 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:29.190 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:29.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.190 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.190 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:45 2.0 e51/17 2:0fbfa3dc:ns::rbd_data.133e813ab5bc.0000000000000007:head v 51'24, mlcod=51'22)", 2024-09-18T04:39:29.190 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.581892+0000", 2024-09-18T04:39:29.190 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 46.497807035999998, 2024-09-18T04:39:29.190 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015646006, 2024-09-18T04:39:29.190 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:29.190 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:29.190 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:29.190 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.190 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:29.190 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581892+0000", 2024-09-18T04:39:29.190 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:29.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.191 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.191 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:29.191 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581892+0000", 2024-09-18T04:39:29.191 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:29.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.191 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.191 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:29.191 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581890+0000", 2024-09-18T04:39:29.191 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:29.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.191 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.191 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:29.191 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586188+0000", 2024-09-18T04:39:29.192 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0042985769999999996 2024-09-18T04:39:29.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.192 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.192 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:29.192 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586210+0000", 2024-09-18T04:39:29.192 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1069000000000001e-05 2024-09-18T04:39:29.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.192 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.192 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:29.192 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586221+0000", 2024-09-18T04:39:29.192 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1287000000000001e-05 2024-09-18T04:39:29.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.192 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.192 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:29.193 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586313+0000", 2024-09-18T04:39:29.193 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1935999999999995e-05 2024-09-18T04:39:29.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.193 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.193 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:29.193 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586368+0000", 2024-09-18T04:39:29.193 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5578000000000002e-05 2024-09-18T04:39:29.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.193 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.193 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:29.193 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.597473+0000", 2024-09-18T04:39:29.193 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011105139999999999 2024-09-18T04:39:29.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.193 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.193 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:29.194 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.597538+0000", 2024-09-18T04:39:29.194 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4133000000000004e-05 2024-09-18T04:39:29.194 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:29.194 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:29.194 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:29.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.194 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.194 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:53 2.4 2:39f7f174:ns::rbd_data.133e813ab5bc.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:29.194 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.584263+0000", 2024-09-18T04:39:29.194 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 46.495435162, 2024-09-18T04:39:29.194 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018701180000000001, 2024-09-18T04:39:29.194 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:29.194 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:29.194 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:29.195 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:29.195 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:29.195 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 53 2024-09-18T04:39:29.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.195 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:29.195 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.195 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:29.195 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584263+0000", 2024-09-18T04:39:29.195 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:29.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.195 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.195 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:29.195 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584263+0000", 2024-09-18T04:39:29.195 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:29.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.196 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.196 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:29.196 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584262+0000", 2024-09-18T04:39:29.196 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:29.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.196 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.196 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:29.196 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.591617+0000", 2024-09-18T04:39:29.196 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0073550600000000001 2024-09-18T04:39:29.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.196 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.196 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:29.196 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.591637+0000", 2024-09-18T04:39:29.197 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0432999999999999e-05 2024-09-18T04:39:29.197 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.197 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.197 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:29.197 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.591646+0000", 2024-09-18T04:39:29.197 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7849999999999997e-06 2024-09-18T04:39:29.197 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.197 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.197 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:29.197 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593478+0000", 2024-09-18T04:39:29.197 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001831495 2024-09-18T04:39:29.197 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.197 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.197 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:29.197 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593661+0000", 2024-09-18T04:39:29.198 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018364999999999999 2024-09-18T04:39:29.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.198 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.198 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:29.198 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593834+0000", 2024-09-18T04:39:29.198 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000173065 2024-09-18T04:39:29.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.198 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.198 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:29.198 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.600670+0000", 2024-09-18T04:39:29.198 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0068354970000000003 2024-09-18T04:39:29.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.198 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.198 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:29.199 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602851+0000", 2024-09-18T04:39:29.199 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002181653 2024-09-18T04:39:29.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.199 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.199 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:29.199 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602930+0000", 2024-09-18T04:39:29.199 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8045000000000002e-05 2024-09-18T04:39:29.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.199 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.199 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:29.200 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602965+0000", 2024-09-18T04:39:29.200 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5102999999999999e-05 2024-09-18T04:39:29.200 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:29.200 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:29.200 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:29.200 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.200 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.200 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:56 2.6 2:72453c3e:ns::rbd_data.133e813ab5bc.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:29.200 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.595988+0000", 2024-09-18T04:39:29.200 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 46.483710760000001, 2024-09-18T04:39:29.200 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.030296172, 2024-09-18T04:39:29.200 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:29.200 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:29.201 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:29.201 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:29.201 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:29.201 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 56 2024-09-18T04:39:29.201 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.201 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:29.201 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.201 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:29.201 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.595988+0000", 2024-09-18T04:39:29.201 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:29.201 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.201 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.201 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:29.201 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.595988+0000", 2024-09-18T04:39:29.201 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:29.202 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.202 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.202 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:29.202 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.595986+0000", 2024-09-18T04:39:29.202 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:29.202 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.202 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.202 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:29.202 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605262+0000", 2024-09-18T04:39:29.202 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0092767240000000001 2024-09-18T04:39:29.202 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.202 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.202 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:29.202 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605282+0000", 2024-09-18T04:39:29.203 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9307e-05 2024-09-18T04:39:29.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.203 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.203 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:29.203 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605292+0000", 2024-09-18T04:39:29.203 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9510000000000001e-06 2024-09-18T04:39:29.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.203 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.203 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:29.203 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605369+0000", 2024-09-18T04:39:29.203 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7846e-05 2024-09-18T04:39:29.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.203 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.204 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:29.204 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605668+0000", 2024-09-18T04:39:29.204 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00029896099999999999 2024-09-18T04:39:29.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.204 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.204 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:29.204 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605906+0000", 2024-09-18T04:39:29.204 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000237758 2024-09-18T04:39:29.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.204 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.204 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:29.204 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.613532+0000", 2024-09-18T04:39:29.204 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0076260590000000001 2024-09-18T04:39:29.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.205 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.205 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:29.205 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.626116+0000", 2024-09-18T04:39:29.205 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012584092999999999 2024-09-18T04:39:29.205 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.205 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.205 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:29.205 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.626216+0000", 2024-09-18T04:39:29.205 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9630999999999995e-05 2024-09-18T04:39:29.205 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:29.205 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:29.205 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:29.205 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.626284+0000", 2024-09-18T04:39:29.205 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8034999999999995e-05 2024-09-18T04:39:29.206 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:29.206 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:29.206 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:29.206 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:29.206 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:29.206 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:29.206 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:39:29.207 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:29.145+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:29.256 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:29.252+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:29.323 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:29.323 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:29.323 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:39:29.324 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:29.334 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:39:29.354 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:29.349+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:29.433 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T04:39:29.443 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:29.443 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:29.443 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:39:29.443 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:39:29.443 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:29.453 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:39:29.454 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:29.449+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:29.555 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:29.552+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:29.562 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:29.562 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:39:29.562 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:39:29.563 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:39:29.563 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.563 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:39:29.563 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:39:29.563 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 121.769616089, 2024-09-18T04:39:29.563 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017399218000000001, 2024-09-18T04:39:29.563 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:29.563 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:29.563 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:29.563 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:39:29.564 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:39:29.564 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 114 2024-09-18T04:39:29.564 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.564 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:29.564 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.564 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:29.564 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:39:29.564 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:29.564 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.564 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.564 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:29.564 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:39:29.565 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:29.565 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.565 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.565 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:29.565 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786900+0000", 2024-09-18T04:39:29.565 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:29.565 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.565 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.565 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:29.565 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786908+0000", 2024-09-18T04:39:29.565 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8199999999999997e-06 2024-09-18T04:39:29.565 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.565 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.566 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:29.566 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786925+0000", 2024-09-18T04:39:29.566 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6849000000000001e-05 2024-09-18T04:39:29.566 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.566 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.566 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:29.566 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786934+0000", 2024-09-18T04:39:29.566 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6299999999999993e-06 2024-09-18T04:39:29.566 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.566 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.567 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:29.567 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800926+0000", 2024-09-18T04:39:29.567 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013991632 2024-09-18T04:39:29.567 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.567 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.567 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:29.567 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801060+0000", 2024-09-18T04:39:29.567 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013378199999999999 2024-09-18T04:39:29.567 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.568 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.568 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:29.568 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801166+0000", 2024-09-18T04:39:29.568 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010607099999999999 2024-09-18T04:39:29.568 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.568 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.568 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:29.569 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803026+0000", 2024-09-18T04:39:29.569 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001860469 2024-09-18T04:39:29.569 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.569 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.569 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:29.569 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804245+0000", 2024-09-18T04:39:29.569 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0012185900000000001 2024-09-18T04:39:29.569 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.569 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.569 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:29.569 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804278+0000", 2024-09-18T04:39:29.569 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3272000000000003e-05 2024-09-18T04:39:29.570 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.570 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.570 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:29.570 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804300+0000", 2024-09-18T04:39:29.570 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2266e-05 2024-09-18T04:39:29.570 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:29.570 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:29.570 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:29.570 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.570 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.570 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:39:29.570 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:39:29.570 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 121.768703191, 2024-09-18T04:39:29.571 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017521662, 2024-09-18T04:39:29.571 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:29.571 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:29.571 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:29.571 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:39:29.571 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:39:29.571 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 116 2024-09-18T04:39:29.571 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.571 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:29.571 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.571 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:29.571 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:39:29.571 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:29.572 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.572 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.572 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:29.572 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:39:29.572 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:29.572 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.572 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.573 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:29.573 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787813+0000", 2024-09-18T04:39:29.573 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:29.573 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.573 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.573 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:29.573 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787824+0000", 2024-09-18T04:39:29.573 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1157e-05 2024-09-18T04:39:29.573 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.573 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.573 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:29.574 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787840+0000", 2024-09-18T04:39:29.574 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6269e-05 2024-09-18T04:39:29.574 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.574 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.574 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:29.574 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787853+0000", 2024-09-18T04:39:29.574 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3156000000000001e-05 2024-09-18T04:39:29.574 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.574 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.574 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:29.574 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803110+0000", 2024-09-18T04:39:29.574 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015256736 2024-09-18T04:39:29.574 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.574 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.575 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:29.575 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803197+0000", 2024-09-18T04:39:29.575 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7195999999999997e-05 2024-09-18T04:39:29.575 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.575 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.575 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:29.575 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803308+0000", 2024-09-18T04:39:29.575 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111005 2024-09-18T04:39:29.575 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.575 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.575 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:29.575 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804316+0000", 2024-09-18T04:39:29.575 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010073949999999999 2024-09-18T04:39:29.575 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.576 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.576 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:29.576 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805232+0000", 2024-09-18T04:39:29.576 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091636300000000001 2024-09-18T04:39:29.576 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.576 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.576 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:29.576 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805277+0000", 2024-09-18T04:39:29.576 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4864000000000003e-05 2024-09-18T04:39:29.576 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.576 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.576 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:29.576 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805336+0000", 2024-09-18T04:39:29.576 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8968000000000002e-05 2024-09-18T04:39:29.576 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:29.577 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:29.577 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:29.577 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.577 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.577 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:39:29.577 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:39:29.577 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 121.768651136, 2024-09-18T04:39:29.577 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017983453, 2024-09-18T04:39:29.577 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:29.577 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:29.577 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:29.577 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:39:29.577 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:39:29.577 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 117 2024-09-18T04:39:29.578 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.578 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:29.578 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.578 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:29.578 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:39:29.578 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:29.578 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.578 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.578 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:29.578 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:39:29.578 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:29.578 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.578 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.578 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:29.578 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787865+0000", 2024-09-18T04:39:29.579 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:29.579 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.579 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.579 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:29.579 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787889+0000", 2024-09-18T04:39:29.579 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4244999999999999e-05 2024-09-18T04:39:29.579 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.579 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.579 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:29.579 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787905+0000", 2024-09-18T04:39:29.580 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5739999999999998e-05 2024-09-18T04:39:29.580 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.580 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.580 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:29.580 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787918+0000", 2024-09-18T04:39:29.580 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.252e-05 2024-09-18T04:39:29.580 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.580 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.580 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:29.580 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803693+0000", 2024-09-18T04:39:29.580 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015774872999999998 2024-09-18T04:39:29.580 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.580 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.580 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:29.581 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803775+0000", 2024-09-18T04:39:29.581 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2761000000000003e-05 2024-09-18T04:39:29.581 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.581 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.581 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:29.581 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803887+0000", 2024-09-18T04:39:29.581 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111708 2024-09-18T04:39:29.581 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.581 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.581 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:29.581 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804823+0000", 2024-09-18T04:39:29.581 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00093625100000000003 2024-09-18T04:39:29.581 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.582 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.582 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:29.582 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805804+0000", 2024-09-18T04:39:29.582 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00098114699999999992 2024-09-18T04:39:29.582 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.582 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.582 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:29.582 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805831+0000", 2024-09-18T04:39:29.582 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6401999999999999e-05 2024-09-18T04:39:29.582 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.582 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.582 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:29.582 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805850+0000", 2024-09-18T04:39:29.582 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8862e-05 2024-09-18T04:39:29.583 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:29.583 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:29.583 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:29.583 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.583 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.583 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:36 2.7 2:fe1b02ad:::rbd_data.125fc09df6d1.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:29.583 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:39:29.583 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 47.290795025999998, 2024-09-18T04:39:29.583 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.040387804999999999, 2024-09-18T04:39:29.583 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:29.583 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:29.583 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:29.583 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:29.583 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:29.583 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 36 2024-09-18T04:39:29.584 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.584 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:29.584 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.584 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:29.584 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:39:29.584 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:29.584 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.584 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.584 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:29.585 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:39:29.585 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:29.585 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.585 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.585 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:29.585 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265720+0000", 2024-09-18T04:39:29.585 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:29.585 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.585 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.586 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:29.586 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272590+0000", 2024-09-18T04:39:29.586 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.006870508 2024-09-18T04:39:29.586 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.586 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.586 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:29.586 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272623+0000", 2024-09-18T04:39:29.586 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2716000000000003e-05 2024-09-18T04:39:29.586 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.586 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.586 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:29.586 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272637+0000", 2024-09-18T04:39:29.586 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4175999999999999e-05 2024-09-18T04:39:29.586 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.587 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.587 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:29.587 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272779+0000", 2024-09-18T04:39:29.587 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000141392 2024-09-18T04:39:29.587 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.587 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.587 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:29.587 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273031+0000", 2024-09-18T04:39:29.587 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00025249099999999998 2024-09-18T04:39:29.587 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.587 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.587 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:39:29.587 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273219+0000", 2024-09-18T04:39:29.587 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000187531 2024-09-18T04:39:29.588 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.588 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.588 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:29.588 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.302806+0000", 2024-09-18T04:39:29.588 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.029587553999999999 2024-09-18T04:39:29.588 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.588 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.588 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:29.588 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306004+0000", 2024-09-18T04:39:29.588 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0031982629999999998 2024-09-18T04:39:29.588 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.588 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.588 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:29.588 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306071+0000", 2024-09-18T04:39:29.588 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6829999999999995e-05 2024-09-18T04:39:29.589 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.589 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.589 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:29.589 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306110+0000", 2024-09-18T04:39:29.589 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8874000000000001e-05 2024-09-18T04:39:29.589 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:29.589 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:29.589 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:29.589 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.589 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.589 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:38 2.0 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:29.589 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:39:29.589 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 47.283841488999997, 2024-09-18T04:39:29.589 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.033754348000000003, 2024-09-18T04:39:29.590 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:29.590 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:29.590 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:29.590 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:29.590 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:29.590 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 38 2024-09-18T04:39:29.590 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.590 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:29.590 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.590 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:29.590 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:39:29.590 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:29.590 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.590 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.590 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:29.591 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:39:29.591 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:29.591 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.591 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.591 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:29.591 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272673+0000", 2024-09-18T04:39:29.591 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:29.591 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.591 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.591 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:29.591 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277166+0000", 2024-09-18T04:39:29.591 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044922249999999999 2024-09-18T04:39:29.591 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.591 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.592 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:29.592 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277189+0000", 2024-09-18T04:39:29.592 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3283000000000001e-05 2024-09-18T04:39:29.592 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.592 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.592 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:29.592 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277205+0000", 2024-09-18T04:39:29.592 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6305000000000001e-05 2024-09-18T04:39:29.592 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.592 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.592 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:29.592 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277290+0000", 2024-09-18T04:39:29.592 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5135000000000004e-05 2024-09-18T04:39:29.592 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.593 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.593 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:29.593 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277531+0000", 2024-09-18T04:39:29.593 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00024014699999999999 2024-09-18T04:39:29.593 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.593 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.593 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:29.593 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277793+0000", 2024-09-18T04:39:29.593 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00026224499999999998 2024-09-18T04:39:29.593 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.593 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.593 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:29.593 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303990+0000", 2024-09-18T04:39:29.593 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.026197295999999998 2024-09-18T04:39:29.594 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.594 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.594 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:29.594 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306340+0000", 2024-09-18T04:39:29.594 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.002349524 2024-09-18T04:39:29.594 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.594 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.594 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:29.594 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306394+0000", 2024-09-18T04:39:29.594 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3934999999999998e-05 2024-09-18T04:39:29.594 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.594 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.594 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:29.594 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306430+0000", 2024-09-18T04:39:29.595 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6690000000000003e-05 2024-09-18T04:39:29.595 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:29.595 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:29.595 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:29.595 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.595 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.595 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:39 2.7 2:e9c909c5:::rbd_data.125fc09df6d1.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:29.595 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:39:29.595 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 47.279281884, 2024-09-18T04:39:29.595 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.029110673, 2024-09-18T04:39:29.595 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:29.595 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:29.595 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:29.595 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:29.596 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:29.596 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 39 2024-09-18T04:39:29.596 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.596 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:29.596 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.596 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:29.596 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:39:29.596 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:29.596 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.596 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.596 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:29.596 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:39:29.596 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:29.596 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.597 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.597 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:29.597 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277234+0000", 2024-09-18T04:39:29.597 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:29.597 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.597 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.597 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:29.597 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281351+0000", 2024-09-18T04:39:29.597 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0041170920000000001 2024-09-18T04:39:29.597 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.597 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.597 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:29.597 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281375+0000", 2024-09-18T04:39:29.597 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.446e-05 2024-09-18T04:39:29.598 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.598 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.598 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:29.598 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281390+0000", 2024-09-18T04:39:29.598 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4309e-05 2024-09-18T04:39:29.598 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.598 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.598 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:29.598 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.282306+0000", 2024-09-18T04:39:29.598 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091662000000000004 2024-09-18T04:39:29.598 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.598 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.598 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:29.598 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.282613+0000", 2024-09-18T04:39:29.599 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00030663799999999997 2024-09-18T04:39:29.599 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.599 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.599 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:39:29.599 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283001+0000", 2024-09-18T04:39:29.599 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00038801599999999998 2024-09-18T04:39:29.599 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.599 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.599 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:29.599 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306239+0000", 2024-09-18T04:39:29.599 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023238466999999999 2024-09-18T04:39:29.599 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.599 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.599 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:29.599 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306259+0000", 2024-09-18T04:39:29.600 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9837999999999999e-05 2024-09-18T04:39:29.600 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.600 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.600 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:29.600 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306299+0000", 2024-09-18T04:39:29.600 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0194000000000002e-05 2024-09-18T04:39:29.600 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.600 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.600 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:29.600 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306346+0000", 2024-09-18T04:39:29.600 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6845000000000003e-05 2024-09-18T04:39:29.600 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:29.600 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:29.601 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:29.601 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.601 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.601 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:46 2.6 e51/17 2:7ba25245:::rbd_data.125fc09df6d1.0000000000000006:head v 51'2, mlcod=0'0)", 2024-09-18T04:39:29.601 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:39:29.601 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 47.243360752000001, 2024-09-18T04:39:29.601 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019843098, 2024-09-18T04:39:29.601 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:29.601 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:29.601 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:29.601 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.601 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:29.601 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:39:29.601 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:29.601 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.602 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.602 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:29.602 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:39:29.602 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:29.602 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.602 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.602 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:29.602 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313156+0000", 2024-09-18T04:39:29.602 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:29.602 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.602 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.602 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:29.602 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320144+0000", 2024-09-18T04:39:29.602 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.006988345 2024-09-18T04:39:29.603 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.603 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.603 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:29.603 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320175+0000", 2024-09-18T04:39:29.603 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.0861999999999997e-05 2024-09-18T04:39:29.603 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.603 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.603 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:29.603 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320188+0000", 2024-09-18T04:39:29.603 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2714999999999999e-05 2024-09-18T04:39:29.603 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.603 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.603 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:29.603 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320267+0000", 2024-09-18T04:39:29.603 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9586999999999995e-05 2024-09-18T04:39:29.604 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.604 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.604 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:29.604 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320311+0000", 2024-09-18T04:39:29.604 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3529000000000003e-05 2024-09-18T04:39:29.604 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.604 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.604 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:29.604 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.332949+0000", 2024-09-18T04:39:29.604 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012638267999999999 2024-09-18T04:39:29.604 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.604 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.604 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:29.605 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.333000+0000", 2024-09-18T04:39:29.605 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0736999999999999e-05 2024-09-18T04:39:29.605 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:29.605 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:29.605 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:29.605 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.605 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.605 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:57 2.0 2:0cdd070e:::rbd_data.125fc09df6d1.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:29.605 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:39:29.605 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 47.207503177, 2024-09-18T04:39:29.605 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.033329974999999998, 2024-09-18T04:39:29.605 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:29.605 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:29.606 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:29.606 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:29.606 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:29.606 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 57 2024-09-18T04:39:29.606 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.606 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:29.606 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.606 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:29.606 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:39:29.606 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:29.607 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.607 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.607 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:29.607 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:39:29.607 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:29.607 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.607 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.607 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:29.607 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349013+0000", 2024-09-18T04:39:29.607 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:29.607 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.608 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.608 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:29.608 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356138+0000", 2024-09-18T04:39:29.608 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071258010000000002 2024-09-18T04:39:29.608 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.608 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.608 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:29.608 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356168+0000", 2024-09-18T04:39:29.608 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9679000000000002e-05 2024-09-18T04:39:29.608 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.608 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.608 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:29.608 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356181+0000", 2024-09-18T04:39:29.608 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3164e-05 2024-09-18T04:39:29.609 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.609 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.609 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:29.609 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356254+0000", 2024-09-18T04:39:29.609 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3182999999999994e-05 2024-09-18T04:39:29.609 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.609 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.609 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:29.609 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356453+0000", 2024-09-18T04:39:29.609 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019876599999999999 2024-09-18T04:39:29.609 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.609 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.609 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:29.609 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356627+0000", 2024-09-18T04:39:29.609 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017340199999999999 2024-09-18T04:39:29.610 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.610 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.610 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:29.610 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.369200+0000", 2024-09-18T04:39:29.610 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012573302 2024-09-18T04:39:29.610 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.610 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.610 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:29.610 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382262+0000", 2024-09-18T04:39:29.611 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013062549 2024-09-18T04:39:29.611 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.611 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.611 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:29.611 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382310+0000", 2024-09-18T04:39:29.611 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7366e-05 2024-09-18T04:39:29.611 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.611 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.611 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:29.611 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382344+0000", 2024-09-18T04:39:29.611 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4319999999999997e-05 2024-09-18T04:39:29.611 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:29.611 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:29.611 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:29.611 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.612 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.612 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:59 2.0 2:19c52cde:::rbd_data.125fc09df6d1.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:29.612 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:39:29.612 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 47.200312011999998, 2024-09-18T04:39:29.612 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.027137247999999999, 2024-09-18T04:39:29.612 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:29.612 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:29.612 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:29.612 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:29.612 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:29.612 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 59 2024-09-18T04:39:29.612 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.612 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:29.613 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.613 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:29.613 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:39:29.613 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:29.613 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.613 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.613 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:29.613 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:39:29.613 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:29.613 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.613 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.613 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:29.614 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356204+0000", 2024-09-18T04:39:29.614 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:29.614 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.614 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.614 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:29.614 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362661+0000", 2024-09-18T04:39:29.614 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0064565129999999997 2024-09-18T04:39:29.614 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.614 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.614 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:29.614 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362684+0000", 2024-09-18T04:39:29.614 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3688000000000001e-05 2024-09-18T04:39:29.614 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.614 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.615 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:29.615 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362699+0000", 2024-09-18T04:39:29.615 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4798e-05 2024-09-18T04:39:29.615 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.615 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.615 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:29.615 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363016+0000", 2024-09-18T04:39:29.615 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031737300000000003 2024-09-18T04:39:29.615 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.615 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.616 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:29.616 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363201+0000", 2024-09-18T04:39:29.616 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000184131 2024-09-18T04:39:29.616 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.616 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.616 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:29.616 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363343+0000", 2024-09-18T04:39:29.616 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014236699999999999 2024-09-18T04:39:29.616 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.616 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.616 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:29.616 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.372189+0000", 2024-09-18T04:39:29.616 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0088464819999999993 2024-09-18T04:39:29.616 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.617 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.617 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:29.617 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383269+0000", 2024-09-18T04:39:29.617 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011079594 2024-09-18T04:39:29.617 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.617 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.617 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:29.617 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383314+0000", 2024-09-18T04:39:29.617 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4932000000000003e-05 2024-09-18T04:39:29.617 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.617 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.617 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:29.618 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383343+0000", 2024-09-18T04:39:29.618 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8600999999999999e-05 2024-09-18T04:39:29.618 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:29.618 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:29.618 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:29.618 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.618 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.618 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:40 2.6 e51/17 2:784d6cd1:ns::rbd_data.133e813ab5bc.0000000000000005:head v 51'7, mlcod=51'5)", 2024-09-18T04:39:29.618 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:39:29.618 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 47.004062277999999, 2024-09-18T04:39:29.618 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020497496, 2024-09-18T04:39:29.618 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:29.618 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:29.618 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:29.619 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.619 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:29.619 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:39:29.619 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:29.619 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.619 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.619 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:29.619 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:39:29.619 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:29.619 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.619 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.619 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:29.619 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552454+0000", 2024-09-18T04:39:29.619 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:29.619 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.620 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.620 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:29.620 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556448+0000", 2024-09-18T04:39:29.620 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0039940990000000001 2024-09-18T04:39:29.620 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.620 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.620 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:29.620 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556481+0000", 2024-09-18T04:39:29.620 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3778000000000001e-05 2024-09-18T04:39:29.620 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.620 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.620 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:29.620 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556493+0000", 2024-09-18T04:39:29.620 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1622e-05 2024-09-18T04:39:29.621 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.621 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.621 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:29.621 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556556+0000", 2024-09-18T04:39:29.621 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2582999999999994e-05 2024-09-18T04:39:29.621 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.621 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.621 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:29.621 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556592+0000", 2024-09-18T04:39:29.621 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5880999999999997e-05 2024-09-18T04:39:29.622 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.622 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.622 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:29.622 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.572786+0000", 2024-09-18T04:39:29.622 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016194571000000001 2024-09-18T04:39:29.622 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.622 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.622 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:29.622 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.572953+0000", 2024-09-18T04:39:29.622 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000166436 2024-09-18T04:39:29.622 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:29.623 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:29.623 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:29.623 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.623 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.623 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:45 2.0 2:0fbfa3dc:ns::rbd_data.133e813ab5bc.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:29.623 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:39:29.623 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 46.992847978999997, 2024-09-18T04:39:29.623 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.036558168000000002, 2024-09-18T04:39:29.623 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:29.623 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:29.623 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:29.623 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:29.623 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:29.624 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 45 2024-09-18T04:39:29.624 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.624 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:29.624 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.624 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:29.624 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:39:29.624 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:29.624 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.624 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.624 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:29.624 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:39:29.624 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:29.624 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.624 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.625 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:29.625 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:39:29.625 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:29.625 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.625 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.625 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:29.625 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578594+0000", 2024-09-18T04:39:29.625 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014927147999999999 2024-09-18T04:39:29.625 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.625 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.626 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:29.626 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578612+0000", 2024-09-18T04:39:29.626 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7623999999999999e-05 2024-09-18T04:39:29.626 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.626 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.626 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:29.626 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578620+0000", 2024-09-18T04:39:29.626 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9070000000000007e-06 2024-09-18T04:39:29.627 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.627 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.627 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:29.627 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578702+0000", 2024-09-18T04:39:29.627 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2349999999999996e-05 2024-09-18T04:39:29.627 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.627 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.627 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:29.627 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.579028+0000", 2024-09-18T04:39:29.627 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032622899999999998 2024-09-18T04:39:29.628 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.628 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.628 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:29.628 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.579343+0000", 2024-09-18T04:39:29.628 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031448899999999999 2024-09-18T04:39:29.628 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.628 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.628 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:29.629 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.590078+0000", 2024-09-18T04:39:29.629 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010734946 2024-09-18T04:39:29.629 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.629 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.629 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:29.629 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600089+0000", 2024-09-18T04:39:29.629 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010011195000000001 2024-09-18T04:39:29.629 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.629 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.629 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:29.629 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600177+0000", 2024-09-18T04:39:29.629 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8425999999999997e-05 2024-09-18T04:39:29.629 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.629 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.630 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:29.630 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600228+0000", 2024-09-18T04:39:29.630 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0216999999999997e-05 2024-09-18T04:39:29.630 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:29.630 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:29.630 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:29.630 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.630 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.630 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:54 2.0 2:04f17ef6:ns::rbd_data.133e813ab5bc.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:29.630 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:39:29.630 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 46.971276142000001, 2024-09-18T04:39:29.631 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.024612999, 2024-09-18T04:39:29.631 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:29.631 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:29.631 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:29.631 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:29.631 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:29.631 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 54 2024-09-18T04:39:29.631 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.631 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:29.632 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.632 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:29.632 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:39:29.632 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:29.632 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.632 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.632 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:29.632 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:39:29.633 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:29.633 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.633 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.633 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:29.633 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585239+0000", 2024-09-18T04:39:29.633 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:29.633 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.633 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.633 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:29.633 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593702+0000", 2024-09-18T04:39:29.634 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0084632990000000005 2024-09-18T04:39:29.634 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.634 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.634 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:29.634 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593719+0000", 2024-09-18T04:39:29.634 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.664e-05 2024-09-18T04:39:29.634 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.634 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.634 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:29.634 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593727+0000", 2024-09-18T04:39:29.635 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6839999999999996e-06 2024-09-18T04:39:29.635 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.635 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.635 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:29.635 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593767+0000", 2024-09-18T04:39:29.635 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0330999999999997e-05 2024-09-18T04:39:29.635 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.635 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.635 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:29.635 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593905+0000", 2024-09-18T04:39:29.635 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013754800000000001 2024-09-18T04:39:29.636 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.636 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.636 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:29.636 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.594013+0000", 2024-09-18T04:39:29.636 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000108805 2024-09-18T04:39:29.636 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.636 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.636 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:29.636 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604678+0000", 2024-09-18T04:39:29.636 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010664391 2024-09-18T04:39:29.636 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.636 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.636 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:29.637 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609662+0000", 2024-09-18T04:39:29.637 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0049846100000000004 2024-09-18T04:39:29.637 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.637 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.637 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:29.637 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609788+0000", 2024-09-18T04:39:29.637 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012518800000000001 2024-09-18T04:39:29.637 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.637 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.637 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:29.637 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609854+0000", 2024-09-18T04:39:29.637 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6556000000000004e-05 2024-09-18T04:39:29.637 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:29.637 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:29.638 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:29.638 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.638 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.638 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:58 2.7 2:fc7c6cd3:ns::rbd_data.133e813ab5bc.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:29.638 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:39:29.638 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 46.958433712999998, 2024-09-18T04:39:29.638 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.027690566999999999, 2024-09-18T04:39:29.638 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:29.638 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:29.638 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:29.638 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:29.639 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:29.639 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 58 2024-09-18T04:39:29.639 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.639 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:29.639 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.639 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:29.639 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:39:29.639 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:29.639 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.639 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.640 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:29.640 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:39:29.640 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:29.640 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.640 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.640 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:29.640 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598082+0000", 2024-09-18T04:39:29.640 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:29.640 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.640 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.640 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:29.641 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604832+0000", 2024-09-18T04:39:29.641 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0067504510000000002 2024-09-18T04:39:29.641 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.641 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.641 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:29.641 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604852+0000", 2024-09-18T04:39:29.641 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0228999999999999e-05 2024-09-18T04:39:29.641 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.642 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.642 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:29.642 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604870+0000", 2024-09-18T04:39:29.642 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7952e-05 2024-09-18T04:39:29.642 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.642 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.642 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:29.642 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604936+0000", 2024-09-18T04:39:29.642 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5431000000000004e-05 2024-09-18T04:39:29.642 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.642 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.643 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:29.643 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605085+0000", 2024-09-18T04:39:29.643 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014910300000000001 2024-09-18T04:39:29.643 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.643 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.643 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:39:29.643 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605205+0000", 2024-09-18T04:39:29.643 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000119887 2024-09-18T04:39:29.643 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.643 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.643 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:29.643 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625586+0000", 2024-09-18T04:39:29.643 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020381399000000001 2024-09-18T04:39:29.644 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.644 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.644 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:29.644 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625628+0000", 2024-09-18T04:39:29.644 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1873999999999999e-05 2024-09-18T04:39:29.644 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.644 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.644 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:29.644 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625722+0000", 2024-09-18T04:39:29.644 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3787000000000002e-05 2024-09-18T04:39:29.645 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.645 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.645 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:29.645 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625774+0000", 2024-09-18T04:39:29.645 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2503999999999997e-05 2024-09-18T04:39:29.645 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:29.645 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:29.645 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:29.645 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.646 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.646 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:56 2.6 e51/17 2:72453c3e:ns::rbd_data.133e813ab5bc.000000000000000d:head v 51'10, mlcod=51'8)", 2024-09-18T04:39:29.646 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:39:29.646 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 46.951236604999998, 2024-09-18T04:39:29.646 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019766552999999999, 2024-09-18T04:39:29.646 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:29.646 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:29.646 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:29.646 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.647 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:29.647 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:39:29.647 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:29.647 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.647 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.647 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:29.647 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:39:29.647 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:29.647 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.647 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.648 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:29.648 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605279+0000", 2024-09-18T04:39:29.648 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:29.648 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.648 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.648 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:29.648 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609417+0000", 2024-09-18T04:39:29.648 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0041378059999999999 2024-09-18T04:39:29.649 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.649 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.649 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:29.649 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609443+0000", 2024-09-18T04:39:29.649 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6098e-05 2024-09-18T04:39:29.649 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.649 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.649 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:29.649 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609458+0000", 2024-09-18T04:39:29.650 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4671000000000001e-05 2024-09-18T04:39:29.650 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.650 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.650 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:29.650 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609553+0000", 2024-09-18T04:39:29.650 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5265000000000004e-05 2024-09-18T04:39:29.650 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.650 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.650 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:29.650 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609583+0000", 2024-09-18T04:39:29.651 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9839e-05 2024-09-18T04:39:29.651 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.651 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.651 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:29.651 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625004+0000", 2024-09-18T04:39:29.651 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015420639999999999 2024-09-18T04:39:29.651 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.651 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.651 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:29.651 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625047+0000", 2024-09-18T04:39:29.651 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3665999999999998e-05 2024-09-18T04:39:29.651 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:29.652 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:29.652 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:29.652 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.652 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.652 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:99 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:29.652 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.582718+0000", 2024-09-18T04:39:29.652 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 32.973799696, 2024-09-18T04:39:29.652 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019362575, 2024-09-18T04:39:29.652 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:29.652 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:29.653 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:29.653 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:29.653 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:29.653 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 99 2024-09-18T04:39:29.653 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.653 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:29.654 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.654 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:29.654 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582718+0000", 2024-09-18T04:39:29.654 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:29.654 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.654 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.654 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:29.654 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582718+0000", 2024-09-18T04:39:29.654 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:29.654 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.654 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.655 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:29.655 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582717+0000", 2024-09-18T04:39:29.655 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:29.655 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.655 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.655 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:29.655 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582720+0000", 2024-09-18T04:39:29.655 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2239999999999998e-06 2024-09-18T04:39:29.655 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.655 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.655 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:29.656 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582723+0000", 2024-09-18T04:39:29.656 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.258e-06 2024-09-18T04:39:29.656 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.656 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.656 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:29.656 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582726+0000", 2024-09-18T04:39:29.656 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8870000000000001e-06 2024-09-18T04:39:29.656 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.656 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.656 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:29.656 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.592018+0000", 2024-09-18T04:39:29.656 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0092926420000000003 2024-09-18T04:39:29.657 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.657 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.657 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:29.657 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.592214+0000", 2024-09-18T04:39:29.657 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000195267 2024-09-18T04:39:29.657 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.657 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.657 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:29.657 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.592913+0000", 2024-09-18T04:39:29.657 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00069924600000000005 2024-09-18T04:39:29.658 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.658 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.658 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:29.658 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597493+0000", 2024-09-18T04:39:29.658 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045797629999999997 2024-09-18T04:39:29.658 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.658 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.658 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:29.658 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.602009+0000", 2024-09-18T04:39:29.658 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045159559999999998 2024-09-18T04:39:29.658 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.658 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.659 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:29.659 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.602042+0000", 2024-09-18T04:39:29.659 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3084999999999999e-05 2024-09-18T04:39:29.659 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.659 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.659 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:29.659 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.602080+0000", 2024-09-18T04:39:29.659 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8600000000000003e-05 2024-09-18T04:39:29.659 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:29.659 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:29.660 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:29.660 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.660 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.660 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:104 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:29.660 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.583165+0000", 2024-09-18T04:39:29.660 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 32.973352398000003, 2024-09-18T04:39:29.660 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023679603, 2024-09-18T04:39:29.660 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:29.660 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:29.660 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:29.661 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:29.661 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:29.661 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 104 2024-09-18T04:39:29.661 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.661 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:29.661 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.661 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:29.661 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583165+0000", 2024-09-18T04:39:29.661 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:29.661 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.662 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.662 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:29.662 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583165+0000", 2024-09-18T04:39:29.662 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:29.662 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.662 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.662 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:29.662 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583164+0000", 2024-09-18T04:39:29.662 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:29.662 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.662 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.663 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:29.663 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583167+0000", 2024-09-18T04:39:29.663 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9210000000000001e-06 2024-09-18T04:39:29.663 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.663 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.663 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:29.663 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583171+0000", 2024-09-18T04:39:29.663 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9759999999999997e-06 2024-09-18T04:39:29.663 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.663 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.663 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:29.664 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583174+0000", 2024-09-18T04:39:29.664 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9960000000000002e-06 2024-09-18T04:39:29.664 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.664 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.664 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:29.664 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.594373+0000", 2024-09-18T04:39:29.664 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011198665999999999 2024-09-18T04:39:29.664 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.664 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.664 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:29.664 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.594434+0000", 2024-09-18T04:39:29.664 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.1446999999999996e-05 2024-09-18T04:39:29.664 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.665 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.665 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:29.665 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.594692+0000", 2024-09-18T04:39:29.665 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00025756599999999998 2024-09-18T04:39:29.665 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.665 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.665 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:29.665 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599302+0000", 2024-09-18T04:39:29.665 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0046099640000000002 2024-09-18T04:39:29.665 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.665 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.665 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:29.666 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606710+0000", 2024-09-18T04:39:29.666 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0074077739999999998 2024-09-18T04:39:29.666 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.666 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.666 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:29.666 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606798+0000", 2024-09-18T04:39:29.666 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8133999999999993e-05 2024-09-18T04:39:29.666 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.666 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.666 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:29.666 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606845+0000", 2024-09-18T04:39:29.666 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6653000000000002e-05 2024-09-18T04:39:29.666 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:29.667 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:29.667 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:29.667 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.667 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.667 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:105 2.0 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:29.667 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.583193+0000", 2024-09-18T04:39:29.667 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 32.973324779000002, 2024-09-18T04:39:29.667 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023713649, 2024-09-18T04:39:29.667 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:29.667 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:29.667 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:29.667 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:29.668 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:29.668 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 105 2024-09-18T04:39:29.668 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.668 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:29.668 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.668 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:29.668 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583193+0000", 2024-09-18T04:39:29.668 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:29.668 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.669 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.669 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:29.669 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583193+0000", 2024-09-18T04:39:29.669 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:29.669 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.669 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.669 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:29.669 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583192+0000", 2024-09-18T04:39:29.669 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:29.670 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.670 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.670 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:29.670 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583194+0000", 2024-09-18T04:39:29.670 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.305e-06 2024-09-18T04:39:29.670 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.670 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.670 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:29.670 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583198+0000", 2024-09-18T04:39:29.671 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9369999999999997e-06 2024-09-18T04:39:29.671 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.671 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.671 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:29.671 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583202+0000", 2024-09-18T04:39:29.671 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6480000000000001e-06 2024-09-18T04:39:29.671 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.671 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.671 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:29.671 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.595304+0000", 2024-09-18T04:39:29.671 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012102063999999999 2024-09-18T04:39:29.671 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.671 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.672 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:29.672 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.595390+0000", 2024-09-18T04:39:29.672 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5493000000000003e-05 2024-09-18T04:39:29.672 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.672 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.672 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:29.672 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.595509+0000", 2024-09-18T04:39:29.672 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011966199999999999 2024-09-18T04:39:29.672 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.672 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.672 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:29.672 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599358+0000", 2024-09-18T04:39:29.672 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0038486689999999999 2024-09-18T04:39:29.672 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.673 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.673 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:29.673 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606859+0000", 2024-09-18T04:39:29.673 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0075010789999999999 2024-09-18T04:39:29.673 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.673 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.673 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:29.673 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606887+0000", 2024-09-18T04:39:29.673 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7773e-05 2024-09-18T04:39:29.673 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.673 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.673 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:29.673 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606906+0000", 2024-09-18T04:39:29.674 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9417000000000001e-05 2024-09-18T04:39:29.674 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:29.674 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:29.674 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:29.674 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.674 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.674 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:106 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:29.674 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.584425+0000", 2024-09-18T04:39:29.674 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 32.972092482999997, 2024-09-18T04:39:29.674 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.022529293999999998, 2024-09-18T04:39:29.674 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:29.675 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:29.675 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:29.675 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:29.675 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:29.675 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 106 2024-09-18T04:39:29.675 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.675 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:29.675 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.675 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:29.675 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584425+0000", 2024-09-18T04:39:29.675 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:29.676 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.676 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.676 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:29.676 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584425+0000", 2024-09-18T04:39:29.676 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:29.676 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.676 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.676 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:29.676 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584424+0000", 2024-09-18T04:39:29.676 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:29.676 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.676 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.677 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:29.677 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584430+0000", 2024-09-18T04:39:29.677 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6389999999999997e-06 2024-09-18T04:39:29.677 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.677 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.677 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:29.677 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584456+0000", 2024-09-18T04:39:29.677 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6188999999999999e-05 2024-09-18T04:39:29.677 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.677 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.677 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:29.677 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584463+0000", 2024-09-18T04:39:29.677 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7340000000000002e-06 2024-09-18T04:39:29.678 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.678 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.678 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:29.678 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597537+0000", 2024-09-18T04:39:29.678 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013074161000000001 2024-09-18T04:39:29.678 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.678 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.678 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:29.678 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597598+0000", 2024-09-18T04:39:29.678 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0930000000000001e-05 2024-09-18T04:39:29.678 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.679 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.679 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:29.679 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597922+0000", 2024-09-18T04:39:29.679 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032473000000000002 2024-09-18T04:39:29.679 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.679 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.679 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:29.679 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.601103+0000", 2024-09-18T04:39:29.679 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0031809849999999999 2024-09-18T04:39:29.679 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.680 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.680 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:29.680 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606912+0000", 2024-09-18T04:39:29.680 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058083550000000003 2024-09-18T04:39:29.680 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.680 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.680 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:29.680 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606935+0000", 2024-09-18T04:39:29.681 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2759999999999999e-05 2024-09-18T04:39:29.681 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.681 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.681 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:29.681 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606954+0000", 2024-09-18T04:39:29.681 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9602999999999998e-05 2024-09-18T04:39:29.681 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:29.681 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:29.682 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:29.682 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.682 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.682 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:109 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:29.682 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.585231+0000", 2024-09-18T04:39:29.682 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 32.971285862999999, 2024-09-18T04:39:29.682 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.021767496000000001, 2024-09-18T04:39:29.682 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:29.682 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:29.682 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:29.682 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:29.682 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:29.682 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 109 2024-09-18T04:39:29.683 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.683 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:29.683 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.683 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:29.683 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585231+0000", 2024-09-18T04:39:29.683 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:29.683 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.683 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.683 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:29.683 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585231+0000", 2024-09-18T04:39:29.683 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:29.683 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.684 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.684 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:29.684 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585230+0000", 2024-09-18T04:39:29.684 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:29.684 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.684 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.684 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:29.684 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585237+0000", 2024-09-18T04:39:29.684 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.618e-06 2024-09-18T04:39:29.684 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.684 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.685 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:29.685 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585256+0000", 2024-09-18T04:39:29.685 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9048000000000001e-05 2024-09-18T04:39:29.685 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.685 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.685 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:29.685 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585274+0000", 2024-09-18T04:39:29.685 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8153000000000001e-05 2024-09-18T04:39:29.685 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.685 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.685 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:29.685 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.598284+0000", 2024-09-18T04:39:29.685 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013009682999999999 2024-09-18T04:39:29.685 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.686 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.686 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:29.686 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.598383+0000", 2024-09-18T04:39:29.686 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.8722999999999997e-05 2024-09-18T04:39:29.686 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.686 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.686 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:29.686 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.598829+0000", 2024-09-18T04:39:29.686 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00044623999999999998 2024-09-18T04:39:29.686 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.686 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.686 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:29.686 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.601149+0000", 2024-09-18T04:39:29.686 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0023204150000000002 2024-09-18T04:39:29.687 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.687 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.687 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:29.687 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606960+0000", 2024-09-18T04:39:29.687 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058106390000000003 2024-09-18T04:39:29.687 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.687 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.687 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:29.687 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606982+0000", 2024-09-18T04:39:29.687 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2252e-05 2024-09-18T04:39:29.687 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.687 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.687 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:29.687 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606999+0000", 2024-09-18T04:39:29.687 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6871000000000002e-05 2024-09-18T04:39:29.688 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:29.688 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:29.688 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:29.688 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.688 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.688 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:110 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:29.688 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.586640+0000", 2024-09-18T04:39:29.688 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 32.969877543999999, 2024-09-18T04:39:29.688 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020400911000000001, 2024-09-18T04:39:29.688 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:29.688 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:29.688 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:29.688 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:29.688 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:29.689 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 110 2024-09-18T04:39:29.689 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.689 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:29.689 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.689 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:29.689 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586640+0000", 2024-09-18T04:39:29.689 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:29.689 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.689 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.689 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:29.689 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586640+0000", 2024-09-18T04:39:29.689 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:29.689 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.690 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.690 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:29.690 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586639+0000", 2024-09-18T04:39:29.690 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:29.690 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.690 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.690 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:29.690 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586652+0000", 2024-09-18T04:39:29.690 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2482e-05 2024-09-18T04:39:29.690 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.690 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.690 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:29.691 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586660+0000", 2024-09-18T04:39:29.691 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1659999999999994e-06 2024-09-18T04:39:29.691 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.691 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.691 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:29.691 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586666+0000", 2024-09-18T04:39:29.691 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6900000000000003e-06 2024-09-18T04:39:29.691 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.691 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.691 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:29.691 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599381+0000", 2024-09-18T04:39:29.691 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012714328 2024-09-18T04:39:29.691 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.692 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.692 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:29.692 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599436+0000", 2024-09-18T04:39:29.692 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4802999999999997e-05 2024-09-18T04:39:29.692 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.692 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.692 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:29.692 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599629+0000", 2024-09-18T04:39:29.692 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000193921 2024-09-18T04:39:29.692 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.692 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.693 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:29.693 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.601174+0000", 2024-09-18T04:39:29.693 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001544857 2024-09-18T04:39:29.693 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.693 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.693 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:29.693 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.607004+0000", 2024-09-18T04:39:29.693 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058300820000000003 2024-09-18T04:39:29.693 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.693 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.693 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:29.693 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.607024+0000", 2024-09-18T04:39:29.694 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9888e-05 2024-09-18T04:39:29.694 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:29.694 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:29.694 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:29.694 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.607041+0000", 2024-09-18T04:39:29.694 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6450999999999999e-05 2024-09-18T04:39:29.694 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:29.694 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:29.694 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:29.694 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:29.694 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:29.694 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:29.696 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:39:29.696 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:39:29.697 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:39:29.698 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:29.689+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:29.698 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --op export --pgid 2.5 --file /var/log/ceph/exp.2.5.2 2024-09-18T04:39:29.699 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:39:29.699 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:39:29.700 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:39:29.700 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T04:39:29.701 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:39:29.701 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:39:29.701 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-18T04:39:29.701 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:39:29.701 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:39:29.702 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:39:29.702 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T04:39:29.703 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-18T04:39:29.797 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:29.789+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:29.820 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:29.821 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:29.830 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:39:29.897 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:29.889+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:29.936 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:29.936 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:29.946 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:39:29.998 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:29.993+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:30.053 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:30.053 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:30.062 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:39:30.099 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:30.093+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:30.171 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:30.171 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:30.181 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:39:30.222 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:30.217+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:30.301 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:30.293+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:30.329 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:30.330 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:30.339 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:39:30.402 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:30.400+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:30.441 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:30.441 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:30.451 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:39:30.503 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:30.497+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:30.577 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:30.577 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:30.587 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:39:30.603 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:30.600+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:30.644 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd device --device-type nbd list 2024-09-18T04:39:30.645 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false grep '^19491\b' 2024-09-18T04:39:30.645 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep '^19491\b' 2024-09-18T04:39:30.645 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-09-18T04:39:30.694 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:30.694 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:30.705 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:39:30.705 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:30.697+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:30.713 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:39:30.713 INFO:tasks.workunit.client.0.smithi115.stderr:+ ps -C rbd-nbd 2024-09-18T04:39:30.714 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false grep '^ *19491\b' 2024-09-18T04:39:30.714 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep '^ *19491\b' 2024-09-18T04:39:30.723 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:39:30.723 INFO:tasks.workunit.client.0.smithi115.stderr:+ sleep 0.5 2024-09-18T04:39:30.806 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:30.801+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:30.807 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:30.807 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:30.818 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:39:30.882 INFO:teuthology.orchestra.run.smithi028.stderr:Exporting 2.5 info 2.5( v 70'58 (0'0,70'58] local-lis/les=28/29 n=7 ec=17/17 lis/c=28/28 les/c/f=29/29/0 sis=28) 2024-09-18T04:39:30.893 INFO:teuthology.orchestra.run.smithi028.stderr:Read #2:a06f51e4:ns::rbd_data.133e813ab5bc.000000000000000f:head# 2024-09-18T04:39:30.903 INFO:teuthology.orchestra.run.smithi028.stderr:Read #2:b227e6b1:::rbd_data.125fc09df6d1.000000000000000e:head# 2024-09-18T04:39:30.907 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:30.905+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:30.909 INFO:teuthology.orchestra.run.smithi028.stderr:Read #2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head# 2024-09-18T04:39:30.916 INFO:teuthology.orchestra.run.smithi028.stderr:Read #2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:head# 2024-09-18T04:39:30.925 INFO:teuthology.orchestra.run.smithi028.stderr:Read #2:bb27d472:::rbd_object_map.125fc09df6d1.0000000000000006:head# 2024-09-18T04:39:30.925 INFO:teuthology.orchestra.run.smithi028.stderr:Read #2:bd5b8dd8:ns::rbd_data.133e813ab5bc.0000000000000002:head# 2024-09-18T04:39:30.929 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:30.929 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:30.931 INFO:teuthology.orchestra.run.smithi028.stderr:Read #2:bf39b8e4:ns::rbd_data.133e813ab5bc.0000000000000003:head# 2024-09-18T04:39:30.937 INFO:teuthology.orchestra.run.smithi028.stderr:Export successful 2024-09-18T04:39:30.940 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:39:31.008 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:31.001+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:31.038 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:31.038 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:31.047 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:39:31.109 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:31.101+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:31.158 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:31.159 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:31.170 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:39:31.214 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:31.209+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:31.226 INFO:tasks.workunit.client.0.smithi115.stderr:+ LOG_FILE=/tmp/tmp.WJMNJxIdAN/rbd-nbd.log 2024-09-18T04:39:31.226 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T04:39:31.226 INFO:tasks.workunit.client.0.smithi115.stderr:++ _sudo rbd device --device-type nbd map --quiesce rbd/testrbdnbd17342 --log-file=/tmp/tmp.WJMNJxIdAN/rbd-nbd.log 2024-09-18T04:39:31.226 INFO:tasks.workunit.client.0.smithi115.stderr:++ local cmd 2024-09-18T04:39:31.227 INFO:tasks.workunit.client.0.smithi115.stderr:+++ id -u 2024-09-18T04:39:31.228 INFO:tasks.workunit.client.0.smithi115.stderr:++ '[' 1000 -eq 0 ']' 2024-09-18T04:39:31.229 INFO:tasks.workunit.client.0.smithi115.stderr:+++ which rbd 2024-09-18T04:39:31.232 INFO:tasks.workunit.client.0.smithi115.stderr:++ cmd=/usr/bin/rbd 2024-09-18T04:39:31.232 INFO:tasks.workunit.client.0.smithi115.stderr:++ shift 2024-09-18T04:39:31.232 INFO:tasks.workunit.client.0.smithi115.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map --quiesce rbd/testrbdnbd17342 --log-file=/tmp/tmp.WJMNJxIdAN/rbd-nbd.log 2024-09-18T04:39:31.286 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:31.286 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:31.296 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:39:31.311 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:31.305+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:31.401 INFO:tasks.workunit.client.0.smithi115.stderr:+ DEV=/dev/nbd0 2024-09-18T04:39:31.401 INFO:tasks.workunit.client.0.smithi115.stderr:+ get_pid rbd 2024-09-18T04:39:31.401 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pool=rbd 2024-09-18T04:39:31.401 INFO:tasks.workunit.client.0.smithi115.stderr:+ local ns= 2024-09-18T04:39:31.401 INFO:tasks.workunit.client.0.smithi115.stderr:++ rbd device --device-type nbd --format xml list 2024-09-18T04:39:31.402 INFO:tasks.workunit.client.0.smithi115.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd17342'\''][device='\''/dev/nbd0'\'']/id' 2024-09-18T04:39:31.412 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:31.405+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:31.414 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:31.414 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:31.424 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:39:31.488 INFO:tasks.workunit.client.0.smithi115.stderr:+ PID=19676 2024-09-18T04:39:31.488 INFO:tasks.workunit.client.0.smithi115.stderr:+ test -n 19676 2024-09-18T04:39:31.488 INFO:tasks.workunit.client.0.smithi115.stderr:+ ps -p 19676 -C rbd-nbd 2024-09-18T04:39:31.500 INFO:tasks.workunit.client.0.smithi115.stdout: PID TTY TIME CMD 2024-09-18T04:39:31.500 INFO:tasks.workunit.client.0.smithi115.stdout: 19676 ? 00:00:00 rbd-nbd 2024-09-18T04:39:31.500 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo mkfs /dev/nbd0 2024-09-18T04:39:31.500 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:39:31.501 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:39:31.502 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:39:31.502 INFO:tasks.workunit.client.0.smithi115.stderr:++ which mkfs 2024-09-18T04:39:31.504 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/sbin/mkfs 2024-09-18T04:39:31.504 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:39:31.504 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/sbin/mkfs /dev/nbd0 2024-09-18T04:39:31.513 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:31.505+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:31.524 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --force --op remove --pgid 2.5 2024-09-18T04:39:31.579 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:31.580 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:31.589 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:39:31.634 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:31.629+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:31.665 INFO:tasks.workunit.client.0.smithi115.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-18T04:39:31.700 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:31.700 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:31.703 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T04:39:31.712 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:39:31.713 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:31.708+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:31.740 INFO:tasks.workunit.client.0.smithi115.stdout:Discarding device blocks: 4096/16384 done 2024-09-18T04:39:31.740 INFO:tasks.workunit.client.0.smithi115.stdout:Creating filesystem with 16384 4k blocks and 16384 inodes 2024-09-18T04:39:31.740 INFO:tasks.workunit.client.0.smithi115.stdout: 2024-09-18T04:39:31.740 INFO:tasks.workunit.client.0.smithi115.stdout:Allocating group tables: 0/1 done 2024-09-18T04:39:31.758 INFO:tasks.workunit.client.0.smithi115.stdout:Writing inode tables: 0/1 done 2024-09-18T04:39:31.786 INFO:tasks.workunit.client.0.smithi115.stdout:Writing superblocks and filesystem accounting information: 0/1 done 2024-09-18T04:39:31.786 INFO:tasks.workunit.client.0.smithi115.stdout: 2024-09-18T04:39:31.788 INFO:tasks.workunit.client.0.smithi115.stderr:+ mkdir /tmp/tmp.WJMNJxIdAN/mnt 2024-09-18T04:39:31.790 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo mount /dev/nbd0 /tmp/tmp.WJMNJxIdAN/mnt 2024-09-18T04:39:31.790 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:39:31.791 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:39:31.793 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:39:31.794 INFO:tasks.workunit.client.0.smithi115.stderr:++ which mount 2024-09-18T04:39:31.796 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/bin/mount 2024-09-18T04:39:31.796 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:39:31.796 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/bin/mount /dev/nbd0 /tmp/tmp.WJMNJxIdAN/mnt 2024-09-18T04:39:31.811 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:31.811 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:31.815 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:31.809+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:31.821 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:39:31.848 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd snap create rbd/testrbdnbd17342@quiesce3 2024-09-18T04:39:31.852 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:39:31.872 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:39:31.896 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:39:31.915 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:31.909+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:31.927 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:39:31.928 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:31.928 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:39:31.928 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:39:31.928 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:39:31.928 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:31.928 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.5431.0:1 3.3 3:d80af3cb:::rbd_id.testrbdnbd17342:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-18T04:39:31.928 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:13.542818+0000", 2024-09-18T04:39:31.928 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 18.382704607000001, 2024-09-18T04:39:31.928 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048493400000000002, 2024-09-18T04:39:31.929 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:31.929 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:39:31.929 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:31.929 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.5431", 2024-09-18T04:39:31.929 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3147136083", 2024-09-18T04:39:31.929 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 1 2024-09-18T04:39:31.929 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:31.929 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:31.930 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:31.930 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:31.930 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542818+0000", 2024-09-18T04:39:31.930 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:31.930 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:31.930 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:31.930 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:31.930 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542818+0000", 2024-09-18T04:39:31.930 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:31.930 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:31.930 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:31.930 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:31.930 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542816+0000", 2024-09-18T04:39:31.931 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:31.931 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:31.931 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:31.931 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:31.931 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542828+0000", 2024-09-18T04:39:31.931 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2699e-05 2024-09-18T04:39:31.931 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:31.931 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:31.931 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:31.931 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542852+0000", 2024-09-18T04:39:31.931 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3176e-05 2024-09-18T04:39:31.931 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:31.931 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:31.932 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:31.932 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542874+0000", 2024-09-18T04:39:31.932 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1967000000000002e-05 2024-09-18T04:39:31.932 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:31.932 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:31.932 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:31.932 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542984+0000", 2024-09-18T04:39:31.932 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011026099999999999 2024-09-18T04:39:31.932 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:31.932 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:31.932 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:31.932 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543161+0000", 2024-09-18T04:39:31.932 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017720299999999999 2024-09-18T04:39:31.933 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:31.933 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:31.933 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:31.933 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543303+0000", 2024-09-18T04:39:31.933 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014174700000000001 2024-09-18T04:39:31.933 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:31.933 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:31.933 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:31.933 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:31.933 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:31.933 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.5431.0:2 3.3 3:d80af3cb:::rbd_id.testrbdnbd17342:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-18T04:39:31.933 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:13.543564+0000", 2024-09-18T04:39:31.934 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 18.381958121, 2024-09-18T04:39:31.934 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010399929999999999, 2024-09-18T04:39:31.934 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:31.934 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:39:31.934 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:31.934 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.5431", 2024-09-18T04:39:31.934 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3147136083", 2024-09-18T04:39:31.934 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 2 2024-09-18T04:39:31.934 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:31.934 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:31.934 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:31.934 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:31.935 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543564+0000", 2024-09-18T04:39:31.935 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:31.935 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:31.935 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:31.935 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:31.935 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543564+0000", 2024-09-18T04:39:31.935 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:31.935 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:31.935 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:31.935 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:31.935 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543563+0000", 2024-09-18T04:39:31.935 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:31.935 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:31.935 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:31.936 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:31.936 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543571+0000", 2024-09-18T04:39:31.936 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4270000000000008e-06 2024-09-18T04:39:31.936 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:31.936 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:31.936 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:31.936 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543591+0000", 2024-09-18T04:39:31.936 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.929e-05 2024-09-18T04:39:31.936 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:31.936 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:31.936 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:31.936 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543600+0000", 2024-09-18T04:39:31.936 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.1390000000000001e-06 2024-09-18T04:39:31.936 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:31.937 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:31.937 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:31.937 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543724+0000", 2024-09-18T04:39:31.937 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012458400000000001 2024-09-18T04:39:31.937 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:31.937 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:31.937 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:31.937 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543968+0000", 2024-09-18T04:39:31.937 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00024323199999999999 2024-09-18T04:39:31.937 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:31.937 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:31.937 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:31.937 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.544604+0000", 2024-09-18T04:39:31.937 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00063675799999999996 2024-09-18T04:39:31.938 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:31.938 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:31.938 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:31.938 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:31.938 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:31.938 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:31.939 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:39:31.950 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:39:31.951 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:39:31.951 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:39:31.951 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-18T04:39:31.951 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:39:31.951 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:39:31.951 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:39:31.955 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T04:39:31.956 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T04:39:32.017 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:32.012+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:32.043 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:32.043 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:32.043 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:39:32.043 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:32.053 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:39:32.117 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:32.109+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:32.159 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:32.160 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:32.160 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:39:32.160 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:39:32.160 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:32.169 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:39:32.218 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:32.213+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:32.281 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:32.281 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:39:32.282 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:39:32.282 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:39:32.282 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.282 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-18T04:39:32.282 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:39:32.282 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 117.806250249, 2024-09-18T04:39:32.282 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0095781189999999995, 2024-09-18T04:39:32.282 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:32.282 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:32.282 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:32.282 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:39:32.282 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:39:32.282 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 1 2024-09-18T04:39:32.282 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.283 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:32.283 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.283 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:32.283 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:39:32.283 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:32.283 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.283 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.283 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:32.283 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:39:32.283 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:32.283 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.283 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.283 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:32.283 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469862+0000", 2024-09-18T04:39:32.284 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:32.284 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.284 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.284 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:32.284 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469879+0000", 2024-09-18T04:39:32.284 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7431999999999998e-05 2024-09-18T04:39:32.284 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.284 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.285 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:32.285 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469896+0000", 2024-09-18T04:39:32.285 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6742999999999999e-05 2024-09-18T04:39:32.285 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.285 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.285 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:32.285 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469915+0000", 2024-09-18T04:39:32.286 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8991000000000002e-05 2024-09-18T04:39:32.286 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.286 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.286 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:32.286 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470397+0000", 2024-09-18T04:39:32.286 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048167799999999998 2024-09-18T04:39:32.286 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.286 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.286 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-09-18T04:39:32.286 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470412+0000", 2024-09-18T04:39:32.286 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5526e-05 2024-09-18T04:39:32.286 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.286 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.287 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:32.287 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473279+0000", 2024-09-18T04:39:32.287 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0028667469999999998 2024-09-18T04:39:32.287 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.287 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.287 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:32.287 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473441+0000", 2024-09-18T04:39:32.287 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000162082 2024-09-18T04:39:32.287 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.287 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.287 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:32.287 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473573+0000", 2024-09-18T04:39:32.287 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000132032 2024-09-18T04:39:32.288 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.288 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.288 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:32.288 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474826+0000", 2024-09-18T04:39:32.288 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001252717 2024-09-18T04:39:32.288 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.288 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.288 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:32.288 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479307+0000", 2024-09-18T04:39:32.288 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044813220000000003 2024-09-18T04:39:32.288 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.288 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.288 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:32.289 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479366+0000", 2024-09-18T04:39:32.289 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9024999999999998e-05 2024-09-18T04:39:32.289 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.289 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.289 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:32.289 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479443+0000", 2024-09-18T04:39:32.289 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6339000000000002e-05 2024-09-18T04:39:32.289 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:32.289 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:32.289 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:32.289 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.290 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.290 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-18T04:39:32.290 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:39:32.290 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 115.793241829, 2024-09-18T04:39:32.290 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082699560000000002, 2024-09-18T04:39:32.290 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:32.290 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:32.290 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:32.290 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:39:32.290 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:39:32.290 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 7 2024-09-18T04:39:32.290 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.290 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:32.291 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.291 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:32.291 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:39:32.291 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:32.291 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.291 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.291 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:32.291 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:39:32.291 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:32.291 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.291 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.291 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:32.291 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482870+0000", 2024-09-18T04:39:32.292 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999967 2024-09-18T04:39:32.292 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.292 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.292 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:32.292 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482889+0000", 2024-09-18T04:39:32.292 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.965e-05 2024-09-18T04:39:32.292 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.292 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.292 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:32.292 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482938+0000", 2024-09-18T04:39:32.292 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8548999999999999e-05 2024-09-18T04:39:32.292 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.292 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.293 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:32.293 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482963+0000", 2024-09-18T04:39:32.293 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5075999999999998e-05 2024-09-18T04:39:32.293 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.293 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.293 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:32.293 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483121+0000", 2024-09-18T04:39:32.293 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015819400000000001 2024-09-18T04:39:32.293 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.293 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.293 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op must wait for map", 2024-09-18T04:39:32.293 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483138+0000", 2024-09-18T04:39:32.293 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7068999999999999e-05 2024-09-18T04:39:32.294 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.294 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.294 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:32.294 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488666+0000", 2024-09-18T04:39:32.294 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0055282049999999996 2024-09-18T04:39:32.294 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.294 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.294 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:32.294 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488783+0000", 2024-09-18T04:39:32.294 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000116737 2024-09-18T04:39:32.295 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.295 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.295 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:32.295 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488991+0000", 2024-09-18T04:39:32.295 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020781400000000001 2024-09-18T04:39:32.295 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.295 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.295 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:32.295 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490554+0000", 2024-09-18T04:39:32.295 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0015628370000000001 2024-09-18T04:39:32.295 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.295 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.295 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:32.296 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491019+0000", 2024-09-18T04:39:32.296 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000465608 2024-09-18T04:39:32.296 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.296 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.296 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:32.296 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491087+0000", 2024-09-18T04:39:32.296 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7988e-05 2024-09-18T04:39:32.296 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.296 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.296 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:32.296 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491143+0000", 2024-09-18T04:39:32.296 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5538999999999998e-05 2024-09-18T04:39:32.296 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:32.296 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:32.297 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:32.297 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.297 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.297 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:34 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:32.297 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.261758+0000", 2024-09-18T04:39:32.297 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 50.014356677999999, 2024-09-18T04:39:32.297 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.042024191000000002, 2024-09-18T04:39:32.297 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:32.297 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:32.297 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:32.297 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:32.297 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:32.297 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 34 2024-09-18T04:39:32.298 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.298 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:32.298 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.298 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:32.298 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.261758+0000", 2024-09-18T04:39:32.298 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:32.298 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.298 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.298 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:32.298 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.261758+0000", 2024-09-18T04:39:32.298 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:32.298 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.298 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.298 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:32.299 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.261756+0000", 2024-09-18T04:39:32.299 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:32.299 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.299 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.299 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:32.299 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.270785+0000", 2024-09-18T04:39:32.299 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0090292340000000006 2024-09-18T04:39:32.299 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.299 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.299 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:32.299 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.270890+0000", 2024-09-18T04:39:32.299 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000104541 2024-09-18T04:39:32.299 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.299 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.300 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:32.300 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.270991+0000", 2024-09-18T04:39:32.300 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000100778 2024-09-18T04:39:32.300 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.300 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.300 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:32.300 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.271157+0000", 2024-09-18T04:39:32.300 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016653999999999999 2024-09-18T04:39:32.300 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.300 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.300 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:32.300 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.271591+0000", 2024-09-18T04:39:32.300 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00043409500000000001 2024-09-18T04:39:32.301 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.301 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.301 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:32.301 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272017+0000", 2024-09-18T04:39:32.301 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000425769 2024-09-18T04:39:32.301 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.301 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.301 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:32.301 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.294246+0000", 2024-09-18T04:39:32.301 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.022229237999999998 2024-09-18T04:39:32.301 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.301 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.301 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:32.301 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303571+0000", 2024-09-18T04:39:32.302 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0093250239999999995 2024-09-18T04:39:32.302 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.302 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.302 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:32.302 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303705+0000", 2024-09-18T04:39:32.302 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000133831 2024-09-18T04:39:32.302 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.302 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.302 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:32.302 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303782+0000", 2024-09-18T04:39:32.302 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7156000000000005e-05 2024-09-18T04:39:32.302 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:32.302 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:32.302 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:32.303 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.303 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.303 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:36 2.7 e51/32 2:fe1b02ad:::rbd_data.125fc09df6d1.0000000000000002:head v 51'1, mlcod=0'0)", 2024-09-18T04:39:32.303 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.273441+0000", 2024-09-18T04:39:32.303 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 50.002673696999999, 2024-09-18T04:39:32.303 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.032400768000000003, 2024-09-18T04:39:32.303 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:32.303 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:32.303 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:32.303 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.303 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:32.303 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273441+0000", 2024-09-18T04:39:32.303 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:32.303 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.304 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.304 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:32.304 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273441+0000", 2024-09-18T04:39:32.304 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:32.304 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.304 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.304 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:32.304 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273440+0000", 2024-09-18T04:39:32.304 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:32.304 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.304 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.304 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:32.304 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278723+0000", 2024-09-18T04:39:32.305 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0052829909999999999 2024-09-18T04:39:32.305 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.305 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.305 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:32.305 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278752+0000", 2024-09-18T04:39:32.305 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9178000000000001e-05 2024-09-18T04:39:32.305 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.305 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.305 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:32.305 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278761+0000", 2024-09-18T04:39:32.305 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7900000000000005e-06 2024-09-18T04:39:32.305 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.305 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.305 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:32.306 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278864+0000", 2024-09-18T04:39:32.306 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000102699 2024-09-18T04:39:32.306 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.306 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.307 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:32.307 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278900+0000", 2024-09-18T04:39:32.307 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6001999999999999e-05 2024-09-18T04:39:32.307 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.307 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.307 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:32.307 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.305793+0000", 2024-09-18T04:39:32.307 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.026893344 2024-09-18T04:39:32.307 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.307 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.307 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:32.307 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.305842+0000", 2024-09-18T04:39:32.307 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8720000000000001e-05 2024-09-18T04:39:32.308 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:32.308 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:32.308 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:32.308 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.308 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.308 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:39 2.7 e51/32 2:e9c909c5:::rbd_data.125fc09df6d1.0000000000000005:head v 51'2, mlcod=0'0)", 2024-09-18T04:39:32.308 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.283175+0000", 2024-09-18T04:39:32.308 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 49.992940007999998, 2024-09-18T04:39:32.309 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.022864102000000001, 2024-09-18T04:39:32.309 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:32.309 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:32.309 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:32.309 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.309 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:32.309 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283175+0000", 2024-09-18T04:39:32.309 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:32.309 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.309 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.309 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:32.309 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283175+0000", 2024-09-18T04:39:32.309 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:32.309 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.310 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.310 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:32.310 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283173+0000", 2024-09-18T04:39:32.310 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:32.310 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.310 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.310 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:32.310 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.285951+0000", 2024-09-18T04:39:32.310 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0027784540000000001 2024-09-18T04:39:32.310 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.310 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.311 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:32.311 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.285970+0000", 2024-09-18T04:39:32.311 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8855000000000002e-05 2024-09-18T04:39:32.311 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.311 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.311 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:32.311 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.285984+0000", 2024-09-18T04:39:32.311 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4252e-05 2024-09-18T04:39:32.311 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.311 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.311 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:32.311 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.286041+0000", 2024-09-18T04:39:32.312 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6431999999999998e-05 2024-09-18T04:39:32.312 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.312 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.312 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:32.312 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.286086+0000", 2024-09-18T04:39:32.312 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5423000000000002e-05 2024-09-18T04:39:32.312 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.312 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.312 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:32.312 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.305998+0000", 2024-09-18T04:39:32.312 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019911697999999999 2024-09-18T04:39:32.312 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.312 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.313 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:32.313 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306039+0000", 2024-09-18T04:39:32.313 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0973999999999998e-05 2024-09-18T04:39:32.313 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:32.313 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:32.313 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:32.313 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.313 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.314 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:50 2.3 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:32.314 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.312563+0000", 2024-09-18T04:39:32.314 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 49.963551981000002, 2024-09-18T04:39:32.314 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.032636001999999997, 2024-09-18T04:39:32.314 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:32.314 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:32.314 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:32.314 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:32.315 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:32.315 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 50 2024-09-18T04:39:32.315 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.315 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:32.315 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.315 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:32.315 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.312563+0000", 2024-09-18T04:39:32.315 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:32.315 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.316 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.316 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:32.316 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.312563+0000", 2024-09-18T04:39:32.316 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:32.316 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.316 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.316 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:32.316 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.312561+0000", 2024-09-18T04:39:32.316 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:32.316 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.316 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.317 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:32.317 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321709+0000", 2024-09-18T04:39:32.317 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0091486650000000003 2024-09-18T04:39:32.317 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.317 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.317 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:32.317 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321739+0000", 2024-09-18T04:39:32.317 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9666e-05 2024-09-18T04:39:32.317 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.317 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.317 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:32.317 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321751+0000", 2024-09-18T04:39:32.317 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2086000000000001e-05 2024-09-18T04:39:32.318 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.318 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.318 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:32.318 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321833+0000", 2024-09-18T04:39:32.318 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1842999999999996e-05 2024-09-18T04:39:32.318 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.318 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.318 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:32.318 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321981+0000", 2024-09-18T04:39:32.318 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000147441 2024-09-18T04:39:32.318 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.318 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.318 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:32.318 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.322109+0000", 2024-09-18T04:39:32.319 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012851200000000001 2024-09-18T04:39:32.319 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.319 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.319 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:32.319 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.332829+0000", 2024-09-18T04:39:32.319 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010720225999999999 2024-09-18T04:39:32.319 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.319 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.319 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:32.319 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.345039+0000", 2024-09-18T04:39:32.319 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012209831000000001 2024-09-18T04:39:32.319 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.319 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.319 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:32.320 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.345120+0000", 2024-09-18T04:39:32.320 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1145999999999999e-05 2024-09-18T04:39:32.320 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.320 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.320 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:32.320 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.345199+0000", 2024-09-18T04:39:32.320 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8546000000000003e-05 2024-09-18T04:39:32.320 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:32.320 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:32.320 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:32.320 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.320 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.320 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:30 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:32.321 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.516122+0000", 2024-09-18T04:39:32.321 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 49.759992818999997, 2024-09-18T04:39:32.321 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0091865030000000004, 2024-09-18T04:39:32.321 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:32.321 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:32.321 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:32.321 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:32.321 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:32.321 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 30 2024-09-18T04:39:32.321 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.321 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:32.321 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.321 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:32.321 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516122+0000", 2024-09-18T04:39:32.322 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:32.322 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.322 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.322 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:32.322 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516122+0000", 2024-09-18T04:39:32.322 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:32.322 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.322 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.322 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:32.322 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516120+0000", 2024-09-18T04:39:32.322 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:32.322 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.322 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.322 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:32.323 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516129+0000", 2024-09-18T04:39:32.323 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.85e-06 2024-09-18T04:39:32.323 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.323 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.323 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:32.323 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516155+0000", 2024-09-18T04:39:32.323 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5882000000000001e-05 2024-09-18T04:39:32.323 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.323 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.323 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:32.323 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516175+0000", 2024-09-18T04:39:32.323 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.022e-05 2024-09-18T04:39:32.323 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.323 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.324 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:32.324 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519780+0000", 2024-09-18T04:39:32.324 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003604187 2024-09-18T04:39:32.324 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.324 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.324 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:32.324 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519964+0000", 2024-09-18T04:39:32.324 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00018391999999999999 2024-09-18T04:39:32.324 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.324 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.324 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:32.324 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.520794+0000", 2024-09-18T04:39:32.324 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00083078100000000005 2024-09-18T04:39:32.324 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.325 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.325 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:32.325 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.524129+0000", 2024-09-18T04:39:32.325 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033342419999999998 2024-09-18T04:39:32.325 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.325 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.325 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:32.325 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525239+0000", 2024-09-18T04:39:32.325 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001110778 2024-09-18T04:39:32.325 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.325 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.325 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:32.325 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525277+0000", 2024-09-18T04:39:32.326 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7596999999999999e-05 2024-09-18T04:39:32.326 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.326 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.326 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:32.326 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525308+0000", 2024-09-18T04:39:32.326 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.1535999999999998e-05 2024-09-18T04:39:32.326 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:32.326 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:32.326 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:32.326 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.326 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.326 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:32 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:32.326 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.519217+0000", 2024-09-18T04:39:32.326 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 49.756897498999997, 2024-09-18T04:39:32.326 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082266370000000002, 2024-09-18T04:39:32.327 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:32.327 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:32.327 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:32.327 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:32.327 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:32.327 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 32 2024-09-18T04:39:32.327 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.327 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:32.327 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.327 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:32.327 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519217+0000", 2024-09-18T04:39:32.327 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:32.328 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.328 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.328 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:32.328 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519217+0000", 2024-09-18T04:39:32.328 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:32.328 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.328 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.328 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:32.328 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519215+0000", 2024-09-18T04:39:32.328 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:32.328 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.328 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.328 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:32.329 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519229+0000", 2024-09-18T04:39:32.329 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3329999999999999e-05 2024-09-18T04:39:32.329 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.329 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.329 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:32.329 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519268+0000", 2024-09-18T04:39:32.329 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9533000000000002e-05 2024-09-18T04:39:32.329 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.329 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.329 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:32.329 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519285+0000", 2024-09-18T04:39:32.329 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6446000000000001e-05 2024-09-18T04:39:32.329 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.329 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.330 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:32.330 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.522358+0000", 2024-09-18T04:39:32.330 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030734310000000002 2024-09-18T04:39:32.330 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.330 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.330 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:32.330 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.522551+0000", 2024-09-18T04:39:32.330 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019317600000000001 2024-09-18T04:39:32.330 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.330 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.330 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:32.330 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.523262+0000", 2024-09-18T04:39:32.330 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00071069999999999998 2024-09-18T04:39:32.330 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.331 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.331 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:32.331 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525318+0000", 2024-09-18T04:39:32.331 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0020561469999999999 2024-09-18T04:39:32.331 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.331 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.331 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:32.331 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.527376+0000", 2024-09-18T04:39:32.331 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0020584969999999998 2024-09-18T04:39:32.331 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.331 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.331 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:32.331 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.527416+0000", 2024-09-18T04:39:32.332 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9054000000000003e-05 2024-09-18T04:39:32.332 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.332 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.332 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:32.332 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.527444+0000", 2024-09-18T04:39:32.332 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8347000000000002e-05 2024-09-18T04:39:32.332 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:32.332 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:32.332 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:32.332 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.332 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.332 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:42 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:32.332 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.551160+0000", 2024-09-18T04:39:32.333 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 49.724955107, 2024-09-18T04:39:32.333 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010692757000000001, 2024-09-18T04:39:32.333 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:32.333 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:32.333 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:32.333 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:32.333 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:32.333 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 42 2024-09-18T04:39:32.333 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.333 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:32.333 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.333 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:32.333 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551160+0000", 2024-09-18T04:39:32.334 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:32.334 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.334 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.334 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:32.334 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551160+0000", 2024-09-18T04:39:32.334 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:32.334 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.334 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.334 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:32.334 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551159+0000", 2024-09-18T04:39:32.334 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:32.334 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.334 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.334 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:32.335 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551166+0000", 2024-09-18T04:39:32.335 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1029999999999996e-06 2024-09-18T04:39:32.335 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.335 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.335 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:32.335 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551177+0000", 2024-09-18T04:39:32.335 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1384e-05 2024-09-18T04:39:32.335 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.335 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.335 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:32.335 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551185+0000", 2024-09-18T04:39:32.335 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1990000000000006e-06 2024-09-18T04:39:32.335 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.335 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.336 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:32.336 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552021+0000", 2024-09-18T04:39:32.336 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00083574400000000003 2024-09-18T04:39:32.336 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.336 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.336 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:32.336 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552339+0000", 2024-09-18T04:39:32.336 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031771399999999997 2024-09-18T04:39:32.336 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.336 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.336 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:32.336 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.553140+0000", 2024-09-18T04:39:32.336 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00080088100000000003 2024-09-18T04:39:32.337 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.337 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.337 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:32.337 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.554585+0000", 2024-09-18T04:39:32.337 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001445801 2024-09-18T04:39:32.337 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.337 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.337 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:32.337 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.561711+0000", 2024-09-18T04:39:32.337 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071260109999999998 2024-09-18T04:39:32.337 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.337 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.337 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:32.337 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.561790+0000", 2024-09-18T04:39:32.338 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8967000000000005e-05 2024-09-18T04:39:32.338 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.338 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.338 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:32.338 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.561852+0000", 2024-09-18T04:39:32.338 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2045999999999995e-05 2024-09-18T04:39:32.338 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:32.338 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:32.338 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:32.338 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.338 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.338 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:58 2.7 e51/32 2:fc7c6cd3:ns::rbd_data.133e813ab5bc.000000000000000e:head v 51'7, mlcod=51'5)", 2024-09-18T04:39:32.338 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.605333+0000", 2024-09-18T04:39:32.338 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 49.670781902000002, 2024-09-18T04:39:32.339 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020085902999999999, 2024-09-18T04:39:32.339 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:32.339 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:32.339 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:32.339 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.339 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:32.339 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605333+0000", 2024-09-18T04:39:32.339 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:32.339 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.339 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.339 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:32.339 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605333+0000", 2024-09-18T04:39:32.340 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:32.340 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.340 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.340 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:32.340 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605332+0000", 2024-09-18T04:39:32.340 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:32.340 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.340 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.340 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:32.340 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608667+0000", 2024-09-18T04:39:32.340 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033350300000000001 2024-09-18T04:39:32.340 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.340 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.341 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:32.341 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608706+0000", 2024-09-18T04:39:32.341 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8951e-05 2024-09-18T04:39:32.341 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.341 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.341 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:32.341 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608725+0000", 2024-09-18T04:39:32.341 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9675999999999999e-05 2024-09-18T04:39:32.341 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.341 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.341 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:32.341 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608798+0000", 2024-09-18T04:39:32.341 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3135999999999999e-05 2024-09-18T04:39:32.341 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.342 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.342 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:32.342 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608824+0000", 2024-09-18T04:39:32.342 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5284000000000001e-05 2024-09-18T04:39:32.342 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.342 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.342 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:32.342 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625370+0000", 2024-09-18T04:39:32.343 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016545975000000001 2024-09-18T04:39:32.343 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.343 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.343 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:32.343 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625419+0000", 2024-09-18T04:39:32.343 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9076999999999998e-05 2024-09-18T04:39:32.343 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:32.343 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:32.343 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:32.343 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.343 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.343 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:611 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:32.343 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.069561+0000", 2024-09-18T04:39:32.343 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 41.206553348, 2024-09-18T04:39:32.344 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0072330299999999997, 2024-09-18T04:39:32.344 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:32.344 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:32.344 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:32.344 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:39:32.344 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:32.344 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 611 2024-09-18T04:39:32.344 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.344 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:32.345 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.345 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:32.345 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.069561+0000", 2024-09-18T04:39:32.345 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:32.345 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.345 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.345 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:32.345 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.069561+0000", 2024-09-18T04:39:32.345 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:32.345 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.345 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.345 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:32.346 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.069559+0000", 2024-09-18T04:39:32.346 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:32.346 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.346 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.346 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:32.346 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070801+0000", 2024-09-18T04:39:32.346 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0012423359999999999 2024-09-18T04:39:32.346 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.346 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.347 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:32.347 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070828+0000", 2024-09-18T04:39:32.347 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6387e-05 2024-09-18T04:39:32.347 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.347 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.347 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:32.347 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070840+0000", 2024-09-18T04:39:32.347 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2689e-05 2024-09-18T04:39:32.347 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.347 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.347 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:32.347 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070990+0000", 2024-09-18T04:39:32.348 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014946800000000001 2024-09-18T04:39:32.348 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.348 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.348 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:32.348 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.071128+0000", 2024-09-18T04:39:32.348 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013796699999999999 2024-09-18T04:39:32.348 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.348 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.348 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:32.348 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.071318+0000", 2024-09-18T04:39:32.348 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000190106 2024-09-18T04:39:32.349 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.349 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.349 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:32.349 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.075711+0000", 2024-09-18T04:39:32.349 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.004393002 2024-09-18T04:39:32.349 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.349 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.349 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:32.349 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.076665+0000", 2024-09-18T04:39:32.349 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00095390399999999999 2024-09-18T04:39:32.350 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.350 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.350 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:32.350 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.076742+0000", 2024-09-18T04:39:32.350 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7112000000000003e-05 2024-09-18T04:39:32.350 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.350 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.350 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:32.350 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.076794+0000", 2024-09-18T04:39:32.350 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2475999999999997e-05 2024-09-18T04:39:32.350 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:32.351 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:32.351 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:32.351 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.351 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.351 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:687 2.3 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:32.351 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.487994+0000", 2024-09-18T04:39:32.351 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 40.788120364999997, 2024-09-18T04:39:32.351 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0074167690000000001, 2024-09-18T04:39:32.351 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:32.352 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:32.352 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:32.352 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:39:32.352 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:32.352 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 687 2024-09-18T04:39:32.352 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.352 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:32.352 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.352 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:32.353 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.487994+0000", 2024-09-18T04:39:32.353 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:32.353 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.353 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.353 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:32.353 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.487994+0000", 2024-09-18T04:39:32.353 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:32.353 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.353 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.353 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:32.353 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.487993+0000", 2024-09-18T04:39:32.353 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:32.353 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.353 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.354 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:32.354 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.488979+0000", 2024-09-18T04:39:32.354 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00098620700000000001 2024-09-18T04:39:32.354 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.354 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.354 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:32.354 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489002+0000", 2024-09-18T04:39:32.354 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2500000000000001e-05 2024-09-18T04:39:32.354 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.354 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.354 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:32.354 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489020+0000", 2024-09-18T04:39:32.354 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7691000000000001e-05 2024-09-18T04:39:32.354 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.355 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.355 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:32.355 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489091+0000", 2024-09-18T04:39:32.355 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1307e-05 2024-09-18T04:39:32.355 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.355 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.355 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:32.355 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489219+0000", 2024-09-18T04:39:32.355 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012766000000000001 2024-09-18T04:39:32.355 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.355 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.355 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:32.355 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489387+0000", 2024-09-18T04:39:32.355 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016855400000000001 2024-09-18T04:39:32.355 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.356 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.356 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:32.356 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495243+0000", 2024-09-18T04:39:32.356 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058563189999999996 2024-09-18T04:39:32.356 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.356 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.356 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:32.356 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495266+0000", 2024-09-18T04:39:32.356 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2810999999999999e-05 2024-09-18T04:39:32.357 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.357 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.357 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:32.357 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495332+0000", 2024-09-18T04:39:32.357 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5469999999999995e-05 2024-09-18T04:39:32.357 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.357 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.357 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:32.357 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495411+0000", 2024-09-18T04:39:32.357 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9395999999999997e-05 2024-09-18T04:39:32.357 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:32.357 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:32.357 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:32.358 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.358 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.358 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:689 2.3 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:32.358 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.500070+0000", 2024-09-18T04:39:32.358 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 40.776044548999998, 2024-09-18T04:39:32.358 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071096650000000003, 2024-09-18T04:39:32.358 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:32.358 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:32.358 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:32.358 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:39:32.358 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:32.358 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 689 2024-09-18T04:39:32.358 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.359 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:32.359 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.359 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:32.359 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.500070+0000", 2024-09-18T04:39:32.359 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:32.359 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.359 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.359 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:32.359 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.500070+0000", 2024-09-18T04:39:32.359 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:32.359 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.359 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.359 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:32.359 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.500069+0000", 2024-09-18T04:39:32.360 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:32.360 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.360 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.360 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:32.360 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501082+0000", 2024-09-18T04:39:32.360 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010136570000000001 2024-09-18T04:39:32.360 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.360 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.360 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:32.360 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501104+0000", 2024-09-18T04:39:32.360 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1883e-05 2024-09-18T04:39:32.360 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.360 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.360 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:32.361 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501115+0000", 2024-09-18T04:39:32.361 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0414000000000001e-05 2024-09-18T04:39:32.361 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.361 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.361 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:32.361 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501178+0000", 2024-09-18T04:39:32.361 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3128999999999998e-05 2024-09-18T04:39:32.361 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.361 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.361 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:32.361 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501257+0000", 2024-09-18T04:39:32.361 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9592999999999995e-05 2024-09-18T04:39:32.361 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.361 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.362 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:32.362 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501366+0000", 2024-09-18T04:39:32.362 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000108643 2024-09-18T04:39:32.362 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.362 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.362 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:32.362 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.504041+0000", 2024-09-18T04:39:32.362 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0026750889999999999 2024-09-18T04:39:32.362 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.362 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.362 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:32.362 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.507097+0000", 2024-09-18T04:39:32.362 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030556070000000001 2024-09-18T04:39:32.362 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.363 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.363 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:32.363 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.507140+0000", 2024-09-18T04:39:32.363 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3745000000000002e-05 2024-09-18T04:39:32.363 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.363 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.363 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:32.363 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.507180+0000", 2024-09-18T04:39:32.363 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9475999999999999e-05 2024-09-18T04:39:32.363 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:32.363 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:32.363 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:32.363 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.364 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.364 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4972.0:13 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:32.364 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.948263+0000", 2024-09-18T04:39:32.364 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 40.327851758000001, 2024-09-18T04:39:32.364 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0091358970000000005, 2024-09-18T04:39:32.364 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:32.364 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:39:32.364 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:32.364 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4972", 2024-09-18T04:39:32.364 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2280251472", 2024-09-18T04:39:32.364 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 13 2024-09-18T04:39:32.364 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.364 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:32.365 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.365 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:32.365 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948263+0000", 2024-09-18T04:39:32.365 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:32.365 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.365 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.365 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:32.365 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948263+0000", 2024-09-18T04:39:32.365 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:32.365 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.365 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.365 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:32.365 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948261+0000", 2024-09-18T04:39:32.365 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:32.366 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.366 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.366 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:32.366 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948273+0000", 2024-09-18T04:39:32.366 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1576999999999999e-05 2024-09-18T04:39:32.366 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.366 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.366 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:32.366 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948305+0000", 2024-09-18T04:39:32.366 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2209000000000003e-05 2024-09-18T04:39:32.366 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.366 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.366 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:32.366 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948325+0000", 2024-09-18T04:39:32.366 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.022e-05 2024-09-18T04:39:32.367 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.367 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.367 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:32.367 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948455+0000", 2024-09-18T04:39:32.367 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000129624 2024-09-18T04:39:32.367 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.367 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.367 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:32.367 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948598+0000", 2024-09-18T04:39:32.367 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014376699999999999 2024-09-18T04:39:32.367 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.367 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.367 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:32.367 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.957399+0000", 2024-09-18T04:39:32.368 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0088004299999999997 2024-09-18T04:39:32.368 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:32.368 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:32.368 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:32.368 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.368 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.368 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:101 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:32.368 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.576367+0000", 2024-09-18T04:39:32.368 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 35.699748049999997, 2024-09-18T04:39:32.368 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0078671020000000008, 2024-09-18T04:39:32.368 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:32.368 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:32.368 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:32.369 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:32.369 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:32.369 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 101 2024-09-18T04:39:32.369 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.369 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:32.369 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.369 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:32.369 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576367+0000", 2024-09-18T04:39:32.369 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:32.369 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.369 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.369 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:32.369 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576367+0000", 2024-09-18T04:39:32.370 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:32.370 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.370 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.370 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:32.370 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576364+0000", 2024-09-18T04:39:32.370 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999971 2024-09-18T04:39:32.370 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.370 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.370 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:32.370 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576378+0000", 2024-09-18T04:39:32.370 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3879999999999999e-05 2024-09-18T04:39:32.370 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.370 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.371 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:32.371 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576410+0000", 2024-09-18T04:39:32.371 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2286000000000002e-05 2024-09-18T04:39:32.371 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.371 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.371 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:32.371 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576425+0000", 2024-09-18T04:39:32.371 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5290000000000001e-05 2024-09-18T04:39:32.371 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.371 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.371 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:32.371 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576551+0000", 2024-09-18T04:39:32.371 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012555199999999999 2024-09-18T04:39:32.371 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.371 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.372 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:32.372 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576719+0000", 2024-09-18T04:39:32.372 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016787400000000001 2024-09-18T04:39:32.372 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.372 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.372 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:32.372 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576951+0000", 2024-09-18T04:39:32.372 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00023252400000000001 2024-09-18T04:39:32.372 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.372 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.372 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:32.372 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.581459+0000", 2024-09-18T04:39:32.372 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045074410000000001 2024-09-18T04:39:32.372 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.373 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.373 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:32.373 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584076+0000", 2024-09-18T04:39:32.373 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0026173860000000002 2024-09-18T04:39:32.373 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.373 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.373 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:32.373 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584160+0000", 2024-09-18T04:39:32.374 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4191999999999996e-05 2024-09-18T04:39:32.374 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.374 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.374 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:32.374 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584234+0000", 2024-09-18T04:39:32.374 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.339e-05 2024-09-18T04:39:32.374 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:32.374 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:32.374 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:32.374 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.374 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.374 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:44 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:32.374 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.550448+0000", 2024-09-18T04:39:32.374 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 31.725666861000001, 2024-09-18T04:39:32.375 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082117590000000008, 2024-09-18T04:39:32.375 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:32.375 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:32.375 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:32.375 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:39:32.375 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:32.375 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 44 2024-09-18T04:39:32.375 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.375 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:32.375 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.375 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:32.376 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.550448+0000", 2024-09-18T04:39:32.376 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:32.376 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.376 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.376 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:32.376 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.550448+0000", 2024-09-18T04:39:32.376 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:32.376 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.376 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.376 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:32.377 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.550445+0000", 2024-09-18T04:39:32.377 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999971 2024-09-18T04:39:32.377 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.377 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.377 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:32.377 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.551904+0000", 2024-09-18T04:39:32.377 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001458322 2024-09-18T04:39:32.377 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.377 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.377 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:32.377 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.551938+0000", 2024-09-18T04:39:32.377 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4359000000000001e-05 2024-09-18T04:39:32.378 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.378 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.378 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:32.378 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.551959+0000", 2024-09-18T04:39:32.378 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1403000000000002e-05 2024-09-18T04:39:32.378 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.378 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.378 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:32.378 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.552049+0000", 2024-09-18T04:39:32.379 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9970999999999996e-05 2024-09-18T04:39:32.379 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.379 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.379 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:32.379 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.552190+0000", 2024-09-18T04:39:32.379 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000140271 2024-09-18T04:39:32.379 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.379 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.379 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:32.379 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.552385+0000", 2024-09-18T04:39:32.379 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019498700000000001 2024-09-18T04:39:32.380 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.380 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.380 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:32.380 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.556308+0000", 2024-09-18T04:39:32.380 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00392333 2024-09-18T04:39:32.380 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.380 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.380 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:32.380 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.558479+0000", 2024-09-18T04:39:32.380 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0021715889999999998 2024-09-18T04:39:32.380 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.380 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.381 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:32.381 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.558616+0000", 2024-09-18T04:39:32.381 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013613399999999999 2024-09-18T04:39:32.381 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.381 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.381 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:32.381 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.558660+0000", 2024-09-18T04:39:32.381 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4053e-05 2024-09-18T04:39:32.381 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:32.382 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:32.382 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:32.382 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.382 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.382 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:46 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:32.382 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.564089+0000", 2024-09-18T04:39:32.382 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 31.712025732000001, 2024-09-18T04:39:32.382 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0069437090000000002, 2024-09-18T04:39:32.382 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:32.382 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:32.382 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:32.383 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:39:32.383 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:32.383 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 46 2024-09-18T04:39:32.383 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.383 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:32.383 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.383 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:32.383 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.564089+0000", 2024-09-18T04:39:32.383 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:32.383 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.384 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.384 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:32.384 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.564089+0000", 2024-09-18T04:39:32.384 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:32.384 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.384 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.384 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:32.384 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.564087+0000", 2024-09-18T04:39:32.384 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:32.385 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.385 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.385 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:32.385 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565126+0000", 2024-09-18T04:39:32.385 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001038902 2024-09-18T04:39:32.385 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.385 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.385 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:32.385 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565148+0000", 2024-09-18T04:39:32.385 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1911000000000001e-05 2024-09-18T04:39:32.386 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.386 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.386 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:32.386 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565157+0000", 2024-09-18T04:39:32.386 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6220000000000004e-06 2024-09-18T04:39:32.386 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.386 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.386 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:32.386 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565261+0000", 2024-09-18T04:39:32.386 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000103063 2024-09-18T04:39:32.386 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.386 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.386 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:32.386 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565394+0000", 2024-09-18T04:39:32.387 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000133032 2024-09-18T04:39:32.387 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.387 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.387 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:32.387 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565564+0000", 2024-09-18T04:39:32.387 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017068000000000001 2024-09-18T04:39:32.387 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.387 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.387 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:32.387 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.570556+0000", 2024-09-18T04:39:32.387 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0049920620000000002 2024-09-18T04:39:32.387 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.387 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.387 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:32.388 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.570885+0000", 2024-09-18T04:39:32.388 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032827900000000001 2024-09-18T04:39:32.388 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.388 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.388 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:32.388 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.570959+0000", 2024-09-18T04:39:32.388 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3955999999999995e-05 2024-09-18T04:39:32.388 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.388 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.388 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:32.388 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.571033+0000", 2024-09-18T04:39:32.388 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4191999999999997e-05 2024-09-18T04:39:32.388 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:32.388 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:32.388 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:32.389 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.389 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.389 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:100 2.4 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:32.389 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.856389+0000", 2024-09-18T04:39:32.389 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 31.419725620000001, 2024-09-18T04:39:32.389 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0081480800000000003, 2024-09-18T04:39:32.389 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:32.389 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:32.389 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:32.390 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:39:32.390 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:32.390 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 100 2024-09-18T04:39:32.390 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.392 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:32.393 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.393 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:32.393 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.856389+0000", 2024-09-18T04:39:32.393 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:32.393 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.393 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.393 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:32.393 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.856389+0000", 2024-09-18T04:39:32.393 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:32.393 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.393 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.393 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:32.393 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.856388+0000", 2024-09-18T04:39:32.393 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:32.394 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.394 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.394 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:32.394 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857295+0000", 2024-09-18T04:39:32.394 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00090725599999999995 2024-09-18T04:39:32.394 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.394 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.394 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:32.395 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857311+0000", 2024-09-18T04:39:32.395 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6092000000000001e-05 2024-09-18T04:39:32.395 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.395 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.395 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:32.395 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857319+0000", 2024-09-18T04:39:32.395 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9200000000000004e-06 2024-09-18T04:39:32.395 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.395 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.395 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:32.395 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857388+0000", 2024-09-18T04:39:32.395 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8591999999999997e-05 2024-09-18T04:39:32.396 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.396 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.396 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:32.396 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857535+0000", 2024-09-18T04:39:32.396 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014684299999999999 2024-09-18T04:39:32.396 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.396 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.396 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:32.396 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857747+0000", 2024-09-18T04:39:32.396 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000212361 2024-09-18T04:39:32.396 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.396 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.397 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:32.397 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.860831+0000", 2024-09-18T04:39:32.397 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030841789999999998 2024-09-18T04:39:32.397 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.397 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.397 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:32.397 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.864361+0000", 2024-09-18T04:39:32.397 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0035298780000000002 2024-09-18T04:39:32.397 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.397 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.397 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:32.397 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.864456+0000", 2024-09-18T04:39:32.397 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4557999999999998e-05 2024-09-18T04:39:32.398 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.398 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.398 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:32.398 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.864537+0000", 2024-09-18T04:39:32.398 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1577999999999997e-05 2024-09-18T04:39:32.398 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:32.398 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:32.398 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:32.398 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.398 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.398 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:102 2.4 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:32.398 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.869773+0000", 2024-09-18T04:39:32.398 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 31.406341962999999, 2024-09-18T04:39:32.399 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0084430359999999992, 2024-09-18T04:39:32.399 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:32.399 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:32.399 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:32.399 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:39:32.399 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:32.399 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 102 2024-09-18T04:39:32.399 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.399 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:32.399 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.399 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:32.399 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.869773+0000", 2024-09-18T04:39:32.399 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:32.400 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.400 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.400 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:32.400 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.869773+0000", 2024-09-18T04:39:32.400 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:32.400 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.400 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.400 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:32.400 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.869771+0000", 2024-09-18T04:39:32.400 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:32.400 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.400 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.400 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:32.400 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870629+0000", 2024-09-18T04:39:32.401 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00085760999999999997 2024-09-18T04:39:32.401 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.401 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.401 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:32.401 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870647+0000", 2024-09-18T04:39:32.401 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8363999999999999e-05 2024-09-18T04:39:32.401 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.401 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.401 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:32.401 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870655+0000", 2024-09-18T04:39:32.401 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0099999999999995e-06 2024-09-18T04:39:32.401 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.402 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.402 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:32.402 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870803+0000", 2024-09-18T04:39:32.402 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014798 2024-09-18T04:39:32.402 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.402 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.402 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:32.402 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870951+0000", 2024-09-18T04:39:32.402 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014715999999999999 2024-09-18T04:39:32.402 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.402 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.402 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:32.402 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.871145+0000", 2024-09-18T04:39:32.402 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000194253 2024-09-18T04:39:32.403 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.403 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.403 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:32.403 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.877033+0000", 2024-09-18T04:39:32.403 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058883099999999999 2024-09-18T04:39:32.403 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.403 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.403 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:32.403 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.878078+0000", 2024-09-18T04:39:32.403 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001044747 2024-09-18T04:39:32.403 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.403 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.403 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:32.404 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.878160+0000", 2024-09-18T04:39:32.404 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1904999999999996e-05 2024-09-18T04:39:32.404 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.404 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.404 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:32.404 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.878216+0000", 2024-09-18T04:39:32.404 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6082000000000003e-05 2024-09-18T04:39:32.404 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:32.404 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:32.404 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:32.404 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.404 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.404 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:145 2.4 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:32.405 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:01.096935+0000", 2024-09-18T04:39:32.405 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 31.179180183, 2024-09-18T04:39:32.405 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0069937869999999996, 2024-09-18T04:39:32.405 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:32.405 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:32.405 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:32.405 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:39:32.405 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:32.405 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 145 2024-09-18T04:39:32.405 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.405 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:32.405 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.405 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:32.406 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.096935+0000", 2024-09-18T04:39:32.406 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:32.406 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.406 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.406 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:32.406 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.096935+0000", 2024-09-18T04:39:32.406 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:32.406 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.406 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.406 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:32.406 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.096932+0000", 2024-09-18T04:39:32.406 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:32.406 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.407 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.407 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:32.407 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097815+0000", 2024-09-18T04:39:32.407 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00088276000000000001 2024-09-18T04:39:32.407 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.407 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.407 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:32.407 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097842+0000", 2024-09-18T04:39:32.407 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6806999999999999e-05 2024-09-18T04:39:32.407 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.407 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.407 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:32.407 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097852+0000", 2024-09-18T04:39:32.407 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0548e-05 2024-09-18T04:39:32.408 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.408 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.408 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:32.408 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097931+0000", 2024-09-18T04:39:32.408 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8417000000000005e-05 2024-09-18T04:39:32.408 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.408 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.408 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:32.408 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.098047+0000", 2024-09-18T04:39:32.408 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011611 2024-09-18T04:39:32.408 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.408 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.408 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:32.409 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.098172+0000", 2024-09-18T04:39:32.409 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000125482 2024-09-18T04:39:32.409 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.409 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.409 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:32.409 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.100888+0000", 2024-09-18T04:39:32.409 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0027154430000000001 2024-09-18T04:39:32.409 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.409 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.409 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:32.409 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.103814+0000", 2024-09-18T04:39:32.409 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0029259619999999998 2024-09-18T04:39:32.409 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.409 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.410 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:32.410 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.103891+0000", 2024-09-18T04:39:32.410 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7619000000000003e-05 2024-09-18T04:39:32.410 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:32.410 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:32.410 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:32.410 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.103928+0000", 2024-09-18T04:39:32.410 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6900999999999998e-05 2024-09-18T04:39:32.410 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:32.411 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:32.411 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:32.411 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:32.411 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:32.411 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:32.412 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:32.409+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:32.413 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:39:32.513 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:32.509+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:32.540 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:32.540 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:32.540 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:39:32.541 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:32.550 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:39:32.614 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:32.608+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:32.656 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-18T04:39:32.670 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:32.670 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:32.670 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:39:32.670 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:39:32.670 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:32.671 INFO:tasks.workunit.client.0.smithi115.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-18T04:39:32.671 INFO:teuthology.orchestra.run.smithi028.stdout: marking collection for removal 2024-09-18T04:39:32.672 INFO:teuthology.orchestra.run.smithi028.stdout:setting '_remove' omap key 2024-09-18T04:39:32.672 INFO:teuthology.orchestra.run.smithi028.stdout:finish_remove_pgs 2.5_head removing 2.5 2024-09-18T04:39:32.680 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo dd if=/tmp/tmp.WJMNJxIdAN/data of=/tmp/tmp.WJMNJxIdAN/mnt/test bs=1M count=1 oflag=direct 2024-09-18T04:39:32.680 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:39:32.680 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:39:32.682 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:39:32.683 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:39:32.683 INFO:tasks.workunit.client.0.smithi115.stderr:++ which dd 2024-09-18T04:39:32.684 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/bin/dd 2024-09-18T04:39:32.684 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:39:32.684 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.WJMNJxIdAN/data of=/tmp/tmp.WJMNJxIdAN/mnt/test bs=1M count=1 oflag=direct 2024-09-18T04:39:32.692 INFO:teuthology.orchestra.run.smithi028.stdout:Remove successful 2024-09-18T04:39:32.700 INFO:tasks.workunit.client.0.smithi115.stderr:1+0 records in 2024-09-18T04:39:32.700 INFO:tasks.workunit.client.0.smithi115.stderr:1+0 records out 2024-09-18T04:39:32.700 INFO:tasks.workunit.client.0.smithi115.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00333019 s, 315 MB/s 2024-09-18T04:39:32.701 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo umount /tmp/tmp.WJMNJxIdAN/mnt 2024-09-18T04:39:32.701 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:39:32.701 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:39:32.704 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:39:32.704 INFO:tasks.workunit.client.0.smithi115.stderr:++ which umount 2024-09-18T04:39:32.706 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/bin/umount 2024-09-18T04:39:32.706 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:39:32.706 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/bin/umount /tmp/tmp.WJMNJxIdAN/mnt 2024-09-18T04:39:32.715 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:32.709+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:32.739 INFO:tasks.workunit.client.0.smithi115.stderr:+ unmap_device /dev/nbd0 19676 2024-09-18T04:39:32.739 INFO:tasks.workunit.client.0.smithi115.stderr:+ local args=/dev/nbd0 2024-09-18T04:39:32.739 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pid=19676 2024-09-18T04:39:32.739 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-18T04:39:32.739 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:39:32.740 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:39:32.741 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:39:32.742 INFO:tasks.workunit.client.0.smithi115.stderr:++ which rbd 2024-09-18T04:39:32.744 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/bin/rbd 2024-09-18T04:39:32.745 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:39:32.745 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-18T04:39:32.815 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:32.815 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:39:32.815 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:39:32.815 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:39:32.815 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.815 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:37 2.5 e51/28 2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:head v 51'1, mlcod=0'0)", 2024-09-18T04:39:32.816 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:39:32.816 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 50.518489219999999, 2024-09-18T04:39:32.816 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012279973, 2024-09-18T04:39:32.816 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:32.816 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:32.816 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:32.816 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.816 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:32.816 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:39:32.816 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:32.816 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.816 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.816 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:32.816 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:39:32.817 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:32.817 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.817 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.817 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:32.817 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292468+0000", 2024-09-18T04:39:32.817 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:32.817 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.817 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.817 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:32.817 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295837+0000", 2024-09-18T04:39:32.817 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0033691699999999999 2024-09-18T04:39:32.817 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.817 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.818 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:32.818 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295860+0000", 2024-09-18T04:39:32.818 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2756999999999999e-05 2024-09-18T04:39:32.818 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.818 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.818 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:32.818 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295872+0000", 2024-09-18T04:39:32.818 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2442e-05 2024-09-18T04:39:32.818 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.818 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.818 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:32.818 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295934+0000", 2024-09-18T04:39:32.818 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1187999999999998e-05 2024-09-18T04:39:32.819 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.819 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.819 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:32.819 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295967+0000", 2024-09-18T04:39:32.819 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3185999999999997e-05 2024-09-18T04:39:32.819 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.819 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.819 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:32.820 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304697+0000", 2024-09-18T04:39:32.820 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0087298059999999997 2024-09-18T04:39:32.820 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.820 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.820 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:32.820 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304749+0000", 2024-09-18T04:39:32.821 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1922000000000002e-05 2024-09-18T04:39:32.821 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:32.821 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:32.821 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:32.821 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.821 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.822 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:49 2.5 e51/28 2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head v 51'2, mlcod=0'0)", 2024-09-18T04:39:32.822 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:39:32.822 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 50.485162381999999, 2024-09-18T04:39:32.822 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018033041999999999, 2024-09-18T04:39:32.822 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:32.822 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:32.822 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:32.823 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.823 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:32.823 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:39:32.823 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:32.823 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.823 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.823 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:32.824 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:39:32.824 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:32.824 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.824 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.824 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:32.824 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325793+0000", 2024-09-18T04:39:32.824 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:32.824 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.825 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.825 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:32.825 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329038+0000", 2024-09-18T04:39:32.825 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0032444010000000001 2024-09-18T04:39:32.825 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.825 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.825 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:32.825 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329065+0000", 2024-09-18T04:39:32.825 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7500000000000001e-05 2024-09-18T04:39:32.825 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.825 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.825 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:32.825 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329077+0000", 2024-09-18T04:39:32.826 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1919999999999999e-05 2024-09-18T04:39:32.826 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.826 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.826 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:32.826 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329149+0000", 2024-09-18T04:39:32.826 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2349999999999997e-05 2024-09-18T04:39:32.826 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.826 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.826 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:32.826 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329192+0000", 2024-09-18T04:39:32.826 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2744000000000003e-05 2024-09-18T04:39:32.826 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.826 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.826 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:32.827 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343773+0000", 2024-09-18T04:39:32.827 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014581246000000001 2024-09-18T04:39:32.827 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.827 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.827 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:32.827 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343829+0000", 2024-09-18T04:39:32.827 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5143999999999999e-05 2024-09-18T04:39:32.827 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:32.827 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:32.827 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:32.827 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.827 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.827 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:47 2.4 e51/17 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 51'3, mlcod=51'1)", 2024-09-18T04:39:32.827 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:39:32.828 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 50.484582117000002, 2024-09-18T04:39:32.828 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018960186, 2024-09-18T04:39:32.828 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:32.828 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:32.828 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:32.828 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.828 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:32.828 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:39:32.828 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:32.828 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.828 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.829 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:32.829 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:39:32.829 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:32.829 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.829 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.829 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:32.830 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326374+0000", 2024-09-18T04:39:32.830 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:32.830 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.830 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.830 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:32.830 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330883+0000", 2024-09-18T04:39:32.830 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0045084740000000002 2024-09-18T04:39:32.831 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.831 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.831 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:32.831 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330901+0000", 2024-09-18T04:39:32.831 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7890999999999999e-05 2024-09-18T04:39:32.831 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.831 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.831 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:32.832 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330912+0000", 2024-09-18T04:39:32.832 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1041e-05 2024-09-18T04:39:32.832 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.832 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.832 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:32.832 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.331003+0000", 2024-09-18T04:39:32.832 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1323999999999995e-05 2024-09-18T04:39:32.833 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.833 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.833 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:32.833 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.331030+0000", 2024-09-18T04:39:32.833 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7351999999999998e-05 2024-09-18T04:39:32.833 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.833 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.833 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:32.833 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345287+0000", 2024-09-18T04:39:32.833 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014256332 2024-09-18T04:39:32.833 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.834 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.834 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:32.834 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345336+0000", 2024-09-18T04:39:32.834 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9356e-05 2024-09-18T04:39:32.834 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:32.834 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:32.834 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:32.834 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.834 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.834 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:58 2.5 e51/28 2:b227e6b1:::rbd_data.125fc09df6d1.000000000000000e:head v 51'3, mlcod=51'1)", 2024-09-18T04:39:32.834 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:39:32.834 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 50.44415549, 2024-09-18T04:39:32.834 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016513678, 2024-09-18T04:39:32.834 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:32.835 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:32.835 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:32.835 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.835 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:32.835 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:39:32.835 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:32.835 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.835 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.835 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:32.835 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:39:32.835 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:32.835 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.835 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.835 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:32.836 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366800+0000", 2024-09-18T04:39:32.836 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:32.836 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.836 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.836 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:32.836 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368644+0000", 2024-09-18T04:39:32.836 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001844399 2024-09-18T04:39:32.836 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.836 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.836 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:32.836 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368681+0000", 2024-09-18T04:39:32.837 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6526999999999998e-05 2024-09-18T04:39:32.837 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.837 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.837 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:32.837 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368698+0000", 2024-09-18T04:39:32.837 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7272999999999999e-05 2024-09-18T04:39:32.837 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.837 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.837 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:32.837 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368773+0000", 2024-09-18T04:39:32.837 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4862000000000003e-05 2024-09-18T04:39:32.837 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.837 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.837 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:32.838 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368801+0000", 2024-09-18T04:39:32.838 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7925000000000001e-05 2024-09-18T04:39:32.838 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.838 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.838 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:32.838 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383261+0000", 2024-09-18T04:39:32.838 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014460377999999999 2024-09-18T04:39:32.839 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.839 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.839 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:32.839 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383316+0000", 2024-09-18T04:39:32.839 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4623999999999997e-05 2024-09-18T04:39:32.839 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:32.839 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:32.839 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:32.839 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.839 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.839 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:56 2.4 e51/17 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 51'4, mlcod=51'2)", 2024-09-18T04:39:32.840 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:39:32.840 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 50.442151463999998, 2024-09-18T04:39:32.840 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015105221, 2024-09-18T04:39:32.840 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:32.840 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:32.840 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:32.840 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.840 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:32.840 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:39:32.840 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:32.840 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.840 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.840 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:32.841 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:39:32.841 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:32.841 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.841 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.841 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:32.841 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368805+0000", 2024-09-18T04:39:32.841 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:32.841 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.841 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.841 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:32.841 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371603+0000", 2024-09-18T04:39:32.841 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027982660000000002 2024-09-18T04:39:32.841 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.841 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.842 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:32.842 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371633+0000", 2024-09-18T04:39:32.842 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0073000000000002e-05 2024-09-18T04:39:32.842 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.842 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.842 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:32.842 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371652+0000", 2024-09-18T04:39:32.843 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8597000000000002e-05 2024-09-18T04:39:32.843 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.843 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.843 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:32.843 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371752+0000", 2024-09-18T04:39:32.843 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9772999999999995e-05 2024-09-18T04:39:32.844 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.844 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.844 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:32.844 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371793+0000", 2024-09-18T04:39:32.844 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1173e-05 2024-09-18T04:39:32.844 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.844 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.844 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:32.845 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383875+0000", 2024-09-18T04:39:32.845 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012081852000000001 2024-09-18T04:39:32.845 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.845 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.845 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:32.845 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383912+0000", 2024-09-18T04:39:32.845 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7045000000000002e-05 2024-09-18T04:39:32.845 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:32.845 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:32.845 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:32.845 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.845 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.846 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:36 2.5 e51/28 2:bd5b8dd8:ns::rbd_data.133e813ab5bc.0000000000000002:head v 51'4, mlcod=51'2)", 2024-09-18T04:39:32.846 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:39:32.846 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 50.273432473, 2024-09-18T04:39:32.846 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020758161000000001, 2024-09-18T04:39:32.846 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:32.846 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:32.846 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:32.846 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.846 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:32.846 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:39:32.846 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:32.846 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.846 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.846 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:32.847 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:39:32.847 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:32.847 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.847 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.847 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:32.847 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537523+0000", 2024-09-18T04:39:32.847 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:32.847 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.847 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.847 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:32.848 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541578+0000", 2024-09-18T04:39:32.848 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0040549710000000001 2024-09-18T04:39:32.848 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.848 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.848 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:32.848 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541608+0000", 2024-09-18T04:39:32.848 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9349e-05 2024-09-18T04:39:32.848 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.849 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.849 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:32.849 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541621+0000", 2024-09-18T04:39:32.849 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2736e-05 2024-09-18T04:39:32.849 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.850 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.850 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:32.850 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541715+0000", 2024-09-18T04:39:32.850 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4197999999999995e-05 2024-09-18T04:39:32.850 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.850 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.850 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:32.850 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541755+0000", 2024-09-18T04:39:32.850 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9818999999999999e-05 2024-09-18T04:39:32.850 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.850 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.850 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:32.850 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558217+0000", 2024-09-18T04:39:32.850 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016462234999999999 2024-09-18T04:39:32.851 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.851 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.851 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:32.851 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558284+0000", 2024-09-18T04:39:32.851 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6774999999999997e-05 2024-09-18T04:39:32.851 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:32.851 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:32.851 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:32.851 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.851 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.851 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:39 2.4 e51/17 2:304364d1:ns::rbd_data.133e813ab5bc.0000000000000004:head v 51'5, mlcod=51'3)", 2024-09-18T04:39:32.851 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:39:32.851 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 50.265293346, 2024-09-18T04:39:32.851 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016777558000000001, 2024-09-18T04:39:32.852 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:32.852 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:32.852 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:32.852 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.852 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:32.852 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:39:32.852 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:32.852 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.852 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.852 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:32.852 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:39:32.852 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:32.852 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.853 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.853 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:32.853 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545663+0000", 2024-09-18T04:39:32.853 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:32.853 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.853 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.853 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:32.853 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548286+0000", 2024-09-18T04:39:32.853 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026233390000000001 2024-09-18T04:39:32.853 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.853 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.853 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:32.853 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548305+0000", 2024-09-18T04:39:32.854 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8425000000000001e-05 2024-09-18T04:39:32.854 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.854 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.854 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:32.854 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548316+0000", 2024-09-18T04:39:32.854 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1163e-05 2024-09-18T04:39:32.854 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.854 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.854 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:32.854 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548379+0000", 2024-09-18T04:39:32.854 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2780000000000005e-05 2024-09-18T04:39:32.854 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.855 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.855 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:32.855 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548399+0000", 2024-09-18T04:39:32.855 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0080000000000001e-05 2024-09-18T04:39:32.855 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.855 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.855 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:32.855 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562383+0000", 2024-09-18T04:39:32.855 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013984671000000001 2024-09-18T04:39:32.855 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.855 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.855 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:32.855 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562442+0000", 2024-09-18T04:39:32.855 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8700999999999999e-05 2024-09-18T04:39:32.856 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:32.856 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:32.856 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:32.856 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.856 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.856 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:37 2.5 e51/28 2:bf39b8e4:ns::rbd_data.133e813ab5bc.0000000000000003:head v 51'5, mlcod=51'2)", 2024-09-18T04:39:32.856 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:39:32.856 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 50.262425077000003, 2024-09-18T04:39:32.856 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014264492, 2024-09-18T04:39:32.856 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:32.856 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:32.856 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:32.856 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.857 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:32.857 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:39:32.857 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:32.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.857 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.857 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:32.857 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:39:32.857 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:32.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.857 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.857 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:32.857 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548531+0000", 2024-09-18T04:39:32.857 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:32.858 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.858 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.858 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:32.858 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551500+0000", 2024-09-18T04:39:32.858 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0029688129999999998 2024-09-18T04:39:32.858 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.858 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.858 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:32.858 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551516+0000", 2024-09-18T04:39:32.858 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5994e-05 2024-09-18T04:39:32.858 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.858 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.858 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:32.858 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551526+0000", 2024-09-18T04:39:32.859 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9299999999999998e-06 2024-09-18T04:39:32.859 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.859 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.859 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:32.859 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551577+0000", 2024-09-18T04:39:32.859 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0871000000000001e-05 2024-09-18T04:39:32.859 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.859 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.859 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:32.859 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551602+0000", 2024-09-18T04:39:32.859 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4813000000000001e-05 2024-09-18T04:39:32.859 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.859 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.860 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:32.860 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562741+0000", 2024-09-18T04:39:32.860 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011139428999999999 2024-09-18T04:39:32.860 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.860 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.860 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:32.860 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562797+0000", 2024-09-18T04:39:32.860 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6329000000000002e-05 2024-09-18T04:39:32.860 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:32.860 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:32.860 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:32.860 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.860 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.860 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:49 2.4 e51/17 2:20f0b0c9:ns::rbd_data.133e813ab5bc.0000000000000009:head v 51'6, mlcod=51'4)", 2024-09-18T04:39:32.861 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:39:32.861 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 50.230990519000002, 2024-09-18T04:39:32.861 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013849046, 2024-09-18T04:39:32.861 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:32.861 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:32.861 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:32.861 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.861 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:32.861 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:39:32.861 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:32.861 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.861 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.861 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:32.862 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:39:32.862 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:32.862 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.862 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.862 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:32.862 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579966+0000", 2024-09-18T04:39:32.862 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:32.862 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.862 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.862 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:32.862 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582324+0000", 2024-09-18T04:39:32.862 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023578890000000002 2024-09-18T04:39:32.862 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.862 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.863 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:32.863 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582355+0000", 2024-09-18T04:39:32.863 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1485000000000001e-05 2024-09-18T04:39:32.863 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.863 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.863 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:32.863 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582371+0000", 2024-09-18T04:39:32.863 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6147999999999999e-05 2024-09-18T04:39:32.863 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.863 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.863 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:32.863 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582467+0000", 2024-09-18T04:39:32.863 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6059999999999998e-05 2024-09-18T04:39:32.863 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.864 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.864 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:32.864 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582505+0000", 2024-09-18T04:39:32.864 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7679000000000003e-05 2024-09-18T04:39:32.864 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.864 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.864 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:32.864 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593743+0000", 2024-09-18T04:39:32.864 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011238356 2024-09-18T04:39:32.864 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.864 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.864 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:32.864 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593816+0000", 2024-09-18T04:39:32.865 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2935000000000006e-05 2024-09-18T04:39:32.865 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:32.865 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:32.865 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:32.865 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.865 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.865 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:53 2.4 e51/17 2:39f7f174:ns::rbd_data.133e813ab5bc.000000000000000b:head v 51'7, mlcod=51'4)", 2024-09-18T04:39:32.865 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:39:32.865 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 50.216289363999998, 2024-09-18T04:39:32.865 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0079914159999999995, 2024-09-18T04:39:32.865 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:32.865 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:32.865 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:32.865 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.866 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:32.866 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:39:32.866 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:32.866 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.866 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.866 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:32.866 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:39:32.866 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:32.866 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.866 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.866 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:32.866 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594667+0000", 2024-09-18T04:39:32.866 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:32.866 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.867 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.867 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:32.867 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596446+0000", 2024-09-18T04:39:32.867 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001779033 2024-09-18T04:39:32.867 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.867 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.867 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:32.867 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596472+0000", 2024-09-18T04:39:32.867 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6120000000000001e-05 2024-09-18T04:39:32.867 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.867 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.867 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:32.867 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596500+0000", 2024-09-18T04:39:32.867 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7518e-05 2024-09-18T04:39:32.868 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.868 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.868 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:32.868 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596564+0000", 2024-09-18T04:39:32.868 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3880999999999994e-05 2024-09-18T04:39:32.868 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.868 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.868 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:32.868 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596596+0000", 2024-09-18T04:39:32.868 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2638999999999997e-05 2024-09-18T04:39:32.868 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.868 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.868 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:32.869 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602605+0000", 2024-09-18T04:39:32.869 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0060080699999999999 2024-09-18T04:39:32.869 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.869 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.869 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:32.869 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602660+0000", 2024-09-18T04:39:32.869 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5396000000000003e-05 2024-09-18T04:39:32.869 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:32.869 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:32.869 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:32.869 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.869 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.869 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:60 2.5 e51/28 2:a06f51e4:ns::rbd_data.133e813ab5bc.000000000000000f:head v 51'6, mlcod=51'4)", 2024-09-18T04:39:32.870 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:39:32.870 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 50.201472379000002, 2024-09-18T04:39:32.870 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0093954989999999999, 2024-09-18T04:39:32.870 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:32.870 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:32.870 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:32.870 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.870 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:32.870 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:39:32.870 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:32.870 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.870 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.870 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:32.870 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:39:32.871 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:32.871 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.871 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.871 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:32.871 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609483+0000", 2024-09-18T04:39:32.871 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:32.871 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.871 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.871 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:32.871 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611767+0000", 2024-09-18T04:39:32.871 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002283724 2024-09-18T04:39:32.871 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.871 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.871 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:32.872 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611791+0000", 2024-09-18T04:39:32.872 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3547e-05 2024-09-18T04:39:32.872 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.872 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.872 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:32.873 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611800+0000", 2024-09-18T04:39:32.873 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4979999999999995e-06 2024-09-18T04:39:32.873 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.873 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.873 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:32.873 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611864+0000", 2024-09-18T04:39:32.873 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3974999999999996e-05 2024-09-18T04:39:32.873 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.873 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.873 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:32.873 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611887+0000", 2024-09-18T04:39:32.873 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2633000000000002e-05 2024-09-18T04:39:32.873 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.873 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.874 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:32.874 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.618777+0000", 2024-09-18T04:39:32.874 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0068897979999999999 2024-09-18T04:39:32.874 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.874 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.874 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:32.874 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.618881+0000", 2024-09-18T04:39:32.874 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010437699999999999 2024-09-18T04:39:32.874 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:32.874 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:32.874 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:32.874 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.874 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.874 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:663 2.4 e57/17 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 57'13, mlcod=57'11)", 2024-09-18T04:39:32.875 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.355300+0000", 2024-09-18T04:39:32.875 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 41.455657444000003, 2024-09-18T04:39:32.875 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0049192760000000002, 2024-09-18T04:39:32.875 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:32.875 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:32.875 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:32.875 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.875 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:32.875 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355300+0000", 2024-09-18T04:39:32.875 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:32.875 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.875 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.875 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:32.876 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355300+0000", 2024-09-18T04:39:32.876 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:32.876 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.876 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.876 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:32.876 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355299+0000", 2024-09-18T04:39:32.876 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:32.876 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.876 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.876 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:32.876 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355876+0000", 2024-09-18T04:39:32.876 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00057730900000000003 2024-09-18T04:39:32.876 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.876 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.877 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:32.877 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355897+0000", 2024-09-18T04:39:32.877 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0477000000000001e-05 2024-09-18T04:39:32.877 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.877 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.877 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:32.877 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355905+0000", 2024-09-18T04:39:32.877 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5739999999999996e-06 2024-09-18T04:39:32.877 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.877 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.877 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:32.877 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355989+0000", 2024-09-18T04:39:32.877 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4002e-05 2024-09-18T04:39:32.877 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.878 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.878 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:32.878 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.356041+0000", 2024-09-18T04:39:32.878 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1347000000000002e-05 2024-09-18T04:39:32.878 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.878 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.878 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:32.878 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.360172+0000", 2024-09-18T04:39:32.878 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0041315090000000002 2024-09-18T04:39:32.878 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.878 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.878 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:32.878 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.360220+0000", 2024-09-18T04:39:32.879 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7584999999999999e-05 2024-09-18T04:39:32.879 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:32.879 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:32.879 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:32.879 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.879 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.879 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:679 2.5 e57/28 2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head v 57'20, mlcod=57'18)", 2024-09-18T04:39:32.879 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.445316+0000", 2024-09-18T04:39:32.879 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 41.365641967000002, 2024-09-18T04:39:32.879 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0047016760000000001, 2024-09-18T04:39:32.879 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:32.879 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:32.879 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:32.879 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.880 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:32.880 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445316+0000", 2024-09-18T04:39:32.880 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:32.880 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.880 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.880 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:32.880 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445316+0000", 2024-09-18T04:39:32.880 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:32.880 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.880 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.880 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:32.880 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445314+0000", 2024-09-18T04:39:32.880 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:32.880 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.881 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.881 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:32.881 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445947+0000", 2024-09-18T04:39:32.881 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00063298900000000003 2024-09-18T04:39:32.881 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.881 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.881 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:32.881 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445968+0000", 2024-09-18T04:39:32.881 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.054e-05 2024-09-18T04:39:32.881 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.881 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.881 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:32.881 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445981+0000", 2024-09-18T04:39:32.882 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2697000000000001e-05 2024-09-18T04:39:32.882 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.882 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.882 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:32.882 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.446102+0000", 2024-09-18T04:39:32.882 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012189300000000001 2024-09-18T04:39:32.882 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.882 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.882 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:32.882 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.446151+0000", 2024-09-18T04:39:32.882 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8279000000000003e-05 2024-09-18T04:39:32.882 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.882 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.882 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:32.883 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.449953+0000", 2024-09-18T04:39:32.883 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003801979 2024-09-18T04:39:32.883 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.883 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.883 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:32.883 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.450018+0000", 2024-09-18T04:39:32.883 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4873e-05 2024-09-18T04:39:32.883 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:32.883 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:32.883 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:32.883 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.883 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.883 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:699 2.4 e57/17 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 57'17, mlcod=57'15)", 2024-09-18T04:39:32.883 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.566073+0000", 2024-09-18T04:39:32.883 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 41.244884454000001, 2024-09-18T04:39:32.884 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055398380000000001, 2024-09-18T04:39:32.884 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:32.884 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:32.884 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:32.884 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.884 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:32.884 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566073+0000", 2024-09-18T04:39:32.884 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:32.884 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.884 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.884 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:32.884 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566073+0000", 2024-09-18T04:39:32.884 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:32.884 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.885 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.885 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:32.885 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566071+0000", 2024-09-18T04:39:32.885 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:32.885 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.885 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.885 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:32.885 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566637+0000", 2024-09-18T04:39:32.885 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00056516699999999999 2024-09-18T04:39:32.885 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.885 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.885 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:32.885 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566668+0000", 2024-09-18T04:39:32.885 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1544999999999997e-05 2024-09-18T04:39:32.886 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.886 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.886 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:32.886 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566685+0000", 2024-09-18T04:39:32.886 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6857000000000001e-05 2024-09-18T04:39:32.886 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.886 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.886 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:32.886 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566816+0000", 2024-09-18T04:39:32.886 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013092799999999999 2024-09-18T04:39:32.886 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.886 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.886 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:32.887 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566875+0000", 2024-09-18T04:39:32.887 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9583999999999997e-05 2024-09-18T04:39:32.887 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.887 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.887 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:32.887 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.571567+0000", 2024-09-18T04:39:32.887 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.004691682 2024-09-18T04:39:32.887 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.887 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.887 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:32.887 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.571613+0000", 2024-09-18T04:39:32.887 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6125000000000003e-05 2024-09-18T04:39:32.887 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:32.887 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:32.888 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:32.888 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.888 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.888 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:98 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'27, mlcod=60'25)", 2024-09-18T04:39:32.888 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.847978+0000", 2024-09-18T04:39:32.888 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 31.962979895, 2024-09-18T04:39:32.888 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0044617709999999998, 2024-09-18T04:39:32.888 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:32.888 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:32.888 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:32.888 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.888 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:32.888 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.847978+0000", 2024-09-18T04:39:32.889 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:32.889 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.889 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.889 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:32.889 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.847978+0000", 2024-09-18T04:39:32.889 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:32.889 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.889 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.889 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:32.889 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.847977+0000", 2024-09-18T04:39:32.889 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:32.889 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.889 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.889 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:32.890 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849038+0000", 2024-09-18T04:39:32.890 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0010614820000000001 2024-09-18T04:39:32.890 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.890 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.890 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:32.890 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849065+0000", 2024-09-18T04:39:32.890 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7127999999999998e-05 2024-09-18T04:39:32.890 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.890 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.890 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:32.890 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849082+0000", 2024-09-18T04:39:32.890 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7170999999999999e-05 2024-09-18T04:39:32.890 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.890 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.891 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:32.891 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849188+0000", 2024-09-18T04:39:32.891 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010605800000000001 2024-09-18T04:39:32.891 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.891 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.891 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:32.891 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849233+0000", 2024-09-18T04:39:32.891 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4579000000000001e-05 2024-09-18T04:39:32.891 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.891 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.891 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:32.891 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.852370+0000", 2024-09-18T04:39:32.891 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00313751 2024-09-18T04:39:32.891 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.892 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.892 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:32.892 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.852440+0000", 2024-09-18T04:39:32.892 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9238000000000004e-05 2024-09-18T04:39:32.892 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:32.892 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:32.892 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:32.892 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.892 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.892 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:100 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'29, mlcod=64'27)", 2024-09-18T04:39:32.892 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.858766+0000", 2024-09-18T04:39:32.892 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 31.952191566, 2024-09-18T04:39:32.892 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0061964840000000004, 2024-09-18T04:39:32.893 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:32.893 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:32.893 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:32.893 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.893 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:32.893 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.858766+0000", 2024-09-18T04:39:32.893 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:32.893 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.893 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.893 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:32.893 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.858766+0000", 2024-09-18T04:39:32.893 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:32.893 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.893 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.894 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:32.894 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.858765+0000", 2024-09-18T04:39:32.894 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:32.894 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.894 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.894 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:32.894 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861612+0000", 2024-09-18T04:39:32.894 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028473169999999998 2024-09-18T04:39:32.894 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.894 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.894 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:32.894 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861643+0000", 2024-09-18T04:39:32.894 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1356000000000003e-05 2024-09-18T04:39:32.894 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.895 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.895 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:32.895 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861656+0000", 2024-09-18T04:39:32.895 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2899e-05 2024-09-18T04:39:32.895 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.895 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.895 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:32.895 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861773+0000", 2024-09-18T04:39:32.895 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011637999999999999 2024-09-18T04:39:32.895 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.895 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.895 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:32.895 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861808+0000", 2024-09-18T04:39:32.895 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5002999999999997e-05 2024-09-18T04:39:32.896 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.896 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.896 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:32.896 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.864876+0000", 2024-09-18T04:39:32.896 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030680719999999998 2024-09-18T04:39:32.896 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.896 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.896 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:32.896 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.864963+0000", 2024-09-18T04:39:32.896 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7114000000000007e-05 2024-09-18T04:39:32.896 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:32.896 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:32.896 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:32.897 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.897 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.897 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:102 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'31, mlcod=64'29)", 2024-09-18T04:39:32.897 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.872098+0000", 2024-09-18T04:39:32.897 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 31.938860308999999, 2024-09-18T04:39:32.897 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0064676450000000002, 2024-09-18T04:39:32.897 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:32.897 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:32.897 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:32.897 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.897 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:32.897 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.872098+0000", 2024-09-18T04:39:32.897 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:32.898 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.898 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.898 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:32.898 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.872098+0000", 2024-09-18T04:39:32.898 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:32.898 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.898 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.898 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:32.898 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.872096+0000", 2024-09-18T04:39:32.898 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:32.898 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.898 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.898 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:32.898 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875176+0000", 2024-09-18T04:39:32.899 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030799809999999999 2024-09-18T04:39:32.899 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.899 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.899 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:32.899 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875204+0000", 2024-09-18T04:39:32.899 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8025999999999999e-05 2024-09-18T04:39:32.899 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.899 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.899 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:32.899 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875216+0000", 2024-09-18T04:39:32.899 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2072999999999999e-05 2024-09-18T04:39:32.899 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.899 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.899 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:32.900 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875307+0000", 2024-09-18T04:39:32.900 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0373999999999999e-05 2024-09-18T04:39:32.900 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.900 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.900 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:32.900 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875349+0000", 2024-09-18T04:39:32.900 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2299000000000003e-05 2024-09-18T04:39:32.900 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.900 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.900 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:32.900 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.878496+0000", 2024-09-18T04:39:32.900 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0031468960000000002 2024-09-18T04:39:32.900 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.900 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.901 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:32.901 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.878565+0000", 2024-09-18T04:39:32.901 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9465e-05 2024-09-18T04:39:32.901 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:32.901 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:32.901 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:32.901 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.901 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.901 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:104 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'33, mlcod=64'31)", 2024-09-18T04:39:32.901 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.885100+0000", 2024-09-18T04:39:32.901 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 31.925857774000001, 2024-09-18T04:39:32.901 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0051888159999999997, 2024-09-18T04:39:32.901 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:32.902 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:32.902 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:32.902 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.902 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:32.902 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.885100+0000", 2024-09-18T04:39:32.902 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:32.902 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.902 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.902 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:32.902 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.885100+0000", 2024-09-18T04:39:32.902 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:32.902 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.902 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.902 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:32.903 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.885098+0000", 2024-09-18T04:39:32.903 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:32.903 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.903 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.903 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:32.903 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886809+0000", 2024-09-18T04:39:32.903 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017110459999999999 2024-09-18T04:39:32.903 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.903 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.903 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:32.903 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886842+0000", 2024-09-18T04:39:32.903 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2638999999999997e-05 2024-09-18T04:39:32.903 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.903 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.904 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:32.904 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886855+0000", 2024-09-18T04:39:32.904 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3380000000000001e-05 2024-09-18T04:39:32.904 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.904 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.904 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:32.904 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886939+0000", 2024-09-18T04:39:32.904 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3733999999999995e-05 2024-09-18T04:39:32.904 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.904 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.904 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:32.904 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886981+0000", 2024-09-18T04:39:32.904 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1956000000000003e-05 2024-09-18T04:39:32.904 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.905 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.905 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:32.905 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.890201+0000", 2024-09-18T04:39:32.905 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003219927 2024-09-18T04:39:32.905 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.905 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.905 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:32.905 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.890289+0000", 2024-09-18T04:39:32.905 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7997999999999993e-05 2024-09-18T04:39:32.905 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:32.905 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:32.905 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:32.905 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.905 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.906 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:145 2.4 e64/62 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 64'37, mlcod=64'35)", 2024-09-18T04:39:32.906 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:01.099398+0000", 2024-09-18T04:39:32.906 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 31.711560286000001, 2024-09-18T04:39:32.906 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0050078459999999998, 2024-09-18T04:39:32.906 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:32.906 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:32.906 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:32.906 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.906 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:32.906 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.099398+0000", 2024-09-18T04:39:32.906 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:32.906 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.906 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.906 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:32.907 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.099398+0000", 2024-09-18T04:39:32.907 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:32.907 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.907 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.907 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:32.907 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.099396+0000", 2024-09-18T04:39:32.907 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:32.907 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.907 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.907 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:32.907 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100199+0000", 2024-09-18T04:39:32.907 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00080338000000000002 2024-09-18T04:39:32.907 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.907 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.908 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:32.908 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100230+0000", 2024-09-18T04:39:32.908 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0716000000000002e-05 2024-09-18T04:39:32.908 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.908 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.908 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:32.908 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100243+0000", 2024-09-18T04:39:32.908 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3359e-05 2024-09-18T04:39:32.908 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.908 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.908 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:32.908 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100337+0000", 2024-09-18T04:39:32.908 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4387000000000003e-05 2024-09-18T04:39:32.908 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.909 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.909 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:32.909 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100396+0000", 2024-09-18T04:39:32.909 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8164000000000001e-05 2024-09-18T04:39:32.909 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.909 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.909 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:32.909 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.104309+0000", 2024-09-18T04:39:32.909 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0039137800000000004 2024-09-18T04:39:32.909 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.909 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.909 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:32.909 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.104405+0000", 2024-09-18T04:39:32.909 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6039999999999995e-05 2024-09-18T04:39:32.910 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:32.910 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:32.910 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:32.910 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.910 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.910 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:147 2.4 e64/62 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 64'39, mlcod=64'37)", 2024-09-18T04:39:32.910 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:01.110658+0000", 2024-09-18T04:39:32.910 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 31.700300197000001, 2024-09-18T04:39:32.910 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0046316910000000003, 2024-09-18T04:39:32.910 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:32.910 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:32.910 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:32.910 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.911 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:32.911 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.110658+0000", 2024-09-18T04:39:32.911 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:32.911 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.911 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.911 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:32.911 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.110658+0000", 2024-09-18T04:39:32.911 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:32.911 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.911 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.911 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:32.911 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.110656+0000", 2024-09-18T04:39:32.911 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:32.911 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.912 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.912 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:32.912 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111444+0000", 2024-09-18T04:39:32.912 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00078762400000000001 2024-09-18T04:39:32.912 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.912 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.912 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:32.912 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111474+0000", 2024-09-18T04:39:32.912 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0085000000000001e-05 2024-09-18T04:39:32.912 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.912 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.912 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:32.912 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111486+0000", 2024-09-18T04:39:32.912 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2645e-05 2024-09-18T04:39:32.913 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.913 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.913 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:32.913 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111581+0000", 2024-09-18T04:39:32.913 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4432000000000007e-05 2024-09-18T04:39:32.913 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.913 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.913 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:32.913 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111631+0000", 2024-09-18T04:39:32.913 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9901000000000002e-05 2024-09-18T04:39:32.913 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.913 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.913 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:32.913 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.115209+0000", 2024-09-18T04:39:32.914 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003577916 2024-09-18T04:39:32.914 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:32.914 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:32.914 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:32.914 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.115289+0000", 2024-09-18T04:39:32.914 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0704999999999994e-05 2024-09-18T04:39:32.914 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:32.914 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:32.914 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:32.914 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:32.914 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:32.914 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:32.916 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:32.808+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:32.916 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:32.908+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:32.916 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd device --device-type nbd list 2024-09-18T04:39:32.916 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false grep '^19676\b' 2024-09-18T04:39:32.916 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep '^19676\b' 2024-09-18T04:39:32.917 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:39:32.920 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:39:32.921 INFO:tasks.workunit.client.0.smithi115.stderr:+ ps -C rbd-nbd 2024-09-18T04:39:32.921 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false grep '^ *19676\b' 2024-09-18T04:39:32.921 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep '^ *19676\b' 2024-09-18T04:39:32.927 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:39:32.928 INFO:tasks.workunit.client.0.smithi115.stderr:+ sleep 0.5 2024-09-18T04:39:33.016 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:33.012+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:33.025 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:33.025 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:33.025 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:39:33.025 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:33.035 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:39:33.118 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:33.112+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:33.136 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:33.136 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:33.136 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:39:33.137 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:39:33.137 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:33.146 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:39:33.218 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:33.216+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:33.248 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.2.5.2 2024-09-18T04:39:33.257 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:33.257 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:39:33.258 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:39:33.258 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:39:33.258 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.258 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:33 2.1 e51/24 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head v 51'4, mlcod=51'2)", 2024-09-18T04:39:33.258 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:39:33.258 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 50.973368696000001, 2024-09-18T04:39:33.258 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018702106, 2024-09-18T04:39:33.258 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:33.258 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:33.258 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:33.258 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.259 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:33.259 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:39:33.259 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:33.259 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.259 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.259 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:33.259 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:39:33.259 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:33.259 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.259 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.259 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:33.259 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279197+0000", 2024-09-18T04:39:33.260 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:33.260 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.260 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.260 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:33.260 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282682+0000", 2024-09-18T04:39:33.260 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0034848689999999998 2024-09-18T04:39:33.260 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.260 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.260 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:33.260 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282709+0000", 2024-09-18T04:39:33.260 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7599000000000001e-05 2024-09-18T04:39:33.260 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.261 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.261 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:33.261 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282720+0000", 2024-09-18T04:39:33.261 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.092e-05 2024-09-18T04:39:33.261 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.261 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.261 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:33.261 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282791+0000", 2024-09-18T04:39:33.261 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1268999999999998e-05 2024-09-18T04:39:33.261 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.261 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.262 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:33.262 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282843+0000", 2024-09-18T04:39:33.262 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1771999999999998e-05 2024-09-18T04:39:33.262 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.262 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.262 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:33.262 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.297830+0000", 2024-09-18T04:39:33.262 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014987228999999999 2024-09-18T04:39:33.262 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.262 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.262 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:33.262 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.297901+0000", 2024-09-18T04:39:33.263 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0209999999999994e-05 2024-09-18T04:39:33.263 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:33.263 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:33.263 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:33.263 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.263 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.263 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:38 2.0 e51/17 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head v 51'10, mlcod=51'8)", 2024-09-18T04:39:33.263 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:39:33.263 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 50.971950266, 2024-09-18T04:39:33.263 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.023984472, 2024-09-18T04:39:33.263 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:33.263 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:33.264 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:33.264 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.264 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:33.264 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:39:33.264 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:33.264 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.264 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.264 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:33.264 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:39:33.264 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:33.265 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.265 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.265 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:33.265 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280614+0000", 2024-09-18T04:39:33.265 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:33.265 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.265 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.265 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:33.265 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293094+0000", 2024-09-18T04:39:33.265 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012479571 2024-09-18T04:39:33.265 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.265 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.266 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:33.266 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293111+0000", 2024-09-18T04:39:33.266 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7354999999999999e-05 2024-09-18T04:39:33.266 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.266 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.266 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:33.266 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293121+0000", 2024-09-18T04:39:33.266 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6979999999999993e-06 2024-09-18T04:39:33.266 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.266 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.266 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:33.267 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293180+0000", 2024-09-18T04:39:33.267 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9333000000000003e-05 2024-09-18T04:39:33.267 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.267 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.267 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:33.267 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293203+0000", 2024-09-18T04:39:33.267 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2633000000000002e-05 2024-09-18T04:39:33.267 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.267 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.267 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:33.267 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304534+0000", 2024-09-18T04:39:33.268 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011330761 2024-09-18T04:39:33.268 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.268 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.268 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:33.268 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304601+0000", 2024-09-18T04:39:33.268 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7657000000000006e-05 2024-09-18T04:39:33.268 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:33.268 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:33.268 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:33.268 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.268 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.268 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:46 2.6 2:7ba25245:::rbd_data.125fc09df6d1.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:33.269 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:39:33.269 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 50.944407466000001, 2024-09-18T04:39:33.269 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.035604595000000003, 2024-09-18T04:39:33.269 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:33.269 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:33.269 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:33.269 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:33.269 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:33.269 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 46 2024-09-18T04:39:33.269 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.269 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:33.269 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.270 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:33.270 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:39:33.270 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:33.270 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.270 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.270 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:33.270 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:39:33.270 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:33.270 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.270 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.270 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:33.271 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308158+0000", 2024-09-18T04:39:33.271 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:33.271 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.271 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.271 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:33.271 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313191+0000", 2024-09-18T04:39:33.271 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0050323929999999996 2024-09-18T04:39:33.271 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.271 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.271 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:33.271 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313210+0000", 2024-09-18T04:39:33.271 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9673e-05 2024-09-18T04:39:33.272 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.272 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.272 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:33.272 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313221+0000", 2024-09-18T04:39:33.272 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0444e-05 2024-09-18T04:39:33.272 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.272 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.272 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:33.272 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313294+0000", 2024-09-18T04:39:33.272 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2650000000000004e-05 2024-09-18T04:39:33.272 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.273 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.273 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:33.273 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313493+0000", 2024-09-18T04:39:33.273 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019968300000000001 2024-09-18T04:39:33.273 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.273 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.273 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:33.273 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313666+0000", 2024-09-18T04:39:33.273 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00017248599999999999 2024-09-18T04:39:33.273 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.273 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.274 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:33.274 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.323175+0000", 2024-09-18T04:39:33.274 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0095088819999999998 2024-09-18T04:39:33.274 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.274 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.274 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:33.274 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343651+0000", 2024-09-18T04:39:33.274 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020475892999999998 2024-09-18T04:39:33.274 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.274 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.274 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:33.275 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343715+0000", 2024-09-18T04:39:33.275 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4443999999999995e-05 2024-09-18T04:39:33.275 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.275 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.275 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:33.275 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343764+0000", 2024-09-18T04:39:33.275 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9280000000000003e-05 2024-09-18T04:39:33.275 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:33.275 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:33.275 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:33.275 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.276 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.276 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:47 2.4 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:33.276 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:39:33.276 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 50.939325429, 2024-09-18T04:39:33.276 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.032570693999999997, 2024-09-18T04:39:33.276 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:33.276 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:33.276 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:33.276 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:33.276 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:33.276 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 47 2024-09-18T04:39:33.277 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.277 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:33.277 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.277 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:33.277 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:39:33.277 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:33.277 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.277 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.277 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:33.277 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:39:33.277 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:33.277 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.278 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.278 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:33.278 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313240+0000", 2024-09-18T04:39:33.278 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:33.278 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.278 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.278 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:33.278 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325825+0000", 2024-09-18T04:39:33.278 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012584425999999999 2024-09-18T04:39:33.278 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.278 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.279 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:33.279 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325848+0000", 2024-09-18T04:39:33.279 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3025999999999999e-05 2024-09-18T04:39:33.279 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.279 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.279 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:33.279 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325863+0000", 2024-09-18T04:39:33.279 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5152e-05 2024-09-18T04:39:33.279 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.279 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.279 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:33.280 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325938+0000", 2024-09-18T04:39:33.280 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4982999999999997e-05 2024-09-18T04:39:33.280 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.280 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.280 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:33.280 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326090+0000", 2024-09-18T04:39:33.280 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015207900000000001 2024-09-18T04:39:33.280 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.280 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.280 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:33.280 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326242+0000", 2024-09-18T04:39:33.281 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015163500000000001 2024-09-18T04:39:33.281 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.281 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.281 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:33.281 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343015+0000", 2024-09-18T04:39:33.281 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016773414 2024-09-18T04:39:33.281 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.281 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.281 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:33.281 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345664+0000", 2024-09-18T04:39:33.281 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026487870000000001 2024-09-18T04:39:33.282 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.282 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.282 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:33.282 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345739+0000", 2024-09-18T04:39:33.282 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5018999999999994e-05 2024-09-18T04:39:33.282 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.282 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.282 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:33.282 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345812+0000", 2024-09-18T04:39:33.282 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3547000000000006e-05 2024-09-18T04:39:33.282 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:33.282 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:33.282 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:33.282 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.283 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.283 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:48 2.6 2:61db7a3a:::rbd_data.125fc09df6d1.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:33.283 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:39:33.283 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 50.926668757000002, 2024-09-18T04:39:33.283 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.025064388, 2024-09-18T04:39:33.283 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:33.283 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:33.283 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:33.283 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:33.283 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:33.283 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 48 2024-09-18T04:39:33.283 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.283 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:33.283 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.284 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:33.284 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:39:33.284 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:33.284 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.284 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.284 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:33.284 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:39:33.284 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:33.284 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.284 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.284 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:33.284 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325897+0000", 2024-09-18T04:39:33.284 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:33.284 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.284 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.285 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:33.285 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333697+0000", 2024-09-18T04:39:33.285 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0078002649999999998 2024-09-18T04:39:33.285 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.285 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.285 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:33.285 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333716+0000", 2024-09-18T04:39:33.285 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8910999999999999e-05 2024-09-18T04:39:33.285 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.285 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.285 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:33.285 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333728+0000", 2024-09-18T04:39:33.285 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1936000000000001e-05 2024-09-18T04:39:33.285 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.286 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.286 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:33.286 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333792+0000", 2024-09-18T04:39:33.286 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4010999999999994e-05 2024-09-18T04:39:33.286 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.286 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.286 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:33.286 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.334082+0000", 2024-09-18T04:39:33.286 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00028955799999999998 2024-09-18T04:39:33.286 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.286 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.286 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:33.286 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.334326+0000", 2024-09-18T04:39:33.286 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00024399099999999999 2024-09-18T04:39:33.286 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.287 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.287 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:33.287 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350745+0000", 2024-09-18T04:39:33.287 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016419474999999999 2024-09-18T04:39:33.287 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.287 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.287 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:33.287 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350808+0000", 2024-09-18T04:39:33.287 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3244999999999995e-05 2024-09-18T04:39:33.287 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.287 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.287 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:33.287 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350892+0000", 2024-09-18T04:39:33.287 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4074999999999997e-05 2024-09-18T04:39:33.287 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.288 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.288 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:33.288 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350963+0000", 2024-09-18T04:39:33.288 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0412000000000003e-05 2024-09-18T04:39:33.288 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:33.288 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:33.288 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:33.288 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.288 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.288 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:51 2.6 2:775d7b11:::rbd_data.125fc09df6d1.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:33.288 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:39:33.288 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 50.918816620999998, 2024-09-18T04:39:33.288 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.023824221, 2024-09-18T04:39:33.288 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:33.289 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:33.289 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:33.289 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:33.289 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:33.289 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 51 2024-09-18T04:39:33.289 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.289 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:33.289 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.289 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:33.289 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:39:33.289 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:33.289 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.289 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.290 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:33.290 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:39:33.290 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:33.290 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.290 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.290 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:33.290 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333749+0000", 2024-09-18T04:39:33.290 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:33.290 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.290 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.290 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:33.290 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340899+0000", 2024-09-18T04:39:33.290 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0071500340000000004 2024-09-18T04:39:33.290 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.290 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.291 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:33.291 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340929+0000", 2024-09-18T04:39:33.291 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0093000000000001e-05 2024-09-18T04:39:33.291 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.291 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.291 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:33.291 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340943+0000", 2024-09-18T04:39:33.291 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4019000000000001e-05 2024-09-18T04:39:33.291 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.291 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.291 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:33.291 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343834+0000", 2024-09-18T04:39:33.291 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028918120000000001 2024-09-18T04:39:33.292 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.292 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.292 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:33.292 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344020+0000", 2024-09-18T04:39:33.292 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000185892 2024-09-18T04:39:33.292 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.292 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.292 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:33.292 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344190+0000", 2024-09-18T04:39:33.292 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016946800000000001 2024-09-18T04:39:33.292 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.292 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.292 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:33.292 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.351291+0000", 2024-09-18T04:39:33.293 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.007100805 2024-09-18T04:39:33.293 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.293 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.293 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:33.293 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357470+0000", 2024-09-18T04:39:33.293 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0061797570000000001 2024-09-18T04:39:33.293 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.293 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.293 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:33.293 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357533+0000", 2024-09-18T04:39:33.293 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2906999999999999e-05 2024-09-18T04:39:33.293 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.293 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.293 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:33.294 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357575+0000", 2024-09-18T04:39:33.294 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1403999999999999e-05 2024-09-18T04:39:33.294 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:33.294 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:33.294 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:33.294 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.294 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.294 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:56 2.4 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:33.294 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:39:33.294 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 50.902771528000002, 2024-09-18T04:39:33.294 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.034295870999999999, 2024-09-18T04:39:33.294 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:33.294 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:33.294 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:33.295 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:33.295 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:33.295 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 56 2024-09-18T04:39:33.295 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.295 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:33.295 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.295 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:33.295 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:39:33.295 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:33.295 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.295 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.295 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:33.295 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:39:33.295 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:33.296 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.296 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.296 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:33.296 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349794+0000", 2024-09-18T04:39:33.296 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:33.296 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.296 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.296 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:33.296 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366632+0000", 2024-09-18T04:39:33.296 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01683755 2024-09-18T04:39:33.296 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.296 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.296 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:33.296 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366669+0000", 2024-09-18T04:39:33.297 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7351000000000002e-05 2024-09-18T04:39:33.297 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.297 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.297 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:33.297 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366686+0000", 2024-09-18T04:39:33.297 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6507999999999998e-05 2024-09-18T04:39:33.297 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.297 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.297 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:33.297 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366962+0000", 2024-09-18T04:39:33.297 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00027655900000000001 2024-09-18T04:39:33.297 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.297 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.298 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:33.298 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.367452+0000", 2024-09-18T04:39:33.298 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00049025299999999996 2024-09-18T04:39:33.298 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.298 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.298 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:33.298 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368113+0000", 2024-09-18T04:39:33.298 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00066093399999999996 2024-09-18T04:39:33.298 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.298 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.298 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:33.298 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383719+0000", 2024-09-18T04:39:33.298 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015606106999999999 2024-09-18T04:39:33.298 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.299 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.299 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:33.299 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384035+0000", 2024-09-18T04:39:33.299 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00031590399999999999 2024-09-18T04:39:33.299 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.299 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.299 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:33.299 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384067+0000", 2024-09-18T04:39:33.299 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.129e-05 2024-09-18T04:39:33.299 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.299 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.299 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:33.299 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384091+0000", 2024-09-18T04:39:33.299 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4788000000000001e-05 2024-09-18T04:39:33.299 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:33.300 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:33.300 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:33.300 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.300 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.300 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:57 2.0 e51/17 2:0cdd070e:::rbd_data.125fc09df6d1.000000000000000d:head v 51'21, mlcod=51'19)", 2024-09-18T04:39:33.300 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:39:33.300 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 50.894944041000002, 2024-09-18T04:39:33.300 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.025216311000000002, 2024-09-18T04:39:33.300 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:33.300 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:33.300 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:33.300 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.300 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:33.300 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:39:33.300 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:33.301 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.301 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.301 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:33.301 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:39:33.301 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:33.301 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.301 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.301 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:33.301 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:39:33.301 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:33.301 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.301 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.301 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:33.302 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363844+0000", 2024-09-18T04:39:33.302 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0062218450000000002 2024-09-18T04:39:33.302 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.302 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.302 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:33.302 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363879+0000", 2024-09-18T04:39:33.302 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4382999999999999e-05 2024-09-18T04:39:33.302 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.302 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.302 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:33.302 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363893+0000", 2024-09-18T04:39:33.302 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4348999999999999e-05 2024-09-18T04:39:33.302 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.302 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.303 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:33.303 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363977+0000", 2024-09-18T04:39:33.303 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3499999999999997e-05 2024-09-18T04:39:33.303 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.303 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.303 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:33.303 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364009+0000", 2024-09-18T04:39:33.303 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2292000000000002e-05 2024-09-18T04:39:33.303 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.303 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.303 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:33.303 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.382781+0000", 2024-09-18T04:39:33.303 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018771626 2024-09-18T04:39:33.303 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.303 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.304 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:33.304 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.382839+0000", 2024-09-18T04:39:33.304 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8758000000000002e-05 2024-09-18T04:39:33.304 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:33.304 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:33.304 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:33.304 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.304 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.304 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:59 2.0 e51/17 2:19c52cde:::rbd_data.125fc09df6d1.000000000000000f:head v 51'22, mlcod=51'19)", 2024-09-18T04:39:33.304 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:39:33.304 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 50.888284114999998, 2024-09-18T04:39:33.304 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.019497112, 2024-09-18T04:39:33.304 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:33.304 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:33.305 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:33.305 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.305 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:33.305 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:39:33.305 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:33.305 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.305 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.305 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:33.305 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:39:33.305 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:33.305 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.305 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.305 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:33.306 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364282+0000", 2024-09-18T04:39:33.306 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:33.306 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.306 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.306 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:33.306 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371823+0000", 2024-09-18T04:39:33.306 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0075411829999999999 2024-09-18T04:39:33.306 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.306 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.306 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:33.307 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371837+0000", 2024-09-18T04:39:33.307 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4336e-05 2024-09-18T04:39:33.307 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.307 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.307 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:33.307 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371846+0000", 2024-09-18T04:39:33.307 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9770000000000006e-06 2024-09-18T04:39:33.307 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.307 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.307 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:33.307 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.373566+0000", 2024-09-18T04:39:33.307 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017192539999999999 2024-09-18T04:39:33.307 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.307 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.308 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:33.308 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.373595+0000", 2024-09-18T04:39:33.308 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8975999999999998e-05 2024-09-18T04:39:33.308 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.308 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.308 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:33.308 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383740+0000", 2024-09-18T04:39:33.308 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01014525 2024-09-18T04:39:33.308 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.308 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.308 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:33.308 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383780+0000", 2024-09-18T04:39:33.308 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0275999999999999e-05 2024-09-18T04:39:33.308 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:33.308 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:33.309 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:33.309 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.309 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.309 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:34 2.6 2:66289171:ns::rbd_data.133e813ab5bc.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:33.309 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.524015+0000", 2024-09-18T04:39:33.309 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 50.728552506, 2024-09-18T04:39:33.309 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027582111999999999, 2024-09-18T04:39:33.309 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:33.309 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:33.309 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:33.309 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:33.309 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:33.309 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 34 2024-09-18T04:39:33.310 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.310 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:33.310 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.310 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:33.310 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524015+0000", 2024-09-18T04:39:33.310 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:33.310 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.310 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.310 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:33.310 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524015+0000", 2024-09-18T04:39:33.310 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:33.310 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.310 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.310 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:33.310 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524012+0000", 2024-09-18T04:39:33.311 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:33.311 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.311 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.311 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:33.311 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534956+0000", 2024-09-18T04:39:33.311 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010943403000000001 2024-09-18T04:39:33.311 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.311 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.311 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:33.311 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534978+0000", 2024-09-18T04:39:33.311 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1965999999999999e-05 2024-09-18T04:39:33.311 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.311 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.311 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:33.312 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534990+0000", 2024-09-18T04:39:33.312 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2065999999999999e-05 2024-09-18T04:39:33.312 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.312 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.312 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:33.312 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535065+0000", 2024-09-18T04:39:33.312 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5523000000000002e-05 2024-09-18T04:39:33.312 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.312 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.312 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:33.312 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535208+0000", 2024-09-18T04:39:33.312 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014309500000000001 2024-09-18T04:39:33.312 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.312 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.313 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:33.313 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535343+0000", 2024-09-18T04:39:33.313 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013437400000000001 2024-09-18T04:39:33.313 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.313 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.313 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:33.313 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.546576+0000", 2024-09-18T04:39:33.313 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011233425999999999 2024-09-18T04:39:33.313 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.313 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.313 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:33.313 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551498+0000", 2024-09-18T04:39:33.313 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0049219349999999997 2024-09-18T04:39:33.313 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.313 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.314 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:33.314 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551554+0000", 2024-09-18T04:39:33.314 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6400999999999998e-05 2024-09-18T04:39:33.314 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.314 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.314 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:33.314 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551597+0000", 2024-09-18T04:39:33.314 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2283000000000002e-05 2024-09-18T04:39:33.314 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:33.314 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:33.314 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:33.314 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.314 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.314 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:33 2.1 e51/24 2:8b40f1c5:ns::rbd_data.133e813ab5bc.0000000000000000:head v 51'7, mlcod=51'5)", 2024-09-18T04:39:33.314 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.531767+0000", 2024-09-18T04:39:33.315 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 50.720800589, 2024-09-18T04:39:33.315 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015370295000000001, 2024-09-18T04:39:33.315 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:33.315 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:33.315 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:33.315 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.315 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:33.315 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531767+0000", 2024-09-18T04:39:33.315 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:33.315 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.315 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.315 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:33.315 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531767+0000", 2024-09-18T04:39:33.315 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:33.316 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.316 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.316 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:33.316 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531765+0000", 2024-09-18T04:39:33.316 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:33.316 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.316 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.316 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:33.316 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534258+0000", 2024-09-18T04:39:33.316 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002493561 2024-09-18T04:39:33.316 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.316 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.316 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:33.316 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534288+0000", 2024-09-18T04:39:33.317 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.976e-05 2024-09-18T04:39:33.317 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.317 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.317 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:33.317 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534300+0000", 2024-09-18T04:39:33.317 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1928e-05 2024-09-18T04:39:33.317 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.317 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.317 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:33.317 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534367+0000", 2024-09-18T04:39:33.317 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7045999999999995e-05 2024-09-18T04:39:33.317 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.317 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.317 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:33.318 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534394+0000", 2024-09-18T04:39:33.318 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6903999999999999e-05 2024-09-18T04:39:33.318 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.318 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.318 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:33.318 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.547073+0000", 2024-09-18T04:39:33.318 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012679365 2024-09-18T04:39:33.318 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.318 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.318 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:33.318 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.547137+0000", 2024-09-18T04:39:33.318 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3691999999999999e-05 2024-09-18T04:39:33.318 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:33.318 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:33.318 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:33.319 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.319 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.319 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:39 2.4 2:304364d1:ns::rbd_data.133e813ab5bc.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:33.319 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.535013+0000", 2024-09-18T04:39:33.319 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 50.717553654, 2024-09-18T04:39:33.319 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027764487000000001, 2024-09-18T04:39:33.319 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:33.319 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:33.319 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:33.319 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:33.319 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:33.319 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 39 2024-09-18T04:39:33.319 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.319 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:33.320 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.320 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:33.320 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535013+0000", 2024-09-18T04:39:33.320 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:33.320 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.320 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.320 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:33.320 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535013+0000", 2024-09-18T04:39:33.320 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:33.320 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.320 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.320 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:33.320 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535012+0000", 2024-09-18T04:39:33.320 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:33.321 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.321 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.321 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:33.321 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545016+0000", 2024-09-18T04:39:33.321 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01000322 2024-09-18T04:39:33.321 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.321 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.321 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:33.321 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545035+0000", 2024-09-18T04:39:33.321 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9845999999999999e-05 2024-09-18T04:39:33.321 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.321 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.321 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:33.321 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545045+0000", 2024-09-18T04:39:33.322 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3640000000000005e-06 2024-09-18T04:39:33.322 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.322 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.322 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:33.322 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545152+0000", 2024-09-18T04:39:33.322 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000107025 2024-09-18T04:39:33.322 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.322 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.322 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:33.322 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545338+0000", 2024-09-18T04:39:33.322 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018643500000000001 2024-09-18T04:39:33.322 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.322 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.322 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:33.323 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545492+0000", 2024-09-18T04:39:33.323 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015393199999999999 2024-09-18T04:39:33.323 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.323 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.323 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:33.323 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.553549+0000", 2024-09-18T04:39:33.323 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0080569170000000002 2024-09-18T04:39:33.323 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.323 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.323 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:33.323 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562693+0000", 2024-09-18T04:39:33.323 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0091441609999999996 2024-09-18T04:39:33.323 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.323 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.323 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:33.324 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562748+0000", 2024-09-18T04:39:33.324 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5047000000000003e-05 2024-09-18T04:39:33.324 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.324 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.324 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:33.324 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562778+0000", 2024-09-18T04:39:33.324 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9652e-05 2024-09-18T04:39:33.324 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:33.324 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:33.324 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:33.324 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.324 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.324 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:40 2.6 2:784d6cd1:ns::rbd_data.133e813ab5bc.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:33.324 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.545062+0000", 2024-09-18T04:39:33.325 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 50.707504602999997, 2024-09-18T04:39:33.325 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.032771623, 2024-09-18T04:39:33.325 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:33.325 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:33.325 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:33.325 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:33.325 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:33.325 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 40 2024-09-18T04:39:33.325 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.325 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:33.325 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.325 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:33.325 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545062+0000", 2024-09-18T04:39:33.325 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:33.326 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.326 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.326 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:33.326 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545062+0000", 2024-09-18T04:39:33.326 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:33.326 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.326 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.326 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:33.326 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545061+0000", 2024-09-18T04:39:33.326 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:33.326 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.326 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.326 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:33.326 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552477+0000", 2024-09-18T04:39:33.327 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.007415914 2024-09-18T04:39:33.327 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.327 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.327 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:33.327 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552500+0000", 2024-09-18T04:39:33.327 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3546000000000001e-05 2024-09-18T04:39:33.327 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.327 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.327 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:33.327 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552515+0000", 2024-09-18T04:39:33.328 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4606999999999999e-05 2024-09-18T04:39:33.328 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.328 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.328 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:33.328 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552661+0000", 2024-09-18T04:39:33.328 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000146007 2024-09-18T04:39:33.328 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.328 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.328 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:33.328 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552876+0000", 2024-09-18T04:39:33.329 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000214618 2024-09-18T04:39:33.329 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.329 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.329 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:33.329 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.553061+0000", 2024-09-18T04:39:33.329 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018564100000000001 2024-09-18T04:39:33.329 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.329 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.329 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:33.329 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.565612+0000", 2024-09-18T04:39:33.329 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012550301999999999 2024-09-18T04:39:33.329 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.329 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.329 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:33.330 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577695+0000", 2024-09-18T04:39:33.330 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012083493000000001 2024-09-18T04:39:33.330 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.330 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.330 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:33.330 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577772+0000", 2024-09-18T04:39:33.330 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7158999999999998e-05 2024-09-18T04:39:33.330 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.330 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.330 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:33.330 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577834+0000", 2024-09-18T04:39:33.330 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1851000000000001e-05 2024-09-18T04:39:33.330 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:33.330 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:33.331 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:33.331 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.331 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.331 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:43 2.6 2:759f27b6:ns::rbd_data.133e813ab5bc.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:33.331 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.555868+0000", 2024-09-18T04:39:33.331 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 50.696699109000001, 2024-09-18T04:39:33.331 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.022332497999999999, 2024-09-18T04:39:33.331 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:33.331 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:33.331 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:33.331 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:33.331 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:33.331 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 43 2024-09-18T04:39:33.331 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.332 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:33.332 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.332 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:33.332 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555868+0000", 2024-09-18T04:39:33.332 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:33.332 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.332 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.332 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:33.332 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555868+0000", 2024-09-18T04:39:33.332 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:33.332 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.332 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.332 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:33.332 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555865+0000", 2024-09-18T04:39:33.332 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999971 2024-09-18T04:39:33.333 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.333 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.333 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:33.333 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561031+0000", 2024-09-18T04:39:33.333 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0051658770000000001 2024-09-18T04:39:33.333 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.333 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.333 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:33.333 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561048+0000", 2024-09-18T04:39:33.333 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6569e-05 2024-09-18T04:39:33.333 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.333 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.333 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:33.333 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561055+0000", 2024-09-18T04:39:33.334 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.328e-06 2024-09-18T04:39:33.334 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.334 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.334 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:33.334 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561114+0000", 2024-09-18T04:39:33.334 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9540000000000003e-05 2024-09-18T04:39:33.334 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.334 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.334 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:33.334 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561314+0000", 2024-09-18T04:39:33.334 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019965799999999999 2024-09-18T04:39:33.334 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.334 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.334 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:33.335 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561473+0000", 2024-09-18T04:39:33.335 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000158904 2024-09-18T04:39:33.335 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.335 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.335 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:33.335 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.567015+0000", 2024-09-18T04:39:33.335 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055420859999999999 2024-09-18T04:39:33.335 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.335 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.335 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:33.335 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577982+0000", 2024-09-18T04:39:33.335 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010966872000000001 2024-09-18T04:39:33.335 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.335 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.335 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:33.336 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578162+0000", 2024-09-18T04:39:33.336 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018050999999999999 2024-09-18T04:39:33.336 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.336 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.336 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:33.336 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578200+0000", 2024-09-18T04:39:33.336 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8000999999999998e-05 2024-09-18T04:39:33.336 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:33.336 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:33.336 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:33.336 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.336 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.336 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:48 2.6 2:7c225365:ns::rbd_data.133e813ab5bc.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:33.337 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:39:33.337 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 50.688899886000002, 2024-09-18T04:39:33.337 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.024576377, 2024-09-18T04:39:33.337 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:33.337 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:33.337 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:33.337 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:33.337 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:33.337 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 48 2024-09-18T04:39:33.337 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.337 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:33.337 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.337 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:33.337 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:39:33.338 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:33.338 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.338 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.338 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:33.338 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:39:33.338 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:33.338 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.338 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.338 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:33.338 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.563666+0000", 2024-09-18T04:39:33.338 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:33.338 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.338 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.338 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:33.338 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570331+0000", 2024-09-18T04:39:33.339 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0066653329999999998 2024-09-18T04:39:33.339 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.339 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.339 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:33.339 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570348+0000", 2024-09-18T04:39:33.339 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6507e-05 2024-09-18T04:39:33.339 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.339 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.339 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:33.339 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570355+0000", 2024-09-18T04:39:33.339 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2749999999999998e-06 2024-09-18T04:39:33.339 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.339 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.339 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:33.340 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570396+0000", 2024-09-18T04:39:33.340 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0809000000000002e-05 2024-09-18T04:39:33.340 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.340 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.340 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:33.340 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570571+0000", 2024-09-18T04:39:33.340 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000174823 2024-09-18T04:39:33.340 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.340 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.340 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:33.340 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570722+0000", 2024-09-18T04:39:33.340 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015154599999999999 2024-09-18T04:39:33.340 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.340 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.340 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:33.341 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579756+0000", 2024-09-18T04:39:33.341 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0090337000000000004 2024-09-18T04:39:33.341 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.341 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.341 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:33.341 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588093+0000", 2024-09-18T04:39:33.341 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.008336553 2024-09-18T04:39:33.341 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.341 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.341 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:33.341 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588173+0000", 2024-09-18T04:39:33.341 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0333999999999994e-05 2024-09-18T04:39:33.341 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.341 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.342 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:33.342 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588244+0000", 2024-09-18T04:39:33.342 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0573000000000003e-05 2024-09-18T04:39:33.342 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:33.342 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:33.342 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:33.342 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.342 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.342 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:49 2.4 2:20f0b0c9:ns::rbd_data.133e813ab5bc.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:33.342 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.570370+0000", 2024-09-18T04:39:33.342 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 50.682197559000002, 2024-09-18T04:39:33.342 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.029633981, 2024-09-18T04:39:33.342 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:33.342 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:33.342 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:33.343 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:33.343 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:33.343 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 49 2024-09-18T04:39:33.343 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.343 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:33.343 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.343 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:33.343 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570370+0000", 2024-09-18T04:39:33.343 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:33.343 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.343 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.343 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:33.343 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570370+0000", 2024-09-18T04:39:33.343 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:33.344 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.344 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.344 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:33.344 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570369+0000", 2024-09-18T04:39:33.344 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:33.344 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.344 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.344 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:33.344 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579090+0000", 2024-09-18T04:39:33.344 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0087212339999999996 2024-09-18T04:39:33.344 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.344 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.344 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:33.344 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579109+0000", 2024-09-18T04:39:33.345 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9120999999999999e-05 2024-09-18T04:39:33.345 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.345 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.345 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:33.345 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579123+0000", 2024-09-18T04:39:33.345 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4382000000000001e-05 2024-09-18T04:39:33.345 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.345 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.345 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:33.345 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579221+0000", 2024-09-18T04:39:33.345 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.7503999999999993e-05 2024-09-18T04:39:33.345 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.345 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.345 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:33.345 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579553+0000", 2024-09-18T04:39:33.346 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00033254200000000002 2024-09-18T04:39:33.346 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.346 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.346 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:33.346 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579817+0000", 2024-09-18T04:39:33.346 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00026317099999999998 2024-09-18T04:39:33.346 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.346 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.346 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:33.346 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.599905+0000", 2024-09-18T04:39:33.346 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020088575000000001 2024-09-18T04:39:33.346 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.346 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.346 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:33.347 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.599923+0000", 2024-09-18T04:39:33.347 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7425999999999999e-05 2024-09-18T04:39:33.347 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.347 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.347 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:33.347 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.599973+0000", 2024-09-18T04:39:33.347 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0649999999999998e-05 2024-09-18T04:39:33.347 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.347 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.347 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:33.347 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.600004+0000", 2024-09-18T04:39:33.347 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0361999999999998e-05 2024-09-18T04:39:33.347 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:33.347 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:33.347 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:33.348 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.348 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.348 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:50 2.1 e51/24 2:8c706374:ns::rbd_data.133e813ab5bc.000000000000000a:head v 51'8, mlcod=51'6)", 2024-09-18T04:39:33.348 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.577029+0000", 2024-09-18T04:39:33.348 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 50.675538187999997, 2024-09-18T04:39:33.348 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.017362855, 2024-09-18T04:39:33.348 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:33.348 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:33.348 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:33.348 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.348 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:33.348 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577029+0000", 2024-09-18T04:39:33.348 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:33.348 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.349 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.349 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:33.349 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577029+0000", 2024-09-18T04:39:33.349 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:33.349 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.349 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.349 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:33.349 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577027+0000", 2024-09-18T04:39:33.349 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:33.349 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.349 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.349 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:33.349 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578880+0000", 2024-09-18T04:39:33.349 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0018532080000000001 2024-09-18T04:39:33.350 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.350 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.350 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:33.350 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578906+0000", 2024-09-18T04:39:33.350 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5573999999999999e-05 2024-09-18T04:39:33.350 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.350 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.350 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:33.350 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578920+0000", 2024-09-18T04:39:33.350 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4483e-05 2024-09-18T04:39:33.350 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.350 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.350 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:33.350 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578983+0000", 2024-09-18T04:39:33.350 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2248000000000004e-05 2024-09-18T04:39:33.351 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.351 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.351 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:33.351 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579016+0000", 2024-09-18T04:39:33.351 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3364000000000001e-05 2024-09-18T04:39:33.351 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.351 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.351 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:33.351 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594342+0000", 2024-09-18T04:39:33.351 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01532564 2024-09-18T04:39:33.351 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.351 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.351 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:33.351 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594392+0000", 2024-09-18T04:39:33.352 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0068999999999998e-05 2024-09-18T04:39:33.352 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:33.352 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:33.352 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:33.352 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.352 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.352 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:45 2.0 e51/17 2:0fbfa3dc:ns::rbd_data.133e813ab5bc.0000000000000007:head v 51'24, mlcod=51'22)", 2024-09-18T04:39:33.352 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.581892+0000", 2024-09-18T04:39:33.352 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 50.670675516000003, 2024-09-18T04:39:33.352 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015646006, 2024-09-18T04:39:33.352 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:33.352 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:33.352 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:33.352 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.352 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:33.353 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581892+0000", 2024-09-18T04:39:33.353 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:33.353 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.353 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.353 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:33.353 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581892+0000", 2024-09-18T04:39:33.353 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:33.353 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.353 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.353 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:33.353 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581890+0000", 2024-09-18T04:39:33.353 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:33.353 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.354 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.354 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:33.354 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586188+0000", 2024-09-18T04:39:33.354 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0042985769999999996 2024-09-18T04:39:33.354 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.354 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.354 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:33.354 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586210+0000", 2024-09-18T04:39:33.354 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1069000000000001e-05 2024-09-18T04:39:33.354 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.354 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.354 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:33.354 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586221+0000", 2024-09-18T04:39:33.354 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1287000000000001e-05 2024-09-18T04:39:33.354 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.355 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.355 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:33.355 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586313+0000", 2024-09-18T04:39:33.355 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1935999999999995e-05 2024-09-18T04:39:33.355 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.355 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.355 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:33.355 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586368+0000", 2024-09-18T04:39:33.355 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5578000000000002e-05 2024-09-18T04:39:33.355 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.356 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.356 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:33.356 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.597473+0000", 2024-09-18T04:39:33.356 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011105139999999999 2024-09-18T04:39:33.356 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.356 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.356 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:33.356 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.597538+0000", 2024-09-18T04:39:33.356 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4133000000000004e-05 2024-09-18T04:39:33.356 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:33.356 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:33.356 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:33.356 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.356 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.357 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:53 2.4 2:39f7f174:ns::rbd_data.133e813ab5bc.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:33.357 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.584263+0000", 2024-09-18T04:39:33.357 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 50.668303641999998, 2024-09-18T04:39:33.357 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018701180000000001, 2024-09-18T04:39:33.357 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:33.357 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:33.357 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:33.357 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:33.357 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:33.357 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 53 2024-09-18T04:39:33.357 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.357 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:33.357 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.357 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:33.358 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584263+0000", 2024-09-18T04:39:33.358 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:33.358 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.358 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.358 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:33.358 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584263+0000", 2024-09-18T04:39:33.358 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:33.358 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.358 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.358 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:33.358 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584262+0000", 2024-09-18T04:39:33.358 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:33.358 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.358 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.358 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:33.359 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.591617+0000", 2024-09-18T04:39:33.359 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0073550600000000001 2024-09-18T04:39:33.359 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.359 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.359 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:33.359 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.591637+0000", 2024-09-18T04:39:33.359 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0432999999999999e-05 2024-09-18T04:39:33.359 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.359 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.359 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:33.359 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.591646+0000", 2024-09-18T04:39:33.359 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7849999999999997e-06 2024-09-18T04:39:33.359 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.359 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.360 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:33.360 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593478+0000", 2024-09-18T04:39:33.360 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001831495 2024-09-18T04:39:33.360 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.360 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.360 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:33.360 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593661+0000", 2024-09-18T04:39:33.360 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018364999999999999 2024-09-18T04:39:33.360 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.360 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.361 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:33.361 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593834+0000", 2024-09-18T04:39:33.361 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000173065 2024-09-18T04:39:33.361 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.361 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.361 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:33.361 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.600670+0000", 2024-09-18T04:39:33.361 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0068354970000000003 2024-09-18T04:39:33.361 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.361 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.361 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:33.361 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602851+0000", 2024-09-18T04:39:33.361 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002181653 2024-09-18T04:39:33.361 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.362 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.362 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:33.362 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602930+0000", 2024-09-18T04:39:33.362 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8045000000000002e-05 2024-09-18T04:39:33.362 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.362 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.362 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:33.362 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602965+0000", 2024-09-18T04:39:33.362 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5102999999999999e-05 2024-09-18T04:39:33.362 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:33.362 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:33.362 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:33.362 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.362 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.363 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:56 2.6 2:72453c3e:ns::rbd_data.133e813ab5bc.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:33.363 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.595988+0000", 2024-09-18T04:39:33.363 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 50.656579239999999, 2024-09-18T04:39:33.363 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.030296172, 2024-09-18T04:39:33.363 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:33.363 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:33.363 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:33.363 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:33.363 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:33.363 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 56 2024-09-18T04:39:33.363 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.363 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:33.363 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.363 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:33.363 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.595988+0000", 2024-09-18T04:39:33.364 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:33.364 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.364 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.364 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:33.364 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.595988+0000", 2024-09-18T04:39:33.364 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:33.364 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.364 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.364 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:33.364 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.595986+0000", 2024-09-18T04:39:33.364 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:33.364 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.364 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.364 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:33.365 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605262+0000", 2024-09-18T04:39:33.365 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0092767240000000001 2024-09-18T04:39:33.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.365 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.365 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:33.365 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605282+0000", 2024-09-18T04:39:33.365 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9307e-05 2024-09-18T04:39:33.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.365 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.365 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:33.365 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605292+0000", 2024-09-18T04:39:33.365 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9510000000000001e-06 2024-09-18T04:39:33.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.365 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.366 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:33.366 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605369+0000", 2024-09-18T04:39:33.366 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7846e-05 2024-09-18T04:39:33.366 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.366 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.366 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:33.366 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605668+0000", 2024-09-18T04:39:33.366 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00029896099999999999 2024-09-18T04:39:33.366 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.366 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.366 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:33.366 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605906+0000", 2024-09-18T04:39:33.366 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000237758 2024-09-18T04:39:33.366 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.366 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.367 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:33.367 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.613532+0000", 2024-09-18T04:39:33.367 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0076260590000000001 2024-09-18T04:39:33.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.367 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.367 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:33.367 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.626116+0000", 2024-09-18T04:39:33.367 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012584092999999999 2024-09-18T04:39:33.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.367 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.367 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:33.367 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.626216+0000", 2024-09-18T04:39:33.367 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9630999999999995e-05 2024-09-18T04:39:33.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:33.368 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:33.368 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:33.368 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.626284+0000", 2024-09-18T04:39:33.368 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8034999999999995e-05 2024-09-18T04:39:33.368 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:33.368 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:33.368 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:33.368 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:33.368 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:33.368 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:33.370 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:39:33.370 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:33.364+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:33.429 INFO:tasks.workunit.client.0.smithi115.stderr:+ DEV= 2024-09-18T04:39:33.430 INFO:tasks.workunit.client.0.smithi115.stderr:+ cat /tmp/tmp.WJMNJxIdAN/rbd-nbd.log 2024-09-18T04:39:33.430 INFO:tasks.workunit.client.0.smithi115.stdout:2024-09-18T04:39:31.323+0000 7f05295ae3c0 0 ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable), process rbd-nbd, pid 19667 2024-09-18T04:39:33.431 INFO:tasks.workunit.client.0.smithi115.stdout:2024-09-18T04:39:31.323+0000 7f05295ae3c0 0 pidfile_write: ignore empty --pid-file 2024-09-18T04:39:33.431 INFO:tasks.workunit.client.0.smithi115.stdout:2024-09-18T04:39:31.431+0000 7f05260e0700 0 -- 172.21.15.115:0/1707046328 >> [v2:172.21.15.82:6808/2746051132,v1:172.21.15.82:6809/2746051132] conn(0x7f050c011550 msgr2=0x7f050c013f50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-18T04:39:33.431 INFO:tasks.workunit.client.0.smithi115.stdout:2024-09-18T04:39:31.671+0000 7f05280e4700 0 -- 172.21.15.115:0/1707046328 >> [v2:172.21.15.28:6800/511141401,v1:172.21.15.28:6802/511141401] conn(0x7f050c0089a0 msgr2=0x7f050c00ae30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-18T04:39:33.431 INFO:tasks.workunit.client.0.smithi115.stdout:2024-09-18T04:39:32.823+0000 7f0507fff700 0 rbd-nbd: disconnect request received 2024-09-18T04:39:33.431 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false grep 'quiesce failed' /tmp/tmp.WJMNJxIdAN/rbd-nbd.log 2024-09-18T04:39:33.431 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep 'quiesce failed' /tmp/tmp.WJMNJxIdAN/rbd-nbd.log 2024-09-18T04:39:33.433 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:39:33.434 INFO:tasks.workunit.client.0.smithi115.stderr:++ _sudo rbd device --device-type nbd --options try-netlink,show-cookie map rbd/testrbdnbd17342 2024-09-18T04:39:33.434 INFO:tasks.workunit.client.0.smithi115.stderr:++ local cmd 2024-09-18T04:39:33.435 INFO:tasks.workunit.client.0.smithi115.stderr:+++ id -u 2024-09-18T04:39:33.436 INFO:tasks.workunit.client.0.smithi115.stderr:++ '[' 1000 -eq 0 ']' 2024-09-18T04:39:33.437 INFO:tasks.workunit.client.0.smithi115.stderr:+++ which rbd 2024-09-18T04:39:33.439 INFO:tasks.workunit.client.0.smithi115.stderr:++ cmd=/usr/bin/rbd 2024-09-18T04:39:33.440 INFO:tasks.workunit.client.0.smithi115.stderr:++ shift 2024-09-18T04:39:33.440 INFO:tasks.workunit.client.0.smithi115.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd --options try-netlink,show-cookie map rbd/testrbdnbd17342 2024-09-18T04:39:33.470 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:33.468+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:33.486 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:33.486 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:33.486 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:39:33.486 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:33.496 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:39:33.571 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:33.564+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:33.602 INFO:tasks.workunit.client.0.smithi115.stderr:+ OUT=/dev/nbd0 2024-09-18T04:39:33.603 INFO:tasks.workunit.client.0.smithi115.stderr:+ read DEV COOKIE 2024-09-18T04:39:33.603 INFO:tasks.workunit.client.0.smithi115.stderr:+ get_pid rbd 2024-09-18T04:39:33.603 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pool=rbd 2024-09-18T04:39:33.603 INFO:tasks.workunit.client.0.smithi115.stderr:+ local ns= 2024-09-18T04:39:33.604 INFO:tasks.workunit.client.0.smithi115.stderr:++ rbd device --device-type nbd --format xml list 2024-09-18T04:39:33.605 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:33.605 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:33.605 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:39:33.605 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:39:33.605 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:33.605 INFO:tasks.workunit.client.0.smithi115.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd17342'\''][device='\''/dev/nbd0'\'']/id' 2024-09-18T04:39:33.619 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:39:33.659 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-09-18T04:39:33.672 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:33.664+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:33.683 INFO:tasks.workunit.client.0.smithi115.stderr:+ PID=19794 2024-09-18T04:39:33.684 INFO:tasks.workunit.client.0.smithi115.stderr:+ test -n 19794 2024-09-18T04:39:33.684 INFO:tasks.workunit.client.0.smithi115.stderr:+ ps -p 19794 -C rbd-nbd 2024-09-18T04:39:33.689 INFO:tasks.workunit.client.0.smithi115.stdout: PID TTY TIME CMD 2024-09-18T04:39:33.690 INFO:tasks.workunit.client.0.smithi115.stdout: 19794 ? 00:00:00 rbd-nbd 2024-09-18T04:39:33.690 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo mount /dev/nbd0 /tmp/tmp.WJMNJxIdAN/mnt 2024-09-18T04:39:33.690 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:39:33.690 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:39:33.692 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:39:33.693 INFO:tasks.workunit.client.0.smithi115.stderr:++ which mount 2024-09-18T04:39:33.695 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/bin/mount 2024-09-18T04:39:33.695 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:39:33.695 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/bin/mount /dev/nbd0 /tmp/tmp.WJMNJxIdAN/mnt 2024-09-18T04:39:33.727 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo rbd device detach rbd/testrbdnbd17342 --device-type nbd 2024-09-18T04:39:33.727 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:39:33.727 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:39:33.729 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:39:33.730 INFO:tasks.workunit.client.0.smithi115.stderr:++ which rbd 2024-09-18T04:39:33.732 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:33.732 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:39:33.733 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:39:33.733 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:39:33.733 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.733 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:39:33.733 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:39:33.733 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 125.939079052, 2024-09-18T04:39:33.733 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017399218000000001, 2024-09-18T04:39:33.733 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:33.733 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:33.733 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:33.733 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:39:33.734 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:39:33.734 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 114 2024-09-18T04:39:33.734 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.734 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:33.734 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.734 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:33.734 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:39:33.734 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:33.734 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.734 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.734 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:33.734 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:39:33.735 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:33.735 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.735 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.735 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:33.735 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786900+0000", 2024-09-18T04:39:33.735 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:33.735 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.735 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.735 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:33.735 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786908+0000", 2024-09-18T04:39:33.735 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8199999999999997e-06 2024-09-18T04:39:33.735 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.735 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.736 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:33.736 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786925+0000", 2024-09-18T04:39:33.736 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6849000000000001e-05 2024-09-18T04:39:33.736 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.736 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.736 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:33.736 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786934+0000", 2024-09-18T04:39:33.736 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6299999999999993e-06 2024-09-18T04:39:33.736 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.736 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.736 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:33.736 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800926+0000", 2024-09-18T04:39:33.736 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013991632 2024-09-18T04:39:33.737 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.737 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.737 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:33.737 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801060+0000", 2024-09-18T04:39:33.737 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013378199999999999 2024-09-18T04:39:33.737 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.737 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.737 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:33.737 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801166+0000", 2024-09-18T04:39:33.737 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010607099999999999 2024-09-18T04:39:33.737 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.737 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.738 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:33.738 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803026+0000", 2024-09-18T04:39:33.738 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001860469 2024-09-18T04:39:33.738 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.738 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.738 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:33.738 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804245+0000", 2024-09-18T04:39:33.738 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0012185900000000001 2024-09-18T04:39:33.738 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.738 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.738 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:33.739 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804278+0000", 2024-09-18T04:39:33.739 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3272000000000003e-05 2024-09-18T04:39:33.739 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.739 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.739 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:33.739 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804300+0000", 2024-09-18T04:39:33.739 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2266e-05 2024-09-18T04:39:33.739 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:33.739 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:33.739 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:33.739 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.739 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.740 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:39:33.740 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:39:33.740 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 125.938166154, 2024-09-18T04:39:33.740 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017521662, 2024-09-18T04:39:33.740 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:33.740 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:33.740 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:33.740 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:39:33.740 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:39:33.740 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 116 2024-09-18T04:39:33.741 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.741 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:33.741 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.741 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:33.741 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:39:33.741 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:33.741 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.741 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.741 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:33.741 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:39:33.742 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:33.742 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.742 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.742 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:33.742 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787813+0000", 2024-09-18T04:39:33.742 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:33.742 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.742 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.742 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:33.742 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787824+0000", 2024-09-18T04:39:33.742 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1157e-05 2024-09-18T04:39:33.743 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.743 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.743 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:33.743 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787840+0000", 2024-09-18T04:39:33.743 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6269e-05 2024-09-18T04:39:33.743 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.743 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.743 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:33.743 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787853+0000", 2024-09-18T04:39:33.743 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3156000000000001e-05 2024-09-18T04:39:33.743 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.743 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.744 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:33.744 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803110+0000", 2024-09-18T04:39:33.744 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015256736 2024-09-18T04:39:33.744 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.744 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.744 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:33.744 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803197+0000", 2024-09-18T04:39:33.744 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7195999999999997e-05 2024-09-18T04:39:33.744 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.744 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.744 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:33.745 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803308+0000", 2024-09-18T04:39:33.745 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111005 2024-09-18T04:39:33.745 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.745 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.745 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:33.745 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804316+0000", 2024-09-18T04:39:33.745 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010073949999999999 2024-09-18T04:39:33.746 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.746 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.746 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:33.746 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805232+0000", 2024-09-18T04:39:33.746 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091636300000000001 2024-09-18T04:39:33.747 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.747 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.747 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:33.747 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805277+0000", 2024-09-18T04:39:33.747 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4864000000000003e-05 2024-09-18T04:39:33.747 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.747 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.747 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:33.748 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805336+0000", 2024-09-18T04:39:33.748 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8968000000000002e-05 2024-09-18T04:39:33.748 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:33.748 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:33.748 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:33.748 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.748 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.748 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:39:33.748 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:39:33.749 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 125.938114099, 2024-09-18T04:39:33.749 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017983453, 2024-09-18T04:39:33.749 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:33.749 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:33.749 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:33.749 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:39:33.750 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:39:33.750 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 117 2024-09-18T04:39:33.750 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.750 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:33.750 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.750 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:33.751 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:39:33.751 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:33.751 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.751 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.751 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:33.751 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:39:33.751 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:33.751 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.751 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.752 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:33.752 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787865+0000", 2024-09-18T04:39:33.752 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:33.752 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.752 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.752 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:33.752 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787889+0000", 2024-09-18T04:39:33.752 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4244999999999999e-05 2024-09-18T04:39:33.753 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.753 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.753 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:33.753 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787905+0000", 2024-09-18T04:39:33.753 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5739999999999998e-05 2024-09-18T04:39:33.754 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.754 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.754 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:33.754 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787918+0000", 2024-09-18T04:39:33.754 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.252e-05 2024-09-18T04:39:33.754 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.754 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.754 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:33.754 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803693+0000", 2024-09-18T04:39:33.754 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015774872999999998 2024-09-18T04:39:33.754 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.755 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.755 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:33.755 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803775+0000", 2024-09-18T04:39:33.755 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2761000000000003e-05 2024-09-18T04:39:33.755 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.755 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.755 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:33.755 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803887+0000", 2024-09-18T04:39:33.755 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111708 2024-09-18T04:39:33.755 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.755 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.755 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:33.755 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804823+0000", 2024-09-18T04:39:33.755 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00093625100000000003 2024-09-18T04:39:33.756 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.756 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.756 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:33.756 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805804+0000", 2024-09-18T04:39:33.756 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00098114699999999992 2024-09-18T04:39:33.756 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.756 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.756 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:33.756 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805831+0000", 2024-09-18T04:39:33.756 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6401999999999999e-05 2024-09-18T04:39:33.756 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.757 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.757 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:33.757 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805850+0000", 2024-09-18T04:39:33.757 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8862e-05 2024-09-18T04:39:33.757 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:33.757 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:33.757 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:33.757 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.757 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.757 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:36 2.7 2:fe1b02ad:::rbd_data.125fc09df6d1.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:33.757 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:39:33.757 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 51.460257988999999, 2024-09-18T04:39:33.757 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.040387804999999999, 2024-09-18T04:39:33.758 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:33.758 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:33.758 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:33.758 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:33.758 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:33.758 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 36 2024-09-18T04:39:33.758 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.758 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:33.758 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.758 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:33.758 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:39:33.758 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:33.758 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.759 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.759 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:33.759 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:39:33.759 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:33.759 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.759 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.759 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:33.759 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265720+0000", 2024-09-18T04:39:33.759 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:33.759 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.759 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.759 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:33.759 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272590+0000", 2024-09-18T04:39:33.759 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.006870508 2024-09-18T04:39:33.760 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.760 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.760 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:33.760 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272623+0000", 2024-09-18T04:39:33.760 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2716000000000003e-05 2024-09-18T04:39:33.760 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.760 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.760 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:33.760 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272637+0000", 2024-09-18T04:39:33.760 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4175999999999999e-05 2024-09-18T04:39:33.760 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.760 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.760 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:33.761 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272779+0000", 2024-09-18T04:39:33.761 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000141392 2024-09-18T04:39:33.761 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.761 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.761 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:33.761 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273031+0000", 2024-09-18T04:39:33.761 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00025249099999999998 2024-09-18T04:39:33.761 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.761 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.761 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:39:33.761 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273219+0000", 2024-09-18T04:39:33.762 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000187531 2024-09-18T04:39:33.762 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.762 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.762 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:33.762 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.302806+0000", 2024-09-18T04:39:33.762 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.029587553999999999 2024-09-18T04:39:33.762 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.762 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.762 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:33.762 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306004+0000", 2024-09-18T04:39:33.763 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0031982629999999998 2024-09-18T04:39:33.763 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.763 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.763 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:33.763 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306071+0000", 2024-09-18T04:39:33.763 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6829999999999995e-05 2024-09-18T04:39:33.764 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.764 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.764 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:33.764 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306110+0000", 2024-09-18T04:39:33.764 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8874000000000001e-05 2024-09-18T04:39:33.765 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:33.765 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:33.765 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:33.765 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.766 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.766 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:38 2.0 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:33.766 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:39:33.766 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 51.453304451999998, 2024-09-18T04:39:33.766 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.033754348000000003, 2024-09-18T04:39:33.766 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:33.766 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:33.766 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:33.766 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:33.766 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:33.766 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 38 2024-09-18T04:39:33.767 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.767 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:33.767 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.767 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:33.767 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:39:33.768 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:33.768 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.768 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.768 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:33.768 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:39:33.768 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:33.768 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.769 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.769 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:33.769 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272673+0000", 2024-09-18T04:39:33.769 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:33.769 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.769 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.769 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:33.769 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277166+0000", 2024-09-18T04:39:33.769 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044922249999999999 2024-09-18T04:39:33.770 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.770 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.770 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:33.770 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277189+0000", 2024-09-18T04:39:33.770 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3283000000000001e-05 2024-09-18T04:39:33.770 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.770 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.770 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:33.770 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277205+0000", 2024-09-18T04:39:33.770 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6305000000000001e-05 2024-09-18T04:39:33.770 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.770 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.771 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:33.771 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277290+0000", 2024-09-18T04:39:33.771 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5135000000000004e-05 2024-09-18T04:39:33.771 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.771 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.771 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:33.771 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277531+0000", 2024-09-18T04:39:33.771 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00024014699999999999 2024-09-18T04:39:33.771 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.772 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.772 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:33.772 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277793+0000", 2024-09-18T04:39:33.772 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00026224499999999998 2024-09-18T04:39:33.772 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.772 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.772 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:33.772 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303990+0000", 2024-09-18T04:39:33.772 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.026197295999999998 2024-09-18T04:39:33.772 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.773 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.773 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:33.773 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306340+0000", 2024-09-18T04:39:33.773 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.002349524 2024-09-18T04:39:33.773 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.773 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.773 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:33.773 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306394+0000", 2024-09-18T04:39:33.773 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3934999999999998e-05 2024-09-18T04:39:33.773 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.773 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.774 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:33.774 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306430+0000", 2024-09-18T04:39:33.774 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6690000000000003e-05 2024-09-18T04:39:33.774 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:33.774 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:33.774 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:33.774 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.774 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.774 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:39 2.7 2:e9c909c5:::rbd_data.125fc09df6d1.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:33.774 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:39:33.774 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 51.448744847, 2024-09-18T04:39:33.774 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.029110673, 2024-09-18T04:39:33.775 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:33.775 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:33.775 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:33.775 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:33.775 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:33.775 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 39 2024-09-18T04:39:33.775 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.775 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:33.776 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.776 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:33.776 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:39:33.776 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:33.776 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.776 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.776 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:33.776 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:39:33.777 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:33.777 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.777 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.777 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:33.778 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277234+0000", 2024-09-18T04:39:33.778 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:33.778 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.778 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.778 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:33.778 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281351+0000", 2024-09-18T04:39:33.778 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0041170920000000001 2024-09-18T04:39:33.778 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.778 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.778 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:33.779 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281375+0000", 2024-09-18T04:39:33.779 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.446e-05 2024-09-18T04:39:33.779 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.779 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.779 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:33.779 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281390+0000", 2024-09-18T04:39:33.779 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4309e-05 2024-09-18T04:39:33.779 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.779 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.779 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:33.780 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.282306+0000", 2024-09-18T04:39:33.780 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091662000000000004 2024-09-18T04:39:33.780 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.780 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.780 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:33.780 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.282613+0000", 2024-09-18T04:39:33.781 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00030663799999999997 2024-09-18T04:39:33.781 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.781 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.781 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:39:33.781 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283001+0000", 2024-09-18T04:39:33.781 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00038801599999999998 2024-09-18T04:39:33.782 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.782 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.782 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:33.782 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306239+0000", 2024-09-18T04:39:33.782 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023238466999999999 2024-09-18T04:39:33.782 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.782 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.782 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:33.783 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306259+0000", 2024-09-18T04:39:33.783 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9837999999999999e-05 2024-09-18T04:39:33.783 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.783 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.784 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:33.784 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306299+0000", 2024-09-18T04:39:33.784 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0194000000000002e-05 2024-09-18T04:39:33.784 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.784 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.784 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:33.784 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306346+0000", 2024-09-18T04:39:33.784 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6845000000000003e-05 2024-09-18T04:39:33.784 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:33.784 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:33.784 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:33.784 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.785 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.785 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:46 2.6 e51/17 2:7ba25245:::rbd_data.125fc09df6d1.0000000000000006:head v 51'2, mlcod=0'0)", 2024-09-18T04:39:33.785 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:39:33.785 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 51.412823715000002, 2024-09-18T04:39:33.785 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019843098, 2024-09-18T04:39:33.785 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:33.785 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:33.785 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:33.786 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.787 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:33.787 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:39:33.787 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:33.787 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.787 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.787 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:33.787 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:39:33.787 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:33.787 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.787 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.787 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:33.787 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313156+0000", 2024-09-18T04:39:33.788 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:33.788 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.788 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.789 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:33.789 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320144+0000", 2024-09-18T04:39:33.789 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.006988345 2024-09-18T04:39:33.789 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.789 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.789 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:33.789 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320175+0000", 2024-09-18T04:39:33.789 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.0861999999999997e-05 2024-09-18T04:39:33.789 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.789 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.789 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:33.789 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320188+0000", 2024-09-18T04:39:33.790 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2714999999999999e-05 2024-09-18T04:39:33.790 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.790 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.790 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:33.790 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320267+0000", 2024-09-18T04:39:33.790 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9586999999999995e-05 2024-09-18T04:39:33.790 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.790 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.790 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:33.790 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320311+0000", 2024-09-18T04:39:33.790 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3529000000000003e-05 2024-09-18T04:39:33.791 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.791 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.791 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:33.791 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.332949+0000", 2024-09-18T04:39:33.791 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012638267999999999 2024-09-18T04:39:33.791 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.791 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.791 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:33.791 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.333000+0000", 2024-09-18T04:39:33.792 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0736999999999999e-05 2024-09-18T04:39:33.792 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:33.792 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:33.792 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:33.792 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.793 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.793 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:57 2.0 2:0cdd070e:::rbd_data.125fc09df6d1.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:33.793 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:39:33.793 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 51.37696614, 2024-09-18T04:39:33.793 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.033329974999999998, 2024-09-18T04:39:33.793 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:33.793 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:33.793 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:33.793 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:33.794 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:33.794 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 57 2024-09-18T04:39:33.794 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.794 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:33.794 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.795 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:33.795 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:39:33.795 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:33.795 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.796 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.796 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:33.796 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:39:33.796 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:33.796 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.796 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.796 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:33.797 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349013+0000", 2024-09-18T04:39:33.797 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:33.797 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.797 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.797 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:33.798 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356138+0000", 2024-09-18T04:39:33.798 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071258010000000002 2024-09-18T04:39:33.798 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.798 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.798 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:33.798 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356168+0000", 2024-09-18T04:39:33.799 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9679000000000002e-05 2024-09-18T04:39:33.799 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.799 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.799 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:33.799 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356181+0000", 2024-09-18T04:39:33.799 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3164e-05 2024-09-18T04:39:33.799 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.799 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.799 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:33.799 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356254+0000", 2024-09-18T04:39:33.800 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3182999999999994e-05 2024-09-18T04:39:33.800 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.800 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.800 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:33.800 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356453+0000", 2024-09-18T04:39:33.800 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019876599999999999 2024-09-18T04:39:33.800 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.800 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.800 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:33.800 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356627+0000", 2024-09-18T04:39:33.800 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017340199999999999 2024-09-18T04:39:33.800 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.800 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.800 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:33.801 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.369200+0000", 2024-09-18T04:39:33.801 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012573302 2024-09-18T04:39:33.801 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.801 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.801 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:33.801 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382262+0000", 2024-09-18T04:39:33.801 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013062549 2024-09-18T04:39:33.801 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.801 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.801 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:33.801 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382310+0000", 2024-09-18T04:39:33.802 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7366e-05 2024-09-18T04:39:33.802 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.802 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.802 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:33.802 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382344+0000", 2024-09-18T04:39:33.802 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4319999999999997e-05 2024-09-18T04:39:33.802 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:33.802 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:33.802 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:33.802 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.802 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.803 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:59 2.0 2:19c52cde:::rbd_data.125fc09df6d1.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:33.803 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:39:33.803 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 51.369774974999999, 2024-09-18T04:39:33.803 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.027137247999999999, 2024-09-18T04:39:33.803 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:33.803 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:33.803 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:33.803 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:33.803 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:33.803 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 59 2024-09-18T04:39:33.803 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.804 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:33.804 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.804 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:33.804 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:39:33.805 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:33.805 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.805 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.805 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:33.806 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:39:33.806 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:33.806 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.806 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.806 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:33.806 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356204+0000", 2024-09-18T04:39:33.806 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:33.807 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.807 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.807 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:33.807 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362661+0000", 2024-09-18T04:39:33.807 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0064565129999999997 2024-09-18T04:39:33.807 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.807 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.807 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:33.807 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362684+0000", 2024-09-18T04:39:33.807 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3688000000000001e-05 2024-09-18T04:39:33.807 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.807 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.808 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:33.808 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362699+0000", 2024-09-18T04:39:33.808 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4798e-05 2024-09-18T04:39:33.808 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.808 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.808 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:33.809 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363016+0000", 2024-09-18T04:39:33.809 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031737300000000003 2024-09-18T04:39:33.809 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.809 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.809 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:33.809 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363201+0000", 2024-09-18T04:39:33.809 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000184131 2024-09-18T04:39:33.809 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.809 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.809 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:33.810 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363343+0000", 2024-09-18T04:39:33.810 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014236699999999999 2024-09-18T04:39:33.810 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.810 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.810 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:33.811 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.372189+0000", 2024-09-18T04:39:33.811 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0088464819999999993 2024-09-18T04:39:33.811 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.811 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.811 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:33.811 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383269+0000", 2024-09-18T04:39:33.811 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011079594 2024-09-18T04:39:33.811 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.811 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.811 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:33.811 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383314+0000", 2024-09-18T04:39:33.811 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4932000000000003e-05 2024-09-18T04:39:33.812 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.812 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.812 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:33.812 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383343+0000", 2024-09-18T04:39:33.812 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8600999999999999e-05 2024-09-18T04:39:33.812 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:33.812 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:33.812 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:33.812 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.813 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.813 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:40 2.6 e51/17 2:784d6cd1:ns::rbd_data.133e813ab5bc.0000000000000005:head v 51'7, mlcod=51'5)", 2024-09-18T04:39:33.813 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:39:33.813 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 51.173525241, 2024-09-18T04:39:33.813 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020497496, 2024-09-18T04:39:33.813 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:33.813 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:33.813 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:33.814 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.814 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:33.814 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:39:33.814 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:33.814 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.814 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.814 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:33.814 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:39:33.814 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:33.814 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.814 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.815 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:33.815 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552454+0000", 2024-09-18T04:39:33.815 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:33.815 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.815 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.815 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:33.815 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556448+0000", 2024-09-18T04:39:33.815 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0039940990000000001 2024-09-18T04:39:33.815 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.815 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.815 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:33.815 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556481+0000", 2024-09-18T04:39:33.815 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3778000000000001e-05 2024-09-18T04:39:33.816 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.816 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.816 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:33.816 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556493+0000", 2024-09-18T04:39:33.816 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1622e-05 2024-09-18T04:39:33.816 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.816 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.816 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:33.816 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556556+0000", 2024-09-18T04:39:33.816 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2582999999999994e-05 2024-09-18T04:39:33.816 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.816 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.816 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:33.816 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556592+0000", 2024-09-18T04:39:33.817 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5880999999999997e-05 2024-09-18T04:39:33.817 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.817 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.817 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:33.817 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.572786+0000", 2024-09-18T04:39:33.817 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016194571000000001 2024-09-18T04:39:33.817 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.817 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.817 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:33.818 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.572953+0000", 2024-09-18T04:39:33.823 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000166436 2024-09-18T04:39:33.823 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:33.823 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:33.823 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:33.824 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.824 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.824 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:45 2.0 2:0fbfa3dc:ns::rbd_data.133e813ab5bc.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:33.824 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:39:33.824 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 51.162310941999998, 2024-09-18T04:39:33.824 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.036558168000000002, 2024-09-18T04:39:33.824 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:33.825 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:33.825 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:33.825 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:33.825 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:33.825 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 45 2024-09-18T04:39:33.825 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.825 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:33.825 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.825 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:33.825 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:39:33.825 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:33.825 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.826 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.826 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:33.826 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:39:33.826 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:33.826 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.826 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.826 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:33.826 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:39:33.826 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:33.826 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.826 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.826 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:33.826 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578594+0000", 2024-09-18T04:39:33.826 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014927147999999999 2024-09-18T04:39:33.827 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.827 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.827 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:33.827 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578612+0000", 2024-09-18T04:39:33.827 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7623999999999999e-05 2024-09-18T04:39:33.827 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.827 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.827 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:33.827 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578620+0000", 2024-09-18T04:39:33.828 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9070000000000007e-06 2024-09-18T04:39:33.828 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.828 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.828 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:33.828 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578702+0000", 2024-09-18T04:39:33.829 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2349999999999996e-05 2024-09-18T04:39:33.829 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.829 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.829 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:33.829 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.579028+0000", 2024-09-18T04:39:33.829 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032622899999999998 2024-09-18T04:39:33.829 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.829 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.830 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:33.830 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.579343+0000", 2024-09-18T04:39:33.830 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031448899999999999 2024-09-18T04:39:33.830 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.831 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.831 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:33.831 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.590078+0000", 2024-09-18T04:39:33.831 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010734946 2024-09-18T04:39:33.831 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.831 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.831 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:33.831 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600089+0000", 2024-09-18T04:39:33.832 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010011195000000001 2024-09-18T04:39:33.832 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.832 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.832 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:33.832 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600177+0000", 2024-09-18T04:39:33.833 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8425999999999997e-05 2024-09-18T04:39:33.833 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.833 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.833 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:33.833 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600228+0000", 2024-09-18T04:39:33.833 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0216999999999997e-05 2024-09-18T04:39:33.833 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:33.833 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:33.834 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:33.834 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.834 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.834 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:54 2.0 2:04f17ef6:ns::rbd_data.133e813ab5bc.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:33.834 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:39:33.834 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 51.140739105000002, 2024-09-18T04:39:33.834 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.024612999, 2024-09-18T04:39:33.834 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:33.834 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:33.834 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:33.835 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:33.835 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:33.835 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 54 2024-09-18T04:39:33.835 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.835 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:33.835 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.835 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:33.835 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:39:33.835 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:33.835 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.835 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.836 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:33.836 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:39:33.836 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:33.836 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.836 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.837 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:33.837 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585239+0000", 2024-09-18T04:39:33.837 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:33.837 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.837 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.838 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:33.838 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593702+0000", 2024-09-18T04:39:33.838 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0084632990000000005 2024-09-18T04:39:33.838 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.838 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.839 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:33.839 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593719+0000", 2024-09-18T04:39:33.839 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.664e-05 2024-09-18T04:39:33.839 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.839 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.840 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:33.840 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593727+0000", 2024-09-18T04:39:33.840 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6839999999999996e-06 2024-09-18T04:39:33.840 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.840 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.840 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:33.841 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593767+0000", 2024-09-18T04:39:33.841 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0330999999999997e-05 2024-09-18T04:39:33.841 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.841 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.841 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:33.842 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593905+0000", 2024-09-18T04:39:33.842 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013754800000000001 2024-09-18T04:39:33.842 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.842 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.842 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:33.843 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.594013+0000", 2024-09-18T04:39:33.843 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000108805 2024-09-18T04:39:33.843 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.843 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.843 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:33.844 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604678+0000", 2024-09-18T04:39:33.844 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010664391 2024-09-18T04:39:33.844 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.844 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.845 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:33.845 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609662+0000", 2024-09-18T04:39:33.845 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0049846100000000004 2024-09-18T04:39:33.845 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.845 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.845 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:33.846 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609788+0000", 2024-09-18T04:39:33.846 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012518800000000001 2024-09-18T04:39:33.847 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.847 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.847 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:33.847 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609854+0000", 2024-09-18T04:39:33.847 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6556000000000004e-05 2024-09-18T04:39:33.847 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:33.847 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:33.848 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:33.848 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.848 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.848 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:58 2.7 2:fc7c6cd3:ns::rbd_data.133e813ab5bc.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:33.848 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:39:33.848 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 51.127896675999999, 2024-09-18T04:39:33.848 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.027690566999999999, 2024-09-18T04:39:33.848 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:33.848 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:33.848 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:33.848 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:33.848 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:33.848 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 58 2024-09-18T04:39:33.849 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.849 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:33.849 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.849 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:33.849 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:39:33.849 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:33.849 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.849 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.849 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:33.849 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:39:33.849 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:33.849 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.850 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.850 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:33.850 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598082+0000", 2024-09-18T04:39:33.850 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:33.850 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.851 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.851 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:33.851 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604832+0000", 2024-09-18T04:39:33.851 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0067504510000000002 2024-09-18T04:39:33.851 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.851 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.852 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:33.852 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604852+0000", 2024-09-18T04:39:33.852 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0228999999999999e-05 2024-09-18T04:39:33.852 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.852 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.853 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:33.853 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604870+0000", 2024-09-18T04:39:33.853 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7952e-05 2024-09-18T04:39:33.853 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.853 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.853 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:33.854 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604936+0000", 2024-09-18T04:39:33.854 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5431000000000004e-05 2024-09-18T04:39:33.854 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.854 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.854 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:33.855 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605085+0000", 2024-09-18T04:39:33.855 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014910300000000001 2024-09-18T04:39:33.855 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.855 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.855 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:39:33.855 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605205+0000", 2024-09-18T04:39:33.856 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000119887 2024-09-18T04:39:33.856 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.856 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.856 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:33.856 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625586+0000", 2024-09-18T04:39:33.857 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020381399000000001 2024-09-18T04:39:33.857 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.857 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.857 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:33.858 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625628+0000", 2024-09-18T04:39:33.858 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1873999999999999e-05 2024-09-18T04:39:33.858 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.858 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.858 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:33.859 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625722+0000", 2024-09-18T04:39:33.859 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3787000000000002e-05 2024-09-18T04:39:33.859 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.859 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.859 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:33.860 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625774+0000", 2024-09-18T04:39:33.860 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2503999999999997e-05 2024-09-18T04:39:33.860 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:33.860 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:33.861 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:33.861 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.861 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.861 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:56 2.6 e51/17 2:72453c3e:ns::rbd_data.133e813ab5bc.000000000000000d:head v 51'10, mlcod=51'8)", 2024-09-18T04:39:33.861 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:39:33.862 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 51.120699567999999, 2024-09-18T04:39:33.862 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019766552999999999, 2024-09-18T04:39:33.862 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:33.862 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:33.862 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:33.863 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.863 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:33.863 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:39:33.863 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:33.863 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.863 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.863 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:33.863 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:39:33.864 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:33.864 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.864 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.864 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:33.864 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605279+0000", 2024-09-18T04:39:33.864 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:33.864 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.864 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.864 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:33.864 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609417+0000", 2024-09-18T04:39:33.865 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0041378059999999999 2024-09-18T04:39:33.865 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.865 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.865 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:33.865 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609443+0000", 2024-09-18T04:39:33.865 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6098e-05 2024-09-18T04:39:33.865 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.865 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.865 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:33.865 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609458+0000", 2024-09-18T04:39:33.865 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4671000000000001e-05 2024-09-18T04:39:33.866 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.866 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.866 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:33.866 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609553+0000", 2024-09-18T04:39:33.866 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5265000000000004e-05 2024-09-18T04:39:33.866 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.866 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.866 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:33.866 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609583+0000", 2024-09-18T04:39:33.866 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9839e-05 2024-09-18T04:39:33.866 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.867 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.867 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:33.867 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625004+0000", 2024-09-18T04:39:33.867 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015420639999999999 2024-09-18T04:39:33.867 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.867 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.867 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:33.867 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625047+0000", 2024-09-18T04:39:33.867 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3665999999999998e-05 2024-09-18T04:39:33.867 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:33.867 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:33.867 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:33.867 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.868 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.868 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:99 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:33.868 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.582718+0000", 2024-09-18T04:39:33.868 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 37.143262659000001, 2024-09-18T04:39:33.868 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019362575, 2024-09-18T04:39:33.868 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:33.868 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:33.868 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:33.868 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:33.868 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:33.868 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 99 2024-09-18T04:39:33.868 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.868 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:33.868 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.869 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:33.869 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582718+0000", 2024-09-18T04:39:33.869 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:33.869 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.869 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.869 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:33.869 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582718+0000", 2024-09-18T04:39:33.869 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:33.869 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.869 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.869 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:33.869 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582717+0000", 2024-09-18T04:39:33.869 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:33.869 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.870 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.870 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:33.870 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582720+0000", 2024-09-18T04:39:33.870 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2239999999999998e-06 2024-09-18T04:39:33.870 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.870 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.870 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:33.870 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582723+0000", 2024-09-18T04:39:33.870 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.258e-06 2024-09-18T04:39:33.870 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.870 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.870 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:33.870 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582726+0000", 2024-09-18T04:39:33.871 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8870000000000001e-06 2024-09-18T04:39:33.871 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.871 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.871 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:33.871 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.592018+0000", 2024-09-18T04:39:33.871 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0092926420000000003 2024-09-18T04:39:33.871 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.871 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.871 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:33.871 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.592214+0000", 2024-09-18T04:39:33.871 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000195267 2024-09-18T04:39:33.872 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.872 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.872 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:33.872 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.592913+0000", 2024-09-18T04:39:33.872 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00069924600000000005 2024-09-18T04:39:33.872 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.872 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.872 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:33.872 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597493+0000", 2024-09-18T04:39:33.872 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045797629999999997 2024-09-18T04:39:33.872 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.873 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.873 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:33.873 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.602009+0000", 2024-09-18T04:39:33.873 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045159559999999998 2024-09-18T04:39:33.873 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.873 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.873 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:33.873 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.602042+0000", 2024-09-18T04:39:33.873 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3084999999999999e-05 2024-09-18T04:39:33.873 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.874 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.874 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:33.874 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.602080+0000", 2024-09-18T04:39:33.874 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8600000000000003e-05 2024-09-18T04:39:33.874 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:33.874 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:33.874 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:33.874 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.874 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.874 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:104 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:33.875 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.583165+0000", 2024-09-18T04:39:33.875 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 37.142815360999997, 2024-09-18T04:39:33.875 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023679603, 2024-09-18T04:39:33.875 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:33.875 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:33.875 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:33.875 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:33.875 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:33.875 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 104 2024-09-18T04:39:33.875 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.876 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:33.876 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.876 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:33.876 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583165+0000", 2024-09-18T04:39:33.876 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:33.876 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.876 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.876 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:33.876 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583165+0000", 2024-09-18T04:39:33.876 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:33.877 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.877 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.877 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:33.877 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583164+0000", 2024-09-18T04:39:33.877 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:33.877 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.877 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.877 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:33.877 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583167+0000", 2024-09-18T04:39:33.877 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9210000000000001e-06 2024-09-18T04:39:33.877 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.877 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.878 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:33.878 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583171+0000", 2024-09-18T04:39:33.878 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9759999999999997e-06 2024-09-18T04:39:33.878 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.878 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.878 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:33.878 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583174+0000", 2024-09-18T04:39:33.878 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9960000000000002e-06 2024-09-18T04:39:33.878 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.878 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.878 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:33.878 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.594373+0000", 2024-09-18T04:39:33.879 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011198665999999999 2024-09-18T04:39:33.879 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.879 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.879 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:33.879 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.594434+0000", 2024-09-18T04:39:33.879 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.1446999999999996e-05 2024-09-18T04:39:33.879 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.879 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.879 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:33.879 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.594692+0000", 2024-09-18T04:39:33.879 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00025756599999999998 2024-09-18T04:39:33.879 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.879 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.880 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:33.880 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599302+0000", 2024-09-18T04:39:33.880 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0046099640000000002 2024-09-18T04:39:33.880 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.880 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.880 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:33.880 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606710+0000", 2024-09-18T04:39:33.880 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0074077739999999998 2024-09-18T04:39:33.880 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.880 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.880 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:33.880 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606798+0000", 2024-09-18T04:39:33.880 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8133999999999993e-05 2024-09-18T04:39:33.881 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.881 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.881 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:33.881 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606845+0000", 2024-09-18T04:39:33.881 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6653000000000002e-05 2024-09-18T04:39:33.881 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:33.881 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:33.881 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:33.881 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.881 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.881 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:105 2.0 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:33.881 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.583193+0000", 2024-09-18T04:39:33.881 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 37.142787742000003, 2024-09-18T04:39:33.881 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023713649, 2024-09-18T04:39:33.882 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:33.882 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:33.882 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:33.882 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:33.882 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:33.882 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 105 2024-09-18T04:39:33.882 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.882 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:33.882 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.882 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:33.882 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583193+0000", 2024-09-18T04:39:33.883 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:33.883 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.883 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.883 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:33.883 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583193+0000", 2024-09-18T04:39:33.883 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:33.883 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.883 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.883 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:33.883 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583192+0000", 2024-09-18T04:39:33.883 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:33.883 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.883 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.884 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:33.884 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583194+0000", 2024-09-18T04:39:33.884 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.305e-06 2024-09-18T04:39:33.884 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.884 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.884 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:33.884 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583198+0000", 2024-09-18T04:39:33.884 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9369999999999997e-06 2024-09-18T04:39:33.884 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.884 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.884 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:33.884 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583202+0000", 2024-09-18T04:39:33.884 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6480000000000001e-06 2024-09-18T04:39:33.884 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.885 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.885 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:33.885 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.595304+0000", 2024-09-18T04:39:33.885 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012102063999999999 2024-09-18T04:39:33.885 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.885 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.885 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:33.885 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.595390+0000", 2024-09-18T04:39:33.885 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5493000000000003e-05 2024-09-18T04:39:33.885 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.885 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.885 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:33.885 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.595509+0000", 2024-09-18T04:39:33.886 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011966199999999999 2024-09-18T04:39:33.886 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.886 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.886 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:33.886 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599358+0000", 2024-09-18T04:39:33.886 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0038486689999999999 2024-09-18T04:39:33.886 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.886 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.886 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:33.886 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606859+0000", 2024-09-18T04:39:33.886 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0075010789999999999 2024-09-18T04:39:33.886 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.886 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.886 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:33.887 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606887+0000", 2024-09-18T04:39:33.887 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7773e-05 2024-09-18T04:39:33.887 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.887 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.887 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:33.887 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606906+0000", 2024-09-18T04:39:33.887 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9417000000000001e-05 2024-09-18T04:39:33.887 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:33.887 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:33.887 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:33.887 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.887 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.888 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:106 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:33.888 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.584425+0000", 2024-09-18T04:39:33.888 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 37.141555445999998, 2024-09-18T04:39:33.888 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.022529293999999998, 2024-09-18T04:39:33.888 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:33.888 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:33.888 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:33.888 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:33.888 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:33.888 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 106 2024-09-18T04:39:33.888 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.888 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:33.889 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.889 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:33.889 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584425+0000", 2024-09-18T04:39:33.889 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:33.889 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.889 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.889 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:33.889 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584425+0000", 2024-09-18T04:39:33.889 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:33.889 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.889 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.889 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:33.889 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584424+0000", 2024-09-18T04:39:33.890 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:33.890 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.890 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.890 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:33.890 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584430+0000", 2024-09-18T04:39:33.890 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6389999999999997e-06 2024-09-18T04:39:33.890 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.890 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.890 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:33.890 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584456+0000", 2024-09-18T04:39:33.890 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6188999999999999e-05 2024-09-18T04:39:33.891 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.891 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.891 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:33.891 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584463+0000", 2024-09-18T04:39:33.891 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7340000000000002e-06 2024-09-18T04:39:33.891 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.891 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.891 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:33.891 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597537+0000", 2024-09-18T04:39:33.891 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013074161000000001 2024-09-18T04:39:33.891 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.891 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.892 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:33.892 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597598+0000", 2024-09-18T04:39:33.892 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0930000000000001e-05 2024-09-18T04:39:33.892 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.892 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.892 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:33.892 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597922+0000", 2024-09-18T04:39:33.892 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032473000000000002 2024-09-18T04:39:33.892 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.892 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.893 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:33.893 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.601103+0000", 2024-09-18T04:39:33.893 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0031809849999999999 2024-09-18T04:39:33.893 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.893 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.893 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:33.893 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606912+0000", 2024-09-18T04:39:33.893 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058083550000000003 2024-09-18T04:39:33.893 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.893 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.893 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:33.894 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606935+0000", 2024-09-18T04:39:33.894 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2759999999999999e-05 2024-09-18T04:39:33.894 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.894 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.894 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:33.894 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606954+0000", 2024-09-18T04:39:33.894 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9602999999999998e-05 2024-09-18T04:39:33.894 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:33.894 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:33.894 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:33.895 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.895 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.895 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:109 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:33.895 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.585231+0000", 2024-09-18T04:39:33.895 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 37.140748825999999, 2024-09-18T04:39:33.895 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.021767496000000001, 2024-09-18T04:39:33.895 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:33.895 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:33.895 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:33.896 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:33.896 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:33.896 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 109 2024-09-18T04:39:33.896 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.896 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:33.896 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.896 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:33.896 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585231+0000", 2024-09-18T04:39:33.896 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:33.896 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.896 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.897 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:33.897 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585231+0000", 2024-09-18T04:39:33.897 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:33.897 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.897 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.897 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:33.897 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585230+0000", 2024-09-18T04:39:33.897 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:33.897 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.897 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.898 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:33.898 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585237+0000", 2024-09-18T04:39:33.898 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.618e-06 2024-09-18T04:39:33.898 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.898 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.898 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:33.898 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585256+0000", 2024-09-18T04:39:33.898 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9048000000000001e-05 2024-09-18T04:39:33.898 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.898 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.898 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:33.899 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585274+0000", 2024-09-18T04:39:33.899 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8153000000000001e-05 2024-09-18T04:39:33.899 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.899 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.899 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:33.899 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.598284+0000", 2024-09-18T04:39:33.899 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013009682999999999 2024-09-18T04:39:33.899 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.899 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.899 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:33.900 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.598383+0000", 2024-09-18T04:39:33.900 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.8722999999999997e-05 2024-09-18T04:39:33.900 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.900 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.900 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:33.900 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.598829+0000", 2024-09-18T04:39:33.900 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00044623999999999998 2024-09-18T04:39:33.900 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.900 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.900 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:33.901 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.601149+0000", 2024-09-18T04:39:33.901 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0023204150000000002 2024-09-18T04:39:33.901 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.901 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.901 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:33.901 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606960+0000", 2024-09-18T04:39:33.901 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058106390000000003 2024-09-18T04:39:33.901 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.901 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.901 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:33.901 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606982+0000", 2024-09-18T04:39:33.901 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2252e-05 2024-09-18T04:39:33.902 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.902 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.902 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:33.902 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606999+0000", 2024-09-18T04:39:33.902 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6871000000000002e-05 2024-09-18T04:39:33.902 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:33.902 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:33.902 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:33.902 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.902 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.903 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:110 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:33.903 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.586640+0000", 2024-09-18T04:39:33.903 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 37.139340507, 2024-09-18T04:39:33.903 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020400911000000001, 2024-09-18T04:39:33.903 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:33.903 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:33.903 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:33.903 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:33.903 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:33.903 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 110 2024-09-18T04:39:33.904 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.904 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:33.904 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.904 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:33.904 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586640+0000", 2024-09-18T04:39:33.904 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:33.904 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.904 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.904 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:33.904 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586640+0000", 2024-09-18T04:39:33.904 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:33.904 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.905 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.905 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:33.905 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586639+0000", 2024-09-18T04:39:33.905 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:33.905 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.905 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.905 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:33.905 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586652+0000", 2024-09-18T04:39:33.905 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2482e-05 2024-09-18T04:39:33.905 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.905 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.906 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:33.906 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586660+0000", 2024-09-18T04:39:33.906 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1659999999999994e-06 2024-09-18T04:39:33.906 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.906 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.906 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:33.906 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586666+0000", 2024-09-18T04:39:33.906 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6900000000000003e-06 2024-09-18T04:39:33.907 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.907 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.907 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:33.907 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599381+0000", 2024-09-18T04:39:33.907 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012714328 2024-09-18T04:39:33.907 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.907 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.907 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:33.907 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599436+0000", 2024-09-18T04:39:33.907 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4802999999999997e-05 2024-09-18T04:39:33.908 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.908 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.908 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:33.908 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599629+0000", 2024-09-18T04:39:33.908 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000193921 2024-09-18T04:39:33.908 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.908 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.908 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:33.908 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.601174+0000", 2024-09-18T04:39:33.908 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001544857 2024-09-18T04:39:33.908 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.908 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.909 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:33.909 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.607004+0000", 2024-09-18T04:39:33.909 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058300820000000003 2024-09-18T04:39:33.909 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.909 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.909 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:33.909 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.607024+0000", 2024-09-18T04:39:33.909 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9888e-05 2024-09-18T04:39:33.909 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:33.909 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:33.909 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:33.909 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.607041+0000", 2024-09-18T04:39:33.909 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6450999999999999e-05 2024-09-18T04:39:33.909 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:33.909 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:33.910 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:33.910 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:33.910 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:33.910 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:33.912 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:39:33.912 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/bin/rbd 2024-09-18T04:39:33.913 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:39:33.913 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/bin/rbd device detach rbd/testrbdnbd17342 --device-type nbd 2024-09-18T04:39:33.913 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false get_pid rbd 2024-09-18T04:39:33.913 INFO:tasks.workunit.client.0.smithi115.stderr:+ get_pid rbd 2024-09-18T04:39:33.913 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pool=rbd 2024-09-18T04:39:33.913 INFO:tasks.workunit.client.0.smithi115.stderr:+ local ns= 2024-09-18T04:39:33.913 INFO:tasks.workunit.client.0.smithi115.stderr:++ rbd device --device-type nbd --format xml list 2024-09-18T04:39:33.913 INFO:tasks.workunit.client.0.smithi115.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd17342'\''][device='\''/dev/nbd0'\'']/id' 2024-09-18T04:39:33.913 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:33.908+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:33.922 INFO:tasks.workunit.client.0.smithi115.stderr:+ PID= 2024-09-18T04:39:33.923 INFO:tasks.workunit.client.0.smithi115.stderr:+ test -n '' 2024-09-18T04:39:33.923 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 1 2024-09-18T04:39:33.923 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:39:33.923 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false _sudo rbd device attach --device /dev/nbd0 rbd/testrbdnbd17342 --device-type nbd 2024-09-18T04:39:33.923 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo rbd device attach --device /dev/nbd0 rbd/testrbdnbd17342 --device-type nbd 2024-09-18T04:39:33.923 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:39:33.923 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:39:33.924 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:39:33.925 INFO:tasks.workunit.client.0.smithi115.stderr:++ which rbd 2024-09-18T04:39:33.927 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/bin/rbd 2024-09-18T04:39:33.927 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:39:33.927 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/bin/rbd device attach --device /dev/nbd0 rbd/testrbdnbd17342 --device-type nbd 2024-09-18T04:39:33.957 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T04:39:33.969 INFO:tasks.workunit.client.0.smithi115.stderr:rbd: could not validate attach request 2024-09-18T04:39:33.969 INFO:tasks.workunit.client.0.smithi115.stderr:rbd: mismatching the image and the device may lead to data corruption 2024-09-18T04:39:33.969 INFO:tasks.workunit.client.0.smithi115.stderr:rbd: must specify --cookie or --force to proceed 2024-09-18T04:39:33.972 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:39:33.972 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T04:39:33.972 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo rbd device attach --device /dev/nbd0 rbd/testrbdnbd17342 --device-type nbd --force 2024-09-18T04:39:33.972 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:39:33.973 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:39:33.974 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:39:33.975 INFO:tasks.workunit.client.0.smithi115.stderr:++ which rbd 2024-09-18T04:39:33.976 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/bin/rbd 2024-09-18T04:39:33.976 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:39:33.976 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/bin/rbd device attach --device /dev/nbd0 rbd/testrbdnbd17342 --device-type nbd --force 2024-09-18T04:39:34.012 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:34.004+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:34.068 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:34.068 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:34.068 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:39:34.068 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:34.078 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:39:34.112 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:34.108+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:34.114 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:39:34.114 INFO:tasks.workunit.client.0.smithi115.stdout:/dev/nbd0 2024-09-18T04:39:34.124 INFO:tasks.workunit.client.0.smithi115.stderr:+ get_pid rbd 2024-09-18T04:39:34.124 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pool=rbd 2024-09-18T04:39:34.125 INFO:tasks.workunit.client.0.smithi115.stderr:+ local ns= 2024-09-18T04:39:34.125 INFO:tasks.workunit.client.0.smithi115.stderr:++ rbd device --device-type nbd --format xml list 2024-09-18T04:39:34.125 INFO:tasks.workunit.client.0.smithi115.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd17342'\''][device='\''/dev/nbd0'\'']/id' 2024-09-18T04:39:34.138 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:39:34.166 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:39:34.181 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:34.181 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:34.181 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:39:34.182 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:39:34.182 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:34.190 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:39:34.191 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:39:34.211 INFO:tasks.workunit.client.0.smithi115.stderr:+ PID=19865 2024-09-18T04:39:34.212 INFO:tasks.workunit.client.0.smithi115.stderr:+ test -n 19865 2024-09-18T04:39:34.212 INFO:tasks.workunit.client.0.smithi115.stderr:+ ps -p 19865 -C rbd-nbd 2024-09-18T04:39:34.212 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:39:34.213 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:39:34.213 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:39:34.213 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-18T04:39:34.213 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:39:34.213 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:39:34.213 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:39:34.214 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:34.204+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:34.215 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T04:39:34.216 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T04:39:34.220 INFO:tasks.workunit.client.0.smithi115.stdout: PID TTY TIME CMD 2024-09-18T04:39:34.220 INFO:tasks.workunit.client.0.smithi115.stdout: 19865 ? 00:00:00 rbd-nbd 2024-09-18T04:39:34.220 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo rbd device detach /dev/nbd0 --device-type nbd 2024-09-18T04:39:34.220 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:39:34.221 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:39:34.223 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:39:34.223 INFO:tasks.workunit.client.0.smithi115.stderr:++ which rbd 2024-09-18T04:39:34.230 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/bin/rbd 2024-09-18T04:39:34.230 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:39:34.230 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/bin/rbd device detach /dev/nbd0 --device-type nbd 2024-09-18T04:39:34.294 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:34.294 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:34.305 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:39:34.314 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:34.312+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:34.321 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false get_pid rbd 2024-09-18T04:39:34.321 INFO:tasks.workunit.client.0.smithi115.stderr:+ get_pid rbd 2024-09-18T04:39:34.321 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pool=rbd 2024-09-18T04:39:34.321 INFO:tasks.workunit.client.0.smithi115.stderr:+ local ns= 2024-09-18T04:39:34.321 INFO:tasks.workunit.client.0.smithi115.stderr:++ rbd device --device-type nbd --format xml list 2024-09-18T04:39:34.322 INFO:tasks.workunit.client.0.smithi115.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd17342'\''][device='\''/dev/nbd0'\'']/id' 2024-09-18T04:39:34.406 INFO:tasks.workunit.client.0.smithi115.stderr:+ PID= 2024-09-18T04:39:34.406 INFO:tasks.workunit.client.0.smithi115.stderr:+ test -n '' 2024-09-18T04:39:34.406 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 1 2024-09-18T04:39:34.406 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:39:34.406 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T04:39:34.407 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo rbd device attach --device /dev/nbd0 rbd/testrbdnbd17342 --device-type nbd --force 2024-09-18T04:39:34.407 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:39:34.407 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:39:34.407 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:39:34.407 INFO:tasks.workunit.client.0.smithi115.stderr:++ which rbd 2024-09-18T04:39:34.408 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:34.408 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:34.409 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/bin/rbd 2024-09-18T04:39:34.409 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:39:34.409 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/bin/rbd device attach --device /dev/nbd0 rbd/testrbdnbd17342 --device-type nbd --force 2024-09-18T04:39:34.415 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:34.408+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:34.418 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:39:34.422 INFO:teuthology.orchestra.run.smithi028.stdout:get_pg_num_history pg_num_history pg_num_history(e76 pg_nums {1={14=1},2={17=8},3={52=8}} deleted_pools ) 2024-09-18T04:39:34.422 INFO:teuthology.orchestra.run.smithi028.stdout:Importing pgid 2.5 2024-09-18T04:39:34.431 INFO:teuthology.orchestra.run.smithi028.stdout:Write #2:a06f51e4:ns::rbd_data.133e813ab5bc.000000000000000f:head# 2024-09-18T04:39:34.437 INFO:teuthology.orchestra.run.smithi028.stdout:snapset 0=[]:{} 2024-09-18T04:39:34.442 INFO:teuthology.orchestra.run.smithi028.stdout:Write #2:b227e6b1:::rbd_data.125fc09df6d1.000000000000000e:head# 2024-09-18T04:39:34.443 INFO:teuthology.orchestra.run.smithi028.stdout:snapset 0=[]:{} 2024-09-18T04:39:34.450 INFO:teuthology.orchestra.run.smithi028.stdout:Write #2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head# 2024-09-18T04:39:34.451 INFO:teuthology.orchestra.run.smithi028.stdout:snapset 0=[]:{} 2024-09-18T04:39:34.456 INFO:teuthology.orchestra.run.smithi028.stdout:Write #2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:head# 2024-09-18T04:39:34.457 INFO:teuthology.orchestra.run.smithi028.stdout:snapset 0=[]:{} 2024-09-18T04:39:34.464 INFO:teuthology.orchestra.run.smithi028.stdout:Write #2:bb27d472:::rbd_object_map.125fc09df6d1.0000000000000006:head# 2024-09-18T04:39:34.464 INFO:teuthology.orchestra.run.smithi028.stdout:snapset 0=[]:{} 2024-09-18T04:39:34.465 INFO:teuthology.orchestra.run.smithi028.stdout:Write #2:bd5b8dd8:ns::rbd_data.133e813ab5bc.0000000000000002:head# 2024-09-18T04:39:34.466 INFO:teuthology.orchestra.run.smithi028.stdout:snapset 0=[]:{} 2024-09-18T04:39:34.471 INFO:teuthology.orchestra.run.smithi028.stdout:Write #2:bf39b8e4:ns::rbd_data.133e813ab5bc.0000000000000003:head# 2024-09-18T04:39:34.472 INFO:teuthology.orchestra.run.smithi028.stdout:snapset 0=[]:{} 2024-09-18T04:39:34.478 INFO:teuthology.orchestra.run.smithi028.stdout:write_pg epoch 70 info 2.5( v 70'58 (0'0,70'58] local-lis/les=28/29 n=7 ec=17/17 lis/c=28/28 les/c/f=29/29/0 sis=28) 2024-09-18T04:39:34.479 INFO:teuthology.orchestra.run.smithi028.stdout:Import successful 2024-09-18T04:39:34.516 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:34.508+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:34.540 INFO:tasks.workunit.client.0.smithi115.stdout:/dev/nbd0 2024-09-18T04:39:34.553 INFO:tasks.workunit.client.0.smithi115.stderr:+ get_pid rbd 2024-09-18T04:39:34.553 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pool=rbd 2024-09-18T04:39:34.553 INFO:tasks.workunit.client.0.smithi115.stderr:+ local ns= 2024-09-18T04:39:34.554 INFO:tasks.workunit.client.0.smithi115.stderr:++ rbd device --device-type nbd --format xml list 2024-09-18T04:39:34.554 INFO:tasks.workunit.client.0.smithi115.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd17342'\''][device='\''/dev/nbd0'\'']/id' 2024-09-18T04:39:34.574 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:34.574 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:34.583 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:39:34.617 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:34.608+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:34.642 INFO:tasks.workunit.client.0.smithi115.stderr:+ PID=19923 2024-09-18T04:39:34.643 INFO:tasks.workunit.client.0.smithi115.stderr:+ test -n 19923 2024-09-18T04:39:34.643 INFO:tasks.workunit.client.0.smithi115.stderr:+ ps -p 19923 -C rbd-nbd 2024-09-18T04:39:34.652 INFO:tasks.workunit.client.0.smithi115.stdout: PID TTY TIME CMD 2024-09-18T04:39:34.652 INFO:tasks.workunit.client.0.smithi115.stdout: 19923 ? 00:00:00 rbd-nbd 2024-09-18T04:39:34.653 INFO:tasks.workunit.client.0.smithi115.stderr:+ ls /tmp/tmp.WJMNJxIdAN/mnt/ 2024-09-18T04:39:34.656 INFO:tasks.workunit.client.0.smithi115.stdout:lost+found 2024-09-18T04:39:34.656 INFO:tasks.workunit.client.0.smithi115.stdout:test 2024-09-18T04:39:34.656 INFO:tasks.workunit.client.0.smithi115.stderr:+ dd if=/tmp/tmp.WJMNJxIdAN/mnt/test of=/dev/null bs=1M count=1 2024-09-18T04:39:34.666 INFO:tasks.workunit.client.0.smithi115.stderr:1+0 records in 2024-09-18T04:39:34.666 INFO:tasks.workunit.client.0.smithi115.stderr:1+0 records out 2024-09-18T04:39:34.666 INFO:tasks.workunit.client.0.smithi115.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00903086 s, 116 MB/s 2024-09-18T04:39:34.666 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo dd if=/tmp/tmp.WJMNJxIdAN/data of=/tmp/tmp.WJMNJxIdAN/mnt/test1 bs=1M count=1 oflag=direct 2024-09-18T04:39:34.666 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:39:34.667 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:39:34.668 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:39:34.669 INFO:tasks.workunit.client.0.smithi115.stderr:++ which dd 2024-09-18T04:39:34.671 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/bin/dd 2024-09-18T04:39:34.671 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:39:34.671 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.WJMNJxIdAN/data of=/tmp/tmp.WJMNJxIdAN/mnt/test1 bs=1M count=1 oflag=direct 2024-09-18T04:39:34.687 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:34.687 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:34.698 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:39:34.702 INFO:tasks.workunit.client.0.smithi115.stderr:1+0 records in 2024-09-18T04:39:34.702 INFO:tasks.workunit.client.0.smithi115.stderr:1+0 records out 2024-09-18T04:39:34.702 INFO:tasks.workunit.client.0.smithi115.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220071 s, 47.6 MB/s 2024-09-18T04:39:34.704 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo umount /tmp/tmp.WJMNJxIdAN/mnt 2024-09-18T04:39:34.704 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:39:34.704 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:39:34.706 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:39:34.707 INFO:tasks.workunit.client.0.smithi115.stderr:++ which umount 2024-09-18T04:39:34.709 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/bin/umount 2024-09-18T04:39:34.709 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:39:34.709 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/bin/umount /tmp/tmp.WJMNJxIdAN/mnt 2024-09-18T04:39:34.728 INFO:tasks.workunit.client.0.smithi115.stderr:+ unmap_device /dev/nbd0 19923 2024-09-18T04:39:34.728 INFO:tasks.workunit.client.0.smithi115.stderr:+ local args=/dev/nbd0 2024-09-18T04:39:34.728 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pid=19923 2024-09-18T04:39:34.728 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-18T04:39:34.729 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:39:34.729 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:39:34.730 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:39:34.730 INFO:tasks.workunit.client.0.smithi115.stderr:++ which rbd 2024-09-18T04:39:34.733 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/bin/rbd 2024-09-18T04:39:34.733 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:39:34.733 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-18T04:39:34.742 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:34.736+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:34.818 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:34.818 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:34.819 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:34.816+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:34.829 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:39:34.839 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd device --device-type nbd list 2024-09-18T04:39:34.839 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false grep '^19923\b' 2024-09-18T04:39:34.840 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep '^19923\b' 2024-09-18T04:39:34.911 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:39:34.911 INFO:tasks.workunit.client.0.smithi115.stderr:+ ps -C rbd-nbd 2024-09-18T04:39:34.912 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false grep '^ *19923\b' 2024-09-18T04:39:34.912 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep '^ *19923\b' 2024-09-18T04:39:34.919 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:34.916+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:34.922 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:39:34.922 INFO:tasks.workunit.client.0.smithi115.stderr:+ sleep 0.5 2024-09-18T04:39:34.934 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:34.934 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:34.944 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:39:35.020 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:35.016+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:35.040 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f /var/log/ceph/exp.2.5.2 2024-09-18T04:39:35.097 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:35.097 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:35.107 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:39:35.121 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:35.112+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:35.221 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:35.216+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:35.260 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:35.261 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:35.271 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:39:35.322 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:35.316+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:35.377 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:35.377 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:35.387 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:39:35.423 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:35.420+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:35.424 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T04:39:35.424 INFO:tasks.workunit.client.0.smithi115.stderr:+ DEV= 2024-09-18T04:39:35.425 INFO:tasks.workunit.client.0.smithi115.stderr:++ rbd snap ls rbd/testrbdnbd17342 2024-09-18T04:39:35.425 INFO:tasks.workunit.client.0.smithi115.stderr:++ awk '$2 == "snap" {print $1}' 2024-09-18T04:39:35.497 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:35.498 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:35.502 INFO:tasks.workunit.client.0.smithi115.stderr:+ SNAPID=4 2024-09-18T04:39:35.503 INFO:tasks.workunit.client.0.smithi115.stderr:++ _sudo rbd device --device-type nbd --options try-netlink,show-cookie map --snap-id 4 rbd/testrbdnbd17342 2024-09-18T04:39:35.503 INFO:tasks.workunit.client.0.smithi115.stderr:++ local cmd 2024-09-18T04:39:35.503 INFO:tasks.workunit.client.0.smithi115.stderr:+++ id -u 2024-09-18T04:39:35.505 INFO:tasks.workunit.client.0.smithi115.stderr:++ '[' 1000 -eq 0 ']' 2024-09-18T04:39:35.506 INFO:tasks.workunit.client.0.smithi115.stderr:+++ which rbd 2024-09-18T04:39:35.508 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:39:35.508 INFO:tasks.workunit.client.0.smithi115.stderr:++ cmd=/usr/bin/rbd 2024-09-18T04:39:35.508 INFO:tasks.workunit.client.0.smithi115.stderr:++ shift 2024-09-18T04:39:35.508 INFO:tasks.workunit.client.0.smithi115.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd --options try-netlink,show-cookie map --snap-id 4 rbd/testrbdnbd17342 2024-09-18T04:39:35.554 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:35.548+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:35.624 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:35.616+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:35.625 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:35.625 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:35.636 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:39:35.674 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-18T04:39:35.681 INFO:tasks.workunit.client.0.smithi115.stderr:+ OUT=/dev/nbd0 2024-09-18T04:39:35.682 INFO:tasks.workunit.client.0.smithi115.stderr:+ read DEV COOKIE 2024-09-18T04:39:35.682 INFO:tasks.workunit.client.0.smithi115.stderr:+ get_pid rbd 2024-09-18T04:39:35.682 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pool=rbd 2024-09-18T04:39:35.682 INFO:tasks.workunit.client.0.smithi115.stderr:+ local ns= 2024-09-18T04:39:35.682 INFO:tasks.workunit.client.0.smithi115.stderr:++ rbd device --device-type nbd --format xml list 2024-09-18T04:39:35.683 INFO:tasks.workunit.client.0.smithi115.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd17342'\''][device='\''/dev/nbd0'\'']/id' 2024-09-18T04:39:35.725 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:35.716+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:35.747 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:35.747 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:35.757 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:39:35.765 INFO:tasks.workunit.client.0.smithi115.stderr:+ PID=20021 2024-09-18T04:39:35.765 INFO:tasks.workunit.client.0.smithi115.stderr:+ test -n 20021 2024-09-18T04:39:35.765 INFO:tasks.workunit.client.0.smithi115.stderr:+ ps -p 20021 -C rbd-nbd 2024-09-18T04:39:35.776 INFO:tasks.workunit.client.0.smithi115.stdout: PID TTY TIME CMD 2024-09-18T04:39:35.776 INFO:tasks.workunit.client.0.smithi115.stdout: 20021 ? 00:00:00 rbd-nbd 2024-09-18T04:39:35.777 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo rbd device detach rbd/testrbdnbd17342 --snap-id 4 --device-type nbd 2024-09-18T04:39:35.777 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:39:35.777 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:39:35.779 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:39:35.779 INFO:tasks.workunit.client.0.smithi115.stderr:++ which rbd 2024-09-18T04:39:35.781 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/bin/rbd 2024-09-18T04:39:35.781 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:39:35.781 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/bin/rbd device detach rbd/testrbdnbd17342 --snap-id 4 --device-type nbd 2024-09-18T04:39:35.826 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:35.824+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:35.876 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false get_pid rbd 2024-09-18T04:39:35.876 INFO:tasks.workunit.client.0.smithi115.stderr:+ get_pid rbd 2024-09-18T04:39:35.876 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pool=rbd 2024-09-18T04:39:35.877 INFO:tasks.workunit.client.0.smithi115.stderr:+ local ns= 2024-09-18T04:39:35.877 INFO:tasks.workunit.client.0.smithi115.stderr:++ rbd device --device-type nbd --format xml list 2024-09-18T04:39:35.877 INFO:tasks.workunit.client.0.smithi115.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd17342'\''][device='\''/dev/nbd0'\'']/id' 2024-09-18T04:39:35.904 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:35.904 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:35.914 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:39:35.927 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:35.920+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:35.957 INFO:tasks.workunit.client.0.smithi115.stderr:+ PID= 2024-09-18T04:39:35.958 INFO:tasks.workunit.client.0.smithi115.stderr:+ test -n '' 2024-09-18T04:39:35.958 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 1 2024-09-18T04:39:35.958 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:39:35.958 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false _sudo rbd device attach --device /dev/nbd0 --snap-id 4 rbd/testrbdnbd17342 --device-type nbd 2024-09-18T04:39:35.958 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo rbd device attach --device /dev/nbd0 --snap-id 4 rbd/testrbdnbd17342 --device-type nbd 2024-09-18T04:39:35.958 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:39:35.958 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:39:35.959 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:39:35.959 INFO:tasks.workunit.client.0.smithi115.stderr:++ which rbd 2024-09-18T04:39:35.962 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/bin/rbd 2024-09-18T04:39:35.962 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:39:35.962 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/bin/rbd device attach --device /dev/nbd0 --snap-id 4 rbd/testrbdnbd17342 --device-type nbd 2024-09-18T04:39:36.005 INFO:tasks.workunit.client.0.smithi115.stderr:rbd: could not validate attach request 2024-09-18T04:39:36.005 INFO:tasks.workunit.client.0.smithi115.stderr:rbd: mismatching the image and the device may lead to data corruption 2024-09-18T04:39:36.005 INFO:tasks.workunit.client.0.smithi115.stderr:rbd: must specify --cookie or --force to proceed 2024-09-18T04:39:36.008 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:39:36.008 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T04:39:36.008 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo rbd device attach --device /dev/nbd0 --snap-id 4 rbd/testrbdnbd17342 --device-type nbd --force 2024-09-18T04:39:36.008 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:39:36.009 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:39:36.010 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:39:36.011 INFO:tasks.workunit.client.0.smithi115.stderr:++ which rbd 2024-09-18T04:39:36.013 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/bin/rbd 2024-09-18T04:39:36.013 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:39:36.013 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/bin/rbd device attach --device /dev/nbd0 --snap-id 4 rbd/testrbdnbd17342 --device-type nbd --force 2024-09-18T04:39:36.022 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:36.022 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:36.027 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:36.024+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:36.031 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:39:36.128 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:36.120+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:36.169 INFO:tasks.workunit.client.0.smithi115.stdout:/dev/nbd0 2024-09-18T04:39:36.180 INFO:tasks.workunit.client.0.smithi115.stderr:+ get_pid rbd 2024-09-18T04:39:36.180 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pool=rbd 2024-09-18T04:39:36.180 INFO:tasks.workunit.client.0.smithi115.stderr:+ local ns= 2024-09-18T04:39:36.181 INFO:tasks.workunit.client.0.smithi115.stderr:++ rbd device --device-type nbd --format xml list 2024-09-18T04:39:36.181 INFO:tasks.workunit.client.0.smithi115.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd17342'\''][device='\''/dev/nbd0'\'']/id' 2024-09-18T04:39:36.184 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:36.184 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:36.195 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:39:36.217 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T04:39:36.246 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:36.240+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:36.260 INFO:tasks.workunit.client.0.smithi115.stderr:+ PID=20086 2024-09-18T04:39:36.260 INFO:tasks.workunit.client.0.smithi115.stderr:+ test -n 20086 2024-09-18T04:39:36.260 INFO:tasks.workunit.client.0.smithi115.stderr:+ ps -p 20086 -C rbd-nbd 2024-09-18T04:39:36.265 INFO:tasks.workunit.client.0.smithi115.stdout: PID TTY TIME CMD 2024-09-18T04:39:36.267 INFO:tasks.workunit.client.0.smithi115.stdout: 20086 ? 00:00:00 rbd-nbd 2024-09-18T04:39:36.267 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo rbd device detach /dev/nbd0 --device-type nbd 2024-09-18T04:39:36.267 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:39:36.267 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:39:36.268 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:39:36.268 INFO:tasks.workunit.client.0.smithi115.stderr:++ which rbd 2024-09-18T04:39:36.269 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/bin/rbd 2024-09-18T04:39:36.269 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:39:36.269 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/bin/rbd device detach /dev/nbd0 --device-type nbd 2024-09-18T04:39:36.297 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:36.297 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:36.306 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:39:36.330 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:36.324+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:36.364 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false get_pid rbd 2024-09-18T04:39:36.364 INFO:tasks.workunit.client.0.smithi115.stderr:+ get_pid rbd 2024-09-18T04:39:36.364 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pool=rbd 2024-09-18T04:39:36.364 INFO:tasks.workunit.client.0.smithi115.stderr:+ local ns= 2024-09-18T04:39:36.364 INFO:tasks.workunit.client.0.smithi115.stderr:++ rbd device --device-type nbd --format xml list 2024-09-18T04:39:36.365 INFO:tasks.workunit.client.0.smithi115.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd17342'\''][device='\''/dev/nbd0'\'']/id' 2024-09-18T04:39:36.370 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:39:36.393 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:39:36.414 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:36.414 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:36.416 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:39:36.424 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:39:36.431 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:36.428+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:36.439 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:39:36.446 INFO:tasks.workunit.client.0.smithi115.stderr:+ PID= 2024-09-18T04:39:36.446 INFO:tasks.workunit.client.0.smithi115.stderr:+ test -n '' 2024-09-18T04:39:36.446 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 1 2024-09-18T04:39:36.447 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:39:36.447 INFO:tasks.workunit.client.0.smithi115.stderr:+ DEV= 2024-09-18T04:39:36.447 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd config image set rbd/testrbdnbd17342 rbd_discard_granularity_bytes 4096 2024-09-18T04:39:36.460 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:39:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:39:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:39:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-18T04:39:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:39:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:39:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:39:36.464 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T04:39:36.464 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T04:39:36.532 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:36.524+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:36.536 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:36.537 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:36.537 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:39:36.537 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:39:36.537 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:36.547 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:39:36.549 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd feature enable rbd/testrbdnbd17342 journaling 2024-09-18T04:39:36.633 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:36.628+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:36.655 INFO:tasks.workunit.client.0.smithi115.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd17342 2024-09-18T04:39:36.656 INFO:tasks.workunit.client.0.smithi115.stderr:++ local cmd 2024-09-18T04:39:36.656 INFO:tasks.workunit.client.0.smithi115.stderr:+++ id -u 2024-09-18T04:39:36.658 INFO:tasks.workunit.client.0.smithi115.stderr:++ '[' 1000 -eq 0 ']' 2024-09-18T04:39:36.658 INFO:tasks.workunit.client.0.smithi115.stderr:+++ which rbd 2024-09-18T04:39:36.661 INFO:tasks.workunit.client.0.smithi115.stderr:++ cmd=/usr/bin/rbd 2024-09-18T04:39:36.661 INFO:tasks.workunit.client.0.smithi115.stderr:++ shift 2024-09-18T04:39:36.661 INFO:tasks.workunit.client.0.smithi115.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd17342 2024-09-18T04:39:36.672 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-09-18T04:39:36.705 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:36.705 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:39:36.705 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:39:36.705 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:39:36.705 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.705 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-18T04:39:36.705 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:39:36.705 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 122.229892514, 2024-09-18T04:39:36.706 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0095781189999999995, 2024-09-18T04:39:36.706 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:36.706 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:36.706 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:36.706 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:39:36.706 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:39:36.706 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 1 2024-09-18T04:39:36.706 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.706 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:36.706 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.706 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:36.706 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:39:36.706 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:36.706 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.707 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.707 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:36.707 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:39:36.708 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:36.708 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.708 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.708 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:36.708 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469862+0000", 2024-09-18T04:39:36.708 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:36.708 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.708 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.708 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:36.708 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469879+0000", 2024-09-18T04:39:36.708 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7431999999999998e-05 2024-09-18T04:39:36.708 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.709 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.709 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:36.709 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469896+0000", 2024-09-18T04:39:36.709 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6742999999999999e-05 2024-09-18T04:39:36.709 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.709 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.709 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:36.709 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469915+0000", 2024-09-18T04:39:36.709 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8991000000000002e-05 2024-09-18T04:39:36.709 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.709 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.709 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:36.709 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470397+0000", 2024-09-18T04:39:36.709 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048167799999999998 2024-09-18T04:39:36.710 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.710 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.710 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-09-18T04:39:36.710 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470412+0000", 2024-09-18T04:39:36.710 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5526e-05 2024-09-18T04:39:36.710 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.710 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.710 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:36.710 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473279+0000", 2024-09-18T04:39:36.710 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0028667469999999998 2024-09-18T04:39:36.710 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.710 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.710 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:36.711 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473441+0000", 2024-09-18T04:39:36.711 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000162082 2024-09-18T04:39:36.711 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.711 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.711 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:36.711 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473573+0000", 2024-09-18T04:39:36.711 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000132032 2024-09-18T04:39:36.711 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.711 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.711 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:36.711 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474826+0000", 2024-09-18T04:39:36.711 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001252717 2024-09-18T04:39:36.711 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.712 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.712 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:36.712 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479307+0000", 2024-09-18T04:39:36.712 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044813220000000003 2024-09-18T04:39:36.712 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.712 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.712 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:36.712 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479366+0000", 2024-09-18T04:39:36.712 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9024999999999998e-05 2024-09-18T04:39:36.712 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.712 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.712 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:36.712 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479443+0000", 2024-09-18T04:39:36.712 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6339000000000002e-05 2024-09-18T04:39:36.713 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:36.713 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:36.713 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:36.713 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.713 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.713 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-18T04:39:36.713 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:39:36.713 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 120.21688409399999, 2024-09-18T04:39:36.713 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082699560000000002, 2024-09-18T04:39:36.713 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:36.713 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:36.713 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:36.713 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:39:36.713 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:39:36.714 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 7 2024-09-18T04:39:36.714 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.714 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:36.714 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.714 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:36.714 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:39:36.714 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:36.714 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.714 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.714 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:36.714 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:39:36.714 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:36.714 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.714 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.714 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:36.715 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482870+0000", 2024-09-18T04:39:36.715 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999967 2024-09-18T04:39:36.715 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.715 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.715 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:36.715 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482889+0000", 2024-09-18T04:39:36.715 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.965e-05 2024-09-18T04:39:36.715 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.715 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.715 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:36.715 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482938+0000", 2024-09-18T04:39:36.715 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8548999999999999e-05 2024-09-18T04:39:36.715 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.715 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.716 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:36.716 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482963+0000", 2024-09-18T04:39:36.716 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5075999999999998e-05 2024-09-18T04:39:36.716 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.716 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.716 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:36.716 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483121+0000", 2024-09-18T04:39:36.716 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015819400000000001 2024-09-18T04:39:36.716 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.716 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.716 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op must wait for map", 2024-09-18T04:39:36.716 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483138+0000", 2024-09-18T04:39:36.716 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7068999999999999e-05 2024-09-18T04:39:36.716 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.717 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.717 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:36.717 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488666+0000", 2024-09-18T04:39:36.717 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0055282049999999996 2024-09-18T04:39:36.717 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.717 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.717 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:36.717 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488783+0000", 2024-09-18T04:39:36.717 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000116737 2024-09-18T04:39:36.717 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.717 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.717 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:36.717 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488991+0000", 2024-09-18T04:39:36.717 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020781400000000001 2024-09-18T04:39:36.718 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.718 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.718 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:36.718 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490554+0000", 2024-09-18T04:39:36.718 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0015628370000000001 2024-09-18T04:39:36.718 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.718 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.718 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:36.718 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491019+0000", 2024-09-18T04:39:36.718 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000465608 2024-09-18T04:39:36.718 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.718 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.718 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:36.718 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491087+0000", 2024-09-18T04:39:36.719 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7988e-05 2024-09-18T04:39:36.719 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.719 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.719 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:36.719 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491143+0000", 2024-09-18T04:39:36.719 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5538999999999998e-05 2024-09-18T04:39:36.719 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:36.719 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:36.719 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:36.720 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.720 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.720 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:34 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:36.720 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.261758+0000", 2024-09-18T04:39:36.720 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 54.437998942999997, 2024-09-18T04:39:36.720 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.042024191000000002, 2024-09-18T04:39:36.720 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:36.720 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:36.720 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:36.720 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:36.720 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:36.720 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 34 2024-09-18T04:39:36.721 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.721 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:36.721 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.721 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:36.721 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.261758+0000", 2024-09-18T04:39:36.721 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:36.721 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.721 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.721 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:36.721 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.261758+0000", 2024-09-18T04:39:36.721 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:36.721 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.721 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.722 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:36.722 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.261756+0000", 2024-09-18T04:39:36.722 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:36.722 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.722 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.722 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:36.722 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.270785+0000", 2024-09-18T04:39:36.722 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0090292340000000006 2024-09-18T04:39:36.722 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.722 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.722 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:36.722 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.270890+0000", 2024-09-18T04:39:36.722 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000104541 2024-09-18T04:39:36.722 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.723 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.723 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:36.723 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.270991+0000", 2024-09-18T04:39:36.723 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000100778 2024-09-18T04:39:36.723 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.723 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.723 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:36.723 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.271157+0000", 2024-09-18T04:39:36.723 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016653999999999999 2024-09-18T04:39:36.723 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.723 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.723 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:36.723 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.271591+0000", 2024-09-18T04:39:36.724 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00043409500000000001 2024-09-18T04:39:36.724 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.724 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.724 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:36.724 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272017+0000", 2024-09-18T04:39:36.724 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000425769 2024-09-18T04:39:36.724 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.724 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.724 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:36.724 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.294246+0000", 2024-09-18T04:39:36.724 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.022229237999999998 2024-09-18T04:39:36.724 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.724 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.725 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:36.725 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303571+0000", 2024-09-18T04:39:36.725 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0093250239999999995 2024-09-18T04:39:36.725 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.725 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.725 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:36.725 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303705+0000", 2024-09-18T04:39:36.725 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000133831 2024-09-18T04:39:36.725 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.725 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.725 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:36.725 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303782+0000", 2024-09-18T04:39:36.725 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7156000000000005e-05 2024-09-18T04:39:36.725 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:36.726 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:36.726 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:36.726 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.726 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.726 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:36 2.7 e51/32 2:fe1b02ad:::rbd_data.125fc09df6d1.0000000000000002:head v 51'1, mlcod=0'0)", 2024-09-18T04:39:36.726 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.273441+0000", 2024-09-18T04:39:36.726 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 54.426315961999997, 2024-09-18T04:39:36.726 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.032400768000000003, 2024-09-18T04:39:36.726 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:36.726 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:36.726 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:36.726 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.726 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:36.727 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273441+0000", 2024-09-18T04:39:36.727 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:36.727 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.727 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.727 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:36.727 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273441+0000", 2024-09-18T04:39:36.727 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:36.727 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.727 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.727 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:36.727 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273440+0000", 2024-09-18T04:39:36.727 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:36.727 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.727 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.728 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:36.728 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278723+0000", 2024-09-18T04:39:36.728 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0052829909999999999 2024-09-18T04:39:36.728 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.728 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.728 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:36.728 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278752+0000", 2024-09-18T04:39:36.728 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9178000000000001e-05 2024-09-18T04:39:36.728 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.728 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.728 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:36.728 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278761+0000", 2024-09-18T04:39:36.728 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7900000000000005e-06 2024-09-18T04:39:36.729 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.729 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.729 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:36.729 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278864+0000", 2024-09-18T04:39:36.729 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000102699 2024-09-18T04:39:36.729 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.729 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.729 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:36.729 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278900+0000", 2024-09-18T04:39:36.729 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6001999999999999e-05 2024-09-18T04:39:36.729 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.729 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.729 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:36.729 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.305793+0000", 2024-09-18T04:39:36.730 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.026893344 2024-09-18T04:39:36.730 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.730 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.730 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:36.730 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.305842+0000", 2024-09-18T04:39:36.730 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8720000000000001e-05 2024-09-18T04:39:36.730 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:36.730 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:36.730 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:36.730 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.730 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.730 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:39 2.7 e51/32 2:e9c909c5:::rbd_data.125fc09df6d1.0000000000000005:head v 51'2, mlcod=0'0)", 2024-09-18T04:39:36.730 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.283175+0000", 2024-09-18T04:39:36.730 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 54.416582273000003, 2024-09-18T04:39:36.731 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.022864102000000001, 2024-09-18T04:39:36.731 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:36.731 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:36.731 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:36.731 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.731 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:36.731 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283175+0000", 2024-09-18T04:39:36.731 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:36.731 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.731 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.731 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:36.731 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283175+0000", 2024-09-18T04:39:36.731 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:36.732 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.732 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.732 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:36.732 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283173+0000", 2024-09-18T04:39:36.732 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:36.732 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.732 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.732 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:36.732 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.285951+0000", 2024-09-18T04:39:36.732 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0027784540000000001 2024-09-18T04:39:36.732 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.732 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.732 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:36.733 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.285970+0000", 2024-09-18T04:39:36.733 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8855000000000002e-05 2024-09-18T04:39:36.733 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.733 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.733 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:36.733 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.285984+0000", 2024-09-18T04:39:36.733 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4252e-05 2024-09-18T04:39:36.733 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.733 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.733 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:36.733 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.286041+0000", 2024-09-18T04:39:36.733 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6431999999999998e-05 2024-09-18T04:39:36.733 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.734 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.734 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:36.734 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.286086+0000", 2024-09-18T04:39:36.734 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5423000000000002e-05 2024-09-18T04:39:36.734 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.734 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.734 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:36.734 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.305998+0000", 2024-09-18T04:39:36.734 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019911697999999999 2024-09-18T04:39:36.734 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.734 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.734 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:36.734 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306039+0000", 2024-09-18T04:39:36.734 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0973999999999998e-05 2024-09-18T04:39:36.735 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:36.735 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:36.735 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:36.735 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.735 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.735 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:50 2.3 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:36.735 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.312563+0000", 2024-09-18T04:39:36.735 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 54.387194246, 2024-09-18T04:39:36.735 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.032636001999999997, 2024-09-18T04:39:36.735 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:36.735 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:36.735 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:36.735 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:36.736 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:36.736 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 50 2024-09-18T04:39:36.736 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.736 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:36.736 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.736 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:36.736 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.312563+0000", 2024-09-18T04:39:36.736 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:36.736 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.736 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.736 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:36.736 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.312563+0000", 2024-09-18T04:39:36.737 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:36.737 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.737 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.737 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:36.737 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.312561+0000", 2024-09-18T04:39:36.737 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:36.737 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.737 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.737 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:36.737 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321709+0000", 2024-09-18T04:39:36.737 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0091486650000000003 2024-09-18T04:39:36.737 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.737 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.738 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:36.738 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321739+0000", 2024-09-18T04:39:36.738 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9666e-05 2024-09-18T04:39:36.738 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.738 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.738 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:36.738 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321751+0000", 2024-09-18T04:39:36.738 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2086000000000001e-05 2024-09-18T04:39:36.738 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.738 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.738 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:36.738 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321833+0000", 2024-09-18T04:39:36.738 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1842999999999996e-05 2024-09-18T04:39:36.738 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.738 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.739 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:36.739 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321981+0000", 2024-09-18T04:39:36.739 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000147441 2024-09-18T04:39:36.739 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.739 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.739 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:36.739 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.322109+0000", 2024-09-18T04:39:36.739 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012851200000000001 2024-09-18T04:39:36.739 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.739 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.739 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:36.739 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.332829+0000", 2024-09-18T04:39:36.739 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010720225999999999 2024-09-18T04:39:36.739 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.740 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.740 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:36.740 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.345039+0000", 2024-09-18T04:39:36.740 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012209831000000001 2024-09-18T04:39:36.740 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.740 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.740 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:36.740 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.345120+0000", 2024-09-18T04:39:36.740 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1145999999999999e-05 2024-09-18T04:39:36.740 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.740 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.740 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:36.740 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.345199+0000", 2024-09-18T04:39:36.741 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8546000000000003e-05 2024-09-18T04:39:36.741 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:36.741 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:36.741 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:36.741 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.741 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.741 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:30 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:36.741 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.516122+0000", 2024-09-18T04:39:36.741 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 54.183635084000002, 2024-09-18T04:39:36.741 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0091865030000000004, 2024-09-18T04:39:36.741 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:36.741 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:36.741 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:36.741 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:36.742 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:36.742 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 30 2024-09-18T04:39:36.742 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.742 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:36.742 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.742 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:36.742 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516122+0000", 2024-09-18T04:39:36.742 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:36.742 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.742 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.742 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:36.742 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516122+0000", 2024-09-18T04:39:36.742 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:36.742 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.743 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.743 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:36.743 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516120+0000", 2024-09-18T04:39:36.743 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:36.743 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.743 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.743 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:36.743 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516129+0000", 2024-09-18T04:39:36.743 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.85e-06 2024-09-18T04:39:36.743 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.743 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.743 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:36.743 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516155+0000", 2024-09-18T04:39:36.743 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5882000000000001e-05 2024-09-18T04:39:36.743 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.744 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.744 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:36.744 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516175+0000", 2024-09-18T04:39:36.744 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.022e-05 2024-09-18T04:39:36.744 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.744 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.744 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:36.744 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519780+0000", 2024-09-18T04:39:36.744 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003604187 2024-09-18T04:39:36.744 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.744 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.744 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:36.744 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519964+0000", 2024-09-18T04:39:36.745 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00018391999999999999 2024-09-18T04:39:36.745 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.745 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.745 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:36.745 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.520794+0000", 2024-09-18T04:39:36.745 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00083078100000000005 2024-09-18T04:39:36.745 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.745 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.745 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:36.745 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.524129+0000", 2024-09-18T04:39:36.745 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033342419999999998 2024-09-18T04:39:36.745 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.745 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.746 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:36.746 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525239+0000", 2024-09-18T04:39:36.746 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001110778 2024-09-18T04:39:36.746 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.746 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.746 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:36.746 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525277+0000", 2024-09-18T04:39:36.746 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7596999999999999e-05 2024-09-18T04:39:36.746 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.746 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.746 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:36.746 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525308+0000", 2024-09-18T04:39:36.746 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.1535999999999998e-05 2024-09-18T04:39:36.746 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:36.747 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:36.747 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:36.747 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.747 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.747 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:32 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:36.747 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.519217+0000", 2024-09-18T04:39:36.747 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 54.180539764000002, 2024-09-18T04:39:36.747 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082266370000000002, 2024-09-18T04:39:36.747 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:36.747 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:36.747 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:36.747 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:36.747 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:36.747 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 32 2024-09-18T04:39:36.747 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.748 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:36.748 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.748 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:36.748 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519217+0000", 2024-09-18T04:39:36.748 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:36.748 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.748 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.748 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:36.748 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519217+0000", 2024-09-18T04:39:36.748 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:36.748 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.748 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.748 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:36.749 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519215+0000", 2024-09-18T04:39:36.749 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:36.749 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.749 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.749 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:36.749 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519229+0000", 2024-09-18T04:39:36.749 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3329999999999999e-05 2024-09-18T04:39:36.749 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.749 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.749 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:36.749 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519268+0000", 2024-09-18T04:39:36.749 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9533000000000002e-05 2024-09-18T04:39:36.749 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.749 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.750 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:36.750 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519285+0000", 2024-09-18T04:39:36.750 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6446000000000001e-05 2024-09-18T04:39:36.750 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.750 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.750 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:36.750 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.522358+0000", 2024-09-18T04:39:36.750 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030734310000000002 2024-09-18T04:39:36.750 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.751 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.751 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:36.751 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.522551+0000", 2024-09-18T04:39:36.751 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019317600000000001 2024-09-18T04:39:36.751 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.751 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.751 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:36.751 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.523262+0000", 2024-09-18T04:39:36.751 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00071069999999999998 2024-09-18T04:39:36.751 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.751 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.751 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:36.751 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525318+0000", 2024-09-18T04:39:36.751 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0020561469999999999 2024-09-18T04:39:36.752 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.752 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.752 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:36.752 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.527376+0000", 2024-09-18T04:39:36.752 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0020584969999999998 2024-09-18T04:39:36.752 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.752 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.752 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:36.752 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.527416+0000", 2024-09-18T04:39:36.752 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9054000000000003e-05 2024-09-18T04:39:36.752 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.752 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.752 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:36.752 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.527444+0000", 2024-09-18T04:39:36.753 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8347000000000002e-05 2024-09-18T04:39:36.753 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:36.753 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:36.753 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:36.753 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.753 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.753 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:42 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:36.753 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.551160+0000", 2024-09-18T04:39:36.753 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 54.148597371999998, 2024-09-18T04:39:36.753 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010692757000000001, 2024-09-18T04:39:36.753 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:36.753 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:36.753 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:36.753 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:36.754 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:36.754 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 42 2024-09-18T04:39:36.754 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.754 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:36.754 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.754 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:36.754 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551160+0000", 2024-09-18T04:39:36.754 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:36.754 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.754 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.754 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:36.754 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551160+0000", 2024-09-18T04:39:36.754 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:36.754 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.755 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.755 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:36.755 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551159+0000", 2024-09-18T04:39:36.755 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:36.755 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.755 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.755 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:36.755 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551166+0000", 2024-09-18T04:39:36.755 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1029999999999996e-06 2024-09-18T04:39:36.755 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.755 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.755 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:36.755 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551177+0000", 2024-09-18T04:39:36.755 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1384e-05 2024-09-18T04:39:36.756 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.756 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.756 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:36.756 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551185+0000", 2024-09-18T04:39:36.756 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1990000000000006e-06 2024-09-18T04:39:36.756 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.756 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.756 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:36.756 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552021+0000", 2024-09-18T04:39:36.756 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00083574400000000003 2024-09-18T04:39:36.756 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.756 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.756 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:36.756 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552339+0000", 2024-09-18T04:39:36.757 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031771399999999997 2024-09-18T04:39:36.757 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.757 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.757 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:36.757 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.553140+0000", 2024-09-18T04:39:36.757 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00080088100000000003 2024-09-18T04:39:36.757 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.757 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.757 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:36.757 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.554585+0000", 2024-09-18T04:39:36.757 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001445801 2024-09-18T04:39:36.757 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.757 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.757 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:36.758 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.561711+0000", 2024-09-18T04:39:36.758 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071260109999999998 2024-09-18T04:39:36.758 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.758 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.758 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:36.758 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.561790+0000", 2024-09-18T04:39:36.758 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8967000000000005e-05 2024-09-18T04:39:36.758 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.758 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.758 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:36.758 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.561852+0000", 2024-09-18T04:39:36.758 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2045999999999995e-05 2024-09-18T04:39:36.758 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:36.758 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:36.758 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:36.759 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.759 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.759 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:58 2.7 e51/32 2:fc7c6cd3:ns::rbd_data.133e813ab5bc.000000000000000e:head v 51'7, mlcod=51'5)", 2024-09-18T04:39:36.759 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.605333+0000", 2024-09-18T04:39:36.759 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 54.094424167, 2024-09-18T04:39:36.759 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020085902999999999, 2024-09-18T04:39:36.759 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:36.759 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:36.759 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:36.759 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.759 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:36.759 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605333+0000", 2024-09-18T04:39:36.759 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:36.759 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.760 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.760 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:36.760 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605333+0000", 2024-09-18T04:39:36.760 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:36.760 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.760 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.760 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:36.760 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605332+0000", 2024-09-18T04:39:36.760 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:36.760 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.760 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.760 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:36.760 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608667+0000", 2024-09-18T04:39:36.760 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033350300000000001 2024-09-18T04:39:36.761 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.761 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.761 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:36.761 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608706+0000", 2024-09-18T04:39:36.761 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8951e-05 2024-09-18T04:39:36.761 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.761 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.761 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:36.761 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608725+0000", 2024-09-18T04:39:36.761 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9675999999999999e-05 2024-09-18T04:39:36.761 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.761 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.761 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:36.761 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608798+0000", 2024-09-18T04:39:36.762 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3135999999999999e-05 2024-09-18T04:39:36.762 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.762 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.762 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:36.762 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608824+0000", 2024-09-18T04:39:36.762 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5284000000000001e-05 2024-09-18T04:39:36.762 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.762 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.762 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:36.762 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625370+0000", 2024-09-18T04:39:36.762 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016545975000000001 2024-09-18T04:39:36.762 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.762 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.763 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:36.763 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625419+0000", 2024-09-18T04:39:36.763 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9076999999999998e-05 2024-09-18T04:39:36.763 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:36.763 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:36.763 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:36.763 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.763 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.763 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:611 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:36.763 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.069561+0000", 2024-09-18T04:39:36.763 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 45.630195612999998, 2024-09-18T04:39:36.763 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0072330299999999997, 2024-09-18T04:39:36.763 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:36.763 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:36.763 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:36.764 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:39:36.764 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:36.764 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 611 2024-09-18T04:39:36.764 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.764 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:36.764 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.764 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:36.764 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.069561+0000", 2024-09-18T04:39:36.764 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:36.764 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.764 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.764 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:36.764 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.069561+0000", 2024-09-18T04:39:36.765 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:36.765 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.765 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.765 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:36.765 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.069559+0000", 2024-09-18T04:39:36.765 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:36.765 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.765 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.765 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:36.765 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070801+0000", 2024-09-18T04:39:36.765 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0012423359999999999 2024-09-18T04:39:36.765 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.765 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.765 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:36.766 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070828+0000", 2024-09-18T04:39:36.766 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6387e-05 2024-09-18T04:39:36.766 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.766 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.766 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:36.766 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070840+0000", 2024-09-18T04:39:36.766 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2689e-05 2024-09-18T04:39:36.766 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.766 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.766 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:36.766 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070990+0000", 2024-09-18T04:39:36.766 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014946800000000001 2024-09-18T04:39:36.766 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.766 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.767 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:36.767 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.071128+0000", 2024-09-18T04:39:36.767 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013796699999999999 2024-09-18T04:39:36.767 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.767 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.767 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:36.767 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.071318+0000", 2024-09-18T04:39:36.767 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000190106 2024-09-18T04:39:36.767 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.767 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.767 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:36.767 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.075711+0000", 2024-09-18T04:39:36.767 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.004393002 2024-09-18T04:39:36.767 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.768 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.768 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:36.768 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.076665+0000", 2024-09-18T04:39:36.768 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00095390399999999999 2024-09-18T04:39:36.768 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.768 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.768 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:36.768 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.076742+0000", 2024-09-18T04:39:36.768 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7112000000000003e-05 2024-09-18T04:39:36.768 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.768 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.768 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:36.768 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.076794+0000", 2024-09-18T04:39:36.768 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2475999999999997e-05 2024-09-18T04:39:36.769 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:36.769 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:36.769 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:36.769 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.769 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.769 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:687 2.3 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:36.769 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.487994+0000", 2024-09-18T04:39:36.769 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 45.211762630000003, 2024-09-18T04:39:36.769 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0074167690000000001, 2024-09-18T04:39:36.769 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:36.769 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:36.769 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:36.769 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:39:36.770 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:36.770 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 687 2024-09-18T04:39:36.770 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.770 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:36.770 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.770 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:36.770 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.487994+0000", 2024-09-18T04:39:36.770 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:36.770 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.770 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.770 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:36.770 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.487994+0000", 2024-09-18T04:39:36.770 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:36.770 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.771 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.771 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:36.771 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.487993+0000", 2024-09-18T04:39:36.771 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:36.771 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.771 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.771 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:36.771 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.488979+0000", 2024-09-18T04:39:36.771 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00098620700000000001 2024-09-18T04:39:36.771 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.771 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.771 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:36.771 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489002+0000", 2024-09-18T04:39:36.771 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2500000000000001e-05 2024-09-18T04:39:36.772 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.772 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.772 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:36.772 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489020+0000", 2024-09-18T04:39:36.772 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7691000000000001e-05 2024-09-18T04:39:36.772 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.772 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.772 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:36.772 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489091+0000", 2024-09-18T04:39:36.772 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1307e-05 2024-09-18T04:39:36.772 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.772 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.772 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:36.772 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489219+0000", 2024-09-18T04:39:36.772 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012766000000000001 2024-09-18T04:39:36.773 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.773 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.773 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:36.773 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489387+0000", 2024-09-18T04:39:36.773 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016855400000000001 2024-09-18T04:39:36.773 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.773 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.773 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:36.773 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495243+0000", 2024-09-18T04:39:36.773 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058563189999999996 2024-09-18T04:39:36.773 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.773 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.773 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:36.773 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495266+0000", 2024-09-18T04:39:36.774 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2810999999999999e-05 2024-09-18T04:39:36.774 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.774 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.774 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:36.774 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495332+0000", 2024-09-18T04:39:36.774 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5469999999999995e-05 2024-09-18T04:39:36.774 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.774 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.774 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:36.774 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495411+0000", 2024-09-18T04:39:36.774 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9395999999999997e-05 2024-09-18T04:39:36.774 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:36.774 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:36.774 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:36.775 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.775 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.775 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:689 2.3 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:36.775 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.500070+0000", 2024-09-18T04:39:36.775 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 45.199686814000003, 2024-09-18T04:39:36.775 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071096650000000003, 2024-09-18T04:39:36.775 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:36.775 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:36.775 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:36.775 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:39:36.775 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:36.775 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 689 2024-09-18T04:39:36.775 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.775 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:36.776 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.776 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:36.776 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.500070+0000", 2024-09-18T04:39:36.776 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:36.776 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.776 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.776 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:36.776 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.500070+0000", 2024-09-18T04:39:36.776 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:36.776 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.776 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.776 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:36.776 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.500069+0000", 2024-09-18T04:39:36.776 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:36.777 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.777 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.777 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:36.777 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501082+0000", 2024-09-18T04:39:36.777 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010136570000000001 2024-09-18T04:39:36.777 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.777 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.777 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:36.777 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501104+0000", 2024-09-18T04:39:36.777 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1883e-05 2024-09-18T04:39:36.777 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.777 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.777 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:36.777 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501115+0000", 2024-09-18T04:39:36.778 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0414000000000001e-05 2024-09-18T04:39:36.778 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.778 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.778 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:36.778 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501178+0000", 2024-09-18T04:39:36.778 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3128999999999998e-05 2024-09-18T04:39:36.778 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.778 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.778 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:36.778 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501257+0000", 2024-09-18T04:39:36.778 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9592999999999995e-05 2024-09-18T04:39:36.778 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.778 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.778 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:36.779 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501366+0000", 2024-09-18T04:39:36.779 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000108643 2024-09-18T04:39:36.779 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.779 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.779 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:36.779 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.504041+0000", 2024-09-18T04:39:36.779 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0026750889999999999 2024-09-18T04:39:36.779 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.779 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.779 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:36.779 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.507097+0000", 2024-09-18T04:39:36.779 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030556070000000001 2024-09-18T04:39:36.779 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.779 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.780 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:36.780 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.507140+0000", 2024-09-18T04:39:36.780 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3745000000000002e-05 2024-09-18T04:39:36.780 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.780 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.780 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:36.780 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.507180+0000", 2024-09-18T04:39:36.780 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9475999999999999e-05 2024-09-18T04:39:36.780 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:36.780 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:36.780 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:36.780 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.780 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.780 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4972.0:13 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:36.781 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.948263+0000", 2024-09-18T04:39:36.781 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 44.751494022999999, 2024-09-18T04:39:36.781 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0091358970000000005, 2024-09-18T04:39:36.781 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:36.781 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:39:36.781 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:36.781 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4972", 2024-09-18T04:39:36.781 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2280251472", 2024-09-18T04:39:36.781 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 13 2024-09-18T04:39:36.781 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.781 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:36.781 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.781 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:36.782 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948263+0000", 2024-09-18T04:39:36.782 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:36.782 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.782 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.782 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:36.782 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948263+0000", 2024-09-18T04:39:36.782 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:36.782 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.782 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.782 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:36.782 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948261+0000", 2024-09-18T04:39:36.782 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:36.782 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.782 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.783 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:36.783 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948273+0000", 2024-09-18T04:39:36.783 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1576999999999999e-05 2024-09-18T04:39:36.783 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.783 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.783 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:36.783 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948305+0000", 2024-09-18T04:39:36.783 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2209000000000003e-05 2024-09-18T04:39:36.783 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.783 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.783 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:36.783 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948325+0000", 2024-09-18T04:39:36.783 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.022e-05 2024-09-18T04:39:36.783 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.784 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.784 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:36.784 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948455+0000", 2024-09-18T04:39:36.784 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000129624 2024-09-18T04:39:36.784 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.784 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.784 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:36.784 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948598+0000", 2024-09-18T04:39:36.784 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014376699999999999 2024-09-18T04:39:36.784 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.784 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.784 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:36.784 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.957399+0000", 2024-09-18T04:39:36.784 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0088004299999999997 2024-09-18T04:39:36.785 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:36.785 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:36.785 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:36.785 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.785 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.785 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:101 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:36.785 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.576367+0000", 2024-09-18T04:39:36.785 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 40.123390315000002, 2024-09-18T04:39:36.785 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0078671020000000008, 2024-09-18T04:39:36.785 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:36.785 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:36.785 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:36.785 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:36.785 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:36.786 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 101 2024-09-18T04:39:36.786 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.786 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:36.786 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.786 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:36.786 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576367+0000", 2024-09-18T04:39:36.786 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:36.786 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.786 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.786 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:36.786 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576367+0000", 2024-09-18T04:39:36.786 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:36.786 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.786 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.787 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:36.787 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576364+0000", 2024-09-18T04:39:36.787 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999971 2024-09-18T04:39:36.787 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.787 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.787 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:36.787 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576378+0000", 2024-09-18T04:39:36.787 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3879999999999999e-05 2024-09-18T04:39:36.787 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.787 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.787 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:36.787 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576410+0000", 2024-09-18T04:39:36.787 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2286000000000002e-05 2024-09-18T04:39:36.787 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.788 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.788 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:36.788 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576425+0000", 2024-09-18T04:39:36.788 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5290000000000001e-05 2024-09-18T04:39:36.788 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.788 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.788 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:36.788 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576551+0000", 2024-09-18T04:39:36.788 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012555199999999999 2024-09-18T04:39:36.788 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.788 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.788 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:36.788 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576719+0000", 2024-09-18T04:39:36.788 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016787400000000001 2024-09-18T04:39:36.789 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.789 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.789 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:36.789 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576951+0000", 2024-09-18T04:39:36.789 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00023252400000000001 2024-09-18T04:39:36.789 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.789 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.789 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:36.789 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.581459+0000", 2024-09-18T04:39:36.789 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045074410000000001 2024-09-18T04:39:36.789 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.789 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.789 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:36.789 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584076+0000", 2024-09-18T04:39:36.790 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0026173860000000002 2024-09-18T04:39:36.790 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.790 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.790 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:36.790 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584160+0000", 2024-09-18T04:39:36.790 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4191999999999996e-05 2024-09-18T04:39:36.790 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.790 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.790 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:36.790 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584234+0000", 2024-09-18T04:39:36.790 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.339e-05 2024-09-18T04:39:36.790 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:36.790 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:36.790 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:36.791 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.791 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.791 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:44 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:36.791 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.550448+0000", 2024-09-18T04:39:36.791 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 36.149309125999999, 2024-09-18T04:39:36.791 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082117590000000008, 2024-09-18T04:39:36.791 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:36.791 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:36.791 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:36.791 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:39:36.791 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:36.791 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 44 2024-09-18T04:39:36.791 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.791 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:36.792 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.792 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:36.792 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.550448+0000", 2024-09-18T04:39:36.792 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:36.792 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.792 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.792 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:36.792 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.550448+0000", 2024-09-18T04:39:36.792 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:36.792 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.792 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.792 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:36.792 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.550445+0000", 2024-09-18T04:39:36.793 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999971 2024-09-18T04:39:36.793 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.793 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.793 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:36.793 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.551904+0000", 2024-09-18T04:39:36.793 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001458322 2024-09-18T04:39:36.793 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.793 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.793 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:36.793 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.551938+0000", 2024-09-18T04:39:36.793 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4359000000000001e-05 2024-09-18T04:39:36.793 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.793 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.793 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:36.793 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.551959+0000", 2024-09-18T04:39:36.794 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1403000000000002e-05 2024-09-18T04:39:36.794 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.794 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.794 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:36.794 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.552049+0000", 2024-09-18T04:39:36.794 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9970999999999996e-05 2024-09-18T04:39:36.794 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.794 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.794 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:36.794 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.552190+0000", 2024-09-18T04:39:36.794 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000140271 2024-09-18T04:39:36.794 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.794 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.794 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:36.795 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.552385+0000", 2024-09-18T04:39:36.795 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019498700000000001 2024-09-18T04:39:36.795 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.795 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.795 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:36.795 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.556308+0000", 2024-09-18T04:39:36.795 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00392333 2024-09-18T04:39:36.795 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.795 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.795 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:36.795 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.558479+0000", 2024-09-18T04:39:36.795 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0021715889999999998 2024-09-18T04:39:36.795 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.795 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.796 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:36.796 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.558616+0000", 2024-09-18T04:39:36.796 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013613399999999999 2024-09-18T04:39:36.796 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.796 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.796 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:36.796 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.558660+0000", 2024-09-18T04:39:36.796 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4053e-05 2024-09-18T04:39:36.796 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:36.796 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:36.796 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:36.796 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.796 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.797 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:46 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:36.797 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.564089+0000", 2024-09-18T04:39:36.797 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 36.135667996999999, 2024-09-18T04:39:36.797 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0069437090000000002, 2024-09-18T04:39:36.797 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:36.797 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:36.797 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:36.797 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:39:36.797 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:36.797 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 46 2024-09-18T04:39:36.797 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.797 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:36.797 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.797 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:36.797 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.564089+0000", 2024-09-18T04:39:36.798 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:36.798 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.798 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.798 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:36.798 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.564089+0000", 2024-09-18T04:39:36.798 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:36.798 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.798 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.798 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:36.798 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.564087+0000", 2024-09-18T04:39:36.798 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:36.798 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.798 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.798 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:36.799 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565126+0000", 2024-09-18T04:39:36.799 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001038902 2024-09-18T04:39:36.799 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.799 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.799 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:36.799 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565148+0000", 2024-09-18T04:39:36.799 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1911000000000001e-05 2024-09-18T04:39:36.799 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.799 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.799 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:36.799 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565157+0000", 2024-09-18T04:39:36.799 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6220000000000004e-06 2024-09-18T04:39:36.799 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.799 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.800 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:36.800 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565261+0000", 2024-09-18T04:39:36.800 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000103063 2024-09-18T04:39:36.800 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.800 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.800 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:36.800 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565394+0000", 2024-09-18T04:39:36.800 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000133032 2024-09-18T04:39:36.800 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.800 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.800 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:36.800 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565564+0000", 2024-09-18T04:39:36.800 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017068000000000001 2024-09-18T04:39:36.800 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.800 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.801 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:36.801 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.570556+0000", 2024-09-18T04:39:36.801 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0049920620000000002 2024-09-18T04:39:36.801 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.801 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.801 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:36.801 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.570885+0000", 2024-09-18T04:39:36.801 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032827900000000001 2024-09-18T04:39:36.801 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.801 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.801 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:36.801 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.570959+0000", 2024-09-18T04:39:36.801 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3955999999999995e-05 2024-09-18T04:39:36.801 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.802 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.802 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:36.802 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.571033+0000", 2024-09-18T04:39:36.802 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4191999999999997e-05 2024-09-18T04:39:36.802 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:36.802 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:36.802 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:36.802 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.802 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.802 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:100 2.4 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:36.802 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.856389+0000", 2024-09-18T04:39:36.802 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 35.843367884999999, 2024-09-18T04:39:36.802 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0081480800000000003, 2024-09-18T04:39:36.803 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:36.803 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:36.803 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:36.803 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:39:36.803 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:36.803 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 100 2024-09-18T04:39:36.803 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.803 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:36.803 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.803 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:36.803 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.856389+0000", 2024-09-18T04:39:36.803 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:36.803 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.804 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.804 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:36.804 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.856389+0000", 2024-09-18T04:39:36.804 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:36.804 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.804 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.804 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:36.804 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.856388+0000", 2024-09-18T04:39:36.804 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:36.804 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.804 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.804 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:36.804 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857295+0000", 2024-09-18T04:39:36.804 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00090725599999999995 2024-09-18T04:39:36.805 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.805 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.805 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:36.805 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857311+0000", 2024-09-18T04:39:36.805 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6092000000000001e-05 2024-09-18T04:39:36.805 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.805 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.805 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:36.805 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857319+0000", 2024-09-18T04:39:36.805 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9200000000000004e-06 2024-09-18T04:39:36.805 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.805 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.805 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:36.805 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857388+0000", 2024-09-18T04:39:36.806 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8591999999999997e-05 2024-09-18T04:39:36.806 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.806 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.806 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:36.806 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857535+0000", 2024-09-18T04:39:36.806 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014684299999999999 2024-09-18T04:39:36.806 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.806 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.806 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:36.806 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857747+0000", 2024-09-18T04:39:36.806 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000212361 2024-09-18T04:39:36.806 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.806 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.807 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:36.807 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.860831+0000", 2024-09-18T04:39:36.807 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030841789999999998 2024-09-18T04:39:36.807 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.807 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.807 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:36.807 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.864361+0000", 2024-09-18T04:39:36.807 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0035298780000000002 2024-09-18T04:39:36.807 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.807 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.807 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:36.807 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.864456+0000", 2024-09-18T04:39:36.807 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4557999999999998e-05 2024-09-18T04:39:36.807 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.808 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.808 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:36.808 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.864537+0000", 2024-09-18T04:39:36.808 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1577999999999997e-05 2024-09-18T04:39:36.808 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:36.808 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:36.808 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:36.808 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.808 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.808 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:102 2.4 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:36.808 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.869773+0000", 2024-09-18T04:39:36.808 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 35.829984228000001, 2024-09-18T04:39:36.808 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0084430359999999992, 2024-09-18T04:39:36.809 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:36.809 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:36.809 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:36.809 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:39:36.809 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:36.809 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 102 2024-09-18T04:39:36.809 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.809 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:36.809 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.809 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:36.809 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.869773+0000", 2024-09-18T04:39:36.809 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:36.809 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.809 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.809 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:36.810 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.869773+0000", 2024-09-18T04:39:36.810 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:36.810 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.810 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.810 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:36.810 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.869771+0000", 2024-09-18T04:39:36.810 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:36.810 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.810 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.810 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:36.810 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870629+0000", 2024-09-18T04:39:36.810 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00085760999999999997 2024-09-18T04:39:36.810 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.810 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.811 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:36.811 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870647+0000", 2024-09-18T04:39:36.811 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8363999999999999e-05 2024-09-18T04:39:36.811 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.811 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.811 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:36.811 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870655+0000", 2024-09-18T04:39:36.811 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0099999999999995e-06 2024-09-18T04:39:36.811 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.811 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.811 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:36.811 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870803+0000", 2024-09-18T04:39:36.811 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014798 2024-09-18T04:39:36.811 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.812 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.812 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:36.812 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870951+0000", 2024-09-18T04:39:36.812 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014715999999999999 2024-09-18T04:39:36.812 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.812 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.812 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:36.812 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.871145+0000", 2024-09-18T04:39:36.812 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000194253 2024-09-18T04:39:36.812 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.812 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.812 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:36.812 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.877033+0000", 2024-09-18T04:39:36.813 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058883099999999999 2024-09-18T04:39:36.813 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.813 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.813 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:36.813 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.878078+0000", 2024-09-18T04:39:36.813 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001044747 2024-09-18T04:39:36.813 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.813 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.813 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:36.813 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.878160+0000", 2024-09-18T04:39:36.813 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1904999999999996e-05 2024-09-18T04:39:36.813 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.813 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.814 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:36.814 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.878216+0000", 2024-09-18T04:39:36.814 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6082000000000003e-05 2024-09-18T04:39:36.814 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:36.814 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:36.814 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:36.814 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.814 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.814 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:145 2.4 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:36.814 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:01.096935+0000", 2024-09-18T04:39:36.814 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 35.602822447999998, 2024-09-18T04:39:36.814 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0069937869999999996, 2024-09-18T04:39:36.814 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:36.814 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:36.815 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:36.815 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:39:36.815 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:36.815 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 145 2024-09-18T04:39:36.815 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.815 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:36.815 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.815 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:36.815 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.096935+0000", 2024-09-18T04:39:36.815 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:36.815 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.815 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.815 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:36.815 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.096935+0000", 2024-09-18T04:39:36.816 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:36.816 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.816 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.816 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:36.816 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.096932+0000", 2024-09-18T04:39:36.816 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:36.816 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.816 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.816 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:36.816 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097815+0000", 2024-09-18T04:39:36.816 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00088276000000000001 2024-09-18T04:39:36.816 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.816 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.817 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:36.817 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097842+0000", 2024-09-18T04:39:36.817 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6806999999999999e-05 2024-09-18T04:39:36.817 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.817 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.817 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:36.817 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097852+0000", 2024-09-18T04:39:36.817 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0548e-05 2024-09-18T04:39:36.817 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.817 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.817 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:36.817 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097931+0000", 2024-09-18T04:39:36.817 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8417000000000005e-05 2024-09-18T04:39:36.818 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.818 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.818 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:36.818 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.098047+0000", 2024-09-18T04:39:36.818 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011611 2024-09-18T04:39:36.818 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.818 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.818 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:36.818 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.098172+0000", 2024-09-18T04:39:36.818 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000125482 2024-09-18T04:39:36.818 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.818 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.818 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:36.818 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.100888+0000", 2024-09-18T04:39:36.819 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0027154430000000001 2024-09-18T04:39:36.819 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.819 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.819 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:36.819 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.103814+0000", 2024-09-18T04:39:36.819 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0029259619999999998 2024-09-18T04:39:36.819 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.819 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.819 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:36.819 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.103891+0000", 2024-09-18T04:39:36.819 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7619000000000003e-05 2024-09-18T04:39:36.819 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:36.820 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:36.820 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:36.820 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.103928+0000", 2024-09-18T04:39:36.820 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6900999999999998e-05 2024-09-18T04:39:36.820 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:36.820 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:36.820 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:36.820 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:36.820 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:36.820 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:36.822 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:36.816+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:36.822 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:39:36.923 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:36.916+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:36.940 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:36.941 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:36.941 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:39:36.941 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:36.951 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:39:37.024 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:37.016+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:37.061 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:37.061 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:37.061 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:39:37.061 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:39:37.061 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:37.072 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:39:37.124 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:37.116+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:37.184 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:37.184 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:39:37.184 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:39:37.185 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:39:37.185 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.185 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:37 2.5 e51/28 2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:head v 51'1, mlcod=0'0)", 2024-09-18T04:39:37.185 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:39:37.185 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 54.887419168000001, 2024-09-18T04:39:37.185 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012279973, 2024-09-18T04:39:37.185 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:37.185 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:37.185 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:37.185 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.185 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:37.185 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:39:37.186 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.186 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.186 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:37.186 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:39:37.186 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.186 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.186 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:37.186 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292468+0000", 2024-09-18T04:39:37.186 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:37.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.186 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.186 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:37.186 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295837+0000", 2024-09-18T04:39:37.186 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0033691699999999999 2024-09-18T04:39:37.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.187 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.187 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:37.187 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295860+0000", 2024-09-18T04:39:37.187 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2756999999999999e-05 2024-09-18T04:39:37.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.187 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.187 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:37.187 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295872+0000", 2024-09-18T04:39:37.187 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2442e-05 2024-09-18T04:39:37.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.187 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.187 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:37.187 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295934+0000", 2024-09-18T04:39:37.187 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1187999999999998e-05 2024-09-18T04:39:37.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.188 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.188 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:37.188 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295967+0000", 2024-09-18T04:39:37.188 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3185999999999997e-05 2024-09-18T04:39:37.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.188 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.188 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:37.188 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304697+0000", 2024-09-18T04:39:37.188 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0087298059999999997 2024-09-18T04:39:37.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.188 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.188 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:37.188 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304749+0000", 2024-09-18T04:39:37.188 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1922000000000002e-05 2024-09-18T04:39:37.189 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.189 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:37.189 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.189 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.189 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:49 2.5 e51/28 2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head v 51'2, mlcod=0'0)", 2024-09-18T04:39:37.189 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:39:37.189 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 54.85409233, 2024-09-18T04:39:37.189 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018033041999999999, 2024-09-18T04:39:37.189 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:37.189 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:37.189 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:37.189 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.189 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:37.190 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:39:37.190 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.190 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.190 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:37.190 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:39:37.190 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.190 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.190 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:37.190 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325793+0000", 2024-09-18T04:39:37.190 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:37.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.190 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.190 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:37.191 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329038+0000", 2024-09-18T04:39:37.191 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0032444010000000001 2024-09-18T04:39:37.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.191 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.191 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:37.191 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329065+0000", 2024-09-18T04:39:37.191 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7500000000000001e-05 2024-09-18T04:39:37.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.191 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.191 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:37.191 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329077+0000", 2024-09-18T04:39:37.191 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1919999999999999e-05 2024-09-18T04:39:37.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.191 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.192 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:37.192 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329149+0000", 2024-09-18T04:39:37.192 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2349999999999997e-05 2024-09-18T04:39:37.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.192 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.192 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:37.192 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329192+0000", 2024-09-18T04:39:37.192 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2744000000000003e-05 2024-09-18T04:39:37.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.192 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.192 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:37.192 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343773+0000", 2024-09-18T04:39:37.192 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014581246000000001 2024-09-18T04:39:37.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.193 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.193 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:37.193 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343829+0000", 2024-09-18T04:39:37.193 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5143999999999999e-05 2024-09-18T04:39:37.193 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.193 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:37.193 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.193 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.193 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:47 2.4 e51/17 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 51'3, mlcod=51'1)", 2024-09-18T04:39:37.194 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:39:37.194 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 54.853512064999997, 2024-09-18T04:39:37.194 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018960186, 2024-09-18T04:39:37.194 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:37.194 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:37.194 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:37.194 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.194 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:37.194 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:39:37.194 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.194 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.194 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:37.194 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:39:37.195 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.195 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.195 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:37.195 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326374+0000", 2024-09-18T04:39:37.195 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:37.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.195 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.195 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:37.195 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330883+0000", 2024-09-18T04:39:37.195 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0045084740000000002 2024-09-18T04:39:37.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.195 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.195 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:37.196 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330901+0000", 2024-09-18T04:39:37.196 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7890999999999999e-05 2024-09-18T04:39:37.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.196 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.196 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:37.196 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330912+0000", 2024-09-18T04:39:37.196 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1041e-05 2024-09-18T04:39:37.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.196 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.196 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:37.196 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.331003+0000", 2024-09-18T04:39:37.196 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1323999999999995e-05 2024-09-18T04:39:37.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.196 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.197 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:37.197 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.331030+0000", 2024-09-18T04:39:37.197 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7351999999999998e-05 2024-09-18T04:39:37.197 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.197 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.197 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:37.197 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345287+0000", 2024-09-18T04:39:37.197 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014256332 2024-09-18T04:39:37.197 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.197 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.197 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:37.197 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345336+0000", 2024-09-18T04:39:37.197 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9356e-05 2024-09-18T04:39:37.197 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.198 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:37.198 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.198 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.198 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:58 2.5 e51/28 2:b227e6b1:::rbd_data.125fc09df6d1.000000000000000e:head v 51'3, mlcod=51'1)", 2024-09-18T04:39:37.198 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:39:37.198 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 54.813085438000002, 2024-09-18T04:39:37.198 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016513678, 2024-09-18T04:39:37.198 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:37.198 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:37.198 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:37.198 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.198 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:37.198 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:39:37.199 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.199 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.199 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:37.199 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:39:37.199 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.199 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.199 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:37.199 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366800+0000", 2024-09-18T04:39:37.199 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:37.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.199 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.199 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:37.199 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368644+0000", 2024-09-18T04:39:37.200 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001844399 2024-09-18T04:39:37.200 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.200 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.200 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:37.200 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368681+0000", 2024-09-18T04:39:37.200 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6526999999999998e-05 2024-09-18T04:39:37.200 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.200 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.200 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:37.200 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368698+0000", 2024-09-18T04:39:37.200 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7272999999999999e-05 2024-09-18T04:39:37.200 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.200 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.200 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:37.200 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368773+0000", 2024-09-18T04:39:37.201 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4862000000000003e-05 2024-09-18T04:39:37.201 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.201 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.201 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:37.201 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368801+0000", 2024-09-18T04:39:37.201 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7925000000000001e-05 2024-09-18T04:39:37.201 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.201 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.201 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:37.201 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383261+0000", 2024-09-18T04:39:37.201 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014460377999999999 2024-09-18T04:39:37.201 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.201 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.201 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:37.202 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383316+0000", 2024-09-18T04:39:37.202 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4623999999999997e-05 2024-09-18T04:39:37.202 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.202 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:37.202 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.202 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.202 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.202 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:56 2.4 e51/17 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 51'4, mlcod=51'2)", 2024-09-18T04:39:37.202 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:39:37.202 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 54.811081412, 2024-09-18T04:39:37.202 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015105221, 2024-09-18T04:39:37.202 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:37.202 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:37.202 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:37.203 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.203 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:37.203 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:39:37.203 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.203 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.203 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:37.203 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:39:37.203 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.203 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.203 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:37.203 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368805+0000", 2024-09-18T04:39:37.203 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:37.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.204 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.204 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:37.204 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371603+0000", 2024-09-18T04:39:37.204 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027982660000000002 2024-09-18T04:39:37.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.204 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.204 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:37.204 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371633+0000", 2024-09-18T04:39:37.204 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0073000000000002e-05 2024-09-18T04:39:37.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.204 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.204 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:37.204 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371652+0000", 2024-09-18T04:39:37.204 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8597000000000002e-05 2024-09-18T04:39:37.205 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.205 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.205 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:37.205 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371752+0000", 2024-09-18T04:39:37.205 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9772999999999995e-05 2024-09-18T04:39:37.205 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.205 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.205 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:37.205 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371793+0000", 2024-09-18T04:39:37.205 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1173e-05 2024-09-18T04:39:37.205 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.205 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.205 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:37.205 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383875+0000", 2024-09-18T04:39:37.205 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012081852000000001 2024-09-18T04:39:37.206 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.206 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.206 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:37.206 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383912+0000", 2024-09-18T04:39:37.206 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7045000000000002e-05 2024-09-18T04:39:37.206 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.206 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:37.206 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.206 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.206 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.206 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:36 2.5 e51/28 2:bd5b8dd8:ns::rbd_data.133e813ab5bc.0000000000000002:head v 51'4, mlcod=51'2)", 2024-09-18T04:39:37.206 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:39:37.206 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 54.642362421000001, 2024-09-18T04:39:37.206 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020758161000000001, 2024-09-18T04:39:37.207 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:37.207 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:37.207 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:37.207 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.207 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:37.207 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:39:37.207 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.207 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.207 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.207 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:37.207 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:39:37.207 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.207 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.207 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.207 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:37.207 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537523+0000", 2024-09-18T04:39:37.208 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:37.208 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.208 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.208 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:37.208 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541578+0000", 2024-09-18T04:39:37.208 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0040549710000000001 2024-09-18T04:39:37.208 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.208 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.208 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:37.208 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541608+0000", 2024-09-18T04:39:37.208 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9349e-05 2024-09-18T04:39:37.208 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.208 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.208 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:37.208 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541621+0000", 2024-09-18T04:39:37.209 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2736e-05 2024-09-18T04:39:37.209 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.209 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.209 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:37.209 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541715+0000", 2024-09-18T04:39:37.209 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4197999999999995e-05 2024-09-18T04:39:37.209 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.209 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.209 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:37.209 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541755+0000", 2024-09-18T04:39:37.209 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9818999999999999e-05 2024-09-18T04:39:37.209 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.209 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.210 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:37.210 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558217+0000", 2024-09-18T04:39:37.210 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016462234999999999 2024-09-18T04:39:37.210 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.211 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.211 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:37.211 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558284+0000", 2024-09-18T04:39:37.211 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6774999999999997e-05 2024-09-18T04:39:37.211 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.211 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:37.211 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.211 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.211 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.211 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:39 2.4 e51/17 2:304364d1:ns::rbd_data.133e813ab5bc.0000000000000004:head v 51'5, mlcod=51'3)", 2024-09-18T04:39:37.211 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:39:37.211 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 54.634223294000002, 2024-09-18T04:39:37.211 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016777558000000001, 2024-09-18T04:39:37.211 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:37.212 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:37.212 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:37.212 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.212 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:37.212 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:39:37.212 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.212 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.212 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.212 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:37.212 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:39:37.212 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.212 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.212 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.212 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:37.212 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545663+0000", 2024-09-18T04:39:37.213 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:37.213 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.215 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.215 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:37.215 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548286+0000", 2024-09-18T04:39:37.216 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026233390000000001 2024-09-18T04:39:37.216 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.216 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.216 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:37.216 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548305+0000", 2024-09-18T04:39:37.216 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8425000000000001e-05 2024-09-18T04:39:37.216 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.216 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.216 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:37.216 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548316+0000", 2024-09-18T04:39:37.216 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1163e-05 2024-09-18T04:39:37.216 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.216 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.217 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:37.217 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548379+0000", 2024-09-18T04:39:37.217 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2780000000000005e-05 2024-09-18T04:39:37.217 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.217 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.217 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:37.217 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548399+0000", 2024-09-18T04:39:37.217 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0080000000000001e-05 2024-09-18T04:39:37.217 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.217 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.217 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:37.217 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562383+0000", 2024-09-18T04:39:37.217 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013984671000000001 2024-09-18T04:39:37.217 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.218 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.218 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:37.218 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562442+0000", 2024-09-18T04:39:37.218 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8700999999999999e-05 2024-09-18T04:39:37.218 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.218 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:37.218 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.218 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.218 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.218 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:37 2.5 e51/28 2:bf39b8e4:ns::rbd_data.133e813ab5bc.0000000000000003:head v 51'5, mlcod=51'2)", 2024-09-18T04:39:37.218 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:39:37.218 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 54.631355024999998, 2024-09-18T04:39:37.218 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014264492, 2024-09-18T04:39:37.218 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:37.218 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:37.219 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:37.219 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.219 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:37.219 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:39:37.219 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.219 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.219 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.219 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:37.219 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:39:37.219 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.219 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.219 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.219 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:37.219 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548531+0000", 2024-09-18T04:39:37.219 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:37.220 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.220 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.220 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:37.220 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551500+0000", 2024-09-18T04:39:37.220 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0029688129999999998 2024-09-18T04:39:37.220 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.220 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.220 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:37.220 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551516+0000", 2024-09-18T04:39:37.220 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5994e-05 2024-09-18T04:39:37.220 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.220 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.220 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:37.220 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551526+0000", 2024-09-18T04:39:37.221 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9299999999999998e-06 2024-09-18T04:39:37.221 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.221 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.221 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:37.221 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551577+0000", 2024-09-18T04:39:37.221 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0871000000000001e-05 2024-09-18T04:39:37.221 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.221 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.221 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:37.221 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551602+0000", 2024-09-18T04:39:37.221 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4813000000000001e-05 2024-09-18T04:39:37.221 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.221 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.222 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:37.222 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562741+0000", 2024-09-18T04:39:37.222 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011139428999999999 2024-09-18T04:39:37.222 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.222 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.222 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:37.222 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562797+0000", 2024-09-18T04:39:37.222 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6329000000000002e-05 2024-09-18T04:39:37.222 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.222 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:37.222 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.222 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.222 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.222 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:49 2.4 e51/17 2:20f0b0c9:ns::rbd_data.133e813ab5bc.0000000000000009:head v 51'6, mlcod=51'4)", 2024-09-18T04:39:37.223 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:39:37.223 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 54.599920466999997, 2024-09-18T04:39:37.223 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013849046, 2024-09-18T04:39:37.223 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:37.223 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:37.223 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:37.223 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.223 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:37.223 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:39:37.223 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.223 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.223 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.223 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:37.223 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:39:37.224 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.224 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.224 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:37.224 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579966+0000", 2024-09-18T04:39:37.224 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:37.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.224 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.224 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:37.224 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582324+0000", 2024-09-18T04:39:37.224 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023578890000000002 2024-09-18T04:39:37.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.224 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.224 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:37.225 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582355+0000", 2024-09-18T04:39:37.225 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1485000000000001e-05 2024-09-18T04:39:37.225 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.225 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.225 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:37.225 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582371+0000", 2024-09-18T04:39:37.225 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6147999999999999e-05 2024-09-18T04:39:37.225 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.225 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.225 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:37.225 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582467+0000", 2024-09-18T04:39:37.225 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6059999999999998e-05 2024-09-18T04:39:37.225 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.225 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.226 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:37.226 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582505+0000", 2024-09-18T04:39:37.226 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7679000000000003e-05 2024-09-18T04:39:37.226 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.226 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.226 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:37.226 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593743+0000", 2024-09-18T04:39:37.226 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011238356 2024-09-18T04:39:37.226 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.226 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.226 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:37.226 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593816+0000", 2024-09-18T04:39:37.226 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2935000000000006e-05 2024-09-18T04:39:37.226 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.227 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:37.227 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.227 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.227 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.227 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:53 2.4 e51/17 2:39f7f174:ns::rbd_data.133e813ab5bc.000000000000000b:head v 51'7, mlcod=51'4)", 2024-09-18T04:39:37.227 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:39:37.227 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 54.585219312, 2024-09-18T04:39:37.227 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0079914159999999995, 2024-09-18T04:39:37.227 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:37.227 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:37.227 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:37.227 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.227 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:37.227 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:39:37.227 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.228 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.228 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.228 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:37.228 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:39:37.228 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.228 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.228 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.228 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:37.228 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594667+0000", 2024-09-18T04:39:37.228 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:37.228 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.228 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.228 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:37.228 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596446+0000", 2024-09-18T04:39:37.228 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001779033 2024-09-18T04:39:37.229 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.229 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.229 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:37.229 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596472+0000", 2024-09-18T04:39:37.229 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6120000000000001e-05 2024-09-18T04:39:37.229 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.229 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.229 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:37.229 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596500+0000", 2024-09-18T04:39:37.229 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7518e-05 2024-09-18T04:39:37.229 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.229 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.229 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:37.229 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596564+0000", 2024-09-18T04:39:37.230 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3880999999999994e-05 2024-09-18T04:39:37.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.230 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.230 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:37.230 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596596+0000", 2024-09-18T04:39:37.230 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2638999999999997e-05 2024-09-18T04:39:37.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.230 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.230 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:37.230 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602605+0000", 2024-09-18T04:39:37.230 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0060080699999999999 2024-09-18T04:39:37.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.230 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.230 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:37.231 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602660+0000", 2024-09-18T04:39:37.231 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5396000000000003e-05 2024-09-18T04:39:37.231 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.231 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:37.231 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.231 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.231 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.231 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:60 2.5 e51/28 2:a06f51e4:ns::rbd_data.133e813ab5bc.000000000000000f:head v 51'6, mlcod=51'4)", 2024-09-18T04:39:37.231 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:39:37.231 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 54.570402326999996, 2024-09-18T04:39:37.231 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0093954989999999999, 2024-09-18T04:39:37.231 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:37.231 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:37.231 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:37.232 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.232 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:37.232 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:39:37.232 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.232 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.232 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.232 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:37.232 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:39:37.232 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.232 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.232 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.232 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:37.232 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609483+0000", 2024-09-18T04:39:37.232 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:37.233 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.233 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.233 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:37.233 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611767+0000", 2024-09-18T04:39:37.233 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002283724 2024-09-18T04:39:37.233 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.233 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.233 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:37.233 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611791+0000", 2024-09-18T04:39:37.233 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3547e-05 2024-09-18T04:39:37.233 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.233 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.233 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:37.233 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611800+0000", 2024-09-18T04:39:37.234 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4979999999999995e-06 2024-09-18T04:39:37.234 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.234 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.234 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:37.234 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611864+0000", 2024-09-18T04:39:37.234 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3974999999999996e-05 2024-09-18T04:39:37.234 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.234 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.234 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:37.234 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611887+0000", 2024-09-18T04:39:37.234 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2633000000000002e-05 2024-09-18T04:39:37.234 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.234 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.234 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:37.234 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.618777+0000", 2024-09-18T04:39:37.235 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0068897979999999999 2024-09-18T04:39:37.235 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.235 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.235 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:37.235 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.618881+0000", 2024-09-18T04:39:37.235 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010437699999999999 2024-09-18T04:39:37.235 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.235 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:37.235 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.235 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.235 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.235 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:663 2.4 e57/17 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 57'13, mlcod=57'11)", 2024-09-18T04:39:37.235 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.355300+0000", 2024-09-18T04:39:37.235 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 45.824587391999998, 2024-09-18T04:39:37.235 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0049192760000000002, 2024-09-18T04:39:37.236 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:37.236 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:37.236 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:37.236 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.236 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:37.236 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355300+0000", 2024-09-18T04:39:37.236 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.236 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.236 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.236 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:37.236 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355300+0000", 2024-09-18T04:39:37.236 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.236 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.236 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.236 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:37.237 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355299+0000", 2024-09-18T04:39:37.237 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:37.237 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.237 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.237 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:37.237 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355876+0000", 2024-09-18T04:39:37.237 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00057730900000000003 2024-09-18T04:39:37.237 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.237 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.237 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:37.237 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355897+0000", 2024-09-18T04:39:37.237 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0477000000000001e-05 2024-09-18T04:39:37.237 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.238 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.238 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:37.238 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355905+0000", 2024-09-18T04:39:37.238 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5739999999999996e-06 2024-09-18T04:39:37.238 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.238 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.238 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:37.238 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355989+0000", 2024-09-18T04:39:37.238 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4002e-05 2024-09-18T04:39:37.238 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.238 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.238 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:37.238 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.356041+0000", 2024-09-18T04:39:37.238 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1347000000000002e-05 2024-09-18T04:39:37.238 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.239 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.239 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:37.239 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.360172+0000", 2024-09-18T04:39:37.239 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0041315090000000002 2024-09-18T04:39:37.239 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.239 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.239 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:37.239 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.360220+0000", 2024-09-18T04:39:37.239 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7584999999999999e-05 2024-09-18T04:39:37.239 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.239 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:37.240 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.240 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.240 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.240 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:679 2.5 e57/28 2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head v 57'20, mlcod=57'18)", 2024-09-18T04:39:37.240 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.445316+0000", 2024-09-18T04:39:37.240 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 45.734571914999997, 2024-09-18T04:39:37.240 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0047016760000000001, 2024-09-18T04:39:37.240 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:37.240 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:37.240 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:37.240 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.240 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:37.240 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445316+0000", 2024-09-18T04:39:37.241 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.241 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.241 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.241 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:37.241 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445316+0000", 2024-09-18T04:39:37.241 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.241 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.241 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.241 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:37.241 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445314+0000", 2024-09-18T04:39:37.241 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:37.241 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.241 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.241 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:37.241 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445947+0000", 2024-09-18T04:39:37.242 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00063298900000000003 2024-09-18T04:39:37.242 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.242 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.242 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:37.242 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445968+0000", 2024-09-18T04:39:37.242 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.054e-05 2024-09-18T04:39:37.242 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.242 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.242 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:37.242 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445981+0000", 2024-09-18T04:39:37.243 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2697000000000001e-05 2024-09-18T04:39:37.243 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.243 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.243 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:37.243 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.446102+0000", 2024-09-18T04:39:37.243 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012189300000000001 2024-09-18T04:39:37.243 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.243 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.243 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:37.243 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.446151+0000", 2024-09-18T04:39:37.243 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8279000000000003e-05 2024-09-18T04:39:37.243 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.243 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.244 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:37.244 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.449953+0000", 2024-09-18T04:39:37.244 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003801979 2024-09-18T04:39:37.244 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.244 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.244 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:37.244 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.450018+0000", 2024-09-18T04:39:37.244 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4873e-05 2024-09-18T04:39:37.244 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.244 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:37.244 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.244 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.244 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.245 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:699 2.4 e57/17 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 57'17, mlcod=57'15)", 2024-09-18T04:39:37.245 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.566073+0000", 2024-09-18T04:39:37.245 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 45.613814402000003, 2024-09-18T04:39:37.245 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055398380000000001, 2024-09-18T04:39:37.245 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:37.245 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:37.245 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:37.245 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.245 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:37.245 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566073+0000", 2024-09-18T04:39:37.245 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.245 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.245 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.245 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:37.246 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566073+0000", 2024-09-18T04:39:37.246 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.246 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.246 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.246 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:37.246 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566071+0000", 2024-09-18T04:39:37.246 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:37.246 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.246 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.246 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:37.246 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566637+0000", 2024-09-18T04:39:37.246 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00056516699999999999 2024-09-18T04:39:37.246 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.246 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.246 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:37.247 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566668+0000", 2024-09-18T04:39:37.247 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1544999999999997e-05 2024-09-18T04:39:37.247 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.247 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.247 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:37.247 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566685+0000", 2024-09-18T04:39:37.247 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6857000000000001e-05 2024-09-18T04:39:37.247 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.247 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.247 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:37.247 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566816+0000", 2024-09-18T04:39:37.247 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013092799999999999 2024-09-18T04:39:37.247 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.247 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.248 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:37.248 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566875+0000", 2024-09-18T04:39:37.248 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9583999999999997e-05 2024-09-18T04:39:37.248 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.248 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.248 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:37.248 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.571567+0000", 2024-09-18T04:39:37.248 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.004691682 2024-09-18T04:39:37.248 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.248 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.248 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:37.248 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.571613+0000", 2024-09-18T04:39:37.248 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6125000000000003e-05 2024-09-18T04:39:37.249 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.249 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:37.249 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.249 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.249 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.249 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:98 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'27, mlcod=60'25)", 2024-09-18T04:39:37.249 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.847978+0000", 2024-09-18T04:39:37.249 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 36.331909842999998, 2024-09-18T04:39:37.249 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0044617709999999998, 2024-09-18T04:39:37.249 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:37.249 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:37.249 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:37.249 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.249 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:37.249 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.847978+0000", 2024-09-18T04:39:37.250 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.250 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.250 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.250 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:37.250 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.847978+0000", 2024-09-18T04:39:37.250 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.250 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.250 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.250 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:37.250 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.847977+0000", 2024-09-18T04:39:37.250 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:37.250 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849038+0000", 2024-09-18T04:39:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0010614820000000001 2024-09-18T04:39:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849065+0000", 2024-09-18T04:39:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7127999999999998e-05 2024-09-18T04:39:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849082+0000", 2024-09-18T04:39:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7170999999999999e-05 2024-09-18T04:39:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849188+0000", 2024-09-18T04:39:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010605800000000001 2024-09-18T04:39:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849233+0000", 2024-09-18T04:39:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4579000000000001e-05 2024-09-18T04:39:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:37.253 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.852370+0000", 2024-09-18T04:39:37.253 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00313751 2024-09-18T04:39:37.253 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.253 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.253 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:37.253 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.852440+0000", 2024-09-18T04:39:37.253 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9238000000000004e-05 2024-09-18T04:39:37.253 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.253 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:37.253 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.253 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.253 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.253 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:100 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'29, mlcod=64'27)", 2024-09-18T04:39:37.253 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.858766+0000", 2024-09-18T04:39:37.254 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 36.321121513999998, 2024-09-18T04:39:37.254 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0061964840000000004, 2024-09-18T04:39:37.254 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:37.254 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:37.254 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:37.254 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.254 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:37.254 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.858766+0000", 2024-09-18T04:39:37.254 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.254 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.254 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.254 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:37.254 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.858766+0000", 2024-09-18T04:39:37.254 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.254 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.255 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.255 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:37.255 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.858765+0000", 2024-09-18T04:39:37.255 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:37.255 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.255 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.255 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:37.255 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861612+0000", 2024-09-18T04:39:37.255 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028473169999999998 2024-09-18T04:39:37.255 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.255 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.255 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:37.255 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861643+0000", 2024-09-18T04:39:37.255 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1356000000000003e-05 2024-09-18T04:39:37.255 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.256 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.256 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:37.256 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861656+0000", 2024-09-18T04:39:37.256 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2899e-05 2024-09-18T04:39:37.256 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.256 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.256 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:37.256 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861773+0000", 2024-09-18T04:39:37.256 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011637999999999999 2024-09-18T04:39:37.256 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.256 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.256 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:37.256 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861808+0000", 2024-09-18T04:39:37.256 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5002999999999997e-05 2024-09-18T04:39:37.256 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.256 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.257 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:37.257 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.864876+0000", 2024-09-18T04:39:37.257 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030680719999999998 2024-09-18T04:39:37.257 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.257 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.257 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:37.257 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.864963+0000", 2024-09-18T04:39:37.257 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7114000000000007e-05 2024-09-18T04:39:37.257 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.257 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:37.257 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.257 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.257 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.258 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:102 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'31, mlcod=64'29)", 2024-09-18T04:39:37.258 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.872098+0000", 2024-09-18T04:39:37.258 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 36.307790257000001, 2024-09-18T04:39:37.258 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0064676450000000002, 2024-09-18T04:39:37.258 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:37.258 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:37.258 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:37.258 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.258 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:37.258 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.872098+0000", 2024-09-18T04:39:37.258 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.258 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.258 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.258 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:37.259 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.872098+0000", 2024-09-18T04:39:37.259 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.259 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.259 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.259 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:37.259 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.872096+0000", 2024-09-18T04:39:37.259 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:37.259 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.259 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.259 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:37.259 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875176+0000", 2024-09-18T04:39:37.259 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030799809999999999 2024-09-18T04:39:37.259 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.259 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875204+0000", 2024-09-18T04:39:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8025999999999999e-05 2024-09-18T04:39:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875216+0000", 2024-09-18T04:39:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2072999999999999e-05 2024-09-18T04:39:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875307+0000", 2024-09-18T04:39:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0373999999999999e-05 2024-09-18T04:39:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.261 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.261 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:37.261 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875349+0000", 2024-09-18T04:39:37.261 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2299000000000003e-05 2024-09-18T04:39:37.261 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.261 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.261 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:37.261 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.878496+0000", 2024-09-18T04:39:37.261 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0031468960000000002 2024-09-18T04:39:37.261 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.261 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.261 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:37.261 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.878565+0000", 2024-09-18T04:39:37.262 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9465e-05 2024-09-18T04:39:37.262 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.262 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:37.262 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.262 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.262 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.262 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:104 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'33, mlcod=64'31)", 2024-09-18T04:39:37.262 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.885100+0000", 2024-09-18T04:39:37.262 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 36.294787722000002, 2024-09-18T04:39:37.262 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0051888159999999997, 2024-09-18T04:39:37.262 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:37.262 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:37.262 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:37.263 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.263 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:37.263 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.885100+0000", 2024-09-18T04:39:37.263 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.263 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.263 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.263 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:37.263 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.885100+0000", 2024-09-18T04:39:37.263 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.263 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.263 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.263 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:37.263 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.885098+0000", 2024-09-18T04:39:37.263 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:37.264 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.264 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.264 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:37.264 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886809+0000", 2024-09-18T04:39:37.264 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017110459999999999 2024-09-18T04:39:37.264 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.264 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.264 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:37.264 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886842+0000", 2024-09-18T04:39:37.264 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2638999999999997e-05 2024-09-18T04:39:37.264 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.264 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.264 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:37.264 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886855+0000", 2024-09-18T04:39:37.264 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3380000000000001e-05 2024-09-18T04:39:37.265 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.265 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.265 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:37.265 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886939+0000", 2024-09-18T04:39:37.265 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3733999999999995e-05 2024-09-18T04:39:37.265 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.265 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.265 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:37.265 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886981+0000", 2024-09-18T04:39:37.265 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1956000000000003e-05 2024-09-18T04:39:37.265 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.265 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.265 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:37.265 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.890201+0000", 2024-09-18T04:39:37.266 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003219927 2024-09-18T04:39:37.266 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.266 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.266 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:37.266 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.890289+0000", 2024-09-18T04:39:37.266 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7997999999999993e-05 2024-09-18T04:39:37.266 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.266 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:37.266 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.266 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.266 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.266 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:145 2.4 e64/62 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 64'37, mlcod=64'35)", 2024-09-18T04:39:37.266 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:01.099398+0000", 2024-09-18T04:39:37.266 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 36.080490234000003, 2024-09-18T04:39:37.266 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0050078459999999998, 2024-09-18T04:39:37.267 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:37.267 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:37.267 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:37.267 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.267 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:37.267 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.099398+0000", 2024-09-18T04:39:37.267 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.267 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.267 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.267 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:37.267 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.099398+0000", 2024-09-18T04:39:37.267 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.267 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.267 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.267 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:37.268 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.099396+0000", 2024-09-18T04:39:37.268 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:37.268 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.268 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.268 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:37.268 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100199+0000", 2024-09-18T04:39:37.268 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00080338000000000002 2024-09-18T04:39:37.268 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.268 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.268 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:37.268 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100230+0000", 2024-09-18T04:39:37.268 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0716000000000002e-05 2024-09-18T04:39:37.268 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.268 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.269 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:37.269 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100243+0000", 2024-09-18T04:39:37.269 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3359e-05 2024-09-18T04:39:37.269 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.269 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.269 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:37.269 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100337+0000", 2024-09-18T04:39:37.269 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4387000000000003e-05 2024-09-18T04:39:37.269 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.269 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.269 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:37.269 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100396+0000", 2024-09-18T04:39:37.269 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8164000000000001e-05 2024-09-18T04:39:37.269 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.270 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.270 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:37.270 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.104309+0000", 2024-09-18T04:39:37.270 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0039137800000000004 2024-09-18T04:39:37.270 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.270 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.270 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:37.270 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.104405+0000", 2024-09-18T04:39:37.270 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6039999999999995e-05 2024-09-18T04:39:37.270 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.270 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:37.270 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.270 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.270 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.270 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:147 2.4 e64/62 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 64'39, mlcod=64'37)", 2024-09-18T04:39:37.270 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:01.110658+0000", 2024-09-18T04:39:37.271 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 36.069230144999999, 2024-09-18T04:39:37.271 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0046316910000000003, 2024-09-18T04:39:37.271 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:37.271 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:37.271 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:37.271 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.271 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:37.271 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.110658+0000", 2024-09-18T04:39:37.271 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.271 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.271 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.271 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:37.271 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.110658+0000", 2024-09-18T04:39:37.271 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.271 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.272 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.272 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:37.272 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.110656+0000", 2024-09-18T04:39:37.272 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:37.272 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.272 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.272 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:37.272 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111444+0000", 2024-09-18T04:39:37.272 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00078762400000000001 2024-09-18T04:39:37.272 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.272 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.272 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:37.272 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111474+0000", 2024-09-18T04:39:37.272 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0085000000000001e-05 2024-09-18T04:39:37.272 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.273 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.273 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:37.273 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111486+0000", 2024-09-18T04:39:37.273 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2645e-05 2024-09-18T04:39:37.273 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.273 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.273 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:37.273 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111581+0000", 2024-09-18T04:39:37.273 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4432000000000007e-05 2024-09-18T04:39:37.273 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.273 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.273 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:37.273 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111631+0000", 2024-09-18T04:39:37.273 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9901000000000002e-05 2024-09-18T04:39:37.274 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.274 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.274 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:37.274 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.115209+0000", 2024-09-18T04:39:37.274 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003577916 2024-09-18T04:39:37.274 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.274 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.274 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:37.274 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.115289+0000", 2024-09-18T04:39:37.274 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0704999999999994e-05 2024-09-18T04:39:37.274 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.274 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:37.274 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.274 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.274 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:37.275 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:37.276 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:39:37.276 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:37.268+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:37.377 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:37.372+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:37.389 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:37.390 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:37.390 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:39:37.390 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:37.400 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:39:37.477 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:37.472+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:37.502 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:37.502 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:37.503 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:39:37.503 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:39:37.503 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:37.512 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:39:37.579 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:37.576+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:37.620 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:37.621 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:39:37.621 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:39:37.621 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:39:37.621 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.621 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:33 2.1 e51/24 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head v 51'4, mlcod=51'2)", 2024-09-18T04:39:37.621 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:39:37.621 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 55.338181226000003, 2024-09-18T04:39:37.621 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018702106, 2024-09-18T04:39:37.621 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:37.621 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:37.621 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:37.621 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.621 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:37.621 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:39:37.622 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.622 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.622 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:37.622 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:39:37.622 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.622 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.622 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:37.622 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279197+0000", 2024-09-18T04:39:37.622 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:37.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.622 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.623 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:37.623 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282682+0000", 2024-09-18T04:39:37.623 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0034848689999999998 2024-09-18T04:39:37.623 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.623 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.623 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:37.623 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282709+0000", 2024-09-18T04:39:37.623 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7599000000000001e-05 2024-09-18T04:39:37.623 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.623 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.623 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:37.623 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282720+0000", 2024-09-18T04:39:37.623 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.092e-05 2024-09-18T04:39:37.623 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.624 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.624 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:37.624 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282791+0000", 2024-09-18T04:39:37.624 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1268999999999998e-05 2024-09-18T04:39:37.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.624 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.624 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:37.624 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282843+0000", 2024-09-18T04:39:37.624 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1771999999999998e-05 2024-09-18T04:39:37.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.624 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.624 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:37.624 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.297830+0000", 2024-09-18T04:39:37.624 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014987228999999999 2024-09-18T04:39:37.625 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.625 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.625 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:37.625 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.297901+0000", 2024-09-18T04:39:37.625 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0209999999999994e-05 2024-09-18T04:39:37.625 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.625 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:37.625 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.625 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.625 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.625 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:38 2.0 e51/17 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head v 51'10, mlcod=51'8)", 2024-09-18T04:39:37.625 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:39:37.625 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 55.336762796000002, 2024-09-18T04:39:37.626 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.023984472, 2024-09-18T04:39:37.626 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:37.626 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:37.626 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:37.626 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.626 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:37.626 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:39:37.626 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.626 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.626 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.626 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:37.626 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:39:37.626 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.626 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.627 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.627 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:37.627 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280614+0000", 2024-09-18T04:39:37.627 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:37.627 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.627 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.627 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:37.627 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293094+0000", 2024-09-18T04:39:37.627 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012479571 2024-09-18T04:39:37.627 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.627 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.627 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:37.627 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293111+0000", 2024-09-18T04:39:37.627 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7354999999999999e-05 2024-09-18T04:39:37.628 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.628 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.628 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:37.628 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293121+0000", 2024-09-18T04:39:37.628 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6979999999999993e-06 2024-09-18T04:39:37.628 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.628 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.628 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:37.628 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293180+0000", 2024-09-18T04:39:37.628 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9333000000000003e-05 2024-09-18T04:39:37.628 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.628 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.628 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:37.628 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293203+0000", 2024-09-18T04:39:37.629 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2633000000000002e-05 2024-09-18T04:39:37.629 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.629 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.629 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:37.629 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304534+0000", 2024-09-18T04:39:37.629 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011330761 2024-09-18T04:39:37.629 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.629 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.629 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:37.629 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304601+0000", 2024-09-18T04:39:37.629 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7657000000000006e-05 2024-09-18T04:39:37.629 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.629 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:37.629 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.630 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.630 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.630 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:46 2.6 2:7ba25245:::rbd_data.125fc09df6d1.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:37.630 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:39:37.630 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 55.309219996000003, 2024-09-18T04:39:37.630 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.035604595000000003, 2024-09-18T04:39:37.630 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:37.630 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:37.630 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:37.630 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:37.630 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:37.630 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 46 2024-09-18T04:39:37.630 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.630 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:37.631 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.631 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:37.631 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:39:37.631 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.631 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.631 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:37.631 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:39:37.631 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.631 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.631 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:37.631 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308158+0000", 2024-09-18T04:39:37.632 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:37.632 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.632 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.632 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:37.632 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313191+0000", 2024-09-18T04:39:37.632 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0050323929999999996 2024-09-18T04:39:37.632 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.632 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.632 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:37.632 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313210+0000", 2024-09-18T04:39:37.632 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9673e-05 2024-09-18T04:39:37.632 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.632 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.632 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:37.633 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313221+0000", 2024-09-18T04:39:37.633 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0444e-05 2024-09-18T04:39:37.633 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.633 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.633 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:37.633 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313294+0000", 2024-09-18T04:39:37.633 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2650000000000004e-05 2024-09-18T04:39:37.633 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.633 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.633 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:37.633 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313493+0000", 2024-09-18T04:39:37.633 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019968300000000001 2024-09-18T04:39:37.633 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.633 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.634 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:37.634 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313666+0000", 2024-09-18T04:39:37.634 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00017248599999999999 2024-09-18T04:39:37.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.634 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.634 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:37.634 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.323175+0000", 2024-09-18T04:39:37.634 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0095088819999999998 2024-09-18T04:39:37.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.634 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.634 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:37.634 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343651+0000", 2024-09-18T04:39:37.634 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020475892999999998 2024-09-18T04:39:37.635 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.635 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.635 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:37.635 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343715+0000", 2024-09-18T04:39:37.635 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4443999999999995e-05 2024-09-18T04:39:37.635 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.635 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.635 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:37.635 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343764+0000", 2024-09-18T04:39:37.635 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9280000000000003e-05 2024-09-18T04:39:37.635 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.635 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:37.635 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.635 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.636 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.636 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:47 2.4 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:37.636 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:39:37.636 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 55.304137959000002, 2024-09-18T04:39:37.636 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.032570693999999997, 2024-09-18T04:39:37.636 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:37.636 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:37.636 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:37.636 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:37.636 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:37.636 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 47 2024-09-18T04:39:37.636 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.636 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:37.636 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.637 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:37.637 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:39:37.637 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.637 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.637 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:37.637 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:39:37.637 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.637 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.637 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:37.637 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313240+0000", 2024-09-18T04:39:37.637 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:37.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.638 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.638 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:37.638 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325825+0000", 2024-09-18T04:39:37.638 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012584425999999999 2024-09-18T04:39:37.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.638 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.638 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:37.638 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325848+0000", 2024-09-18T04:39:37.638 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3025999999999999e-05 2024-09-18T04:39:37.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.638 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.638 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:37.638 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325863+0000", 2024-09-18T04:39:37.639 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5152e-05 2024-09-18T04:39:37.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.639 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.639 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:37.639 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325938+0000", 2024-09-18T04:39:37.639 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4982999999999997e-05 2024-09-18T04:39:37.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.639 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.639 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:37.639 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326090+0000", 2024-09-18T04:39:37.639 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015207900000000001 2024-09-18T04:39:37.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.639 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.639 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:37.640 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326242+0000", 2024-09-18T04:39:37.640 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015163500000000001 2024-09-18T04:39:37.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.640 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.640 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:37.640 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343015+0000", 2024-09-18T04:39:37.640 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016773414 2024-09-18T04:39:37.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.640 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.640 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:37.640 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345664+0000", 2024-09-18T04:39:37.640 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026487870000000001 2024-09-18T04:39:37.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.640 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.641 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:37.641 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345739+0000", 2024-09-18T04:39:37.641 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5018999999999994e-05 2024-09-18T04:39:37.641 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.641 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.641 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:37.641 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345812+0000", 2024-09-18T04:39:37.641 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3547000000000006e-05 2024-09-18T04:39:37.641 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.641 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:37.641 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.641 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.641 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.641 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:48 2.6 2:61db7a3a:::rbd_data.125fc09df6d1.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:37.642 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:39:37.642 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 55.291481287000003, 2024-09-18T04:39:37.642 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.025064388, 2024-09-18T04:39:37.642 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:37.642 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:37.642 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:37.642 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:37.642 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:37.642 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 48 2024-09-18T04:39:37.642 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.642 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:37.642 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.642 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:37.642 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:39:37.643 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.643 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.643 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.643 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:37.643 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:39:37.643 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.643 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.643 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.643 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:37.643 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325897+0000", 2024-09-18T04:39:37.643 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:37.643 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.643 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.643 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:37.644 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333697+0000", 2024-09-18T04:39:37.644 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0078002649999999998 2024-09-18T04:39:37.644 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.644 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.644 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:37.644 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333716+0000", 2024-09-18T04:39:37.644 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8910999999999999e-05 2024-09-18T04:39:37.644 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.644 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.644 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:37.644 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333728+0000", 2024-09-18T04:39:37.644 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1936000000000001e-05 2024-09-18T04:39:37.644 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.644 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.645 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:37.645 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333792+0000", 2024-09-18T04:39:37.645 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4010999999999994e-05 2024-09-18T04:39:37.645 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.645 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.645 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:37.645 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.334082+0000", 2024-09-18T04:39:37.645 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00028955799999999998 2024-09-18T04:39:37.645 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.645 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.645 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:37.645 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.334326+0000", 2024-09-18T04:39:37.645 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00024399099999999999 2024-09-18T04:39:37.646 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.646 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.646 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:37.646 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350745+0000", 2024-09-18T04:39:37.646 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016419474999999999 2024-09-18T04:39:37.646 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.646 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.646 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:37.646 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350808+0000", 2024-09-18T04:39:37.646 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3244999999999995e-05 2024-09-18T04:39:37.646 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.646 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.646 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:37.646 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350892+0000", 2024-09-18T04:39:37.647 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4074999999999997e-05 2024-09-18T04:39:37.647 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.647 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.647 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:37.647 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350963+0000", 2024-09-18T04:39:37.647 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0412000000000003e-05 2024-09-18T04:39:37.647 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.647 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:37.647 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.647 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.647 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.647 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:51 2.6 2:775d7b11:::rbd_data.125fc09df6d1.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:37.647 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:39:37.647 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 55.283629151, 2024-09-18T04:39:37.648 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.023824221, 2024-09-18T04:39:37.648 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:37.648 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:37.648 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:37.648 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:37.648 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:37.648 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 51 2024-09-18T04:39:37.648 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.648 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:37.648 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.648 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:37.648 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:39:37.648 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.649 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.649 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.649 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:37.649 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:39:37.649 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.649 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.649 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.649 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:37.649 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333749+0000", 2024-09-18T04:39:37.649 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:37.649 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.649 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.649 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:37.649 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340899+0000", 2024-09-18T04:39:37.650 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0071500340000000004 2024-09-18T04:39:37.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.650 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.650 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:37.650 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340929+0000", 2024-09-18T04:39:37.650 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0093000000000001e-05 2024-09-18T04:39:37.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.650 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.650 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:37.650 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340943+0000", 2024-09-18T04:39:37.650 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4019000000000001e-05 2024-09-18T04:39:37.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.650 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.650 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:37.650 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343834+0000", 2024-09-18T04:39:37.651 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028918120000000001 2024-09-18T04:39:37.651 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.651 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.651 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:37.651 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344020+0000", 2024-09-18T04:39:37.651 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000185892 2024-09-18T04:39:37.651 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.651 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.651 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:37.651 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344190+0000", 2024-09-18T04:39:37.651 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016946800000000001 2024-09-18T04:39:37.651 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.651 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.651 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:37.652 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.351291+0000", 2024-09-18T04:39:37.652 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.007100805 2024-09-18T04:39:37.652 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.652 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.652 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:37.652 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357470+0000", 2024-09-18T04:39:37.652 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0061797570000000001 2024-09-18T04:39:37.652 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.652 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.652 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:37.652 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357533+0000", 2024-09-18T04:39:37.652 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2906999999999999e-05 2024-09-18T04:39:37.652 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.652 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.653 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:37.653 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357575+0000", 2024-09-18T04:39:37.653 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1403999999999999e-05 2024-09-18T04:39:37.653 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.653 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:37.653 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.653 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.653 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.653 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:56 2.4 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:37.653 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:39:37.653 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 55.267584057999997, 2024-09-18T04:39:37.653 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.034295870999999999, 2024-09-18T04:39:37.653 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:37.653 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:37.654 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:37.654 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:37.654 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:37.654 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 56 2024-09-18T04:39:37.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.654 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:37.654 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.654 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:37.654 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:39:37.654 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.654 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.654 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:37.654 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:39:37.655 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.655 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.655 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:37.655 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349794+0000", 2024-09-18T04:39:37.655 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:37.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.655 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.655 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:37.655 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366632+0000", 2024-09-18T04:39:37.655 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01683755 2024-09-18T04:39:37.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.655 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.655 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:37.656 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366669+0000", 2024-09-18T04:39:37.656 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7351000000000002e-05 2024-09-18T04:39:37.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.656 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.656 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:37.656 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366686+0000", 2024-09-18T04:39:37.656 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6507999999999998e-05 2024-09-18T04:39:37.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.656 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.656 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:37.656 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366962+0000", 2024-09-18T04:39:37.656 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00027655900000000001 2024-09-18T04:39:37.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.656 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.657 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:37.657 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.367452+0000", 2024-09-18T04:39:37.657 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00049025299999999996 2024-09-18T04:39:37.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.657 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.657 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:37.657 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368113+0000", 2024-09-18T04:39:37.657 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00066093399999999996 2024-09-18T04:39:37.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.657 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.657 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:37.657 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383719+0000", 2024-09-18T04:39:37.657 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015606106999999999 2024-09-18T04:39:37.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.658 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.658 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:37.658 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384035+0000", 2024-09-18T04:39:37.658 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00031590399999999999 2024-09-18T04:39:37.658 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.658 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.658 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:37.658 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384067+0000", 2024-09-18T04:39:37.658 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.129e-05 2024-09-18T04:39:37.658 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.658 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.658 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:37.658 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384091+0000", 2024-09-18T04:39:37.658 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4788000000000001e-05 2024-09-18T04:39:37.658 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.659 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:37.659 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.659 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.659 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:57 2.0 e51/17 2:0cdd070e:::rbd_data.125fc09df6d1.000000000000000d:head v 51'21, mlcod=51'19)", 2024-09-18T04:39:37.659 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:39:37.659 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 55.259756570999997, 2024-09-18T04:39:37.659 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.025216311000000002, 2024-09-18T04:39:37.659 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:37.659 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:37.659 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:37.659 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.659 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:37.659 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:39:37.660 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.660 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.660 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.660 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:37.660 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:39:37.660 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.660 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.660 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.660 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:37.660 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:39:37.660 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:37.660 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.660 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.661 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:37.661 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363844+0000", 2024-09-18T04:39:37.661 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0062218450000000002 2024-09-18T04:39:37.661 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.661 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.661 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:37.661 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363879+0000", 2024-09-18T04:39:37.661 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4382999999999999e-05 2024-09-18T04:39:37.661 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.661 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.661 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:37.661 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363893+0000", 2024-09-18T04:39:37.661 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4348999999999999e-05 2024-09-18T04:39:37.661 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.662 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.662 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:37.662 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363977+0000", 2024-09-18T04:39:37.662 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3499999999999997e-05 2024-09-18T04:39:37.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.662 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.662 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:37.662 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364009+0000", 2024-09-18T04:39:37.662 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2292000000000002e-05 2024-09-18T04:39:37.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.662 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.662 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:37.662 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.382781+0000", 2024-09-18T04:39:37.662 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018771626 2024-09-18T04:39:37.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.663 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.663 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:37.663 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.382839+0000", 2024-09-18T04:39:37.663 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8758000000000002e-05 2024-09-18T04:39:37.663 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.663 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:37.663 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.663 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.663 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.663 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:59 2.0 e51/17 2:19c52cde:::rbd_data.125fc09df6d1.000000000000000f:head v 51'22, mlcod=51'19)", 2024-09-18T04:39:37.663 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:39:37.663 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 55.253096644999999, 2024-09-18T04:39:37.663 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.019497112, 2024-09-18T04:39:37.663 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:37.664 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:37.664 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:37.664 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.664 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:37.664 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:39:37.664 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.664 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.664 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.664 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:37.664 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:39:37.664 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.664 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.664 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.664 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:37.665 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364282+0000", 2024-09-18T04:39:37.665 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:37.665 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.665 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.665 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:37.665 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371823+0000", 2024-09-18T04:39:37.665 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0075411829999999999 2024-09-18T04:39:37.665 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.665 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.665 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:37.665 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371837+0000", 2024-09-18T04:39:37.665 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4336e-05 2024-09-18T04:39:37.665 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.665 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.666 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:37.666 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371846+0000", 2024-09-18T04:39:37.666 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9770000000000006e-06 2024-09-18T04:39:37.666 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.666 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.666 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:37.666 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.373566+0000", 2024-09-18T04:39:37.666 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017192539999999999 2024-09-18T04:39:37.666 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.666 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.666 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:37.666 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.373595+0000", 2024-09-18T04:39:37.666 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8975999999999998e-05 2024-09-18T04:39:37.666 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.667 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.667 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:37.667 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383740+0000", 2024-09-18T04:39:37.667 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01014525 2024-09-18T04:39:37.667 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.667 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.667 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:37.667 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383780+0000", 2024-09-18T04:39:37.667 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0275999999999999e-05 2024-09-18T04:39:37.667 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.667 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:37.667 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.667 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.667 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.667 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:34 2.6 2:66289171:ns::rbd_data.133e813ab5bc.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:37.668 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.524015+0000", 2024-09-18T04:39:37.668 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 55.093365036000002, 2024-09-18T04:39:37.668 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027582111999999999, 2024-09-18T04:39:37.668 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:37.668 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:37.668 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:37.668 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:37.668 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:37.668 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 34 2024-09-18T04:39:37.668 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.668 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:37.668 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.668 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:37.668 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524015+0000", 2024-09-18T04:39:37.669 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.669 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.669 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.669 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:37.669 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524015+0000", 2024-09-18T04:39:37.669 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.669 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.669 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.669 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:37.669 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524012+0000", 2024-09-18T04:39:37.669 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:37.669 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.669 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.669 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:37.670 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534956+0000", 2024-09-18T04:39:37.670 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010943403000000001 2024-09-18T04:39:37.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.670 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.670 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:37.670 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534978+0000", 2024-09-18T04:39:37.670 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1965999999999999e-05 2024-09-18T04:39:37.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.670 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.670 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:37.670 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534990+0000", 2024-09-18T04:39:37.670 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2065999999999999e-05 2024-09-18T04:39:37.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.670 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.671 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:37.671 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535065+0000", 2024-09-18T04:39:37.671 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5523000000000002e-05 2024-09-18T04:39:37.671 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.671 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.671 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:37.671 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535208+0000", 2024-09-18T04:39:37.671 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014309500000000001 2024-09-18T04:39:37.671 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.671 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.671 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:37.671 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535343+0000", 2024-09-18T04:39:37.671 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013437400000000001 2024-09-18T04:39:37.671 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.672 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.672 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:37.672 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.546576+0000", 2024-09-18T04:39:37.672 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011233425999999999 2024-09-18T04:39:37.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.672 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.672 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:37.672 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551498+0000", 2024-09-18T04:39:37.672 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0049219349999999997 2024-09-18T04:39:37.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.672 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.672 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:37.672 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551554+0000", 2024-09-18T04:39:37.672 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6400999999999998e-05 2024-09-18T04:39:37.673 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.673 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.673 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:37.673 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551597+0000", 2024-09-18T04:39:37.673 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2283000000000002e-05 2024-09-18T04:39:37.673 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.673 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:37.673 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.673 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.673 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.673 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:33 2.1 e51/24 2:8b40f1c5:ns::rbd_data.133e813ab5bc.0000000000000000:head v 51'7, mlcod=51'5)", 2024-09-18T04:39:37.673 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.531767+0000", 2024-09-18T04:39:37.673 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 55.085613119000001, 2024-09-18T04:39:37.673 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015370295000000001, 2024-09-18T04:39:37.674 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:37.674 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:37.674 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:37.674 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.674 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:37.674 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531767+0000", 2024-09-18T04:39:37.674 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.674 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.674 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:37.674 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531767+0000", 2024-09-18T04:39:37.674 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.674 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.675 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:37.675 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531765+0000", 2024-09-18T04:39:37.675 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:37.675 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.675 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.675 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:37.675 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534258+0000", 2024-09-18T04:39:37.675 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002493561 2024-09-18T04:39:37.675 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.675 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.675 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:37.675 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534288+0000", 2024-09-18T04:39:37.675 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.976e-05 2024-09-18T04:39:37.676 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.676 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.676 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:37.676 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534300+0000", 2024-09-18T04:39:37.676 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1928e-05 2024-09-18T04:39:37.676 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.676 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.676 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:37.676 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534367+0000", 2024-09-18T04:39:37.676 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7045999999999995e-05 2024-09-18T04:39:37.676 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.676 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.676 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:37.676 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534394+0000", 2024-09-18T04:39:37.677 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6903999999999999e-05 2024-09-18T04:39:37.677 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.677 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.677 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:37.677 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.547073+0000", 2024-09-18T04:39:37.677 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012679365 2024-09-18T04:39:37.677 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.677 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.677 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:37.677 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.547137+0000", 2024-09-18T04:39:37.677 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3691999999999999e-05 2024-09-18T04:39:37.677 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.677 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:37.677 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.678 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.678 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.678 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:39 2.4 2:304364d1:ns::rbd_data.133e813ab5bc.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:37.678 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.535013+0000", 2024-09-18T04:39:37.678 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 55.082366184000001, 2024-09-18T04:39:37.678 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027764487000000001, 2024-09-18T04:39:37.678 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:37.678 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:37.678 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:37.678 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:37.678 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:37.678 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 39 2024-09-18T04:39:37.678 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.678 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:37.679 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.679 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:37.679 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535013+0000", 2024-09-18T04:39:37.679 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.679 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.679 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:37.679 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535013+0000", 2024-09-18T04:39:37.679 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.679 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.679 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:37.679 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535012+0000", 2024-09-18T04:39:37.680 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:37.680 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.680 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.680 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:37.680 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545016+0000", 2024-09-18T04:39:37.680 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01000322 2024-09-18T04:39:37.680 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.680 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.680 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:37.680 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545035+0000", 2024-09-18T04:39:37.680 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9845999999999999e-05 2024-09-18T04:39:37.680 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.680 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.680 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:37.681 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545045+0000", 2024-09-18T04:39:37.681 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3640000000000005e-06 2024-09-18T04:39:37.681 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.681 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.681 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:37.681 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545152+0000", 2024-09-18T04:39:37.681 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000107025 2024-09-18T04:39:37.681 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.681 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.681 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:37.681 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545338+0000", 2024-09-18T04:39:37.681 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018643500000000001 2024-09-18T04:39:37.681 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.681 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.681 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:37.682 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545492+0000", 2024-09-18T04:39:37.682 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015393199999999999 2024-09-18T04:39:37.682 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.682 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.682 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:37.682 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.553549+0000", 2024-09-18T04:39:37.682 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0080569170000000002 2024-09-18T04:39:37.682 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.682 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.682 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:37.682 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562693+0000", 2024-09-18T04:39:37.682 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0091441609999999996 2024-09-18T04:39:37.682 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.682 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.683 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:37.683 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562748+0000", 2024-09-18T04:39:37.683 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5047000000000003e-05 2024-09-18T04:39:37.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.683 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.683 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:37.683 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562778+0000", 2024-09-18T04:39:37.683 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9652e-05 2024-09-18T04:39:37.683 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.683 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:37.683 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.683 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.683 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:40 2.6 2:784d6cd1:ns::rbd_data.133e813ab5bc.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:37.684 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.545062+0000", 2024-09-18T04:39:37.684 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 55.072317132999999, 2024-09-18T04:39:37.684 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.032771623, 2024-09-18T04:39:37.684 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:37.684 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:37.684 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:37.684 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:37.684 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:37.684 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 40 2024-09-18T04:39:37.684 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.684 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:37.684 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.684 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:37.684 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545062+0000", 2024-09-18T04:39:37.685 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.685 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.685 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.685 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:37.685 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545062+0000", 2024-09-18T04:39:37.685 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.685 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.685 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.685 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:37.685 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545061+0000", 2024-09-18T04:39:37.685 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:37.685 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.685 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.685 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:37.686 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552477+0000", 2024-09-18T04:39:37.686 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.007415914 2024-09-18T04:39:37.686 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.686 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.686 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:37.686 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552500+0000", 2024-09-18T04:39:37.686 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3546000000000001e-05 2024-09-18T04:39:37.686 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.686 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.686 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:37.686 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552515+0000", 2024-09-18T04:39:37.686 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4606999999999999e-05 2024-09-18T04:39:37.686 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.686 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.686 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:37.687 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552661+0000", 2024-09-18T04:39:37.687 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000146007 2024-09-18T04:39:37.687 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.687 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.687 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:37.687 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552876+0000", 2024-09-18T04:39:37.687 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000214618 2024-09-18T04:39:37.687 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.687 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.687 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:37.687 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.553061+0000", 2024-09-18T04:39:37.687 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018564100000000001 2024-09-18T04:39:37.687 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.687 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.688 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:37.688 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.565612+0000", 2024-09-18T04:39:37.688 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012550301999999999 2024-09-18T04:39:37.688 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.688 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.688 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:37.688 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577695+0000", 2024-09-18T04:39:37.688 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012083493000000001 2024-09-18T04:39:37.688 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.688 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.688 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:37.688 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577772+0000", 2024-09-18T04:39:37.688 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7158999999999998e-05 2024-09-18T04:39:37.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.689 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.689 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:37.689 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577834+0000", 2024-09-18T04:39:37.689 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1851000000000001e-05 2024-09-18T04:39:37.689 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.689 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:37.689 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.689 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.689 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:43 2.6 2:759f27b6:ns::rbd_data.133e813ab5bc.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:37.689 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.555868+0000", 2024-09-18T04:39:37.689 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 55.061511639000003, 2024-09-18T04:39:37.689 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.022332497999999999, 2024-09-18T04:39:37.689 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:37.690 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:37.690 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:37.690 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:37.690 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:37.690 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 43 2024-09-18T04:39:37.690 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.690 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:37.690 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.690 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:37.690 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555868+0000", 2024-09-18T04:39:37.690 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.690 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.690 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.690 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:37.691 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555868+0000", 2024-09-18T04:39:37.691 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.691 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.691 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:37.691 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555865+0000", 2024-09-18T04:39:37.691 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999971 2024-09-18T04:39:37.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.691 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.691 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:37.691 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561031+0000", 2024-09-18T04:39:37.691 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0051658770000000001 2024-09-18T04:39:37.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.691 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.692 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:37.692 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561048+0000", 2024-09-18T04:39:37.692 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6569e-05 2024-09-18T04:39:37.692 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.692 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.692 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:37.692 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561055+0000", 2024-09-18T04:39:37.692 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.328e-06 2024-09-18T04:39:37.692 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.692 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.692 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:37.692 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561114+0000", 2024-09-18T04:39:37.692 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9540000000000003e-05 2024-09-18T04:39:37.693 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.693 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.693 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:37.693 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561314+0000", 2024-09-18T04:39:37.693 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019965799999999999 2024-09-18T04:39:37.693 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.693 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.693 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:37.693 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561473+0000", 2024-09-18T04:39:37.693 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000158904 2024-09-18T04:39:37.693 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.693 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.693 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:37.693 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.567015+0000", 2024-09-18T04:39:37.694 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055420859999999999 2024-09-18T04:39:37.694 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.694 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.694 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:37.694 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577982+0000", 2024-09-18T04:39:37.694 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010966872000000001 2024-09-18T04:39:37.694 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.694 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.694 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:37.694 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578162+0000", 2024-09-18T04:39:37.694 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018050999999999999 2024-09-18T04:39:37.694 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.694 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.694 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:37.695 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578200+0000", 2024-09-18T04:39:37.695 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8000999999999998e-05 2024-09-18T04:39:37.695 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.695 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:37.695 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.695 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.695 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:48 2.6 2:7c225365:ns::rbd_data.133e813ab5bc.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:37.695 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:39:37.695 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 55.053712416000003, 2024-09-18T04:39:37.695 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.024576377, 2024-09-18T04:39:37.695 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:37.695 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:37.695 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:37.696 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:37.696 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:37.696 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 48 2024-09-18T04:39:37.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.696 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:37.696 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.696 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:37.696 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:39:37.696 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.696 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.696 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:37.696 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:39:37.696 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.697 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.697 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:37.697 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.563666+0000", 2024-09-18T04:39:37.697 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:37.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.697 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.697 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:37.697 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570331+0000", 2024-09-18T04:39:37.697 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0066653329999999998 2024-09-18T04:39:37.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.697 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.697 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:37.697 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570348+0000", 2024-09-18T04:39:37.698 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6507e-05 2024-09-18T04:39:37.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.698 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.698 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:37.698 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570355+0000", 2024-09-18T04:39:37.698 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2749999999999998e-06 2024-09-18T04:39:37.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.698 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.698 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:37.698 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570396+0000", 2024-09-18T04:39:37.698 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0809000000000002e-05 2024-09-18T04:39:37.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.698 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.699 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:37.699 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570571+0000", 2024-09-18T04:39:37.699 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000174823 2024-09-18T04:39:37.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.699 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.699 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:37.699 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570722+0000", 2024-09-18T04:39:37.699 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015154599999999999 2024-09-18T04:39:37.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.699 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.699 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:37.699 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579756+0000", 2024-09-18T04:39:37.699 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0090337000000000004 2024-09-18T04:39:37.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.700 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.700 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:37.700 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588093+0000", 2024-09-18T04:39:37.700 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.008336553 2024-09-18T04:39:37.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.700 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.700 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:37.700 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588173+0000", 2024-09-18T04:39:37.700 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0333999999999994e-05 2024-09-18T04:39:37.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.700 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.700 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:37.700 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588244+0000", 2024-09-18T04:39:37.701 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0573000000000003e-05 2024-09-18T04:39:37.701 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.701 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:37.701 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.701 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.701 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:49 2.4 2:20f0b0c9:ns::rbd_data.133e813ab5bc.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:37.701 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.570370+0000", 2024-09-18T04:39:37.701 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 55.047010088999997, 2024-09-18T04:39:37.701 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.029633981, 2024-09-18T04:39:37.701 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:37.701 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:37.701 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:37.701 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:37.701 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:37.702 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 49 2024-09-18T04:39:37.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.702 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:37.702 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.702 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:37.702 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570370+0000", 2024-09-18T04:39:37.702 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.702 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.702 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:37.702 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570370+0000", 2024-09-18T04:39:37.702 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.703 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.703 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:37.703 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570369+0000", 2024-09-18T04:39:37.703 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:37.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.703 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.703 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:37.703 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579090+0000", 2024-09-18T04:39:37.703 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0087212339999999996 2024-09-18T04:39:37.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.703 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.703 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:37.703 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579109+0000", 2024-09-18T04:39:37.703 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9120999999999999e-05 2024-09-18T04:39:37.704 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.704 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.704 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:37.704 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579123+0000", 2024-09-18T04:39:37.704 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4382000000000001e-05 2024-09-18T04:39:37.704 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.704 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.704 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:37.704 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579221+0000", 2024-09-18T04:39:37.704 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.7503999999999993e-05 2024-09-18T04:39:37.704 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.704 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.704 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:37.704 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579553+0000", 2024-09-18T04:39:37.705 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00033254200000000002 2024-09-18T04:39:37.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.705 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.705 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:37.705 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579817+0000", 2024-09-18T04:39:37.705 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00026317099999999998 2024-09-18T04:39:37.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.705 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.705 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:37.705 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.599905+0000", 2024-09-18T04:39:37.705 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020088575000000001 2024-09-18T04:39:37.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.705 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.706 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:37.706 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.599923+0000", 2024-09-18T04:39:37.706 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7425999999999999e-05 2024-09-18T04:39:37.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.706 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.706 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:37.706 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.599973+0000", 2024-09-18T04:39:37.706 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0649999999999998e-05 2024-09-18T04:39:37.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.706 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.706 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:37.706 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.600004+0000", 2024-09-18T04:39:37.706 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0361999999999998e-05 2024-09-18T04:39:37.706 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.707 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:37.707 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.707 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.707 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:50 2.1 e51/24 2:8c706374:ns::rbd_data.133e813ab5bc.000000000000000a:head v 51'8, mlcod=51'6)", 2024-09-18T04:39:37.707 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.577029+0000", 2024-09-18T04:39:37.707 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 55.040350717999999, 2024-09-18T04:39:37.707 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.017362855, 2024-09-18T04:39:37.707 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:37.707 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:37.707 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:37.707 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.707 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:37.707 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577029+0000", 2024-09-18T04:39:37.708 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.708 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.708 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:37.708 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577029+0000", 2024-09-18T04:39:37.708 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.708 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.708 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:37.708 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577027+0000", 2024-09-18T04:39:37.708 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:37.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.708 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.708 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:37.709 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578880+0000", 2024-09-18T04:39:37.709 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0018532080000000001 2024-09-18T04:39:37.709 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.709 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.709 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:37.709 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578906+0000", 2024-09-18T04:39:37.709 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5573999999999999e-05 2024-09-18T04:39:37.709 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.709 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.709 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:37.709 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578920+0000", 2024-09-18T04:39:37.709 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4483e-05 2024-09-18T04:39:37.709 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.709 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.710 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:37.710 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578983+0000", 2024-09-18T04:39:37.710 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2248000000000004e-05 2024-09-18T04:39:37.710 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.710 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.710 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:37.710 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579016+0000", 2024-09-18T04:39:37.710 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3364000000000001e-05 2024-09-18T04:39:37.710 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.710 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.710 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:37.710 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594342+0000", 2024-09-18T04:39:37.710 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01532564 2024-09-18T04:39:37.710 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.710 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.711 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:37.711 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594392+0000", 2024-09-18T04:39:37.711 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0068999999999998e-05 2024-09-18T04:39:37.711 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.711 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:37.711 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.711 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.711 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:45 2.0 e51/17 2:0fbfa3dc:ns::rbd_data.133e813ab5bc.0000000000000007:head v 51'24, mlcod=51'22)", 2024-09-18T04:39:37.711 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.581892+0000", 2024-09-18T04:39:37.711 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 55.035488045999998, 2024-09-18T04:39:37.711 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015646006, 2024-09-18T04:39:37.711 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:37.711 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:37.712 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:37.712 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.712 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:37.712 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581892+0000", 2024-09-18T04:39:37.712 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.712 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.712 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.712 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:37.712 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581892+0000", 2024-09-18T04:39:37.712 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.712 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.712 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.712 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:37.712 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581890+0000", 2024-09-18T04:39:37.712 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:37.713 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.713 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.713 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:37.713 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586188+0000", 2024-09-18T04:39:37.713 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0042985769999999996 2024-09-18T04:39:37.713 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.713 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.713 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:37.713 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586210+0000", 2024-09-18T04:39:37.713 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1069000000000001e-05 2024-09-18T04:39:37.713 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.713 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.713 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:37.713 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586221+0000", 2024-09-18T04:39:37.714 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1287000000000001e-05 2024-09-18T04:39:37.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.714 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.714 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:37.714 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586313+0000", 2024-09-18T04:39:37.714 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1935999999999995e-05 2024-09-18T04:39:37.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.714 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.714 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:37.714 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586368+0000", 2024-09-18T04:39:37.714 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5578000000000002e-05 2024-09-18T04:39:37.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.714 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.714 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:37.714 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.597473+0000", 2024-09-18T04:39:37.715 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011105139999999999 2024-09-18T04:39:37.715 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.715 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.715 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:37.715 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.597538+0000", 2024-09-18T04:39:37.715 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4133000000000004e-05 2024-09-18T04:39:37.715 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.715 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:37.715 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.715 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.715 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.715 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:53 2.4 2:39f7f174:ns::rbd_data.133e813ab5bc.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:37.715 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.584263+0000", 2024-09-18T04:39:37.715 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 55.033116172, 2024-09-18T04:39:37.716 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018701180000000001, 2024-09-18T04:39:37.716 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:37.716 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:37.716 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:37.716 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:37.716 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:37.716 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 53 2024-09-18T04:39:37.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.716 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:37.716 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.716 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:37.716 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584263+0000", 2024-09-18T04:39:37.716 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.717 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.717 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.717 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:37.717 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584263+0000", 2024-09-18T04:39:37.717 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.717 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.717 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.717 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:37.717 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584262+0000", 2024-09-18T04:39:37.717 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:37.717 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.717 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.717 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:37.717 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.591617+0000", 2024-09-18T04:39:37.717 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0073550600000000001 2024-09-18T04:39:37.718 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.718 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.718 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:37.718 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.591637+0000", 2024-09-18T04:39:37.718 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0432999999999999e-05 2024-09-18T04:39:37.718 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.718 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.718 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:37.718 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.591646+0000", 2024-09-18T04:39:37.718 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7849999999999997e-06 2024-09-18T04:39:37.718 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.718 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.718 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:37.718 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593478+0000", 2024-09-18T04:39:37.719 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001831495 2024-09-18T04:39:37.719 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.719 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.719 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:37.719 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593661+0000", 2024-09-18T04:39:37.719 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018364999999999999 2024-09-18T04:39:37.719 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.719 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.719 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:37.719 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593834+0000", 2024-09-18T04:39:37.719 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000173065 2024-09-18T04:39:37.719 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.719 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.719 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:37.720 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.600670+0000", 2024-09-18T04:39:37.720 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0068354970000000003 2024-09-18T04:39:37.720 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.720 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.720 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:37.720 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602851+0000", 2024-09-18T04:39:37.720 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002181653 2024-09-18T04:39:37.720 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.720 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.720 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:37.720 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602930+0000", 2024-09-18T04:39:37.720 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8045000000000002e-05 2024-09-18T04:39:37.720 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.720 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.721 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:37.721 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602965+0000", 2024-09-18T04:39:37.721 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5102999999999999e-05 2024-09-18T04:39:37.721 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.721 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:37.721 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.721 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.721 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.721 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:56 2.6 2:72453c3e:ns::rbd_data.133e813ab5bc.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:37.721 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.595988+0000", 2024-09-18T04:39:37.721 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 55.021391770000001, 2024-09-18T04:39:37.721 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.030296172, 2024-09-18T04:39:37.721 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:37.721 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:37.722 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:37.722 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:37.722 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:37.722 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 56 2024-09-18T04:39:37.722 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.722 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:37.722 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.722 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:37.722 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.595988+0000", 2024-09-18T04:39:37.722 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.722 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.722 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.722 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:37.722 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.595988+0000", 2024-09-18T04:39:37.723 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:37.723 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.723 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.723 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:37.723 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.595986+0000", 2024-09-18T04:39:37.723 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:37.723 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.723 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.723 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:37.723 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605262+0000", 2024-09-18T04:39:37.723 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0092767240000000001 2024-09-18T04:39:37.723 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.723 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.723 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:37.723 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605282+0000", 2024-09-18T04:39:37.724 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9307e-05 2024-09-18T04:39:37.724 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.724 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.724 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:37.724 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605292+0000", 2024-09-18T04:39:37.724 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9510000000000001e-06 2024-09-18T04:39:37.724 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.724 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.724 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:37.724 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605369+0000", 2024-09-18T04:39:37.724 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7846e-05 2024-09-18T04:39:37.724 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.724 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.724 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:37.725 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605668+0000", 2024-09-18T04:39:37.725 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00029896099999999999 2024-09-18T04:39:37.725 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.725 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.725 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:37.725 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605906+0000", 2024-09-18T04:39:37.725 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000237758 2024-09-18T04:39:37.725 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.725 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.725 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:37.725 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.613532+0000", 2024-09-18T04:39:37.725 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0076260590000000001 2024-09-18T04:39:37.725 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.725 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.725 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:37.726 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.626116+0000", 2024-09-18T04:39:37.726 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012584092999999999 2024-09-18T04:39:37.726 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.726 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.726 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:37.726 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.626216+0000", 2024-09-18T04:39:37.726 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9630999999999995e-05 2024-09-18T04:39:37.726 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:37.726 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:37.726 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:37.726 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.626284+0000", 2024-09-18T04:39:37.726 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8034999999999995e-05 2024-09-18T04:39:37.726 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.726 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:37.727 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.727 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:37.727 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:37.727 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:37.728 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:39:37.729 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:37.724+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:37.829 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:37.820+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:37.845 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:37.845 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:37.845 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:39:37.845 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:37.856 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:39:37.931 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:37.928+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:37.958 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:37.958 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:37.958 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:39:37.959 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:39:37.959 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:37.968 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:39:38.032 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:38.024+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:38.081 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:38.081 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:39:38.081 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:39:38.081 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:39:38.081 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.081 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:39:38.081 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:39:38.081 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 130.28819522399999, 2024-09-18T04:39:38.081 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017399218000000001, 2024-09-18T04:39:38.082 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:38.082 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:38.082 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:38.082 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:39:38.082 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:39:38.082 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 114 2024-09-18T04:39:38.082 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.082 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:38.082 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.082 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:38.082 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:39:38.083 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:38.083 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.083 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.083 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:38.083 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:39:38.083 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:38.083 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.083 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.083 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:38.083 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786900+0000", 2024-09-18T04:39:38.083 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:38.083 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.083 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.083 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:38.084 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786908+0000", 2024-09-18T04:39:38.084 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8199999999999997e-06 2024-09-18T04:39:38.084 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.084 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.084 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:38.084 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786925+0000", 2024-09-18T04:39:38.084 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6849000000000001e-05 2024-09-18T04:39:38.084 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.084 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.084 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:38.084 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786934+0000", 2024-09-18T04:39:38.084 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6299999999999993e-06 2024-09-18T04:39:38.084 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.084 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.085 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:38.085 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800926+0000", 2024-09-18T04:39:38.085 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013991632 2024-09-18T04:39:38.085 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.085 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.085 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:38.085 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801060+0000", 2024-09-18T04:39:38.085 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013378199999999999 2024-09-18T04:39:38.085 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.085 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.085 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:38.085 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801166+0000", 2024-09-18T04:39:38.086 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010607099999999999 2024-09-18T04:39:38.086 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.086 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.086 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:38.086 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803026+0000", 2024-09-18T04:39:38.086 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001860469 2024-09-18T04:39:38.086 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.086 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.086 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:38.086 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804245+0000", 2024-09-18T04:39:38.086 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0012185900000000001 2024-09-18T04:39:38.086 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.086 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.087 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:38.087 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804278+0000", 2024-09-18T04:39:38.087 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3272000000000003e-05 2024-09-18T04:39:38.087 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.087 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.087 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:38.087 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804300+0000", 2024-09-18T04:39:38.087 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2266e-05 2024-09-18T04:39:38.087 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:38.087 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:38.087 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:38.087 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.087 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.088 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:39:38.088 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:39:38.088 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 130.287282326, 2024-09-18T04:39:38.088 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017521662, 2024-09-18T04:39:38.088 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:38.088 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:38.088 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:38.088 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:39:38.088 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:39:38.088 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 116 2024-09-18T04:39:38.088 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.088 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:38.088 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.088 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:38.089 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:39:38.089 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:38.089 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.089 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.089 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:38.089 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:39:38.089 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:38.089 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.089 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.089 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:38.089 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787813+0000", 2024-09-18T04:39:38.089 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:38.090 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.090 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.090 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:38.090 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787824+0000", 2024-09-18T04:39:38.090 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1157e-05 2024-09-18T04:39:38.090 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.090 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.090 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:38.090 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787840+0000", 2024-09-18T04:39:38.090 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6269e-05 2024-09-18T04:39:38.090 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.090 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.090 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:38.090 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787853+0000", 2024-09-18T04:39:38.091 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3156000000000001e-05 2024-09-18T04:39:38.091 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.091 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.091 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:38.091 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803110+0000", 2024-09-18T04:39:38.091 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015256736 2024-09-18T04:39:38.091 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.091 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.091 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:38.091 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803197+0000", 2024-09-18T04:39:38.091 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7195999999999997e-05 2024-09-18T04:39:38.091 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.091 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.091 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:38.092 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803308+0000", 2024-09-18T04:39:38.092 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111005 2024-09-18T04:39:38.092 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.092 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.092 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:38.092 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804316+0000", 2024-09-18T04:39:38.092 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010073949999999999 2024-09-18T04:39:38.092 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.092 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.092 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:38.092 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805232+0000", 2024-09-18T04:39:38.092 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091636300000000001 2024-09-18T04:39:38.093 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.093 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.093 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:38.093 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805277+0000", 2024-09-18T04:39:38.093 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4864000000000003e-05 2024-09-18T04:39:38.093 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.093 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.093 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:38.093 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805336+0000", 2024-09-18T04:39:38.093 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8968000000000002e-05 2024-09-18T04:39:38.093 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:38.093 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:38.094 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:38.094 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.094 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.094 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:39:38.094 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:39:38.094 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 130.287230271, 2024-09-18T04:39:38.094 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017983453, 2024-09-18T04:39:38.094 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:38.094 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:38.094 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:38.094 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:39:38.094 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:39:38.094 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 117 2024-09-18T04:39:38.094 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.095 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:38.095 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.095 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:38.095 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:39:38.095 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:38.095 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.095 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.095 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:38.095 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:39:38.095 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:38.095 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.095 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.096 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:38.096 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787865+0000", 2024-09-18T04:39:38.096 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:38.096 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.096 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.096 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:38.096 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787889+0000", 2024-09-18T04:39:38.096 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4244999999999999e-05 2024-09-18T04:39:38.096 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.096 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.096 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:38.096 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787905+0000", 2024-09-18T04:39:38.097 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5739999999999998e-05 2024-09-18T04:39:38.097 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.097 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.097 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:38.097 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787918+0000", 2024-09-18T04:39:38.097 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.252e-05 2024-09-18T04:39:38.097 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.097 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.097 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:38.097 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803693+0000", 2024-09-18T04:39:38.097 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015774872999999998 2024-09-18T04:39:38.097 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.098 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.098 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:38.098 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803775+0000", 2024-09-18T04:39:38.098 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2761000000000003e-05 2024-09-18T04:39:38.098 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.098 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.098 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:38.098 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803887+0000", 2024-09-18T04:39:38.098 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111708 2024-09-18T04:39:38.098 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.098 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.098 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:38.098 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804823+0000", 2024-09-18T04:39:38.098 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00093625100000000003 2024-09-18T04:39:38.099 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.099 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.099 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:38.099 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805804+0000", 2024-09-18T04:39:38.099 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00098114699999999992 2024-09-18T04:39:38.099 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.099 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.099 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:38.099 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805831+0000", 2024-09-18T04:39:38.099 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6401999999999999e-05 2024-09-18T04:39:38.099 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.099 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.099 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:38.099 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805850+0000", 2024-09-18T04:39:38.100 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8862e-05 2024-09-18T04:39:38.100 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:38.100 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:38.100 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:38.100 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.100 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.100 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:36 2.7 2:fe1b02ad:::rbd_data.125fc09df6d1.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:38.100 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:39:38.100 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 55.809374161000001, 2024-09-18T04:39:38.100 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.040387804999999999, 2024-09-18T04:39:38.100 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:38.100 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:38.101 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:38.101 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:38.101 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:38.101 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 36 2024-09-18T04:39:38.101 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.101 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:38.101 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.101 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:38.101 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:39:38.101 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:38.101 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.101 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.101 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:38.101 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:39:38.102 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:38.102 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.102 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.102 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:38.102 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265720+0000", 2024-09-18T04:39:38.102 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:38.102 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.102 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.102 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:38.102 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272590+0000", 2024-09-18T04:39:38.102 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.006870508 2024-09-18T04:39:38.102 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.102 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.102 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:38.103 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272623+0000", 2024-09-18T04:39:38.103 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2716000000000003e-05 2024-09-18T04:39:38.103 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.103 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.103 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:38.103 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272637+0000", 2024-09-18T04:39:38.103 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4175999999999999e-05 2024-09-18T04:39:38.103 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.103 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.103 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:38.103 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272779+0000", 2024-09-18T04:39:38.103 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000141392 2024-09-18T04:39:38.103 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.103 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.104 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:38.104 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273031+0000", 2024-09-18T04:39:38.104 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00025249099999999998 2024-09-18T04:39:38.104 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.104 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.104 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:39:38.104 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273219+0000", 2024-09-18T04:39:38.104 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000187531 2024-09-18T04:39:38.104 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.104 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.104 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:38.104 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.302806+0000", 2024-09-18T04:39:38.104 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.029587553999999999 2024-09-18T04:39:38.104 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.105 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.105 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:38.105 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306004+0000", 2024-09-18T04:39:38.105 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0031982629999999998 2024-09-18T04:39:38.105 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.105 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.105 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:38.105 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306071+0000", 2024-09-18T04:39:38.105 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6829999999999995e-05 2024-09-18T04:39:38.105 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.105 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.105 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:38.105 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306110+0000", 2024-09-18T04:39:38.105 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8874000000000001e-05 2024-09-18T04:39:38.106 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:38.106 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:38.106 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:38.106 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.106 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.106 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:38 2.0 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:38.106 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:39:38.106 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 55.802420624, 2024-09-18T04:39:38.106 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.033754348000000003, 2024-09-18T04:39:38.106 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:38.106 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:38.106 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:38.106 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:38.106 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:38.107 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 38 2024-09-18T04:39:38.107 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.107 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:38.107 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.107 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:38.107 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:39:38.107 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:38.107 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.107 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.107 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:38.107 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:39:38.107 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:38.107 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.107 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.107 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:38.108 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272673+0000", 2024-09-18T04:39:38.108 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:38.108 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.108 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.108 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:38.108 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277166+0000", 2024-09-18T04:39:38.108 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044922249999999999 2024-09-18T04:39:38.108 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.108 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.108 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:38.108 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277189+0000", 2024-09-18T04:39:38.108 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3283000000000001e-05 2024-09-18T04:39:38.108 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.108 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.109 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:38.109 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277205+0000", 2024-09-18T04:39:38.109 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6305000000000001e-05 2024-09-18T04:39:38.109 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.109 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.109 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:38.109 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277290+0000", 2024-09-18T04:39:38.109 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5135000000000004e-05 2024-09-18T04:39:38.109 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.109 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.109 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:38.109 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277531+0000", 2024-09-18T04:39:38.109 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00024014699999999999 2024-09-18T04:39:38.109 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.110 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.110 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:38.110 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277793+0000", 2024-09-18T04:39:38.110 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00026224499999999998 2024-09-18T04:39:38.110 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.110 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.110 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:38.110 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303990+0000", 2024-09-18T04:39:38.110 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.026197295999999998 2024-09-18T04:39:38.110 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.110 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.110 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:38.110 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306340+0000", 2024-09-18T04:39:38.110 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.002349524 2024-09-18T04:39:38.111 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.111 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.111 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:38.111 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306394+0000", 2024-09-18T04:39:38.111 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3934999999999998e-05 2024-09-18T04:39:38.111 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.111 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.111 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:38.111 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306430+0000", 2024-09-18T04:39:38.111 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6690000000000003e-05 2024-09-18T04:39:38.111 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:38.111 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:38.111 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:38.111 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.112 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.112 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:39 2.7 2:e9c909c5:::rbd_data.125fc09df6d1.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:38.112 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:39:38.112 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 55.797861019000003, 2024-09-18T04:39:38.112 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.029110673, 2024-09-18T04:39:38.112 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:38.112 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:38.112 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:38.112 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:38.112 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:38.112 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 39 2024-09-18T04:39:38.112 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.112 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:38.112 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.113 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:38.113 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:39:38.113 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:38.113 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.113 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.113 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:38.113 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:39:38.113 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:38.113 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.113 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.113 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:38.113 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277234+0000", 2024-09-18T04:39:38.113 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:38.114 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.114 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.114 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:38.114 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281351+0000", 2024-09-18T04:39:38.114 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0041170920000000001 2024-09-18T04:39:38.114 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.114 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.114 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:38.114 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281375+0000", 2024-09-18T04:39:38.114 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.446e-05 2024-09-18T04:39:38.114 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.114 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.114 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:38.114 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281390+0000", 2024-09-18T04:39:38.115 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4309e-05 2024-09-18T04:39:38.115 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.115 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.115 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:38.115 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.282306+0000", 2024-09-18T04:39:38.115 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091662000000000004 2024-09-18T04:39:38.115 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.115 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.115 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:38.115 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.282613+0000", 2024-09-18T04:39:38.115 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00030663799999999997 2024-09-18T04:39:38.115 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.115 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.115 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:39:38.116 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283001+0000", 2024-09-18T04:39:38.116 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00038801599999999998 2024-09-18T04:39:38.116 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.116 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.116 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:38.116 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306239+0000", 2024-09-18T04:39:38.116 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023238466999999999 2024-09-18T04:39:38.116 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.116 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.116 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:38.116 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306259+0000", 2024-09-18T04:39:38.116 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9837999999999999e-05 2024-09-18T04:39:38.116 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.116 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.117 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:38.117 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306299+0000", 2024-09-18T04:39:38.117 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0194000000000002e-05 2024-09-18T04:39:38.117 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.117 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.117 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:38.117 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306346+0000", 2024-09-18T04:39:38.117 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6845000000000003e-05 2024-09-18T04:39:38.117 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:38.117 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:38.117 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:38.117 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.117 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.117 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:46 2.6 e51/17 2:7ba25245:::rbd_data.125fc09df6d1.0000000000000006:head v 51'2, mlcod=0'0)", 2024-09-18T04:39:38.118 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:39:38.118 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 55.761939886999997, 2024-09-18T04:39:38.118 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019843098, 2024-09-18T04:39:38.118 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:38.118 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:38.118 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:38.118 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.118 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:38.118 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:39:38.118 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:38.118 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.118 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.118 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:38.118 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:39:38.119 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:38.119 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.119 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.119 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:38.119 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313156+0000", 2024-09-18T04:39:38.119 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:38.119 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.119 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.119 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:38.119 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320144+0000", 2024-09-18T04:39:38.119 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.006988345 2024-09-18T04:39:38.119 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.119 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.120 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:38.120 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320175+0000", 2024-09-18T04:39:38.120 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.0861999999999997e-05 2024-09-18T04:39:38.120 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.120 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.120 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:38.120 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320188+0000", 2024-09-18T04:39:38.120 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2714999999999999e-05 2024-09-18T04:39:38.120 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.120 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.120 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:38.120 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320267+0000", 2024-09-18T04:39:38.120 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9586999999999995e-05 2024-09-18T04:39:38.120 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.121 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.121 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:38.121 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320311+0000", 2024-09-18T04:39:38.121 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3529000000000003e-05 2024-09-18T04:39:38.121 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.121 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.121 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:38.121 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.332949+0000", 2024-09-18T04:39:38.121 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012638267999999999 2024-09-18T04:39:38.121 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.121 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.121 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:38.121 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.333000+0000", 2024-09-18T04:39:38.121 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0736999999999999e-05 2024-09-18T04:39:38.122 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:38.122 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:38.122 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:38.122 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.122 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.122 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:57 2.0 2:0cdd070e:::rbd_data.125fc09df6d1.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:38.122 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:39:38.122 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 55.726082312000003, 2024-09-18T04:39:38.122 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.033329974999999998, 2024-09-18T04:39:38.122 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:38.122 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:38.122 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:38.122 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:38.122 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:38.123 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 57 2024-09-18T04:39:38.123 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.123 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:38.123 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.123 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:38.123 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:39:38.123 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:38.123 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.123 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.123 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:38.123 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:39:38.123 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:38.123 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.123 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.124 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:38.124 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349013+0000", 2024-09-18T04:39:38.124 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:38.124 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.124 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.124 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:38.124 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356138+0000", 2024-09-18T04:39:38.124 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071258010000000002 2024-09-18T04:39:38.124 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.124 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.124 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:38.124 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356168+0000", 2024-09-18T04:39:38.124 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9679000000000002e-05 2024-09-18T04:39:38.125 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.125 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.125 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:38.125 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356181+0000", 2024-09-18T04:39:38.125 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3164e-05 2024-09-18T04:39:38.125 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.125 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.125 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:38.125 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356254+0000", 2024-09-18T04:39:38.125 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3182999999999994e-05 2024-09-18T04:39:38.125 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.125 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.126 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:38.126 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356453+0000", 2024-09-18T04:39:38.126 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019876599999999999 2024-09-18T04:39:38.126 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.126 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.126 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:38.126 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356627+0000", 2024-09-18T04:39:38.126 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017340199999999999 2024-09-18T04:39:38.126 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.126 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.126 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:38.126 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.369200+0000", 2024-09-18T04:39:38.126 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012573302 2024-09-18T04:39:38.127 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.127 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.127 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:38.127 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382262+0000", 2024-09-18T04:39:38.127 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013062549 2024-09-18T04:39:38.127 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.127 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.127 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:38.127 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382310+0000", 2024-09-18T04:39:38.127 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7366e-05 2024-09-18T04:39:38.127 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.127 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.127 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:38.127 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382344+0000", 2024-09-18T04:39:38.128 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4319999999999997e-05 2024-09-18T04:39:38.128 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:38.128 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:38.128 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:38.128 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.128 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.128 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:59 2.0 2:19c52cde:::rbd_data.125fc09df6d1.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:38.128 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:39:38.128 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 55.718891147000001, 2024-09-18T04:39:38.128 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.027137247999999999, 2024-09-18T04:39:38.128 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:38.128 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:38.128 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:38.129 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:38.129 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:38.129 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 59 2024-09-18T04:39:38.129 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.129 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:38.129 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.129 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:38.129 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:39:38.129 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:38.129 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.129 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.129 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:38.129 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:39:38.130 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:38.130 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.130 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.130 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:38.130 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356204+0000", 2024-09-18T04:39:38.130 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:38.130 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.130 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.130 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:38.130 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362661+0000", 2024-09-18T04:39:38.130 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0064565129999999997 2024-09-18T04:39:38.130 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.130 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.131 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:38.131 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362684+0000", 2024-09-18T04:39:38.131 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3688000000000001e-05 2024-09-18T04:39:38.131 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.131 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.131 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:38.131 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362699+0000", 2024-09-18T04:39:38.131 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4798e-05 2024-09-18T04:39:38.131 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.131 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.131 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:38.131 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363016+0000", 2024-09-18T04:39:38.131 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031737300000000003 2024-09-18T04:39:38.131 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.132 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.132 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:38.132 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363201+0000", 2024-09-18T04:39:38.132 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000184131 2024-09-18T04:39:38.132 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.132 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.132 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:38.132 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363343+0000", 2024-09-18T04:39:38.132 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014236699999999999 2024-09-18T04:39:38.132 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.132 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.132 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:38.133 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.372189+0000", 2024-09-18T04:39:38.133 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0088464819999999993 2024-09-18T04:39:38.133 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.133 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.133 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:38.133 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383269+0000", 2024-09-18T04:39:38.133 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011079594 2024-09-18T04:39:38.133 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.133 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.133 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:38.133 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383314+0000", 2024-09-18T04:39:38.133 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4932000000000003e-05 2024-09-18T04:39:38.133 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.134 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.134 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:38.134 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383343+0000", 2024-09-18T04:39:38.134 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8600999999999999e-05 2024-09-18T04:39:38.134 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:38.134 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:38.134 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:38.134 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.134 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.134 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:40 2.6 e51/17 2:784d6cd1:ns::rbd_data.133e813ab5bc.0000000000000005:head v 51'7, mlcod=51'5)", 2024-09-18T04:39:38.134 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:39:38.134 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 55.522641413000002, 2024-09-18T04:39:38.134 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020497496, 2024-09-18T04:39:38.134 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:38.135 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:38.135 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:38.135 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.135 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:38.135 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:39:38.135 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:38.135 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.135 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.135 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:38.135 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:39:38.135 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:38.135 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.135 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.135 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:38.136 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552454+0000", 2024-09-18T04:39:38.136 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:38.136 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.136 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.136 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:38.136 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556448+0000", 2024-09-18T04:39:38.136 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0039940990000000001 2024-09-18T04:39:38.136 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.136 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.136 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:38.136 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556481+0000", 2024-09-18T04:39:38.136 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3778000000000001e-05 2024-09-18T04:39:38.136 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.136 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.136 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:38.137 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556493+0000", 2024-09-18T04:39:38.137 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1622e-05 2024-09-18T04:39:38.137 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.137 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.137 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:38.137 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556556+0000", 2024-09-18T04:39:38.137 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2582999999999994e-05 2024-09-18T04:39:38.137 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.137 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.137 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:38.137 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556592+0000", 2024-09-18T04:39:38.137 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5880999999999997e-05 2024-09-18T04:39:38.137 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.138 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.138 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:38.138 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.572786+0000", 2024-09-18T04:39:38.138 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016194571000000001 2024-09-18T04:39:38.138 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.138 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.138 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:38.138 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.572953+0000", 2024-09-18T04:39:38.138 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000166436 2024-09-18T04:39:38.138 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:38.138 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:38.138 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:38.138 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.138 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.139 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:45 2.0 2:0fbfa3dc:ns::rbd_data.133e813ab5bc.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:38.139 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:39:38.139 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 55.511427114, 2024-09-18T04:39:38.139 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.036558168000000002, 2024-09-18T04:39:38.139 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:38.139 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:38.139 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:38.139 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:38.139 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:38.139 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 45 2024-09-18T04:39:38.139 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.139 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:38.139 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.140 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:38.140 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:39:38.140 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:38.140 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.140 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.140 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:38.140 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:39:38.140 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:38.140 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.140 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.140 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:38.140 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:39:38.140 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:38.140 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.141 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.141 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:38.141 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578594+0000", 2024-09-18T04:39:38.141 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014927147999999999 2024-09-18T04:39:38.141 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.141 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.141 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:38.141 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578612+0000", 2024-09-18T04:39:38.141 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7623999999999999e-05 2024-09-18T04:39:38.141 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.141 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.141 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:38.142 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578620+0000", 2024-09-18T04:39:38.142 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9070000000000007e-06 2024-09-18T04:39:38.142 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.142 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.142 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:38.142 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578702+0000", 2024-09-18T04:39:38.142 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2349999999999996e-05 2024-09-18T04:39:38.142 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.142 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.142 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:38.142 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.579028+0000", 2024-09-18T04:39:38.142 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032622899999999998 2024-09-18T04:39:38.142 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.143 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.143 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:38.143 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.579343+0000", 2024-09-18T04:39:38.143 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031448899999999999 2024-09-18T04:39:38.143 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.143 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.143 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:38.143 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.590078+0000", 2024-09-18T04:39:38.143 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010734946 2024-09-18T04:39:38.143 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.143 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.143 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:38.143 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600089+0000", 2024-09-18T04:39:38.144 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010011195000000001 2024-09-18T04:39:38.144 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.144 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.144 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:38.144 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600177+0000", 2024-09-18T04:39:38.144 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8425999999999997e-05 2024-09-18T04:39:38.144 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.144 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.145 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:38.145 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600228+0000", 2024-09-18T04:39:38.145 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0216999999999997e-05 2024-09-18T04:39:38.145 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:38.145 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:38.145 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:38.145 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.145 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.145 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:54 2.0 2:04f17ef6:ns::rbd_data.133e813ab5bc.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:38.145 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:39:38.145 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 55.489855276999997, 2024-09-18T04:39:38.145 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.024612999, 2024-09-18T04:39:38.146 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:38.146 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:38.146 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:38.146 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:38.146 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:38.146 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 54 2024-09-18T04:39:38.146 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.146 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:38.146 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.146 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:38.146 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:39:38.146 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:38.147 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.147 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.147 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:38.147 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:39:38.147 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:38.147 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.147 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.147 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:38.147 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585239+0000", 2024-09-18T04:39:38.147 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:38.147 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.147 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.147 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:38.147 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593702+0000", 2024-09-18T04:39:38.148 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0084632990000000005 2024-09-18T04:39:38.148 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.148 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.148 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:38.148 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593719+0000", 2024-09-18T04:39:38.148 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.664e-05 2024-09-18T04:39:38.148 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.148 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.148 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:38.148 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593727+0000", 2024-09-18T04:39:38.148 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6839999999999996e-06 2024-09-18T04:39:38.148 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.148 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.148 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:38.149 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593767+0000", 2024-09-18T04:39:38.149 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0330999999999997e-05 2024-09-18T04:39:38.149 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.149 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.149 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:38.149 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593905+0000", 2024-09-18T04:39:38.149 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013754800000000001 2024-09-18T04:39:38.149 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.149 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.149 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:38.149 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.594013+0000", 2024-09-18T04:39:38.149 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000108805 2024-09-18T04:39:38.149 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.149 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.149 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:38.150 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604678+0000", 2024-09-18T04:39:38.150 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010664391 2024-09-18T04:39:38.150 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.150 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.150 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:38.150 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609662+0000", 2024-09-18T04:39:38.150 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0049846100000000004 2024-09-18T04:39:38.150 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.150 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.150 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:38.150 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609788+0000", 2024-09-18T04:39:38.150 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012518800000000001 2024-09-18T04:39:38.150 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.151 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.151 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:38.151 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609854+0000", 2024-09-18T04:39:38.151 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6556000000000004e-05 2024-09-18T04:39:38.151 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:38.151 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:38.151 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:38.151 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.151 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.151 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:58 2.7 2:fc7c6cd3:ns::rbd_data.133e813ab5bc.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:38.151 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:39:38.151 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 55.477012848000001, 2024-09-18T04:39:38.151 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.027690566999999999, 2024-09-18T04:39:38.151 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:38.151 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:38.152 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:38.152 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:38.152 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:38.152 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 58 2024-09-18T04:39:38.152 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.152 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:38.152 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.152 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:38.152 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:39:38.152 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:38.152 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.152 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.152 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:38.153 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:39:38.153 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:38.153 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.153 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.153 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:38.153 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598082+0000", 2024-09-18T04:39:38.153 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:38.153 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.153 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.153 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:38.153 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604832+0000", 2024-09-18T04:39:38.153 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0067504510000000002 2024-09-18T04:39:38.153 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.154 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.154 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:38.154 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604852+0000", 2024-09-18T04:39:38.154 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0228999999999999e-05 2024-09-18T04:39:38.154 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.154 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.154 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:38.154 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604870+0000", 2024-09-18T04:39:38.154 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7952e-05 2024-09-18T04:39:38.154 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.154 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.154 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:38.154 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604936+0000", 2024-09-18T04:39:38.155 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5431000000000004e-05 2024-09-18T04:39:38.155 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.155 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.155 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:38.155 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605085+0000", 2024-09-18T04:39:38.155 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014910300000000001 2024-09-18T04:39:38.155 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.155 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.155 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:39:38.155 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605205+0000", 2024-09-18T04:39:38.155 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000119887 2024-09-18T04:39:38.155 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.155 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.156 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:38.156 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625586+0000", 2024-09-18T04:39:38.156 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020381399000000001 2024-09-18T04:39:38.156 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.156 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.156 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:38.156 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625628+0000", 2024-09-18T04:39:38.156 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1873999999999999e-05 2024-09-18T04:39:38.156 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.156 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.156 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:38.156 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625722+0000", 2024-09-18T04:39:38.156 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3787000000000002e-05 2024-09-18T04:39:38.157 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.157 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.157 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:38.157 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625774+0000", 2024-09-18T04:39:38.157 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2503999999999997e-05 2024-09-18T04:39:38.157 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:38.157 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:38.157 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:38.157 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.157 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.157 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:56 2.6 e51/17 2:72453c3e:ns::rbd_data.133e813ab5bc.000000000000000d:head v 51'10, mlcod=51'8)", 2024-09-18T04:39:38.157 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:39:38.157 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 55.469815740000001, 2024-09-18T04:39:38.157 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019766552999999999, 2024-09-18T04:39:38.158 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:38.158 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:38.158 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:38.158 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.158 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:38.158 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:39:38.158 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:38.158 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.158 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.158 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:38.158 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:39:38.158 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:38.158 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.159 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.159 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:38.159 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605279+0000", 2024-09-18T04:39:38.159 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:38.159 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.159 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.159 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:38.159 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609417+0000", 2024-09-18T04:39:38.159 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0041378059999999999 2024-09-18T04:39:38.159 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.159 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.159 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:38.159 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609443+0000", 2024-09-18T04:39:38.160 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6098e-05 2024-09-18T04:39:38.160 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.160 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.160 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:38.160 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609458+0000", 2024-09-18T04:39:38.160 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4671000000000001e-05 2024-09-18T04:39:38.160 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.160 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.160 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:38.160 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609553+0000", 2024-09-18T04:39:38.160 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5265000000000004e-05 2024-09-18T04:39:38.160 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.160 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.160 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:38.161 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609583+0000", 2024-09-18T04:39:38.161 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9839e-05 2024-09-18T04:39:38.161 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.161 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.161 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:38.161 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625004+0000", 2024-09-18T04:39:38.161 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015420639999999999 2024-09-18T04:39:38.161 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.161 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.161 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:38.161 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625047+0000", 2024-09-18T04:39:38.161 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3665999999999998e-05 2024-09-18T04:39:38.161 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:38.162 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:38.162 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:38.162 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.162 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.162 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:99 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:38.162 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.582718+0000", 2024-09-18T04:39:38.162 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 41.492378831000003, 2024-09-18T04:39:38.162 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019362575, 2024-09-18T04:39:38.162 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:38.162 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:38.162 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:38.162 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:38.162 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:38.162 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 99 2024-09-18T04:39:38.162 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.163 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:38.163 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.163 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:38.163 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582718+0000", 2024-09-18T04:39:38.163 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:38.163 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.163 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.163 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:38.163 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582718+0000", 2024-09-18T04:39:38.163 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:38.163 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.163 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.163 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:38.164 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582717+0000", 2024-09-18T04:39:38.164 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:38.164 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.164 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.164 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:38.164 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582720+0000", 2024-09-18T04:39:38.164 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2239999999999998e-06 2024-09-18T04:39:38.164 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.164 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.164 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:38.164 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582723+0000", 2024-09-18T04:39:38.164 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.258e-06 2024-09-18T04:39:38.164 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.165 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.165 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:38.165 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582726+0000", 2024-09-18T04:39:38.165 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8870000000000001e-06 2024-09-18T04:39:38.165 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.165 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.165 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:38.165 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.592018+0000", 2024-09-18T04:39:38.165 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0092926420000000003 2024-09-18T04:39:38.165 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.165 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.165 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:38.165 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.592214+0000", 2024-09-18T04:39:38.166 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000195267 2024-09-18T04:39:38.166 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.166 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.166 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:38.166 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.592913+0000", 2024-09-18T04:39:38.166 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00069924600000000005 2024-09-18T04:39:38.166 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.166 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.166 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:38.166 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597493+0000", 2024-09-18T04:39:38.166 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045797629999999997 2024-09-18T04:39:38.166 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.166 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.167 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:38.167 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.602009+0000", 2024-09-18T04:39:38.167 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045159559999999998 2024-09-18T04:39:38.167 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.167 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.167 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:38.167 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.602042+0000", 2024-09-18T04:39:38.167 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3084999999999999e-05 2024-09-18T04:39:38.167 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.167 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.167 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:38.167 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.602080+0000", 2024-09-18T04:39:38.167 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8600000000000003e-05 2024-09-18T04:39:38.167 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:38.168 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:38.168 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:38.168 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.168 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.168 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:104 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:38.168 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.583165+0000", 2024-09-18T04:39:38.168 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 41.491931532999999, 2024-09-18T04:39:38.168 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023679603, 2024-09-18T04:39:38.168 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:38.168 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:38.168 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:38.168 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:38.168 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:38.169 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 104 2024-09-18T04:39:38.169 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.169 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:38.169 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.169 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:38.169 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583165+0000", 2024-09-18T04:39:38.169 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:38.169 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.169 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.169 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:38.169 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583165+0000", 2024-09-18T04:39:38.169 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:38.169 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.169 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.170 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:38.170 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583164+0000", 2024-09-18T04:39:38.170 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:38.170 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.170 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.170 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:38.170 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583167+0000", 2024-09-18T04:39:38.170 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9210000000000001e-06 2024-09-18T04:39:38.170 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.170 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.170 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:38.170 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583171+0000", 2024-09-18T04:39:38.170 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9759999999999997e-06 2024-09-18T04:39:38.170 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.171 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.171 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:38.171 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583174+0000", 2024-09-18T04:39:38.171 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9960000000000002e-06 2024-09-18T04:39:38.171 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.171 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.171 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:38.171 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.594373+0000", 2024-09-18T04:39:38.171 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011198665999999999 2024-09-18T04:39:38.171 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.171 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.171 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:38.171 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.594434+0000", 2024-09-18T04:39:38.171 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.1446999999999996e-05 2024-09-18T04:39:38.171 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.171 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.172 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:38.172 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.594692+0000", 2024-09-18T04:39:38.172 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00025756599999999998 2024-09-18T04:39:38.172 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.172 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.172 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:38.172 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599302+0000", 2024-09-18T04:39:38.172 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0046099640000000002 2024-09-18T04:39:38.172 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.172 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.172 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:38.172 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606710+0000", 2024-09-18T04:39:38.172 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0074077739999999998 2024-09-18T04:39:38.173 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.173 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.173 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:38.173 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606798+0000", 2024-09-18T04:39:38.173 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8133999999999993e-05 2024-09-18T04:39:38.173 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.173 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.173 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:38.173 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606845+0000", 2024-09-18T04:39:38.173 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6653000000000002e-05 2024-09-18T04:39:38.173 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:38.173 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:38.173 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:38.173 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.174 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.174 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:105 2.0 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:38.174 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.583193+0000", 2024-09-18T04:39:38.174 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 41.491903913999998, 2024-09-18T04:39:38.174 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023713649, 2024-09-18T04:39:38.174 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:38.174 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:38.174 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:38.174 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:38.174 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:38.174 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 105 2024-09-18T04:39:38.174 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.174 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:38.174 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.174 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:38.175 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583193+0000", 2024-09-18T04:39:38.175 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:38.175 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.175 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.175 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:38.175 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583193+0000", 2024-09-18T04:39:38.175 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:38.175 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.175 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.175 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:38.175 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583192+0000", 2024-09-18T04:39:38.175 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:38.175 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.175 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.175 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:38.176 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583194+0000", 2024-09-18T04:39:38.176 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.305e-06 2024-09-18T04:39:38.176 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.176 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.176 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:38.176 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583198+0000", 2024-09-18T04:39:38.176 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9369999999999997e-06 2024-09-18T04:39:38.176 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.176 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.176 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:38.176 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583202+0000", 2024-09-18T04:39:38.176 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6480000000000001e-06 2024-09-18T04:39:38.176 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.176 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.177 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:38.177 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.595304+0000", 2024-09-18T04:39:38.177 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012102063999999999 2024-09-18T04:39:38.177 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.177 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.177 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:38.177 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.595390+0000", 2024-09-18T04:39:38.177 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5493000000000003e-05 2024-09-18T04:39:38.177 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.177 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.177 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:38.177 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.595509+0000", 2024-09-18T04:39:38.177 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011966199999999999 2024-09-18T04:39:38.178 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.178 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.178 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:38.178 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599358+0000", 2024-09-18T04:39:38.178 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0038486689999999999 2024-09-18T04:39:38.178 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.178 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.178 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:38.178 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606859+0000", 2024-09-18T04:39:38.178 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0075010789999999999 2024-09-18T04:39:38.178 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.178 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.178 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:38.178 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606887+0000", 2024-09-18T04:39:38.179 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7773e-05 2024-09-18T04:39:38.179 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.179 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.179 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:38.179 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606906+0000", 2024-09-18T04:39:38.179 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9417000000000001e-05 2024-09-18T04:39:38.179 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:38.179 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:38.179 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:38.179 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.179 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.179 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:106 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:38.179 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.584425+0000", 2024-09-18T04:39:38.179 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 41.490671618, 2024-09-18T04:39:38.180 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.022529293999999998, 2024-09-18T04:39:38.180 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:38.180 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:38.180 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:38.180 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:38.180 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:38.180 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 106 2024-09-18T04:39:38.181 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.181 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:38.181 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.181 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:38.181 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584425+0000", 2024-09-18T04:39:38.181 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:38.181 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.181 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.181 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:38.181 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584425+0000", 2024-09-18T04:39:38.181 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:38.181 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.182 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.182 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:38.182 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584424+0000", 2024-09-18T04:39:38.182 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:38.182 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.182 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.182 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:38.182 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584430+0000", 2024-09-18T04:39:38.182 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6389999999999997e-06 2024-09-18T04:39:38.182 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.182 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.182 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:38.182 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584456+0000", 2024-09-18T04:39:38.182 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6188999999999999e-05 2024-09-18T04:39:38.183 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.183 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.183 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:38.183 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584463+0000", 2024-09-18T04:39:38.183 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7340000000000002e-06 2024-09-18T04:39:38.183 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.183 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.183 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:38.183 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597537+0000", 2024-09-18T04:39:38.184 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013074161000000001 2024-09-18T04:39:38.184 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.184 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.185 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:38.185 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597598+0000", 2024-09-18T04:39:38.185 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0930000000000001e-05 2024-09-18T04:39:38.185 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.186 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.186 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:38.186 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597922+0000", 2024-09-18T04:39:38.186 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032473000000000002 2024-09-18T04:39:38.187 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.187 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.187 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:38.187 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.601103+0000", 2024-09-18T04:39:38.187 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0031809849999999999 2024-09-18T04:39:38.187 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.187 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.187 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:38.187 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606912+0000", 2024-09-18T04:39:38.187 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058083550000000003 2024-09-18T04:39:38.187 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.187 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.187 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:38.188 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606935+0000", 2024-09-18T04:39:38.188 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2759999999999999e-05 2024-09-18T04:39:38.188 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.188 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.188 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:38.188 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606954+0000", 2024-09-18T04:39:38.188 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9602999999999998e-05 2024-09-18T04:39:38.188 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:38.188 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:38.189 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:38.189 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.189 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.189 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:109 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:38.189 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.585231+0000", 2024-09-18T04:39:38.190 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 41.489864998000002, 2024-09-18T04:39:38.190 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.021767496000000001, 2024-09-18T04:39:38.190 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:38.190 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:38.190 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:38.191 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:38.191 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:38.191 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 109 2024-09-18T04:39:38.191 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.191 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:38.191 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.191 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:38.192 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585231+0000", 2024-09-18T04:39:38.192 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:38.192 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.192 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.192 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:38.192 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585231+0000", 2024-09-18T04:39:38.192 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:38.192 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.192 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.192 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:38.192 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585230+0000", 2024-09-18T04:39:38.192 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:38.193 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.193 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.193 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:38.193 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585237+0000", 2024-09-18T04:39:38.193 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.618e-06 2024-09-18T04:39:38.193 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.193 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.193 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:38.193 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585256+0000", 2024-09-18T04:39:38.193 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9048000000000001e-05 2024-09-18T04:39:38.193 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.194 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.194 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:38.194 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585274+0000", 2024-09-18T04:39:38.194 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8153000000000001e-05 2024-09-18T04:39:38.194 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.194 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.194 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:38.194 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.598284+0000", 2024-09-18T04:39:38.194 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013009682999999999 2024-09-18T04:39:38.194 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.194 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.194 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:38.195 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.598383+0000", 2024-09-18T04:39:38.195 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.8722999999999997e-05 2024-09-18T04:39:38.195 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.195 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.195 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:38.195 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.598829+0000", 2024-09-18T04:39:38.195 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00044623999999999998 2024-09-18T04:39:38.195 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.195 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.195 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:38.196 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.601149+0000", 2024-09-18T04:39:38.196 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0023204150000000002 2024-09-18T04:39:38.196 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.196 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.196 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:38.196 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606960+0000", 2024-09-18T04:39:38.196 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058106390000000003 2024-09-18T04:39:38.196 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.196 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.197 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:38.197 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606982+0000", 2024-09-18T04:39:38.197 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2252e-05 2024-09-18T04:39:38.197 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.197 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.197 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:38.197 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606999+0000", 2024-09-18T04:39:38.197 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6871000000000002e-05 2024-09-18T04:39:38.197 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:38.197 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:38.198 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:38.198 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.198 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.198 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:110 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:38.198 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.586640+0000", 2024-09-18T04:39:38.198 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 41.488456679000002, 2024-09-18T04:39:38.198 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020400911000000001, 2024-09-18T04:39:38.198 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:38.198 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:38.198 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:38.198 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:38.199 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:38.199 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 110 2024-09-18T04:39:38.199 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.199 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:38.199 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.199 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:38.199 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586640+0000", 2024-09-18T04:39:38.199 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:38.199 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.199 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.199 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:38.199 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586640+0000", 2024-09-18T04:39:38.199 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:38.200 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.200 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.200 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:38.200 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586639+0000", 2024-09-18T04:39:38.200 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:38.200 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.200 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.200 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:38.200 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586652+0000", 2024-09-18T04:39:38.200 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2482e-05 2024-09-18T04:39:38.200 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.200 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.201 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:38.201 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586660+0000", 2024-09-18T04:39:38.201 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1659999999999994e-06 2024-09-18T04:39:38.201 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.201 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.201 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:38.201 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586666+0000", 2024-09-18T04:39:38.201 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6900000000000003e-06 2024-09-18T04:39:38.201 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.201 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.201 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:38.201 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599381+0000", 2024-09-18T04:39:38.201 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012714328 2024-09-18T04:39:38.201 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.201 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.202 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:38.202 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599436+0000", 2024-09-18T04:39:38.202 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4802999999999997e-05 2024-09-18T04:39:38.202 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.202 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.202 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:38.202 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599629+0000", 2024-09-18T04:39:38.202 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000193921 2024-09-18T04:39:38.202 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.202 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.202 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:38.202 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.601174+0000", 2024-09-18T04:39:38.202 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001544857 2024-09-18T04:39:38.202 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.203 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.203 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:38.203 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.607004+0000", 2024-09-18T04:39:38.203 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058300820000000003 2024-09-18T04:39:38.203 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.203 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.203 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:38.203 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.607024+0000", 2024-09-18T04:39:38.203 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9888e-05 2024-09-18T04:39:38.203 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.203 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.203 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:38.203 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.607041+0000", 2024-09-18T04:39:38.203 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6450999999999999e-05 2024-09-18T04:39:38.203 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:38.204 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:38.204 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:38.204 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:38.204 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:38.204 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:38.206 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:39:38.206 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:38.200+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:38.307 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:38.304+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:38.321 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:38.321 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:38.321 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:39:38.321 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:38.331 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:39:38.408 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:38.404+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:38.438 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:38.438 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:38.438 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:39:38.438 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:39:38.438 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:38.448 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:39:38.466 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T04:39:38.508 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:38.500+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:38.550 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:38.551 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:39:38.551 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:39:38.551 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:39:38.551 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.551 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.5431.0:1 3.3 3:d80af3cb:::rbd_id.testrbdnbd17342:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-18T04:39:38.551 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:13.542818+0000", 2024-09-18T04:39:38.551 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 25.005834614000001, 2024-09-18T04:39:38.551 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048493400000000002, 2024-09-18T04:39:38.551 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:38.552 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:39:38.552 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:38.552 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.5431", 2024-09-18T04:39:38.552 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3147136083", 2024-09-18T04:39:38.552 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 1 2024-09-18T04:39:38.552 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.552 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:38.552 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.552 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:38.552 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542818+0000", 2024-09-18T04:39:38.552 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:38.552 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.552 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.553 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:38.553 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542818+0000", 2024-09-18T04:39:38.553 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:38.553 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.553 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.553 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:38.553 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542816+0000", 2024-09-18T04:39:38.553 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:38.553 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.553 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.553 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:38.553 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542828+0000", 2024-09-18T04:39:38.553 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2699e-05 2024-09-18T04:39:38.554 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.554 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.554 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:38.554 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542852+0000", 2024-09-18T04:39:38.554 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3176e-05 2024-09-18T04:39:38.554 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.554 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.554 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:38.554 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542874+0000", 2024-09-18T04:39:38.554 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1967000000000002e-05 2024-09-18T04:39:38.554 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.554 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.554 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:38.554 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542984+0000", 2024-09-18T04:39:38.555 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011026099999999999 2024-09-18T04:39:38.555 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.555 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.555 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:38.555 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543161+0000", 2024-09-18T04:39:38.555 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017720299999999999 2024-09-18T04:39:38.555 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.555 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.555 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:38.555 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543303+0000", 2024-09-18T04:39:38.555 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014174700000000001 2024-09-18T04:39:38.555 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:38.555 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:38.555 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:38.556 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.556 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.556 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.5431.0:2 3.3 3:d80af3cb:::rbd_id.testrbdnbd17342:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-18T04:39:38.556 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:13.543564+0000", 2024-09-18T04:39:38.556 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 25.005088128000001, 2024-09-18T04:39:38.556 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010399929999999999, 2024-09-18T04:39:38.556 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:38.556 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:39:38.556 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:38.556 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.5431", 2024-09-18T04:39:38.556 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3147136083", 2024-09-18T04:39:38.556 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 2 2024-09-18T04:39:38.556 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.557 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:38.557 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.557 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:38.557 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543564+0000", 2024-09-18T04:39:38.557 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:38.557 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.557 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.557 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:38.557 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543564+0000", 2024-09-18T04:39:38.557 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:38.558 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.558 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.558 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:38.558 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543563+0000", 2024-09-18T04:39:38.558 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:38.558 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.558 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.558 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:38.558 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543571+0000", 2024-09-18T04:39:38.558 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4270000000000008e-06 2024-09-18T04:39:38.558 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.558 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.558 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:38.558 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543591+0000", 2024-09-18T04:39:38.559 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.929e-05 2024-09-18T04:39:38.559 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.559 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.559 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:38.559 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543600+0000", 2024-09-18T04:39:38.559 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.1390000000000001e-06 2024-09-18T04:39:38.559 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.559 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.559 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:38.559 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543724+0000", 2024-09-18T04:39:38.559 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012458400000000001 2024-09-18T04:39:38.559 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.559 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.560 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:38.560 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543968+0000", 2024-09-18T04:39:38.560 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00024323199999999999 2024-09-18T04:39:38.560 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:38.560 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:38.560 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:38.560 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.544604+0000", 2024-09-18T04:39:38.560 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00063675799999999996 2024-09-18T04:39:38.560 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:38.560 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:38.560 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:38.560 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:38.560 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:38.561 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:38.562 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:39:38.609 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:38.600+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:38.623 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:39:38.648 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:39:38.670 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:38.670 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:38.670 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:39:38.670 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:38.673 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:39:38.680 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:39:38.687 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-18T04:39:38.696 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:39:38.710 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:38.708+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:38.729 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:39:38.730 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:39:38.730 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:39:38.730 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-18T04:39:38.730 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:39:38.730 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:39:38.730 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:39:38.733 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T04:39:38.734 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T04:39:38.811 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:38.804+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:38.831 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:38.831 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:38.843 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:39:38.913 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:38.908+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:38.948 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:38.948 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:38.958 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:39:39.014 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:39.008+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:39.068 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:39.068 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:39.078 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:39:39.708 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-09-18T04:39:39.710 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:39.704+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:39.809 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:39.804+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:39.825 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:39.826 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:39.836 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:39:39.911 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:39.908+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:39.948 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:39.948 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:39.956 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:39:40.011 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:40.008+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:40.061 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 2] out_osds: [4] dead_osds: [2] live_osds: [0, 1, 3, 4, 5] 2024-09-18T04:39:40.061 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T04:39:40.061 INFO:tasks.thrashosds.thrasher:Reviving osd 2 2024-09-18T04:39:40.062 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-18T04:39:40.062 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-18T04:39:40.063 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:40.064 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:40.065 INFO:tasks.ceph.osd.2:Started 2024-09-18T04:39:40.065 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:39:40.074 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:39:40.123 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:40.120+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:40.165 INFO:teuthology.orchestra.run.smithi028.stderr:no valid command found; 10 closest matches: 2024-09-18T04:39:40.166 INFO:teuthology.orchestra.run.smithi028.stderr:0 2024-09-18T04:39:40.166 INFO:teuthology.orchestra.run.smithi028.stderr:1 2024-09-18T04:39:40.167 INFO:teuthology.orchestra.run.smithi028.stderr:2 2024-09-18T04:39:40.167 INFO:teuthology.orchestra.run.smithi028.stderr:abort 2024-09-18T04:39:40.167 INFO:teuthology.orchestra.run.smithi028.stderr:assert 2024-09-18T04:39:40.167 INFO:teuthology.orchestra.run.smithi028.stderr:config diff 2024-09-18T04:39:40.167 INFO:teuthology.orchestra.run.smithi028.stderr:config diff get 2024-09-18T04:39:40.167 INFO:teuthology.orchestra.run.smithi028.stderr:config get 2024-09-18T04:39:40.167 INFO:teuthology.orchestra.run.smithi028.stderr:config help [] 2024-09-18T04:39:40.167 INFO:teuthology.orchestra.run.smithi028.stderr:config set ... 2024-09-18T04:39:40.167 INFO:teuthology.orchestra.run.smithi028.stderr:admin_socket: invalid command 2024-09-18T04:39:40.168 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-18T04:39:40.168 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-09-18T04:39:40.214 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:40.208+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:40.230 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:40.230 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:40.240 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:39:40.315 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:40.308+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:40.348 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:40.349 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:40.359 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:39:40.416 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:40.408+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:40.474 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:40.474 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:40.484 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:39:40.516 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:40.508+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:40.589 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:40.589 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:40.598 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:39:40.642 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:40.636+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:40.700 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:40.696+0000 7fb3e7610880 -1 Falling back to public interface 2024-09-18T04:39:40.702 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:40.703 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:40.712 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:39:40.718 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:40.712+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:40.735 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T04:39:40.815 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:40.815 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:40.818 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:40.812+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:40.824 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:39:40.895 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:39:40.917 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:39:40.918 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:40.912+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:40.941 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:39:40.964 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:39:40.979 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:40.979 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:40.979 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:39:40.979 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:39:40.979 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:40.989 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:39:40.990 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:39:40.990 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:39:40.990 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:39:40.990 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-18T04:39:40.990 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:39:40.990 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:39:40.990 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:39:40.993 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T04:39:40.993 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T04:39:41.034 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:41.028+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:41.092 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:41.092 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:39:41.092 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:39:41.092 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:39:41.092 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.092 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.5431.0:1 3.3 3:d80af3cb:::rbd_id.testrbdnbd17342:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-18T04:39:41.092 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:13.542818+0000", 2024-09-18T04:39:41.093 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 27.547127880000001, 2024-09-18T04:39:41.093 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048493400000000002, 2024-09-18T04:39:41.093 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:41.093 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:39:41.093 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:41.093 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.5431", 2024-09-18T04:39:41.093 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3147136083", 2024-09-18T04:39:41.093 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 1 2024-09-18T04:39:41.093 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.093 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:41.093 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.093 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:41.093 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542818+0000", 2024-09-18T04:39:41.093 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:41.094 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.094 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.094 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:41.094 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542818+0000", 2024-09-18T04:39:41.094 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:41.094 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.094 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.094 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:41.094 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542816+0000", 2024-09-18T04:39:41.094 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:41.094 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.094 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.094 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:41.094 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542828+0000", 2024-09-18T04:39:41.095 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2699e-05 2024-09-18T04:39:41.095 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.095 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.095 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:41.095 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542852+0000", 2024-09-18T04:39:41.095 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3176e-05 2024-09-18T04:39:41.095 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.095 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.095 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:41.095 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542874+0000", 2024-09-18T04:39:41.095 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1967000000000002e-05 2024-09-18T04:39:41.095 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.095 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.095 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:41.095 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542984+0000", 2024-09-18T04:39:41.096 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011026099999999999 2024-09-18T04:39:41.096 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.096 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.096 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:41.096 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543161+0000", 2024-09-18T04:39:41.096 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017720299999999999 2024-09-18T04:39:41.096 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.096 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.096 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:41.096 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543303+0000", 2024-09-18T04:39:41.096 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014174700000000001 2024-09-18T04:39:41.096 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:41.096 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:41.096 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:41.097 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.097 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.097 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.5431.0:2 3.3 3:d80af3cb:::rbd_id.testrbdnbd17342:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-18T04:39:41.097 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:13.543564+0000", 2024-09-18T04:39:41.097 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 27.546381394000001, 2024-09-18T04:39:41.097 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010399929999999999, 2024-09-18T04:39:41.097 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:41.097 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:39:41.097 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:41.097 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.5431", 2024-09-18T04:39:41.097 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3147136083", 2024-09-18T04:39:41.097 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 2 2024-09-18T04:39:41.097 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.097 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:41.098 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.098 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:41.098 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543564+0000", 2024-09-18T04:39:41.098 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:41.098 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.098 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.098 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:41.098 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543564+0000", 2024-09-18T04:39:41.098 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:41.098 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.098 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.098 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:41.098 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543563+0000", 2024-09-18T04:39:41.098 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:41.098 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.099 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.099 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:41.099 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543571+0000", 2024-09-18T04:39:41.099 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4270000000000008e-06 2024-09-18T04:39:41.099 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.099 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.099 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:41.099 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543591+0000", 2024-09-18T04:39:41.099 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.929e-05 2024-09-18T04:39:41.099 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.099 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.099 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:41.099 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543600+0000", 2024-09-18T04:39:41.099 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.1390000000000001e-06 2024-09-18T04:39:41.099 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.100 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.100 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:41.100 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543724+0000", 2024-09-18T04:39:41.100 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012458400000000001 2024-09-18T04:39:41.100 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.100 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.100 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:41.100 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543968+0000", 2024-09-18T04:39:41.100 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00024323199999999999 2024-09-18T04:39:41.100 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.100 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.100 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:41.100 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.544604+0000", 2024-09-18T04:39:41.100 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00063675799999999996 2024-09-18T04:39:41.100 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:41.101 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:41.101 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:41.101 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:41.101 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:41.101 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:41.102 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:39:41.119 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:41.112+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:41.209 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:41.209 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:41.209 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:39:41.209 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:41.218 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:39:41.262 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:41.256+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:41.320 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:41.312+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:41.370 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:41.370 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:41.371 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:39:41.371 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:39:41.371 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:41.380 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:39:41.421 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:41.412+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:41.488 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:41.488 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:39:41.488 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:39:41.488 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:39:41.488 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.488 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-18T04:39:41.488 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:39:41.488 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 127.012205971, 2024-09-18T04:39:41.488 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0095781189999999995, 2024-09-18T04:39:41.488 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:41.488 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:41.489 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:41.489 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:39:41.489 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:39:41.489 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 1 2024-09-18T04:39:41.489 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.489 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:41.489 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.489 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:41.489 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:39:41.489 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:41.489 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.489 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.489 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:41.489 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:39:41.490 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:41.490 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.490 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.490 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:41.490 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469862+0000", 2024-09-18T04:39:41.490 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:41.490 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.490 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.490 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:41.490 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469879+0000", 2024-09-18T04:39:41.490 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7431999999999998e-05 2024-09-18T04:39:41.490 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.490 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.490 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:41.491 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469896+0000", 2024-09-18T04:39:41.491 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6742999999999999e-05 2024-09-18T04:39:41.491 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.491 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.491 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:41.491 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469915+0000", 2024-09-18T04:39:41.491 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8991000000000002e-05 2024-09-18T04:39:41.491 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.491 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.491 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:41.491 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470397+0000", 2024-09-18T04:39:41.491 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048167799999999998 2024-09-18T04:39:41.491 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.491 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.491 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-09-18T04:39:41.492 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470412+0000", 2024-09-18T04:39:41.492 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5526e-05 2024-09-18T04:39:41.492 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.492 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.492 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:41.492 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473279+0000", 2024-09-18T04:39:41.492 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0028667469999999998 2024-09-18T04:39:41.492 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.492 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.492 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:41.492 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473441+0000", 2024-09-18T04:39:41.492 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000162082 2024-09-18T04:39:41.492 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.492 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.493 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:41.493 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473573+0000", 2024-09-18T04:39:41.493 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000132032 2024-09-18T04:39:41.493 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.493 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.493 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:41.493 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474826+0000", 2024-09-18T04:39:41.493 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001252717 2024-09-18T04:39:41.493 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.493 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.493 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:41.493 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479307+0000", 2024-09-18T04:39:41.493 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044813220000000003 2024-09-18T04:39:41.493 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.494 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.494 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:41.494 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479366+0000", 2024-09-18T04:39:41.494 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9024999999999998e-05 2024-09-18T04:39:41.494 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.494 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.494 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:41.494 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479443+0000", 2024-09-18T04:39:41.494 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6339000000000002e-05 2024-09-18T04:39:41.494 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:41.494 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:41.494 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:41.494 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.494 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.495 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-18T04:39:41.495 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:39:41.495 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 124.99919755099999, 2024-09-18T04:39:41.495 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082699560000000002, 2024-09-18T04:39:41.495 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:41.495 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:41.495 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:41.495 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:39:41.495 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:39:41.495 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 7 2024-09-18T04:39:41.495 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.495 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:41.495 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.495 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:41.495 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:39:41.496 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:41.496 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.496 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.496 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:41.496 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:39:41.496 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:41.496 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.496 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.496 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:41.496 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482870+0000", 2024-09-18T04:39:41.496 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999967 2024-09-18T04:39:41.496 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.496 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.496 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:41.496 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482889+0000", 2024-09-18T04:39:41.497 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.965e-05 2024-09-18T04:39:41.497 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.497 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.497 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:41.497 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482938+0000", 2024-09-18T04:39:41.497 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8548999999999999e-05 2024-09-18T04:39:41.497 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.497 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.497 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:41.497 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482963+0000", 2024-09-18T04:39:41.497 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5075999999999998e-05 2024-09-18T04:39:41.497 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.497 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.497 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:41.498 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483121+0000", 2024-09-18T04:39:41.498 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015819400000000001 2024-09-18T04:39:41.498 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.498 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.498 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op must wait for map", 2024-09-18T04:39:41.498 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483138+0000", 2024-09-18T04:39:41.498 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7068999999999999e-05 2024-09-18T04:39:41.498 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.498 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.498 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:41.498 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488666+0000", 2024-09-18T04:39:41.498 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0055282049999999996 2024-09-18T04:39:41.498 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.498 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.498 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:41.499 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488783+0000", 2024-09-18T04:39:41.499 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000116737 2024-09-18T04:39:41.499 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.499 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.499 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:41.499 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488991+0000", 2024-09-18T04:39:41.499 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020781400000000001 2024-09-18T04:39:41.499 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.499 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.499 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:41.499 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490554+0000", 2024-09-18T04:39:41.499 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0015628370000000001 2024-09-18T04:39:41.499 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.499 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.499 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:41.500 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491019+0000", 2024-09-18T04:39:41.500 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000465608 2024-09-18T04:39:41.500 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.500 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.500 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:41.500 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491087+0000", 2024-09-18T04:39:41.500 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7988e-05 2024-09-18T04:39:41.500 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.500 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.500 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:41.500 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491143+0000", 2024-09-18T04:39:41.500 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5538999999999998e-05 2024-09-18T04:39:41.500 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:41.500 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:41.501 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:41.501 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.501 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.501 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:34 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:41.501 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.261758+0000", 2024-09-18T04:39:41.501 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 59.220312399999997, 2024-09-18T04:39:41.501 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.042024191000000002, 2024-09-18T04:39:41.501 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:41.501 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:41.501 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:41.501 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:41.501 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:41.501 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 34 2024-09-18T04:39:41.501 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.502 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:41.502 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.502 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:41.502 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.261758+0000", 2024-09-18T04:39:41.502 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:41.502 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.502 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.502 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:41.502 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.261758+0000", 2024-09-18T04:39:41.502 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:41.502 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.502 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.502 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:41.502 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.261756+0000", 2024-09-18T04:39:41.503 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:41.503 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.503 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.503 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:41.503 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.270785+0000", 2024-09-18T04:39:41.503 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0090292340000000006 2024-09-18T04:39:41.503 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.503 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.503 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:41.503 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.270890+0000", 2024-09-18T04:39:41.503 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000104541 2024-09-18T04:39:41.503 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.503 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.503 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:41.503 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.270991+0000", 2024-09-18T04:39:41.504 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000100778 2024-09-18T04:39:41.504 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.504 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.504 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:41.504 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.271157+0000", 2024-09-18T04:39:41.504 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016653999999999999 2024-09-18T04:39:41.504 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.504 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.504 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:41.504 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.271591+0000", 2024-09-18T04:39:41.504 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00043409500000000001 2024-09-18T04:39:41.504 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.505 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.505 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:41.505 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272017+0000", 2024-09-18T04:39:41.505 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000425769 2024-09-18T04:39:41.505 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.505 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.505 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:41.505 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.294246+0000", 2024-09-18T04:39:41.505 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.022229237999999998 2024-09-18T04:39:41.505 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.505 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.505 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:41.505 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303571+0000", 2024-09-18T04:39:41.505 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0093250239999999995 2024-09-18T04:39:41.505 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.506 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.506 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:41.506 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303705+0000", 2024-09-18T04:39:41.506 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000133831 2024-09-18T04:39:41.506 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.506 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.506 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:41.506 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303782+0000", 2024-09-18T04:39:41.506 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7156000000000005e-05 2024-09-18T04:39:41.506 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:41.507 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:41.507 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:41.507 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.507 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.507 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:36 2.7 e51/32 2:fe1b02ad:::rbd_data.125fc09df6d1.0000000000000002:head v 51'1, mlcod=0'0)", 2024-09-18T04:39:41.507 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.273441+0000", 2024-09-18T04:39:41.507 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 59.208629418999998, 2024-09-18T04:39:41.507 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.032400768000000003, 2024-09-18T04:39:41.507 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:41.507 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:41.507 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:41.508 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.508 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:41.508 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273441+0000", 2024-09-18T04:39:41.508 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:41.508 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.508 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.508 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:41.508 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273441+0000", 2024-09-18T04:39:41.508 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:41.508 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.508 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.509 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:41.509 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273440+0000", 2024-09-18T04:39:41.509 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:41.509 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.509 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.509 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:41.509 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278723+0000", 2024-09-18T04:39:41.509 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0052829909999999999 2024-09-18T04:39:41.509 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.509 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.509 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:41.509 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278752+0000", 2024-09-18T04:39:41.509 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9178000000000001e-05 2024-09-18T04:39:41.509 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.509 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.510 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:41.510 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278761+0000", 2024-09-18T04:39:41.510 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7900000000000005e-06 2024-09-18T04:39:41.510 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.510 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.510 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:41.510 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278864+0000", 2024-09-18T04:39:41.510 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000102699 2024-09-18T04:39:41.510 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.510 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.510 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:41.510 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278900+0000", 2024-09-18T04:39:41.510 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6001999999999999e-05 2024-09-18T04:39:41.510 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.511 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.511 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:41.511 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.305793+0000", 2024-09-18T04:39:41.511 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.026893344 2024-09-18T04:39:41.511 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.511 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.511 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:41.511 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.305842+0000", 2024-09-18T04:39:41.511 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8720000000000001e-05 2024-09-18T04:39:41.511 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:41.511 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:41.511 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:41.511 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.511 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.511 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:39 2.7 e51/32 2:e9c909c5:::rbd_data.125fc09df6d1.0000000000000005:head v 51'2, mlcod=0'0)", 2024-09-18T04:39:41.512 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.283175+0000", 2024-09-18T04:39:41.512 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 59.198895729999997, 2024-09-18T04:39:41.512 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.022864102000000001, 2024-09-18T04:39:41.512 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:41.512 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:41.512 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:41.512 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.512 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:41.512 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283175+0000", 2024-09-18T04:39:41.512 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:41.512 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.512 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.512 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:41.512 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283175+0000", 2024-09-18T04:39:41.513 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:41.513 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.513 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.513 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:41.513 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283173+0000", 2024-09-18T04:39:41.513 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:41.513 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.513 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.513 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:41.513 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.285951+0000", 2024-09-18T04:39:41.513 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0027784540000000001 2024-09-18T04:39:41.513 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.513 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.513 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:41.514 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.285970+0000", 2024-09-18T04:39:41.514 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8855000000000002e-05 2024-09-18T04:39:41.514 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.514 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.514 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:41.514 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.285984+0000", 2024-09-18T04:39:41.514 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4252e-05 2024-09-18T04:39:41.514 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.514 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.514 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:41.514 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.286041+0000", 2024-09-18T04:39:41.514 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6431999999999998e-05 2024-09-18T04:39:41.514 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.514 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.514 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:41.515 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.286086+0000", 2024-09-18T04:39:41.515 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5423000000000002e-05 2024-09-18T04:39:41.515 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.515 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.515 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:41.515 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.305998+0000", 2024-09-18T04:39:41.515 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019911697999999999 2024-09-18T04:39:41.515 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.515 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.515 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:41.515 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306039+0000", 2024-09-18T04:39:41.515 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0973999999999998e-05 2024-09-18T04:39:41.515 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:41.515 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:41.516 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:41.516 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.516 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.516 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:50 2.3 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:41.516 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.312563+0000", 2024-09-18T04:39:41.516 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 59.169507703000001, 2024-09-18T04:39:41.516 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.032636001999999997, 2024-09-18T04:39:41.516 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:41.516 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:41.516 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:41.516 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:41.516 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:41.516 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 50 2024-09-18T04:39:41.517 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.517 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:41.517 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.517 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:41.517 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.312563+0000", 2024-09-18T04:39:41.517 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:41.517 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.517 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.517 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:41.517 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.312563+0000", 2024-09-18T04:39:41.517 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:41.517 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.517 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.517 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:41.517 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.312561+0000", 2024-09-18T04:39:41.518 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:41.518 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.518 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.518 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:41.518 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321709+0000", 2024-09-18T04:39:41.518 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0091486650000000003 2024-09-18T04:39:41.518 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.518 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.518 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:41.518 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321739+0000", 2024-09-18T04:39:41.518 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9666e-05 2024-09-18T04:39:41.518 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.518 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.518 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:41.519 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321751+0000", 2024-09-18T04:39:41.519 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2086000000000001e-05 2024-09-18T04:39:41.519 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.519 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.519 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:41.519 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321833+0000", 2024-09-18T04:39:41.519 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1842999999999996e-05 2024-09-18T04:39:41.519 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.519 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.519 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:41.519 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321981+0000", 2024-09-18T04:39:41.519 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000147441 2024-09-18T04:39:41.519 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.519 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.520 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:41.520 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.322109+0000", 2024-09-18T04:39:41.520 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012851200000000001 2024-09-18T04:39:41.520 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.520 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.520 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:41.520 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.332829+0000", 2024-09-18T04:39:41.520 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010720225999999999 2024-09-18T04:39:41.520 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.520 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.520 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:41.520 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.345039+0000", 2024-09-18T04:39:41.520 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012209831000000001 2024-09-18T04:39:41.521 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.521 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.521 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:41.521 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.345120+0000", 2024-09-18T04:39:41.521 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1145999999999999e-05 2024-09-18T04:39:41.521 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.521 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.521 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:41.521 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.345199+0000", 2024-09-18T04:39:41.521 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8546000000000003e-05 2024-09-18T04:39:41.521 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:41.521 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:41.521 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:41.521 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.521 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.522 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:30 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:41.522 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.516122+0000", 2024-09-18T04:39:41.522 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 58.965948541000003, 2024-09-18T04:39:41.522 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0091865030000000004, 2024-09-18T04:39:41.522 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:41.522 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:41.522 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:41.522 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:41.522 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:41.522 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 30 2024-09-18T04:39:41.522 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.522 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:41.522 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.523 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:41.523 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516122+0000", 2024-09-18T04:39:41.523 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:41.523 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.523 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.523 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:41.523 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516122+0000", 2024-09-18T04:39:41.523 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:41.523 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.523 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.523 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:41.523 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516120+0000", 2024-09-18T04:39:41.524 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:41.524 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.524 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.524 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:41.524 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516129+0000", 2024-09-18T04:39:41.524 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.85e-06 2024-09-18T04:39:41.524 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.524 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.524 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:41.524 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516155+0000", 2024-09-18T04:39:41.524 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5882000000000001e-05 2024-09-18T04:39:41.524 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.524 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.524 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:41.524 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516175+0000", 2024-09-18T04:39:41.525 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.022e-05 2024-09-18T04:39:41.525 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.525 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.525 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:41.525 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519780+0000", 2024-09-18T04:39:41.525 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003604187 2024-09-18T04:39:41.525 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.525 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.525 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:41.525 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519964+0000", 2024-09-18T04:39:41.525 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00018391999999999999 2024-09-18T04:39:41.525 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.525 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.525 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:41.526 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.520794+0000", 2024-09-18T04:39:41.526 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00083078100000000005 2024-09-18T04:39:41.526 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.526 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.526 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:41.526 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.524129+0000", 2024-09-18T04:39:41.526 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033342419999999998 2024-09-18T04:39:41.526 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.526 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.526 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:41.526 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525239+0000", 2024-09-18T04:39:41.526 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001110778 2024-09-18T04:39:41.526 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.527 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.527 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:41.527 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525277+0000", 2024-09-18T04:39:41.527 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7596999999999999e-05 2024-09-18T04:39:41.527 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.527 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.527 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:41.527 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525308+0000", 2024-09-18T04:39:41.527 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.1535999999999998e-05 2024-09-18T04:39:41.527 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:41.528 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:41.528 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:41.528 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.528 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.528 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:32 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:41.528 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.519217+0000", 2024-09-18T04:39:41.528 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 58.962853221000003, 2024-09-18T04:39:41.528 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082266370000000002, 2024-09-18T04:39:41.528 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:41.528 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:41.529 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:41.529 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:41.529 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:41.529 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 32 2024-09-18T04:39:41.529 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.529 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:41.529 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.529 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:41.529 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519217+0000", 2024-09-18T04:39:41.529 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:41.529 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.529 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.529 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:41.529 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519217+0000", 2024-09-18T04:39:41.530 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:41.530 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.530 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.530 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:41.530 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519215+0000", 2024-09-18T04:39:41.530 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:41.530 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.530 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.530 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:41.530 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519229+0000", 2024-09-18T04:39:41.530 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3329999999999999e-05 2024-09-18T04:39:41.530 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.530 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.530 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:41.530 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519268+0000", 2024-09-18T04:39:41.531 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9533000000000002e-05 2024-09-18T04:39:41.531 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.531 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.531 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:41.531 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519285+0000", 2024-09-18T04:39:41.531 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6446000000000001e-05 2024-09-18T04:39:41.531 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.531 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.531 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:41.531 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.522358+0000", 2024-09-18T04:39:41.531 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030734310000000002 2024-09-18T04:39:41.531 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.531 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.531 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:41.532 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.522551+0000", 2024-09-18T04:39:41.532 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019317600000000001 2024-09-18T04:39:41.532 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.532 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.532 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:41.532 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.523262+0000", 2024-09-18T04:39:41.532 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00071069999999999998 2024-09-18T04:39:41.532 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.532 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.532 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:41.532 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525318+0000", 2024-09-18T04:39:41.532 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0020561469999999999 2024-09-18T04:39:41.532 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.532 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.532 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:41.533 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.527376+0000", 2024-09-18T04:39:41.533 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0020584969999999998 2024-09-18T04:39:41.533 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.533 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.533 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:41.533 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.527416+0000", 2024-09-18T04:39:41.533 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9054000000000003e-05 2024-09-18T04:39:41.533 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.533 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.533 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:41.533 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.527444+0000", 2024-09-18T04:39:41.533 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8347000000000002e-05 2024-09-18T04:39:41.534 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:41.534 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:41.534 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:41.534 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.534 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.534 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:42 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:41.534 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.551160+0000", 2024-09-18T04:39:41.534 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 58.930910828999998, 2024-09-18T04:39:41.534 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010692757000000001, 2024-09-18T04:39:41.534 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:41.534 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:41.534 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:41.534 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:41.534 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:41.535 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 42 2024-09-18T04:39:41.535 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.535 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:41.535 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.535 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:41.535 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551160+0000", 2024-09-18T04:39:41.535 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:41.535 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.535 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.535 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:41.535 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551160+0000", 2024-09-18T04:39:41.535 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:41.535 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.535 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.536 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:41.536 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551159+0000", 2024-09-18T04:39:41.536 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:41.536 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.536 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.536 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:41.536 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551166+0000", 2024-09-18T04:39:41.536 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1029999999999996e-06 2024-09-18T04:39:41.536 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.536 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.536 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:41.536 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551177+0000", 2024-09-18T04:39:41.536 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1384e-05 2024-09-18T04:39:41.536 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.536 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.537 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:41.537 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551185+0000", 2024-09-18T04:39:41.537 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1990000000000006e-06 2024-09-18T04:39:41.537 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.537 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.537 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:41.537 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552021+0000", 2024-09-18T04:39:41.537 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00083574400000000003 2024-09-18T04:39:41.537 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.537 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.537 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:41.537 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552339+0000", 2024-09-18T04:39:41.537 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031771399999999997 2024-09-18T04:39:41.537 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.538 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.538 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:41.538 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.553140+0000", 2024-09-18T04:39:41.538 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00080088100000000003 2024-09-18T04:39:41.538 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.538 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.538 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:41.538 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.554585+0000", 2024-09-18T04:39:41.538 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001445801 2024-09-18T04:39:41.538 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.538 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.538 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:41.538 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.561711+0000", 2024-09-18T04:39:41.538 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071260109999999998 2024-09-18T04:39:41.538 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.539 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.539 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:41.539 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.561790+0000", 2024-09-18T04:39:41.539 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8967000000000005e-05 2024-09-18T04:39:41.539 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.539 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.539 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:41.539 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.561852+0000", 2024-09-18T04:39:41.539 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2045999999999995e-05 2024-09-18T04:39:41.539 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:41.539 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:41.539 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:41.539 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.539 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.540 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:58 2.7 e51/32 2:fc7c6cd3:ns::rbd_data.133e813ab5bc.000000000000000e:head v 51'7, mlcod=51'5)", 2024-09-18T04:39:41.540 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.605333+0000", 2024-09-18T04:39:41.540 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 58.876737624, 2024-09-18T04:39:41.540 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020085902999999999, 2024-09-18T04:39:41.540 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:41.540 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:41.540 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:41.540 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.540 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:41.540 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605333+0000", 2024-09-18T04:39:41.540 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:41.540 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.540 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.540 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:41.541 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605333+0000", 2024-09-18T04:39:41.541 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:41.541 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.541 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.541 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:41.541 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605332+0000", 2024-09-18T04:39:41.541 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:41.541 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.541 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.541 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:41.541 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608667+0000", 2024-09-18T04:39:41.541 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033350300000000001 2024-09-18T04:39:41.541 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.541 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.542 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:41.542 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608706+0000", 2024-09-18T04:39:41.542 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8951e-05 2024-09-18T04:39:41.542 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.542 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.542 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:41.542 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608725+0000", 2024-09-18T04:39:41.542 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9675999999999999e-05 2024-09-18T04:39:41.542 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.542 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.542 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:41.542 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608798+0000", 2024-09-18T04:39:41.542 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3135999999999999e-05 2024-09-18T04:39:41.542 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.542 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.543 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:41.543 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608824+0000", 2024-09-18T04:39:41.543 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5284000000000001e-05 2024-09-18T04:39:41.543 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.543 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.543 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:41.543 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625370+0000", 2024-09-18T04:39:41.543 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016545975000000001 2024-09-18T04:39:41.543 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.543 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.543 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:41.543 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625419+0000", 2024-09-18T04:39:41.543 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9076999999999998e-05 2024-09-18T04:39:41.543 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:41.544 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:41.544 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:41.544 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.544 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.544 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:611 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:41.544 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.069561+0000", 2024-09-18T04:39:41.544 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 50.412509069999999, 2024-09-18T04:39:41.544 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0072330299999999997, 2024-09-18T04:39:41.544 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:41.544 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:41.544 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:41.544 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:39:41.544 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:41.544 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 611 2024-09-18T04:39:41.544 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.545 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:41.545 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.545 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:41.545 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.069561+0000", 2024-09-18T04:39:41.545 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:41.545 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.545 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.545 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:41.545 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.069561+0000", 2024-09-18T04:39:41.545 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:41.545 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.545 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.545 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:41.545 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.069559+0000", 2024-09-18T04:39:41.546 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:41.546 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.546 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.546 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:41.546 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070801+0000", 2024-09-18T04:39:41.546 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0012423359999999999 2024-09-18T04:39:41.546 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.546 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.546 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:41.546 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070828+0000", 2024-09-18T04:39:41.546 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6387e-05 2024-09-18T04:39:41.546 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.546 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.546 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:41.546 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070840+0000", 2024-09-18T04:39:41.547 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2689e-05 2024-09-18T04:39:41.547 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.547 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.547 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:41.547 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070990+0000", 2024-09-18T04:39:41.547 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014946800000000001 2024-09-18T04:39:41.547 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.547 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.547 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:41.547 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.071128+0000", 2024-09-18T04:39:41.547 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013796699999999999 2024-09-18T04:39:41.547 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.547 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.547 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:41.548 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.071318+0000", 2024-09-18T04:39:41.548 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000190106 2024-09-18T04:39:41.548 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.548 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.548 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:41.548 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.075711+0000", 2024-09-18T04:39:41.548 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.004393002 2024-09-18T04:39:41.548 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.548 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.548 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:41.548 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.076665+0000", 2024-09-18T04:39:41.548 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00095390399999999999 2024-09-18T04:39:41.548 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.548 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.549 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:41.549 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.076742+0000", 2024-09-18T04:39:41.549 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7112000000000003e-05 2024-09-18T04:39:41.549 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.549 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.549 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:41.549 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.076794+0000", 2024-09-18T04:39:41.549 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2475999999999997e-05 2024-09-18T04:39:41.549 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:41.549 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:41.549 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:41.549 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.549 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.549 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:687 2.3 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:41.550 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.487994+0000", 2024-09-18T04:39:41.550 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 49.994076087000003, 2024-09-18T04:39:41.550 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0074167690000000001, 2024-09-18T04:39:41.550 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:41.550 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:41.550 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:41.550 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:39:41.550 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:41.550 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 687 2024-09-18T04:39:41.550 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.550 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:41.550 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.550 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:41.550 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.487994+0000", 2024-09-18T04:39:41.550 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:41.551 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.551 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.551 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:41.551 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.487994+0000", 2024-09-18T04:39:41.551 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:41.551 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.551 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.551 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:41.551 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.487993+0000", 2024-09-18T04:39:41.551 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:41.551 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.551 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.551 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:41.551 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.488979+0000", 2024-09-18T04:39:41.551 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00098620700000000001 2024-09-18T04:39:41.552 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.552 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.552 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:41.552 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489002+0000", 2024-09-18T04:39:41.552 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2500000000000001e-05 2024-09-18T04:39:41.552 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.552 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.552 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:41.552 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489020+0000", 2024-09-18T04:39:41.552 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7691000000000001e-05 2024-09-18T04:39:41.552 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.552 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.552 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:41.552 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489091+0000", 2024-09-18T04:39:41.553 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1307e-05 2024-09-18T04:39:41.553 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.553 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.553 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:41.553 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489219+0000", 2024-09-18T04:39:41.553 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012766000000000001 2024-09-18T04:39:41.553 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.553 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.553 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:41.553 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489387+0000", 2024-09-18T04:39:41.553 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016855400000000001 2024-09-18T04:39:41.553 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.553 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.553 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:41.553 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495243+0000", 2024-09-18T04:39:41.554 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058563189999999996 2024-09-18T04:39:41.554 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.554 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.554 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:41.554 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495266+0000", 2024-09-18T04:39:41.554 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2810999999999999e-05 2024-09-18T04:39:41.554 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.554 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.554 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:41.554 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495332+0000", 2024-09-18T04:39:41.554 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5469999999999995e-05 2024-09-18T04:39:41.554 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.554 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.554 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:41.555 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495411+0000", 2024-09-18T04:39:41.555 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9395999999999997e-05 2024-09-18T04:39:41.555 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:41.555 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:41.555 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:41.555 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.555 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.555 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:689 2.3 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:41.555 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.500070+0000", 2024-09-18T04:39:41.555 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 49.982000270999997, 2024-09-18T04:39:41.555 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071096650000000003, 2024-09-18T04:39:41.555 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:41.555 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:41.555 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:41.555 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:39:41.555 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:41.556 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 689 2024-09-18T04:39:41.556 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.556 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:41.556 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.556 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:41.556 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.500070+0000", 2024-09-18T04:39:41.556 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:41.556 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.556 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.556 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:41.556 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.500070+0000", 2024-09-18T04:39:41.556 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:41.556 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.557 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.557 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:41.557 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.500069+0000", 2024-09-18T04:39:41.557 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:41.557 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.557 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.557 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:41.557 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501082+0000", 2024-09-18T04:39:41.557 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010136570000000001 2024-09-18T04:39:41.557 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.557 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.557 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:41.557 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501104+0000", 2024-09-18T04:39:41.557 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1883e-05 2024-09-18T04:39:41.557 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.558 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.558 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:41.558 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501115+0000", 2024-09-18T04:39:41.558 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0414000000000001e-05 2024-09-18T04:39:41.558 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.558 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.558 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:41.558 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501178+0000", 2024-09-18T04:39:41.558 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3128999999999998e-05 2024-09-18T04:39:41.558 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.558 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.558 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:41.558 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501257+0000", 2024-09-18T04:39:41.558 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9592999999999995e-05 2024-09-18T04:39:41.558 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.559 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.559 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:41.559 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501366+0000", 2024-09-18T04:39:41.559 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000108643 2024-09-18T04:39:41.559 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.559 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.559 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:41.559 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.504041+0000", 2024-09-18T04:39:41.559 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0026750889999999999 2024-09-18T04:39:41.559 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.559 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.559 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:41.559 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.507097+0000", 2024-09-18T04:39:41.559 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030556070000000001 2024-09-18T04:39:41.560 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.560 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.560 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:41.560 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.507140+0000", 2024-09-18T04:39:41.560 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3745000000000002e-05 2024-09-18T04:39:41.560 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.560 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.560 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:41.560 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.507180+0000", 2024-09-18T04:39:41.560 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9475999999999999e-05 2024-09-18T04:39:41.560 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:41.560 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:41.560 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:41.560 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.561 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.561 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4972.0:13 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:41.561 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.948263+0000", 2024-09-18T04:39:41.561 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 49.53380748, 2024-09-18T04:39:41.561 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0091358970000000005, 2024-09-18T04:39:41.561 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:41.561 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:39:41.561 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:41.561 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4972", 2024-09-18T04:39:41.561 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2280251472", 2024-09-18T04:39:41.561 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 13 2024-09-18T04:39:41.561 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.561 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:41.561 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.562 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:41.562 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948263+0000", 2024-09-18T04:39:41.562 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:41.562 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.562 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.562 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:41.562 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948263+0000", 2024-09-18T04:39:41.562 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:41.562 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.562 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.562 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:41.562 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948261+0000", 2024-09-18T04:39:41.562 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:41.562 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.562 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.563 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:41.563 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948273+0000", 2024-09-18T04:39:41.563 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1576999999999999e-05 2024-09-18T04:39:41.563 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.563 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.563 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:41.563 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948305+0000", 2024-09-18T04:39:41.563 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2209000000000003e-05 2024-09-18T04:39:41.563 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.563 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.563 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:41.563 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948325+0000", 2024-09-18T04:39:41.563 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.022e-05 2024-09-18T04:39:41.563 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.564 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.564 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:41.564 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948455+0000", 2024-09-18T04:39:41.564 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000129624 2024-09-18T04:39:41.564 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.564 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.564 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:41.564 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948598+0000", 2024-09-18T04:39:41.564 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014376699999999999 2024-09-18T04:39:41.564 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.564 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.564 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:41.564 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.957399+0000", 2024-09-18T04:39:41.564 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0088004299999999997 2024-09-18T04:39:41.565 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:41.565 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:41.565 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:41.565 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.565 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.565 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:101 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:41.565 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.576367+0000", 2024-09-18T04:39:41.565 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 44.905703772000003, 2024-09-18T04:39:41.565 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0078671020000000008, 2024-09-18T04:39:41.565 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:41.565 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:41.565 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:41.565 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:41.565 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:41.566 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 101 2024-09-18T04:39:41.566 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.566 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:41.566 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.566 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:41.566 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576367+0000", 2024-09-18T04:39:41.566 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:41.566 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.566 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.566 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:41.566 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576367+0000", 2024-09-18T04:39:41.566 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:41.566 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.566 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.566 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:41.567 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576364+0000", 2024-09-18T04:39:41.567 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999971 2024-09-18T04:39:41.567 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.567 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.567 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:41.567 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576378+0000", 2024-09-18T04:39:41.567 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3879999999999999e-05 2024-09-18T04:39:41.567 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.567 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.567 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:41.567 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576410+0000", 2024-09-18T04:39:41.567 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2286000000000002e-05 2024-09-18T04:39:41.567 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.567 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.568 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:41.568 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576425+0000", 2024-09-18T04:39:41.568 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5290000000000001e-05 2024-09-18T04:39:41.568 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.568 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.568 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:41.568 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576551+0000", 2024-09-18T04:39:41.568 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012555199999999999 2024-09-18T04:39:41.568 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.568 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.568 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:41.568 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576719+0000", 2024-09-18T04:39:41.568 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016787400000000001 2024-09-18T04:39:41.568 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.569 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.569 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:41.569 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576951+0000", 2024-09-18T04:39:41.569 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00023252400000000001 2024-09-18T04:39:41.569 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.569 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.569 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:41.569 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.581459+0000", 2024-09-18T04:39:41.569 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045074410000000001 2024-09-18T04:39:41.569 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.569 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.569 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:41.569 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584076+0000", 2024-09-18T04:39:41.569 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0026173860000000002 2024-09-18T04:39:41.570 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.570 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.570 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:41.570 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584160+0000", 2024-09-18T04:39:41.570 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4191999999999996e-05 2024-09-18T04:39:41.570 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.570 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.570 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:41.570 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584234+0000", 2024-09-18T04:39:41.570 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.339e-05 2024-09-18T04:39:41.570 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:41.570 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:41.570 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:41.570 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.570 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.571 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:44 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:41.571 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.550448+0000", 2024-09-18T04:39:41.571 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 40.931622582999999, 2024-09-18T04:39:41.571 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082117590000000008, 2024-09-18T04:39:41.571 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:41.571 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:41.571 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:41.571 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:39:41.571 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:41.571 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 44 2024-09-18T04:39:41.571 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.571 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:41.571 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.571 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:41.572 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.550448+0000", 2024-09-18T04:39:41.572 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:41.572 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.572 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.572 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:41.572 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.550448+0000", 2024-09-18T04:39:41.572 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:41.572 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.572 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.572 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:41.572 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.550445+0000", 2024-09-18T04:39:41.572 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999971 2024-09-18T04:39:41.572 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.572 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.573 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:41.573 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.551904+0000", 2024-09-18T04:39:41.573 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001458322 2024-09-18T04:39:41.573 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.573 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.573 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:41.573 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.551938+0000", 2024-09-18T04:39:41.573 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4359000000000001e-05 2024-09-18T04:39:41.573 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.573 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.573 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:41.573 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.551959+0000", 2024-09-18T04:39:41.573 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1403000000000002e-05 2024-09-18T04:39:41.573 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.574 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.574 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:41.574 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.552049+0000", 2024-09-18T04:39:41.574 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9970999999999996e-05 2024-09-18T04:39:41.574 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.574 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.574 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:41.574 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.552190+0000", 2024-09-18T04:39:41.574 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000140271 2024-09-18T04:39:41.574 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.574 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.574 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:41.574 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.552385+0000", 2024-09-18T04:39:41.574 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019498700000000001 2024-09-18T04:39:41.575 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.575 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.575 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:41.575 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.556308+0000", 2024-09-18T04:39:41.575 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00392333 2024-09-18T04:39:41.575 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.575 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.575 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:41.575 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.558479+0000", 2024-09-18T04:39:41.575 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0021715889999999998 2024-09-18T04:39:41.575 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.575 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.575 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:41.575 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.558616+0000", 2024-09-18T04:39:41.575 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013613399999999999 2024-09-18T04:39:41.576 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.576 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.576 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:41.576 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.558660+0000", 2024-09-18T04:39:41.576 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4053e-05 2024-09-18T04:39:41.576 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:41.576 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:41.576 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:41.576 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.576 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.576 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:46 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:41.576 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.564089+0000", 2024-09-18T04:39:41.576 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 40.917981454, 2024-09-18T04:39:41.577 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0069437090000000002, 2024-09-18T04:39:41.577 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:41.577 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:41.577 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:41.577 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:39:41.577 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:41.577 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 46 2024-09-18T04:39:41.577 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.577 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:41.577 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.577 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:41.577 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.564089+0000", 2024-09-18T04:39:41.577 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:41.577 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.577 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.578 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:41.578 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.564089+0000", 2024-09-18T04:39:41.578 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:41.578 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.578 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.578 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:41.578 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.564087+0000", 2024-09-18T04:39:41.578 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:41.578 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.578 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.578 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:41.578 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565126+0000", 2024-09-18T04:39:41.578 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001038902 2024-09-18T04:39:41.578 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.579 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.579 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:41.579 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565148+0000", 2024-09-18T04:39:41.579 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1911000000000001e-05 2024-09-18T04:39:41.579 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.579 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.579 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:41.579 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565157+0000", 2024-09-18T04:39:41.579 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6220000000000004e-06 2024-09-18T04:39:41.579 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.579 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.579 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:41.579 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565261+0000", 2024-09-18T04:39:41.579 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000103063 2024-09-18T04:39:41.579 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.580 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.580 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:41.580 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565394+0000", 2024-09-18T04:39:41.580 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000133032 2024-09-18T04:39:41.580 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.580 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.580 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:41.580 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565564+0000", 2024-09-18T04:39:41.580 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017068000000000001 2024-09-18T04:39:41.580 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.580 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.580 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:41.580 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.570556+0000", 2024-09-18T04:39:41.580 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0049920620000000002 2024-09-18T04:39:41.581 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.581 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.581 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:41.581 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.570885+0000", 2024-09-18T04:39:41.581 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032827900000000001 2024-09-18T04:39:41.581 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.581 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.581 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:41.581 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.570959+0000", 2024-09-18T04:39:41.581 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3955999999999995e-05 2024-09-18T04:39:41.581 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.581 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.581 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:41.581 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.571033+0000", 2024-09-18T04:39:41.581 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4191999999999997e-05 2024-09-18T04:39:41.582 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:41.582 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:41.582 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:41.582 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.582 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.582 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:100 2.4 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:41.582 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.856389+0000", 2024-09-18T04:39:41.582 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 40.625681342, 2024-09-18T04:39:41.582 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0081480800000000003, 2024-09-18T04:39:41.582 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:41.582 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:41.582 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:41.582 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:39:41.583 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:41.583 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 100 2024-09-18T04:39:41.583 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.583 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:41.583 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.583 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:41.583 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.856389+0000", 2024-09-18T04:39:41.583 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:41.583 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.583 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.583 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:41.583 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.856389+0000", 2024-09-18T04:39:41.583 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:41.583 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.584 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.584 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:41.584 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.856388+0000", 2024-09-18T04:39:41.584 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:41.584 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.584 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.584 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:41.584 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857295+0000", 2024-09-18T04:39:41.584 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00090725599999999995 2024-09-18T04:39:41.584 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.584 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.584 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:41.584 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857311+0000", 2024-09-18T04:39:41.584 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6092000000000001e-05 2024-09-18T04:39:41.585 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.585 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.585 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:41.585 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857319+0000", 2024-09-18T04:39:41.585 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9200000000000004e-06 2024-09-18T04:39:41.585 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.585 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.585 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:41.585 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857388+0000", 2024-09-18T04:39:41.585 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8591999999999997e-05 2024-09-18T04:39:41.585 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.585 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.585 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:41.585 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857535+0000", 2024-09-18T04:39:41.585 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014684299999999999 2024-09-18T04:39:41.586 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.586 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.586 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:41.586 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857747+0000", 2024-09-18T04:39:41.586 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000212361 2024-09-18T04:39:41.586 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.586 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.586 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:41.586 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.860831+0000", 2024-09-18T04:39:41.586 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030841789999999998 2024-09-18T04:39:41.586 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.586 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.586 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:41.586 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.864361+0000", 2024-09-18T04:39:41.587 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0035298780000000002 2024-09-18T04:39:41.587 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.587 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.587 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:41.587 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.864456+0000", 2024-09-18T04:39:41.587 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4557999999999998e-05 2024-09-18T04:39:41.587 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.587 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.587 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:41.587 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.864537+0000", 2024-09-18T04:39:41.587 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1577999999999997e-05 2024-09-18T04:39:41.587 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:41.587 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:41.587 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:41.588 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.588 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.588 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:102 2.4 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:41.588 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.869773+0000", 2024-09-18T04:39:41.588 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 40.612297685000001, 2024-09-18T04:39:41.588 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0084430359999999992, 2024-09-18T04:39:41.588 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:41.588 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:41.588 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:41.588 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:39:41.588 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:41.588 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 102 2024-09-18T04:39:41.588 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.588 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:41.589 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.589 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:41.589 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.869773+0000", 2024-09-18T04:39:41.589 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:41.589 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.589 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.589 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:41.589 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.869773+0000", 2024-09-18T04:39:41.589 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:41.589 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.589 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.589 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:41.589 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.869771+0000", 2024-09-18T04:39:41.589 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:41.589 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.590 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.590 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:41.590 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870629+0000", 2024-09-18T04:39:41.590 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00085760999999999997 2024-09-18T04:39:41.590 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.590 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.590 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:41.590 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870647+0000", 2024-09-18T04:39:41.590 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8363999999999999e-05 2024-09-18T04:39:41.590 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.590 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.590 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:41.590 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870655+0000", 2024-09-18T04:39:41.590 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0099999999999995e-06 2024-09-18T04:39:41.591 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.591 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.591 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:41.591 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870803+0000", 2024-09-18T04:39:41.591 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014798 2024-09-18T04:39:41.591 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.591 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.591 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:41.591 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870951+0000", 2024-09-18T04:39:41.591 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014715999999999999 2024-09-18T04:39:41.591 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.591 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.591 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:41.591 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.871145+0000", 2024-09-18T04:39:41.591 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000194253 2024-09-18T04:39:41.592 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.592 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.592 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:41.592 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.877033+0000", 2024-09-18T04:39:41.592 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058883099999999999 2024-09-18T04:39:41.592 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.592 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.592 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:41.592 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.878078+0000", 2024-09-18T04:39:41.592 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001044747 2024-09-18T04:39:41.592 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.592 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.592 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:41.592 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.878160+0000", 2024-09-18T04:39:41.593 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1904999999999996e-05 2024-09-18T04:39:41.593 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.593 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.593 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:41.593 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.878216+0000", 2024-09-18T04:39:41.593 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6082000000000003e-05 2024-09-18T04:39:41.593 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:41.593 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:41.593 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:41.593 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.593 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.593 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:145 2.4 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:41.593 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:01.096935+0000", 2024-09-18T04:39:41.593 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 40.385135904999999, 2024-09-18T04:39:41.593 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0069937869999999996, 2024-09-18T04:39:41.594 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:41.594 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:41.594 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:41.594 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:39:41.594 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:41.594 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 145 2024-09-18T04:39:41.594 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.594 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:41.594 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.594 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:41.594 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.096935+0000", 2024-09-18T04:39:41.594 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:41.594 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.594 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.594 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:41.595 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.096935+0000", 2024-09-18T04:39:41.595 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:41.595 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.595 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.595 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:41.595 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.096932+0000", 2024-09-18T04:39:41.595 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:41.595 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.595 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.595 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:41.595 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097815+0000", 2024-09-18T04:39:41.595 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00088276000000000001 2024-09-18T04:39:41.595 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.595 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.595 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:41.596 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097842+0000", 2024-09-18T04:39:41.596 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6806999999999999e-05 2024-09-18T04:39:41.596 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.596 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.596 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:41.596 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097852+0000", 2024-09-18T04:39:41.596 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0548e-05 2024-09-18T04:39:41.596 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.596 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.596 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:41.596 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097931+0000", 2024-09-18T04:39:41.596 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8417000000000005e-05 2024-09-18T04:39:41.596 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.597 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.597 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:41.597 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.098047+0000", 2024-09-18T04:39:41.597 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011611 2024-09-18T04:39:41.597 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.597 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.597 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:41.597 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.098172+0000", 2024-09-18T04:39:41.597 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000125482 2024-09-18T04:39:41.597 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.597 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.597 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:41.597 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.100888+0000", 2024-09-18T04:39:41.597 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0027154430000000001 2024-09-18T04:39:41.597 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.598 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.598 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:41.598 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.103814+0000", 2024-09-18T04:39:41.598 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0029259619999999998 2024-09-18T04:39:41.598 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.598 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.598 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:41.598 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.103891+0000", 2024-09-18T04:39:41.598 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7619000000000003e-05 2024-09-18T04:39:41.598 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:41.598 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:41.598 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:41.598 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.103928+0000", 2024-09-18T04:39:41.598 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6900999999999998e-05 2024-09-18T04:39:41.598 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:41.599 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:41.599 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:41.599 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:41.599 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:41.599 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:41.600 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:41.596+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:41.601 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:39:41.701 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:41.696+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:41.710 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-18T04:39:41.719 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:41.719 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:41.719 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:39:41.719 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:41.730 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:39:41.802 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:41.796+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:41.881 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:41.881 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:41.881 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:39:41.881 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:39:41.881 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:41.891 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:39:41.903 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:41.900+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:42.000 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:42.001 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:39:42.001 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:39:42.001 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:39:42.001 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.002 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:37 2.5 e51/28 2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:head v 51'1, mlcod=0'0)", 2024-09-18T04:39:42.002 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:39:42.002 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 59.703696530999999, 2024-09-18T04:39:42.002 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012279973, 2024-09-18T04:39:42.002 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:42.002 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.002 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:42.002 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.002 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:42.002 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:39:42.002 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.002 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.003 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:42.003 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:39:42.003 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.003 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.003 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:42.003 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292468+0000", 2024-09-18T04:39:42.003 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:42.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.003 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.003 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:42.003 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295837+0000", 2024-09-18T04:39:42.003 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0033691699999999999 2024-09-18T04:39:42.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.003 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.004 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:42.004 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295860+0000", 2024-09-18T04:39:42.004 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2756999999999999e-05 2024-09-18T04:39:42.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.004 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.004 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.004 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295872+0000", 2024-09-18T04:39:42.004 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2442e-05 2024-09-18T04:39:42.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.004 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.004 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:42.004 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295934+0000", 2024-09-18T04:39:42.004 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1187999999999998e-05 2024-09-18T04:39:42.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.005 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.005 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:42.005 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295967+0000", 2024-09-18T04:39:42.005 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3185999999999997e-05 2024-09-18T04:39:42.005 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.005 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.005 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:42.005 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304697+0000", 2024-09-18T04:39:42.005 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0087298059999999997 2024-09-18T04:39:42.005 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.005 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.005 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:42.005 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304749+0000", 2024-09-18T04:39:42.005 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1922000000000002e-05 2024-09-18T04:39:42.006 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.006 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:42.006 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.006 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.006 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.006 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:49 2.5 e51/28 2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head v 51'2, mlcod=0'0)", 2024-09-18T04:39:42.006 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:39:42.006 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 59.670369692999998, 2024-09-18T04:39:42.006 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018033041999999999, 2024-09-18T04:39:42.006 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:42.006 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.006 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:42.007 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.007 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:42.007 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:39:42.007 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.008 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.008 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:42.008 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:39:42.008 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.009 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.009 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.009 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:42.009 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325793+0000", 2024-09-18T04:39:42.009 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:42.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.010 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.010 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:42.010 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329038+0000", 2024-09-18T04:39:42.010 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0032444010000000001 2024-09-18T04:39:42.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.010 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.010 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:42.010 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329065+0000", 2024-09-18T04:39:42.010 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7500000000000001e-05 2024-09-18T04:39:42.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.010 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.011 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.011 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329077+0000", 2024-09-18T04:39:42.011 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1919999999999999e-05 2024-09-18T04:39:42.011 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.011 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.011 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:42.011 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329149+0000", 2024-09-18T04:39:42.011 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2349999999999997e-05 2024-09-18T04:39:42.011 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.012 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.012 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:42.012 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329192+0000", 2024-09-18T04:39:42.012 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2744000000000003e-05 2024-09-18T04:39:42.012 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.012 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.012 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:42.012 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343773+0000", 2024-09-18T04:39:42.013 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014581246000000001 2024-09-18T04:39:42.013 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.013 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.013 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:42.013 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343829+0000", 2024-09-18T04:39:42.013 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5143999999999999e-05 2024-09-18T04:39:42.013 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.013 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:42.013 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.013 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.013 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.013 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:47 2.4 e51/17 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 51'3, mlcod=51'1)", 2024-09-18T04:39:42.013 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:39:42.013 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 59.669789428000001, 2024-09-18T04:39:42.013 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018960186, 2024-09-18T04:39:42.014 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:42.014 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.014 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:42.014 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.014 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:42.014 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:39:42.014 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.014 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.014 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.014 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:42.014 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:39:42.014 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.014 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.014 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.015 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:42.015 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326374+0000", 2024-09-18T04:39:42.015 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:42.015 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.015 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.015 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:42.015 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330883+0000", 2024-09-18T04:39:42.015 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0045084740000000002 2024-09-18T04:39:42.015 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.015 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.015 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:42.015 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330901+0000", 2024-09-18T04:39:42.015 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7890999999999999e-05 2024-09-18T04:39:42.015 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.015 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.016 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.016 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330912+0000", 2024-09-18T04:39:42.016 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1041e-05 2024-09-18T04:39:42.016 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.016 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.016 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:42.016 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.331003+0000", 2024-09-18T04:39:42.016 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1323999999999995e-05 2024-09-18T04:39:42.016 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.016 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.016 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:42.016 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.331030+0000", 2024-09-18T04:39:42.016 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7351999999999998e-05 2024-09-18T04:39:42.016 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.016 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.017 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:42.017 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345287+0000", 2024-09-18T04:39:42.017 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014256332 2024-09-18T04:39:42.017 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.017 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.017 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:42.017 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345336+0000", 2024-09-18T04:39:42.017 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9356e-05 2024-09-18T04:39:42.017 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.017 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:42.017 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.017 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.017 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.017 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:58 2.5 e51/28 2:b227e6b1:::rbd_data.125fc09df6d1.000000000000000e:head v 51'3, mlcod=51'1)", 2024-09-18T04:39:42.018 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:39:42.018 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 59.629362800999999, 2024-09-18T04:39:42.018 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016513678, 2024-09-18T04:39:42.018 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:42.018 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.018 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:42.018 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.018 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:42.018 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:39:42.018 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.018 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.018 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.018 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:42.018 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:39:42.018 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.019 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.019 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.019 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:42.019 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366800+0000", 2024-09-18T04:39:42.019 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:42.019 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.019 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.019 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:42.019 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368644+0000", 2024-09-18T04:39:42.019 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001844399 2024-09-18T04:39:42.019 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.019 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.019 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:42.019 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368681+0000", 2024-09-18T04:39:42.019 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6526999999999998e-05 2024-09-18T04:39:42.020 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.020 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.020 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.020 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368698+0000", 2024-09-18T04:39:42.020 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7272999999999999e-05 2024-09-18T04:39:42.020 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.020 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.020 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:42.020 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368773+0000", 2024-09-18T04:39:42.020 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4862000000000003e-05 2024-09-18T04:39:42.020 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.021 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.021 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:42.021 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368801+0000", 2024-09-18T04:39:42.021 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7925000000000001e-05 2024-09-18T04:39:42.021 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.021 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.021 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:42.021 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383261+0000", 2024-09-18T04:39:42.021 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014460377999999999 2024-09-18T04:39:42.021 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.021 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.021 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:42.021 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383316+0000", 2024-09-18T04:39:42.021 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4623999999999997e-05 2024-09-18T04:39:42.021 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.022 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:42.022 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.022 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.022 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.022 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:56 2.4 e51/17 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 51'4, mlcod=51'2)", 2024-09-18T04:39:42.022 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:39:42.022 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 59.627358774999998, 2024-09-18T04:39:42.022 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015105221, 2024-09-18T04:39:42.022 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:42.022 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.022 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:42.022 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.022 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:42.022 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:39:42.023 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.023 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.023 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.023 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:42.023 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:39:42.023 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.023 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.023 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.023 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:42.023 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368805+0000", 2024-09-18T04:39:42.024 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:42.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.024 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.024 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:42.024 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371603+0000", 2024-09-18T04:39:42.025 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027982660000000002 2024-09-18T04:39:42.025 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.025 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.025 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:42.025 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371633+0000", 2024-09-18T04:39:42.025 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0073000000000002e-05 2024-09-18T04:39:42.026 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.026 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.026 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.026 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371652+0000", 2024-09-18T04:39:42.026 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8597000000000002e-05 2024-09-18T04:39:42.026 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.026 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.026 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:42.026 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371752+0000", 2024-09-18T04:39:42.026 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9772999999999995e-05 2024-09-18T04:39:42.026 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.026 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.027 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:42.027 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371793+0000", 2024-09-18T04:39:42.027 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1173e-05 2024-09-18T04:39:42.027 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.027 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.027 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:42.027 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383875+0000", 2024-09-18T04:39:42.027 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012081852000000001 2024-09-18T04:39:42.027 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.027 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.027 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:42.028 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383912+0000", 2024-09-18T04:39:42.028 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7045000000000002e-05 2024-09-18T04:39:42.028 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.028 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:42.028 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.028 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.028 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.028 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:36 2.5 e51/28 2:bd5b8dd8:ns::rbd_data.133e813ab5bc.0000000000000002:head v 51'4, mlcod=51'2)", 2024-09-18T04:39:42.028 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:39:42.028 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 59.458639783999999, 2024-09-18T04:39:42.028 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020758161000000001, 2024-09-18T04:39:42.028 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:42.028 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.029 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:42.029 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.029 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:42.029 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:39:42.029 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.029 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.029 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.029 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:42.029 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:39:42.029 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.029 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.029 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.029 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:42.029 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537523+0000", 2024-09-18T04:39:42.029 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:42.030 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.030 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.030 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:42.030 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541578+0000", 2024-09-18T04:39:42.030 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0040549710000000001 2024-09-18T04:39:42.030 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.030 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.030 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:42.030 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541608+0000", 2024-09-18T04:39:42.030 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9349e-05 2024-09-18T04:39:42.030 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.030 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.030 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.030 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541621+0000", 2024-09-18T04:39:42.031 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2736e-05 2024-09-18T04:39:42.031 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.031 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.031 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:42.031 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541715+0000", 2024-09-18T04:39:42.031 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4197999999999995e-05 2024-09-18T04:39:42.031 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.031 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.031 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:42.031 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541755+0000", 2024-09-18T04:39:42.031 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9818999999999999e-05 2024-09-18T04:39:42.031 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.031 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.031 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:42.031 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558217+0000", 2024-09-18T04:39:42.031 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016462234999999999 2024-09-18T04:39:42.032 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.032 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.032 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:42.032 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558284+0000", 2024-09-18T04:39:42.032 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6774999999999997e-05 2024-09-18T04:39:42.032 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.032 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:42.032 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.032 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.032 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.032 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:39 2.4 e51/17 2:304364d1:ns::rbd_data.133e813ab5bc.0000000000000004:head v 51'5, mlcod=51'3)", 2024-09-18T04:39:42.032 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:39:42.032 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 59.450500656999999, 2024-09-18T04:39:42.032 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016777558000000001, 2024-09-18T04:39:42.033 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:42.033 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.033 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:42.033 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.033 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:42.033 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:39:42.033 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.033 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.033 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.033 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:42.033 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:39:42.033 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.033 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.033 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.033 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:42.034 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545663+0000", 2024-09-18T04:39:42.034 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:42.034 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.034 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.034 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:42.034 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548286+0000", 2024-09-18T04:39:42.034 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026233390000000001 2024-09-18T04:39:42.034 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.034 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.034 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:42.034 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548305+0000", 2024-09-18T04:39:42.034 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8425000000000001e-05 2024-09-18T04:39:42.034 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.034 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.035 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.035 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548316+0000", 2024-09-18T04:39:42.035 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1163e-05 2024-09-18T04:39:42.035 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.035 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.035 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:42.035 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548379+0000", 2024-09-18T04:39:42.035 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2780000000000005e-05 2024-09-18T04:39:42.035 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.035 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.035 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:42.035 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548399+0000", 2024-09-18T04:39:42.035 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0080000000000001e-05 2024-09-18T04:39:42.035 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.036 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.036 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:42.036 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562383+0000", 2024-09-18T04:39:42.036 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013984671000000001 2024-09-18T04:39:42.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.036 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.036 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:42.036 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562442+0000", 2024-09-18T04:39:42.036 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8700999999999999e-05 2024-09-18T04:39:42.036 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.036 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:42.036 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.036 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.037 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:37 2.5 e51/28 2:bf39b8e4:ns::rbd_data.133e813ab5bc.0000000000000003:head v 51'5, mlcod=51'2)", 2024-09-18T04:39:42.037 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:39:42.037 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 59.447632388000002, 2024-09-18T04:39:42.037 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014264492, 2024-09-18T04:39:42.037 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:42.037 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.037 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:42.037 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.037 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:42.037 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:39:42.037 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.037 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.037 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.037 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:42.037 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:39:42.038 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.038 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.038 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.038 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:42.038 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548531+0000", 2024-09-18T04:39:42.038 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:42.038 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.038 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.038 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:42.038 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551500+0000", 2024-09-18T04:39:42.039 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0029688129999999998 2024-09-18T04:39:42.039 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.039 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.039 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:42.039 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551516+0000", 2024-09-18T04:39:42.039 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5994e-05 2024-09-18T04:39:42.039 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.039 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.039 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.039 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551526+0000", 2024-09-18T04:39:42.039 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9299999999999998e-06 2024-09-18T04:39:42.039 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.039 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.039 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:42.040 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551577+0000", 2024-09-18T04:39:42.040 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0871000000000001e-05 2024-09-18T04:39:42.040 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.040 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.040 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:42.040 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551602+0000", 2024-09-18T04:39:42.040 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4813000000000001e-05 2024-09-18T04:39:42.040 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.040 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.040 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:42.040 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562741+0000", 2024-09-18T04:39:42.040 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011139428999999999 2024-09-18T04:39:42.040 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.040 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.040 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:42.041 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562797+0000", 2024-09-18T04:39:42.041 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6329000000000002e-05 2024-09-18T04:39:42.041 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.041 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:42.041 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.041 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.041 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.041 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:49 2.4 e51/17 2:20f0b0c9:ns::rbd_data.133e813ab5bc.0000000000000009:head v 51'6, mlcod=51'4)", 2024-09-18T04:39:42.041 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:39:42.041 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 59.416197830000002, 2024-09-18T04:39:42.041 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013849046, 2024-09-18T04:39:42.041 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:42.041 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.041 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:42.042 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.042 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:42.042 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:39:42.042 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.042 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.042 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:42.042 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:39:42.042 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.042 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.042 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:42.042 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579966+0000", 2024-09-18T04:39:42.042 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:42.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.043 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.043 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:42.043 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582324+0000", 2024-09-18T04:39:42.043 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023578890000000002 2024-09-18T04:39:42.043 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.043 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.043 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:42.043 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582355+0000", 2024-09-18T04:39:42.043 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1485000000000001e-05 2024-09-18T04:39:42.043 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.043 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.043 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.043 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582371+0000", 2024-09-18T04:39:42.043 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6147999999999999e-05 2024-09-18T04:39:42.043 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.044 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.044 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:42.044 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582467+0000", 2024-09-18T04:39:42.044 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6059999999999998e-05 2024-09-18T04:39:42.044 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.044 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.044 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:42.044 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582505+0000", 2024-09-18T04:39:42.044 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7679000000000003e-05 2024-09-18T04:39:42.044 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.044 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.044 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:42.044 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593743+0000", 2024-09-18T04:39:42.044 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011238356 2024-09-18T04:39:42.045 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.045 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.045 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:42.045 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593816+0000", 2024-09-18T04:39:42.045 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2935000000000006e-05 2024-09-18T04:39:42.045 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.045 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:42.045 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.045 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.045 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.045 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:53 2.4 e51/17 2:39f7f174:ns::rbd_data.133e813ab5bc.000000000000000b:head v 51'7, mlcod=51'4)", 2024-09-18T04:39:42.045 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:39:42.045 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 59.401496674999997, 2024-09-18T04:39:42.045 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0079914159999999995, 2024-09-18T04:39:42.045 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:42.046 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.046 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:42.046 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.046 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:42.046 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:39:42.046 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.046 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.046 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.046 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:42.046 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:39:42.046 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.046 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.046 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.046 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:42.046 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594667+0000", 2024-09-18T04:39:42.047 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:42.047 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.047 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.047 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:42.047 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596446+0000", 2024-09-18T04:39:42.047 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001779033 2024-09-18T04:39:42.047 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.047 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.047 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:42.047 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596472+0000", 2024-09-18T04:39:42.047 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6120000000000001e-05 2024-09-18T04:39:42.047 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.047 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.047 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.047 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596500+0000", 2024-09-18T04:39:42.048 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7518e-05 2024-09-18T04:39:42.048 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.048 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.048 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:42.048 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596564+0000", 2024-09-18T04:39:42.048 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3880999999999994e-05 2024-09-18T04:39:42.048 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.048 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.048 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:42.048 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596596+0000", 2024-09-18T04:39:42.048 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2638999999999997e-05 2024-09-18T04:39:42.048 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.048 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.048 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:42.049 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602605+0000", 2024-09-18T04:39:42.049 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0060080699999999999 2024-09-18T04:39:42.049 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.049 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.049 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:42.049 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602660+0000", 2024-09-18T04:39:42.049 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5396000000000003e-05 2024-09-18T04:39:42.049 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.049 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:42.049 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.049 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.049 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.049 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:60 2.5 e51/28 2:a06f51e4:ns::rbd_data.133e813ab5bc.000000000000000f:head v 51'6, mlcod=51'4)", 2024-09-18T04:39:42.049 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:39:42.049 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 59.386679690000001, 2024-09-18T04:39:42.050 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0093954989999999999, 2024-09-18T04:39:42.050 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:42.050 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.050 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:42.050 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.050 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:42.050 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:39:42.050 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.050 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.050 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.050 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:42.050 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:39:42.050 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.050 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.050 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.051 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:42.051 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609483+0000", 2024-09-18T04:39:42.051 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:42.051 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.051 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.051 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:42.051 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611767+0000", 2024-09-18T04:39:42.051 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002283724 2024-09-18T04:39:42.051 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.051 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.051 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:42.051 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611791+0000", 2024-09-18T04:39:42.051 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3547e-05 2024-09-18T04:39:42.051 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.051 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.052 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.052 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611800+0000", 2024-09-18T04:39:42.052 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4979999999999995e-06 2024-09-18T04:39:42.052 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.052 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.052 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:42.052 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611864+0000", 2024-09-18T04:39:42.052 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3974999999999996e-05 2024-09-18T04:39:42.052 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.052 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.052 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:42.052 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611887+0000", 2024-09-18T04:39:42.052 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2633000000000002e-05 2024-09-18T04:39:42.052 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.053 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.053 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:42.053 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.618777+0000", 2024-09-18T04:39:42.053 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0068897979999999999 2024-09-18T04:39:42.053 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.053 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.053 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:42.053 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.618881+0000", 2024-09-18T04:39:42.053 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010437699999999999 2024-09-18T04:39:42.053 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.053 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:42.053 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.053 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.053 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.053 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:663 2.4 e57/17 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 57'13, mlcod=57'11)", 2024-09-18T04:39:42.054 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.355300+0000", 2024-09-18T04:39:42.054 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 50.640864755000003, 2024-09-18T04:39:42.054 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0049192760000000002, 2024-09-18T04:39:42.054 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:42.054 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.054 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:42.054 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.054 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:42.054 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355300+0000", 2024-09-18T04:39:42.054 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.054 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.054 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.054 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:42.054 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355300+0000", 2024-09-18T04:39:42.055 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.055 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.055 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.055 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:42.055 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355299+0000", 2024-09-18T04:39:42.055 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:42.055 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.055 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.055 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:42.055 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355876+0000", 2024-09-18T04:39:42.055 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00057730900000000003 2024-09-18T04:39:42.055 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.055 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.055 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:42.055 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355897+0000", 2024-09-18T04:39:42.056 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0477000000000001e-05 2024-09-18T04:39:42.056 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.056 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.056 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.056 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355905+0000", 2024-09-18T04:39:42.056 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5739999999999996e-06 2024-09-18T04:39:42.056 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.056 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.056 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:42.056 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355989+0000", 2024-09-18T04:39:42.056 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4002e-05 2024-09-18T04:39:42.056 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.056 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.056 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:42.057 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.356041+0000", 2024-09-18T04:39:42.057 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1347000000000002e-05 2024-09-18T04:39:42.057 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.057 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.057 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:42.057 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.360172+0000", 2024-09-18T04:39:42.057 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0041315090000000002 2024-09-18T04:39:42.057 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.057 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.057 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:42.057 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.360220+0000", 2024-09-18T04:39:42.057 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7584999999999999e-05 2024-09-18T04:39:42.057 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.057 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:42.057 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.058 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.058 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.058 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:679 2.5 e57/28 2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head v 57'20, mlcod=57'18)", 2024-09-18T04:39:42.058 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.445316+0000", 2024-09-18T04:39:42.058 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 50.550849278000001, 2024-09-18T04:39:42.058 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0047016760000000001, 2024-09-18T04:39:42.058 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:42.058 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.058 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:42.058 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.058 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:42.058 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445316+0000", 2024-09-18T04:39:42.058 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.058 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.058 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.059 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:42.059 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445316+0000", 2024-09-18T04:39:42.059 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.059 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.059 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.059 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:42.059 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445314+0000", 2024-09-18T04:39:42.059 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:42.059 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.059 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.059 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:42.059 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445947+0000", 2024-09-18T04:39:42.059 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00063298900000000003 2024-09-18T04:39:42.060 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.060 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.060 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:42.060 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445968+0000", 2024-09-18T04:39:42.060 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.054e-05 2024-09-18T04:39:42.060 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.060 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.060 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.060 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445981+0000", 2024-09-18T04:39:42.060 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2697000000000001e-05 2024-09-18T04:39:42.060 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.060 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.061 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:42.061 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.446102+0000", 2024-09-18T04:39:42.061 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012189300000000001 2024-09-18T04:39:42.061 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.061 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.061 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:42.061 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.446151+0000", 2024-09-18T04:39:42.061 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8279000000000003e-05 2024-09-18T04:39:42.061 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.061 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.061 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:42.061 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.449953+0000", 2024-09-18T04:39:42.061 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003801979 2024-09-18T04:39:42.061 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.062 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.062 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:42.062 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.450018+0000", 2024-09-18T04:39:42.062 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4873e-05 2024-09-18T04:39:42.062 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.062 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:42.062 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.062 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.062 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.062 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:699 2.4 e57/17 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 57'17, mlcod=57'15)", 2024-09-18T04:39:42.062 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.566073+0000", 2024-09-18T04:39:42.062 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 50.430091765, 2024-09-18T04:39:42.062 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055398380000000001, 2024-09-18T04:39:42.062 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:42.062 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.063 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:42.063 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.063 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:42.063 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566073+0000", 2024-09-18T04:39:42.063 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.063 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.063 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.063 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:42.063 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566073+0000", 2024-09-18T04:39:42.063 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.063 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.063 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.063 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:42.063 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566071+0000", 2024-09-18T04:39:42.063 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:42.064 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.064 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.064 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:42.064 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566637+0000", 2024-09-18T04:39:42.064 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00056516699999999999 2024-09-18T04:39:42.064 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.064 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.064 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:42.064 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566668+0000", 2024-09-18T04:39:42.064 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1544999999999997e-05 2024-09-18T04:39:42.064 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.064 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.064 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.064 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566685+0000", 2024-09-18T04:39:42.065 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6857000000000001e-05 2024-09-18T04:39:42.065 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.065 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.065 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:42.065 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566816+0000", 2024-09-18T04:39:42.065 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013092799999999999 2024-09-18T04:39:42.065 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.065 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.065 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:42.065 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566875+0000", 2024-09-18T04:39:42.065 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9583999999999997e-05 2024-09-18T04:39:42.065 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.065 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.065 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:42.065 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.571567+0000", 2024-09-18T04:39:42.066 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.004691682 2024-09-18T04:39:42.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.066 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.066 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:42.066 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.571613+0000", 2024-09-18T04:39:42.066 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6125000000000003e-05 2024-09-18T04:39:42.066 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.066 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:42.066 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.066 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.066 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:98 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'27, mlcod=60'25)", 2024-09-18T04:39:42.066 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.847978+0000", 2024-09-18T04:39:42.066 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 41.148187206000003, 2024-09-18T04:39:42.067 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0044617709999999998, 2024-09-18T04:39:42.067 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:42.067 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.067 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:42.067 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.067 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:42.067 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.847978+0000", 2024-09-18T04:39:42.067 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.067 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.067 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.067 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:42.067 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.847978+0000", 2024-09-18T04:39:42.067 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.067 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.067 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.068 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:42.068 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.847977+0000", 2024-09-18T04:39:42.068 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:42.068 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.068 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.068 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:42.068 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849038+0000", 2024-09-18T04:39:42.068 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0010614820000000001 2024-09-18T04:39:42.068 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.068 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.068 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:42.068 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849065+0000", 2024-09-18T04:39:42.068 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7127999999999998e-05 2024-09-18T04:39:42.068 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.068 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.069 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.069 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849082+0000", 2024-09-18T04:39:42.069 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7170999999999999e-05 2024-09-18T04:39:42.069 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.069 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.069 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:42.069 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849188+0000", 2024-09-18T04:39:42.069 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010605800000000001 2024-09-18T04:39:42.069 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.069 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.069 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:42.069 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849233+0000", 2024-09-18T04:39:42.069 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4579000000000001e-05 2024-09-18T04:39:42.069 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.069 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.070 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:42.070 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.852370+0000", 2024-09-18T04:39:42.070 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00313751 2024-09-18T04:39:42.070 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.070 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.070 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:42.070 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.852440+0000", 2024-09-18T04:39:42.070 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9238000000000004e-05 2024-09-18T04:39:42.070 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.070 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:42.070 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.070 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.070 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.070 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:100 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'29, mlcod=64'27)", 2024-09-18T04:39:42.070 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.858766+0000", 2024-09-18T04:39:42.071 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 41.137398877000003, 2024-09-18T04:39:42.071 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0061964840000000004, 2024-09-18T04:39:42.071 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:42.071 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.071 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:42.071 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.071 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:42.071 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.858766+0000", 2024-09-18T04:39:42.071 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.071 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.071 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.071 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:42.071 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.858766+0000", 2024-09-18T04:39:42.071 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.071 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.072 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.072 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:42.072 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.858765+0000", 2024-09-18T04:39:42.072 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:42.072 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.072 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.072 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:42.072 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861612+0000", 2024-09-18T04:39:42.072 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028473169999999998 2024-09-18T04:39:42.072 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.072 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.072 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:42.072 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861643+0000", 2024-09-18T04:39:42.072 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1356000000000003e-05 2024-09-18T04:39:42.073 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.073 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.073 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.073 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861656+0000", 2024-09-18T04:39:42.073 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2899e-05 2024-09-18T04:39:42.073 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.073 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.073 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:42.073 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861773+0000", 2024-09-18T04:39:42.073 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011637999999999999 2024-09-18T04:39:42.073 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.073 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.073 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:42.073 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861808+0000", 2024-09-18T04:39:42.074 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5002999999999997e-05 2024-09-18T04:39:42.074 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.074 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.074 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:42.074 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.864876+0000", 2024-09-18T04:39:42.074 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030680719999999998 2024-09-18T04:39:42.074 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.074 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.074 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:42.074 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.864963+0000", 2024-09-18T04:39:42.074 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7114000000000007e-05 2024-09-18T04:39:42.074 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.074 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:42.074 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.074 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.075 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.075 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:102 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'31, mlcod=64'29)", 2024-09-18T04:39:42.075 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.872098+0000", 2024-09-18T04:39:42.075 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 41.124067619999998, 2024-09-18T04:39:42.075 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0064676450000000002, 2024-09-18T04:39:42.075 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:42.075 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.075 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:42.075 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.075 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:42.075 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.872098+0000", 2024-09-18T04:39:42.075 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.075 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.075 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.075 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:42.076 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.872098+0000", 2024-09-18T04:39:42.076 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.076 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.076 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.076 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:42.076 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.872096+0000", 2024-09-18T04:39:42.076 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:42.076 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.076 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.076 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:42.076 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875176+0000", 2024-09-18T04:39:42.076 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030799809999999999 2024-09-18T04:39:42.076 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.076 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.077 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:42.077 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875204+0000", 2024-09-18T04:39:42.077 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8025999999999999e-05 2024-09-18T04:39:42.077 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.077 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.077 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.077 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875216+0000", 2024-09-18T04:39:42.077 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2072999999999999e-05 2024-09-18T04:39:42.077 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.077 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.077 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:42.077 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875307+0000", 2024-09-18T04:39:42.077 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0373999999999999e-05 2024-09-18T04:39:42.077 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.077 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.078 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:42.078 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875349+0000", 2024-09-18T04:39:42.078 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2299000000000003e-05 2024-09-18T04:39:42.078 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.078 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.078 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:42.078 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.878496+0000", 2024-09-18T04:39:42.078 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0031468960000000002 2024-09-18T04:39:42.078 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.078 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.078 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:42.078 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.878565+0000", 2024-09-18T04:39:42.078 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9465e-05 2024-09-18T04:39:42.078 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.079 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:42.079 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.079 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.079 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.079 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:104 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'33, mlcod=64'31)", 2024-09-18T04:39:42.079 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.885100+0000", 2024-09-18T04:39:42.079 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 41.111065085, 2024-09-18T04:39:42.079 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0051888159999999997, 2024-09-18T04:39:42.079 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:42.079 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.079 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:42.079 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.079 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:42.079 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.885100+0000", 2024-09-18T04:39:42.079 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.080 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.080 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.080 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:42.080 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.885100+0000", 2024-09-18T04:39:42.080 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.080 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.080 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.080 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:42.080 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.885098+0000", 2024-09-18T04:39:42.080 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:42.080 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.081 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.081 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:42.081 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886809+0000", 2024-09-18T04:39:42.081 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017110459999999999 2024-09-18T04:39:42.081 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.081 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.081 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:42.081 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886842+0000", 2024-09-18T04:39:42.081 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2638999999999997e-05 2024-09-18T04:39:42.081 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.081 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.081 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.081 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886855+0000", 2024-09-18T04:39:42.081 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3380000000000001e-05 2024-09-18T04:39:42.082 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.082 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.082 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:42.082 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886939+0000", 2024-09-18T04:39:42.082 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3733999999999995e-05 2024-09-18T04:39:42.082 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.082 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.082 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:42.082 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886981+0000", 2024-09-18T04:39:42.082 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1956000000000003e-05 2024-09-18T04:39:42.082 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.082 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.082 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:42.082 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.890201+0000", 2024-09-18T04:39:42.082 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003219927 2024-09-18T04:39:42.083 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.083 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.083 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:42.083 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.890289+0000", 2024-09-18T04:39:42.083 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7997999999999993e-05 2024-09-18T04:39:42.083 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.083 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:42.083 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.083 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.083 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.083 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:145 2.4 e64/62 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 64'37, mlcod=64'35)", 2024-09-18T04:39:42.083 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:01.099398+0000", 2024-09-18T04:39:42.083 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 40.896767597, 2024-09-18T04:39:42.083 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0050078459999999998, 2024-09-18T04:39:42.084 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:42.084 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.084 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:42.084 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.084 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:42.084 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.099398+0000", 2024-09-18T04:39:42.084 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.084 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.084 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.084 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:42.084 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.099398+0000", 2024-09-18T04:39:42.084 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.084 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.084 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.084 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:42.085 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.099396+0000", 2024-09-18T04:39:42.085 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:42.085 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.085 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.085 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:42.085 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100199+0000", 2024-09-18T04:39:42.085 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00080338000000000002 2024-09-18T04:39:42.085 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.085 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.085 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:42.085 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100230+0000", 2024-09-18T04:39:42.085 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0716000000000002e-05 2024-09-18T04:39:42.085 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.085 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.085 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.086 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100243+0000", 2024-09-18T04:39:42.086 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3359e-05 2024-09-18T04:39:42.086 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.086 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.086 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:42.086 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100337+0000", 2024-09-18T04:39:42.086 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4387000000000003e-05 2024-09-18T04:39:42.086 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.086 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.086 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:42.086 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100396+0000", 2024-09-18T04:39:42.086 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8164000000000001e-05 2024-09-18T04:39:42.086 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.086 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.087 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:42.087 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.104309+0000", 2024-09-18T04:39:42.087 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0039137800000000004 2024-09-18T04:39:42.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.087 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.087 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:42.087 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.104405+0000", 2024-09-18T04:39:42.087 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6039999999999995e-05 2024-09-18T04:39:42.087 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.087 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:42.087 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.087 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.087 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:147 2.4 e64/62 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 64'39, mlcod=64'37)", 2024-09-18T04:39:42.087 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:01.110658+0000", 2024-09-18T04:39:42.088 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 40.885507508000003, 2024-09-18T04:39:42.088 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0046316910000000003, 2024-09-18T04:39:42.088 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:42.088 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.088 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:42.088 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.088 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:42.088 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.110658+0000", 2024-09-18T04:39:42.088 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.088 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.088 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.088 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:42.088 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.110658+0000", 2024-09-18T04:39:42.088 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.089 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.089 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.089 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:42.089 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.110656+0000", 2024-09-18T04:39:42.089 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:42.089 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.089 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.089 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:42.089 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111444+0000", 2024-09-18T04:39:42.089 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00078762400000000001 2024-09-18T04:39:42.089 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.090 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.090 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:42.090 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111474+0000", 2024-09-18T04:39:42.090 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0085000000000001e-05 2024-09-18T04:39:42.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.090 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.090 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.090 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111486+0000", 2024-09-18T04:39:42.090 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2645e-05 2024-09-18T04:39:42.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.090 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.090 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:42.090 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111581+0000", 2024-09-18T04:39:42.090 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4432000000000007e-05 2024-09-18T04:39:42.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.091 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.091 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:42.091 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111631+0000", 2024-09-18T04:39:42.091 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9901000000000002e-05 2024-09-18T04:39:42.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.091 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.091 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:42.091 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.115209+0000", 2024-09-18T04:39:42.091 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003577916 2024-09-18T04:39:42.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.091 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.091 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:42.091 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.115289+0000", 2024-09-18T04:39:42.091 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0704999999999994e-05 2024-09-18T04:39:42.092 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.092 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:42.092 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.092 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.092 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:42.092 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:42.093 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:39:42.094 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:42.084+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:42.194 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:42.188+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:42.204 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:42.204 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:42.205 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:39:42.205 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:42.214 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:39:42.295 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:42.292+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:42.319 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:42.319 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:42.319 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:39:42.320 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:39:42.320 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:42.330 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:39:42.397 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:42.392+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:42.446 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:42.446 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:39:42.446 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:39:42.447 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:39:42.447 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.447 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:33 2.1 e51/24 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head v 51'4, mlcod=51'2)", 2024-09-18T04:39:42.447 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:39:42.447 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 60.161088231999997, 2024-09-18T04:39:42.448 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018702106, 2024-09-18T04:39:42.448 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:42.448 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.448 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:42.448 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.448 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:42.449 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:39:42.449 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.449 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.449 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.449 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:42.450 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:39:42.450 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.450 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.450 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.450 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:42.450 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279197+0000", 2024-09-18T04:39:42.451 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:42.451 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.453 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.453 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:42.453 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282682+0000", 2024-09-18T04:39:42.453 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0034848689999999998 2024-09-18T04:39:42.453 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.453 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.453 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:42.454 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282709+0000", 2024-09-18T04:39:42.454 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7599000000000001e-05 2024-09-18T04:39:42.454 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.454 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.454 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.454 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282720+0000", 2024-09-18T04:39:42.454 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.092e-05 2024-09-18T04:39:42.454 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.454 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.454 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:42.454 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282791+0000", 2024-09-18T04:39:42.454 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1268999999999998e-05 2024-09-18T04:39:42.454 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.454 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.455 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:42.455 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282843+0000", 2024-09-18T04:39:42.455 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1771999999999998e-05 2024-09-18T04:39:42.455 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.455 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.455 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:42.455 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.297830+0000", 2024-09-18T04:39:42.455 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014987228999999999 2024-09-18T04:39:42.455 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.455 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.455 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:42.455 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.297901+0000", 2024-09-18T04:39:42.455 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0209999999999994e-05 2024-09-18T04:39:42.455 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.455 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:42.456 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.456 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.456 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.456 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:38 2.0 e51/17 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head v 51'10, mlcod=51'8)", 2024-09-18T04:39:42.456 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:39:42.456 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 60.159669802000003, 2024-09-18T04:39:42.456 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.023984472, 2024-09-18T04:39:42.456 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:42.456 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.456 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:42.456 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.456 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:42.456 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:39:42.457 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.457 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.457 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.457 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:42.457 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:39:42.457 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.457 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.457 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.457 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:42.457 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280614+0000", 2024-09-18T04:39:42.457 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:42.457 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.457 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.457 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:42.457 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293094+0000", 2024-09-18T04:39:42.458 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012479571 2024-09-18T04:39:42.458 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.458 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.458 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:42.458 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293111+0000", 2024-09-18T04:39:42.458 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7354999999999999e-05 2024-09-18T04:39:42.458 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.458 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.458 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.458 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293121+0000", 2024-09-18T04:39:42.458 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6979999999999993e-06 2024-09-18T04:39:42.458 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.458 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.458 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:42.458 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293180+0000", 2024-09-18T04:39:42.459 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9333000000000003e-05 2024-09-18T04:39:42.459 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.459 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.459 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:42.459 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293203+0000", 2024-09-18T04:39:42.459 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2633000000000002e-05 2024-09-18T04:39:42.459 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.459 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.459 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:42.459 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304534+0000", 2024-09-18T04:39:42.459 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011330761 2024-09-18T04:39:42.459 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.459 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.459 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:42.460 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304601+0000", 2024-09-18T04:39:42.460 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7657000000000006e-05 2024-09-18T04:39:42.460 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.460 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:42.460 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.460 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.460 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.460 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:46 2.6 2:7ba25245:::rbd_data.125fc09df6d1.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:42.460 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:39:42.460 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 60.132127001999997, 2024-09-18T04:39:42.460 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.035604595000000003, 2024-09-18T04:39:42.460 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:42.460 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.460 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:42.461 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:42.461 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:42.461 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 46 2024-09-18T04:39:42.461 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.461 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:42.461 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.461 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:42.461 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:39:42.461 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.461 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.461 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.461 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:42.461 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:39:42.461 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.462 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.462 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.462 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:42.462 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308158+0000", 2024-09-18T04:39:42.462 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:42.462 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.462 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.462 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:42.462 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313191+0000", 2024-09-18T04:39:42.462 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0050323929999999996 2024-09-18T04:39:42.462 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.462 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.462 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:42.462 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313210+0000", 2024-09-18T04:39:42.462 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9673e-05 2024-09-18T04:39:42.463 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.463 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.463 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.463 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313221+0000", 2024-09-18T04:39:42.463 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0444e-05 2024-09-18T04:39:42.463 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.463 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.463 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:42.463 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313294+0000", 2024-09-18T04:39:42.463 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2650000000000004e-05 2024-09-18T04:39:42.463 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.463 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.463 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:42.463 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313493+0000", 2024-09-18T04:39:42.463 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019968300000000001 2024-09-18T04:39:42.464 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.464 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.464 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:42.464 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313666+0000", 2024-09-18T04:39:42.464 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00017248599999999999 2024-09-18T04:39:42.464 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.464 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.464 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:42.464 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.323175+0000", 2024-09-18T04:39:42.464 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0095088819999999998 2024-09-18T04:39:42.464 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.464 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.464 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:42.464 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343651+0000", 2024-09-18T04:39:42.465 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020475892999999998 2024-09-18T04:39:42.465 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.465 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.465 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:42.465 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343715+0000", 2024-09-18T04:39:42.465 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4443999999999995e-05 2024-09-18T04:39:42.465 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.465 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.465 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:42.465 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343764+0000", 2024-09-18T04:39:42.465 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9280000000000003e-05 2024-09-18T04:39:42.465 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.465 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:42.466 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.466 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.466 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.466 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:47 2.4 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:42.466 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:39:42.466 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 60.127044965000003, 2024-09-18T04:39:42.466 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.032570693999999997, 2024-09-18T04:39:42.466 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:42.466 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.466 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:42.466 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:42.466 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:42.466 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 47 2024-09-18T04:39:42.466 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.466 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:42.467 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.467 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:42.467 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:39:42.467 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.467 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.467 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.467 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:42.467 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:39:42.467 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.467 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.467 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.467 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:42.467 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313240+0000", 2024-09-18T04:39:42.467 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:42.468 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.468 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.468 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:42.468 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325825+0000", 2024-09-18T04:39:42.468 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012584425999999999 2024-09-18T04:39:42.468 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.468 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.468 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:42.468 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325848+0000", 2024-09-18T04:39:42.468 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3025999999999999e-05 2024-09-18T04:39:42.468 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.468 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.468 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.468 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325863+0000", 2024-09-18T04:39:42.469 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5152e-05 2024-09-18T04:39:42.469 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.469 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.469 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:42.469 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325938+0000", 2024-09-18T04:39:42.469 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4982999999999997e-05 2024-09-18T04:39:42.469 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.469 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.469 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:42.469 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326090+0000", 2024-09-18T04:39:42.469 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015207900000000001 2024-09-18T04:39:42.469 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.469 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.469 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:42.469 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326242+0000", 2024-09-18T04:39:42.470 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015163500000000001 2024-09-18T04:39:42.470 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.470 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.470 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:42.470 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343015+0000", 2024-09-18T04:39:42.470 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016773414 2024-09-18T04:39:42.470 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.470 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.470 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:42.470 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345664+0000", 2024-09-18T04:39:42.470 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026487870000000001 2024-09-18T04:39:42.470 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.470 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.470 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:42.470 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345739+0000", 2024-09-18T04:39:42.471 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5018999999999994e-05 2024-09-18T04:39:42.471 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.471 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.471 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:42.471 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345812+0000", 2024-09-18T04:39:42.471 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3547000000000006e-05 2024-09-18T04:39:42.471 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.471 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:42.471 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.471 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.471 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.471 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:48 2.6 2:61db7a3a:::rbd_data.125fc09df6d1.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:42.471 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:39:42.471 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 60.114388292999998, 2024-09-18T04:39:42.472 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.025064388, 2024-09-18T04:39:42.472 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:42.472 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.472 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:42.472 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:42.472 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:42.472 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 48 2024-09-18T04:39:42.472 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.472 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:42.472 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.473 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:42.473 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:39:42.473 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.473 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.473 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.473 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:42.473 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:39:42.473 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.473 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.473 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.473 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:42.473 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325897+0000", 2024-09-18T04:39:42.473 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:42.473 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.473 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.474 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:42.474 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333697+0000", 2024-09-18T04:39:42.474 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0078002649999999998 2024-09-18T04:39:42.474 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.474 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.474 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:42.474 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333716+0000", 2024-09-18T04:39:42.474 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8910999999999999e-05 2024-09-18T04:39:42.474 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.474 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.474 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.474 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333728+0000", 2024-09-18T04:39:42.474 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1936000000000001e-05 2024-09-18T04:39:42.474 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.475 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.475 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:42.475 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333792+0000", 2024-09-18T04:39:42.475 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4010999999999994e-05 2024-09-18T04:39:42.475 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.475 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.475 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:42.475 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.334082+0000", 2024-09-18T04:39:42.475 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00028955799999999998 2024-09-18T04:39:42.475 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.475 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.475 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:42.475 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.334326+0000", 2024-09-18T04:39:42.475 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00024399099999999999 2024-09-18T04:39:42.475 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.476 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.476 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:42.476 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350745+0000", 2024-09-18T04:39:42.476 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016419474999999999 2024-09-18T04:39:42.476 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.476 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.476 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:42.476 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350808+0000", 2024-09-18T04:39:42.476 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3244999999999995e-05 2024-09-18T04:39:42.476 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.476 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.476 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:42.476 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350892+0000", 2024-09-18T04:39:42.477 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4074999999999997e-05 2024-09-18T04:39:42.477 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.477 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.477 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:42.477 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350963+0000", 2024-09-18T04:39:42.477 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0412000000000003e-05 2024-09-18T04:39:42.477 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.477 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:42.477 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.477 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.477 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.477 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:51 2.6 2:775d7b11:::rbd_data.125fc09df6d1.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:42.477 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:39:42.477 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 60.106536157000001, 2024-09-18T04:39:42.478 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.023824221, 2024-09-18T04:39:42.478 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:42.478 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.478 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:42.478 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:42.478 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:42.478 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 51 2024-09-18T04:39:42.478 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.478 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:42.478 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.478 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:42.478 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:39:42.478 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.478 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.479 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.479 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:42.479 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:39:42.479 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.479 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.479 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.479 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:42.479 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333749+0000", 2024-09-18T04:39:42.479 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:42.479 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.479 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.479 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:42.479 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340899+0000", 2024-09-18T04:39:42.479 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0071500340000000004 2024-09-18T04:39:42.479 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.480 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.480 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:42.480 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340929+0000", 2024-09-18T04:39:42.480 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0093000000000001e-05 2024-09-18T04:39:42.480 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.480 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.480 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.480 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340943+0000", 2024-09-18T04:39:42.480 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4019000000000001e-05 2024-09-18T04:39:42.480 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.480 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.480 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:42.480 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343834+0000", 2024-09-18T04:39:42.480 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028918120000000001 2024-09-18T04:39:42.481 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.481 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.481 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:42.481 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344020+0000", 2024-09-18T04:39:42.481 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000185892 2024-09-18T04:39:42.481 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.481 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.481 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:42.481 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344190+0000", 2024-09-18T04:39:42.481 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016946800000000001 2024-09-18T04:39:42.481 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.481 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.481 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:42.481 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.351291+0000", 2024-09-18T04:39:42.482 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.007100805 2024-09-18T04:39:42.482 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.482 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.482 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:42.482 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357470+0000", 2024-09-18T04:39:42.482 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0061797570000000001 2024-09-18T04:39:42.482 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.482 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.482 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:42.482 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357533+0000", 2024-09-18T04:39:42.482 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2906999999999999e-05 2024-09-18T04:39:42.482 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.482 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.482 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:42.483 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357575+0000", 2024-09-18T04:39:42.483 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1403999999999999e-05 2024-09-18T04:39:42.483 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.483 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:42.483 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.483 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.483 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.483 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:56 2.4 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:42.483 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:39:42.483 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 60.090491063999998, 2024-09-18T04:39:42.483 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.034295870999999999, 2024-09-18T04:39:42.483 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:42.483 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.483 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:42.484 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:42.484 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:42.484 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 56 2024-09-18T04:39:42.484 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.484 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:42.484 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.484 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:42.484 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:39:42.484 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.484 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.484 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.484 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:42.484 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:39:42.484 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.485 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.485 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.485 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:42.485 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349794+0000", 2024-09-18T04:39:42.485 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:42.485 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.485 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.485 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:42.485 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366632+0000", 2024-09-18T04:39:42.485 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01683755 2024-09-18T04:39:42.485 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.485 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.485 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:42.485 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366669+0000", 2024-09-18T04:39:42.486 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7351000000000002e-05 2024-09-18T04:39:42.486 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.486 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.486 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.486 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366686+0000", 2024-09-18T04:39:42.486 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6507999999999998e-05 2024-09-18T04:39:42.486 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.486 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.486 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:42.486 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366962+0000", 2024-09-18T04:39:42.486 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00027655900000000001 2024-09-18T04:39:42.486 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.486 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.486 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:42.486 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.367452+0000", 2024-09-18T04:39:42.487 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00049025299999999996 2024-09-18T04:39:42.487 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.487 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.487 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:42.487 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368113+0000", 2024-09-18T04:39:42.487 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00066093399999999996 2024-09-18T04:39:42.487 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.487 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.487 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:42.487 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383719+0000", 2024-09-18T04:39:42.487 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015606106999999999 2024-09-18T04:39:42.487 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.487 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.487 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:42.487 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384035+0000", 2024-09-18T04:39:42.488 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00031590399999999999 2024-09-18T04:39:42.488 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.488 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.488 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:42.488 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384067+0000", 2024-09-18T04:39:42.488 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.129e-05 2024-09-18T04:39:42.488 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.488 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.488 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:42.488 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384091+0000", 2024-09-18T04:39:42.488 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4788000000000001e-05 2024-09-18T04:39:42.488 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.488 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:42.488 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.489 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.489 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.489 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:57 2.0 e51/17 2:0cdd070e:::rbd_data.125fc09df6d1.000000000000000d:head v 51'21, mlcod=51'19)", 2024-09-18T04:39:42.489 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:39:42.489 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 60.082663576999998, 2024-09-18T04:39:42.489 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.025216311000000002, 2024-09-18T04:39:42.489 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:42.489 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.489 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:42.489 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.489 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:42.489 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:39:42.489 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.489 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.489 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.490 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:42.490 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:39:42.490 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.490 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.490 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.490 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:42.490 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:39:42.490 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:42.490 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.490 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.490 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:42.490 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363844+0000", 2024-09-18T04:39:42.490 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0062218450000000002 2024-09-18T04:39:42.490 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.491 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.491 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:42.491 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363879+0000", 2024-09-18T04:39:42.491 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4382999999999999e-05 2024-09-18T04:39:42.491 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.491 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.491 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.491 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363893+0000", 2024-09-18T04:39:42.491 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4348999999999999e-05 2024-09-18T04:39:42.491 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.491 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.491 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:42.491 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363977+0000", 2024-09-18T04:39:42.491 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3499999999999997e-05 2024-09-18T04:39:42.492 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.492 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.492 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:42.492 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364009+0000", 2024-09-18T04:39:42.492 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2292000000000002e-05 2024-09-18T04:39:42.492 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.492 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.492 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:42.492 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.382781+0000", 2024-09-18T04:39:42.492 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018771626 2024-09-18T04:39:42.492 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.492 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.492 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:42.492 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.382839+0000", 2024-09-18T04:39:42.492 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8758000000000002e-05 2024-09-18T04:39:42.493 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.493 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:42.493 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.493 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.493 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.493 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:59 2.0 e51/17 2:19c52cde:::rbd_data.125fc09df6d1.000000000000000f:head v 51'22, mlcod=51'19)", 2024-09-18T04:39:42.493 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:39:42.493 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 60.076003651000001, 2024-09-18T04:39:42.493 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.019497112, 2024-09-18T04:39:42.493 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:42.493 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.493 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:42.493 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.493 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:42.494 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:39:42.494 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.494 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.494 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.494 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:42.494 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:39:42.494 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.494 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.494 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.494 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:42.494 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364282+0000", 2024-09-18T04:39:42.494 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:42.494 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.494 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.494 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:42.495 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371823+0000", 2024-09-18T04:39:42.495 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0075411829999999999 2024-09-18T04:39:42.495 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.495 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.495 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:42.495 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371837+0000", 2024-09-18T04:39:42.495 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4336e-05 2024-09-18T04:39:42.495 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.495 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.495 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.495 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371846+0000", 2024-09-18T04:39:42.495 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9770000000000006e-06 2024-09-18T04:39:42.495 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.495 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.496 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:42.496 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.373566+0000", 2024-09-18T04:39:42.496 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017192539999999999 2024-09-18T04:39:42.496 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.496 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.496 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:42.496 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.373595+0000", 2024-09-18T04:39:42.496 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8975999999999998e-05 2024-09-18T04:39:42.496 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.496 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.496 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:42.496 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383740+0000", 2024-09-18T04:39:42.496 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01014525 2024-09-18T04:39:42.496 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.497 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.497 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:42.497 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383780+0000", 2024-09-18T04:39:42.497 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0275999999999999e-05 2024-09-18T04:39:42.497 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.497 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:42.497 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.497 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.497 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.497 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:34 2.6 2:66289171:ns::rbd_data.133e813ab5bc.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:42.497 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.524015+0000", 2024-09-18T04:39:42.497 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 59.916272042000003, 2024-09-18T04:39:42.497 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027582111999999999, 2024-09-18T04:39:42.497 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:42.498 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.498 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:42.498 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:42.498 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:42.498 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 34 2024-09-18T04:39:42.498 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.498 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:42.498 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.498 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:42.498 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524015+0000", 2024-09-18T04:39:42.498 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.498 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.498 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.498 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:42.498 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524015+0000", 2024-09-18T04:39:42.499 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.499 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.499 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.499 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:42.499 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524012+0000", 2024-09-18T04:39:42.499 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:42.499 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.499 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.499 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:42.499 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534956+0000", 2024-09-18T04:39:42.499 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010943403000000001 2024-09-18T04:39:42.499 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.499 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.499 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:42.499 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534978+0000", 2024-09-18T04:39:42.500 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1965999999999999e-05 2024-09-18T04:39:42.500 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.500 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.500 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.500 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534990+0000", 2024-09-18T04:39:42.500 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2065999999999999e-05 2024-09-18T04:39:42.500 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.500 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.500 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:42.500 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535065+0000", 2024-09-18T04:39:42.500 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5523000000000002e-05 2024-09-18T04:39:42.500 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.500 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.500 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:42.501 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535208+0000", 2024-09-18T04:39:42.501 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014309500000000001 2024-09-18T04:39:42.501 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.501 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.501 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:42.501 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535343+0000", 2024-09-18T04:39:42.501 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013437400000000001 2024-09-18T04:39:42.501 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.501 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.501 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:42.501 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.546576+0000", 2024-09-18T04:39:42.501 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011233425999999999 2024-09-18T04:39:42.501 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.501 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.502 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:42.502 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551498+0000", 2024-09-18T04:39:42.502 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0049219349999999997 2024-09-18T04:39:42.502 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.502 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.502 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:42.502 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551554+0000", 2024-09-18T04:39:42.502 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6400999999999998e-05 2024-09-18T04:39:42.502 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.502 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.502 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:42.502 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551597+0000", 2024-09-18T04:39:42.502 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2283000000000002e-05 2024-09-18T04:39:42.502 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.503 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:42.503 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.503 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.503 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.503 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:33 2.1 e51/24 2:8b40f1c5:ns::rbd_data.133e813ab5bc.0000000000000000:head v 51'7, mlcod=51'5)", 2024-09-18T04:39:42.503 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.531767+0000", 2024-09-18T04:39:42.503 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 59.908520125000003, 2024-09-18T04:39:42.503 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015370295000000001, 2024-09-18T04:39:42.503 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:42.503 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.503 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:42.503 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.503 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:42.503 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531767+0000", 2024-09-18T04:39:42.503 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.504 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.504 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.504 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:42.504 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531767+0000", 2024-09-18T04:39:42.504 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.504 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.504 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.504 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:42.504 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531765+0000", 2024-09-18T04:39:42.504 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:42.504 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.504 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.504 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:42.504 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534258+0000", 2024-09-18T04:39:42.505 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002493561 2024-09-18T04:39:42.505 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.505 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.505 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:42.505 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534288+0000", 2024-09-18T04:39:42.505 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.976e-05 2024-09-18T04:39:42.505 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.505 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.505 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.505 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534300+0000", 2024-09-18T04:39:42.505 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1928e-05 2024-09-18T04:39:42.505 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.505 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.505 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:42.506 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534367+0000", 2024-09-18T04:39:42.506 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7045999999999995e-05 2024-09-18T04:39:42.506 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.506 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.506 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:42.506 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534394+0000", 2024-09-18T04:39:42.506 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6903999999999999e-05 2024-09-18T04:39:42.506 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.506 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.506 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:42.506 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.547073+0000", 2024-09-18T04:39:42.506 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012679365 2024-09-18T04:39:42.506 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.506 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.506 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:42.507 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.547137+0000", 2024-09-18T04:39:42.507 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3691999999999999e-05 2024-09-18T04:39:42.507 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.507 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:42.507 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.507 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.507 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.507 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:39 2.4 2:304364d1:ns::rbd_data.133e813ab5bc.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:42.507 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.535013+0000", 2024-09-18T04:39:42.507 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 59.905273190000003, 2024-09-18T04:39:42.507 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027764487000000001, 2024-09-18T04:39:42.507 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:42.507 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.507 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:42.508 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:42.508 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:42.508 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 39 2024-09-18T04:39:42.508 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.508 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:42.508 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.508 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:42.508 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535013+0000", 2024-09-18T04:39:42.508 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.508 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.508 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.508 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:42.508 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535013+0000", 2024-09-18T04:39:42.508 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.509 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.509 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.509 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:42.509 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535012+0000", 2024-09-18T04:39:42.509 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:42.509 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.509 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.509 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:42.509 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545016+0000", 2024-09-18T04:39:42.509 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01000322 2024-09-18T04:39:42.509 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.509 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.509 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:42.509 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545035+0000", 2024-09-18T04:39:42.510 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9845999999999999e-05 2024-09-18T04:39:42.510 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.510 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.510 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.510 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545045+0000", 2024-09-18T04:39:42.510 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3640000000000005e-06 2024-09-18T04:39:42.510 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.510 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.510 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:42.510 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545152+0000", 2024-09-18T04:39:42.510 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000107025 2024-09-18T04:39:42.510 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.510 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.510 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:42.511 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545338+0000", 2024-09-18T04:39:42.511 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018643500000000001 2024-09-18T04:39:42.511 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.511 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.511 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:42.511 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545492+0000", 2024-09-18T04:39:42.511 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015393199999999999 2024-09-18T04:39:42.511 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.511 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.511 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:42.511 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.553549+0000", 2024-09-18T04:39:42.511 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0080569170000000002 2024-09-18T04:39:42.511 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.511 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.511 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:42.512 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562693+0000", 2024-09-18T04:39:42.512 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0091441609999999996 2024-09-18T04:39:42.512 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.512 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.512 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:42.512 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562748+0000", 2024-09-18T04:39:42.512 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5047000000000003e-05 2024-09-18T04:39:42.512 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.512 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.512 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:42.512 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562778+0000", 2024-09-18T04:39:42.512 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9652e-05 2024-09-18T04:39:42.512 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.512 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:42.513 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.513 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.513 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.513 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:40 2.6 2:784d6cd1:ns::rbd_data.133e813ab5bc.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:42.513 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.545062+0000", 2024-09-18T04:39:42.513 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 59.895224139, 2024-09-18T04:39:42.513 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.032771623, 2024-09-18T04:39:42.513 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:42.513 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.513 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:42.513 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:42.513 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:42.513 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 40 2024-09-18T04:39:42.513 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.513 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:42.514 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.514 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:42.514 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545062+0000", 2024-09-18T04:39:42.514 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.514 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.514 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.514 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:42.514 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545062+0000", 2024-09-18T04:39:42.514 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.514 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.514 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.514 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:42.514 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545061+0000", 2024-09-18T04:39:42.514 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:42.515 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.515 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.515 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:42.515 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552477+0000", 2024-09-18T04:39:42.515 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.007415914 2024-09-18T04:39:42.515 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.515 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.515 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:42.515 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552500+0000", 2024-09-18T04:39:42.515 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3546000000000001e-05 2024-09-18T04:39:42.515 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.515 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.515 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.515 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552515+0000", 2024-09-18T04:39:42.515 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4606999999999999e-05 2024-09-18T04:39:42.516 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.516 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.516 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:42.516 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552661+0000", 2024-09-18T04:39:42.516 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000146007 2024-09-18T04:39:42.516 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.516 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.516 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:42.516 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552876+0000", 2024-09-18T04:39:42.516 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000214618 2024-09-18T04:39:42.516 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.516 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.516 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:42.516 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.553061+0000", 2024-09-18T04:39:42.517 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018564100000000001 2024-09-18T04:39:42.517 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.517 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.517 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:42.517 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.565612+0000", 2024-09-18T04:39:42.517 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012550301999999999 2024-09-18T04:39:42.517 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.517 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.517 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:42.517 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577695+0000", 2024-09-18T04:39:42.517 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012083493000000001 2024-09-18T04:39:42.517 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.517 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.517 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:42.518 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577772+0000", 2024-09-18T04:39:42.518 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7158999999999998e-05 2024-09-18T04:39:42.518 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.518 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.518 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:42.518 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577834+0000", 2024-09-18T04:39:42.518 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1851000000000001e-05 2024-09-18T04:39:42.518 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.518 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:42.518 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.518 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.518 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.518 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:43 2.6 2:759f27b6:ns::rbd_data.133e813ab5bc.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:42.518 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.555868+0000", 2024-09-18T04:39:42.519 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 59.884418644999997, 2024-09-18T04:39:42.519 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.022332497999999999, 2024-09-18T04:39:42.519 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:42.519 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.519 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:42.519 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:42.519 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:42.519 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 43 2024-09-18T04:39:42.519 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.519 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:42.519 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.519 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:42.519 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555868+0000", 2024-09-18T04:39:42.519 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.520 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.520 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.520 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:42.520 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555868+0000", 2024-09-18T04:39:42.520 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.520 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.520 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.520 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:42.520 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555865+0000", 2024-09-18T04:39:42.520 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999971 2024-09-18T04:39:42.520 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.520 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.520 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:42.520 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561031+0000", 2024-09-18T04:39:42.520 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0051658770000000001 2024-09-18T04:39:42.521 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.521 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.521 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:42.521 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561048+0000", 2024-09-18T04:39:42.521 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6569e-05 2024-09-18T04:39:42.521 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.521 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.521 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.521 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561055+0000", 2024-09-18T04:39:42.521 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.328e-06 2024-09-18T04:39:42.521 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.521 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.521 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:42.521 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561114+0000", 2024-09-18T04:39:42.522 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9540000000000003e-05 2024-09-18T04:39:42.522 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.522 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.522 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:42.522 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561314+0000", 2024-09-18T04:39:42.522 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019965799999999999 2024-09-18T04:39:42.522 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.522 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.522 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:42.522 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561473+0000", 2024-09-18T04:39:42.522 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000158904 2024-09-18T04:39:42.522 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.522 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.522 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:42.523 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.567015+0000", 2024-09-18T04:39:42.523 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055420859999999999 2024-09-18T04:39:42.523 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.523 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.523 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:42.523 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577982+0000", 2024-09-18T04:39:42.523 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010966872000000001 2024-09-18T04:39:42.523 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.523 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.523 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:42.523 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578162+0000", 2024-09-18T04:39:42.523 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018050999999999999 2024-09-18T04:39:42.523 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.523 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.524 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:42.524 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578200+0000", 2024-09-18T04:39:42.524 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8000999999999998e-05 2024-09-18T04:39:42.524 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.524 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:42.524 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.524 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.524 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.524 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:48 2.6 2:7c225365:ns::rbd_data.133e813ab5bc.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:42.524 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:39:42.524 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 59.876619421999997, 2024-09-18T04:39:42.524 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.024576377, 2024-09-18T04:39:42.524 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:42.524 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.525 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:42.525 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:42.525 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:42.525 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 48 2024-09-18T04:39:42.525 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.525 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:42.525 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.525 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:42.525 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:39:42.525 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.525 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.525 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.525 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:42.525 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:39:42.525 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.526 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.526 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.526 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:42.526 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.563666+0000", 2024-09-18T04:39:42.526 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:42.526 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.526 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.526 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:42.526 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570331+0000", 2024-09-18T04:39:42.526 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0066653329999999998 2024-09-18T04:39:42.526 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.526 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.526 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:42.526 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570348+0000", 2024-09-18T04:39:42.527 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6507e-05 2024-09-18T04:39:42.527 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.527 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.527 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.527 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570355+0000", 2024-09-18T04:39:42.527 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2749999999999998e-06 2024-09-18T04:39:42.527 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.527 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.527 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:42.527 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570396+0000", 2024-09-18T04:39:42.527 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0809000000000002e-05 2024-09-18T04:39:42.527 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.527 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.527 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:42.527 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570571+0000", 2024-09-18T04:39:42.528 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000174823 2024-09-18T04:39:42.528 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.528 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.528 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:42.528 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570722+0000", 2024-09-18T04:39:42.528 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015154599999999999 2024-09-18T04:39:42.528 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.528 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.528 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:42.528 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579756+0000", 2024-09-18T04:39:42.528 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0090337000000000004 2024-09-18T04:39:42.528 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.528 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.528 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:42.529 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588093+0000", 2024-09-18T04:39:42.529 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.008336553 2024-09-18T04:39:42.529 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.529 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.529 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:42.529 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588173+0000", 2024-09-18T04:39:42.529 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0333999999999994e-05 2024-09-18T04:39:42.529 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.529 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.529 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:42.529 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588244+0000", 2024-09-18T04:39:42.529 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0573000000000003e-05 2024-09-18T04:39:42.529 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.529 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:42.530 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.530 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.530 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.530 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:49 2.4 2:20f0b0c9:ns::rbd_data.133e813ab5bc.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:42.530 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.570370+0000", 2024-09-18T04:39:42.530 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 59.869917094999998, 2024-09-18T04:39:42.530 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.029633981, 2024-09-18T04:39:42.530 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:42.530 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.530 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:42.530 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:42.530 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:42.530 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 49 2024-09-18T04:39:42.530 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.531 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:42.531 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.531 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:42.531 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570370+0000", 2024-09-18T04:39:42.531 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.531 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.531 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.531 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:42.531 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570370+0000", 2024-09-18T04:39:42.531 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.531 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.531 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.531 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:42.531 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570369+0000", 2024-09-18T04:39:42.532 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:42.532 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.532 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.532 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:42.532 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579090+0000", 2024-09-18T04:39:42.532 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0087212339999999996 2024-09-18T04:39:42.532 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.532 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.532 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:42.532 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579109+0000", 2024-09-18T04:39:42.532 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9120999999999999e-05 2024-09-18T04:39:42.532 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.532 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.532 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.533 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579123+0000", 2024-09-18T04:39:42.533 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4382000000000001e-05 2024-09-18T04:39:42.533 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.533 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.533 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:42.533 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579221+0000", 2024-09-18T04:39:42.533 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.7503999999999993e-05 2024-09-18T04:39:42.533 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.533 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.533 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:42.533 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579553+0000", 2024-09-18T04:39:42.533 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00033254200000000002 2024-09-18T04:39:42.533 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.533 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.534 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:42.534 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579817+0000", 2024-09-18T04:39:42.534 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00026317099999999998 2024-09-18T04:39:42.534 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.534 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.534 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:42.534 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.599905+0000", 2024-09-18T04:39:42.534 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020088575000000001 2024-09-18T04:39:42.534 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.534 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.534 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:42.534 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.599923+0000", 2024-09-18T04:39:42.534 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7425999999999999e-05 2024-09-18T04:39:42.534 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.535 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.535 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:42.535 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.599973+0000", 2024-09-18T04:39:42.535 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0649999999999998e-05 2024-09-18T04:39:42.535 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.535 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.535 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:42.535 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.600004+0000", 2024-09-18T04:39:42.535 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0361999999999998e-05 2024-09-18T04:39:42.535 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.535 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:42.535 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.535 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.535 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.535 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:50 2.1 e51/24 2:8c706374:ns::rbd_data.133e813ab5bc.000000000000000a:head v 51'8, mlcod=51'6)", 2024-09-18T04:39:42.536 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.577029+0000", 2024-09-18T04:39:42.536 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 59.863257724, 2024-09-18T04:39:42.536 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.017362855, 2024-09-18T04:39:42.536 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:42.536 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.536 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:42.536 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.536 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:42.536 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577029+0000", 2024-09-18T04:39:42.536 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.536 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.536 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.536 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:42.536 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577029+0000", 2024-09-18T04:39:42.537 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.537 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.537 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.537 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:42.537 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577027+0000", 2024-09-18T04:39:42.537 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:42.537 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.537 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.537 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:42.537 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578880+0000", 2024-09-18T04:39:42.537 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0018532080000000001 2024-09-18T04:39:42.537 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.537 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.537 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:42.538 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578906+0000", 2024-09-18T04:39:42.538 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5573999999999999e-05 2024-09-18T04:39:42.538 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.538 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.538 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.538 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578920+0000", 2024-09-18T04:39:42.538 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4483e-05 2024-09-18T04:39:42.538 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.538 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.538 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:42.538 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578983+0000", 2024-09-18T04:39:42.538 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2248000000000004e-05 2024-09-18T04:39:42.538 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.538 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.539 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:42.539 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579016+0000", 2024-09-18T04:39:42.539 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3364000000000001e-05 2024-09-18T04:39:42.539 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.539 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.539 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:42.539 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594342+0000", 2024-09-18T04:39:42.539 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01532564 2024-09-18T04:39:42.539 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.539 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.539 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:42.539 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594392+0000", 2024-09-18T04:39:42.539 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0068999999999998e-05 2024-09-18T04:39:42.539 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.539 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:42.540 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.540 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.540 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.540 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:45 2.0 e51/17 2:0fbfa3dc:ns::rbd_data.133e813ab5bc.0000000000000007:head v 51'24, mlcod=51'22)", 2024-09-18T04:39:42.540 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.581892+0000", 2024-09-18T04:39:42.540 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 59.858395051999999, 2024-09-18T04:39:42.540 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015646006, 2024-09-18T04:39:42.540 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:42.540 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.540 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:42.540 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.540 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:42.540 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581892+0000", 2024-09-18T04:39:42.540 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.541 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.541 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.541 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:42.541 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581892+0000", 2024-09-18T04:39:42.541 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.541 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.541 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.541 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:42.541 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581890+0000", 2024-09-18T04:39:42.541 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:42.541 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.541 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.541 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:42.541 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586188+0000", 2024-09-18T04:39:42.541 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0042985769999999996 2024-09-18T04:39:42.542 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.542 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.542 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:42.542 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586210+0000", 2024-09-18T04:39:42.542 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1069000000000001e-05 2024-09-18T04:39:42.542 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.542 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.542 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.542 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586221+0000", 2024-09-18T04:39:42.542 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1287000000000001e-05 2024-09-18T04:39:42.542 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.543 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.543 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:42.543 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586313+0000", 2024-09-18T04:39:42.543 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1935999999999995e-05 2024-09-18T04:39:42.543 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.543 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.543 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:42.543 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586368+0000", 2024-09-18T04:39:42.543 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5578000000000002e-05 2024-09-18T04:39:42.543 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.543 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.543 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:42.543 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.597473+0000", 2024-09-18T04:39:42.543 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011105139999999999 2024-09-18T04:39:42.544 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.544 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.544 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:42.544 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.597538+0000", 2024-09-18T04:39:42.544 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4133000000000004e-05 2024-09-18T04:39:42.544 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.544 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:42.544 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.544 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.544 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.544 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:53 2.4 2:39f7f174:ns::rbd_data.133e813ab5bc.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:42.544 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.584263+0000", 2024-09-18T04:39:42.544 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 59.856023178000001, 2024-09-18T04:39:42.544 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018701180000000001, 2024-09-18T04:39:42.545 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:42.545 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.545 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:42.545 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:42.545 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:42.545 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 53 2024-09-18T04:39:42.545 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.545 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:42.545 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.545 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:42.545 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584263+0000", 2024-09-18T04:39:42.545 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.545 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.545 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.546 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:42.546 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584263+0000", 2024-09-18T04:39:42.546 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.546 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.546 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.546 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:42.546 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584262+0000", 2024-09-18T04:39:42.546 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:42.546 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.546 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.546 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:42.546 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.591617+0000", 2024-09-18T04:39:42.546 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0073550600000000001 2024-09-18T04:39:42.546 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.546 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.547 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:42.547 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.591637+0000", 2024-09-18T04:39:42.547 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0432999999999999e-05 2024-09-18T04:39:42.547 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.547 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.547 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.547 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.591646+0000", 2024-09-18T04:39:42.547 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7849999999999997e-06 2024-09-18T04:39:42.547 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.547 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.547 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:42.547 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593478+0000", 2024-09-18T04:39:42.547 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001831495 2024-09-18T04:39:42.547 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.548 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.548 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:42.548 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593661+0000", 2024-09-18T04:39:42.548 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018364999999999999 2024-09-18T04:39:42.548 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.548 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.548 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:42.548 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593834+0000", 2024-09-18T04:39:42.548 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000173065 2024-09-18T04:39:42.548 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.548 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.548 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:42.548 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.600670+0000", 2024-09-18T04:39:42.548 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0068354970000000003 2024-09-18T04:39:42.548 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.549 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.549 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:42.549 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602851+0000", 2024-09-18T04:39:42.549 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002181653 2024-09-18T04:39:42.549 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.549 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.549 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:42.549 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602930+0000", 2024-09-18T04:39:42.549 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8045000000000002e-05 2024-09-18T04:39:42.549 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.549 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.549 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:42.549 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602965+0000", 2024-09-18T04:39:42.549 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5102999999999999e-05 2024-09-18T04:39:42.549 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.550 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:42.550 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.550 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.550 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.550 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:56 2.6 2:72453c3e:ns::rbd_data.133e813ab5bc.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:42.550 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.595988+0000", 2024-09-18T04:39:42.550 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 59.844298776000002, 2024-09-18T04:39:42.550 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.030296172, 2024-09-18T04:39:42.550 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:42.550 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.550 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:42.550 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:42.550 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:42.550 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 56 2024-09-18T04:39:42.550 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.551 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:42.551 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.551 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:42.551 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.595988+0000", 2024-09-18T04:39:42.551 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.551 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.551 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.551 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:42.551 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.595988+0000", 2024-09-18T04:39:42.551 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:42.551 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.551 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.551 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:42.551 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.595986+0000", 2024-09-18T04:39:42.551 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:42.552 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.552 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.552 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:42.552 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605262+0000", 2024-09-18T04:39:42.552 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0092767240000000001 2024-09-18T04:39:42.552 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.552 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.552 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:42.552 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605282+0000", 2024-09-18T04:39:42.552 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9307e-05 2024-09-18T04:39:42.552 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.552 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.552 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.552 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605292+0000", 2024-09-18T04:39:42.553 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9510000000000001e-06 2024-09-18T04:39:42.553 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.553 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.553 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:42.553 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605369+0000", 2024-09-18T04:39:42.553 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7846e-05 2024-09-18T04:39:42.553 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.553 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.553 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:42.553 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605668+0000", 2024-09-18T04:39:42.553 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00029896099999999999 2024-09-18T04:39:42.553 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.553 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.553 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:42.553 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605906+0000", 2024-09-18T04:39:42.554 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000237758 2024-09-18T04:39:42.554 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.554 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.554 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:42.554 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.613532+0000", 2024-09-18T04:39:42.554 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0076260590000000001 2024-09-18T04:39:42.554 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.554 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.554 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:42.554 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.626116+0000", 2024-09-18T04:39:42.554 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012584092999999999 2024-09-18T04:39:42.554 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.554 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.554 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:42.554 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.626216+0000", 2024-09-18T04:39:42.555 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9630999999999995e-05 2024-09-18T04:39:42.555 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:42.555 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:42.555 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:42.555 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.626284+0000", 2024-09-18T04:39:42.555 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8034999999999995e-05 2024-09-18T04:39:42.555 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.555 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:42.555 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.555 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:42.555 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:42.555 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:42.557 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:39:42.557 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:42.548+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:42.658 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:42.648+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:42.674 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:42.674 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:42.675 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:39:42.675 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:42.684 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:39:42.700 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-09-18T04:39:42.758 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:42.756+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:42.787 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:42.787 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:42.787 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:39:42.787 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:39:42.787 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:42.797 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:39:42.859 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:42.856+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:42.905 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:42.905 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:39:42.905 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:39:42.905 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:39:42.905 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.905 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:39:42.905 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:39:42.906 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 135.11253412400001, 2024-09-18T04:39:42.906 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017399218000000001, 2024-09-18T04:39:42.906 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:42.906 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.906 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:42.906 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:39:42.906 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:39:42.906 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 114 2024-09-18T04:39:42.906 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.906 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:42.906 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.906 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:42.906 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:39:42.907 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:42.907 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.907 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.907 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:42.907 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:39:42.907 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:42.907 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.907 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.907 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:42.907 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786900+0000", 2024-09-18T04:39:42.907 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:42.907 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.907 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.907 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:42.908 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786908+0000", 2024-09-18T04:39:42.908 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8199999999999997e-06 2024-09-18T04:39:42.908 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.908 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.908 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:42.908 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786925+0000", 2024-09-18T04:39:42.908 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6849000000000001e-05 2024-09-18T04:39:42.908 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.908 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.908 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.908 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786934+0000", 2024-09-18T04:39:42.908 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6299999999999993e-06 2024-09-18T04:39:42.908 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.909 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.909 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:42.909 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800926+0000", 2024-09-18T04:39:42.909 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013991632 2024-09-18T04:39:42.909 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.909 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.909 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:42.909 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801060+0000", 2024-09-18T04:39:42.909 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013378199999999999 2024-09-18T04:39:42.909 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.909 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.909 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:42.909 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801166+0000", 2024-09-18T04:39:42.909 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010607099999999999 2024-09-18T04:39:42.910 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.910 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.910 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:42.910 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803026+0000", 2024-09-18T04:39:42.910 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001860469 2024-09-18T04:39:42.910 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.910 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.910 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:42.910 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804245+0000", 2024-09-18T04:39:42.910 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0012185900000000001 2024-09-18T04:39:42.910 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.910 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.910 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:42.910 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804278+0000", 2024-09-18T04:39:42.911 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3272000000000003e-05 2024-09-18T04:39:42.911 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.911 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.911 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:42.911 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804300+0000", 2024-09-18T04:39:42.911 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2266e-05 2024-09-18T04:39:42.911 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:42.911 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:42.911 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:42.911 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.911 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.911 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:39:42.911 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:39:42.911 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 135.11162122600001, 2024-09-18T04:39:42.912 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017521662, 2024-09-18T04:39:42.912 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:42.912 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.912 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:42.912 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:39:42.912 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:39:42.912 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 116 2024-09-18T04:39:42.912 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.912 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:42.912 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.912 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:42.912 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:39:42.912 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:42.912 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.913 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.913 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:42.913 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:39:42.913 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:42.913 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.913 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.913 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:42.913 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787813+0000", 2024-09-18T04:39:42.913 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:42.913 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.913 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.913 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:42.913 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787824+0000", 2024-09-18T04:39:42.913 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1157e-05 2024-09-18T04:39:42.914 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.914 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.914 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:42.914 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787840+0000", 2024-09-18T04:39:42.914 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6269e-05 2024-09-18T04:39:42.914 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.914 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.914 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.914 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787853+0000", 2024-09-18T04:39:42.914 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3156000000000001e-05 2024-09-18T04:39:42.914 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.914 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.914 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:42.914 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803110+0000", 2024-09-18T04:39:42.914 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015256736 2024-09-18T04:39:42.915 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.915 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.915 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:42.915 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803197+0000", 2024-09-18T04:39:42.915 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7195999999999997e-05 2024-09-18T04:39:42.915 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.915 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.915 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:42.915 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803308+0000", 2024-09-18T04:39:42.915 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111005 2024-09-18T04:39:42.915 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.915 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.915 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:42.915 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804316+0000", 2024-09-18T04:39:42.916 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010073949999999999 2024-09-18T04:39:42.916 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.916 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.916 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:42.916 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805232+0000", 2024-09-18T04:39:42.916 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091636300000000001 2024-09-18T04:39:42.916 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.916 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.916 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:42.916 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805277+0000", 2024-09-18T04:39:42.916 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4864000000000003e-05 2024-09-18T04:39:42.916 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.916 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.916 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:42.917 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805336+0000", 2024-09-18T04:39:42.917 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8968000000000002e-05 2024-09-18T04:39:42.917 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:42.917 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:42.917 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:42.917 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.917 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.917 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:39:42.917 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:39:42.917 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 135.11156917100001, 2024-09-18T04:39:42.917 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017983453, 2024-09-18T04:39:42.917 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:42.917 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.917 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:42.917 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:39:42.918 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:39:42.918 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 117 2024-09-18T04:39:42.918 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.918 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:42.918 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.918 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:42.918 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:39:42.918 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:42.918 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.918 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.918 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:42.918 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:39:42.918 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:42.918 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.919 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.919 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:42.919 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787865+0000", 2024-09-18T04:39:42.919 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:42.919 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.919 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.919 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:42.919 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787889+0000", 2024-09-18T04:39:42.919 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4244999999999999e-05 2024-09-18T04:39:42.919 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.919 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.919 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:42.919 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787905+0000", 2024-09-18T04:39:42.919 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5739999999999998e-05 2024-09-18T04:39:42.920 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.920 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.920 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.920 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787918+0000", 2024-09-18T04:39:42.920 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.252e-05 2024-09-18T04:39:42.920 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.920 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.920 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:42.920 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803693+0000", 2024-09-18T04:39:42.920 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015774872999999998 2024-09-18T04:39:42.921 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.921 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.921 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:42.921 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803775+0000", 2024-09-18T04:39:42.921 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2761000000000003e-05 2024-09-18T04:39:42.921 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.921 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.921 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:42.921 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803887+0000", 2024-09-18T04:39:42.921 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111708 2024-09-18T04:39:42.921 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.921 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.921 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:42.922 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804823+0000", 2024-09-18T04:39:42.922 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00093625100000000003 2024-09-18T04:39:42.922 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.922 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.922 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:42.922 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805804+0000", 2024-09-18T04:39:42.922 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00098114699999999992 2024-09-18T04:39:42.922 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.922 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.922 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:42.922 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805831+0000", 2024-09-18T04:39:42.922 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6401999999999999e-05 2024-09-18T04:39:42.922 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.922 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.922 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:42.923 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805850+0000", 2024-09-18T04:39:42.923 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8862e-05 2024-09-18T04:39:42.923 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:42.923 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:42.923 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:42.923 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.923 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.923 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:36 2.7 2:fe1b02ad:::rbd_data.125fc09df6d1.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:42.923 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:39:42.923 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 60.633713061000002, 2024-09-18T04:39:42.923 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.040387804999999999, 2024-09-18T04:39:42.923 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:42.923 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.924 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:42.924 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:42.924 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:42.924 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 36 2024-09-18T04:39:42.924 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.924 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:42.924 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.924 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:42.924 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:39:42.924 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:42.924 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.924 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.924 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:42.924 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:39:42.925 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:42.925 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.925 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.925 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:42.925 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265720+0000", 2024-09-18T04:39:42.925 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:42.925 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.925 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.925 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:42.925 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272590+0000", 2024-09-18T04:39:42.925 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.006870508 2024-09-18T04:39:42.925 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.925 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.925 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:42.925 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272623+0000", 2024-09-18T04:39:42.926 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2716000000000003e-05 2024-09-18T04:39:42.926 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.926 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.926 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.926 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272637+0000", 2024-09-18T04:39:42.926 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4175999999999999e-05 2024-09-18T04:39:42.926 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.926 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.926 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:42.926 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272779+0000", 2024-09-18T04:39:42.926 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000141392 2024-09-18T04:39:42.926 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.926 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.926 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:42.927 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273031+0000", 2024-09-18T04:39:42.927 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00025249099999999998 2024-09-18T04:39:42.927 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.927 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.927 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:39:42.927 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273219+0000", 2024-09-18T04:39:42.927 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000187531 2024-09-18T04:39:42.927 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.927 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.927 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:42.927 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.302806+0000", 2024-09-18T04:39:42.927 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.029587553999999999 2024-09-18T04:39:42.927 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.927 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.928 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:42.928 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306004+0000", 2024-09-18T04:39:42.928 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0031982629999999998 2024-09-18T04:39:42.928 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.928 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.928 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:42.928 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306071+0000", 2024-09-18T04:39:42.928 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6829999999999995e-05 2024-09-18T04:39:42.928 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.928 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.928 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:42.928 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306110+0000", 2024-09-18T04:39:42.928 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8874000000000001e-05 2024-09-18T04:39:42.928 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:42.928 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:42.929 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:42.929 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.929 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.929 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:38 2.0 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:42.929 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:39:42.929 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 60.626759524000001, 2024-09-18T04:39:42.929 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.033754348000000003, 2024-09-18T04:39:42.929 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:42.929 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.929 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:42.929 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:42.929 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:42.929 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 38 2024-09-18T04:39:42.929 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.930 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:42.930 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.930 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:42.930 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:39:42.930 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:42.930 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.930 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.930 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:42.930 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:39:42.930 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:42.930 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.930 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.930 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:42.930 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272673+0000", 2024-09-18T04:39:42.930 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:42.931 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.931 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.931 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:42.931 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277166+0000", 2024-09-18T04:39:42.931 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044922249999999999 2024-09-18T04:39:42.931 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.931 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.931 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:42.931 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277189+0000", 2024-09-18T04:39:42.931 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3283000000000001e-05 2024-09-18T04:39:42.931 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.931 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.931 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.931 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277205+0000", 2024-09-18T04:39:42.932 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6305000000000001e-05 2024-09-18T04:39:42.932 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.932 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.932 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:42.932 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277290+0000", 2024-09-18T04:39:42.932 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5135000000000004e-05 2024-09-18T04:39:42.932 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.932 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.932 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:42.932 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277531+0000", 2024-09-18T04:39:42.932 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00024014699999999999 2024-09-18T04:39:42.932 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.932 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.933 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:42.933 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277793+0000", 2024-09-18T04:39:42.933 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00026224499999999998 2024-09-18T04:39:42.933 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.933 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.933 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:42.933 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303990+0000", 2024-09-18T04:39:42.933 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.026197295999999998 2024-09-18T04:39:42.933 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.933 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.934 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:42.934 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306340+0000", 2024-09-18T04:39:42.934 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.002349524 2024-09-18T04:39:42.934 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.934 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.934 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:42.934 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306394+0000", 2024-09-18T04:39:42.934 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3934999999999998e-05 2024-09-18T04:39:42.934 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.934 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.934 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:42.934 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306430+0000", 2024-09-18T04:39:42.934 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6690000000000003e-05 2024-09-18T04:39:42.934 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:42.935 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:42.935 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:42.935 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.935 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.935 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:39 2.7 2:e9c909c5:::rbd_data.125fc09df6d1.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:42.935 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:39:42.935 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 60.622199919000003, 2024-09-18T04:39:42.935 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.029110673, 2024-09-18T04:39:42.935 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:42.935 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.935 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:42.935 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:42.935 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:42.935 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 39 2024-09-18T04:39:42.936 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.936 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:42.936 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.936 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:42.936 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:39:42.936 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:42.936 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.936 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.936 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:42.936 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:39:42.936 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:42.936 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.936 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.936 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:42.937 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277234+0000", 2024-09-18T04:39:42.937 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:42.937 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.937 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.937 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:42.937 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281351+0000", 2024-09-18T04:39:42.937 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0041170920000000001 2024-09-18T04:39:42.937 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.937 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.937 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:42.937 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281375+0000", 2024-09-18T04:39:42.937 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.446e-05 2024-09-18T04:39:42.937 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.938 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.938 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.938 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281390+0000", 2024-09-18T04:39:42.938 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4309e-05 2024-09-18T04:39:42.938 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.938 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.938 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:42.938 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.282306+0000", 2024-09-18T04:39:42.938 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091662000000000004 2024-09-18T04:39:42.938 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.938 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.938 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:42.938 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.282613+0000", 2024-09-18T04:39:42.938 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00030663799999999997 2024-09-18T04:39:42.938 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.939 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.939 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:39:42.939 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283001+0000", 2024-09-18T04:39:42.939 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00038801599999999998 2024-09-18T04:39:42.939 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.939 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.939 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:42.939 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306239+0000", 2024-09-18T04:39:42.939 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023238466999999999 2024-09-18T04:39:42.939 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.939 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.939 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:42.939 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306259+0000", 2024-09-18T04:39:42.939 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9837999999999999e-05 2024-09-18T04:39:42.940 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.940 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.940 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:42.940 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306299+0000", 2024-09-18T04:39:42.940 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0194000000000002e-05 2024-09-18T04:39:42.940 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.940 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.940 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:42.940 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306346+0000", 2024-09-18T04:39:42.940 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6845000000000003e-05 2024-09-18T04:39:42.940 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:42.940 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:42.940 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:42.940 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.941 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.941 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:46 2.6 e51/17 2:7ba25245:::rbd_data.125fc09df6d1.0000000000000006:head v 51'2, mlcod=0'0)", 2024-09-18T04:39:42.941 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:39:42.941 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 60.586278786999998, 2024-09-18T04:39:42.941 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019843098, 2024-09-18T04:39:42.941 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:42.941 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.941 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:42.941 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.941 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:42.941 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:39:42.941 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:42.941 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.942 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.942 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:42.942 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:39:42.942 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:42.942 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.942 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.942 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:42.942 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313156+0000", 2024-09-18T04:39:42.942 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:42.942 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.942 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.942 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:42.942 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320144+0000", 2024-09-18T04:39:42.942 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.006988345 2024-09-18T04:39:42.942 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.943 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.943 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:42.943 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320175+0000", 2024-09-18T04:39:42.943 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.0861999999999997e-05 2024-09-18T04:39:42.943 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.943 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.943 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.943 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320188+0000", 2024-09-18T04:39:42.943 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2714999999999999e-05 2024-09-18T04:39:42.943 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.943 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.943 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:42.943 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320267+0000", 2024-09-18T04:39:42.943 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9586999999999995e-05 2024-09-18T04:39:42.944 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.944 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.944 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:42.944 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320311+0000", 2024-09-18T04:39:42.944 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3529000000000003e-05 2024-09-18T04:39:42.944 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.944 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.944 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:42.944 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.332949+0000", 2024-09-18T04:39:42.944 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012638267999999999 2024-09-18T04:39:42.944 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.944 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.944 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:42.945 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.333000+0000", 2024-09-18T04:39:42.945 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0736999999999999e-05 2024-09-18T04:39:42.945 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:42.945 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:42.945 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:42.945 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.945 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.945 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:57 2.0 2:0cdd070e:::rbd_data.125fc09df6d1.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:42.945 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:39:42.945 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 60.550421212000003, 2024-09-18T04:39:42.945 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.033329974999999998, 2024-09-18T04:39:42.945 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:42.945 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.945 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:42.945 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:42.946 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:42.946 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 57 2024-09-18T04:39:42.946 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.946 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:42.946 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.946 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:42.946 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:39:42.946 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:42.946 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.946 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.946 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:42.946 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:39:42.946 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:42.947 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.947 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.947 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:42.947 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349013+0000", 2024-09-18T04:39:42.947 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:42.947 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.947 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.947 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:42.947 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356138+0000", 2024-09-18T04:39:42.947 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071258010000000002 2024-09-18T04:39:42.947 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.947 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.947 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:42.947 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356168+0000", 2024-09-18T04:39:42.948 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9679000000000002e-05 2024-09-18T04:39:42.948 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.948 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.948 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.948 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356181+0000", 2024-09-18T04:39:42.948 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3164e-05 2024-09-18T04:39:42.948 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.948 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.948 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:42.948 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356254+0000", 2024-09-18T04:39:42.948 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3182999999999994e-05 2024-09-18T04:39:42.948 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.948 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.948 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:42.949 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356453+0000", 2024-09-18T04:39:42.949 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019876599999999999 2024-09-18T04:39:42.949 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.949 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.949 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:42.949 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356627+0000", 2024-09-18T04:39:42.949 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017340199999999999 2024-09-18T04:39:42.949 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.949 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.949 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:42.949 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.369200+0000", 2024-09-18T04:39:42.949 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012573302 2024-09-18T04:39:42.949 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.950 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.950 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:42.950 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382262+0000", 2024-09-18T04:39:42.950 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013062549 2024-09-18T04:39:42.950 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.950 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.950 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:42.950 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382310+0000", 2024-09-18T04:39:42.950 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7366e-05 2024-09-18T04:39:42.950 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.950 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.950 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:42.950 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382344+0000", 2024-09-18T04:39:42.950 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4319999999999997e-05 2024-09-18T04:39:42.951 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:42.951 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:42.951 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:42.951 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.951 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.951 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:59 2.0 2:19c52cde:::rbd_data.125fc09df6d1.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:42.951 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:39:42.951 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 60.543230047000002, 2024-09-18T04:39:42.951 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.027137247999999999, 2024-09-18T04:39:42.951 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:42.951 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.951 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:42.951 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:42.951 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:42.952 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 59 2024-09-18T04:39:42.952 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.952 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:42.952 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.952 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:42.952 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:39:42.952 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:42.952 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.952 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.952 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:42.952 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:39:42.952 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:42.952 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.952 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.953 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:42.953 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356204+0000", 2024-09-18T04:39:42.953 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:42.953 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.953 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.953 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:42.953 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362661+0000", 2024-09-18T04:39:42.953 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0064565129999999997 2024-09-18T04:39:42.953 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.953 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.953 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:42.953 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362684+0000", 2024-09-18T04:39:42.953 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3688000000000001e-05 2024-09-18T04:39:42.953 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.953 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.954 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.954 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362699+0000", 2024-09-18T04:39:42.954 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4798e-05 2024-09-18T04:39:42.954 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.954 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.954 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:42.954 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363016+0000", 2024-09-18T04:39:42.954 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031737300000000003 2024-09-18T04:39:42.954 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.954 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.954 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:42.954 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363201+0000", 2024-09-18T04:39:42.954 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000184131 2024-09-18T04:39:42.954 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.955 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.955 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:42.955 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363343+0000", 2024-09-18T04:39:42.955 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014236699999999999 2024-09-18T04:39:42.955 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.955 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.955 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:42.955 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.372189+0000", 2024-09-18T04:39:42.955 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0088464819999999993 2024-09-18T04:39:42.955 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.955 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.955 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:42.955 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383269+0000", 2024-09-18T04:39:42.955 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011079594 2024-09-18T04:39:42.956 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.956 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.956 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:42.956 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383314+0000", 2024-09-18T04:39:42.956 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4932000000000003e-05 2024-09-18T04:39:42.956 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.956 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.956 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:42.956 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383343+0000", 2024-09-18T04:39:42.956 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8600999999999999e-05 2024-09-18T04:39:42.956 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:42.956 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:42.956 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:42.956 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.956 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.957 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:40 2.6 e51/17 2:784d6cd1:ns::rbd_data.133e813ab5bc.0000000000000005:head v 51'7, mlcod=51'5)", 2024-09-18T04:39:42.957 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:39:42.957 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 60.346980313000003, 2024-09-18T04:39:42.957 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020497496, 2024-09-18T04:39:42.957 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:42.957 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.957 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:42.957 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.957 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:42.957 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:39:42.957 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:42.957 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.957 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.957 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:42.958 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:39:42.958 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:42.958 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.958 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.958 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:42.958 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552454+0000", 2024-09-18T04:39:42.958 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:42.958 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.958 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.958 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:42.958 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556448+0000", 2024-09-18T04:39:42.958 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0039940990000000001 2024-09-18T04:39:42.958 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.958 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.958 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:42.959 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556481+0000", 2024-09-18T04:39:42.959 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3778000000000001e-05 2024-09-18T04:39:42.959 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.959 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.959 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.959 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556493+0000", 2024-09-18T04:39:42.959 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1622e-05 2024-09-18T04:39:42.959 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.959 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.959 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:42.959 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556556+0000", 2024-09-18T04:39:42.959 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2582999999999994e-05 2024-09-18T04:39:42.959 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.959 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.960 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:42.960 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556592+0000", 2024-09-18T04:39:42.960 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5880999999999997e-05 2024-09-18T04:39:42.960 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.960 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.960 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:42.960 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.572786+0000", 2024-09-18T04:39:42.960 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016194571000000001 2024-09-18T04:39:42.960 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.960 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.960 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:42.960 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.572953+0000", 2024-09-18T04:39:42.960 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000166436 2024-09-18T04:39:42.960 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:42.961 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:42.961 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:42.961 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.961 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.961 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:45 2.0 2:0fbfa3dc:ns::rbd_data.133e813ab5bc.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:42.961 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:39:42.961 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 60.335766014000001, 2024-09-18T04:39:42.961 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.036558168000000002, 2024-09-18T04:39:42.961 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:42.961 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.961 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:42.961 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:42.961 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:42.961 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 45 2024-09-18T04:39:42.962 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.962 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:42.962 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.962 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:42.962 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:39:42.962 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:42.962 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.962 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.962 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:42.962 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:39:42.962 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:42.962 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.962 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.962 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:42.963 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:39:42.963 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:42.963 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.963 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.963 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:42.963 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578594+0000", 2024-09-18T04:39:42.963 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014927147999999999 2024-09-18T04:39:42.963 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.963 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.963 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:42.963 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578612+0000", 2024-09-18T04:39:42.963 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7623999999999999e-05 2024-09-18T04:39:42.963 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.963 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.964 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.964 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578620+0000", 2024-09-18T04:39:42.964 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9070000000000007e-06 2024-09-18T04:39:42.964 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.964 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.964 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:42.964 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578702+0000", 2024-09-18T04:39:42.964 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2349999999999996e-05 2024-09-18T04:39:42.964 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.964 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.964 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:42.964 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.579028+0000", 2024-09-18T04:39:42.965 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032622899999999998 2024-09-18T04:39:42.965 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.965 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.965 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:42.965 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.579343+0000", 2024-09-18T04:39:42.965 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031448899999999999 2024-09-18T04:39:42.965 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.965 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.965 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:42.965 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.590078+0000", 2024-09-18T04:39:42.965 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010734946 2024-09-18T04:39:42.966 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.966 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.966 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:42.966 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600089+0000", 2024-09-18T04:39:42.966 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010011195000000001 2024-09-18T04:39:42.966 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.966 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.966 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:42.966 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600177+0000", 2024-09-18T04:39:42.966 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8425999999999997e-05 2024-09-18T04:39:42.966 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.966 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.966 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:42.966 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600228+0000", 2024-09-18T04:39:42.967 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0216999999999997e-05 2024-09-18T04:39:42.967 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:42.967 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:42.967 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:42.967 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.967 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.967 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:54 2.0 2:04f17ef6:ns::rbd_data.133e813ab5bc.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:42.967 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:39:42.967 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 60.314194176999997, 2024-09-18T04:39:42.967 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.024612999, 2024-09-18T04:39:42.967 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:42.967 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.967 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:42.967 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:42.968 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:42.968 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 54 2024-09-18T04:39:42.968 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.968 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:42.968 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.968 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:42.968 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:39:42.968 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:42.968 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.968 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.968 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:42.968 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:39:42.968 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:42.968 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.969 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.969 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:42.969 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585239+0000", 2024-09-18T04:39:42.969 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:42.969 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.969 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.969 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:42.969 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593702+0000", 2024-09-18T04:39:42.969 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0084632990000000005 2024-09-18T04:39:42.969 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.969 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.969 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:42.969 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593719+0000", 2024-09-18T04:39:42.969 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.664e-05 2024-09-18T04:39:42.970 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.970 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.970 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.970 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593727+0000", 2024-09-18T04:39:42.970 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6839999999999996e-06 2024-09-18T04:39:42.970 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.970 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.970 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:42.970 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593767+0000", 2024-09-18T04:39:42.970 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0330999999999997e-05 2024-09-18T04:39:42.970 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.970 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.970 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:42.970 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593905+0000", 2024-09-18T04:39:42.970 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013754800000000001 2024-09-18T04:39:42.971 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.971 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.971 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:42.971 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.594013+0000", 2024-09-18T04:39:42.971 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000108805 2024-09-18T04:39:42.971 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.971 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.971 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:42.971 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604678+0000", 2024-09-18T04:39:42.971 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010664391 2024-09-18T04:39:42.971 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.971 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.971 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:42.971 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609662+0000", 2024-09-18T04:39:42.972 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0049846100000000004 2024-09-18T04:39:42.972 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.972 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.972 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:42.972 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609788+0000", 2024-09-18T04:39:42.972 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012518800000000001 2024-09-18T04:39:42.972 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.972 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.972 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:42.972 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609854+0000", 2024-09-18T04:39:42.972 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6556000000000004e-05 2024-09-18T04:39:42.972 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:42.972 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:42.972 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:42.973 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.973 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.973 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:58 2.7 2:fc7c6cd3:ns::rbd_data.133e813ab5bc.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:42.973 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:39:42.973 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 60.301351748000002, 2024-09-18T04:39:42.973 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.027690566999999999, 2024-09-18T04:39:42.973 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:42.973 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.973 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:42.973 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:42.973 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:42.973 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 58 2024-09-18T04:39:42.973 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.973 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:42.974 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.974 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:42.974 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:39:42.974 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:42.974 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.974 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.974 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:42.974 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:39:42.974 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:42.974 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.974 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.974 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:42.974 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598082+0000", 2024-09-18T04:39:42.974 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:42.975 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.975 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.975 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:42.975 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604832+0000", 2024-09-18T04:39:42.975 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0067504510000000002 2024-09-18T04:39:42.975 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.975 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.975 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:42.975 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604852+0000", 2024-09-18T04:39:42.975 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0228999999999999e-05 2024-09-18T04:39:42.975 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.975 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.975 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.975 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604870+0000", 2024-09-18T04:39:42.975 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7952e-05 2024-09-18T04:39:42.976 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.976 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.976 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:42.976 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604936+0000", 2024-09-18T04:39:42.976 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5431000000000004e-05 2024-09-18T04:39:42.976 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.976 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.976 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:42.976 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605085+0000", 2024-09-18T04:39:42.976 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014910300000000001 2024-09-18T04:39:42.976 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.976 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.976 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:39:42.977 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605205+0000", 2024-09-18T04:39:42.977 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000119887 2024-09-18T04:39:42.977 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.977 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.977 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:42.977 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625586+0000", 2024-09-18T04:39:42.977 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020381399000000001 2024-09-18T04:39:42.977 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.977 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.977 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:42.977 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625628+0000", 2024-09-18T04:39:42.977 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1873999999999999e-05 2024-09-18T04:39:42.977 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.977 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.977 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:42.978 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625722+0000", 2024-09-18T04:39:42.978 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3787000000000002e-05 2024-09-18T04:39:42.978 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.978 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.978 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:42.978 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625774+0000", 2024-09-18T04:39:42.978 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2503999999999997e-05 2024-09-18T04:39:42.978 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:42.978 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:42.978 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:42.978 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.978 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.978 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:56 2.6 e51/17 2:72453c3e:ns::rbd_data.133e813ab5bc.000000000000000d:head v 51'10, mlcod=51'8)", 2024-09-18T04:39:42.978 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:39:42.979 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 60.294154640000002, 2024-09-18T04:39:42.979 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019766552999999999, 2024-09-18T04:39:42.979 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:42.979 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.979 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:42.979 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.979 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:42.979 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:39:42.979 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:42.979 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.979 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.979 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:42.979 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:39:42.979 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:42.980 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.980 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.980 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:42.980 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605279+0000", 2024-09-18T04:39:42.980 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:42.980 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.980 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.980 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:42.980 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609417+0000", 2024-09-18T04:39:42.980 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0041378059999999999 2024-09-18T04:39:42.980 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.980 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.980 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:42.980 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609443+0000", 2024-09-18T04:39:42.981 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6098e-05 2024-09-18T04:39:42.981 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.981 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.981 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.981 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609458+0000", 2024-09-18T04:39:42.981 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4671000000000001e-05 2024-09-18T04:39:42.981 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.981 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.981 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:42.981 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609553+0000", 2024-09-18T04:39:42.981 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5265000000000004e-05 2024-09-18T04:39:42.981 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.981 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.981 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:42.981 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609583+0000", 2024-09-18T04:39:42.982 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9839e-05 2024-09-18T04:39:42.982 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.982 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.982 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:42.982 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625004+0000", 2024-09-18T04:39:42.982 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015420639999999999 2024-09-18T04:39:42.982 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.982 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.982 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:42.982 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625047+0000", 2024-09-18T04:39:42.982 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3665999999999998e-05 2024-09-18T04:39:42.982 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:42.982 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:42.982 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:42.983 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.983 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.983 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:99 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:42.983 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.582718+0000", 2024-09-18T04:39:42.983 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 46.316717730999997, 2024-09-18T04:39:42.983 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019362575, 2024-09-18T04:39:42.983 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:42.983 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.983 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:42.983 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:42.984 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:42.984 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 99 2024-09-18T04:39:42.984 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.984 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:42.984 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.984 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:42.984 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582718+0000", 2024-09-18T04:39:42.984 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:42.984 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.984 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.984 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:42.984 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582718+0000", 2024-09-18T04:39:42.984 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:42.985 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.985 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.985 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:42.985 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582717+0000", 2024-09-18T04:39:42.985 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:42.985 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.985 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.985 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:42.985 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582720+0000", 2024-09-18T04:39:42.985 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2239999999999998e-06 2024-09-18T04:39:42.985 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.985 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.985 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:42.985 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582723+0000", 2024-09-18T04:39:42.986 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.258e-06 2024-09-18T04:39:42.986 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.986 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.986 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.986 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582726+0000", 2024-09-18T04:39:42.986 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8870000000000001e-06 2024-09-18T04:39:42.986 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.986 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.986 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:42.986 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.592018+0000", 2024-09-18T04:39:42.986 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0092926420000000003 2024-09-18T04:39:42.986 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.986 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.986 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:42.986 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.592214+0000", 2024-09-18T04:39:42.987 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000195267 2024-09-18T04:39:42.987 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.987 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.987 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:42.987 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.592913+0000", 2024-09-18T04:39:42.987 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00069924600000000005 2024-09-18T04:39:42.987 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.987 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.987 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:42.987 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597493+0000", 2024-09-18T04:39:42.987 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045797629999999997 2024-09-18T04:39:42.987 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.987 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.988 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:42.988 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.602009+0000", 2024-09-18T04:39:42.988 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045159559999999998 2024-09-18T04:39:42.988 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.988 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.988 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:42.988 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.602042+0000", 2024-09-18T04:39:42.988 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3084999999999999e-05 2024-09-18T04:39:42.988 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.988 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.988 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:42.988 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.602080+0000", 2024-09-18T04:39:42.988 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8600000000000003e-05 2024-09-18T04:39:42.988 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:42.989 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:42.989 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:42.989 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.989 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.989 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:104 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:42.989 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.583165+0000", 2024-09-18T04:39:42.989 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 46.316270433, 2024-09-18T04:39:42.989 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023679603, 2024-09-18T04:39:42.989 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:42.989 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.989 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:42.989 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:42.989 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:42.989 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 104 2024-09-18T04:39:42.990 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.990 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:42.990 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.990 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:42.990 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583165+0000", 2024-09-18T04:39:42.990 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:42.990 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.990 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.990 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:42.990 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583165+0000", 2024-09-18T04:39:42.990 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:42.990 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.990 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.990 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:42.991 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583164+0000", 2024-09-18T04:39:42.991 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:42.991 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.991 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.991 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:42.991 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583167+0000", 2024-09-18T04:39:42.991 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9210000000000001e-06 2024-09-18T04:39:42.991 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.991 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.991 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:42.991 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583171+0000", 2024-09-18T04:39:42.992 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9759999999999997e-06 2024-09-18T04:39:42.992 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.992 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.992 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.992 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583174+0000", 2024-09-18T04:39:42.992 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9960000000000002e-06 2024-09-18T04:39:42.992 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.992 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.992 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:42.992 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.594373+0000", 2024-09-18T04:39:42.992 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011198665999999999 2024-09-18T04:39:42.992 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.992 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.993 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:42.993 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.594434+0000", 2024-09-18T04:39:42.993 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.1446999999999996e-05 2024-09-18T04:39:42.993 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.993 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.993 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:42.993 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.594692+0000", 2024-09-18T04:39:42.993 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00025756599999999998 2024-09-18T04:39:42.993 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.993 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.993 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:42.993 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599302+0000", 2024-09-18T04:39:42.993 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0046099640000000002 2024-09-18T04:39:42.993 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.994 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.994 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:42.994 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606710+0000", 2024-09-18T04:39:42.994 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0074077739999999998 2024-09-18T04:39:42.994 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.994 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.994 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:42.994 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606798+0000", 2024-09-18T04:39:42.994 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8133999999999993e-05 2024-09-18T04:39:42.994 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.994 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.994 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:42.994 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606845+0000", 2024-09-18T04:39:42.994 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6653000000000002e-05 2024-09-18T04:39:42.994 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:42.995 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:42.995 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:42.995 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.995 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.995 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:105 2.0 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:42.995 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.583193+0000", 2024-09-18T04:39:42.995 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 46.316242813999999, 2024-09-18T04:39:42.995 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023713649, 2024-09-18T04:39:42.995 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:42.995 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:42.995 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:42.995 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:42.995 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:42.995 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 105 2024-09-18T04:39:42.996 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.996 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:42.996 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.996 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:42.996 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583193+0000", 2024-09-18T04:39:42.996 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:42.996 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.996 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.996 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:42.996 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583193+0000", 2024-09-18T04:39:42.996 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:42.996 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.996 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.996 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:42.996 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583192+0000", 2024-09-18T04:39:42.997 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:42.997 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.997 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.997 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:42.997 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583194+0000", 2024-09-18T04:39:42.997 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.305e-06 2024-09-18T04:39:42.997 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.997 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.997 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:42.997 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583198+0000", 2024-09-18T04:39:42.997 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9369999999999997e-06 2024-09-18T04:39:42.997 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.997 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.997 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:42.998 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583202+0000", 2024-09-18T04:39:42.998 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6480000000000001e-06 2024-09-18T04:39:42.998 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.998 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.998 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:42.998 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.595304+0000", 2024-09-18T04:39:42.998 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012102063999999999 2024-09-18T04:39:42.998 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.998 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.998 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:42.998 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.595390+0000", 2024-09-18T04:39:42.998 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5493000000000003e-05 2024-09-18T04:39:42.998 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.998 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.999 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:42.999 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.595509+0000", 2024-09-18T04:39:42.999 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011966199999999999 2024-09-18T04:39:42.999 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.999 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.999 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:42.999 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599358+0000", 2024-09-18T04:39:42.999 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0038486689999999999 2024-09-18T04:39:42.999 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.999 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:42.999 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:42.999 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606859+0000", 2024-09-18T04:39:42.999 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0075010789999999999 2024-09-18T04:39:42.999 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:42.999 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:43.000 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:43.000 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606887+0000", 2024-09-18T04:39:43.000 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7773e-05 2024-09-18T04:39:43.000 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:43.000 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:43.000 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:43.000 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606906+0000", 2024-09-18T04:39:43.000 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9417000000000001e-05 2024-09-18T04:39:43.000 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:43.000 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:43.000 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:43.000 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:43.000 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:43.000 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:106 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:43.001 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.584425+0000", 2024-09-18T04:39:43.001 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 46.315010518000001, 2024-09-18T04:39:43.001 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.022529293999999998, 2024-09-18T04:39:43.001 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:43.001 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:43.001 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:43.001 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:43.001 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:43.001 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 106 2024-09-18T04:39:43.001 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:43.001 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:43.001 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:43.001 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:43.001 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584425+0000", 2024-09-18T04:39:43.002 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:43.002 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:43.002 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:43.002 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:43.002 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584425+0000", 2024-09-18T04:39:43.002 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:43.002 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:43.002 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:43.002 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:43.002 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584424+0000", 2024-09-18T04:39:43.002 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:43.002 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:43.002 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:43.002 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:43.002 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584430+0000", 2024-09-18T04:39:43.003 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6389999999999997e-06 2024-09-18T04:39:43.003 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:43.003 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:43.003 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:43.003 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584456+0000", 2024-09-18T04:39:43.003 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6188999999999999e-05 2024-09-18T04:39:43.003 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:43.003 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:43.003 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:43.003 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584463+0000", 2024-09-18T04:39:43.003 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7340000000000002e-06 2024-09-18T04:39:43.003 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:43.003 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:43.003 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:43.003 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597537+0000", 2024-09-18T04:39:43.004 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013074161000000001 2024-09-18T04:39:43.004 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:43.004 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:43.004 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:43.004 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597598+0000", 2024-09-18T04:39:43.004 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0930000000000001e-05 2024-09-18T04:39:43.004 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:43.004 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:43.004 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:43.004 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597922+0000", 2024-09-18T04:39:43.004 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032473000000000002 2024-09-18T04:39:43.004 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:43.004 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:43.004 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:43.005 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.601103+0000", 2024-09-18T04:39:43.005 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0031809849999999999 2024-09-18T04:39:43.005 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:43.005 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:43.005 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:43.005 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606912+0000", 2024-09-18T04:39:43.005 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058083550000000003 2024-09-18T04:39:43.005 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:43.005 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:43.005 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:43.005 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606935+0000", 2024-09-18T04:39:43.005 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2759999999999999e-05 2024-09-18T04:39:43.005 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:43.005 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:43.006 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:43.006 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606954+0000", 2024-09-18T04:39:43.006 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9602999999999998e-05 2024-09-18T04:39:43.006 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:43.006 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:43.006 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:43.006 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:43.006 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:43.006 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:109 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:43.006 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.585231+0000", 2024-09-18T04:39:43.006 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 46.314203898000002, 2024-09-18T04:39:43.006 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.021767496000000001, 2024-09-18T04:39:43.006 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:43.006 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:43.007 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:43.007 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:43.007 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:43.007 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 109 2024-09-18T04:39:43.007 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:43.007 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:43.007 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:43.007 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:43.007 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585231+0000", 2024-09-18T04:39:43.007 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:43.007 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:43.007 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:43.007 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:43.007 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585231+0000", 2024-09-18T04:39:43.007 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:43.008 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:43.008 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:43.008 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:43.008 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585230+0000", 2024-09-18T04:39:43.008 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:43.008 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:43.008 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:43.008 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:43.008 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585237+0000", 2024-09-18T04:39:43.008 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.618e-06 2024-09-18T04:39:43.008 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:43.008 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:43.008 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:43.008 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585256+0000", 2024-09-18T04:39:43.009 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9048000000000001e-05 2024-09-18T04:39:43.009 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:43.009 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:43.009 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:43.009 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585274+0000", 2024-09-18T04:39:43.009 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8153000000000001e-05 2024-09-18T04:39:43.009 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:43.009 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:43.009 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:43.009 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.598284+0000", 2024-09-18T04:39:43.009 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013009682999999999 2024-09-18T04:39:43.009 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:43.009 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:43.009 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:43.010 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.598383+0000", 2024-09-18T04:39:43.010 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.8722999999999997e-05 2024-09-18T04:39:43.010 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:43.010 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:43.010 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:43.010 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.598829+0000", 2024-09-18T04:39:43.010 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00044623999999999998 2024-09-18T04:39:43.010 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:43.010 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:43.010 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:43.010 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.601149+0000", 2024-09-18T04:39:43.010 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0023204150000000002 2024-09-18T04:39:43.010 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:43.010 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:43.011 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:43.011 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606960+0000", 2024-09-18T04:39:43.011 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058106390000000003 2024-09-18T04:39:43.011 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:43.011 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:43.011 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:43.011 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606982+0000", 2024-09-18T04:39:43.011 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2252e-05 2024-09-18T04:39:43.011 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:43.011 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:43.011 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:43.011 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606999+0000", 2024-09-18T04:39:43.011 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6871000000000002e-05 2024-09-18T04:39:43.011 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:43.011 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:43.012 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:43.012 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:43.012 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:43.012 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:110 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:43.012 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.586640+0000", 2024-09-18T04:39:43.012 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 46.312795579000003, 2024-09-18T04:39:43.012 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020400911000000001, 2024-09-18T04:39:43.012 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:43.012 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:43.012 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:43.012 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:43.012 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:43.012 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 110 2024-09-18T04:39:43.012 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:43.013 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:43.013 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:43.013 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:43.013 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586640+0000", 2024-09-18T04:39:43.013 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:43.013 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:43.013 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:43.013 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:43.013 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586640+0000", 2024-09-18T04:39:43.013 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:43.013 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:43.013 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:43.013 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:43.013 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586639+0000", 2024-09-18T04:39:43.013 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:43.014 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:43.014 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:43.014 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:43.014 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586652+0000", 2024-09-18T04:39:43.014 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2482e-05 2024-09-18T04:39:43.014 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:43.014 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:43.014 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:43.014 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586660+0000", 2024-09-18T04:39:43.014 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1659999999999994e-06 2024-09-18T04:39:43.014 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:43.014 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:43.014 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:43.014 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586666+0000", 2024-09-18T04:39:43.015 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6900000000000003e-06 2024-09-18T04:39:43.015 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:43.015 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:43.015 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:43.015 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599381+0000", 2024-09-18T04:39:43.015 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012714328 2024-09-18T04:39:43.015 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:43.015 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:43.015 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:43.015 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599436+0000", 2024-09-18T04:39:43.015 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4802999999999997e-05 2024-09-18T04:39:43.015 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:43.015 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:43.015 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:43.016 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599629+0000", 2024-09-18T04:39:43.016 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000193921 2024-09-18T04:39:43.016 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:43.016 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:43.016 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:43.016 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.601174+0000", 2024-09-18T04:39:43.016 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001544857 2024-09-18T04:39:43.016 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:43.016 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:43.016 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:43.016 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.607004+0000", 2024-09-18T04:39:43.016 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058300820000000003 2024-09-18T04:39:43.016 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:43.016 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:43.017 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:43.017 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.607024+0000", 2024-09-18T04:39:43.017 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9888e-05 2024-09-18T04:39:43.017 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:43.017 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:43.017 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:43.017 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.607041+0000", 2024-09-18T04:39:43.017 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6450999999999999e-05 2024-09-18T04:39:43.017 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:43.017 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:43.017 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:43.017 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:43.017 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:43.017 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:43.018 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T04:39:43.020 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:39:43.020 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:43.012+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:43.120 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:43.112+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:43.134 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:43.134 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:43.134 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:39:43.134 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:43.144 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:39:43.164 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:39:43.183 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:39:43.211 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:39:43.221 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:43.212+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:43.235 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:39:43.252 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:43.253 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:43.258 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:39:43.258 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:39:43.259 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:39:43.259 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-18T04:39:43.259 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:39:43.259 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:39:43.259 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:39:43.262 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:39:43.262 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T04:39:43.263 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T04:39:43.322 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:43.316+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:43.368 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:43.369 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:43.378 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:39:43.423 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:43.416+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:43.524 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:43.520+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:43.533 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:43.533 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:43.543 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:39:43.624 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:43.616+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:43.647 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:43.647 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:43.657 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:39:43.726 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:43.716+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:43.762 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:43.762 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:43.772 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:39:43.827 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:43.824+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:43.889 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:43.889 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:43.899 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:39:43.943 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:43.940+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:44.029 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:44.020+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:44.057 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:44.058 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:44.068 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:39:44.129 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:44.120+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:44.177 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:44.177 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:44.190 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:39:44.234 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:44.232+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:44.252 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:44.248+0000 7fb3e7610880 -1 osd.2 76 log_to_monitors true 2024-09-18T04:39:44.293 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:44.293 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:44.303 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:39:44.331 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:44.324+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:44.407 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:44.407 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:44.417 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:39:44.431 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:44.428+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:44.525 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:44.526 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:44.532 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:44.524+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:44.535 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:39:44.633 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:44.628+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:44.651 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:44.651 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:44.662 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:39:44.715 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-18T04:39:44.734 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:44.728+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:44.742 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:44.740+0000 7fb3daefe700 -1 osd.2 85 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T04:39:44.764 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:44.765 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:44.774 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:39:44.834 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:44.832+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:44.880 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:44.881 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:44.890 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:39:44.936 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:44.928+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:44.998 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:44.999 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:45.008 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:39:45.037 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:45.028+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:45.119 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:45.120 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:45.129 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:39:45.138 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:45.136+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:45.170 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:39:45.238 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:45.232+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:45.264 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T04:39:45.283 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:45.284 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:45.290 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:45.290 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:45.290 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:39:45.290 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:45.294 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:39:45.342 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:45.336+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:45.403 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:39:45.406 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:45.406 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:45.417 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:39:45.429 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:39:45.440 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:45.432+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:45.448 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:39:45.469 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:39:45.491 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:39:45.492 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:39:45.492 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:39:45.492 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-18T04:39:45.492 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:39:45.492 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:39:45.492 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:39:45.496 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T04:39:45.497 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T04:39:45.522 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:45.522 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:45.522 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:39:45.522 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:39:45.522 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:45.532 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:39:45.541 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:45.536+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:39:45.642 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:45.636+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:45.687 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:45.687 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:39:45.687 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:39:45.687 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:39:45.687 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.687 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-18T04:39:45.687 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:39:45.687 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 131.211341226, 2024-09-18T04:39:45.687 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0095781189999999995, 2024-09-18T04:39:45.687 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:45.687 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:45.687 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:45.688 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:39:45.688 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:39:45.688 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 1 2024-09-18T04:39:45.688 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.688 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:45.688 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.688 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:45.688 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:39:45.688 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:45.688 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.688 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.688 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:45.688 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:39:45.689 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:45.689 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.689 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.689 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:45.689 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469862+0000", 2024-09-18T04:39:45.689 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:45.689 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.689 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.689 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:45.689 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469879+0000", 2024-09-18T04:39:45.689 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7431999999999998e-05 2024-09-18T04:39:45.689 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.689 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.690 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:45.690 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469896+0000", 2024-09-18T04:39:45.690 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6742999999999999e-05 2024-09-18T04:39:45.690 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.690 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.690 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:45.690 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469915+0000", 2024-09-18T04:39:45.690 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8991000000000002e-05 2024-09-18T04:39:45.690 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.690 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.690 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:45.690 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470397+0000", 2024-09-18T04:39:45.690 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048167799999999998 2024-09-18T04:39:45.690 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.691 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.691 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-09-18T04:39:45.691 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470412+0000", 2024-09-18T04:39:45.691 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5526e-05 2024-09-18T04:39:45.691 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.691 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.691 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:45.691 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473279+0000", 2024-09-18T04:39:45.691 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0028667469999999998 2024-09-18T04:39:45.691 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.691 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.691 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:45.691 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473441+0000", 2024-09-18T04:39:45.691 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000162082 2024-09-18T04:39:45.692 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.692 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.692 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:45.692 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473573+0000", 2024-09-18T04:39:45.692 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000132032 2024-09-18T04:39:45.692 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.692 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.692 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:45.692 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474826+0000", 2024-09-18T04:39:45.692 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001252717 2024-09-18T04:39:45.692 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.692 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.692 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:45.693 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479307+0000", 2024-09-18T04:39:45.693 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044813220000000003 2024-09-18T04:39:45.693 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.693 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.693 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:45.693 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479366+0000", 2024-09-18T04:39:45.693 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9024999999999998e-05 2024-09-18T04:39:45.693 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.693 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.693 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:45.693 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479443+0000", 2024-09-18T04:39:45.693 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6339000000000002e-05 2024-09-18T04:39:45.693 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:45.693 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:45.694 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:45.694 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.694 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.694 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-18T04:39:45.694 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:39:45.694 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 129.198332806, 2024-09-18T04:39:45.694 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082699560000000002, 2024-09-18T04:39:45.694 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:45.694 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:45.694 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:45.694 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:39:45.694 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:39:45.694 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 7 2024-09-18T04:39:45.695 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.695 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:45.695 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.695 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:45.695 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:39:45.695 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:45.695 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.695 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.695 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:45.695 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:39:45.695 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:45.695 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.695 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.695 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:45.696 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482870+0000", 2024-09-18T04:39:45.696 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999967 2024-09-18T04:39:45.696 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.696 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.696 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:45.696 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482889+0000", 2024-09-18T04:39:45.696 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.965e-05 2024-09-18T04:39:45.696 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.696 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.696 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:45.696 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482938+0000", 2024-09-18T04:39:45.696 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8548999999999999e-05 2024-09-18T04:39:45.696 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.696 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.697 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:45.697 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482963+0000", 2024-09-18T04:39:45.697 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5075999999999998e-05 2024-09-18T04:39:45.697 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.697 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.697 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:45.697 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483121+0000", 2024-09-18T04:39:45.697 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015819400000000001 2024-09-18T04:39:45.697 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.697 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.697 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op must wait for map", 2024-09-18T04:39:45.697 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483138+0000", 2024-09-18T04:39:45.697 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7068999999999999e-05 2024-09-18T04:39:45.697 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.698 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.698 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:45.698 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488666+0000", 2024-09-18T04:39:45.698 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0055282049999999996 2024-09-18T04:39:45.698 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.698 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.698 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:45.698 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488783+0000", 2024-09-18T04:39:45.698 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000116737 2024-09-18T04:39:45.698 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.698 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.698 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:45.698 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488991+0000", 2024-09-18T04:39:45.698 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020781400000000001 2024-09-18T04:39:45.699 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.699 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.699 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:45.699 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490554+0000", 2024-09-18T04:39:45.699 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0015628370000000001 2024-09-18T04:39:45.699 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.699 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.699 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:45.699 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491019+0000", 2024-09-18T04:39:45.699 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000465608 2024-09-18T04:39:45.699 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.699 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.699 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:45.699 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491087+0000", 2024-09-18T04:39:45.700 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7988e-05 2024-09-18T04:39:45.700 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.700 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.700 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:45.700 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491143+0000", 2024-09-18T04:39:45.700 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5538999999999998e-05 2024-09-18T04:39:45.700 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:45.700 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:45.700 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:45.700 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.700 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.700 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:34 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:45.700 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.261758+0000", 2024-09-18T04:39:45.700 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 63.419447654999999, 2024-09-18T04:39:45.701 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.042024191000000002, 2024-09-18T04:39:45.701 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:45.701 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:45.701 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:45.701 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:45.701 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:45.701 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 34 2024-09-18T04:39:45.701 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.701 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:45.701 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.701 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:45.701 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.261758+0000", 2024-09-18T04:39:45.701 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:45.702 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.702 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.702 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:45.702 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.261758+0000", 2024-09-18T04:39:45.702 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:45.702 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.702 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.702 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:45.702 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.261756+0000", 2024-09-18T04:39:45.702 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:45.702 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.702 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.702 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:45.703 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.270785+0000", 2024-09-18T04:39:45.703 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0090292340000000006 2024-09-18T04:39:45.703 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.703 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.703 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:45.703 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.270890+0000", 2024-09-18T04:39:45.703 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000104541 2024-09-18T04:39:45.703 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.703 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.703 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:45.703 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.270991+0000", 2024-09-18T04:39:45.703 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000100778 2024-09-18T04:39:45.703 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.703 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.704 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:45.704 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.271157+0000", 2024-09-18T04:39:45.704 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016653999999999999 2024-09-18T04:39:45.704 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.704 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.704 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:45.704 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.271591+0000", 2024-09-18T04:39:45.704 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00043409500000000001 2024-09-18T04:39:45.704 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.704 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.704 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:45.704 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272017+0000", 2024-09-18T04:39:45.704 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000425769 2024-09-18T04:39:45.705 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.705 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.705 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:45.705 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.294246+0000", 2024-09-18T04:39:45.705 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.022229237999999998 2024-09-18T04:39:45.705 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.705 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.705 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:45.705 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303571+0000", 2024-09-18T04:39:45.705 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0093250239999999995 2024-09-18T04:39:45.705 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.705 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.705 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:45.706 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303705+0000", 2024-09-18T04:39:45.706 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000133831 2024-09-18T04:39:45.706 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.706 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.706 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:45.706 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303782+0000", 2024-09-18T04:39:45.706 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7156000000000005e-05 2024-09-18T04:39:45.706 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:45.706 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:45.706 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:45.706 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.706 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.706 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:36 2.7 e51/32 2:fe1b02ad:::rbd_data.125fc09df6d1.0000000000000002:head v 51'1, mlcod=0'0)", 2024-09-18T04:39:45.706 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.273441+0000", 2024-09-18T04:39:45.707 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 63.407764673999999, 2024-09-18T04:39:45.707 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.032400768000000003, 2024-09-18T04:39:45.707 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:45.707 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:45.707 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:45.707 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.707 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:45.707 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273441+0000", 2024-09-18T04:39:45.707 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:45.707 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.707 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.707 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:45.707 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273441+0000", 2024-09-18T04:39:45.707 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:45.708 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.708 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.708 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:45.708 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273440+0000", 2024-09-18T04:39:45.708 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:45.708 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.708 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.708 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:45.708 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278723+0000", 2024-09-18T04:39:45.708 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0052829909999999999 2024-09-18T04:39:45.708 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.708 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.708 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:45.709 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278752+0000", 2024-09-18T04:39:45.709 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9178000000000001e-05 2024-09-18T04:39:45.709 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.709 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.709 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:45.709 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278761+0000", 2024-09-18T04:39:45.709 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7900000000000005e-06 2024-09-18T04:39:45.709 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.709 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.709 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:45.709 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278864+0000", 2024-09-18T04:39:45.709 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000102699 2024-09-18T04:39:45.709 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.710 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.710 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:45.710 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278900+0000", 2024-09-18T04:39:45.710 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6001999999999999e-05 2024-09-18T04:39:45.710 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.710 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.710 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:45.710 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.305793+0000", 2024-09-18T04:39:45.710 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.026893344 2024-09-18T04:39:45.710 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.710 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.710 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:45.710 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.305842+0000", 2024-09-18T04:39:45.710 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8720000000000001e-05 2024-09-18T04:39:45.710 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:45.711 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:45.711 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:45.711 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.711 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.711 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:39 2.7 e51/32 2:e9c909c5:::rbd_data.125fc09df6d1.0000000000000005:head v 51'2, mlcod=0'0)", 2024-09-18T04:39:45.711 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.283175+0000", 2024-09-18T04:39:45.711 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 63.398030984999998, 2024-09-18T04:39:45.711 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.022864102000000001, 2024-09-18T04:39:45.711 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:45.711 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:45.711 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:45.711 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.711 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:45.712 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283175+0000", 2024-09-18T04:39:45.712 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:45.712 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.712 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.712 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:45.712 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283175+0000", 2024-09-18T04:39:45.712 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:45.712 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.712 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.712 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:45.712 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283173+0000", 2024-09-18T04:39:45.712 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:45.712 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.712 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.713 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:45.713 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.285951+0000", 2024-09-18T04:39:45.713 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0027784540000000001 2024-09-18T04:39:45.713 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.713 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.713 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:45.713 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.285970+0000", 2024-09-18T04:39:45.713 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8855000000000002e-05 2024-09-18T04:39:45.713 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.713 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.713 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:45.713 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.285984+0000", 2024-09-18T04:39:45.713 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4252e-05 2024-09-18T04:39:45.714 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.714 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.714 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:45.714 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.286041+0000", 2024-09-18T04:39:45.714 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6431999999999998e-05 2024-09-18T04:39:45.714 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.714 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.714 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:45.714 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.286086+0000", 2024-09-18T04:39:45.714 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5423000000000002e-05 2024-09-18T04:39:45.714 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.714 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.714 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:45.714 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.305998+0000", 2024-09-18T04:39:45.715 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019911697999999999 2024-09-18T04:39:45.715 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.715 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.715 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:45.715 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306039+0000", 2024-09-18T04:39:45.715 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0973999999999998e-05 2024-09-18T04:39:45.715 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:45.715 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:45.715 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:45.715 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.715 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.715 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:50 2.3 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:45.715 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.312563+0000", 2024-09-18T04:39:45.715 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 63.368642958000002, 2024-09-18T04:39:45.716 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.032636001999999997, 2024-09-18T04:39:45.716 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:45.716 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:45.716 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:45.716 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:45.716 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:45.716 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 50 2024-09-18T04:39:45.716 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.716 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:45.716 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.716 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:45.716 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.312563+0000", 2024-09-18T04:39:45.716 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:45.717 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.717 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.717 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:45.717 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.312563+0000", 2024-09-18T04:39:45.717 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:45.717 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.717 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.717 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:45.717 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.312561+0000", 2024-09-18T04:39:45.717 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:45.717 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.717 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.717 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:45.717 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321709+0000", 2024-09-18T04:39:45.718 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0091486650000000003 2024-09-18T04:39:45.718 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.718 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.718 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:45.718 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321739+0000", 2024-09-18T04:39:45.718 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9666e-05 2024-09-18T04:39:45.718 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.718 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.718 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:45.718 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321751+0000", 2024-09-18T04:39:45.718 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2086000000000001e-05 2024-09-18T04:39:45.718 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.718 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.718 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:45.719 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321833+0000", 2024-09-18T04:39:45.719 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1842999999999996e-05 2024-09-18T04:39:45.719 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.719 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.719 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:45.719 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321981+0000", 2024-09-18T04:39:45.719 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000147441 2024-09-18T04:39:45.719 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.719 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.719 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:45.719 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.322109+0000", 2024-09-18T04:39:45.719 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012851200000000001 2024-09-18T04:39:45.719 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.720 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.720 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:45.720 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.332829+0000", 2024-09-18T04:39:45.720 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010720225999999999 2024-09-18T04:39:45.720 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.720 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.720 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:45.720 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.345039+0000", 2024-09-18T04:39:45.720 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012209831000000001 2024-09-18T04:39:45.720 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.720 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.720 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:45.720 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.345120+0000", 2024-09-18T04:39:45.720 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1145999999999999e-05 2024-09-18T04:39:45.721 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.721 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.721 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:45.721 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.345199+0000", 2024-09-18T04:39:45.721 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8546000000000003e-05 2024-09-18T04:39:45.721 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:45.721 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:45.721 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:45.721 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.721 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.721 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:30 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:45.721 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.516122+0000", 2024-09-18T04:39:45.721 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 63.165083795999998, 2024-09-18T04:39:45.721 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0091865030000000004, 2024-09-18T04:39:45.722 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:45.722 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:45.722 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:45.722 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:45.722 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:45.722 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 30 2024-09-18T04:39:45.722 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.722 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:45.722 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.722 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:45.722 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516122+0000", 2024-09-18T04:39:45.722 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:45.722 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.722 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.723 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:45.723 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516122+0000", 2024-09-18T04:39:45.723 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:45.723 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.723 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.723 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:45.723 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516120+0000", 2024-09-18T04:39:45.723 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:45.723 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.723 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.723 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:45.723 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516129+0000", 2024-09-18T04:39:45.723 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.85e-06 2024-09-18T04:39:45.723 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.724 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.724 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:45.724 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516155+0000", 2024-09-18T04:39:45.724 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5882000000000001e-05 2024-09-18T04:39:45.724 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.724 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.724 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:45.724 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516175+0000", 2024-09-18T04:39:45.724 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.022e-05 2024-09-18T04:39:45.724 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.724 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.724 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:45.724 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519780+0000", 2024-09-18T04:39:45.724 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003604187 2024-09-18T04:39:45.725 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.725 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.725 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:45.725 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519964+0000", 2024-09-18T04:39:45.725 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00018391999999999999 2024-09-18T04:39:45.725 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.725 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.725 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:45.725 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.520794+0000", 2024-09-18T04:39:45.725 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00083078100000000005 2024-09-18T04:39:45.725 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.725 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.725 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:45.725 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.524129+0000", 2024-09-18T04:39:45.726 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033342419999999998 2024-09-18T04:39:45.726 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.726 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.726 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:45.726 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525239+0000", 2024-09-18T04:39:45.726 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001110778 2024-09-18T04:39:45.726 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.726 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.726 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:45.726 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525277+0000", 2024-09-18T04:39:45.726 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7596999999999999e-05 2024-09-18T04:39:45.726 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.726 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.727 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:45.727 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525308+0000", 2024-09-18T04:39:45.727 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.1535999999999998e-05 2024-09-18T04:39:45.727 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:45.727 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:45.727 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:45.727 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.727 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.727 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:32 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:45.727 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.519217+0000", 2024-09-18T04:39:45.727 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 63.161988475999998, 2024-09-18T04:39:45.727 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082266370000000002, 2024-09-18T04:39:45.727 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:45.727 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:45.727 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:45.728 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:45.728 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:45.728 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 32 2024-09-18T04:39:45.728 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.728 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:45.728 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.728 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:45.728 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519217+0000", 2024-09-18T04:39:45.728 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:45.728 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.728 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.728 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:45.728 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519217+0000", 2024-09-18T04:39:45.729 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:45.729 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.729 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.729 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:45.729 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519215+0000", 2024-09-18T04:39:45.729 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:45.729 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.729 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.729 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:45.729 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519229+0000", 2024-09-18T04:39:45.729 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3329999999999999e-05 2024-09-18T04:39:45.729 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.729 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.730 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:45.730 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519268+0000", 2024-09-18T04:39:45.730 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9533000000000002e-05 2024-09-18T04:39:45.730 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.730 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.730 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:45.730 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519285+0000", 2024-09-18T04:39:45.730 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6446000000000001e-05 2024-09-18T04:39:45.730 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.730 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.730 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:45.730 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.522358+0000", 2024-09-18T04:39:45.730 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030734310000000002 2024-09-18T04:39:45.730 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.730 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.731 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:45.731 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.522551+0000", 2024-09-18T04:39:45.731 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019317600000000001 2024-09-18T04:39:45.731 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.731 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.731 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:45.731 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.523262+0000", 2024-09-18T04:39:45.731 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00071069999999999998 2024-09-18T04:39:45.731 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.731 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.731 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:45.731 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525318+0000", 2024-09-18T04:39:45.731 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0020561469999999999 2024-09-18T04:39:45.732 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.732 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.732 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:45.732 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.527376+0000", 2024-09-18T04:39:45.732 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0020584969999999998 2024-09-18T04:39:45.732 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.732 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.732 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:45.732 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.527416+0000", 2024-09-18T04:39:45.732 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9054000000000003e-05 2024-09-18T04:39:45.732 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.732 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.732 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:45.732 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.527444+0000", 2024-09-18T04:39:45.733 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8347000000000002e-05 2024-09-18T04:39:45.733 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:45.733 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:45.733 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:45.733 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.733 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.733 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:42 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:45.733 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.551160+0000", 2024-09-18T04:39:45.733 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 63.130046084, 2024-09-18T04:39:45.733 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010692757000000001, 2024-09-18T04:39:45.734 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:45.734 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:45.734 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:45.734 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:45.734 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:45.734 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 42 2024-09-18T04:39:45.734 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.734 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:45.734 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.734 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:45.734 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551160+0000", 2024-09-18T04:39:45.734 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:45.734 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.734 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.735 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:45.735 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551160+0000", 2024-09-18T04:39:45.735 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:45.735 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.735 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.735 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:45.735 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551159+0000", 2024-09-18T04:39:45.735 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:45.735 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.735 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.735 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:45.735 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551166+0000", 2024-09-18T04:39:45.735 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1029999999999996e-06 2024-09-18T04:39:45.735 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.736 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.736 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:45.736 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551177+0000", 2024-09-18T04:39:45.736 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1384e-05 2024-09-18T04:39:45.736 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.736 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.736 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:45.736 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551185+0000", 2024-09-18T04:39:45.736 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1990000000000006e-06 2024-09-18T04:39:45.736 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.736 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.736 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:45.736 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552021+0000", 2024-09-18T04:39:45.737 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00083574400000000003 2024-09-18T04:39:45.737 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.737 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.737 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:45.737 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552339+0000", 2024-09-18T04:39:45.737 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031771399999999997 2024-09-18T04:39:45.737 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.737 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.737 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:45.737 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.553140+0000", 2024-09-18T04:39:45.737 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00080088100000000003 2024-09-18T04:39:45.737 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.737 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.738 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:45.738 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.554585+0000", 2024-09-18T04:39:45.738 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001445801 2024-09-18T04:39:45.738 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.738 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.738 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:45.738 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.561711+0000", 2024-09-18T04:39:45.738 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071260109999999998 2024-09-18T04:39:45.738 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.738 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.738 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:45.738 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.561790+0000", 2024-09-18T04:39:45.738 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8967000000000005e-05 2024-09-18T04:39:45.738 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.738 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.739 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:45.739 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.561852+0000", 2024-09-18T04:39:45.739 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2045999999999995e-05 2024-09-18T04:39:45.739 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:45.739 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:45.739 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:45.739 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.739 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.739 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:58 2.7 e51/32 2:fc7c6cd3:ns::rbd_data.133e813ab5bc.000000000000000e:head v 51'7, mlcod=51'5)", 2024-09-18T04:39:45.739 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.605333+0000", 2024-09-18T04:39:45.739 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 63.075872879000002, 2024-09-18T04:39:45.739 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020085902999999999, 2024-09-18T04:39:45.739 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:45.739 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:45.740 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:45.740 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.740 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:45.740 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605333+0000", 2024-09-18T04:39:45.740 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:45.740 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.740 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.740 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:45.740 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605333+0000", 2024-09-18T04:39:45.740 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:45.740 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.740 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.740 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:45.740 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605332+0000", 2024-09-18T04:39:45.741 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:45.741 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.741 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.741 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:45.741 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608667+0000", 2024-09-18T04:39:45.741 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033350300000000001 2024-09-18T04:39:45.741 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.741 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.741 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:45.741 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608706+0000", 2024-09-18T04:39:45.741 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8951e-05 2024-09-18T04:39:45.741 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.741 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.742 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:45.742 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608725+0000", 2024-09-18T04:39:45.742 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9675999999999999e-05 2024-09-18T04:39:45.742 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.742 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.742 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:45.742 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608798+0000", 2024-09-18T04:39:45.742 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3135999999999999e-05 2024-09-18T04:39:45.742 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.742 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.742 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:45.742 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608824+0000", 2024-09-18T04:39:45.742 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5284000000000001e-05 2024-09-18T04:39:45.742 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.743 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.743 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:45.743 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625370+0000", 2024-09-18T04:39:45.743 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016545975000000001 2024-09-18T04:39:45.743 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.743 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.743 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:45.743 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625419+0000", 2024-09-18T04:39:45.743 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9076999999999998e-05 2024-09-18T04:39:45.743 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:45.743 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:45.743 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:45.743 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.744 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.744 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:611 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:45.744 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.069561+0000", 2024-09-18T04:39:45.744 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 54.611644325, 2024-09-18T04:39:45.744 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0072330299999999997, 2024-09-18T04:39:45.744 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:45.744 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:45.744 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:45.744 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:39:45.744 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:45.744 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 611 2024-09-18T04:39:45.744 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.744 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:45.744 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.745 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:45.745 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.069561+0000", 2024-09-18T04:39:45.745 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:45.745 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.745 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.745 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:45.745 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.069561+0000", 2024-09-18T04:39:45.745 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:45.745 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.745 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.745 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:45.745 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.069559+0000", 2024-09-18T04:39:45.745 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:45.746 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.746 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.746 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:45.746 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070801+0000", 2024-09-18T04:39:45.746 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0012423359999999999 2024-09-18T04:39:45.746 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.746 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.746 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:45.746 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070828+0000", 2024-09-18T04:39:45.746 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6387e-05 2024-09-18T04:39:45.746 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.746 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.746 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:45.746 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070840+0000", 2024-09-18T04:39:45.746 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2689e-05 2024-09-18T04:39:45.747 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.747 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.747 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:45.747 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070990+0000", 2024-09-18T04:39:45.747 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014946800000000001 2024-09-18T04:39:45.747 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.747 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.747 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:45.747 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.071128+0000", 2024-09-18T04:39:45.747 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013796699999999999 2024-09-18T04:39:45.747 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.747 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.747 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:45.747 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.071318+0000", 2024-09-18T04:39:45.748 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000190106 2024-09-18T04:39:45.748 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.748 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.748 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:45.748 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.075711+0000", 2024-09-18T04:39:45.748 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.004393002 2024-09-18T04:39:45.748 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.748 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.748 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:45.748 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.076665+0000", 2024-09-18T04:39:45.748 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00095390399999999999 2024-09-18T04:39:45.748 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.748 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.748 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:45.749 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.076742+0000", 2024-09-18T04:39:45.749 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7112000000000003e-05 2024-09-18T04:39:45.749 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.749 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.749 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:45.749 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.076794+0000", 2024-09-18T04:39:45.749 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2475999999999997e-05 2024-09-18T04:39:45.749 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:45.749 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:45.749 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:45.749 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.749 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.749 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:687 2.3 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:45.750 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.487994+0000", 2024-09-18T04:39:45.750 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 54.193211341999998, 2024-09-18T04:39:45.750 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0074167690000000001, 2024-09-18T04:39:45.750 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:45.750 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:45.750 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:45.750 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:39:45.750 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:45.750 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 687 2024-09-18T04:39:45.750 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.750 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:45.750 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.750 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:45.750 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.487994+0000", 2024-09-18T04:39:45.751 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:45.751 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.751 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.751 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:45.751 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.487994+0000", 2024-09-18T04:39:45.751 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:45.751 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.751 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.751 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:45.751 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.487993+0000", 2024-09-18T04:39:45.751 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:45.751 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.751 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.751 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:45.752 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.488979+0000", 2024-09-18T04:39:45.752 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00098620700000000001 2024-09-18T04:39:45.752 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.752 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.752 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:45.752 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489002+0000", 2024-09-18T04:39:45.752 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2500000000000001e-05 2024-09-18T04:39:45.752 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.752 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.752 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:45.752 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489020+0000", 2024-09-18T04:39:45.752 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7691000000000001e-05 2024-09-18T04:39:45.752 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.752 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.753 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:45.753 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489091+0000", 2024-09-18T04:39:45.753 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1307e-05 2024-09-18T04:39:45.753 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.753 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.753 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:45.753 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489219+0000", 2024-09-18T04:39:45.753 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012766000000000001 2024-09-18T04:39:45.753 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.753 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.753 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:45.753 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489387+0000", 2024-09-18T04:39:45.753 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016855400000000001 2024-09-18T04:39:45.753 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.754 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.754 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:45.754 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495243+0000", 2024-09-18T04:39:45.754 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058563189999999996 2024-09-18T04:39:45.754 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.754 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.754 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:45.754 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495266+0000", 2024-09-18T04:39:45.754 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2810999999999999e-05 2024-09-18T04:39:45.754 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.754 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.754 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:45.754 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495332+0000", 2024-09-18T04:39:45.754 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5469999999999995e-05 2024-09-18T04:39:45.755 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.755 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.755 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:45.755 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495411+0000", 2024-09-18T04:39:45.755 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9395999999999997e-05 2024-09-18T04:39:45.755 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:45.755 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:45.755 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:45.755 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.755 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.755 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:689 2.3 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:45.755 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.500070+0000", 2024-09-18T04:39:45.755 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 54.181135525999998, 2024-09-18T04:39:45.755 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071096650000000003, 2024-09-18T04:39:45.756 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:45.756 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:45.756 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:45.756 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:39:45.756 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:45.756 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 689 2024-09-18T04:39:45.756 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.756 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:45.756 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.756 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:45.756 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.500070+0000", 2024-09-18T04:39:45.756 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:45.756 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.757 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.757 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:45.757 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.500070+0000", 2024-09-18T04:39:45.757 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:45.757 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.757 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.757 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:45.757 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.500069+0000", 2024-09-18T04:39:45.757 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:45.757 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.757 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.757 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:45.757 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501082+0000", 2024-09-18T04:39:45.758 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010136570000000001 2024-09-18T04:39:45.758 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.758 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.758 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:45.758 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501104+0000", 2024-09-18T04:39:45.758 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1883e-05 2024-09-18T04:39:45.758 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.758 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.758 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:45.758 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501115+0000", 2024-09-18T04:39:45.758 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0414000000000001e-05 2024-09-18T04:39:45.758 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.758 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.759 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:45.759 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501178+0000", 2024-09-18T04:39:45.759 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3128999999999998e-05 2024-09-18T04:39:45.759 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.759 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.759 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:45.759 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501257+0000", 2024-09-18T04:39:45.759 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9592999999999995e-05 2024-09-18T04:39:45.759 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.759 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.759 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:45.759 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501366+0000", 2024-09-18T04:39:45.759 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000108643 2024-09-18T04:39:45.759 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.760 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.760 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:45.760 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.504041+0000", 2024-09-18T04:39:45.760 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0026750889999999999 2024-09-18T04:39:45.760 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.760 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.760 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:45.760 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.507097+0000", 2024-09-18T04:39:45.760 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030556070000000001 2024-09-18T04:39:45.760 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.760 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.760 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:45.760 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.507140+0000", 2024-09-18T04:39:45.761 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3745000000000002e-05 2024-09-18T04:39:45.761 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.761 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.761 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:45.761 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.507180+0000", 2024-09-18T04:39:45.761 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9475999999999999e-05 2024-09-18T04:39:45.761 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:45.761 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:45.761 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:45.761 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.761 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.761 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4972.0:13 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:45.761 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.948263+0000", 2024-09-18T04:39:45.761 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 53.732942735000002, 2024-09-18T04:39:45.762 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0091358970000000005, 2024-09-18T04:39:45.762 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:45.762 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:39:45.762 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:45.762 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4972", 2024-09-18T04:39:45.762 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2280251472", 2024-09-18T04:39:45.762 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 13 2024-09-18T04:39:45.762 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.762 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:45.762 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.762 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:45.762 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948263+0000", 2024-09-18T04:39:45.762 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:45.763 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.763 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.763 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:45.763 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948263+0000", 2024-09-18T04:39:45.763 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:45.763 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.763 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.763 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:45.763 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948261+0000", 2024-09-18T04:39:45.763 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:45.763 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.763 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.763 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:45.763 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948273+0000", 2024-09-18T04:39:45.764 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1576999999999999e-05 2024-09-18T04:39:45.764 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.764 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.764 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:45.764 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948305+0000", 2024-09-18T04:39:45.764 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2209000000000003e-05 2024-09-18T04:39:45.764 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.764 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.764 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:45.764 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948325+0000", 2024-09-18T04:39:45.764 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.022e-05 2024-09-18T04:39:45.764 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.764 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.764 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:45.765 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948455+0000", 2024-09-18T04:39:45.765 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000129624 2024-09-18T04:39:45.765 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.765 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.765 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:45.765 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948598+0000", 2024-09-18T04:39:45.765 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014376699999999999 2024-09-18T04:39:45.765 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.765 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.765 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:45.765 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.957399+0000", 2024-09-18T04:39:45.765 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0088004299999999997 2024-09-18T04:39:45.765 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:45.765 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:45.766 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:45.766 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.766 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.766 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:101 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:45.766 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.576367+0000", 2024-09-18T04:39:45.766 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 49.104839026999997, 2024-09-18T04:39:45.766 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0078671020000000008, 2024-09-18T04:39:45.766 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:45.766 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:45.766 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:45.766 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:45.766 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:45.766 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 101 2024-09-18T04:39:45.766 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.766 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:45.767 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.767 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:45.767 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576367+0000", 2024-09-18T04:39:45.767 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:45.767 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.767 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.767 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:45.767 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576367+0000", 2024-09-18T04:39:45.767 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:45.767 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.767 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.767 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:45.767 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576364+0000", 2024-09-18T04:39:45.767 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999971 2024-09-18T04:39:45.768 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.768 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.768 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:45.768 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576378+0000", 2024-09-18T04:39:45.768 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3879999999999999e-05 2024-09-18T04:39:45.768 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.768 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.768 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:45.768 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576410+0000", 2024-09-18T04:39:45.768 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2286000000000002e-05 2024-09-18T04:39:45.768 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.768 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.768 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:45.768 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576425+0000", 2024-09-18T04:39:45.769 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5290000000000001e-05 2024-09-18T04:39:45.769 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.769 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.769 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:45.769 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576551+0000", 2024-09-18T04:39:45.769 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012555199999999999 2024-09-18T04:39:45.769 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.769 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.769 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:45.769 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576719+0000", 2024-09-18T04:39:45.769 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016787400000000001 2024-09-18T04:39:45.769 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.769 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.769 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:45.770 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576951+0000", 2024-09-18T04:39:45.770 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00023252400000000001 2024-09-18T04:39:45.770 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.770 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.770 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:45.770 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.581459+0000", 2024-09-18T04:39:45.770 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045074410000000001 2024-09-18T04:39:45.770 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.770 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.770 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:45.770 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584076+0000", 2024-09-18T04:39:45.770 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0026173860000000002 2024-09-18T04:39:45.770 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.771 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.771 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:45.771 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584160+0000", 2024-09-18T04:39:45.771 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4191999999999996e-05 2024-09-18T04:39:45.771 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.771 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.771 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:45.771 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584234+0000", 2024-09-18T04:39:45.771 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.339e-05 2024-09-18T04:39:45.771 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:45.771 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:45.771 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:45.771 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.771 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.772 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:44 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:45.772 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.550448+0000", 2024-09-18T04:39:45.772 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 45.130757838000001, 2024-09-18T04:39:45.772 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082117590000000008, 2024-09-18T04:39:45.772 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:45.772 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:45.772 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:45.772 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:39:45.772 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:45.772 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 44 2024-09-18T04:39:45.772 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.772 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:45.772 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.772 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:45.773 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.550448+0000", 2024-09-18T04:39:45.773 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:45.773 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.773 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.773 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:45.773 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.550448+0000", 2024-09-18T04:39:45.773 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:45.773 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.773 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.773 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:45.773 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.550445+0000", 2024-09-18T04:39:45.773 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999971 2024-09-18T04:39:45.773 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.773 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.774 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:45.774 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.551904+0000", 2024-09-18T04:39:45.774 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001458322 2024-09-18T04:39:45.774 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.774 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.774 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:45.774 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.551938+0000", 2024-09-18T04:39:45.774 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4359000000000001e-05 2024-09-18T04:39:45.774 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.774 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.774 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:45.774 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.551959+0000", 2024-09-18T04:39:45.774 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1403000000000002e-05 2024-09-18T04:39:45.774 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.775 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.775 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:45.775 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.552049+0000", 2024-09-18T04:39:45.775 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9970999999999996e-05 2024-09-18T04:39:45.775 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.775 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.775 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:45.775 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.552190+0000", 2024-09-18T04:39:45.775 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000140271 2024-09-18T04:39:45.775 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.775 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.775 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:45.775 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.552385+0000", 2024-09-18T04:39:45.776 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019498700000000001 2024-09-18T04:39:45.776 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.776 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.776 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:45.776 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.556308+0000", 2024-09-18T04:39:45.776 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00392333 2024-09-18T04:39:45.776 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.776 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.776 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:45.776 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.558479+0000", 2024-09-18T04:39:45.776 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0021715889999999998 2024-09-18T04:39:45.776 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.776 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.776 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:45.776 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.558616+0000", 2024-09-18T04:39:45.777 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013613399999999999 2024-09-18T04:39:45.777 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.777 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.777 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:45.777 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.558660+0000", 2024-09-18T04:39:45.777 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4053e-05 2024-09-18T04:39:45.777 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:45.777 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:45.777 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:45.777 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.777 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.777 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:46 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:45.777 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.564089+0000", 2024-09-18T04:39:45.777 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 45.117116709000001, 2024-09-18T04:39:45.778 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0069437090000000002, 2024-09-18T04:39:45.778 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:45.778 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:45.778 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:45.778 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:39:45.778 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:45.778 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 46 2024-09-18T04:39:45.778 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.778 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:45.778 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.778 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:45.778 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.564089+0000", 2024-09-18T04:39:45.778 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:45.779 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.779 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.779 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:45.779 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.564089+0000", 2024-09-18T04:39:45.779 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:45.779 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.779 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.779 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:45.779 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.564087+0000", 2024-09-18T04:39:45.779 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:45.779 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.779 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.779 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:45.779 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565126+0000", 2024-09-18T04:39:45.780 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001038902 2024-09-18T04:39:45.780 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.780 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.780 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:45.780 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565148+0000", 2024-09-18T04:39:45.780 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1911000000000001e-05 2024-09-18T04:39:45.780 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.780 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.780 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:45.780 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565157+0000", 2024-09-18T04:39:45.780 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6220000000000004e-06 2024-09-18T04:39:45.780 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.780 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.780 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:45.781 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565261+0000", 2024-09-18T04:39:45.781 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000103063 2024-09-18T04:39:45.781 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.781 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.781 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:45.781 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565394+0000", 2024-09-18T04:39:45.781 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000133032 2024-09-18T04:39:45.781 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.781 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.781 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:45.781 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565564+0000", 2024-09-18T04:39:45.781 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017068000000000001 2024-09-18T04:39:45.781 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.781 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.782 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:45.782 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.570556+0000", 2024-09-18T04:39:45.782 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0049920620000000002 2024-09-18T04:39:45.782 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.782 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.782 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:45.782 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.570885+0000", 2024-09-18T04:39:45.782 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032827900000000001 2024-09-18T04:39:45.782 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.782 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.782 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:45.782 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.570959+0000", 2024-09-18T04:39:45.782 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3955999999999995e-05 2024-09-18T04:39:45.782 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.783 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.783 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:45.783 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.571033+0000", 2024-09-18T04:39:45.783 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4191999999999997e-05 2024-09-18T04:39:45.783 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:45.783 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:45.783 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:45.783 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.783 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.783 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:100 2.4 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:45.783 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.856389+0000", 2024-09-18T04:39:45.783 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 44.824816597000002, 2024-09-18T04:39:45.783 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0081480800000000003, 2024-09-18T04:39:45.783 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:45.784 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:45.784 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:45.784 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:39:45.784 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:45.784 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 100 2024-09-18T04:39:45.784 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.784 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:45.784 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.784 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:45.784 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.856389+0000", 2024-09-18T04:39:45.784 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:45.785 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.785 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.785 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:45.785 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.856389+0000", 2024-09-18T04:39:45.785 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:45.785 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.785 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.785 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:45.785 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.856388+0000", 2024-09-18T04:39:45.785 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:45.785 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.785 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.785 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:45.785 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857295+0000", 2024-09-18T04:39:45.786 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00090725599999999995 2024-09-18T04:39:45.786 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.786 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.786 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:45.786 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857311+0000", 2024-09-18T04:39:45.786 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6092000000000001e-05 2024-09-18T04:39:45.786 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.786 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.786 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:45.786 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857319+0000", 2024-09-18T04:39:45.786 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9200000000000004e-06 2024-09-18T04:39:45.786 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.786 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.787 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:45.787 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857388+0000", 2024-09-18T04:39:45.787 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8591999999999997e-05 2024-09-18T04:39:45.787 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.787 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.787 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:45.787 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857535+0000", 2024-09-18T04:39:45.787 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014684299999999999 2024-09-18T04:39:45.787 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.787 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.787 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:45.787 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857747+0000", 2024-09-18T04:39:45.787 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000212361 2024-09-18T04:39:45.787 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.788 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.788 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:45.788 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.860831+0000", 2024-09-18T04:39:45.788 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030841789999999998 2024-09-18T04:39:45.788 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.788 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.788 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:45.788 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.864361+0000", 2024-09-18T04:39:45.788 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0035298780000000002 2024-09-18T04:39:45.788 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.788 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.788 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:45.788 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.864456+0000", 2024-09-18T04:39:45.788 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4557999999999998e-05 2024-09-18T04:39:45.788 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.789 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.789 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:45.789 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.864537+0000", 2024-09-18T04:39:45.789 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1577999999999997e-05 2024-09-18T04:39:45.789 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:45.789 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:45.789 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:45.789 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.789 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.789 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:102 2.4 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:45.789 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.869773+0000", 2024-09-18T04:39:45.789 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 44.811432940000003, 2024-09-18T04:39:45.789 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0084430359999999992, 2024-09-18T04:39:45.789 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:45.790 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:45.790 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:45.790 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:39:45.790 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:45.790 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 102 2024-09-18T04:39:45.790 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.790 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:45.790 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.790 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:45.790 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.869773+0000", 2024-09-18T04:39:45.790 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:45.790 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.790 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.790 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:45.791 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.869773+0000", 2024-09-18T04:39:45.791 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:45.791 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.791 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.791 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:45.791 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.869771+0000", 2024-09-18T04:39:45.791 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:45.791 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.791 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.791 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:45.791 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870629+0000", 2024-09-18T04:39:45.791 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00085760999999999997 2024-09-18T04:39:45.791 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.791 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.792 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:45.792 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870647+0000", 2024-09-18T04:39:45.792 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8363999999999999e-05 2024-09-18T04:39:45.792 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.792 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.792 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:45.792 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870655+0000", 2024-09-18T04:39:45.792 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0099999999999995e-06 2024-09-18T04:39:45.792 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.792 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.792 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:45.792 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870803+0000", 2024-09-18T04:39:45.792 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014798 2024-09-18T04:39:45.792 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.793 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.793 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:45.793 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870951+0000", 2024-09-18T04:39:45.793 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014715999999999999 2024-09-18T04:39:45.793 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.793 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.793 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:45.793 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.871145+0000", 2024-09-18T04:39:45.793 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000194253 2024-09-18T04:39:45.793 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.793 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.793 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:45.793 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.877033+0000", 2024-09-18T04:39:45.793 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058883099999999999 2024-09-18T04:39:45.794 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.794 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.794 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:45.794 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.878078+0000", 2024-09-18T04:39:45.794 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001044747 2024-09-18T04:39:45.794 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.794 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.794 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:45.794 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.878160+0000", 2024-09-18T04:39:45.794 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1904999999999996e-05 2024-09-18T04:39:45.794 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.794 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.794 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:45.794 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.878216+0000", 2024-09-18T04:39:45.795 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6082000000000003e-05 2024-09-18T04:39:45.795 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:45.795 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:45.795 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:45.795 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.795 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.795 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:145 2.4 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:45.795 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:01.096935+0000", 2024-09-18T04:39:45.795 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 44.58427116, 2024-09-18T04:39:45.795 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0069937869999999996, 2024-09-18T04:39:45.795 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:45.795 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:45.795 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:45.795 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:39:45.796 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:45.796 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 145 2024-09-18T04:39:45.796 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.796 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:45.796 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.796 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:45.796 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.096935+0000", 2024-09-18T04:39:45.796 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:45.796 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.796 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.796 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:45.796 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.096935+0000", 2024-09-18T04:39:45.796 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:45.797 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.797 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.797 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:45.797 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.096932+0000", 2024-09-18T04:39:45.797 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:45.797 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.797 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.797 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:45.797 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097815+0000", 2024-09-18T04:39:45.797 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00088276000000000001 2024-09-18T04:39:45.797 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.797 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.797 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:45.798 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097842+0000", 2024-09-18T04:39:45.798 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6806999999999999e-05 2024-09-18T04:39:45.798 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.798 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.798 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:45.798 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097852+0000", 2024-09-18T04:39:45.798 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0548e-05 2024-09-18T04:39:45.798 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.798 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.798 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:45.798 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097931+0000", 2024-09-18T04:39:45.798 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8417000000000005e-05 2024-09-18T04:39:45.798 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.798 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.799 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:45.799 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.098047+0000", 2024-09-18T04:39:45.799 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011611 2024-09-18T04:39:45.799 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.799 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.799 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:45.799 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.098172+0000", 2024-09-18T04:39:45.799 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000125482 2024-09-18T04:39:45.799 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.799 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.799 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:45.799 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.100888+0000", 2024-09-18T04:39:45.799 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0027154430000000001 2024-09-18T04:39:45.799 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.800 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.800 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:45.800 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.103814+0000", 2024-09-18T04:39:45.800 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0029259619999999998 2024-09-18T04:39:45.800 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.800 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.800 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:45.800 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.103891+0000", 2024-09-18T04:39:45.800 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7619000000000003e-05 2024-09-18T04:39:45.800 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:45.800 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:45.800 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:45.800 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.103928+0000", 2024-09-18T04:39:45.800 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6900999999999998e-05 2024-09-18T04:39:45.801 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:45.801 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:45.801 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:45.801 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:45.801 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:45.801 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:45.803 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-09-18T04:39:45.803 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:39:45.804 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:45.800+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:45.903 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:45.896+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:45.967 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:45.967 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:45.967 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:39:45.967 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:45.977 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:39:46.004 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:45.996+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:46.082 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:46.082 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:46.082 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:39:46.082 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:39:46.082 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:46.092 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:39:46.105 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:46.100+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:46.202 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:46.202 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:39:46.202 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:39:46.202 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [] 2024-09-18T04:39:46.202 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:46.206 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:46.200+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:46.212 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:39:46.306 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:46.300+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:46.366 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:46.367 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:46.367 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:39:46.367 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:46.377 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:39:46.407 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:46.404+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:46.484 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:46.484 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:46.484 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:39:46.484 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:39:46.485 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:46.494 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:39:46.539 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:46.536+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:46.609 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:46.604+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:46.650 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:46.650 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:39:46.650 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:39:46.650 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:39:46.650 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.651 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:37 2.5 e51/28 2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:head v 51'1, mlcod=0'0)", 2024-09-18T04:39:46.651 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:39:46.651 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 64.354540150999995, 2024-09-18T04:39:46.651 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012279973, 2024-09-18T04:39:46.651 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:46.651 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:46.651 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:46.651 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.651 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:46.651 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:39:46.651 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:46.651 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.651 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.651 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:46.652 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:39:46.652 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:46.652 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.652 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.652 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:46.652 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292468+0000", 2024-09-18T04:39:46.652 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:46.652 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.652 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.652 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:46.652 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295837+0000", 2024-09-18T04:39:46.652 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0033691699999999999 2024-09-18T04:39:46.653 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.653 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.653 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:46.653 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295860+0000", 2024-09-18T04:39:46.653 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2756999999999999e-05 2024-09-18T04:39:46.653 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.653 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.653 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:46.653 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295872+0000", 2024-09-18T04:39:46.653 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2442e-05 2024-09-18T04:39:46.653 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.653 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.653 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:46.654 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295934+0000", 2024-09-18T04:39:46.654 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1187999999999998e-05 2024-09-18T04:39:46.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.654 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.654 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:46.654 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295967+0000", 2024-09-18T04:39:46.654 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3185999999999997e-05 2024-09-18T04:39:46.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.654 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.654 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:46.654 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304697+0000", 2024-09-18T04:39:46.654 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0087298059999999997 2024-09-18T04:39:46.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.654 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.655 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:46.655 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304749+0000", 2024-09-18T04:39:46.655 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1922000000000002e-05 2024-09-18T04:39:46.655 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:46.655 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:46.655 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:46.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.655 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.655 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:49 2.5 e51/28 2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head v 51'2, mlcod=0'0)", 2024-09-18T04:39:46.655 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:39:46.655 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 64.321213313000001, 2024-09-18T04:39:46.655 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018033041999999999, 2024-09-18T04:39:46.655 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:46.656 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:46.656 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:46.656 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.656 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:46.656 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:39:46.656 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:46.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.656 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.656 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:46.656 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:39:46.656 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:46.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.656 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.657 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:46.657 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325793+0000", 2024-09-18T04:39:46.657 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:46.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.657 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.657 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:46.657 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329038+0000", 2024-09-18T04:39:46.657 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0032444010000000001 2024-09-18T04:39:46.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.657 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.657 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:46.657 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329065+0000", 2024-09-18T04:39:46.657 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7500000000000001e-05 2024-09-18T04:39:46.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.658 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.658 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:46.658 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329077+0000", 2024-09-18T04:39:46.658 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1919999999999999e-05 2024-09-18T04:39:46.658 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.658 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.658 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:46.658 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329149+0000", 2024-09-18T04:39:46.658 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2349999999999997e-05 2024-09-18T04:39:46.658 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.658 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.658 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:46.658 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329192+0000", 2024-09-18T04:39:46.659 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2744000000000003e-05 2024-09-18T04:39:46.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.659 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.659 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:46.659 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343773+0000", 2024-09-18T04:39:46.659 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014581246000000001 2024-09-18T04:39:46.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.659 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.659 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:46.659 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343829+0000", 2024-09-18T04:39:46.659 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5143999999999999e-05 2024-09-18T04:39:46.659 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:46.659 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:46.659 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:46.660 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.660 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.660 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:47 2.4 e51/17 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 51'3, mlcod=51'1)", 2024-09-18T04:39:46.660 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:39:46.660 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 64.320633048000005, 2024-09-18T04:39:46.660 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018960186, 2024-09-18T04:39:46.660 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:46.660 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:46.660 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:46.660 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.660 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:46.660 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:39:46.660 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:46.661 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.661 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.661 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:46.661 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:39:46.661 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:46.661 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.661 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.661 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:46.661 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326374+0000", 2024-09-18T04:39:46.661 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:46.661 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.661 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.661 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:46.661 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330883+0000", 2024-09-18T04:39:46.662 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0045084740000000002 2024-09-18T04:39:46.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.662 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.662 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:46.662 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330901+0000", 2024-09-18T04:39:46.662 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7890999999999999e-05 2024-09-18T04:39:46.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.662 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.662 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:46.662 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330912+0000", 2024-09-18T04:39:46.662 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1041e-05 2024-09-18T04:39:46.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.662 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.663 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:46.663 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.331003+0000", 2024-09-18T04:39:46.663 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1323999999999995e-05 2024-09-18T04:39:46.663 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.663 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.663 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:46.663 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.331030+0000", 2024-09-18T04:39:46.663 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7351999999999998e-05 2024-09-18T04:39:46.663 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.663 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.663 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:46.663 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345287+0000", 2024-09-18T04:39:46.663 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014256332 2024-09-18T04:39:46.664 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.664 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.664 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:46.664 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345336+0000", 2024-09-18T04:39:46.664 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9356e-05 2024-09-18T04:39:46.664 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:46.664 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:46.664 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:46.664 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.664 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.664 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:58 2.5 e51/28 2:b227e6b1:::rbd_data.125fc09df6d1.000000000000000e:head v 51'3, mlcod=51'1)", 2024-09-18T04:39:46.664 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:39:46.664 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 64.280206421000003, 2024-09-18T04:39:46.664 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016513678, 2024-09-18T04:39:46.665 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:46.665 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:46.665 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:46.665 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.665 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:46.665 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:39:46.665 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:46.665 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.665 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.665 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:46.665 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:39:46.665 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:46.665 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.666 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.666 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:46.666 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366800+0000", 2024-09-18T04:39:46.666 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:46.666 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.666 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.666 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:46.666 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368644+0000", 2024-09-18T04:39:46.666 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001844399 2024-09-18T04:39:46.666 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.666 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.666 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:46.666 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368681+0000", 2024-09-18T04:39:46.666 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6526999999999998e-05 2024-09-18T04:39:46.667 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.667 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.667 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:46.667 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368698+0000", 2024-09-18T04:39:46.667 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7272999999999999e-05 2024-09-18T04:39:46.667 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.667 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.667 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:46.667 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368773+0000", 2024-09-18T04:39:46.667 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4862000000000003e-05 2024-09-18T04:39:46.667 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.667 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.667 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:46.667 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368801+0000", 2024-09-18T04:39:46.667 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7925000000000001e-05 2024-09-18T04:39:46.668 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.668 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.668 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:46.668 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383261+0000", 2024-09-18T04:39:46.668 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014460377999999999 2024-09-18T04:39:46.668 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.668 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.668 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:46.668 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383316+0000", 2024-09-18T04:39:46.668 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4623999999999997e-05 2024-09-18T04:39:46.668 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:46.668 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:46.669 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:46.669 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.669 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.669 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:56 2.4 e51/17 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 51'4, mlcod=51'2)", 2024-09-18T04:39:46.669 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:39:46.669 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 64.278202394999994, 2024-09-18T04:39:46.669 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015105221, 2024-09-18T04:39:46.669 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:46.669 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:46.669 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:46.669 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.669 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:46.669 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:39:46.669 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:46.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.670 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.670 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:46.670 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:39:46.670 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:46.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.670 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.670 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:46.670 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368805+0000", 2024-09-18T04:39:46.670 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:46.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.670 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.670 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:46.670 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371603+0000", 2024-09-18T04:39:46.671 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027982660000000002 2024-09-18T04:39:46.671 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.671 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.671 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:46.671 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371633+0000", 2024-09-18T04:39:46.671 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0073000000000002e-05 2024-09-18T04:39:46.671 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.671 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.671 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:46.671 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371652+0000", 2024-09-18T04:39:46.671 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8597000000000002e-05 2024-09-18T04:39:46.671 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.671 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.672 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:46.672 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371752+0000", 2024-09-18T04:39:46.672 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9772999999999995e-05 2024-09-18T04:39:46.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.672 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.672 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:46.672 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371793+0000", 2024-09-18T04:39:46.672 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1173e-05 2024-09-18T04:39:46.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.672 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.672 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:46.672 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383875+0000", 2024-09-18T04:39:46.672 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012081852000000001 2024-09-18T04:39:46.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.673 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.673 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:46.673 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383912+0000", 2024-09-18T04:39:46.673 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7045000000000002e-05 2024-09-18T04:39:46.673 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:46.673 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:46.673 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:46.673 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.673 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.673 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:36 2.5 e51/28 2:bd5b8dd8:ns::rbd_data.133e813ab5bc.0000000000000002:head v 51'4, mlcod=51'2)", 2024-09-18T04:39:46.673 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:39:46.673 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 64.109483404000002, 2024-09-18T04:39:46.673 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020758161000000001, 2024-09-18T04:39:46.674 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:46.674 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:46.674 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:46.674 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.674 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:46.674 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:39:46.674 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:46.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.674 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.674 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:46.674 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:39:46.674 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:46.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.674 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.675 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:46.675 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537523+0000", 2024-09-18T04:39:46.675 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:46.675 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.675 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.675 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:46.675 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541578+0000", 2024-09-18T04:39:46.675 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0040549710000000001 2024-09-18T04:39:46.675 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.675 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.675 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:46.675 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541608+0000", 2024-09-18T04:39:46.675 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9349e-05 2024-09-18T04:39:46.675 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.675 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.676 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:46.676 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541621+0000", 2024-09-18T04:39:46.676 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2736e-05 2024-09-18T04:39:46.676 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.676 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.676 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:46.676 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541715+0000", 2024-09-18T04:39:46.676 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4197999999999995e-05 2024-09-18T04:39:46.676 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.676 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.676 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:46.676 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541755+0000", 2024-09-18T04:39:46.676 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9818999999999999e-05 2024-09-18T04:39:46.677 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.677 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.677 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:46.677 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558217+0000", 2024-09-18T04:39:46.677 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016462234999999999 2024-09-18T04:39:46.677 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.677 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.677 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:46.677 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558284+0000", 2024-09-18T04:39:46.677 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6774999999999997e-05 2024-09-18T04:39:46.677 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:46.677 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:46.677 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:46.677 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.678 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.678 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:39 2.4 e51/17 2:304364d1:ns::rbd_data.133e813ab5bc.0000000000000004:head v 51'5, mlcod=51'3)", 2024-09-18T04:39:46.678 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:39:46.678 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 64.101344276999995, 2024-09-18T04:39:46.678 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016777558000000001, 2024-09-18T04:39:46.678 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:46.678 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:46.678 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:46.678 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.678 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:46.678 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:39:46.678 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:46.678 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.678 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.679 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:46.679 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:39:46.679 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:46.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.679 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.679 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:46.679 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545663+0000", 2024-09-18T04:39:46.679 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:46.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.679 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.679 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:46.679 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548286+0000", 2024-09-18T04:39:46.679 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026233390000000001 2024-09-18T04:39:46.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.680 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.680 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:46.680 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548305+0000", 2024-09-18T04:39:46.680 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8425000000000001e-05 2024-09-18T04:39:46.680 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.680 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.680 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:46.680 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548316+0000", 2024-09-18T04:39:46.680 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1163e-05 2024-09-18T04:39:46.680 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.680 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.680 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:46.680 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548379+0000", 2024-09-18T04:39:46.681 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2780000000000005e-05 2024-09-18T04:39:46.681 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.681 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.681 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:46.681 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548399+0000", 2024-09-18T04:39:46.681 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0080000000000001e-05 2024-09-18T04:39:46.681 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.681 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.681 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:46.681 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562383+0000", 2024-09-18T04:39:46.681 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013984671000000001 2024-09-18T04:39:46.681 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.681 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.681 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:46.681 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562442+0000", 2024-09-18T04:39:46.682 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8700999999999999e-05 2024-09-18T04:39:46.682 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:46.682 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:46.682 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:46.682 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.682 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.682 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:37 2.5 e51/28 2:bf39b8e4:ns::rbd_data.133e813ab5bc.0000000000000003:head v 51'5, mlcod=51'2)", 2024-09-18T04:39:46.682 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:39:46.682 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 64.098476008000006, 2024-09-18T04:39:46.682 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014264492, 2024-09-18T04:39:46.682 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:46.682 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:46.682 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:46.682 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.683 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:46.683 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:39:46.683 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:46.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.683 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.683 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:46.683 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:39:46.683 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:46.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.683 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.683 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:46.683 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548531+0000", 2024-09-18T04:39:46.684 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:46.684 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.684 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.684 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:46.684 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551500+0000", 2024-09-18T04:39:46.684 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0029688129999999998 2024-09-18T04:39:46.684 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.684 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.684 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:46.684 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551516+0000", 2024-09-18T04:39:46.684 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5994e-05 2024-09-18T04:39:46.684 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.684 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.685 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:46.685 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551526+0000", 2024-09-18T04:39:46.685 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9299999999999998e-06 2024-09-18T04:39:46.685 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.685 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.685 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:46.685 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551577+0000", 2024-09-18T04:39:46.685 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0871000000000001e-05 2024-09-18T04:39:46.685 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.685 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.685 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:46.685 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551602+0000", 2024-09-18T04:39:46.685 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4813000000000001e-05 2024-09-18T04:39:46.686 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.686 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.686 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:46.686 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562741+0000", 2024-09-18T04:39:46.686 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011139428999999999 2024-09-18T04:39:46.686 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.686 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.686 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:46.686 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562797+0000", 2024-09-18T04:39:46.686 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6329000000000002e-05 2024-09-18T04:39:46.686 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:46.686 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:46.686 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:46.686 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.687 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.687 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:49 2.4 e51/17 2:20f0b0c9:ns::rbd_data.133e813ab5bc.0000000000000009:head v 51'6, mlcod=51'4)", 2024-09-18T04:39:46.687 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:39:46.687 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 64.067041450000005, 2024-09-18T04:39:46.687 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013849046, 2024-09-18T04:39:46.687 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:46.687 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:46.687 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:46.687 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.687 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:46.687 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:39:46.687 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:46.687 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.688 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.688 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:46.688 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:39:46.688 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:46.688 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.688 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.688 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:46.688 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579966+0000", 2024-09-18T04:39:46.688 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:46.688 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.689 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.689 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:46.689 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582324+0000", 2024-09-18T04:39:46.689 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023578890000000002 2024-09-18T04:39:46.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.689 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.689 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:46.689 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582355+0000", 2024-09-18T04:39:46.689 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1485000000000001e-05 2024-09-18T04:39:46.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.689 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.689 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:46.690 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582371+0000", 2024-09-18T04:39:46.690 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6147999999999999e-05 2024-09-18T04:39:46.690 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.690 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.690 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:46.690 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582467+0000", 2024-09-18T04:39:46.690 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6059999999999998e-05 2024-09-18T04:39:46.690 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.690 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.690 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:46.690 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582505+0000", 2024-09-18T04:39:46.690 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7679000000000003e-05 2024-09-18T04:39:46.690 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.690 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.691 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:46.691 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593743+0000", 2024-09-18T04:39:46.691 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011238356 2024-09-18T04:39:46.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.691 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.691 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:46.691 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593816+0000", 2024-09-18T04:39:46.691 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2935000000000006e-05 2024-09-18T04:39:46.691 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:46.691 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:46.691 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:46.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.691 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.691 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:53 2.4 e51/17 2:39f7f174:ns::rbd_data.133e813ab5bc.000000000000000b:head v 51'7, mlcod=51'4)", 2024-09-18T04:39:46.692 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:39:46.692 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 64.052340294999993, 2024-09-18T04:39:46.692 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0079914159999999995, 2024-09-18T04:39:46.692 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:46.692 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:46.692 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:46.692 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.692 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:46.692 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:39:46.692 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:46.692 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.692 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.692 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:46.693 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:39:46.693 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:46.693 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.693 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.693 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:46.693 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594667+0000", 2024-09-18T04:39:46.693 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:46.693 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.693 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.693 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:46.693 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596446+0000", 2024-09-18T04:39:46.693 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001779033 2024-09-18T04:39:46.693 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.693 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.694 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:46.694 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596472+0000", 2024-09-18T04:39:46.694 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6120000000000001e-05 2024-09-18T04:39:46.694 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.694 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.696 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:46.696 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596500+0000", 2024-09-18T04:39:46.696 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7518e-05 2024-09-18T04:39:46.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.696 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.696 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:46.696 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596564+0000", 2024-09-18T04:39:46.696 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3880999999999994e-05 2024-09-18T04:39:46.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.696 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.696 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:46.696 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596596+0000", 2024-09-18T04:39:46.696 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2638999999999997e-05 2024-09-18T04:39:46.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.697 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.697 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:46.697 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602605+0000", 2024-09-18T04:39:46.697 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0060080699999999999 2024-09-18T04:39:46.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.697 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.697 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:46.697 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602660+0000", 2024-09-18T04:39:46.697 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5396000000000003e-05 2024-09-18T04:39:46.697 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:46.697 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:46.697 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:46.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.698 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.698 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:60 2.5 e51/28 2:a06f51e4:ns::rbd_data.133e813ab5bc.000000000000000f:head v 51'6, mlcod=51'4)", 2024-09-18T04:39:46.698 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:39:46.698 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 64.037523309999997, 2024-09-18T04:39:46.698 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0093954989999999999, 2024-09-18T04:39:46.698 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:46.698 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:46.698 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:46.698 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.698 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:46.698 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:39:46.698 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:46.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.699 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.699 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:46.699 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:39:46.699 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:46.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.699 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.699 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:46.699 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609483+0000", 2024-09-18T04:39:46.699 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:46.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.699 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.699 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:46.700 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611767+0000", 2024-09-18T04:39:46.700 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002283724 2024-09-18T04:39:46.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.700 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.700 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:46.700 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611791+0000", 2024-09-18T04:39:46.700 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3547e-05 2024-09-18T04:39:46.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.700 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.700 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:46.700 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611800+0000", 2024-09-18T04:39:46.700 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4979999999999995e-06 2024-09-18T04:39:46.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.701 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.701 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:46.701 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611864+0000", 2024-09-18T04:39:46.701 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3974999999999996e-05 2024-09-18T04:39:46.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.701 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.701 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:46.701 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611887+0000", 2024-09-18T04:39:46.701 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2633000000000002e-05 2024-09-18T04:39:46.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.701 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.701 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:46.701 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.618777+0000", 2024-09-18T04:39:46.701 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0068897979999999999 2024-09-18T04:39:46.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.702 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.702 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:46.702 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.618881+0000", 2024-09-18T04:39:46.702 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010437699999999999 2024-09-18T04:39:46.702 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:46.702 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:46.702 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:46.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.702 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.702 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:663 2.4 e57/17 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 57'13, mlcod=57'11)", 2024-09-18T04:39:46.702 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.355300+0000", 2024-09-18T04:39:46.702 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 55.291708374999999, 2024-09-18T04:39:46.702 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0049192760000000002, 2024-09-18T04:39:46.703 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:46.703 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:46.703 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:46.703 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.703 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:46.703 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355300+0000", 2024-09-18T04:39:46.703 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:46.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.703 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.703 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:46.703 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355300+0000", 2024-09-18T04:39:46.703 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:46.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.704 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.704 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:46.704 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355299+0000", 2024-09-18T04:39:46.704 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:46.704 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.704 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.704 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:46.704 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355876+0000", 2024-09-18T04:39:46.704 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00057730900000000003 2024-09-18T04:39:46.704 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.704 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.704 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:46.705 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355897+0000", 2024-09-18T04:39:46.705 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0477000000000001e-05 2024-09-18T04:39:46.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.705 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.705 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:46.705 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355905+0000", 2024-09-18T04:39:46.705 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5739999999999996e-06 2024-09-18T04:39:46.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.705 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.705 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:46.705 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355989+0000", 2024-09-18T04:39:46.705 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4002e-05 2024-09-18T04:39:46.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.705 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.706 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:46.706 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.356041+0000", 2024-09-18T04:39:46.706 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1347000000000002e-05 2024-09-18T04:39:46.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.706 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.706 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:46.706 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.360172+0000", 2024-09-18T04:39:46.706 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0041315090000000002 2024-09-18T04:39:46.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.706 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.706 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:46.706 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.360220+0000", 2024-09-18T04:39:46.706 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7584999999999999e-05 2024-09-18T04:39:46.706 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:46.707 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:46.707 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:46.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.707 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.707 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:679 2.5 e57/28 2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head v 57'20, mlcod=57'18)", 2024-09-18T04:39:46.707 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.445316+0000", 2024-09-18T04:39:46.707 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 55.201692897999997, 2024-09-18T04:39:46.707 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0047016760000000001, 2024-09-18T04:39:46.707 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:46.707 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:46.707 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:46.707 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.707 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:46.707 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445316+0000", 2024-09-18T04:39:46.708 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:46.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.708 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.708 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:46.708 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445316+0000", 2024-09-18T04:39:46.708 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:46.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.708 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.708 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:46.708 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445314+0000", 2024-09-18T04:39:46.708 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:46.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.708 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.709 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:46.709 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445947+0000", 2024-09-18T04:39:46.709 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00063298900000000003 2024-09-18T04:39:46.709 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.709 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.709 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:46.709 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445968+0000", 2024-09-18T04:39:46.709 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.054e-05 2024-09-18T04:39:46.709 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.709 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.709 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:46.709 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445981+0000", 2024-09-18T04:39:46.710 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2697000000000001e-05 2024-09-18T04:39:46.710 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.710 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.710 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:46.710 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.446102+0000", 2024-09-18T04:39:46.710 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012189300000000001 2024-09-18T04:39:46.710 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.710 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.710 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:46.710 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.446151+0000", 2024-09-18T04:39:46.710 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8279000000000003e-05 2024-09-18T04:39:46.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.711 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.711 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:46.711 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.449953+0000", 2024-09-18T04:39:46.711 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003801979 2024-09-18T04:39:46.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.711 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.711 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:46.711 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.450018+0000", 2024-09-18T04:39:46.711 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4873e-05 2024-09-18T04:39:46.711 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:46.711 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:46.711 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:46.712 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.712 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.712 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:699 2.4 e57/17 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 57'17, mlcod=57'15)", 2024-09-18T04:39:46.712 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.566073+0000", 2024-09-18T04:39:46.712 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 55.080935384999997, 2024-09-18T04:39:46.712 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055398380000000001, 2024-09-18T04:39:46.712 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:46.712 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:46.712 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:46.712 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.712 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:46.712 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566073+0000", 2024-09-18T04:39:46.712 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:46.713 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.713 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.713 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:46.713 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566073+0000", 2024-09-18T04:39:46.713 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:46.713 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.713 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.713 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:46.713 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566071+0000", 2024-09-18T04:39:46.713 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:46.713 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.714 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.714 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:46.714 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566637+0000", 2024-09-18T04:39:46.714 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00056516699999999999 2024-09-18T04:39:46.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.714 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.714 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:46.714 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566668+0000", 2024-09-18T04:39:46.714 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1544999999999997e-05 2024-09-18T04:39:46.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.715 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.715 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:46.715 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566685+0000", 2024-09-18T04:39:46.715 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6857000000000001e-05 2024-09-18T04:39:46.715 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.715 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.715 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:46.715 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566816+0000", 2024-09-18T04:39:46.715 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013092799999999999 2024-09-18T04:39:46.715 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.715 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.715 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:46.715 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566875+0000", 2024-09-18T04:39:46.716 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9583999999999997e-05 2024-09-18T04:39:46.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.716 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.716 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:46.716 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.571567+0000", 2024-09-18T04:39:46.716 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.004691682 2024-09-18T04:39:46.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.716 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.716 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:46.716 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.571613+0000", 2024-09-18T04:39:46.716 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6125000000000003e-05 2024-09-18T04:39:46.716 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:46.716 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:46.717 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:46.717 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.717 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.717 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:98 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'27, mlcod=60'25)", 2024-09-18T04:39:46.717 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.847978+0000", 2024-09-18T04:39:46.717 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 45.799030825999999, 2024-09-18T04:39:46.717 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0044617709999999998, 2024-09-18T04:39:46.717 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:46.717 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:46.717 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:46.717 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.717 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:46.717 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.847978+0000", 2024-09-18T04:39:46.717 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:46.718 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.718 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.718 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:46.718 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.847978+0000", 2024-09-18T04:39:46.718 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:46.718 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.718 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.718 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:46.718 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.847977+0000", 2024-09-18T04:39:46.718 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:46.718 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.718 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.719 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:46.719 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849038+0000", 2024-09-18T04:39:46.719 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0010614820000000001 2024-09-18T04:39:46.719 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.719 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.719 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:46.719 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849065+0000", 2024-09-18T04:39:46.719 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7127999999999998e-05 2024-09-18T04:39:46.719 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.719 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.719 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:46.719 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849082+0000", 2024-09-18T04:39:46.719 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7170999999999999e-05 2024-09-18T04:39:46.720 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.720 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.720 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:46.720 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849188+0000", 2024-09-18T04:39:46.720 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010605800000000001 2024-09-18T04:39:46.720 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.720 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.720 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:46.720 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849233+0000", 2024-09-18T04:39:46.720 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4579000000000001e-05 2024-09-18T04:39:46.720 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.720 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.720 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:46.721 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.852370+0000", 2024-09-18T04:39:46.721 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00313751 2024-09-18T04:39:46.721 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.721 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.721 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:46.721 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.852440+0000", 2024-09-18T04:39:46.721 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9238000000000004e-05 2024-09-18T04:39:46.721 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:46.721 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:46.721 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:46.721 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.721 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.721 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:100 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'29, mlcod=64'27)", 2024-09-18T04:39:46.722 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.858766+0000", 2024-09-18T04:39:46.722 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 45.788242496999999, 2024-09-18T04:39:46.722 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0061964840000000004, 2024-09-18T04:39:46.722 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:46.722 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:46.722 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:46.722 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.722 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:46.722 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.858766+0000", 2024-09-18T04:39:46.722 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:46.722 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.722 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.722 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:46.722 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.858766+0000", 2024-09-18T04:39:46.723 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:46.723 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.723 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.723 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:46.723 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.858765+0000", 2024-09-18T04:39:46.723 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:46.723 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.723 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.723 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:46.723 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861612+0000", 2024-09-18T04:39:46.723 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028473169999999998 2024-09-18T04:39:46.723 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.723 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.724 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:46.724 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861643+0000", 2024-09-18T04:39:46.724 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1356000000000003e-05 2024-09-18T04:39:46.724 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.724 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.724 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:46.724 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861656+0000", 2024-09-18T04:39:46.724 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2899e-05 2024-09-18T04:39:46.724 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.724 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.724 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:46.724 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861773+0000", 2024-09-18T04:39:46.724 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011637999999999999 2024-09-18T04:39:46.725 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.725 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.725 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:46.725 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861808+0000", 2024-09-18T04:39:46.725 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5002999999999997e-05 2024-09-18T04:39:46.725 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.725 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.725 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:46.725 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.864876+0000", 2024-09-18T04:39:46.725 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030680719999999998 2024-09-18T04:39:46.725 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.725 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.725 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:46.726 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.864963+0000", 2024-09-18T04:39:46.726 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7114000000000007e-05 2024-09-18T04:39:46.726 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:46.726 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:46.726 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:46.726 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.726 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.726 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:102 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'31, mlcod=64'29)", 2024-09-18T04:39:46.726 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.872098+0000", 2024-09-18T04:39:46.726 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 45.774911240000002, 2024-09-18T04:39:46.726 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0064676450000000002, 2024-09-18T04:39:46.726 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:46.726 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:46.726 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:46.727 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.727 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:46.727 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.872098+0000", 2024-09-18T04:39:46.727 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:46.727 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.727 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.727 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:46.727 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.872098+0000", 2024-09-18T04:39:46.727 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:46.727 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.727 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.727 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:46.727 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.872096+0000", 2024-09-18T04:39:46.728 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:46.728 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.728 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.728 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:46.728 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875176+0000", 2024-09-18T04:39:46.728 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030799809999999999 2024-09-18T04:39:46.728 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.728 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.728 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:46.728 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875204+0000", 2024-09-18T04:39:46.728 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8025999999999999e-05 2024-09-18T04:39:46.728 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.728 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.729 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:46.729 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875216+0000", 2024-09-18T04:39:46.729 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2072999999999999e-05 2024-09-18T04:39:46.729 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.729 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.729 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:46.729 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875307+0000", 2024-09-18T04:39:46.729 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0373999999999999e-05 2024-09-18T04:39:46.729 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.729 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.729 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:46.730 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875349+0000", 2024-09-18T04:39:46.730 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2299000000000003e-05 2024-09-18T04:39:46.730 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.730 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.730 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:46.730 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.878496+0000", 2024-09-18T04:39:46.730 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0031468960000000002 2024-09-18T04:39:46.730 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.730 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.730 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:46.730 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.878565+0000", 2024-09-18T04:39:46.730 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9465e-05 2024-09-18T04:39:46.730 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:46.730 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:46.731 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:46.731 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.731 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.731 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:104 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'33, mlcod=64'31)", 2024-09-18T04:39:46.731 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.885100+0000", 2024-09-18T04:39:46.731 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 45.761908705000003, 2024-09-18T04:39:46.731 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0051888159999999997, 2024-09-18T04:39:46.731 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:46.731 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:46.731 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:46.731 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.731 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:46.731 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.885100+0000", 2024-09-18T04:39:46.731 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:46.732 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.732 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.732 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:46.732 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.885100+0000", 2024-09-18T04:39:46.732 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:46.732 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.732 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.732 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:46.732 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.885098+0000", 2024-09-18T04:39:46.732 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:46.732 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.732 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.732 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:46.733 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886809+0000", 2024-09-18T04:39:46.733 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017110459999999999 2024-09-18T04:39:46.733 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.733 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.733 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:46.733 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886842+0000", 2024-09-18T04:39:46.733 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2638999999999997e-05 2024-09-18T04:39:46.733 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.733 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.733 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:46.733 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886855+0000", 2024-09-18T04:39:46.733 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3380000000000001e-05 2024-09-18T04:39:46.734 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.734 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.734 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:46.734 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886939+0000", 2024-09-18T04:39:46.734 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3733999999999995e-05 2024-09-18T04:39:46.734 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.734 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.734 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:46.734 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886981+0000", 2024-09-18T04:39:46.734 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1956000000000003e-05 2024-09-18T04:39:46.734 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.734 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.734 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:46.734 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.890201+0000", 2024-09-18T04:39:46.735 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003219927 2024-09-18T04:39:46.735 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.735 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.735 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:46.735 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.890289+0000", 2024-09-18T04:39:46.735 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7997999999999993e-05 2024-09-18T04:39:46.735 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:46.735 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:46.735 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:46.735 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.735 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.735 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:145 2.4 e64/62 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 64'37, mlcod=64'35)", 2024-09-18T04:39:46.735 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:01.099398+0000", 2024-09-18T04:39:46.735 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 45.547611216999996, 2024-09-18T04:39:46.736 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0050078459999999998, 2024-09-18T04:39:46.736 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:46.736 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:46.736 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:46.736 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.736 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:46.736 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.099398+0000", 2024-09-18T04:39:46.736 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:46.736 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.736 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.736 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:46.736 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.099398+0000", 2024-09-18T04:39:46.736 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:46.737 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.737 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.737 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:46.737 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.099396+0000", 2024-09-18T04:39:46.737 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:46.737 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.737 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.737 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:46.737 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100199+0000", 2024-09-18T04:39:46.737 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00080338000000000002 2024-09-18T04:39:46.737 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.737 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.738 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:46.738 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100230+0000", 2024-09-18T04:39:46.738 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0716000000000002e-05 2024-09-18T04:39:46.738 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.738 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.738 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:46.738 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100243+0000", 2024-09-18T04:39:46.738 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3359e-05 2024-09-18T04:39:46.738 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.738 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.738 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:46.738 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100337+0000", 2024-09-18T04:39:46.738 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4387000000000003e-05 2024-09-18T04:39:46.739 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.739 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.739 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:46.739 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100396+0000", 2024-09-18T04:39:46.739 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8164000000000001e-05 2024-09-18T04:39:46.739 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.739 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.739 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:46.739 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.104309+0000", 2024-09-18T04:39:46.739 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0039137800000000004 2024-09-18T04:39:46.739 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.739 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.739 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:46.739 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.104405+0000", 2024-09-18T04:39:46.740 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6039999999999995e-05 2024-09-18T04:39:46.740 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:46.740 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:46.740 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:46.740 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.740 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.740 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:147 2.4 e64/62 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 64'39, mlcod=64'37)", 2024-09-18T04:39:46.740 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:01.110658+0000", 2024-09-18T04:39:46.740 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 45.536351128, 2024-09-18T04:39:46.740 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0046316910000000003, 2024-09-18T04:39:46.740 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:46.740 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:46.740 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:46.741 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.741 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:46.741 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.110658+0000", 2024-09-18T04:39:46.741 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:46.741 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.741 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.741 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:46.741 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.110658+0000", 2024-09-18T04:39:46.741 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:46.741 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.741 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.741 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:46.741 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.110656+0000", 2024-09-18T04:39:46.742 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:46.742 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.742 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.742 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:46.742 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111444+0000", 2024-09-18T04:39:46.742 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00078762400000000001 2024-09-18T04:39:46.742 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.742 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.742 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:46.742 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111474+0000", 2024-09-18T04:39:46.742 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0085000000000001e-05 2024-09-18T04:39:46.742 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.742 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.742 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:46.743 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111486+0000", 2024-09-18T04:39:46.743 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2645e-05 2024-09-18T04:39:46.743 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.743 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.743 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:46.743 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111581+0000", 2024-09-18T04:39:46.743 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4432000000000007e-05 2024-09-18T04:39:46.743 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.743 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.743 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:46.743 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111631+0000", 2024-09-18T04:39:46.743 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9901000000000002e-05 2024-09-18T04:39:46.743 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.743 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.744 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:46.744 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.115209+0000", 2024-09-18T04:39:46.744 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003577916 2024-09-18T04:39:46.744 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:46.744 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:46.744 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:46.744 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.115289+0000", 2024-09-18T04:39:46.744 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0704999999999994e-05 2024-09-18T04:39:46.744 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:46.745 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:46.745 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:46.745 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:46.745 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:46.745 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:46.746 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:39:46.747 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:46.744+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:46.847 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:46.844+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:46.855 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:46.855 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:46.855 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:39:46.855 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:46.865 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:39:46.948 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:46.940+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:46.967 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:46.967 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:46.968 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:39:46.968 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:39:46.968 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:46.978 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:39:47.048 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:47.040+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:47.087 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:47.087 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:39:47.087 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:39:47.087 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:39:47.087 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.087 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:33 2.1 e51/24 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head v 51'4, mlcod=51'2)", 2024-09-18T04:39:47.088 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:39:47.088 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 64.803040393000003, 2024-09-18T04:39:47.088 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018702106, 2024-09-18T04:39:47.088 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:47.088 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:47.088 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:47.088 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.088 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:47.088 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:39:47.088 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:47.088 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.088 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.088 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:47.088 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:39:47.089 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:47.089 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.089 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.089 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:47.089 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279197+0000", 2024-09-18T04:39:47.089 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:47.089 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.089 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.089 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:47.089 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282682+0000", 2024-09-18T04:39:47.089 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0034848689999999998 2024-09-18T04:39:47.089 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.089 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.090 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:47.090 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282709+0000", 2024-09-18T04:39:47.090 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7599000000000001e-05 2024-09-18T04:39:47.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.090 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.090 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:47.090 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282720+0000", 2024-09-18T04:39:47.090 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.092e-05 2024-09-18T04:39:47.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.090 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.090 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:47.090 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282791+0000", 2024-09-18T04:39:47.090 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1268999999999998e-05 2024-09-18T04:39:47.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.091 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.091 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:47.091 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282843+0000", 2024-09-18T04:39:47.091 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1771999999999998e-05 2024-09-18T04:39:47.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.091 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.091 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:47.091 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.297830+0000", 2024-09-18T04:39:47.091 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014987228999999999 2024-09-18T04:39:47.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.091 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.091 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:47.091 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.297901+0000", 2024-09-18T04:39:47.092 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0209999999999994e-05 2024-09-18T04:39:47.092 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:47.092 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:47.092 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:47.092 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.092 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.092 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:38 2.0 e51/17 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head v 51'10, mlcod=51'8)", 2024-09-18T04:39:47.092 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:39:47.092 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 64.801621963000002, 2024-09-18T04:39:47.092 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.023984472, 2024-09-18T04:39:47.092 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:47.092 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:47.093 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:47.093 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.093 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:47.093 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:39:47.093 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:47.093 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.093 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.093 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:47.093 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:39:47.093 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:47.093 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.093 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.094 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:47.094 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280614+0000", 2024-09-18T04:39:47.094 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:47.094 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.094 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.094 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:47.094 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293094+0000", 2024-09-18T04:39:47.094 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012479571 2024-09-18T04:39:47.094 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.094 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.094 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:47.094 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293111+0000", 2024-09-18T04:39:47.094 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7354999999999999e-05 2024-09-18T04:39:47.095 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.095 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.095 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:47.095 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293121+0000", 2024-09-18T04:39:47.095 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6979999999999993e-06 2024-09-18T04:39:47.095 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.095 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.095 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:47.095 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293180+0000", 2024-09-18T04:39:47.095 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9333000000000003e-05 2024-09-18T04:39:47.095 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.095 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.095 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:47.096 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293203+0000", 2024-09-18T04:39:47.096 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2633000000000002e-05 2024-09-18T04:39:47.096 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.096 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.096 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:47.096 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304534+0000", 2024-09-18T04:39:47.096 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011330761 2024-09-18T04:39:47.096 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.096 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.096 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:47.096 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304601+0000", 2024-09-18T04:39:47.096 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7657000000000006e-05 2024-09-18T04:39:47.096 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:47.097 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:47.097 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:47.097 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.097 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.097 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:46 2.6 2:7ba25245:::rbd_data.125fc09df6d1.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:47.097 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:39:47.097 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 64.774079162999996, 2024-09-18T04:39:47.097 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.035604595000000003, 2024-09-18T04:39:47.097 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:47.097 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:47.097 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:47.097 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:47.097 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:47.097 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 46 2024-09-18T04:39:47.098 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.098 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:47.098 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.098 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:47.098 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:39:47.098 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:47.098 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.098 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.098 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:47.098 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:39:47.098 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:47.098 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.098 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.098 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:47.098 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308158+0000", 2024-09-18T04:39:47.099 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:47.099 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.099 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.099 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:47.099 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313191+0000", 2024-09-18T04:39:47.099 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0050323929999999996 2024-09-18T04:39:47.099 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.099 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.099 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:47.099 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313210+0000", 2024-09-18T04:39:47.099 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9673e-05 2024-09-18T04:39:47.099 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.100 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.100 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:47.100 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313221+0000", 2024-09-18T04:39:47.100 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0444e-05 2024-09-18T04:39:47.100 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.100 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.100 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:47.100 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313294+0000", 2024-09-18T04:39:47.100 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2650000000000004e-05 2024-09-18T04:39:47.100 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.101 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.101 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:47.101 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313493+0000", 2024-09-18T04:39:47.101 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019968300000000001 2024-09-18T04:39:47.101 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.101 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.101 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:47.101 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313666+0000", 2024-09-18T04:39:47.101 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00017248599999999999 2024-09-18T04:39:47.101 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.101 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.101 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:47.101 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.323175+0000", 2024-09-18T04:39:47.101 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0095088819999999998 2024-09-18T04:39:47.102 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.102 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.102 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:47.102 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343651+0000", 2024-09-18T04:39:47.102 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020475892999999998 2024-09-18T04:39:47.102 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.102 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.102 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:47.102 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343715+0000", 2024-09-18T04:39:47.102 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4443999999999995e-05 2024-09-18T04:39:47.102 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.102 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.102 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:47.102 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343764+0000", 2024-09-18T04:39:47.103 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9280000000000003e-05 2024-09-18T04:39:47.103 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:47.103 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:47.103 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:47.103 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.103 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.103 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:47 2.4 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:47.103 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:39:47.103 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 64.768997126000002, 2024-09-18T04:39:47.103 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.032570693999999997, 2024-09-18T04:39:47.103 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:47.103 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:47.103 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:47.103 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:47.104 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:47.104 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 47 2024-09-18T04:39:47.104 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.104 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:47.104 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.104 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:47.104 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:39:47.104 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:47.104 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.104 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.104 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:47.104 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:39:47.104 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:47.105 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.105 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.105 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:47.105 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313240+0000", 2024-09-18T04:39:47.105 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:47.105 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.105 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.105 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:47.105 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325825+0000", 2024-09-18T04:39:47.105 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012584425999999999 2024-09-18T04:39:47.105 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.105 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.105 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:47.106 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325848+0000", 2024-09-18T04:39:47.106 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3025999999999999e-05 2024-09-18T04:39:47.106 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.106 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.106 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:47.106 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325863+0000", 2024-09-18T04:39:47.106 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5152e-05 2024-09-18T04:39:47.106 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.106 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.106 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:47.108 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325938+0000", 2024-09-18T04:39:47.108 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4982999999999997e-05 2024-09-18T04:39:47.108 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.108 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.108 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:47.108 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326090+0000", 2024-09-18T04:39:47.108 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015207900000000001 2024-09-18T04:39:47.108 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.108 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.108 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:47.108 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326242+0000", 2024-09-18T04:39:47.108 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015163500000000001 2024-09-18T04:39:47.108 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.109 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.109 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:47.109 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343015+0000", 2024-09-18T04:39:47.109 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016773414 2024-09-18T04:39:47.109 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.109 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.109 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:47.109 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345664+0000", 2024-09-18T04:39:47.109 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026487870000000001 2024-09-18T04:39:47.109 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.109 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.109 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:47.109 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345739+0000", 2024-09-18T04:39:47.109 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5018999999999994e-05 2024-09-18T04:39:47.110 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.110 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.110 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:47.110 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345812+0000", 2024-09-18T04:39:47.110 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3547000000000006e-05 2024-09-18T04:39:47.110 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:47.110 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:47.110 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:47.110 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.110 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.110 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:48 2.6 2:61db7a3a:::rbd_data.125fc09df6d1.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:47.110 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:39:47.110 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 64.756340453999996, 2024-09-18T04:39:47.110 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.025064388, 2024-09-18T04:39:47.111 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:47.111 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:47.111 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:47.111 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:47.111 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:47.111 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 48 2024-09-18T04:39:47.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.111 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:47.111 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.111 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:47.111 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:39:47.111 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:47.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.111 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.112 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:47.112 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:39:47.112 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:47.112 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.112 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.112 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:47.112 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325897+0000", 2024-09-18T04:39:47.112 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:47.112 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.112 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.112 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:47.112 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333697+0000", 2024-09-18T04:39:47.112 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0078002649999999998 2024-09-18T04:39:47.112 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.113 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.113 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:47.113 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333716+0000", 2024-09-18T04:39:47.113 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8910999999999999e-05 2024-09-18T04:39:47.113 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.113 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.113 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:47.113 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333728+0000", 2024-09-18T04:39:47.113 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1936000000000001e-05 2024-09-18T04:39:47.113 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.113 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.113 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:47.113 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333792+0000", 2024-09-18T04:39:47.113 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4010999999999994e-05 2024-09-18T04:39:47.114 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.114 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.114 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:47.114 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.334082+0000", 2024-09-18T04:39:47.114 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00028955799999999998 2024-09-18T04:39:47.114 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.114 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.114 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:47.114 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.334326+0000", 2024-09-18T04:39:47.114 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00024399099999999999 2024-09-18T04:39:47.114 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.114 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.114 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:47.114 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350745+0000", 2024-09-18T04:39:47.115 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016419474999999999 2024-09-18T04:39:47.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.115 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.115 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:47.115 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350808+0000", 2024-09-18T04:39:47.115 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3244999999999995e-05 2024-09-18T04:39:47.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.115 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.115 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:47.115 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350892+0000", 2024-09-18T04:39:47.115 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4074999999999997e-05 2024-09-18T04:39:47.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.115 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.115 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:47.116 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350963+0000", 2024-09-18T04:39:47.116 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0412000000000003e-05 2024-09-18T04:39:47.116 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:47.116 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:47.116 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:47.116 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.116 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.116 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:51 2.6 2:775d7b11:::rbd_data.125fc09df6d1.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:47.116 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:39:47.116 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 64.748488318, 2024-09-18T04:39:47.116 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.023824221, 2024-09-18T04:39:47.116 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:47.116 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:47.116 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:47.117 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:47.117 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:47.117 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 51 2024-09-18T04:39:47.117 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.117 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:47.117 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.117 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:47.117 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:39:47.117 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:47.117 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.117 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.117 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:47.117 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:39:47.118 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:47.118 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.118 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.118 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:47.118 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333749+0000", 2024-09-18T04:39:47.118 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:47.118 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.118 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.118 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:47.118 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340899+0000", 2024-09-18T04:39:47.118 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0071500340000000004 2024-09-18T04:39:47.118 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.118 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.118 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:47.119 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340929+0000", 2024-09-18T04:39:47.119 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0093000000000001e-05 2024-09-18T04:39:47.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.119 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.119 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:47.119 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340943+0000", 2024-09-18T04:39:47.119 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4019000000000001e-05 2024-09-18T04:39:47.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.119 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.119 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:47.119 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343834+0000", 2024-09-18T04:39:47.119 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028918120000000001 2024-09-18T04:39:47.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.119 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.120 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:47.120 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344020+0000", 2024-09-18T04:39:47.120 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000185892 2024-09-18T04:39:47.120 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.120 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.120 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:47.120 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344190+0000", 2024-09-18T04:39:47.120 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016946800000000001 2024-09-18T04:39:47.120 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.120 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.120 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:47.120 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.351291+0000", 2024-09-18T04:39:47.120 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.007100805 2024-09-18T04:39:47.120 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.121 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.121 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:47.121 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357470+0000", 2024-09-18T04:39:47.121 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0061797570000000001 2024-09-18T04:39:47.121 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.121 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.121 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:47.121 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357533+0000", 2024-09-18T04:39:47.121 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2906999999999999e-05 2024-09-18T04:39:47.121 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.121 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.121 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:47.121 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357575+0000", 2024-09-18T04:39:47.121 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1403999999999999e-05 2024-09-18T04:39:47.122 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:47.122 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:47.122 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:47.122 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.122 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.122 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:56 2.4 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:47.122 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:39:47.122 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 64.732443224999997, 2024-09-18T04:39:47.122 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.034295870999999999, 2024-09-18T04:39:47.122 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:47.122 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:47.122 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:47.122 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:47.122 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:47.123 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 56 2024-09-18T04:39:47.123 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.123 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:47.123 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.123 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:47.123 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:39:47.123 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:47.123 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.123 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.123 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:47.123 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:39:47.123 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:47.123 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.123 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.124 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:47.124 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349794+0000", 2024-09-18T04:39:47.124 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:47.124 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.124 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.124 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:47.124 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366632+0000", 2024-09-18T04:39:47.124 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01683755 2024-09-18T04:39:47.124 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.124 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.124 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:47.124 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366669+0000", 2024-09-18T04:39:47.124 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7351000000000002e-05 2024-09-18T04:39:47.125 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.125 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.125 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:47.125 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366686+0000", 2024-09-18T04:39:47.125 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6507999999999998e-05 2024-09-18T04:39:47.125 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.125 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.125 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:47.125 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366962+0000", 2024-09-18T04:39:47.125 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00027655900000000001 2024-09-18T04:39:47.125 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.125 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.125 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:47.126 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.367452+0000", 2024-09-18T04:39:47.126 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00049025299999999996 2024-09-18T04:39:47.126 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.126 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.126 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:47.126 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368113+0000", 2024-09-18T04:39:47.126 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00066093399999999996 2024-09-18T04:39:47.126 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.126 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.126 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:47.126 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383719+0000", 2024-09-18T04:39:47.126 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015606106999999999 2024-09-18T04:39:47.126 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.126 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.127 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:47.127 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384035+0000", 2024-09-18T04:39:47.127 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00031590399999999999 2024-09-18T04:39:47.127 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.127 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.127 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:47.127 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384067+0000", 2024-09-18T04:39:47.127 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.129e-05 2024-09-18T04:39:47.127 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.127 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.127 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:47.127 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384091+0000", 2024-09-18T04:39:47.127 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4788000000000001e-05 2024-09-18T04:39:47.127 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:47.128 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:47.128 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:47.128 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.128 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.128 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:57 2.0 e51/17 2:0cdd070e:::rbd_data.125fc09df6d1.000000000000000d:head v 51'21, mlcod=51'19)", 2024-09-18T04:39:47.128 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:39:47.128 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 64.724615737999997, 2024-09-18T04:39:47.128 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.025216311000000002, 2024-09-18T04:39:47.128 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:47.128 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:47.128 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:47.128 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.128 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:47.129 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:39:47.129 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:47.129 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.129 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.129 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:47.129 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:39:47.129 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:47.129 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.129 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.129 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:47.129 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:39:47.129 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:47.129 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.129 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.130 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:47.130 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363844+0000", 2024-09-18T04:39:47.130 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0062218450000000002 2024-09-18T04:39:47.130 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.130 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.130 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:47.130 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363879+0000", 2024-09-18T04:39:47.130 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4382999999999999e-05 2024-09-18T04:39:47.130 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.130 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.130 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:47.130 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363893+0000", 2024-09-18T04:39:47.130 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4348999999999999e-05 2024-09-18T04:39:47.131 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.131 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.131 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:47.131 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363977+0000", 2024-09-18T04:39:47.131 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3499999999999997e-05 2024-09-18T04:39:47.131 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.131 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.131 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:47.131 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364009+0000", 2024-09-18T04:39:47.131 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2292000000000002e-05 2024-09-18T04:39:47.131 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.131 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.131 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:47.131 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.382781+0000", 2024-09-18T04:39:47.132 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018771626 2024-09-18T04:39:47.132 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.132 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.132 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:47.132 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.382839+0000", 2024-09-18T04:39:47.132 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8758000000000002e-05 2024-09-18T04:39:47.132 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:47.132 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:47.132 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:47.132 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.132 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.132 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:59 2.0 e51/17 2:19c52cde:::rbd_data.125fc09df6d1.000000000000000f:head v 51'22, mlcod=51'19)", 2024-09-18T04:39:47.132 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:39:47.132 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 64.717955812, 2024-09-18T04:39:47.133 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.019497112, 2024-09-18T04:39:47.133 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:47.133 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:47.133 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:47.133 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.133 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:47.133 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:39:47.133 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:47.133 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.133 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.133 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:47.133 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:39:47.133 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:47.133 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.134 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.134 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:47.134 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364282+0000", 2024-09-18T04:39:47.134 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:47.134 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.134 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.134 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:47.134 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371823+0000", 2024-09-18T04:39:47.134 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0075411829999999999 2024-09-18T04:39:47.134 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.134 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.134 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:47.134 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371837+0000", 2024-09-18T04:39:47.134 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4336e-05 2024-09-18T04:39:47.134 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.135 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.135 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:47.135 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371846+0000", 2024-09-18T04:39:47.135 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9770000000000006e-06 2024-09-18T04:39:47.135 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.135 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.135 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:47.135 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.373566+0000", 2024-09-18T04:39:47.135 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017192539999999999 2024-09-18T04:39:47.135 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.135 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.135 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:47.135 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.373595+0000", 2024-09-18T04:39:47.135 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8975999999999998e-05 2024-09-18T04:39:47.136 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.136 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.136 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:47.136 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383740+0000", 2024-09-18T04:39:47.136 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01014525 2024-09-18T04:39:47.136 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.136 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.136 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:47.136 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383780+0000", 2024-09-18T04:39:47.136 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0275999999999999e-05 2024-09-18T04:39:47.136 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:47.136 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:47.137 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:47.137 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.137 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.137 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:34 2.6 2:66289171:ns::rbd_data.133e813ab5bc.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:47.137 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.524015+0000", 2024-09-18T04:39:47.137 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 64.558224202999995, 2024-09-18T04:39:47.137 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027582111999999999, 2024-09-18T04:39:47.137 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:47.137 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:47.137 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:47.137 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:47.137 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:47.137 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 34 2024-09-18T04:39:47.137 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.138 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:47.138 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.138 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:47.138 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524015+0000", 2024-09-18T04:39:47.138 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:47.138 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.138 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.138 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:47.138 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524015+0000", 2024-09-18T04:39:47.138 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:47.138 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.138 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.138 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:47.139 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524012+0000", 2024-09-18T04:39:47.139 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:47.139 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.139 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.139 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:47.139 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534956+0000", 2024-09-18T04:39:47.139 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010943403000000001 2024-09-18T04:39:47.139 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.139 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.139 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:47.139 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534978+0000", 2024-09-18T04:39:47.139 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1965999999999999e-05 2024-09-18T04:39:47.139 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.139 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.140 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:47.140 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534990+0000", 2024-09-18T04:39:47.140 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2065999999999999e-05 2024-09-18T04:39:47.140 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.140 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.140 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:47.140 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535065+0000", 2024-09-18T04:39:47.140 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5523000000000002e-05 2024-09-18T04:39:47.140 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.140 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.140 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:47.140 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535208+0000", 2024-09-18T04:39:47.140 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014309500000000001 2024-09-18T04:39:47.140 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.141 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.141 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:47.141 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535343+0000", 2024-09-18T04:39:47.141 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013437400000000001 2024-09-18T04:39:47.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.141 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.141 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:47.141 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.546576+0000", 2024-09-18T04:39:47.141 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011233425999999999 2024-09-18T04:39:47.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.141 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.141 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:47.141 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551498+0000", 2024-09-18T04:39:47.141 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0049219349999999997 2024-09-18T04:39:47.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.142 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.142 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:47.142 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551554+0000", 2024-09-18T04:39:47.142 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6400999999999998e-05 2024-09-18T04:39:47.142 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.142 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.142 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:47.142 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551597+0000", 2024-09-18T04:39:47.142 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2283000000000002e-05 2024-09-18T04:39:47.142 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:47.142 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:47.142 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:47.142 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.142 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.143 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:33 2.1 e51/24 2:8b40f1c5:ns::rbd_data.133e813ab5bc.0000000000000000:head v 51'7, mlcod=51'5)", 2024-09-18T04:39:47.143 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.531767+0000", 2024-09-18T04:39:47.143 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 64.550472286000002, 2024-09-18T04:39:47.143 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015370295000000001, 2024-09-18T04:39:47.143 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:47.143 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:47.143 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:47.143 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.143 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:47.143 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531767+0000", 2024-09-18T04:39:47.143 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:47.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.143 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.143 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:47.144 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531767+0000", 2024-09-18T04:39:47.144 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:47.144 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.144 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.144 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:47.144 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531765+0000", 2024-09-18T04:39:47.144 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:47.144 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.144 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.144 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:47.144 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534258+0000", 2024-09-18T04:39:47.144 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002493561 2024-09-18T04:39:47.144 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.144 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.145 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:47.145 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534288+0000", 2024-09-18T04:39:47.145 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.976e-05 2024-09-18T04:39:47.145 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.145 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.145 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:47.145 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534300+0000", 2024-09-18T04:39:47.145 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1928e-05 2024-09-18T04:39:47.145 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.145 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.145 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:47.145 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534367+0000", 2024-09-18T04:39:47.145 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7045999999999995e-05 2024-09-18T04:39:47.146 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.146 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.146 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:47.146 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534394+0000", 2024-09-18T04:39:47.146 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6903999999999999e-05 2024-09-18T04:39:47.146 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.146 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.146 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:47.146 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.547073+0000", 2024-09-18T04:39:47.146 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012679365 2024-09-18T04:39:47.146 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.146 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.146 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:47.146 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.547137+0000", 2024-09-18T04:39:47.147 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3691999999999999e-05 2024-09-18T04:39:47.147 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:47.147 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:47.147 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:47.147 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.147 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.147 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:39 2.4 2:304364d1:ns::rbd_data.133e813ab5bc.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:47.147 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.535013+0000", 2024-09-18T04:39:47.147 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 64.547225350999994, 2024-09-18T04:39:47.147 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027764487000000001, 2024-09-18T04:39:47.147 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:47.147 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:47.147 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:47.147 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:47.148 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:47.148 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 39 2024-09-18T04:39:47.148 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.148 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:47.148 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.148 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:47.148 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535013+0000", 2024-09-18T04:39:47.148 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:47.148 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.148 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.148 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:47.148 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535013+0000", 2024-09-18T04:39:47.148 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:47.148 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.149 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.149 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:47.149 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535012+0000", 2024-09-18T04:39:47.149 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:47.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.149 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.149 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:47.149 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545016+0000", 2024-09-18T04:39:47.149 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01000322 2024-09-18T04:39:47.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.149 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.149 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:47.149 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545035+0000", 2024-09-18T04:39:47.149 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9845999999999999e-05 2024-09-18T04:39:47.150 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.150 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.150 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:47.150 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545045+0000", 2024-09-18T04:39:47.150 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3640000000000005e-06 2024-09-18T04:39:47.150 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.150 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.150 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:47.150 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545152+0000", 2024-09-18T04:39:47.150 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000107025 2024-09-18T04:39:47.150 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.150 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.150 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:47.150 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545338+0000", 2024-09-18T04:39:47.151 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018643500000000001 2024-09-18T04:39:47.151 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.151 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.151 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:47.151 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545492+0000", 2024-09-18T04:39:47.151 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015393199999999999 2024-09-18T04:39:47.151 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.151 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.151 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:47.151 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.553549+0000", 2024-09-18T04:39:47.151 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0080569170000000002 2024-09-18T04:39:47.151 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.151 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.151 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:47.152 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562693+0000", 2024-09-18T04:39:47.152 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0091441609999999996 2024-09-18T04:39:47.152 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.152 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.152 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:47.152 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562748+0000", 2024-09-18T04:39:47.152 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5047000000000003e-05 2024-09-18T04:39:47.152 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.152 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.152 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:47.152 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562778+0000", 2024-09-18T04:39:47.152 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9652e-05 2024-09-18T04:39:47.152 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:47.152 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:47.153 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:47.153 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.153 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.153 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:40 2.6 2:784d6cd1:ns::rbd_data.133e813ab5bc.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:47.153 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.545062+0000", 2024-09-18T04:39:47.153 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 64.537176299999999, 2024-09-18T04:39:47.153 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.032771623, 2024-09-18T04:39:47.153 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:47.153 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:47.153 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:47.153 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:47.153 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:47.153 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 40 2024-09-18T04:39:47.153 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.154 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:47.154 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.154 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:47.154 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545062+0000", 2024-09-18T04:39:47.154 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:47.154 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.154 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.154 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:47.154 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545062+0000", 2024-09-18T04:39:47.154 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:47.154 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.154 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.154 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:47.154 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545061+0000", 2024-09-18T04:39:47.155 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:47.155 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.155 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.155 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:47.155 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552477+0000", 2024-09-18T04:39:47.155 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.007415914 2024-09-18T04:39:47.155 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.155 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.155 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:47.155 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552500+0000", 2024-09-18T04:39:47.155 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3546000000000001e-05 2024-09-18T04:39:47.155 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.155 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.155 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:47.156 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552515+0000", 2024-09-18T04:39:47.156 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4606999999999999e-05 2024-09-18T04:39:47.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.156 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.156 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:47.156 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552661+0000", 2024-09-18T04:39:47.156 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000146007 2024-09-18T04:39:47.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.156 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.156 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:47.156 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552876+0000", 2024-09-18T04:39:47.156 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000214618 2024-09-18T04:39:47.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.157 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.157 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:47.157 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.553061+0000", 2024-09-18T04:39:47.157 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018564100000000001 2024-09-18T04:39:47.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.157 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.157 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:47.157 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.565612+0000", 2024-09-18T04:39:47.157 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012550301999999999 2024-09-18T04:39:47.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.157 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.157 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:47.157 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577695+0000", 2024-09-18T04:39:47.157 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012083493000000001 2024-09-18T04:39:47.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.158 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.158 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:47.158 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577772+0000", 2024-09-18T04:39:47.158 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7158999999999998e-05 2024-09-18T04:39:47.158 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.158 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.158 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:47.158 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577834+0000", 2024-09-18T04:39:47.158 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1851000000000001e-05 2024-09-18T04:39:47.159 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:47.159 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:47.159 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:47.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.159 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.159 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:43 2.6 2:759f27b6:ns::rbd_data.133e813ab5bc.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:47.159 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.555868+0000", 2024-09-18T04:39:47.159 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 64.526370806000003, 2024-09-18T04:39:47.159 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.022332497999999999, 2024-09-18T04:39:47.159 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:47.159 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:47.159 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:47.159 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:47.159 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:47.160 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 43 2024-09-18T04:39:47.160 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.160 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:47.160 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.160 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:47.160 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555868+0000", 2024-09-18T04:39:47.160 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:47.160 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.160 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.160 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:47.160 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555868+0000", 2024-09-18T04:39:47.160 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:47.160 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.160 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.161 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:47.161 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555865+0000", 2024-09-18T04:39:47.161 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999971 2024-09-18T04:39:47.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.161 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.161 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:47.161 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561031+0000", 2024-09-18T04:39:47.161 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0051658770000000001 2024-09-18T04:39:47.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.161 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.161 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:47.161 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561048+0000", 2024-09-18T04:39:47.161 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6569e-05 2024-09-18T04:39:47.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.162 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.162 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:47.162 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561055+0000", 2024-09-18T04:39:47.162 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.328e-06 2024-09-18T04:39:47.162 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.162 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.162 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:47.162 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561114+0000", 2024-09-18T04:39:47.162 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9540000000000003e-05 2024-09-18T04:39:47.162 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.162 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.162 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:47.162 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561314+0000", 2024-09-18T04:39:47.162 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019965799999999999 2024-09-18T04:39:47.163 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.163 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.163 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:47.163 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561473+0000", 2024-09-18T04:39:47.163 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000158904 2024-09-18T04:39:47.163 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.163 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.163 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:47.163 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.567015+0000", 2024-09-18T04:39:47.163 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055420859999999999 2024-09-18T04:39:47.163 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.163 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.163 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:47.163 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577982+0000", 2024-09-18T04:39:47.164 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010966872000000001 2024-09-18T04:39:47.164 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.164 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.164 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:47.164 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578162+0000", 2024-09-18T04:39:47.164 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018050999999999999 2024-09-18T04:39:47.164 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.164 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.164 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:47.164 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578200+0000", 2024-09-18T04:39:47.164 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8000999999999998e-05 2024-09-18T04:39:47.164 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:47.164 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:47.164 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:47.165 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.165 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.165 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:48 2.6 2:7c225365:ns::rbd_data.133e813ab5bc.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:47.165 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:39:47.165 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 64.518571582999996, 2024-09-18T04:39:47.165 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.024576377, 2024-09-18T04:39:47.165 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:47.165 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:47.165 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:47.165 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:47.165 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:47.165 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 48 2024-09-18T04:39:47.165 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.165 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:47.166 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.166 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:47.166 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:39:47.166 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:47.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.166 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.166 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:47.166 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:39:47.166 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:47.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.166 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.166 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:47.166 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.563666+0000", 2024-09-18T04:39:47.166 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:47.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.167 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.167 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:47.167 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570331+0000", 2024-09-18T04:39:47.167 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0066653329999999998 2024-09-18T04:39:47.167 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.167 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.167 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:47.167 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570348+0000", 2024-09-18T04:39:47.167 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6507e-05 2024-09-18T04:39:47.167 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.167 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.167 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:47.167 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570355+0000", 2024-09-18T04:39:47.167 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2749999999999998e-06 2024-09-18T04:39:47.168 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.168 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.168 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:47.168 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570396+0000", 2024-09-18T04:39:47.168 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0809000000000002e-05 2024-09-18T04:39:47.168 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.168 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.168 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:47.168 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570571+0000", 2024-09-18T04:39:47.168 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000174823 2024-09-18T04:39:47.168 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.168 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.168 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:47.169 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570722+0000", 2024-09-18T04:39:47.169 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015154599999999999 2024-09-18T04:39:47.169 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.169 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.169 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:47.169 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579756+0000", 2024-09-18T04:39:47.169 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0090337000000000004 2024-09-18T04:39:47.169 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.169 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.169 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:47.169 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588093+0000", 2024-09-18T04:39:47.169 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.008336553 2024-09-18T04:39:47.169 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.170 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.170 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:47.170 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588173+0000", 2024-09-18T04:39:47.170 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0333999999999994e-05 2024-09-18T04:39:47.170 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.170 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.170 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:47.170 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588244+0000", 2024-09-18T04:39:47.170 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0573000000000003e-05 2024-09-18T04:39:47.170 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:47.170 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:47.170 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:47.170 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.170 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.171 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:49 2.4 2:20f0b0c9:ns::rbd_data.133e813ab5bc.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:47.171 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.570370+0000", 2024-09-18T04:39:47.171 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 64.511869255999997, 2024-09-18T04:39:47.171 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.029633981, 2024-09-18T04:39:47.171 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:47.171 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:47.171 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:47.171 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:47.171 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:47.171 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 49 2024-09-18T04:39:47.171 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.171 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:47.171 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.171 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:47.172 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570370+0000", 2024-09-18T04:39:47.172 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:47.172 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.172 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.172 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:47.172 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570370+0000", 2024-09-18T04:39:47.172 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:47.172 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.172 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.172 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:47.172 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570369+0000", 2024-09-18T04:39:47.172 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:47.172 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.172 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.173 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:47.173 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579090+0000", 2024-09-18T04:39:47.173 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0087212339999999996 2024-09-18T04:39:47.173 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.173 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.173 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:47.173 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579109+0000", 2024-09-18T04:39:47.173 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9120999999999999e-05 2024-09-18T04:39:47.173 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.173 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.173 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:47.173 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579123+0000", 2024-09-18T04:39:47.173 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4382000000000001e-05 2024-09-18T04:39:47.173 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.174 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.174 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:47.174 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579221+0000", 2024-09-18T04:39:47.174 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.7503999999999993e-05 2024-09-18T04:39:47.174 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.174 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.174 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:47.174 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579553+0000", 2024-09-18T04:39:47.174 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00033254200000000002 2024-09-18T04:39:47.174 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.174 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.174 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:47.174 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579817+0000", 2024-09-18T04:39:47.175 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00026317099999999998 2024-09-18T04:39:47.175 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.175 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.175 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:47.175 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.599905+0000", 2024-09-18T04:39:47.175 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020088575000000001 2024-09-18T04:39:47.175 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.175 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.175 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:47.175 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.599923+0000", 2024-09-18T04:39:47.175 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7425999999999999e-05 2024-09-18T04:39:47.175 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.175 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.175 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:47.176 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.599973+0000", 2024-09-18T04:39:47.176 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0649999999999998e-05 2024-09-18T04:39:47.176 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.176 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.176 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:47.176 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.600004+0000", 2024-09-18T04:39:47.176 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0361999999999998e-05 2024-09-18T04:39:47.176 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:47.176 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:47.176 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:47.176 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.176 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.176 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:50 2.1 e51/24 2:8c706374:ns::rbd_data.133e813ab5bc.000000000000000a:head v 51'8, mlcod=51'6)", 2024-09-18T04:39:47.177 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.577029+0000", 2024-09-18T04:39:47.177 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 64.505209884999999, 2024-09-18T04:39:47.177 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.017362855, 2024-09-18T04:39:47.177 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:47.177 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:47.177 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:47.177 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.177 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:47.177 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577029+0000", 2024-09-18T04:39:47.177 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:47.177 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.177 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.177 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:47.177 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577029+0000", 2024-09-18T04:39:47.178 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:47.178 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.178 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.178 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:47.178 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577027+0000", 2024-09-18T04:39:47.178 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:47.178 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.178 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.178 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:47.178 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578880+0000", 2024-09-18T04:39:47.178 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0018532080000000001 2024-09-18T04:39:47.178 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.178 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.178 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:47.179 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578906+0000", 2024-09-18T04:39:47.179 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5573999999999999e-05 2024-09-18T04:39:47.179 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.179 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.179 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:47.179 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578920+0000", 2024-09-18T04:39:47.179 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4483e-05 2024-09-18T04:39:47.179 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.179 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.179 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:47.179 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578983+0000", 2024-09-18T04:39:47.179 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2248000000000004e-05 2024-09-18T04:39:47.179 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.179 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.179 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:47.180 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579016+0000", 2024-09-18T04:39:47.180 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3364000000000001e-05 2024-09-18T04:39:47.180 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.180 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.180 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:47.180 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594342+0000", 2024-09-18T04:39:47.180 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01532564 2024-09-18T04:39:47.180 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.180 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.180 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:47.180 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594392+0000", 2024-09-18T04:39:47.180 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0068999999999998e-05 2024-09-18T04:39:47.180 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:47.180 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:47.181 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:47.181 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.181 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.181 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:45 2.0 e51/17 2:0fbfa3dc:ns::rbd_data.133e813ab5bc.0000000000000007:head v 51'24, mlcod=51'22)", 2024-09-18T04:39:47.181 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.581892+0000", 2024-09-18T04:39:47.181 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 64.500347212999998, 2024-09-18T04:39:47.181 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015646006, 2024-09-18T04:39:47.181 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:47.181 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:47.181 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:47.181 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.181 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:47.182 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581892+0000", 2024-09-18T04:39:47.182 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:47.182 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.182 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.182 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:47.182 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581892+0000", 2024-09-18T04:39:47.182 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:47.182 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.182 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.182 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:47.182 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581890+0000", 2024-09-18T04:39:47.182 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:47.182 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.183 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.183 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:47.183 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586188+0000", 2024-09-18T04:39:47.183 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0042985769999999996 2024-09-18T04:39:47.183 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.183 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.183 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:47.183 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586210+0000", 2024-09-18T04:39:47.183 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1069000000000001e-05 2024-09-18T04:39:47.183 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.183 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.183 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:47.183 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586221+0000", 2024-09-18T04:39:47.183 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1287000000000001e-05 2024-09-18T04:39:47.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.184 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.184 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:47.184 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586313+0000", 2024-09-18T04:39:47.184 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1935999999999995e-05 2024-09-18T04:39:47.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.184 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.184 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:47.184 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586368+0000", 2024-09-18T04:39:47.184 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5578000000000002e-05 2024-09-18T04:39:47.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.184 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.184 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:47.184 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.597473+0000", 2024-09-18T04:39:47.185 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011105139999999999 2024-09-18T04:39:47.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.185 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.185 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:47.185 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.597538+0000", 2024-09-18T04:39:47.185 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4133000000000004e-05 2024-09-18T04:39:47.185 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:47.185 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:47.185 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:47.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.185 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.185 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:53 2.4 2:39f7f174:ns::rbd_data.133e813ab5bc.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:47.185 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.584263+0000", 2024-09-18T04:39:47.186 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 64.497975339000007, 2024-09-18T04:39:47.186 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018701180000000001, 2024-09-18T04:39:47.186 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:47.186 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:47.186 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:47.186 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:47.186 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:47.186 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 53 2024-09-18T04:39:47.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.186 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:47.186 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.186 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:47.186 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584263+0000", 2024-09-18T04:39:47.186 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:47.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.187 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.187 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:47.187 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584263+0000", 2024-09-18T04:39:47.187 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:47.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.187 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.187 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:47.187 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584262+0000", 2024-09-18T04:39:47.187 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:47.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.187 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.187 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:47.188 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.591617+0000", 2024-09-18T04:39:47.188 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0073550600000000001 2024-09-18T04:39:47.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.188 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.188 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:47.188 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.591637+0000", 2024-09-18T04:39:47.188 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0432999999999999e-05 2024-09-18T04:39:47.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.188 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.188 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:47.188 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.591646+0000", 2024-09-18T04:39:47.188 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7849999999999997e-06 2024-09-18T04:39:47.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.188 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.189 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:47.189 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593478+0000", 2024-09-18T04:39:47.189 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001831495 2024-09-18T04:39:47.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.189 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.189 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:47.189 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593661+0000", 2024-09-18T04:39:47.189 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018364999999999999 2024-09-18T04:39:47.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.189 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.189 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:47.189 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593834+0000", 2024-09-18T04:39:47.189 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000173065 2024-09-18T04:39:47.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.190 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.190 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:47.190 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.600670+0000", 2024-09-18T04:39:47.190 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0068354970000000003 2024-09-18T04:39:47.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.190 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.190 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:47.190 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602851+0000", 2024-09-18T04:39:47.190 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002181653 2024-09-18T04:39:47.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.190 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.190 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:47.190 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602930+0000", 2024-09-18T04:39:47.190 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8045000000000002e-05 2024-09-18T04:39:47.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.191 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.191 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:47.191 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602965+0000", 2024-09-18T04:39:47.191 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5102999999999999e-05 2024-09-18T04:39:47.191 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:47.191 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:47.191 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:47.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.191 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.191 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:56 2.6 2:72453c3e:ns::rbd_data.133e813ab5bc.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:47.191 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.595988+0000", 2024-09-18T04:39:47.191 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 64.486250936999994, 2024-09-18T04:39:47.191 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.030296172, 2024-09-18T04:39:47.192 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:47.192 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:47.192 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:47.192 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:47.192 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:47.192 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 56 2024-09-18T04:39:47.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.192 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:47.192 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.192 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:47.192 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.595988+0000", 2024-09-18T04:39:47.192 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:47.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.193 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.193 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:47.193 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.595988+0000", 2024-09-18T04:39:47.193 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:47.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.193 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.193 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:47.193 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.595986+0000", 2024-09-18T04:39:47.193 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:47.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.193 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.193 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:47.193 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605262+0000", 2024-09-18T04:39:47.193 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0092767240000000001 2024-09-18T04:39:47.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.194 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.194 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:47.194 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605282+0000", 2024-09-18T04:39:47.194 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9307e-05 2024-09-18T04:39:47.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.194 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.194 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:47.194 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605292+0000", 2024-09-18T04:39:47.194 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9510000000000001e-06 2024-09-18T04:39:47.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.194 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.194 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:47.194 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605369+0000", 2024-09-18T04:39:47.195 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7846e-05 2024-09-18T04:39:47.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.195 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.195 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:47.195 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605668+0000", 2024-09-18T04:39:47.195 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00029896099999999999 2024-09-18T04:39:47.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.195 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.195 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:47.195 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605906+0000", 2024-09-18T04:39:47.195 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000237758 2024-09-18T04:39:47.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.195 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.195 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:47.195 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.613532+0000", 2024-09-18T04:39:47.196 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0076260590000000001 2024-09-18T04:39:47.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.196 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.196 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:47.196 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.626116+0000", 2024-09-18T04:39:47.196 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012584092999999999 2024-09-18T04:39:47.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.196 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.196 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:47.196 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.626216+0000", 2024-09-18T04:39:47.196 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9630999999999995e-05 2024-09-18T04:39:47.197 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:47.197 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:47.197 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:47.197 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.626284+0000", 2024-09-18T04:39:47.197 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8034999999999995e-05 2024-09-18T04:39:47.197 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:47.197 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:47.197 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:47.197 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:47.197 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:47.197 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:47.199 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:39:47.199 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:47.196+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:47.300 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:47.292+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:47.314 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:47.315 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:47.315 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:39:47.315 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:47.325 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:39:47.400 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:47.392+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:47.432 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:47.432 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:47.432 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:39:47.432 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:39:47.432 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:47.442 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:39:47.498 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T04:39:47.501 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:47.496+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:47.597 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:47.597 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:39:47.597 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:39:47.597 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:39:47.597 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.597 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:39:47.598 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:39:47.598 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 139.80441259099999, 2024-09-18T04:39:47.598 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017399218000000001, 2024-09-18T04:39:47.598 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:47.598 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:47.598 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:47.598 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:39:47.598 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:39:47.598 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 114 2024-09-18T04:39:47.598 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.598 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:47.598 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.598 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:47.599 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:39:47.599 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:47.599 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.599 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.599 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:47.599 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:39:47.599 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:47.599 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.599 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.599 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:47.599 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786900+0000", 2024-09-18T04:39:47.599 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:47.599 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.599 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.600 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:47.600 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786908+0000", 2024-09-18T04:39:47.600 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8199999999999997e-06 2024-09-18T04:39:47.600 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.600 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.600 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:47.600 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786925+0000", 2024-09-18T04:39:47.600 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6849000000000001e-05 2024-09-18T04:39:47.600 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.600 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.600 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:47.600 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786934+0000", 2024-09-18T04:39:47.601 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6299999999999993e-06 2024-09-18T04:39:47.601 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.601 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.601 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:47.601 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800926+0000", 2024-09-18T04:39:47.601 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013991632 2024-09-18T04:39:47.601 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.601 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.601 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:47.601 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801060+0000", 2024-09-18T04:39:47.601 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013378199999999999 2024-09-18T04:39:47.601 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.601 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.602 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:47.602 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801166+0000", 2024-09-18T04:39:47.602 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010607099999999999 2024-09-18T04:39:47.602 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.602 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.602 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:47.602 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803026+0000", 2024-09-18T04:39:47.602 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001860469 2024-09-18T04:39:47.602 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.602 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.602 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:47.602 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804245+0000", 2024-09-18T04:39:47.602 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0012185900000000001 2024-09-18T04:39:47.602 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.603 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.603 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:47.603 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804278+0000", 2024-09-18T04:39:47.603 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3272000000000003e-05 2024-09-18T04:39:47.603 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.603 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.603 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:47.603 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804300+0000", 2024-09-18T04:39:47.603 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2266e-05 2024-09-18T04:39:47.603 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:47.603 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:47.603 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:47.603 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.604 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.604 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:39:47.604 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:39:47.604 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 139.80349969299999, 2024-09-18T04:39:47.604 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017521662, 2024-09-18T04:39:47.604 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:47.604 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:47.604 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:47.604 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:39:47.604 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:39:47.604 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 116 2024-09-18T04:39:47.604 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.604 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:47.605 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.605 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:47.605 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:39:47.605 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:47.605 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.605 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.605 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:47.605 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:39:47.605 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:47.605 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.605 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.605 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:47.605 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787813+0000", 2024-09-18T04:39:47.606 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:47.606 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.606 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.606 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:47.606 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787824+0000", 2024-09-18T04:39:47.606 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1157e-05 2024-09-18T04:39:47.606 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.606 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.606 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:47.606 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787840+0000", 2024-09-18T04:39:47.606 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6269e-05 2024-09-18T04:39:47.606 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.606 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.607 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:47.607 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787853+0000", 2024-09-18T04:39:47.607 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3156000000000001e-05 2024-09-18T04:39:47.607 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.607 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.607 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:47.607 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803110+0000", 2024-09-18T04:39:47.607 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015256736 2024-09-18T04:39:47.607 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.607 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.607 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:47.607 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803197+0000", 2024-09-18T04:39:47.607 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7195999999999997e-05 2024-09-18T04:39:47.607 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.607 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.608 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:47.608 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803308+0000", 2024-09-18T04:39:47.608 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111005 2024-09-18T04:39:47.608 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.608 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.608 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:47.608 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804316+0000", 2024-09-18T04:39:47.608 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010073949999999999 2024-09-18T04:39:47.608 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.608 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.608 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:47.608 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805232+0000", 2024-09-18T04:39:47.608 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091636300000000001 2024-09-18T04:39:47.608 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.609 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.609 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:47.609 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805277+0000", 2024-09-18T04:39:47.609 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4864000000000003e-05 2024-09-18T04:39:47.609 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.609 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.609 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:47.609 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805336+0000", 2024-09-18T04:39:47.609 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8968000000000002e-05 2024-09-18T04:39:47.609 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:47.609 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:47.609 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:47.609 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.610 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.610 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:39:47.610 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:39:47.610 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 139.80344763799999, 2024-09-18T04:39:47.610 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017983453, 2024-09-18T04:39:47.610 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:47.610 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:47.610 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:47.610 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:39:47.610 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:39:47.610 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 117 2024-09-18T04:39:47.610 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.610 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:47.610 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.611 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:47.611 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:39:47.611 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:47.611 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.611 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.611 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:47.611 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:39:47.611 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:47.611 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.611 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.611 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:47.611 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787865+0000", 2024-09-18T04:39:47.611 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:47.611 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.612 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.612 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:47.612 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787889+0000", 2024-09-18T04:39:47.612 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4244999999999999e-05 2024-09-18T04:39:47.612 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.612 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.612 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:47.612 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787905+0000", 2024-09-18T04:39:47.612 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5739999999999998e-05 2024-09-18T04:39:47.612 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.612 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.612 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:47.612 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787918+0000", 2024-09-18T04:39:47.612 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.252e-05 2024-09-18T04:39:47.613 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.613 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.613 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:47.613 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803693+0000", 2024-09-18T04:39:47.613 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015774872999999998 2024-09-18T04:39:47.613 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.613 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.613 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:47.613 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803775+0000", 2024-09-18T04:39:47.613 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2761000000000003e-05 2024-09-18T04:39:47.613 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.613 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.613 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:47.614 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803887+0000", 2024-09-18T04:39:47.614 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111708 2024-09-18T04:39:47.614 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.614 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.614 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:47.614 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804823+0000", 2024-09-18T04:39:47.614 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00093625100000000003 2024-09-18T04:39:47.614 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.614 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.614 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:47.614 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805804+0000", 2024-09-18T04:39:47.614 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00098114699999999992 2024-09-18T04:39:47.615 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.615 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.615 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:47.615 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805831+0000", 2024-09-18T04:39:47.615 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6401999999999999e-05 2024-09-18T04:39:47.615 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.615 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.615 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:47.615 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805850+0000", 2024-09-18T04:39:47.615 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8862e-05 2024-09-18T04:39:47.615 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:47.615 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:47.615 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:47.615 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.616 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.616 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:36 2.7 2:fe1b02ad:::rbd_data.125fc09df6d1.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:47.616 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:39:47.616 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 65.325591528000004, 2024-09-18T04:39:47.616 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.040387804999999999, 2024-09-18T04:39:47.616 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:47.616 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:47.616 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:47.616 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:47.616 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:47.616 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 36 2024-09-18T04:39:47.616 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.616 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:47.617 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.617 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:47.617 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:39:47.617 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:47.617 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.617 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.617 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:47.617 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:39:47.617 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:47.617 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.618 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.618 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:47.618 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265720+0000", 2024-09-18T04:39:47.618 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:47.618 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.618 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.618 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:47.618 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272590+0000", 2024-09-18T04:39:47.618 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.006870508 2024-09-18T04:39:47.618 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.618 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.618 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:47.618 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272623+0000", 2024-09-18T04:39:47.618 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2716000000000003e-05 2024-09-18T04:39:47.619 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.619 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.619 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:47.619 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272637+0000", 2024-09-18T04:39:47.619 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4175999999999999e-05 2024-09-18T04:39:47.619 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.619 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.619 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:47.619 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272779+0000", 2024-09-18T04:39:47.619 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000141392 2024-09-18T04:39:47.619 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.619 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.619 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:47.619 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273031+0000", 2024-09-18T04:39:47.619 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00025249099999999998 2024-09-18T04:39:47.620 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.620 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.620 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:39:47.620 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273219+0000", 2024-09-18T04:39:47.620 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000187531 2024-09-18T04:39:47.620 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.620 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.620 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:47.620 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.302806+0000", 2024-09-18T04:39:47.620 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.029587553999999999 2024-09-18T04:39:47.620 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.620 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.620 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:47.621 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306004+0000", 2024-09-18T04:39:47.621 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0031982629999999998 2024-09-18T04:39:47.621 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.621 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.621 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:47.621 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306071+0000", 2024-09-18T04:39:47.621 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6829999999999995e-05 2024-09-18T04:39:47.621 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.621 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.621 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:47.621 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306110+0000", 2024-09-18T04:39:47.621 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8874000000000001e-05 2024-09-18T04:39:47.621 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:47.621 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:47.622 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:47.622 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.622 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.622 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:38 2.0 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:47.622 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:39:47.622 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 65.318637991000003, 2024-09-18T04:39:47.622 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.033754348000000003, 2024-09-18T04:39:47.622 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:47.622 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:47.622 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:47.622 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:47.622 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:47.622 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 38 2024-09-18T04:39:47.622 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.623 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:47.623 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.623 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:47.623 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:39:47.623 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:47.623 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.623 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.623 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:47.623 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:39:47.623 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:47.623 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.623 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.624 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:47.624 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272673+0000", 2024-09-18T04:39:47.624 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:47.624 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.624 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.624 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:47.624 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277166+0000", 2024-09-18T04:39:47.624 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044922249999999999 2024-09-18T04:39:47.624 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.624 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.624 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:47.624 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277189+0000", 2024-09-18T04:39:47.624 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3283000000000001e-05 2024-09-18T04:39:47.625 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.625 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.625 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:47.625 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277205+0000", 2024-09-18T04:39:47.625 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6305000000000001e-05 2024-09-18T04:39:47.625 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.625 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.625 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:47.625 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277290+0000", 2024-09-18T04:39:47.625 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5135000000000004e-05 2024-09-18T04:39:47.625 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.625 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.625 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:47.625 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277531+0000", 2024-09-18T04:39:47.626 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00024014699999999999 2024-09-18T04:39:47.626 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.626 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.626 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:47.626 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277793+0000", 2024-09-18T04:39:47.626 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00026224499999999998 2024-09-18T04:39:47.626 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.626 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.626 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:47.626 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303990+0000", 2024-09-18T04:39:47.626 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.026197295999999998 2024-09-18T04:39:47.626 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.626 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.627 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:47.627 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306340+0000", 2024-09-18T04:39:47.627 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.002349524 2024-09-18T04:39:47.627 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.627 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.627 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:47.627 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306394+0000", 2024-09-18T04:39:47.627 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3934999999999998e-05 2024-09-18T04:39:47.627 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.627 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.627 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:47.627 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306430+0000", 2024-09-18T04:39:47.627 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6690000000000003e-05 2024-09-18T04:39:47.627 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:47.628 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:47.628 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:47.628 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.628 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.628 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:39 2.7 2:e9c909c5:::rbd_data.125fc09df6d1.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:47.628 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:39:47.628 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 65.314078386000006, 2024-09-18T04:39:47.628 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.029110673, 2024-09-18T04:39:47.628 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:47.628 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:47.628 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:47.628 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:47.628 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:47.629 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 39 2024-09-18T04:39:47.629 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.629 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:47.629 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.629 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:47.629 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:39:47.629 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:47.629 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.629 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.629 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:47.629 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:39:47.629 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:47.629 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.629 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.630 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:47.630 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277234+0000", 2024-09-18T04:39:47.630 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:47.630 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.630 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.630 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:47.630 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281351+0000", 2024-09-18T04:39:47.630 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0041170920000000001 2024-09-18T04:39:47.630 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.630 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.630 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:47.630 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281375+0000", 2024-09-18T04:39:47.630 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.446e-05 2024-09-18T04:39:47.630 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.631 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.631 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:47.631 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281390+0000", 2024-09-18T04:39:47.631 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4309e-05 2024-09-18T04:39:47.631 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.631 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.631 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:47.631 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.282306+0000", 2024-09-18T04:39:47.631 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091662000000000004 2024-09-18T04:39:47.631 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.631 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.631 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:47.631 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.282613+0000", 2024-09-18T04:39:47.631 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00030663799999999997 2024-09-18T04:39:47.632 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.632 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.632 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:39:47.632 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283001+0000", 2024-09-18T04:39:47.632 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00038801599999999998 2024-09-18T04:39:47.632 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.632 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.632 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:47.632 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306239+0000", 2024-09-18T04:39:47.632 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023238466999999999 2024-09-18T04:39:47.632 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.632 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.632 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:47.633 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306259+0000", 2024-09-18T04:39:47.633 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9837999999999999e-05 2024-09-18T04:39:47.633 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.633 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.633 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:47.633 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306299+0000", 2024-09-18T04:39:47.633 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0194000000000002e-05 2024-09-18T04:39:47.633 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.633 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.633 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:47.633 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306346+0000", 2024-09-18T04:39:47.633 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6845000000000003e-05 2024-09-18T04:39:47.633 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:47.634 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:47.634 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:47.634 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.634 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.634 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:46 2.6 e51/17 2:7ba25245:::rbd_data.125fc09df6d1.0000000000000006:head v 51'2, mlcod=0'0)", 2024-09-18T04:39:47.634 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:39:47.634 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 65.278157254000007, 2024-09-18T04:39:47.634 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019843098, 2024-09-18T04:39:47.634 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:47.634 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:47.634 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:47.634 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.634 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:47.635 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:39:47.635 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:47.635 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.635 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.635 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:47.635 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:39:47.635 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:47.635 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.635 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.635 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:47.635 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313156+0000", 2024-09-18T04:39:47.635 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:47.635 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.636 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.636 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:47.636 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320144+0000", 2024-09-18T04:39:47.636 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.006988345 2024-09-18T04:39:47.636 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.636 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.636 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:47.636 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320175+0000", 2024-09-18T04:39:47.636 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.0861999999999997e-05 2024-09-18T04:39:47.636 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.636 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.636 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:47.636 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320188+0000", 2024-09-18T04:39:47.637 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2714999999999999e-05 2024-09-18T04:39:47.637 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.637 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.637 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:47.637 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320267+0000", 2024-09-18T04:39:47.637 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9586999999999995e-05 2024-09-18T04:39:47.637 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.637 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.637 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:47.637 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320311+0000", 2024-09-18T04:39:47.637 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3529000000000003e-05 2024-09-18T04:39:47.637 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.637 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.637 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:47.638 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.332949+0000", 2024-09-18T04:39:47.638 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012638267999999999 2024-09-18T04:39:47.638 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.638 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.638 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:47.638 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.333000+0000", 2024-09-18T04:39:47.638 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0736999999999999e-05 2024-09-18T04:39:47.638 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:47.638 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:47.638 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:47.638 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.638 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.638 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:57 2.0 2:0cdd070e:::rbd_data.125fc09df6d1.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:47.638 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:39:47.639 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 65.242299678999998, 2024-09-18T04:39:47.639 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.033329974999999998, 2024-09-18T04:39:47.639 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:47.639 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:47.639 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:47.639 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:47.639 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:47.639 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 57 2024-09-18T04:39:47.639 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.639 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:47.639 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.639 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:47.639 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:39:47.640 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:47.640 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.640 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.640 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:47.640 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:39:47.640 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:47.640 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.640 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.640 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:47.640 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349013+0000", 2024-09-18T04:39:47.640 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:47.640 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.640 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.641 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:47.641 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356138+0000", 2024-09-18T04:39:47.641 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071258010000000002 2024-09-18T04:39:47.641 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.641 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.641 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:47.641 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356168+0000", 2024-09-18T04:39:47.641 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9679000000000002e-05 2024-09-18T04:39:47.641 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.641 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.641 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:47.641 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356181+0000", 2024-09-18T04:39:47.641 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3164e-05 2024-09-18T04:39:47.641 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.642 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.642 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:47.642 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356254+0000", 2024-09-18T04:39:47.642 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3182999999999994e-05 2024-09-18T04:39:47.642 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.642 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.642 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:47.642 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356453+0000", 2024-09-18T04:39:47.642 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019876599999999999 2024-09-18T04:39:47.642 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.642 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.642 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:47.642 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356627+0000", 2024-09-18T04:39:47.642 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017340199999999999 2024-09-18T04:39:47.642 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.643 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.643 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:47.643 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.369200+0000", 2024-09-18T04:39:47.643 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012573302 2024-09-18T04:39:47.643 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.643 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.643 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:47.643 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382262+0000", 2024-09-18T04:39:47.643 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013062549 2024-09-18T04:39:47.643 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.643 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.643 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:47.643 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382310+0000", 2024-09-18T04:39:47.644 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7366e-05 2024-09-18T04:39:47.644 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.644 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.644 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:47.644 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382344+0000", 2024-09-18T04:39:47.644 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4319999999999997e-05 2024-09-18T04:39:47.644 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:47.644 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:47.644 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:47.644 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.644 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.644 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:59 2.0 2:19c52cde:::rbd_data.125fc09df6d1.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:47.644 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:39:47.644 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 65.235108514000004, 2024-09-18T04:39:47.645 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.027137247999999999, 2024-09-18T04:39:47.645 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:47.645 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:47.645 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:47.645 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:47.645 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:47.645 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 59 2024-09-18T04:39:47.645 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.645 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:47.645 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.645 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:47.645 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:39:47.645 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:47.646 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.646 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.646 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:47.646 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:39:47.646 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:47.646 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.646 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.646 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:47.646 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356204+0000", 2024-09-18T04:39:47.646 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:47.646 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.646 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.646 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:47.647 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362661+0000", 2024-09-18T04:39:47.647 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0064565129999999997 2024-09-18T04:39:47.647 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.647 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.647 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:47.647 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362684+0000", 2024-09-18T04:39:47.647 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3688000000000001e-05 2024-09-18T04:39:47.647 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.647 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.647 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:47.648 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362699+0000", 2024-09-18T04:39:47.648 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4798e-05 2024-09-18T04:39:47.648 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.648 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.648 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:47.648 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363016+0000", 2024-09-18T04:39:47.648 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031737300000000003 2024-09-18T04:39:47.648 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.648 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.648 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:47.648 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363201+0000", 2024-09-18T04:39:47.648 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000184131 2024-09-18T04:39:47.648 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.649 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.649 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:47.649 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363343+0000", 2024-09-18T04:39:47.649 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014236699999999999 2024-09-18T04:39:47.649 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.649 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.649 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:47.649 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.372189+0000", 2024-09-18T04:39:47.649 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0088464819999999993 2024-09-18T04:39:47.649 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.649 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.649 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:47.649 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383269+0000", 2024-09-18T04:39:47.649 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011079594 2024-09-18T04:39:47.650 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.650 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.650 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:47.650 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383314+0000", 2024-09-18T04:39:47.650 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4932000000000003e-05 2024-09-18T04:39:47.650 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.650 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.650 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:47.650 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383343+0000", 2024-09-18T04:39:47.650 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8600999999999999e-05 2024-09-18T04:39:47.650 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:47.650 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:47.650 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:47.650 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.651 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.651 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:40 2.6 e51/17 2:784d6cd1:ns::rbd_data.133e813ab5bc.0000000000000005:head v 51'7, mlcod=51'5)", 2024-09-18T04:39:47.651 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:39:47.651 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 65.038858779999998, 2024-09-18T04:39:47.651 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020497496, 2024-09-18T04:39:47.651 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:47.651 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:47.651 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:47.651 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.651 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:47.651 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:39:47.651 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:47.651 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.651 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.652 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:47.652 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:39:47.652 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:47.652 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.652 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.652 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:47.652 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552454+0000", 2024-09-18T04:39:47.652 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:47.652 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.652 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.652 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:47.652 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556448+0000", 2024-09-18T04:39:47.652 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0039940990000000001 2024-09-18T04:39:47.653 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.653 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.653 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:47.653 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556481+0000", 2024-09-18T04:39:47.653 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3778000000000001e-05 2024-09-18T04:39:47.653 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.653 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.653 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:47.653 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556493+0000", 2024-09-18T04:39:47.653 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1622e-05 2024-09-18T04:39:47.653 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.653 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.653 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:47.654 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556556+0000", 2024-09-18T04:39:47.654 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2582999999999994e-05 2024-09-18T04:39:47.654 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.654 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.654 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:47.654 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556592+0000", 2024-09-18T04:39:47.654 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5880999999999997e-05 2024-09-18T04:39:47.654 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.654 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.654 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:47.654 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.572786+0000", 2024-09-18T04:39:47.654 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016194571000000001 2024-09-18T04:39:47.654 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.655 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.655 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:47.655 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.572953+0000", 2024-09-18T04:39:47.655 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000166436 2024-09-18T04:39:47.655 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:47.655 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:47.655 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:47.655 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.655 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.655 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:45 2.0 2:0fbfa3dc:ns::rbd_data.133e813ab5bc.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:47.655 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:39:47.655 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 65.027644480999996, 2024-09-18T04:39:47.655 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.036558168000000002, 2024-09-18T04:39:47.656 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:47.656 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:47.656 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:47.656 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:47.656 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:47.656 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 45 2024-09-18T04:39:47.656 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.656 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:47.656 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.656 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:47.656 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:39:47.656 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:47.656 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.657 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.657 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:47.657 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:39:47.657 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:47.657 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.657 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.657 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:47.657 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:39:47.657 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:47.657 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.657 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.657 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:47.657 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578594+0000", 2024-09-18T04:39:47.658 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014927147999999999 2024-09-18T04:39:47.658 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.658 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.658 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:47.658 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578612+0000", 2024-09-18T04:39:47.658 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7623999999999999e-05 2024-09-18T04:39:47.658 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.658 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.658 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:47.658 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578620+0000", 2024-09-18T04:39:47.658 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9070000000000007e-06 2024-09-18T04:39:47.658 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.658 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.659 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:47.659 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578702+0000", 2024-09-18T04:39:47.659 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2349999999999996e-05 2024-09-18T04:39:47.659 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.659 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.659 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:47.659 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.579028+0000", 2024-09-18T04:39:47.659 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032622899999999998 2024-09-18T04:39:47.659 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.659 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.659 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:47.659 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.579343+0000", 2024-09-18T04:39:47.659 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031448899999999999 2024-09-18T04:39:47.659 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.660 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.660 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:47.660 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.590078+0000", 2024-09-18T04:39:47.660 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010734946 2024-09-18T04:39:47.660 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.660 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.660 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:47.660 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600089+0000", 2024-09-18T04:39:47.660 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010011195000000001 2024-09-18T04:39:47.660 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.660 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.660 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:47.660 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600177+0000", 2024-09-18T04:39:47.661 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8425999999999997e-05 2024-09-18T04:39:47.661 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.661 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.661 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:47.661 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600228+0000", 2024-09-18T04:39:47.661 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0216999999999997e-05 2024-09-18T04:39:47.661 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:47.661 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:47.661 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:47.661 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.661 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.661 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:54 2.0 2:04f17ef6:ns::rbd_data.133e813ab5bc.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:47.661 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:39:47.661 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 65.006072644, 2024-09-18T04:39:47.662 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.024612999, 2024-09-18T04:39:47.662 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:47.662 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:47.662 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:47.662 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:47.662 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:47.662 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 54 2024-09-18T04:39:47.662 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.662 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:47.662 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.662 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:47.662 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:39:47.662 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:47.663 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.663 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.663 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:47.663 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:39:47.663 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:47.663 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.663 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.663 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:47.663 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585239+0000", 2024-09-18T04:39:47.663 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:47.663 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.663 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.663 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:47.663 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593702+0000", 2024-09-18T04:39:47.664 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0084632990000000005 2024-09-18T04:39:47.664 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.664 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.664 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:47.664 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593719+0000", 2024-09-18T04:39:47.664 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.664e-05 2024-09-18T04:39:47.664 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.664 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.664 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:47.664 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593727+0000", 2024-09-18T04:39:47.664 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6839999999999996e-06 2024-09-18T04:39:47.664 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.664 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.665 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:47.665 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593767+0000", 2024-09-18T04:39:47.665 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0330999999999997e-05 2024-09-18T04:39:47.665 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.665 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.665 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:47.665 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593905+0000", 2024-09-18T04:39:47.666 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013754800000000001 2024-09-18T04:39:47.666 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.666 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.666 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:47.666 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.594013+0000", 2024-09-18T04:39:47.666 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000108805 2024-09-18T04:39:47.666 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.666 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.666 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:47.666 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604678+0000", 2024-09-18T04:39:47.666 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010664391 2024-09-18T04:39:47.666 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.666 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.667 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:47.667 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609662+0000", 2024-09-18T04:39:47.667 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0049846100000000004 2024-09-18T04:39:47.667 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.667 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.667 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:47.667 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609788+0000", 2024-09-18T04:39:47.667 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012518800000000001 2024-09-18T04:39:47.667 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.667 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.667 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:47.667 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609854+0000", 2024-09-18T04:39:47.667 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6556000000000004e-05 2024-09-18T04:39:47.668 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:47.668 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:47.668 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:47.668 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.668 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.668 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:58 2.7 2:fc7c6cd3:ns::rbd_data.133e813ab5bc.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:47.668 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:39:47.668 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 64.993230214999997, 2024-09-18T04:39:47.668 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.027690566999999999, 2024-09-18T04:39:47.668 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:47.668 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:47.668 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:47.668 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:47.669 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:47.669 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 58 2024-09-18T04:39:47.669 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.669 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:47.669 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.669 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:47.669 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:39:47.669 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:47.669 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.669 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.669 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:47.669 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:39:47.669 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:47.670 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.670 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.670 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:47.670 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598082+0000", 2024-09-18T04:39:47.670 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:47.670 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.670 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.670 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:47.670 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604832+0000", 2024-09-18T04:39:47.670 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0067504510000000002 2024-09-18T04:39:47.670 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.670 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.670 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:47.671 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604852+0000", 2024-09-18T04:39:47.671 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0228999999999999e-05 2024-09-18T04:39:47.671 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.671 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.671 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:47.671 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604870+0000", 2024-09-18T04:39:47.671 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7952e-05 2024-09-18T04:39:47.671 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.671 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.671 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:47.671 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604936+0000", 2024-09-18T04:39:47.671 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5431000000000004e-05 2024-09-18T04:39:47.671 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.671 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.672 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:47.672 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605085+0000", 2024-09-18T04:39:47.672 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014910300000000001 2024-09-18T04:39:47.672 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.672 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.672 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:39:47.672 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605205+0000", 2024-09-18T04:39:47.672 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000119887 2024-09-18T04:39:47.672 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.672 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.672 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:47.672 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625586+0000", 2024-09-18T04:39:47.672 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020381399000000001 2024-09-18T04:39:47.672 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.673 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.673 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:47.673 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625628+0000", 2024-09-18T04:39:47.673 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1873999999999999e-05 2024-09-18T04:39:47.673 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.673 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.673 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:47.673 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625722+0000", 2024-09-18T04:39:47.673 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3787000000000002e-05 2024-09-18T04:39:47.673 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.673 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.673 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:47.673 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625774+0000", 2024-09-18T04:39:47.673 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2503999999999997e-05 2024-09-18T04:39:47.674 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:47.674 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:47.674 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:47.674 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.674 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.674 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:56 2.6 e51/17 2:72453c3e:ns::rbd_data.133e813ab5bc.000000000000000d:head v 51'10, mlcod=51'8)", 2024-09-18T04:39:47.674 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:39:47.674 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 64.986033106999997, 2024-09-18T04:39:47.674 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019766552999999999, 2024-09-18T04:39:47.674 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:47.674 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:47.674 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:47.674 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.674 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:47.675 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:39:47.675 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:47.675 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.675 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.675 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:47.675 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:39:47.675 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:47.675 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.675 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.675 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:47.675 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605279+0000", 2024-09-18T04:39:47.675 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:47.675 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.675 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.676 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:47.676 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609417+0000", 2024-09-18T04:39:47.676 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0041378059999999999 2024-09-18T04:39:47.676 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.676 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.676 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:47.676 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609443+0000", 2024-09-18T04:39:47.676 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6098e-05 2024-09-18T04:39:47.676 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.676 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.676 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:47.676 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609458+0000", 2024-09-18T04:39:47.676 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4671000000000001e-05 2024-09-18T04:39:47.677 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.677 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.677 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:47.677 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609553+0000", 2024-09-18T04:39:47.677 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5265000000000004e-05 2024-09-18T04:39:47.677 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.677 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.677 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:47.677 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609583+0000", 2024-09-18T04:39:47.678 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9839e-05 2024-09-18T04:39:47.678 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.678 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.678 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:47.678 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625004+0000", 2024-09-18T04:39:47.678 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015420639999999999 2024-09-18T04:39:47.678 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.678 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.678 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:47.678 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625047+0000", 2024-09-18T04:39:47.678 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3665999999999998e-05 2024-09-18T04:39:47.679 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:47.679 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:47.679 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:47.679 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.679 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.679 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:99 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:47.679 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.582718+0000", 2024-09-18T04:39:47.679 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 51.008596197999999, 2024-09-18T04:39:47.679 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019362575, 2024-09-18T04:39:47.679 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:47.679 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:47.680 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:47.680 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:47.680 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:47.680 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 99 2024-09-18T04:39:47.680 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.680 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:47.680 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.680 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:47.680 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582718+0000", 2024-09-18T04:39:47.680 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:47.680 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.681 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.681 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:47.681 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582718+0000", 2024-09-18T04:39:47.681 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:47.681 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.681 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.681 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:47.681 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582717+0000", 2024-09-18T04:39:47.681 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:47.681 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.681 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.681 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:47.682 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582720+0000", 2024-09-18T04:39:47.682 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2239999999999998e-06 2024-09-18T04:39:47.682 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.682 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.682 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:47.682 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582723+0000", 2024-09-18T04:39:47.682 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.258e-06 2024-09-18T04:39:47.682 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.682 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.682 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:47.682 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582726+0000", 2024-09-18T04:39:47.682 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8870000000000001e-06 2024-09-18T04:39:47.683 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.683 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.683 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:47.683 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.592018+0000", 2024-09-18T04:39:47.683 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0092926420000000003 2024-09-18T04:39:47.684 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.684 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.684 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:47.684 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.592214+0000", 2024-09-18T04:39:47.684 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000195267 2024-09-18T04:39:47.684 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.684 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.684 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:47.684 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.592913+0000", 2024-09-18T04:39:47.685 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00069924600000000005 2024-09-18T04:39:47.685 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.685 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.685 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:47.685 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597493+0000", 2024-09-18T04:39:47.685 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045797629999999997 2024-09-18T04:39:47.685 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.685 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.685 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:47.685 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.602009+0000", 2024-09-18T04:39:47.685 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045159559999999998 2024-09-18T04:39:47.686 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.686 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.686 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:47.686 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.602042+0000", 2024-09-18T04:39:47.686 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3084999999999999e-05 2024-09-18T04:39:47.686 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.686 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.686 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:47.686 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.602080+0000", 2024-09-18T04:39:47.686 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8600000000000003e-05 2024-09-18T04:39:47.686 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:47.687 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:47.687 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:47.687 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.687 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.687 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:104 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:47.687 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.583165+0000", 2024-09-18T04:39:47.687 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 51.008148900000002, 2024-09-18T04:39:47.687 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023679603, 2024-09-18T04:39:47.687 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:47.687 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:47.687 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:47.687 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:47.688 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:47.688 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 104 2024-09-18T04:39:47.688 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.688 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:47.688 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.688 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:47.688 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583165+0000", 2024-09-18T04:39:47.688 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:47.688 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.688 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.688 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:47.688 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583165+0000", 2024-09-18T04:39:47.689 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:47.689 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.689 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.689 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:47.689 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583164+0000", 2024-09-18T04:39:47.689 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:47.689 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.689 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.689 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:47.689 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583167+0000", 2024-09-18T04:39:47.689 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9210000000000001e-06 2024-09-18T04:39:47.689 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.690 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.690 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:47.690 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583171+0000", 2024-09-18T04:39:47.690 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9759999999999997e-06 2024-09-18T04:39:47.690 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.690 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.690 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:47.690 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583174+0000", 2024-09-18T04:39:47.690 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9960000000000002e-06 2024-09-18T04:39:47.690 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.690 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.691 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:47.691 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.594373+0000", 2024-09-18T04:39:47.691 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011198665999999999 2024-09-18T04:39:47.691 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.691 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.691 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:47.691 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.594434+0000", 2024-09-18T04:39:47.691 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.1446999999999996e-05 2024-09-18T04:39:47.691 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.691 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.691 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:47.691 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.594692+0000", 2024-09-18T04:39:47.692 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00025756599999999998 2024-09-18T04:39:47.692 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.692 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.692 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:47.692 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599302+0000", 2024-09-18T04:39:47.692 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0046099640000000002 2024-09-18T04:39:47.692 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.692 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.692 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:47.692 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606710+0000", 2024-09-18T04:39:47.692 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0074077739999999998 2024-09-18T04:39:47.693 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.693 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.693 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:47.693 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606798+0000", 2024-09-18T04:39:47.693 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8133999999999993e-05 2024-09-18T04:39:47.693 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.693 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.693 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:47.693 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606845+0000", 2024-09-18T04:39:47.693 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6653000000000002e-05 2024-09-18T04:39:47.693 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:47.694 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:47.694 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:47.694 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.694 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.694 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:105 2.0 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:47.694 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.583193+0000", 2024-09-18T04:39:47.694 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 51.008121281000001, 2024-09-18T04:39:47.694 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023713649, 2024-09-18T04:39:47.694 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:47.694 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:47.694 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:47.695 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:47.695 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:47.695 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 105 2024-09-18T04:39:47.695 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.695 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:47.695 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.695 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:47.695 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583193+0000", 2024-09-18T04:39:47.695 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:47.695 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.695 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.695 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:47.696 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583193+0000", 2024-09-18T04:39:47.696 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:47.696 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.696 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.696 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:47.696 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583192+0000", 2024-09-18T04:39:47.696 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:47.696 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.696 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.696 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:47.697 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583194+0000", 2024-09-18T04:39:47.697 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.305e-06 2024-09-18T04:39:47.697 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.697 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.697 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:47.697 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583198+0000", 2024-09-18T04:39:47.697 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9369999999999997e-06 2024-09-18T04:39:47.697 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.697 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.697 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:47.697 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583202+0000", 2024-09-18T04:39:47.697 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6480000000000001e-06 2024-09-18T04:39:47.698 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.698 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.698 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:47.698 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.595304+0000", 2024-09-18T04:39:47.698 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012102063999999999 2024-09-18T04:39:47.698 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.698 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.698 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:47.698 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.595390+0000", 2024-09-18T04:39:47.698 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5493000000000003e-05 2024-09-18T04:39:47.698 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.699 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.699 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:47.699 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.595509+0000", 2024-09-18T04:39:47.699 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011966199999999999 2024-09-18T04:39:47.699 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.699 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.699 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:47.699 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599358+0000", 2024-09-18T04:39:47.699 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0038486689999999999 2024-09-18T04:39:47.699 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.699 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.699 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:47.700 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606859+0000", 2024-09-18T04:39:47.700 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0075010789999999999 2024-09-18T04:39:47.700 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.700 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.700 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:47.700 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606887+0000", 2024-09-18T04:39:47.700 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7773e-05 2024-09-18T04:39:47.700 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.700 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.700 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:47.700 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606906+0000", 2024-09-18T04:39:47.700 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9417000000000001e-05 2024-09-18T04:39:47.701 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:47.701 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:47.701 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:47.701 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.701 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.701 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:106 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:47.701 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.584425+0000", 2024-09-18T04:39:47.701 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 51.006888985000003, 2024-09-18T04:39:47.701 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.022529293999999998, 2024-09-18T04:39:47.701 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:47.702 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:47.702 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:47.702 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:47.702 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:47.702 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 106 2024-09-18T04:39:47.702 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.702 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:47.702 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.702 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:47.702 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584425+0000", 2024-09-18T04:39:47.702 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:47.703 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.703 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.703 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:47.703 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584425+0000", 2024-09-18T04:39:47.703 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:47.703 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.703 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.703 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:47.703 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584424+0000", 2024-09-18T04:39:47.703 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:47.703 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.704 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.704 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:47.704 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584430+0000", 2024-09-18T04:39:47.704 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6389999999999997e-06 2024-09-18T04:39:47.704 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.704 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.704 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:47.704 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584456+0000", 2024-09-18T04:39:47.704 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6188999999999999e-05 2024-09-18T04:39:47.704 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.704 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.705 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:47.705 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584463+0000", 2024-09-18T04:39:47.705 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7340000000000002e-06 2024-09-18T04:39:47.705 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.705 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.705 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:47.705 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597537+0000", 2024-09-18T04:39:47.705 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013074161000000001 2024-09-18T04:39:47.705 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.705 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.705 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:47.706 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597598+0000", 2024-09-18T04:39:47.706 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0930000000000001e-05 2024-09-18T04:39:47.706 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.706 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.706 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:47.706 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597922+0000", 2024-09-18T04:39:47.706 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032473000000000002 2024-09-18T04:39:47.706 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.706 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.706 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:47.706 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.601103+0000", 2024-09-18T04:39:47.707 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0031809849999999999 2024-09-18T04:39:47.707 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.707 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.707 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:47.707 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606912+0000", 2024-09-18T04:39:47.707 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058083550000000003 2024-09-18T04:39:47.707 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.707 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.707 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:47.707 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606935+0000", 2024-09-18T04:39:47.707 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2759999999999999e-05 2024-09-18T04:39:47.708 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.708 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.708 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:47.708 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606954+0000", 2024-09-18T04:39:47.708 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9602999999999998e-05 2024-09-18T04:39:47.708 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:47.708 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:47.708 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:47.708 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.708 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.708 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:109 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:47.709 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.585231+0000", 2024-09-18T04:39:47.709 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 51.006082364999997, 2024-09-18T04:39:47.709 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.021767496000000001, 2024-09-18T04:39:47.709 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:47.709 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:47.709 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:47.709 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:47.709 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:47.709 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 109 2024-09-18T04:39:47.709 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.709 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:47.710 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.710 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:47.710 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585231+0000", 2024-09-18T04:39:47.710 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:47.710 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.710 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.710 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:47.710 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585231+0000", 2024-09-18T04:39:47.710 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:47.710 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.710 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.710 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:47.710 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585230+0000", 2024-09-18T04:39:47.711 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:47.711 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.711 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.711 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:47.711 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585237+0000", 2024-09-18T04:39:47.711 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.618e-06 2024-09-18T04:39:47.711 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.711 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.711 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:47.711 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585256+0000", 2024-09-18T04:39:47.711 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9048000000000001e-05 2024-09-18T04:39:47.712 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.712 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.712 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:47.712 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585274+0000", 2024-09-18T04:39:47.712 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8153000000000001e-05 2024-09-18T04:39:47.712 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.712 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.712 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:47.712 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.598284+0000", 2024-09-18T04:39:47.712 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013009682999999999 2024-09-18T04:39:47.712 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.712 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.713 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:47.713 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.598383+0000", 2024-09-18T04:39:47.713 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.8722999999999997e-05 2024-09-18T04:39:47.713 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.713 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.713 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:47.713 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.598829+0000", 2024-09-18T04:39:47.713 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00044623999999999998 2024-09-18T04:39:47.713 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.713 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.713 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:47.713 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.601149+0000", 2024-09-18T04:39:47.714 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0023204150000000002 2024-09-18T04:39:47.714 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.714 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.714 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:47.714 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606960+0000", 2024-09-18T04:39:47.714 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058106390000000003 2024-09-18T04:39:47.714 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.714 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.714 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:47.714 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606982+0000", 2024-09-18T04:39:47.714 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2252e-05 2024-09-18T04:39:47.714 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.715 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.715 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:47.715 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606999+0000", 2024-09-18T04:39:47.715 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6871000000000002e-05 2024-09-18T04:39:47.715 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:47.715 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:47.715 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:47.715 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.715 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.715 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:110 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:47.715 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.586640+0000", 2024-09-18T04:39:47.715 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 51.004674045999998, 2024-09-18T04:39:47.716 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020400911000000001, 2024-09-18T04:39:47.716 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:47.716 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:47.716 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:47.716 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:47.716 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:47.716 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 110 2024-09-18T04:39:47.716 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.716 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:47.716 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.716 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:47.717 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586640+0000", 2024-09-18T04:39:47.717 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:47.717 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.717 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.717 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:47.717 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586640+0000", 2024-09-18T04:39:47.717 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:47.717 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.717 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.717 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:47.717 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586639+0000", 2024-09-18T04:39:47.717 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:47.718 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.718 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.718 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:47.718 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586652+0000", 2024-09-18T04:39:47.718 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2482e-05 2024-09-18T04:39:47.718 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.718 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.718 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:47.718 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586660+0000", 2024-09-18T04:39:47.718 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1659999999999994e-06 2024-09-18T04:39:47.718 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.720 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.720 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:47.721 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586666+0000", 2024-09-18T04:39:47.721 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6900000000000003e-06 2024-09-18T04:39:47.721 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.721 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.721 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:47.721 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599381+0000", 2024-09-18T04:39:47.721 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012714328 2024-09-18T04:39:47.721 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.721 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.721 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:47.721 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599436+0000", 2024-09-18T04:39:47.721 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4802999999999997e-05 2024-09-18T04:39:47.722 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.722 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.722 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:47.722 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599629+0000", 2024-09-18T04:39:47.722 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000193921 2024-09-18T04:39:47.722 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.722 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.722 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:47.722 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.601174+0000", 2024-09-18T04:39:47.722 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001544857 2024-09-18T04:39:47.722 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.722 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.723 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:47.723 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.607004+0000", 2024-09-18T04:39:47.723 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058300820000000003 2024-09-18T04:39:47.723 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.723 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.723 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:47.723 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.607024+0000", 2024-09-18T04:39:47.723 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9888e-05 2024-09-18T04:39:47.723 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:47.723 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:47.723 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:47.723 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.607041+0000", 2024-09-18T04:39:47.724 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6450999999999999e-05 2024-09-18T04:39:47.724 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:47.724 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:47.724 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:47.724 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:47.724 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:47.724 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:47.726 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:39:47.726 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:39:47.726 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:39:47.726 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:39:47.727 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:39:47.728 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:47.716+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:47.748 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:39:47.769 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:39:47.771 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:39:47.771 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:39:47.771 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_max_backfills = '' osd_pg_delete_cost = '' (not observed, change may require restart) osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-09-18T04:39:47.772 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:39:47.772 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:39:47.772 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:39:47.805 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-18T04:39:47.827 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:47.820+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:47.843 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:47.843 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:47.853 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:39:47.927 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:47.920+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:48.005 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:48.005 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:48.015 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:39:48.027 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:48.024+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:39:48.124 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:48.124 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:48.129 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:48.120+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:48.134 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:39:48.230 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:48.224+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:48.287 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:48.288 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:48.297 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:39:48.331 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:48.328+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:39:48.400 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:48.400 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:48.410 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:39:48.433 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:48.428+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:39:48.518 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:48.519 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:48.529 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:39:48.535 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:48.528+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:48.635 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:48.632+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:48.647 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:48.647 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:48.657 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:39:48.736 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:48.728+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:48.740 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-09-18T04:39:48.771 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:48.771 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:48.781 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:39:48.837 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:48.832+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:48.885 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:48.886 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:48.896 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:39:48.938 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:48.936+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:49.000 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:49.001 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:49.010 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:39:49.039 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:49.032+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:49.120 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:49.121 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:49.130 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:39:49.139 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:49.136+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:49.235 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:49.236 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:49.240 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:49.236+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:49.245 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:39:49.341 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:49.332+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:49.397 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:49.397 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:49.407 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:39:49.442 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:49.440+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:49.516 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:49.516 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:49.526 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:39:49.543 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:49.540+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:49.634 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:49.634 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:49.644 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:39:49.645 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:49.640+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:49.744 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:49.736+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:49.763 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:49.763 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:49.774 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:39:49.784 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T04:39:49.846 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:49.840+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:49.877 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:49.877 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:49.887 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:39:49.932 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:39:49.947 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:49.940+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:49.952 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:39:49.974 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:39:49.997 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:39:50.001 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:50.002 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:39:50.002 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:39:50.002 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:39:50.002 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.002 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:39:50.002 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:39:50.002 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 142.20865327000001, 2024-09-18T04:39:50.002 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017399218000000001, 2024-09-18T04:39:50.002 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:50.002 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:50.002 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:50.002 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:39:50.003 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:39:50.003 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 114 2024-09-18T04:39:50.003 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.003 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:50.003 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.003 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:50.003 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:39:50.003 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.003 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.003 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.003 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:50.003 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:39:50.003 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.004 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.004 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.004 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:50.004 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786900+0000", 2024-09-18T04:39:50.004 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:50.004 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.004 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.004 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:50.004 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786908+0000", 2024-09-18T04:39:50.004 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8199999999999997e-06 2024-09-18T04:39:50.004 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.005 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.005 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:50.005 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786925+0000", 2024-09-18T04:39:50.005 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6849000000000001e-05 2024-09-18T04:39:50.005 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.005 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.005 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:50.005 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786934+0000", 2024-09-18T04:39:50.005 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6299999999999993e-06 2024-09-18T04:39:50.005 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.005 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.005 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:50.006 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800926+0000", 2024-09-18T04:39:50.006 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013991632 2024-09-18T04:39:50.006 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.006 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.006 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:50.006 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801060+0000", 2024-09-18T04:39:50.006 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013378199999999999 2024-09-18T04:39:50.006 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.006 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.006 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:50.006 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801166+0000", 2024-09-18T04:39:50.006 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010607099999999999 2024-09-18T04:39:50.006 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.007 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.007 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:50.007 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803026+0000", 2024-09-18T04:39:50.007 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001860469 2024-09-18T04:39:50.007 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.007 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.007 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:50.007 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804245+0000", 2024-09-18T04:39:50.007 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0012185900000000001 2024-09-18T04:39:50.007 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.007 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.007 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:50.007 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804278+0000", 2024-09-18T04:39:50.007 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3272000000000003e-05 2024-09-18T04:39:50.008 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.008 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.008 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:50.008 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804300+0000", 2024-09-18T04:39:50.008 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2266e-05 2024-09-18T04:39:50.008 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.008 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:50.008 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.008 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.009 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.009 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:39:50.009 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:39:50.009 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 142.20774037199999, 2024-09-18T04:39:50.009 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017521662, 2024-09-18T04:39:50.009 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:50.009 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:50.009 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:50.009 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:39:50.009 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:39:50.009 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 116 2024-09-18T04:39:50.009 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.009 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:50.010 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.010 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:50.010 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:39:50.010 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.010 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.010 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.010 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:50.010 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:39:50.010 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.010 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.010 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.010 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:50.011 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787813+0000", 2024-09-18T04:39:50.011 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:50.011 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.011 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.011 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:50.011 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787824+0000", 2024-09-18T04:39:50.011 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1157e-05 2024-09-18T04:39:50.011 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.011 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.011 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:50.011 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787840+0000", 2024-09-18T04:39:50.011 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6269e-05 2024-09-18T04:39:50.011 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.011 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.012 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:50.012 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787853+0000", 2024-09-18T04:39:50.012 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3156000000000001e-05 2024-09-18T04:39:50.012 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.012 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.012 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:50.012 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803110+0000", 2024-09-18T04:39:50.012 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015256736 2024-09-18T04:39:50.012 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.012 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.012 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:50.012 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803197+0000", 2024-09-18T04:39:50.012 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7195999999999997e-05 2024-09-18T04:39:50.012 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.013 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.013 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:50.013 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803308+0000", 2024-09-18T04:39:50.013 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111005 2024-09-18T04:39:50.013 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.013 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.013 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:50.013 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804316+0000", 2024-09-18T04:39:50.013 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010073949999999999 2024-09-18T04:39:50.013 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.013 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.013 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:50.014 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805232+0000", 2024-09-18T04:39:50.014 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091636300000000001 2024-09-18T04:39:50.014 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.014 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.014 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:50.014 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805277+0000", 2024-09-18T04:39:50.014 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4864000000000003e-05 2024-09-18T04:39:50.014 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.014 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.014 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:50.014 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805336+0000", 2024-09-18T04:39:50.014 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8968000000000002e-05 2024-09-18T04:39:50.014 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.014 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:50.015 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.015 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.015 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.015 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:39:50.015 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:39:50.015 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 142.20768831699999, 2024-09-18T04:39:50.015 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017983453, 2024-09-18T04:39:50.015 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:50.015 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:50.015 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:50.015 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:39:50.015 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:39:50.015 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 117 2024-09-18T04:39:50.015 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.016 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:50.016 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.016 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:50.016 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:39:50.016 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.016 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.016 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.016 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:50.016 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:39:50.016 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.016 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.016 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.016 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:50.017 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787865+0000", 2024-09-18T04:39:50.017 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:50.017 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.017 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.017 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:50.017 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787889+0000", 2024-09-18T04:39:50.017 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4244999999999999e-05 2024-09-18T04:39:50.017 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.017 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.017 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:50.017 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787905+0000", 2024-09-18T04:39:50.017 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5739999999999998e-05 2024-09-18T04:39:50.017 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.018 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.018 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:50.018 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787918+0000", 2024-09-18T04:39:50.018 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.252e-05 2024-09-18T04:39:50.018 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.018 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.018 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:50.018 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803693+0000", 2024-09-18T04:39:50.018 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015774872999999998 2024-09-18T04:39:50.018 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.018 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.018 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:50.018 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803775+0000", 2024-09-18T04:39:50.019 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2761000000000003e-05 2024-09-18T04:39:50.019 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.019 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.019 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:50.019 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803887+0000", 2024-09-18T04:39:50.019 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111708 2024-09-18T04:39:50.019 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.019 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.019 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:50.019 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804823+0000", 2024-09-18T04:39:50.019 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00093625100000000003 2024-09-18T04:39:50.019 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.019 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.019 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:50.020 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805804+0000", 2024-09-18T04:39:50.020 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00098114699999999992 2024-09-18T04:39:50.020 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.020 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.020 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:50.020 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805831+0000", 2024-09-18T04:39:50.020 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6401999999999999e-05 2024-09-18T04:39:50.020 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.020 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.021 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:50.021 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805850+0000", 2024-09-18T04:39:50.021 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8862e-05 2024-09-18T04:39:50.021 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.021 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:50.021 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.021 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.021 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.021 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:36 2.7 2:fe1b02ad:::rbd_data.125fc09df6d1.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:50.022 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:39:50.022 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 67.729832207000001, 2024-09-18T04:39:50.022 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.040387804999999999, 2024-09-18T04:39:50.022 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:50.022 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:50.022 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:50.022 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:50.022 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:50.022 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 36 2024-09-18T04:39:50.023 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.023 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:50.023 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.023 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:50.023 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:39:50.023 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.023 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.023 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.024 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:50.024 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:39:50.024 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.024 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.024 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.024 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:50.024 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265720+0000", 2024-09-18T04:39:50.024 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:50.024 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.024 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.024 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:50.024 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272590+0000", 2024-09-18T04:39:50.024 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.006870508 2024-09-18T04:39:50.025 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.025 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.025 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:50.025 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272623+0000", 2024-09-18T04:39:50.025 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2716000000000003e-05 2024-09-18T04:39:50.025 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.025 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.025 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:50.025 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272637+0000", 2024-09-18T04:39:50.025 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4175999999999999e-05 2024-09-18T04:39:50.025 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.025 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.025 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:50.025 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272779+0000", 2024-09-18T04:39:50.026 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000141392 2024-09-18T04:39:50.026 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.026 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.026 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:50.026 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273031+0000", 2024-09-18T04:39:50.026 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00025249099999999998 2024-09-18T04:39:50.026 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.026 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.026 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:39:50.026 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273219+0000", 2024-09-18T04:39:50.026 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000187531 2024-09-18T04:39:50.026 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.026 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.026 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:50.027 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.302806+0000", 2024-09-18T04:39:50.027 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.029587553999999999 2024-09-18T04:39:50.027 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.027 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.027 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:50.027 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306004+0000", 2024-09-18T04:39:50.027 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0031982629999999998 2024-09-18T04:39:50.027 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.027 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.027 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:50.027 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306071+0000", 2024-09-18T04:39:50.027 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6829999999999995e-05 2024-09-18T04:39:50.027 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.027 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.028 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:50.028 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306110+0000", 2024-09-18T04:39:50.028 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8874000000000001e-05 2024-09-18T04:39:50.028 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.028 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:50.028 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.028 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.028 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.028 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:38 2.0 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:50.028 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:39:50.028 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 67.72287867, 2024-09-18T04:39:50.028 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.033754348000000003, 2024-09-18T04:39:50.028 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:50.028 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:50.029 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:50.029 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:50.029 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:50.029 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 38 2024-09-18T04:39:50.029 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.029 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:50.029 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.029 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:50.029 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:39:50.029 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.029 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.029 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.029 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:50.029 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:39:50.030 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.030 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.030 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.030 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:50.030 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272673+0000", 2024-09-18T04:39:50.030 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:50.030 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.030 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.030 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:50.030 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277166+0000", 2024-09-18T04:39:50.030 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044922249999999999 2024-09-18T04:39:50.030 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.030 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.031 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:50.031 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277189+0000", 2024-09-18T04:39:50.031 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3283000000000001e-05 2024-09-18T04:39:50.031 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.031 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.031 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:50.031 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277205+0000", 2024-09-18T04:39:50.031 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6305000000000001e-05 2024-09-18T04:39:50.031 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.031 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.031 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:50.031 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277290+0000", 2024-09-18T04:39:50.031 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5135000000000004e-05 2024-09-18T04:39:50.031 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.032 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.032 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:50.032 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277531+0000", 2024-09-18T04:39:50.032 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00024014699999999999 2024-09-18T04:39:50.032 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.032 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.032 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:50.032 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277793+0000", 2024-09-18T04:39:50.032 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00026224499999999998 2024-09-18T04:39:50.032 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.032 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.032 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:50.033 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303990+0000", 2024-09-18T04:39:50.033 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.026197295999999998 2024-09-18T04:39:50.033 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.033 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.033 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:50.033 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306340+0000", 2024-09-18T04:39:50.033 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.002349524 2024-09-18T04:39:50.033 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.033 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.033 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:50.033 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306394+0000", 2024-09-18T04:39:50.033 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3934999999999998e-05 2024-09-18T04:39:50.034 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.034 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.034 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:50.034 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306430+0000", 2024-09-18T04:39:50.034 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6690000000000003e-05 2024-09-18T04:39:50.034 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.034 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:50.034 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.034 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.034 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.034 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:39 2.7 2:e9c909c5:::rbd_data.125fc09df6d1.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:50.034 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:39:50.035 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 67.718319065000003, 2024-09-18T04:39:50.035 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.029110673, 2024-09-18T04:39:50.035 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:50.035 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:50.035 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:50.035 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:50.035 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:50.035 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 39 2024-09-18T04:39:50.035 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.035 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:50.035 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.036 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:50.036 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:39:50.036 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.036 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.036 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.036 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:50.036 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:39:50.036 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.036 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.036 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.036 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:50.037 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277234+0000", 2024-09-18T04:39:50.037 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:50.037 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.037 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.037 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:50.037 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281351+0000", 2024-09-18T04:39:50.037 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0041170920000000001 2024-09-18T04:39:50.037 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.037 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.037 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:50.037 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281375+0000", 2024-09-18T04:39:50.038 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.446e-05 2024-09-18T04:39:50.038 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.038 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.038 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:50.038 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281390+0000", 2024-09-18T04:39:50.038 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4309e-05 2024-09-18T04:39:50.038 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.038 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.038 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:50.039 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.282306+0000", 2024-09-18T04:39:50.039 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091662000000000004 2024-09-18T04:39:50.039 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.039 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.039 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:50.039 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.282613+0000", 2024-09-18T04:39:50.039 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00030663799999999997 2024-09-18T04:39:50.039 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.039 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.039 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:39:50.039 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283001+0000", 2024-09-18T04:39:50.039 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00038801599999999998 2024-09-18T04:39:50.040 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.040 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.040 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:50.040 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306239+0000", 2024-09-18T04:39:50.040 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023238466999999999 2024-09-18T04:39:50.040 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.040 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.040 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:50.040 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306259+0000", 2024-09-18T04:39:50.040 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9837999999999999e-05 2024-09-18T04:39:50.040 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.040 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.040 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:50.040 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306299+0000", 2024-09-18T04:39:50.041 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0194000000000002e-05 2024-09-18T04:39:50.041 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.041 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.041 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:50.041 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306346+0000", 2024-09-18T04:39:50.041 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6845000000000003e-05 2024-09-18T04:39:50.041 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.041 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:50.041 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.041 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.041 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.041 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:46 2.6 e51/17 2:7ba25245:::rbd_data.125fc09df6d1.0000000000000006:head v 51'2, mlcod=0'0)", 2024-09-18T04:39:50.041 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:39:50.042 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 67.682397933000004, 2024-09-18T04:39:50.042 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019843098, 2024-09-18T04:39:50.042 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:50.042 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:50.042 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:50.042 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.042 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:50.042 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:39:50.042 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.042 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.042 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.042 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:50.042 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:39:50.042 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.043 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.043 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.043 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:50.043 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313156+0000", 2024-09-18T04:39:50.043 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:50.043 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.043 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.043 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:50.043 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320144+0000", 2024-09-18T04:39:50.043 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.006988345 2024-09-18T04:39:50.043 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.043 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.043 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:50.043 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320175+0000", 2024-09-18T04:39:50.044 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.0861999999999997e-05 2024-09-18T04:39:50.044 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.044 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.044 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:50.044 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320188+0000", 2024-09-18T04:39:50.044 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2714999999999999e-05 2024-09-18T04:39:50.044 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.044 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.044 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:50.044 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320267+0000", 2024-09-18T04:39:50.044 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9586999999999995e-05 2024-09-18T04:39:50.044 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.044 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.045 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:50.045 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320311+0000", 2024-09-18T04:39:50.045 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3529000000000003e-05 2024-09-18T04:39:50.045 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.045 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.045 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:50.045 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.332949+0000", 2024-09-18T04:39:50.045 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012638267999999999 2024-09-18T04:39:50.045 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.045 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.045 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:50.045 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.333000+0000", 2024-09-18T04:39:50.045 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0736999999999999e-05 2024-09-18T04:39:50.046 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.046 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:50.046 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.046 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.046 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.046 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:57 2.0 2:0cdd070e:::rbd_data.125fc09df6d1.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:50.046 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:39:50.046 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 67.646540357999996, 2024-09-18T04:39:50.046 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.033329974999999998, 2024-09-18T04:39:50.046 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:50.046 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:50.046 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:50.047 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:50.047 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:50.047 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 57 2024-09-18T04:39:50.047 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.047 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:50.047 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.047 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:50.047 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:39:50.047 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.047 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.047 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.047 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:50.047 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:39:50.048 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.048 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.048 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.048 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:50.048 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349013+0000", 2024-09-18T04:39:50.048 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:50.048 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.048 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.048 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:50.048 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356138+0000", 2024-09-18T04:39:50.048 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071258010000000002 2024-09-18T04:39:50.048 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.049 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.049 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:50.049 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356168+0000", 2024-09-18T04:39:50.049 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9679000000000002e-05 2024-09-18T04:39:50.049 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.049 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.049 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:50.049 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356181+0000", 2024-09-18T04:39:50.049 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3164e-05 2024-09-18T04:39:50.049 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.049 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.049 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:50.049 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356254+0000", 2024-09-18T04:39:50.050 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3182999999999994e-05 2024-09-18T04:39:50.050 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.050 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.050 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:50.050 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356453+0000", 2024-09-18T04:39:50.050 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019876599999999999 2024-09-18T04:39:50.050 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.050 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.050 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:50.050 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356627+0000", 2024-09-18T04:39:50.050 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017340199999999999 2024-09-18T04:39:50.050 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.050 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.050 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:50.051 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.369200+0000", 2024-09-18T04:39:50.051 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012573302 2024-09-18T04:39:50.051 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.051 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.051 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:50.051 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382262+0000", 2024-09-18T04:39:50.051 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013062549 2024-09-18T04:39:50.051 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.051 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.051 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:50.051 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382310+0000", 2024-09-18T04:39:50.051 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7366e-05 2024-09-18T04:39:50.051 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.052 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.052 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:50.052 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382344+0000", 2024-09-18T04:39:50.052 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4319999999999997e-05 2024-09-18T04:39:50.052 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.052 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:50.052 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.052 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.052 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.052 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:59 2.0 2:19c52cde:::rbd_data.125fc09df6d1.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:50.052 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:39:50.052 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 67.639349193000001, 2024-09-18T04:39:50.052 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.027137247999999999, 2024-09-18T04:39:50.053 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:50.053 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:50.053 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:50.053 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:50.053 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:50.053 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 59 2024-09-18T04:39:50.053 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.053 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:50.053 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.053 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:50.053 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:39:50.054 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.054 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.054 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.054 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:50.054 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:39:50.054 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.054 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.054 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.054 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:50.054 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356204+0000", 2024-09-18T04:39:50.054 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:50.054 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.054 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.055 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:50.055 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362661+0000", 2024-09-18T04:39:50.055 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0064565129999999997 2024-09-18T04:39:50.055 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.055 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.055 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:50.055 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362684+0000", 2024-09-18T04:39:50.055 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3688000000000001e-05 2024-09-18T04:39:50.055 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.055 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.055 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:50.055 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362699+0000", 2024-09-18T04:39:50.055 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4798e-05 2024-09-18T04:39:50.056 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.056 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.056 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:50.056 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363016+0000", 2024-09-18T04:39:50.056 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031737300000000003 2024-09-18T04:39:50.056 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.056 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.056 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:50.056 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363201+0000", 2024-09-18T04:39:50.056 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000184131 2024-09-18T04:39:50.056 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.056 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.056 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:50.057 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363343+0000", 2024-09-18T04:39:50.057 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014236699999999999 2024-09-18T04:39:50.057 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.057 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.057 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:50.057 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.372189+0000", 2024-09-18T04:39:50.057 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0088464819999999993 2024-09-18T04:39:50.057 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.057 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.057 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:50.057 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383269+0000", 2024-09-18T04:39:50.057 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011079594 2024-09-18T04:39:50.057 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.057 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.058 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:50.058 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383314+0000", 2024-09-18T04:39:50.058 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4932000000000003e-05 2024-09-18T04:39:50.058 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.058 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.058 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:50.058 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383343+0000", 2024-09-18T04:39:50.058 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8600999999999999e-05 2024-09-18T04:39:50.058 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.058 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:50.058 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.058 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.058 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.058 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:40 2.6 e51/17 2:784d6cd1:ns::rbd_data.133e813ab5bc.0000000000000005:head v 51'7, mlcod=51'5)", 2024-09-18T04:39:50.059 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:39:50.059 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 67.443099458999995, 2024-09-18T04:39:50.059 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020497496, 2024-09-18T04:39:50.059 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:50.059 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:50.059 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:50.059 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.059 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:50.059 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:39:50.059 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.059 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.059 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.059 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:50.059 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:39:50.060 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.060 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.060 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.060 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:50.060 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552454+0000", 2024-09-18T04:39:50.060 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:50.060 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.060 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.060 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:50.060 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556448+0000", 2024-09-18T04:39:50.060 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0039940990000000001 2024-09-18T04:39:50.060 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.060 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.060 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:50.061 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556481+0000", 2024-09-18T04:39:50.061 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3778000000000001e-05 2024-09-18T04:39:50.061 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.061 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.061 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:50.061 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556493+0000", 2024-09-18T04:39:50.061 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1622e-05 2024-09-18T04:39:50.061 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.061 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.061 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:50.061 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556556+0000", 2024-09-18T04:39:50.061 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2582999999999994e-05 2024-09-18T04:39:50.061 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.061 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.062 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:50.062 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556592+0000", 2024-09-18T04:39:50.062 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5880999999999997e-05 2024-09-18T04:39:50.062 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.062 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.062 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:50.062 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.572786+0000", 2024-09-18T04:39:50.062 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016194571000000001 2024-09-18T04:39:50.062 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.062 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.062 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:50.062 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.572953+0000", 2024-09-18T04:39:50.062 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000166436 2024-09-18T04:39:50.062 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.063 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:50.063 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.063 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.063 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.063 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:45 2.0 2:0fbfa3dc:ns::rbd_data.133e813ab5bc.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:50.063 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:39:50.063 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 67.431885159999993, 2024-09-18T04:39:50.063 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.036558168000000002, 2024-09-18T04:39:50.063 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:50.063 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:50.063 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:50.063 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:50.063 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:50.063 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 45 2024-09-18T04:39:50.064 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.064 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:50.064 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.064 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:50.064 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:39:50.064 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.064 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.064 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.064 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:50.064 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:39:50.064 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.064 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.064 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.064 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:50.065 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:39:50.065 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:50.065 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.065 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.065 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:50.065 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578594+0000", 2024-09-18T04:39:50.065 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014927147999999999 2024-09-18T04:39:50.065 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.065 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.065 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:50.065 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578612+0000", 2024-09-18T04:39:50.065 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7623999999999999e-05 2024-09-18T04:39:50.066 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.066 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.066 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:50.066 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578620+0000", 2024-09-18T04:39:50.066 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9070000000000007e-06 2024-09-18T04:39:50.066 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.066 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.066 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:50.066 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578702+0000", 2024-09-18T04:39:50.066 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2349999999999996e-05 2024-09-18T04:39:50.066 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.066 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.066 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:50.066 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.579028+0000", 2024-09-18T04:39:50.067 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032622899999999998 2024-09-18T04:39:50.067 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.067 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.067 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:50.067 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.579343+0000", 2024-09-18T04:39:50.067 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031448899999999999 2024-09-18T04:39:50.067 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.067 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.067 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:50.067 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.590078+0000", 2024-09-18T04:39:50.067 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010734946 2024-09-18T04:39:50.067 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.068 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.068 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:50.068 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600089+0000", 2024-09-18T04:39:50.068 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010011195000000001 2024-09-18T04:39:50.068 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.068 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.068 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:50.068 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600177+0000", 2024-09-18T04:39:50.068 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8425999999999997e-05 2024-09-18T04:39:50.068 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.068 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.068 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:50.068 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600228+0000", 2024-09-18T04:39:50.069 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0216999999999997e-05 2024-09-18T04:39:50.069 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.069 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:50.069 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.069 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.069 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.069 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:54 2.0 2:04f17ef6:ns::rbd_data.133e813ab5bc.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:50.069 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:39:50.069 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 67.410313322999997, 2024-09-18T04:39:50.069 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.024612999, 2024-09-18T04:39:50.069 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:50.069 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:50.069 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:50.069 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:50.070 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:50.070 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 54 2024-09-18T04:39:50.070 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.070 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:50.070 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.070 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:50.070 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:39:50.070 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.070 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.070 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.070 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:50.070 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:39:50.070 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.071 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.071 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.071 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:50.071 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585239+0000", 2024-09-18T04:39:50.071 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:50.071 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.071 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.071 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:50.071 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593702+0000", 2024-09-18T04:39:50.071 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0084632990000000005 2024-09-18T04:39:50.071 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.071 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.071 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:50.071 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593719+0000", 2024-09-18T04:39:50.071 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.664e-05 2024-09-18T04:39:50.072 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.072 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.072 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:50.072 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593727+0000", 2024-09-18T04:39:50.072 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6839999999999996e-06 2024-09-18T04:39:50.072 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.072 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.072 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:50.072 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593767+0000", 2024-09-18T04:39:50.072 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0330999999999997e-05 2024-09-18T04:39:50.072 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.072 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.072 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:50.073 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593905+0000", 2024-09-18T04:39:50.073 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013754800000000001 2024-09-18T04:39:50.073 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.073 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.073 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:50.073 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.594013+0000", 2024-09-18T04:39:50.073 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000108805 2024-09-18T04:39:50.073 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.073 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.073 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:50.073 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604678+0000", 2024-09-18T04:39:50.073 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010664391 2024-09-18T04:39:50.073 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.073 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.074 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:50.074 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609662+0000", 2024-09-18T04:39:50.074 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0049846100000000004 2024-09-18T04:39:50.074 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.074 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.074 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:50.074 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609788+0000", 2024-09-18T04:39:50.074 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012518800000000001 2024-09-18T04:39:50.074 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.074 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.074 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:50.074 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609854+0000", 2024-09-18T04:39:50.074 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6556000000000004e-05 2024-09-18T04:39:50.075 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.075 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:50.075 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.075 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.075 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.075 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:58 2.7 2:fc7c6cd3:ns::rbd_data.133e813ab5bc.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:50.075 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:39:50.075 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 67.397470893999994, 2024-09-18T04:39:50.075 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.027690566999999999, 2024-09-18T04:39:50.075 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:50.075 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:50.075 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:50.075 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:50.075 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:50.076 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 58 2024-09-18T04:39:50.076 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.076 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:50.076 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.076 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:50.076 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:39:50.076 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.076 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.076 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.076 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:50.076 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:39:50.076 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.077 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.077 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.077 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:50.077 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598082+0000", 2024-09-18T04:39:50.077 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:50.077 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.077 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.077 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:50.077 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604832+0000", 2024-09-18T04:39:50.077 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0067504510000000002 2024-09-18T04:39:50.077 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.077 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.077 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:50.077 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604852+0000", 2024-09-18T04:39:50.078 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0228999999999999e-05 2024-09-18T04:39:50.078 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.078 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.078 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:50.078 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604870+0000", 2024-09-18T04:39:50.078 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7952e-05 2024-09-18T04:39:50.078 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.078 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.078 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:50.078 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604936+0000", 2024-09-18T04:39:50.078 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5431000000000004e-05 2024-09-18T04:39:50.078 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.078 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.079 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:50.079 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605085+0000", 2024-09-18T04:39:50.079 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014910300000000001 2024-09-18T04:39:50.079 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.079 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.079 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:39:50.079 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605205+0000", 2024-09-18T04:39:50.079 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000119887 2024-09-18T04:39:50.079 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.079 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.079 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:50.079 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625586+0000", 2024-09-18T04:39:50.079 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020381399000000001 2024-09-18T04:39:50.079 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.080 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.080 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:50.080 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625628+0000", 2024-09-18T04:39:50.080 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1873999999999999e-05 2024-09-18T04:39:50.080 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.080 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.080 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:50.080 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625722+0000", 2024-09-18T04:39:50.080 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3787000000000002e-05 2024-09-18T04:39:50.080 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.080 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.080 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:50.080 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625774+0000", 2024-09-18T04:39:50.081 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2503999999999997e-05 2024-09-18T04:39:50.081 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.081 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:50.081 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.081 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.081 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.081 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:56 2.6 e51/17 2:72453c3e:ns::rbd_data.133e813ab5bc.000000000000000d:head v 51'10, mlcod=51'8)", 2024-09-18T04:39:50.081 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:39:50.081 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 67.390273785999995, 2024-09-18T04:39:50.081 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019766552999999999, 2024-09-18T04:39:50.081 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:50.081 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:50.081 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:50.081 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.082 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:50.082 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:39:50.082 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.082 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.082 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.082 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:50.082 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:39:50.082 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.082 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.082 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.082 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:50.082 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605279+0000", 2024-09-18T04:39:50.082 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:50.082 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.083 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.083 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:50.083 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609417+0000", 2024-09-18T04:39:50.083 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0041378059999999999 2024-09-18T04:39:50.083 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.083 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.083 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:50.083 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609443+0000", 2024-09-18T04:39:50.083 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6098e-05 2024-09-18T04:39:50.083 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.083 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.083 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:50.083 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609458+0000", 2024-09-18T04:39:50.083 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4671000000000001e-05 2024-09-18T04:39:50.084 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.084 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.084 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:50.084 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609553+0000", 2024-09-18T04:39:50.084 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5265000000000004e-05 2024-09-18T04:39:50.084 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.084 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.084 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:50.084 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609583+0000", 2024-09-18T04:39:50.084 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9839e-05 2024-09-18T04:39:50.084 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.084 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.084 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:50.085 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625004+0000", 2024-09-18T04:39:50.085 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015420639999999999 2024-09-18T04:39:50.085 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.085 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.085 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:50.085 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625047+0000", 2024-09-18T04:39:50.085 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3665999999999998e-05 2024-09-18T04:39:50.085 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.085 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:50.085 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.085 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.085 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.085 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:99 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:50.086 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.582718+0000", 2024-09-18T04:39:50.086 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 53.412836876999997, 2024-09-18T04:39:50.086 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019362575, 2024-09-18T04:39:50.086 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:50.086 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:50.086 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:50.086 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:50.086 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:50.086 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 99 2024-09-18T04:39:50.087 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.087 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:50.087 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.087 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:50.087 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582718+0000", 2024-09-18T04:39:50.087 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.087 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.087 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.087 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:50.087 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582718+0000", 2024-09-18T04:39:50.087 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.087 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.088 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.088 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:50.088 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582717+0000", 2024-09-18T04:39:50.088 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:50.088 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.088 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.088 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:50.088 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582720+0000", 2024-09-18T04:39:50.088 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2239999999999998e-06 2024-09-18T04:39:50.088 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.088 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.088 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:50.089 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582723+0000", 2024-09-18T04:39:50.089 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.258e-06 2024-09-18T04:39:50.089 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.089 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.089 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:50.089 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582726+0000", 2024-09-18T04:39:50.089 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8870000000000001e-06 2024-09-18T04:39:50.089 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.089 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.089 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:50.090 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.592018+0000", 2024-09-18T04:39:50.090 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0092926420000000003 2024-09-18T04:39:50.090 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.090 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.090 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:50.090 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.592214+0000", 2024-09-18T04:39:50.090 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000195267 2024-09-18T04:39:50.090 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.090 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.090 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:50.090 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.592913+0000", 2024-09-18T04:39:50.091 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00069924600000000005 2024-09-18T04:39:50.091 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.091 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.091 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:50.091 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597493+0000", 2024-09-18T04:39:50.091 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045797629999999997 2024-09-18T04:39:50.091 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.091 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.091 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:50.091 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.602009+0000", 2024-09-18T04:39:50.092 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045159559999999998 2024-09-18T04:39:50.092 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.092 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.092 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:50.092 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.602042+0000", 2024-09-18T04:39:50.092 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3084999999999999e-05 2024-09-18T04:39:50.092 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.092 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.093 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:50.093 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.602080+0000", 2024-09-18T04:39:50.093 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8600000000000003e-05 2024-09-18T04:39:50.093 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.093 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:50.093 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.093 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.093 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.093 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:104 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:50.094 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.583165+0000", 2024-09-18T04:39:50.094 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 53.412389578999999, 2024-09-18T04:39:50.094 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023679603, 2024-09-18T04:39:50.094 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:50.094 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:50.094 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:50.094 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:50.094 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:50.094 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 104 2024-09-18T04:39:50.094 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.095 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:50.095 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.095 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:50.095 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583165+0000", 2024-09-18T04:39:50.095 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.095 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.095 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.095 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:50.095 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583165+0000", 2024-09-18T04:39:50.095 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.095 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.095 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.096 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:50.096 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583164+0000", 2024-09-18T04:39:50.096 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:50.096 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.096 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.096 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:50.096 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583167+0000", 2024-09-18T04:39:50.096 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9210000000000001e-06 2024-09-18T04:39:50.096 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.096 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.096 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:50.097 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583171+0000", 2024-09-18T04:39:50.097 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9759999999999997e-06 2024-09-18T04:39:50.097 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.097 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.097 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:50.097 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583174+0000", 2024-09-18T04:39:50.097 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9960000000000002e-06 2024-09-18T04:39:50.097 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.097 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.097 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:50.097 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.594373+0000", 2024-09-18T04:39:50.097 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011198665999999999 2024-09-18T04:39:50.097 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.098 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.098 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:50.098 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.594434+0000", 2024-09-18T04:39:50.098 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.1446999999999996e-05 2024-09-18T04:39:50.098 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.098 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.098 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:50.098 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.594692+0000", 2024-09-18T04:39:50.098 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00025756599999999998 2024-09-18T04:39:50.098 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.098 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.098 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:50.098 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599302+0000", 2024-09-18T04:39:50.099 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0046099640000000002 2024-09-18T04:39:50.099 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.099 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.099 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:50.099 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606710+0000", 2024-09-18T04:39:50.099 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0074077739999999998 2024-09-18T04:39:50.099 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.099 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.099 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:50.099 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606798+0000", 2024-09-18T04:39:50.099 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8133999999999993e-05 2024-09-18T04:39:50.099 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.099 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.100 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:50.100 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606845+0000", 2024-09-18T04:39:50.100 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6653000000000002e-05 2024-09-18T04:39:50.100 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.100 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:50.100 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.100 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.100 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.100 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:105 2.0 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:50.100 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.583193+0000", 2024-09-18T04:39:50.100 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 53.412361959999998, 2024-09-18T04:39:50.100 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023713649, 2024-09-18T04:39:50.100 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:50.101 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:50.101 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:50.101 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:50.101 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:50.101 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 105 2024-09-18T04:39:50.101 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.101 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:50.101 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.101 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:50.101 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583193+0000", 2024-09-18T04:39:50.101 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.101 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.101 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.101 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:50.101 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583193+0000", 2024-09-18T04:39:50.102 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.102 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.102 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.102 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:50.102 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583192+0000", 2024-09-18T04:39:50.102 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:50.102 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.102 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.102 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:50.102 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583194+0000", 2024-09-18T04:39:50.102 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.305e-06 2024-09-18T04:39:50.102 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.102 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.102 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:50.103 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583198+0000", 2024-09-18T04:39:50.103 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9369999999999997e-06 2024-09-18T04:39:50.103 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.103 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.103 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:50.103 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583202+0000", 2024-09-18T04:39:50.103 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6480000000000001e-06 2024-09-18T04:39:50.103 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.103 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.103 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:50.103 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.595304+0000", 2024-09-18T04:39:50.103 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012102063999999999 2024-09-18T04:39:50.103 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.103 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.104 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:50.104 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.595390+0000", 2024-09-18T04:39:50.104 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5493000000000003e-05 2024-09-18T04:39:50.104 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.104 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.104 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:50.104 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.595509+0000", 2024-09-18T04:39:50.104 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011966199999999999 2024-09-18T04:39:50.104 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.104 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.104 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:50.104 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599358+0000", 2024-09-18T04:39:50.104 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0038486689999999999 2024-09-18T04:39:50.105 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.105 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.105 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:50.105 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606859+0000", 2024-09-18T04:39:50.105 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0075010789999999999 2024-09-18T04:39:50.105 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.105 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.105 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:50.105 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606887+0000", 2024-09-18T04:39:50.105 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7773e-05 2024-09-18T04:39:50.105 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.105 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.105 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:50.105 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606906+0000", 2024-09-18T04:39:50.106 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9417000000000001e-05 2024-09-18T04:39:50.106 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.106 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:50.106 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.106 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.106 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.106 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:106 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:50.106 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.584425+0000", 2024-09-18T04:39:50.106 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 53.411129664000001, 2024-09-18T04:39:50.106 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.022529293999999998, 2024-09-18T04:39:50.106 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:50.106 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:50.106 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:50.107 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:50.107 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:50.107 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 106 2024-09-18T04:39:50.107 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.107 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:50.107 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.107 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:50.107 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584425+0000", 2024-09-18T04:39:50.107 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.107 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.107 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.107 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:50.107 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584425+0000", 2024-09-18T04:39:50.108 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.108 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.108 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.108 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:50.108 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584424+0000", 2024-09-18T04:39:50.108 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:50.108 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.108 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.108 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:50.108 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584430+0000", 2024-09-18T04:39:50.108 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6389999999999997e-06 2024-09-18T04:39:50.108 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.108 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.108 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:50.109 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584456+0000", 2024-09-18T04:39:50.109 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6188999999999999e-05 2024-09-18T04:39:50.109 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.109 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.109 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:50.109 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584463+0000", 2024-09-18T04:39:50.109 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7340000000000002e-06 2024-09-18T04:39:50.109 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.109 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.109 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:50.109 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597537+0000", 2024-09-18T04:39:50.109 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013074161000000001 2024-09-18T04:39:50.109 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.110 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.110 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:50.110 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597598+0000", 2024-09-18T04:39:50.110 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0930000000000001e-05 2024-09-18T04:39:50.110 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.110 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.110 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:50.110 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597922+0000", 2024-09-18T04:39:50.110 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032473000000000002 2024-09-18T04:39:50.110 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.110 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.110 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:50.110 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.601103+0000", 2024-09-18T04:39:50.110 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0031809849999999999 2024-09-18T04:39:50.111 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.111 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.111 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:50.111 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606912+0000", 2024-09-18T04:39:50.111 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058083550000000003 2024-09-18T04:39:50.111 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.111 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.111 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:50.111 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606935+0000", 2024-09-18T04:39:50.111 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2759999999999999e-05 2024-09-18T04:39:50.111 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.111 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.111 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:50.112 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606954+0000", 2024-09-18T04:39:50.112 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9602999999999998e-05 2024-09-18T04:39:50.112 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.112 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:50.112 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.112 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.112 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.112 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:109 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:50.112 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.585231+0000", 2024-09-18T04:39:50.112 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 53.410323044000002, 2024-09-18T04:39:50.112 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.021767496000000001, 2024-09-18T04:39:50.112 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:50.113 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:50.113 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:50.113 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:50.113 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:50.113 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 109 2024-09-18T04:39:50.113 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.113 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:50.113 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.113 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:50.113 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585231+0000", 2024-09-18T04:39:50.113 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.113 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.113 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.114 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:50.114 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585231+0000", 2024-09-18T04:39:50.114 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.114 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.114 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.114 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:50.114 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585230+0000", 2024-09-18T04:39:50.114 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:50.114 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.114 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.114 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:50.114 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585237+0000", 2024-09-18T04:39:50.114 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.618e-06 2024-09-18T04:39:50.115 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.115 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.115 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:50.115 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585256+0000", 2024-09-18T04:39:50.115 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9048000000000001e-05 2024-09-18T04:39:50.115 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.115 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.115 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:50.115 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585274+0000", 2024-09-18T04:39:50.115 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8153000000000001e-05 2024-09-18T04:39:50.115 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.115 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.115 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:50.116 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.598284+0000", 2024-09-18T04:39:50.116 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013009682999999999 2024-09-18T04:39:50.116 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.116 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.116 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:50.116 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.598383+0000", 2024-09-18T04:39:50.116 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.8722999999999997e-05 2024-09-18T04:39:50.116 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.116 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.116 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:50.116 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.598829+0000", 2024-09-18T04:39:50.116 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00044623999999999998 2024-09-18T04:39:50.117 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.117 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.117 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:50.117 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.601149+0000", 2024-09-18T04:39:50.117 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0023204150000000002 2024-09-18T04:39:50.117 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.117 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.117 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:50.117 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606960+0000", 2024-09-18T04:39:50.117 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058106390000000003 2024-09-18T04:39:50.117 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.117 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.117 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:50.117 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606982+0000", 2024-09-18T04:39:50.118 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2252e-05 2024-09-18T04:39:50.118 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.118 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.118 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:50.118 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606999+0000", 2024-09-18T04:39:50.118 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6871000000000002e-05 2024-09-18T04:39:50.118 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.118 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:50.118 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.118 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.118 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.118 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:110 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:50.118 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.586640+0000", 2024-09-18T04:39:50.119 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 53.408914725000002, 2024-09-18T04:39:50.119 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020400911000000001, 2024-09-18T04:39:50.119 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:50.119 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:50.119 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:50.119 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:50.119 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:50.119 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 110 2024-09-18T04:39:50.119 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.119 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:50.119 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.119 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:50.119 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586640+0000", 2024-09-18T04:39:50.119 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.119 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.120 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.120 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:50.120 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586640+0000", 2024-09-18T04:39:50.120 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.120 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.120 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.120 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:50.120 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586639+0000", 2024-09-18T04:39:50.120 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:50.120 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.120 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.120 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:50.120 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586652+0000", 2024-09-18T04:39:50.121 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2482e-05 2024-09-18T04:39:50.121 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.121 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.121 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:50.121 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586660+0000", 2024-09-18T04:39:50.121 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1659999999999994e-06 2024-09-18T04:39:50.121 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.121 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.121 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:50.121 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586666+0000", 2024-09-18T04:39:50.121 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6900000000000003e-06 2024-09-18T04:39:50.121 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.121 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.121 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:50.122 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599381+0000", 2024-09-18T04:39:50.122 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012714328 2024-09-18T04:39:50.122 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.122 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.122 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:50.122 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599436+0000", 2024-09-18T04:39:50.122 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4802999999999997e-05 2024-09-18T04:39:50.122 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.122 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.122 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:50.122 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599629+0000", 2024-09-18T04:39:50.122 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000193921 2024-09-18T04:39:50.123 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.123 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.123 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:50.123 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.601174+0000", 2024-09-18T04:39:50.123 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001544857 2024-09-18T04:39:50.123 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.123 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.123 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:50.123 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.607004+0000", 2024-09-18T04:39:50.123 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058300820000000003 2024-09-18T04:39:50.123 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.123 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.123 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:50.123 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.607024+0000", 2024-09-18T04:39:50.124 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9888e-05 2024-09-18T04:39:50.124 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.124 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.124 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:50.124 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.607041+0000", 2024-09-18T04:39:50.124 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6450999999999999e-05 2024-09-18T04:39:50.124 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.124 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:50.124 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.124 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.124 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:50.124 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:50.126 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:39:50.126 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:39:50.127 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:39:50.128 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:39:50.128 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:39:50.128 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:39:50.128 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:39:50.128 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:39:50.128 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:39:50.128 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:50.120+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:39:50.226 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:50.224+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:39:50.280 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:50.281 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:50.281 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:39:50.281 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:50.291 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:39:50.301 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 2] out_osds: [4] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 2] 2024-09-18T04:39:50.301 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T04:39:50.302 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-18T04:39:50.302 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 1000 0.05 2024-09-18T04:39:50.327 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:50.320+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:50.393 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:50.394 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:50.394 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:39:50.394 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:39:50.394 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:50.403 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:39:50.429 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:50.424+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:39:50.506 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:50.506 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:39:50.506 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:39:50.506 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:39:50.506 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.506 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.5431.0:1 3.3 3:d80af3cb:::rbd_id.testrbdnbd17342:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-18T04:39:50.506 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:13.542818+0000", 2024-09-18T04:39:50.507 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 36.961039892000002, 2024-09-18T04:39:50.507 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048493400000000002, 2024-09-18T04:39:50.507 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:50.507 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:39:50.507 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:50.507 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.5431", 2024-09-18T04:39:50.507 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3147136083", 2024-09-18T04:39:50.507 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 1 2024-09-18T04:39:50.507 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.507 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:50.507 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.507 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:50.507 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542818+0000", 2024-09-18T04:39:50.508 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.508 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.508 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.508 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:50.508 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542818+0000", 2024-09-18T04:39:50.508 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.508 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.508 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.508 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:50.508 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542816+0000", 2024-09-18T04:39:50.508 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:50.508 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.508 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.509 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:50.509 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542828+0000", 2024-09-18T04:39:50.509 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2699e-05 2024-09-18T04:39:50.509 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.509 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.509 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:50.509 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542852+0000", 2024-09-18T04:39:50.509 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3176e-05 2024-09-18T04:39:50.509 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.509 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.509 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:50.509 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542874+0000", 2024-09-18T04:39:50.509 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1967000000000002e-05 2024-09-18T04:39:50.509 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.509 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.510 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:50.510 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542984+0000", 2024-09-18T04:39:50.510 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011026099999999999 2024-09-18T04:39:50.510 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.510 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.510 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:50.510 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543161+0000", 2024-09-18T04:39:50.510 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017720299999999999 2024-09-18T04:39:50.510 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.510 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.510 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:50.510 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543303+0000", 2024-09-18T04:39:50.510 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014174700000000001 2024-09-18T04:39:50.511 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.511 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:50.511 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.511 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.511 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.511 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.5431.0:2 3.3 3:d80af3cb:::rbd_id.testrbdnbd17342:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-18T04:39:50.511 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:13.543564+0000", 2024-09-18T04:39:50.511 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 36.960293405999998, 2024-09-18T04:39:50.511 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010399929999999999, 2024-09-18T04:39:50.511 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:50.511 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:39:50.511 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:50.511 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.5431", 2024-09-18T04:39:50.511 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3147136083", 2024-09-18T04:39:50.512 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 2 2024-09-18T04:39:50.512 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.512 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:50.512 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.512 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:50.512 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543564+0000", 2024-09-18T04:39:50.512 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.512 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.512 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.512 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:50.512 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543564+0000", 2024-09-18T04:39:50.512 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.512 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.512 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.512 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:50.513 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543563+0000", 2024-09-18T04:39:50.513 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:50.513 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.513 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.513 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:50.513 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543571+0000", 2024-09-18T04:39:50.513 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4270000000000008e-06 2024-09-18T04:39:50.513 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.513 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.513 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:50.513 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543591+0000", 2024-09-18T04:39:50.513 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.929e-05 2024-09-18T04:39:50.513 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.513 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.514 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:50.514 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543600+0000", 2024-09-18T04:39:50.514 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.1390000000000001e-06 2024-09-18T04:39:50.514 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.514 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.514 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:50.514 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543724+0000", 2024-09-18T04:39:50.514 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012458400000000001 2024-09-18T04:39:50.514 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.514 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.514 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:50.514 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543968+0000", 2024-09-18T04:39:50.514 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00024323199999999999 2024-09-18T04:39:50.514 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.514 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.515 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:50.515 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.544604+0000", 2024-09-18T04:39:50.515 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00063675799999999996 2024-09-18T04:39:50.515 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.515 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:50.515 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.515 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.515 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:50.515 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:50.516 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:39:50.529 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:50.520+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:50.621 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:50.621 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:50.621 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:39:50.621 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:50.623 INFO:teuthology.orchestra.run.smithi028.stdout:moved 0 / 34 (0%) 2024-09-18T04:39:50.623 INFO:teuthology.orchestra.run.smithi028.stdout:avg 6.8 2024-09-18T04:39:50.623 INFO:teuthology.orchestra.run.smithi028.stdout:stddev 1.46969 -> 1.46969 (expected baseline 2.33238) 2024-09-18T04:39:50.624 INFO:teuthology.orchestra.run.smithi028.stdout:min osd.3 with 5 -> 5 pgs (0.735294 -> 0.735294 * mean) 2024-09-18T04:39:50.624 INFO:teuthology.orchestra.run.smithi028.stdout:max osd.1 with 9 -> 9 pgs (1.32353 -> 1.32353 * mean) 2024-09-18T04:39:50.624 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:39:50.624 INFO:teuthology.orchestra.run.smithi028.stdout:oload 1000 2024-09-18T04:39:50.624 INFO:teuthology.orchestra.run.smithi028.stdout:max_change 0.05 2024-09-18T04:39:50.624 INFO:teuthology.orchestra.run.smithi028.stdout:max_change_osds 4 2024-09-18T04:39:50.624 INFO:teuthology.orchestra.run.smithi028.stdout:average_utilization 0.0009 2024-09-18T04:39:50.624 INFO:teuthology.orchestra.run.smithi028.stdout:overload_utilization 0.0092 2024-09-18T04:39:50.628 INFO:teuthology.orchestra.run.smithi028.stderr:no change 2024-09-18T04:39:50.631 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:39:50.632 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:50.628+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:50.639 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-09-18T04:39:50.639 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 1000 1.0 2024-09-18T04:39:50.731 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:50.728+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:50.753 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-18T04:39:50.784 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:50.784 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:50.784 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:39:50.784 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:39:50.784 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:50.801 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:39:50.832 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:50.828+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:50.909 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:50.909 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:39:50.909 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:39:50.909 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:39:50.909 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.909 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-18T04:39:50.909 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:39:50.910 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 136.43447914199999, 2024-09-18T04:39:50.910 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0095781189999999995, 2024-09-18T04:39:50.910 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:50.910 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:50.910 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:50.910 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:39:50.910 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:39:50.910 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 1 2024-09-18T04:39:50.910 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.910 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:50.910 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.910 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:50.910 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:39:50.910 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.911 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.911 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.911 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:50.911 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:39:50.911 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.911 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.911 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.911 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:50.911 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469862+0000", 2024-09-18T04:39:50.911 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:50.911 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.911 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.911 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:50.911 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469879+0000", 2024-09-18T04:39:50.912 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7431999999999998e-05 2024-09-18T04:39:50.912 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.912 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.912 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:50.912 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469896+0000", 2024-09-18T04:39:50.912 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6742999999999999e-05 2024-09-18T04:39:50.912 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.912 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.912 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:50.912 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469915+0000", 2024-09-18T04:39:50.912 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8991000000000002e-05 2024-09-18T04:39:50.912 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.912 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.913 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:50.913 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470397+0000", 2024-09-18T04:39:50.913 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048167799999999998 2024-09-18T04:39:50.913 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.913 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.913 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-09-18T04:39:50.913 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470412+0000", 2024-09-18T04:39:50.913 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5526e-05 2024-09-18T04:39:50.913 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.913 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.914 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:50.914 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473279+0000", 2024-09-18T04:39:50.914 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0028667469999999998 2024-09-18T04:39:50.914 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.914 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.914 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:50.914 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473441+0000", 2024-09-18T04:39:50.914 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000162082 2024-09-18T04:39:50.914 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.914 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.914 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:50.914 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473573+0000", 2024-09-18T04:39:50.914 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000132032 2024-09-18T04:39:50.914 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.915 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.915 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:50.915 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474826+0000", 2024-09-18T04:39:50.915 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001252717 2024-09-18T04:39:50.915 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.915 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.915 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:50.915 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479307+0000", 2024-09-18T04:39:50.915 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044813220000000003 2024-09-18T04:39:50.915 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.916 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.916 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:50.916 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479366+0000", 2024-09-18T04:39:50.916 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9024999999999998e-05 2024-09-18T04:39:50.916 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.916 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.916 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:50.916 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479443+0000", 2024-09-18T04:39:50.916 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6339000000000002e-05 2024-09-18T04:39:50.916 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.916 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:50.916 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.916 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.917 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.917 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-18T04:39:50.917 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:39:50.917 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 134.42147072200001, 2024-09-18T04:39:50.917 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082699560000000002, 2024-09-18T04:39:50.917 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:50.917 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:50.917 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:50.917 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:39:50.917 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:39:50.917 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 7 2024-09-18T04:39:50.917 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.917 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:50.917 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.918 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:50.918 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:39:50.918 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.918 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.918 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.918 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:50.918 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:39:50.918 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.918 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.918 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.918 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:50.918 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482870+0000", 2024-09-18T04:39:50.918 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999967 2024-09-18T04:39:50.918 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.918 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.919 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:50.919 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482889+0000", 2024-09-18T04:39:50.919 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.965e-05 2024-09-18T04:39:50.919 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.919 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.919 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:50.919 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482938+0000", 2024-09-18T04:39:50.919 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8548999999999999e-05 2024-09-18T04:39:50.919 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.919 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.919 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:50.919 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482963+0000", 2024-09-18T04:39:50.919 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5075999999999998e-05 2024-09-18T04:39:50.919 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.919 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.920 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:50.920 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483121+0000", 2024-09-18T04:39:50.920 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015819400000000001 2024-09-18T04:39:50.920 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.920 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.921 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op must wait for map", 2024-09-18T04:39:50.921 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483138+0000", 2024-09-18T04:39:50.921 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7068999999999999e-05 2024-09-18T04:39:50.921 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.921 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.921 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:50.921 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488666+0000", 2024-09-18T04:39:50.921 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0055282049999999996 2024-09-18T04:39:50.921 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.921 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.921 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:50.922 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488783+0000", 2024-09-18T04:39:50.922 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000116737 2024-09-18T04:39:50.922 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.922 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.922 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:50.922 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488991+0000", 2024-09-18T04:39:50.922 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020781400000000001 2024-09-18T04:39:50.922 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.922 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.922 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:50.922 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490554+0000", 2024-09-18T04:39:50.922 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0015628370000000001 2024-09-18T04:39:50.922 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.922 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.923 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:50.923 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491019+0000", 2024-09-18T04:39:50.923 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000465608 2024-09-18T04:39:50.923 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.923 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.923 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:50.923 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491087+0000", 2024-09-18T04:39:50.923 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7988e-05 2024-09-18T04:39:50.923 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.923 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.923 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:50.923 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491143+0000", 2024-09-18T04:39:50.923 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5538999999999998e-05 2024-09-18T04:39:50.923 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.924 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:50.924 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.924 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.924 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.924 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:34 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:50.924 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.261758+0000", 2024-09-18T04:39:50.924 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 68.642585570999998, 2024-09-18T04:39:50.924 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.042024191000000002, 2024-09-18T04:39:50.924 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:50.924 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:50.924 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:50.924 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:50.925 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:50.925 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 34 2024-09-18T04:39:50.925 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.925 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:50.925 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.925 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:50.925 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.261758+0000", 2024-09-18T04:39:50.925 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.925 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.925 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.926 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:50.926 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.261758+0000", 2024-09-18T04:39:50.926 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.926 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.926 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.926 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:50.926 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.261756+0000", 2024-09-18T04:39:50.926 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:50.926 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.926 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.926 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:50.926 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.270785+0000", 2024-09-18T04:39:50.926 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0090292340000000006 2024-09-18T04:39:50.927 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.927 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.927 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:50.927 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.270890+0000", 2024-09-18T04:39:50.927 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000104541 2024-09-18T04:39:50.927 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.927 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.927 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:50.927 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.270991+0000", 2024-09-18T04:39:50.927 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000100778 2024-09-18T04:39:50.927 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.927 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.928 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:50.928 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.271157+0000", 2024-09-18T04:39:50.928 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016653999999999999 2024-09-18T04:39:50.928 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.928 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.928 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:50.928 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.271591+0000", 2024-09-18T04:39:50.928 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00043409500000000001 2024-09-18T04:39:50.928 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.929 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.929 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:50.929 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272017+0000", 2024-09-18T04:39:50.929 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000425769 2024-09-18T04:39:50.929 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.929 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.929 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:50.929 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.294246+0000", 2024-09-18T04:39:50.929 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.022229237999999998 2024-09-18T04:39:50.929 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.929 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.929 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:50.929 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303571+0000", 2024-09-18T04:39:50.930 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0093250239999999995 2024-09-18T04:39:50.930 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.930 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.930 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:50.930 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303705+0000", 2024-09-18T04:39:50.930 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000133831 2024-09-18T04:39:50.930 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.930 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.930 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:50.930 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303782+0000", 2024-09-18T04:39:50.930 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7156000000000005e-05 2024-09-18T04:39:50.930 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.930 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:50.930 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.931 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.931 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.931 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:36 2.7 e51/32 2:fe1b02ad:::rbd_data.125fc09df6d1.0000000000000002:head v 51'1, mlcod=0'0)", 2024-09-18T04:39:50.931 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.273441+0000", 2024-09-18T04:39:50.931 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 68.630902590000005, 2024-09-18T04:39:50.931 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.032400768000000003, 2024-09-18T04:39:50.931 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:50.931 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:50.931 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:50.931 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.931 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:50.931 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273441+0000", 2024-09-18T04:39:50.932 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.932 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.932 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.932 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:50.932 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273441+0000", 2024-09-18T04:39:50.932 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.932 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.932 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.932 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:50.932 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273440+0000", 2024-09-18T04:39:50.932 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:50.932 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.932 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.933 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:50.933 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278723+0000", 2024-09-18T04:39:50.933 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0052829909999999999 2024-09-18T04:39:50.933 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.933 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.933 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:50.933 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278752+0000", 2024-09-18T04:39:50.933 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9178000000000001e-05 2024-09-18T04:39:50.933 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.933 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.933 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:50.933 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278761+0000", 2024-09-18T04:39:50.933 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7900000000000005e-06 2024-09-18T04:39:50.934 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.934 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.934 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:50.934 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278864+0000", 2024-09-18T04:39:50.934 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000102699 2024-09-18T04:39:50.934 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.934 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.934 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:50.934 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278900+0000", 2024-09-18T04:39:50.935 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6001999999999999e-05 2024-09-18T04:39:50.935 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.935 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.935 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:50.935 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.305793+0000", 2024-09-18T04:39:50.935 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.026893344 2024-09-18T04:39:50.935 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.935 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.935 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:50.935 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.305842+0000", 2024-09-18T04:39:50.935 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8720000000000001e-05 2024-09-18T04:39:50.935 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.935 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:50.936 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.936 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.936 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.936 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:39 2.7 e51/32 2:e9c909c5:::rbd_data.125fc09df6d1.0000000000000005:head v 51'2, mlcod=0'0)", 2024-09-18T04:39:50.936 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.283175+0000", 2024-09-18T04:39:50.936 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 68.621168901000004, 2024-09-18T04:39:50.936 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.022864102000000001, 2024-09-18T04:39:50.936 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:50.936 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:50.936 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:50.936 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.936 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:50.936 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283175+0000", 2024-09-18T04:39:50.936 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.937 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.937 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.937 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:50.937 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283175+0000", 2024-09-18T04:39:50.937 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.937 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.937 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.937 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:50.937 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283173+0000", 2024-09-18T04:39:50.937 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:50.937 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.937 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.937 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:50.938 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.285951+0000", 2024-09-18T04:39:50.938 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0027784540000000001 2024-09-18T04:39:50.938 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.938 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.938 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:50.939 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.285970+0000", 2024-09-18T04:39:50.939 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8855000000000002e-05 2024-09-18T04:39:50.939 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.939 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.939 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:50.939 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.285984+0000", 2024-09-18T04:39:50.939 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4252e-05 2024-09-18T04:39:50.939 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.939 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.939 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:50.939 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.286041+0000", 2024-09-18T04:39:50.940 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6431999999999998e-05 2024-09-18T04:39:50.940 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.940 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.940 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:50.940 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.286086+0000", 2024-09-18T04:39:50.940 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5423000000000002e-05 2024-09-18T04:39:50.940 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.940 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.940 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:50.940 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.305998+0000", 2024-09-18T04:39:50.940 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019911697999999999 2024-09-18T04:39:50.940 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.940 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.941 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:50.941 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306039+0000", 2024-09-18T04:39:50.941 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0973999999999998e-05 2024-09-18T04:39:50.941 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.941 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:50.941 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.941 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.941 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.941 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:50 2.3 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:50.941 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.312563+0000", 2024-09-18T04:39:50.941 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 68.591780873999994, 2024-09-18T04:39:50.941 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.032636001999999997, 2024-09-18T04:39:50.941 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:50.942 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:50.942 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:50.942 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:50.942 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:50.942 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 50 2024-09-18T04:39:50.942 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.942 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:50.942 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.942 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:50.942 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.312563+0000", 2024-09-18T04:39:50.942 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.942 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.942 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.942 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:50.943 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.312563+0000", 2024-09-18T04:39:50.943 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.943 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.943 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.943 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:50.943 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.312561+0000", 2024-09-18T04:39:50.943 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:50.943 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.943 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.943 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:50.943 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321709+0000", 2024-09-18T04:39:50.943 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0091486650000000003 2024-09-18T04:39:50.943 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.943 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.944 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:50.944 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321739+0000", 2024-09-18T04:39:50.944 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9666e-05 2024-09-18T04:39:50.944 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.944 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.944 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:50.944 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321751+0000", 2024-09-18T04:39:50.944 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2086000000000001e-05 2024-09-18T04:39:50.944 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.944 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.944 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:50.944 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321833+0000", 2024-09-18T04:39:50.944 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1842999999999996e-05 2024-09-18T04:39:50.945 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.945 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.945 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:50.945 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321981+0000", 2024-09-18T04:39:50.945 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000147441 2024-09-18T04:39:50.945 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.945 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.945 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:50.945 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.322109+0000", 2024-09-18T04:39:50.945 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012851200000000001 2024-09-18T04:39:50.945 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.945 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.946 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:50.946 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.332829+0000", 2024-09-18T04:39:50.946 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010720225999999999 2024-09-18T04:39:50.946 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.946 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.946 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:50.946 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.345039+0000", 2024-09-18T04:39:50.946 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012209831000000001 2024-09-18T04:39:50.946 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.946 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.946 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:50.946 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.345120+0000", 2024-09-18T04:39:50.946 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1145999999999999e-05 2024-09-18T04:39:50.946 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.947 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.947 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:50.947 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.345199+0000", 2024-09-18T04:39:50.947 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8546000000000003e-05 2024-09-18T04:39:50.947 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.947 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:50.947 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.947 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.947 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.947 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:30 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:50.947 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.516122+0000", 2024-09-18T04:39:50.947 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 68.388221712000004, 2024-09-18T04:39:50.947 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0091865030000000004, 2024-09-18T04:39:50.947 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:50.948 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:50.948 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:50.948 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:50.948 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:50.948 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 30 2024-09-18T04:39:50.948 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.948 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:50.948 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.948 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:50.948 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516122+0000", 2024-09-18T04:39:50.948 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.948 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.948 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.949 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:50.949 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516122+0000", 2024-09-18T04:39:50.949 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.949 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.949 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.949 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:50.949 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516120+0000", 2024-09-18T04:39:50.949 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:50.949 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.949 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.949 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:50.949 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516129+0000", 2024-09-18T04:39:50.949 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.85e-06 2024-09-18T04:39:50.949 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.950 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.950 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:50.950 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516155+0000", 2024-09-18T04:39:50.950 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5882000000000001e-05 2024-09-18T04:39:50.950 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.950 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.950 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:50.950 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516175+0000", 2024-09-18T04:39:50.950 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.022e-05 2024-09-18T04:39:50.950 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.950 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.950 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:50.950 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519780+0000", 2024-09-18T04:39:50.950 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003604187 2024-09-18T04:39:50.951 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.951 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.951 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:50.951 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519964+0000", 2024-09-18T04:39:50.951 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00018391999999999999 2024-09-18T04:39:50.951 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.951 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.951 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:50.951 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.520794+0000", 2024-09-18T04:39:50.951 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00083078100000000005 2024-09-18T04:39:50.951 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.951 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.951 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:50.951 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.524129+0000", 2024-09-18T04:39:50.952 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033342419999999998 2024-09-18T04:39:50.952 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.952 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.952 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:50.952 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525239+0000", 2024-09-18T04:39:50.952 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001110778 2024-09-18T04:39:50.952 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.952 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.952 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:50.952 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525277+0000", 2024-09-18T04:39:50.952 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7596999999999999e-05 2024-09-18T04:39:50.952 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.952 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.953 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:50.953 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525308+0000", 2024-09-18T04:39:50.953 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.1535999999999998e-05 2024-09-18T04:39:50.953 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.953 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:50.953 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.953 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.953 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.953 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:32 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:50.953 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.519217+0000", 2024-09-18T04:39:50.953 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 68.385126392000004, 2024-09-18T04:39:50.953 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082266370000000002, 2024-09-18T04:39:50.953 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:50.953 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:50.953 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:50.954 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:50.954 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:50.954 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 32 2024-09-18T04:39:50.954 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.954 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:50.954 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.954 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:50.954 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519217+0000", 2024-09-18T04:39:50.954 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.954 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.954 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.954 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:50.954 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519217+0000", 2024-09-18T04:39:50.954 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.955 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.955 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.955 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:50.955 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519215+0000", 2024-09-18T04:39:50.955 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:50.955 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.955 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.955 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:50.955 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519229+0000", 2024-09-18T04:39:50.955 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3329999999999999e-05 2024-09-18T04:39:50.955 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.955 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.956 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:50.956 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519268+0000", 2024-09-18T04:39:50.956 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9533000000000002e-05 2024-09-18T04:39:50.956 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.956 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.956 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:50.956 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519285+0000", 2024-09-18T04:39:50.956 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6446000000000001e-05 2024-09-18T04:39:50.956 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.956 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.956 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:50.956 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.522358+0000", 2024-09-18T04:39:50.956 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030734310000000002 2024-09-18T04:39:50.956 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.957 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.957 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:50.957 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.522551+0000", 2024-09-18T04:39:50.957 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019317600000000001 2024-09-18T04:39:50.957 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.957 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.957 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:50.957 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.523262+0000", 2024-09-18T04:39:50.957 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00071069999999999998 2024-09-18T04:39:50.957 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.957 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.957 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:50.957 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525318+0000", 2024-09-18T04:39:50.957 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0020561469999999999 2024-09-18T04:39:50.958 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.958 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.958 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:50.958 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.527376+0000", 2024-09-18T04:39:50.958 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0020584969999999998 2024-09-18T04:39:50.958 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.958 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.958 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:50.958 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.527416+0000", 2024-09-18T04:39:50.958 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9054000000000003e-05 2024-09-18T04:39:50.958 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.958 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.959 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:50.959 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.527444+0000", 2024-09-18T04:39:50.959 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8347000000000002e-05 2024-09-18T04:39:50.959 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.959 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:50.959 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.959 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.959 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.959 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:42 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:50.959 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.551160+0000", 2024-09-18T04:39:50.959 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 68.353183999999999, 2024-09-18T04:39:50.959 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010692757000000001, 2024-09-18T04:39:50.960 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:50.960 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:50.960 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:50.960 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:50.960 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:50.960 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 42 2024-09-18T04:39:50.960 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.960 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:50.960 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.960 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:50.960 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551160+0000", 2024-09-18T04:39:50.960 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.960 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.961 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.961 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:50.961 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551160+0000", 2024-09-18T04:39:50.961 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.961 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.961 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.961 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:50.961 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551159+0000", 2024-09-18T04:39:50.961 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:50.961 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.961 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.961 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:50.961 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551166+0000", 2024-09-18T04:39:50.962 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1029999999999996e-06 2024-09-18T04:39:50.962 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.962 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.962 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:50.962 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551177+0000", 2024-09-18T04:39:50.962 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1384e-05 2024-09-18T04:39:50.962 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.962 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.962 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:50.962 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551185+0000", 2024-09-18T04:39:50.962 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1990000000000006e-06 2024-09-18T04:39:50.962 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.962 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.963 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:50.963 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552021+0000", 2024-09-18T04:39:50.963 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00083574400000000003 2024-09-18T04:39:50.963 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.963 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.963 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:50.963 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552339+0000", 2024-09-18T04:39:50.963 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031771399999999997 2024-09-18T04:39:50.963 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.963 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.963 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:50.963 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.553140+0000", 2024-09-18T04:39:50.963 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00080088100000000003 2024-09-18T04:39:50.963 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.964 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.964 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:50.964 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.554585+0000", 2024-09-18T04:39:50.964 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001445801 2024-09-18T04:39:50.964 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.964 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.964 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:50.964 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.561711+0000", 2024-09-18T04:39:50.964 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071260109999999998 2024-09-18T04:39:50.964 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.964 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.964 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:50.964 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.561790+0000", 2024-09-18T04:39:50.965 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8967000000000005e-05 2024-09-18T04:39:50.965 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.965 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.965 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:50.965 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.561852+0000", 2024-09-18T04:39:50.965 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2045999999999995e-05 2024-09-18T04:39:50.965 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.965 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:50.965 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.965 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.965 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.965 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:58 2.7 e51/32 2:fc7c6cd3:ns::rbd_data.133e813ab5bc.000000000000000e:head v 51'7, mlcod=51'5)", 2024-09-18T04:39:50.965 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.605333+0000", 2024-09-18T04:39:50.965 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 68.299010795000001, 2024-09-18T04:39:50.966 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020085902999999999, 2024-09-18T04:39:50.966 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:50.966 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:50.966 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:50.966 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.966 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:50.966 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605333+0000", 2024-09-18T04:39:50.966 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.966 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.966 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.966 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:50.966 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605333+0000", 2024-09-18T04:39:50.966 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.966 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.967 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.967 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:50.967 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605332+0000", 2024-09-18T04:39:50.967 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:50.967 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.967 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.967 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:50.967 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608667+0000", 2024-09-18T04:39:50.967 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033350300000000001 2024-09-18T04:39:50.967 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.967 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.967 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:50.967 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608706+0000", 2024-09-18T04:39:50.968 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8951e-05 2024-09-18T04:39:50.968 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.968 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.968 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:50.968 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608725+0000", 2024-09-18T04:39:50.968 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9675999999999999e-05 2024-09-18T04:39:50.968 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.968 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.968 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:50.968 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608798+0000", 2024-09-18T04:39:50.968 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3135999999999999e-05 2024-09-18T04:39:50.968 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.968 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.969 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:50.969 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608824+0000", 2024-09-18T04:39:50.969 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5284000000000001e-05 2024-09-18T04:39:50.969 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.969 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.969 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:50.969 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625370+0000", 2024-09-18T04:39:50.969 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016545975000000001 2024-09-18T04:39:50.969 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.969 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.969 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:50.969 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625419+0000", 2024-09-18T04:39:50.969 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9076999999999998e-05 2024-09-18T04:39:50.970 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.970 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:50.970 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.970 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.970 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.970 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:611 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:50.970 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.069561+0000", 2024-09-18T04:39:50.970 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 59.834782240999999, 2024-09-18T04:39:50.970 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0072330299999999997, 2024-09-18T04:39:50.970 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:50.970 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:50.970 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:50.970 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:39:50.971 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:50.971 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 611 2024-09-18T04:39:50.971 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.971 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:50.971 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.971 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:50.971 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.069561+0000", 2024-09-18T04:39:50.971 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.971 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.971 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.971 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:50.971 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.069561+0000", 2024-09-18T04:39:50.971 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.972 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.972 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.972 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:50.972 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.069559+0000", 2024-09-18T04:39:50.972 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:50.972 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.972 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.972 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:50.972 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070801+0000", 2024-09-18T04:39:50.973 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0012423359999999999 2024-09-18T04:39:50.973 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.973 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.973 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:50.973 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070828+0000", 2024-09-18T04:39:50.973 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6387e-05 2024-09-18T04:39:50.973 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.973 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.973 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:50.973 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070840+0000", 2024-09-18T04:39:50.973 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2689e-05 2024-09-18T04:39:50.973 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.974 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.974 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:50.974 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070990+0000", 2024-09-18T04:39:50.974 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014946800000000001 2024-09-18T04:39:50.974 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.974 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.974 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:50.974 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.071128+0000", 2024-09-18T04:39:50.974 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013796699999999999 2024-09-18T04:39:50.974 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.974 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.974 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:50.974 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.071318+0000", 2024-09-18T04:39:50.974 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000190106 2024-09-18T04:39:50.975 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.975 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.975 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:50.975 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.075711+0000", 2024-09-18T04:39:50.975 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.004393002 2024-09-18T04:39:50.975 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.975 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.975 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:50.975 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.076665+0000", 2024-09-18T04:39:50.975 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00095390399999999999 2024-09-18T04:39:50.975 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.975 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.975 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:50.975 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.076742+0000", 2024-09-18T04:39:50.976 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7112000000000003e-05 2024-09-18T04:39:50.976 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.976 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.976 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:50.976 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.076794+0000", 2024-09-18T04:39:50.976 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2475999999999997e-05 2024-09-18T04:39:50.976 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.976 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:50.976 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.976 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.976 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.976 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:687 2.3 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:50.976 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.487994+0000", 2024-09-18T04:39:50.977 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 59.416349257999997, 2024-09-18T04:39:50.977 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0074167690000000001, 2024-09-18T04:39:50.977 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:50.977 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:50.977 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:50.977 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:39:50.977 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:50.977 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 687 2024-09-18T04:39:50.977 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.977 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:50.977 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.977 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:50.977 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.487994+0000", 2024-09-18T04:39:50.978 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.978 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.978 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.978 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:50.978 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.487994+0000", 2024-09-18T04:39:50.978 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.978 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.978 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.978 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:50.978 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.487993+0000", 2024-09-18T04:39:50.978 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:50.978 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.978 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.978 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:50.979 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.488979+0000", 2024-09-18T04:39:50.979 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00098620700000000001 2024-09-18T04:39:50.979 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.979 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.979 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:50.979 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489002+0000", 2024-09-18T04:39:50.979 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2500000000000001e-05 2024-09-18T04:39:50.979 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.979 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.979 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:50.979 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489020+0000", 2024-09-18T04:39:50.979 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7691000000000001e-05 2024-09-18T04:39:50.979 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.980 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.980 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:50.980 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489091+0000", 2024-09-18T04:39:50.980 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1307e-05 2024-09-18T04:39:50.980 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.980 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.980 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:50.980 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489219+0000", 2024-09-18T04:39:50.980 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012766000000000001 2024-09-18T04:39:50.980 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.980 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.980 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:50.980 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489387+0000", 2024-09-18T04:39:50.980 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016855400000000001 2024-09-18T04:39:50.981 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.981 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.981 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:50.981 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495243+0000", 2024-09-18T04:39:50.981 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058563189999999996 2024-09-18T04:39:50.981 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.981 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.981 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:50.981 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495266+0000", 2024-09-18T04:39:50.981 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2810999999999999e-05 2024-09-18T04:39:50.981 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.981 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.981 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:50.981 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495332+0000", 2024-09-18T04:39:50.981 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5469999999999995e-05 2024-09-18T04:39:50.982 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.982 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.982 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:50.982 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495411+0000", 2024-09-18T04:39:50.982 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9395999999999997e-05 2024-09-18T04:39:50.982 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.982 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:50.982 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.982 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.982 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.982 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:689 2.3 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:50.982 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.500070+0000", 2024-09-18T04:39:50.982 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 59.404273441999997, 2024-09-18T04:39:50.982 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071096650000000003, 2024-09-18T04:39:50.982 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:50.983 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:50.983 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:50.983 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:39:50.983 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:50.983 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 689 2024-09-18T04:39:50.983 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.983 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:50.983 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.983 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:50.983 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.500070+0000", 2024-09-18T04:39:50.983 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.983 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.983 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.983 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:50.984 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.500070+0000", 2024-09-18T04:39:50.984 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.984 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.984 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.984 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:50.984 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.500069+0000", 2024-09-18T04:39:50.984 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:50.984 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.984 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.984 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:50.984 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501082+0000", 2024-09-18T04:39:50.984 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010136570000000001 2024-09-18T04:39:50.984 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.985 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.985 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:50.985 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501104+0000", 2024-09-18T04:39:50.985 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1883e-05 2024-09-18T04:39:50.985 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.985 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.985 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:50.985 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501115+0000", 2024-09-18T04:39:50.985 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0414000000000001e-05 2024-09-18T04:39:50.985 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.985 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.985 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:50.985 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501178+0000", 2024-09-18T04:39:50.985 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3128999999999998e-05 2024-09-18T04:39:50.986 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.986 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.986 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:50.986 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501257+0000", 2024-09-18T04:39:50.986 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9592999999999995e-05 2024-09-18T04:39:50.986 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.986 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.986 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:50.986 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501366+0000", 2024-09-18T04:39:50.986 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000108643 2024-09-18T04:39:50.986 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.986 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.986 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:50.987 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.504041+0000", 2024-09-18T04:39:50.987 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0026750889999999999 2024-09-18T04:39:50.987 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.987 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.987 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:50.987 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.507097+0000", 2024-09-18T04:39:50.987 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030556070000000001 2024-09-18T04:39:50.987 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.987 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.987 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:50.987 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.507140+0000", 2024-09-18T04:39:50.987 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3745000000000002e-05 2024-09-18T04:39:50.987 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.988 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.988 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:50.988 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.507180+0000", 2024-09-18T04:39:50.988 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9475999999999999e-05 2024-09-18T04:39:50.988 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.988 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:50.988 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.988 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.988 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.988 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4972.0:13 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:50.988 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.948263+0000", 2024-09-18T04:39:50.988 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 58.956080651000001, 2024-09-18T04:39:50.988 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0091358970000000005, 2024-09-18T04:39:50.989 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:50.989 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:39:50.989 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:50.989 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4972", 2024-09-18T04:39:50.989 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2280251472", 2024-09-18T04:39:50.989 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 13 2024-09-18T04:39:50.989 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.989 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:50.989 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.989 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:50.989 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948263+0000", 2024-09-18T04:39:50.989 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.989 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.989 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.990 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:50.990 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948263+0000", 2024-09-18T04:39:50.990 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.990 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.990 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.990 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:50.990 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948261+0000", 2024-09-18T04:39:50.990 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:50.990 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.990 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.990 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:50.990 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948273+0000", 2024-09-18T04:39:50.990 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1576999999999999e-05 2024-09-18T04:39:50.990 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.991 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.991 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:50.991 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948305+0000", 2024-09-18T04:39:50.991 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2209000000000003e-05 2024-09-18T04:39:50.991 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.991 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.991 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:50.991 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948325+0000", 2024-09-18T04:39:50.991 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.022e-05 2024-09-18T04:39:50.991 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.991 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.991 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:50.991 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948455+0000", 2024-09-18T04:39:50.991 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000129624 2024-09-18T04:39:50.992 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.992 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.992 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:50.992 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948598+0000", 2024-09-18T04:39:50.992 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014376699999999999 2024-09-18T04:39:50.992 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.992 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.992 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:50.992 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.957399+0000", 2024-09-18T04:39:50.992 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0088004299999999997 2024-09-18T04:39:50.992 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.992 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:50.992 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:50.993 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.993 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.993 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:101 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:50.993 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.576367+0000", 2024-09-18T04:39:50.993 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 54.327976943000003, 2024-09-18T04:39:50.993 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0078671020000000008, 2024-09-18T04:39:50.993 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:50.993 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:50.993 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:50.993 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:50.993 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:50.993 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 101 2024-09-18T04:39:50.993 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.993 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:50.994 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.994 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:50.994 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576367+0000", 2024-09-18T04:39:50.994 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.994 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.994 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.994 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:50.994 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576367+0000", 2024-09-18T04:39:50.994 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:50.994 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.994 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.994 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:50.994 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576364+0000", 2024-09-18T04:39:50.994 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999971 2024-09-18T04:39:50.995 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.997 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.997 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:50.997 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576378+0000", 2024-09-18T04:39:50.997 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3879999999999999e-05 2024-09-18T04:39:50.997 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.997 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.997 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:50.997 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576410+0000", 2024-09-18T04:39:50.997 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2286000000000002e-05 2024-09-18T04:39:50.997 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.998 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.998 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:50.998 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576425+0000", 2024-09-18T04:39:50.998 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5290000000000001e-05 2024-09-18T04:39:50.998 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.998 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.998 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:50.998 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576551+0000", 2024-09-18T04:39:50.998 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012555199999999999 2024-09-18T04:39:50.998 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.998 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.998 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:50.998 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576719+0000", 2024-09-18T04:39:50.998 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016787400000000001 2024-09-18T04:39:50.999 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.999 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.999 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:50.999 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576951+0000", 2024-09-18T04:39:50.999 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00023252400000000001 2024-09-18T04:39:50.999 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.999 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.999 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:50.999 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.581459+0000", 2024-09-18T04:39:50.999 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045074410000000001 2024-09-18T04:39:50.999 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:50.999 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:50.999 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:50.999 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584076+0000", 2024-09-18T04:39:51.000 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0026173860000000002 2024-09-18T04:39:51.000 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.000 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.000 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:51.000 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584160+0000", 2024-09-18T04:39:51.000 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4191999999999996e-05 2024-09-18T04:39:51.000 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.000 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.000 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:51.000 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584234+0000", 2024-09-18T04:39:51.000 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.339e-05 2024-09-18T04:39:51.000 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:51.000 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:51.000 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:51.001 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.001 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.001 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:44 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:51.001 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.550448+0000", 2024-09-18T04:39:51.001 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 50.353895754, 2024-09-18T04:39:51.001 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082117590000000008, 2024-09-18T04:39:51.001 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:51.001 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:51.001 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:51.001 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:39:51.001 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:51.001 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 44 2024-09-18T04:39:51.001 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.001 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:51.002 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.002 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:51.002 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.550448+0000", 2024-09-18T04:39:51.002 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:51.002 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.002 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.002 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:51.002 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.550448+0000", 2024-09-18T04:39:51.002 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:51.002 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.002 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.002 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:51.002 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.550445+0000", 2024-09-18T04:39:51.002 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999971 2024-09-18T04:39:51.003 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.003 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.003 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:51.003 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.551904+0000", 2024-09-18T04:39:51.003 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001458322 2024-09-18T04:39:51.003 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.003 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.003 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:51.003 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.551938+0000", 2024-09-18T04:39:51.003 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4359000000000001e-05 2024-09-18T04:39:51.003 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.003 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.004 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:51.004 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.551959+0000", 2024-09-18T04:39:51.004 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1403000000000002e-05 2024-09-18T04:39:51.004 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.004 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.004 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:51.004 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.552049+0000", 2024-09-18T04:39:51.004 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9970999999999996e-05 2024-09-18T04:39:51.004 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.004 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.004 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:51.004 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.552190+0000", 2024-09-18T04:39:51.004 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000140271 2024-09-18T04:39:51.004 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.005 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.005 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:51.005 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.552385+0000", 2024-09-18T04:39:51.005 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019498700000000001 2024-09-18T04:39:51.005 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.005 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.005 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:51.005 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.556308+0000", 2024-09-18T04:39:51.005 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00392333 2024-09-18T04:39:51.005 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.005 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.005 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:51.005 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.558479+0000", 2024-09-18T04:39:51.005 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0021715889999999998 2024-09-18T04:39:51.006 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.006 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.006 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:51.006 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.558616+0000", 2024-09-18T04:39:51.006 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013613399999999999 2024-09-18T04:39:51.006 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.006 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.006 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:51.006 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.558660+0000", 2024-09-18T04:39:51.006 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4053e-05 2024-09-18T04:39:51.006 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:51.006 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:51.006 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:51.006 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.007 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.007 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:46 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:51.007 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.564089+0000", 2024-09-18T04:39:51.007 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 50.340254625, 2024-09-18T04:39:51.007 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0069437090000000002, 2024-09-18T04:39:51.007 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:51.007 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:51.007 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:51.007 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:39:51.007 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:51.007 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 46 2024-09-18T04:39:51.007 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.007 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:51.008 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.008 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:51.008 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.564089+0000", 2024-09-18T04:39:51.008 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:51.008 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.008 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.008 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:51.008 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.564089+0000", 2024-09-18T04:39:51.008 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:51.008 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.008 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.008 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:51.008 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.564087+0000", 2024-09-18T04:39:51.009 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:51.009 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.009 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.009 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:51.009 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565126+0000", 2024-09-18T04:39:51.009 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001038902 2024-09-18T04:39:51.009 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.009 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.009 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:51.009 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565148+0000", 2024-09-18T04:39:51.009 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1911000000000001e-05 2024-09-18T04:39:51.009 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.009 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.009 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:51.010 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565157+0000", 2024-09-18T04:39:51.010 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6220000000000004e-06 2024-09-18T04:39:51.010 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.010 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.010 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:51.010 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565261+0000", 2024-09-18T04:39:51.010 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000103063 2024-09-18T04:39:51.010 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.010 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.010 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:51.010 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565394+0000", 2024-09-18T04:39:51.010 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000133032 2024-09-18T04:39:51.010 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.010 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.010 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:51.011 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565564+0000", 2024-09-18T04:39:51.011 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017068000000000001 2024-09-18T04:39:51.011 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.011 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.011 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:51.011 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.570556+0000", 2024-09-18T04:39:51.011 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0049920620000000002 2024-09-18T04:39:51.011 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.011 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.011 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:51.011 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.570885+0000", 2024-09-18T04:39:51.011 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032827900000000001 2024-09-18T04:39:51.011 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.011 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.012 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:51.012 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.570959+0000", 2024-09-18T04:39:51.012 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3955999999999995e-05 2024-09-18T04:39:51.012 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.012 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.012 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:51.012 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.571033+0000", 2024-09-18T04:39:51.012 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4191999999999997e-05 2024-09-18T04:39:51.012 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:51.012 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:51.012 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:51.012 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.012 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.013 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:100 2.4 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:51.013 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.856389+0000", 2024-09-18T04:39:51.013 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 50.047954513000001, 2024-09-18T04:39:51.013 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0081480800000000003, 2024-09-18T04:39:51.013 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:51.013 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:51.013 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:51.013 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:39:51.013 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:51.013 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 100 2024-09-18T04:39:51.013 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.013 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:51.013 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.014 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:51.014 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.856389+0000", 2024-09-18T04:39:51.014 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:51.014 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.014 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.014 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:51.014 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.856389+0000", 2024-09-18T04:39:51.015 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:51.015 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.015 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.015 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:51.015 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.856388+0000", 2024-09-18T04:39:51.015 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:51.015 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.015 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.015 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:51.015 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857295+0000", 2024-09-18T04:39:51.015 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00090725599999999995 2024-09-18T04:39:51.015 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.015 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.015 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:51.016 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857311+0000", 2024-09-18T04:39:51.016 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6092000000000001e-05 2024-09-18T04:39:51.016 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.016 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.016 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:51.016 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857319+0000", 2024-09-18T04:39:51.016 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9200000000000004e-06 2024-09-18T04:39:51.016 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.016 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.016 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:51.016 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857388+0000", 2024-09-18T04:39:51.016 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8591999999999997e-05 2024-09-18T04:39:51.016 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.017 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.017 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:51.017 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857535+0000", 2024-09-18T04:39:51.017 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014684299999999999 2024-09-18T04:39:51.017 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.017 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.017 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:51.017 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857747+0000", 2024-09-18T04:39:51.017 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000212361 2024-09-18T04:39:51.017 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.017 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.017 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:51.017 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.860831+0000", 2024-09-18T04:39:51.018 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030841789999999998 2024-09-18T04:39:51.018 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.018 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.018 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:51.018 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.864361+0000", 2024-09-18T04:39:51.018 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0035298780000000002 2024-09-18T04:39:51.018 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.018 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.018 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:51.018 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.864456+0000", 2024-09-18T04:39:51.018 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4557999999999998e-05 2024-09-18T04:39:51.018 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.018 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.019 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:51.019 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.864537+0000", 2024-09-18T04:39:51.019 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1577999999999997e-05 2024-09-18T04:39:51.019 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:51.019 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:51.019 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:51.019 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.019 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.019 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:102 2.4 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:51.019 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.869773+0000", 2024-09-18T04:39:51.019 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 50.034570856000002, 2024-09-18T04:39:51.019 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0084430359999999992, 2024-09-18T04:39:51.019 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:51.019 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:51.020 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:51.020 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:39:51.020 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:51.020 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 102 2024-09-18T04:39:51.020 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.020 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:51.020 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.020 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:51.020 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.869773+0000", 2024-09-18T04:39:51.020 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:51.020 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.020 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.020 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:51.020 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.869773+0000", 2024-09-18T04:39:51.020 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:51.021 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.021 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.021 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:51.021 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.869771+0000", 2024-09-18T04:39:51.021 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:51.021 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.021 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.021 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:51.021 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870629+0000", 2024-09-18T04:39:51.021 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00085760999999999997 2024-09-18T04:39:51.021 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.021 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.021 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:51.021 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870647+0000", 2024-09-18T04:39:51.022 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8363999999999999e-05 2024-09-18T04:39:51.022 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.022 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.022 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:51.022 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870655+0000", 2024-09-18T04:39:51.022 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0099999999999995e-06 2024-09-18T04:39:51.022 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.022 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.022 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:51.022 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870803+0000", 2024-09-18T04:39:51.022 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014798 2024-09-18T04:39:51.022 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.022 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.022 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:51.023 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870951+0000", 2024-09-18T04:39:51.023 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014715999999999999 2024-09-18T04:39:51.023 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.023 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.023 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:51.023 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.871145+0000", 2024-09-18T04:39:51.023 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000194253 2024-09-18T04:39:51.023 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.023 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.023 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:51.023 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.877033+0000", 2024-09-18T04:39:51.023 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058883099999999999 2024-09-18T04:39:51.024 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.024 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.024 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:51.024 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.878078+0000", 2024-09-18T04:39:51.024 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001044747 2024-09-18T04:39:51.024 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.024 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.024 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:51.024 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.878160+0000", 2024-09-18T04:39:51.024 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1904999999999996e-05 2024-09-18T04:39:51.024 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.024 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.024 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:51.025 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.878216+0000", 2024-09-18T04:39:51.025 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6082000000000003e-05 2024-09-18T04:39:51.025 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:51.025 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:51.025 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:51.025 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.025 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.025 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:145 2.4 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:51.025 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:01.096935+0000", 2024-09-18T04:39:51.025 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 49.807409075999999, 2024-09-18T04:39:51.026 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0069937869999999996, 2024-09-18T04:39:51.026 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:51.026 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:51.026 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:51.026 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:39:51.026 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:51.026 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 145 2024-09-18T04:39:51.026 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.026 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:51.026 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.026 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:51.026 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.096935+0000", 2024-09-18T04:39:51.026 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:51.026 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.027 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.027 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:51.027 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.096935+0000", 2024-09-18T04:39:51.027 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:51.027 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.027 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.027 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:51.027 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.096932+0000", 2024-09-18T04:39:51.027 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:51.027 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.027 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.027 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:51.027 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097815+0000", 2024-09-18T04:39:51.027 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00088276000000000001 2024-09-18T04:39:51.027 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.028 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.028 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:51.028 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097842+0000", 2024-09-18T04:39:51.028 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6806999999999999e-05 2024-09-18T04:39:51.028 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.028 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.028 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:51.028 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097852+0000", 2024-09-18T04:39:51.028 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0548e-05 2024-09-18T04:39:51.028 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.028 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.028 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:51.028 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097931+0000", 2024-09-18T04:39:51.029 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8417000000000005e-05 2024-09-18T04:39:51.029 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.029 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.029 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:51.029 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.098047+0000", 2024-09-18T04:39:51.029 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011611 2024-09-18T04:39:51.029 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.029 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.029 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:51.029 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.098172+0000", 2024-09-18T04:39:51.029 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000125482 2024-09-18T04:39:51.029 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.029 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.029 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:51.030 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.100888+0000", 2024-09-18T04:39:51.030 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0027154430000000001 2024-09-18T04:39:51.030 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.030 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.030 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:51.030 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.103814+0000", 2024-09-18T04:39:51.030 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0029259619999999998 2024-09-18T04:39:51.030 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.030 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.030 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:51.030 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.103891+0000", 2024-09-18T04:39:51.030 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7619000000000003e-05 2024-09-18T04:39:51.030 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:51.030 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:51.030 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:51.031 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.103928+0000", 2024-09-18T04:39:51.031 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6900999999999998e-05 2024-09-18T04:39:51.031 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:51.031 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:51.031 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:51.031 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:51.031 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:51.031 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:51.032 INFO:teuthology.orchestra.run.smithi028.stdout:moved 0 / 34 (0%) 2024-09-18T04:39:51.032 INFO:teuthology.orchestra.run.smithi028.stdout:avg 6.8 2024-09-18T04:39:51.032 INFO:teuthology.orchestra.run.smithi028.stdout:stddev 1.46969 -> 1.46969 (expected baseline 2.33238) 2024-09-18T04:39:51.032 INFO:teuthology.orchestra.run.smithi028.stdout:min osd.3 with 5 -> 5 pgs (0.735294 -> 0.735294 * mean) 2024-09-18T04:39:51.033 INFO:teuthology.orchestra.run.smithi028.stdout:max osd.1 with 9 -> 9 pgs (1.32353 -> 1.32353 * mean) 2024-09-18T04:39:51.033 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:39:51.033 INFO:teuthology.orchestra.run.smithi028.stdout:oload 1000 2024-09-18T04:39:51.033 INFO:teuthology.orchestra.run.smithi028.stdout:max_change 1 2024-09-18T04:39:51.033 INFO:teuthology.orchestra.run.smithi028.stdout:max_change_osds 4 2024-09-18T04:39:51.033 INFO:teuthology.orchestra.run.smithi028.stdout:average_utilization 0.0009 2024-09-18T04:39:51.033 INFO:teuthology.orchestra.run.smithi028.stdout:overload_utilization 0.0092 2024-09-18T04:39:51.034 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:39:51.035 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:51.024+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:51.036 INFO:teuthology.orchestra.run.smithi028.stderr:no change 2024-09-18T04:39:51.037 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-09-18T04:39:51.037 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 1000 0.05 2024-09-18T04:39:51.134 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:51.128+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:51.145 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:51.146 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:51.146 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:39:51.146 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:51.156 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:39:51.235 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:51.232+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:51.257 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:51.257 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:51.257 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:39:51.257 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:39:51.257 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:51.267 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:39:51.336 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:51.332+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:39:51.340 INFO:teuthology.orchestra.run.smithi028.stdout:moved 0 / 34 (0%) 2024-09-18T04:39:51.340 INFO:teuthology.orchestra.run.smithi028.stdout:avg 6.8 2024-09-18T04:39:51.341 INFO:teuthology.orchestra.run.smithi028.stdout:stddev 1.46969 -> 1.46969 (expected baseline 2.33238) 2024-09-18T04:39:51.341 INFO:teuthology.orchestra.run.smithi028.stdout:min osd.3 with 5 -> 5 pgs (0.735294 -> 0.735294 * mean) 2024-09-18T04:39:51.341 INFO:teuthology.orchestra.run.smithi028.stdout:max osd.1 with 9 -> 9 pgs (1.32353 -> 1.32353 * mean) 2024-09-18T04:39:51.341 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:39:51.341 INFO:teuthology.orchestra.run.smithi028.stdout:oload 1000 2024-09-18T04:39:51.341 INFO:teuthology.orchestra.run.smithi028.stdout:max_change 0.05 2024-09-18T04:39:51.342 INFO:teuthology.orchestra.run.smithi028.stdout:max_change_osds 4 2024-09-18T04:39:51.342 INFO:teuthology.orchestra.run.smithi028.stdout:average_utilization 0.0009 2024-09-18T04:39:51.342 INFO:teuthology.orchestra.run.smithi028.stdout:overload_utilization 0.0092 2024-09-18T04:39:51.344 INFO:teuthology.orchestra.run.smithi028.stderr:no change 2024-09-18T04:39:51.355 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-09-18T04:39:51.355 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 1000 0.05 2024-09-18T04:39:51.380 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:51.380 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:39:51.380 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:39:51.380 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:39:51.381 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.381 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:7, version: 79'62, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:7@79'62, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 8=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-18T04:39:51.381 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.724969+0000", 2024-09-18T04:39:51.381 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.6499584150000004, 2024-09-18T04:39:51.381 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0035525930000000002, 2024-09-18T04:39:51.381 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:51.381 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:51.381 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:51.381 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.381 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:51.382 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.724969+0000", 2024-09-18T04:39:51.382 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.382 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.382 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:51.382 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.724969+0000", 2024-09-18T04:39:51.382 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.382 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.382 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:51.382 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.724968+0000", 2024-09-18T04:39:51.382 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:51.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.383 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.383 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:51.383 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.726245+0000", 2024-09-18T04:39:51.383 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001276395 2024-09-18T04:39:51.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.383 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.383 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:51.383 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.726262+0000", 2024-09-18T04:39:51.383 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6973000000000002e-05 2024-09-18T04:39:51.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.383 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.383 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:51.384 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.726293+0000", 2024-09-18T04:39:51.384 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0886999999999998e-05 2024-09-18T04:39:51.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.384 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.384 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:51.384 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.726403+0000", 2024-09-18T04:39:51.384 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000110288 2024-09-18T04:39:51.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.384 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.384 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:51.384 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.726418+0000", 2024-09-18T04:39:51.384 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5503999999999999e-05 2024-09-18T04:39:51.385 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.385 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.385 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:51.385 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.728522+0000", 2024-09-18T04:39:51.385 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021032630000000002 2024-09-18T04:39:51.385 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.385 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:51.385 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.385 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.385 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.385 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.1 87/86 [PushOp(2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head, version: 81'220, data_included: [4182016~12288], data_size: 12288, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head@81'220, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:4182016, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:39:51.385 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.730366+0000", 2024-09-18T04:39:51.385 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.6445620180000002, 2024-09-18T04:39:51.386 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0032522580000000001, 2024-09-18T04:39:51.386 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:51.386 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:51.386 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:51.386 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.386 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:51.386 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.730366+0000", 2024-09-18T04:39:51.386 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.386 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.386 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:51.386 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.730366+0000", 2024-09-18T04:39:51.386 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.386 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.387 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:51.387 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.730365+0000", 2024-09-18T04:39:51.387 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:51.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.387 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:51.387 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.730385+0000", 2024-09-18T04:39:51.387 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0766000000000001e-05 2024-09-18T04:39:51.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.387 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:51.387 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.730406+0000", 2024-09-18T04:39:51.387 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0706000000000001e-05 2024-09-18T04:39:51.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.388 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.388 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:51.388 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.730423+0000", 2024-09-18T04:39:51.388 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7411e-05 2024-09-18T04:39:51.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.388 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.388 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:51.388 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.730507+0000", 2024-09-18T04:39:51.388 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3388999999999997e-05 2024-09-18T04:39:51.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.388 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.389 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:51.389 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.730523+0000", 2024-09-18T04:39:51.389 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5843999999999999e-05 2024-09-18T04:39:51.389 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.389 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.390 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:51.390 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.733618+0000", 2024-09-18T04:39:51.390 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030951540000000001 2024-09-18T04:39:51.390 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.390 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:51.390 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.390 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.390 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:7, version: 79'62, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:7@79'62, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 8=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:39:51.390 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.732222+0000", 2024-09-18T04:39:51.391 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.6427060579999999, 2024-09-18T04:39:51.391 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030084410000000002, 2024-09-18T04:39:51.391 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:51.391 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:51.391 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:51.391 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.391 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:51.391 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.732222+0000", 2024-09-18T04:39:51.392 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.392 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.392 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:51.392 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.732222+0000", 2024-09-18T04:39:51.392 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.392 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.393 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:51.393 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.732221+0000", 2024-09-18T04:39:51.393 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:51.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.393 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.393 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:51.393 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.733174+0000", 2024-09-18T04:39:51.393 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00095350299999999997 2024-09-18T04:39:51.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.394 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.394 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:51.394 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.733197+0000", 2024-09-18T04:39:51.394 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2925000000000002e-05 2024-09-18T04:39:51.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.394 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.394 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:51.394 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.733218+0000", 2024-09-18T04:39:51.394 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0477999999999999e-05 2024-09-18T04:39:51.395 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.395 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.395 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:51.395 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.733323+0000", 2024-09-18T04:39:51.395 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000105717 2024-09-18T04:39:51.395 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.395 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.395 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:51.395 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.733358+0000", 2024-09-18T04:39:51.395 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4406000000000002e-05 2024-09-18T04:39:51.395 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.395 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.396 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:51.396 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.735230+0000", 2024-09-18T04:39:51.396 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0018720659999999999 2024-09-18T04:39:51.396 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.396 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:51.396 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.396 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.396 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:7, version: 79'62, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:7@79'62, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 8=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:39:51.396 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.743433+0000", 2024-09-18T04:39:51.396 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.6314945769999998, 2024-09-18T04:39:51.397 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003400237, 2024-09-18T04:39:51.397 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:51.397 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:51.397 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:51.397 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.397 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:51.397 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.743433+0000", 2024-09-18T04:39:51.397 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.397 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.397 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:51.397 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.743433+0000", 2024-09-18T04:39:51.398 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.398 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.398 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.398 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:51.398 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.743432+0000", 2024-09-18T04:39:51.398 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:51.398 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.398 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.398 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:51.398 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.744392+0000", 2024-09-18T04:39:51.399 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00095993499999999998 2024-09-18T04:39:51.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.399 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.399 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:51.399 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.744417+0000", 2024-09-18T04:39:51.399 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5249000000000002e-05 2024-09-18T04:39:51.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.399 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.399 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:51.399 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.744434+0000", 2024-09-18T04:39:51.399 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6597999999999999e-05 2024-09-18T04:39:51.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.399 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.399 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:51.400 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.744499+0000", 2024-09-18T04:39:51.400 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4987000000000006e-05 2024-09-18T04:39:51.400 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.400 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.400 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:51.400 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.744506+0000", 2024-09-18T04:39:51.400 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2679999999999997e-06 2024-09-18T04:39:51.400 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.400 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.400 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:51.400 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.746833+0000", 2024-09-18T04:39:51.400 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023270529999999999 2024-09-18T04:39:51.400 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.401 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:51.401 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.401 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.401 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.5 87/86 [PushOp(2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:head, version: 79'60, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:head@79'60, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-18T04:39:51.401 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.747807+0000", 2024-09-18T04:39:51.401 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.6271201030000002, 2024-09-18T04:39:51.401 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00244378, 2024-09-18T04:39:51.401 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:51.401 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:51.401 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:51.401 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.402 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:51.402 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.747807+0000", 2024-09-18T04:39:51.402 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.402 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.402 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.402 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:51.402 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.747807+0000", 2024-09-18T04:39:51.402 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.402 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.402 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.402 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:51.402 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.747806+0000", 2024-09-18T04:39:51.403 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:51.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.403 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.403 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:51.403 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.747825+0000", 2024-09-18T04:39:51.403 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.925e-05 2024-09-18T04:39:51.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.403 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.403 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:51.403 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.747853+0000", 2024-09-18T04:39:51.403 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7174000000000001e-05 2024-09-18T04:39:51.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.404 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.404 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:51.404 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.747887+0000", 2024-09-18T04:39:51.404 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4731999999999999e-05 2024-09-18T04:39:51.404 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.404 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.404 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:51.404 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.747983+0000", 2024-09-18T04:39:51.404 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5964999999999993e-05 2024-09-18T04:39:51.404 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.404 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.404 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:51.405 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.747994+0000", 2024-09-18T04:39:51.405 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0526999999999999e-05 2024-09-18T04:39:51.405 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.405 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.405 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:51.405 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.750251+0000", 2024-09-18T04:39:51.405 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0022573519999999998 2024-09-18T04:39:51.405 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.405 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:51.405 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.405 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.405 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.405 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:7, version: 79'64, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:7@79'64, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-18T04:39:51.406 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.750570+0000", 2024-09-18T04:39:51.406 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.6243577, 2024-09-18T04:39:51.406 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0029248949999999998, 2024-09-18T04:39:51.406 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:51.406 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:51.406 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:51.406 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.406 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:51.406 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.750570+0000", 2024-09-18T04:39:51.406 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.406 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.406 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.406 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:51.406 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.750570+0000", 2024-09-18T04:39:51.406 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.407 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.407 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.407 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:51.407 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.750569+0000", 2024-09-18T04:39:51.407 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:51.407 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.407 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.407 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:51.407 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.751724+0000", 2024-09-18T04:39:51.407 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001155037 2024-09-18T04:39:51.407 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.407 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.407 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:51.408 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.751747+0000", 2024-09-18T04:39:51.408 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2679000000000001e-05 2024-09-18T04:39:51.408 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.408 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.408 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:51.408 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.751761+0000", 2024-09-18T04:39:51.408 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.467e-05 2024-09-18T04:39:51.408 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.408 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.408 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:51.409 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.751839+0000", 2024-09-18T04:39:51.409 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7824000000000006e-05 2024-09-18T04:39:51.409 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.409 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.409 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:51.409 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.751849+0000", 2024-09-18T04:39:51.409 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4159999999999993e-06 2024-09-18T04:39:51.409 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.410 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.410 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:51.410 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.753495+0000", 2024-09-18T04:39:51.410 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0016461830000000001 2024-09-18T04:39:51.410 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.411 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:51.411 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.411 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.411 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.411 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.5 87/86 [PushOp(2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:7, version: 79'59, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:7@79'59, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-18T04:39:51.412 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.752479+0000", 2024-09-18T04:39:51.412 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.6224483330000004, 2024-09-18T04:39:51.412 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027271040000000002, 2024-09-18T04:39:51.412 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:51.412 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:51.412 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:51.413 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.413 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:51.413 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.752479+0000", 2024-09-18T04:39:51.413 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.413 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.414 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.414 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:51.414 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.752479+0000", 2024-09-18T04:39:51.414 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.414 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.415 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.415 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:51.415 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.752478+0000", 2024-09-18T04:39:51.415 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:51.415 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.416 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.416 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:51.416 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.753784+0000", 2024-09-18T04:39:51.416 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0013058869999999999 2024-09-18T04:39:51.416 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.416 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.416 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:51.416 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.753821+0000", 2024-09-18T04:39:51.416 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7268999999999998e-05 2024-09-18T04:39:51.416 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.417 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.417 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:51.417 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.753873+0000", 2024-09-18T04:39:51.417 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1564000000000003e-05 2024-09-18T04:39:51.417 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.417 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.417 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:51.417 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.753963+0000", 2024-09-18T04:39:51.417 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0636000000000004e-05 2024-09-18T04:39:51.417 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.417 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.417 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:51.417 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.753972+0000", 2024-09-18T04:39:51.418 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0529999999999996e-06 2024-09-18T04:39:51.418 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.418 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.418 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:51.418 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.755206+0000", 2024-09-18T04:39:51.418 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001234113 2024-09-18T04:39:51.418 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.418 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:51.418 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.418 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.418 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.419 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:7, version: 79'64, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:7@79'64, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:39:51.419 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.756672+0000", 2024-09-18T04:39:51.419 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.6182552560000003, 2024-09-18T04:39:51.419 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028688770000000001, 2024-09-18T04:39:51.419 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:51.419 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:51.419 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:51.419 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.419 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:51.419 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.756672+0000", 2024-09-18T04:39:51.419 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.419 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.419 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.420 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:51.420 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.756672+0000", 2024-09-18T04:39:51.420 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.420 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.420 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.420 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:51.420 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.756671+0000", 2024-09-18T04:39:51.420 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:51.420 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.420 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.420 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:51.420 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.757623+0000", 2024-09-18T04:39:51.420 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00095182199999999995 2024-09-18T04:39:51.420 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.421 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.421 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:51.421 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.757644+0000", 2024-09-18T04:39:51.421 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0996999999999999e-05 2024-09-18T04:39:51.421 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.421 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.421 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:51.421 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.757657+0000", 2024-09-18T04:39:51.421 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3264999999999999e-05 2024-09-18T04:39:51.421 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.421 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.421 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:51.421 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.757779+0000", 2024-09-18T04:39:51.422 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000121655 2024-09-18T04:39:51.422 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.422 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.422 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:51.422 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.757801+0000", 2024-09-18T04:39:51.422 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2340999999999998e-05 2024-09-18T04:39:51.422 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.422 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.422 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:51.422 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.759541+0000", 2024-09-18T04:39:51.422 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017398120000000001 2024-09-18T04:39:51.422 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.422 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:51.423 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.423 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.423 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.5 87/86 [PushOp(2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:7, version: 79'59, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:7@79'59, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:39:51.423 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.758140+0000", 2024-09-18T04:39:51.423 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.6167873510000002, 2024-09-18T04:39:51.423 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023362539999999998, 2024-09-18T04:39:51.423 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:51.423 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:51.423 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:51.424 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.424 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:51.424 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.758140+0000", 2024-09-18T04:39:51.424 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.424 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.424 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.424 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:51.424 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.758140+0000", 2024-09-18T04:39:51.424 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.424 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.424 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.425 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:51.425 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.758139+0000", 2024-09-18T04:39:51.425 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:51.425 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.425 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.425 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:51.425 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.758850+0000", 2024-09-18T04:39:51.425 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00071123099999999999 2024-09-18T04:39:51.425 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.426 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.426 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:51.426 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.758872+0000", 2024-09-18T04:39:51.426 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2195999999999999e-05 2024-09-18T04:39:51.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.426 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.426 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:51.426 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.758885+0000", 2024-09-18T04:39:51.426 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.327e-05 2024-09-18T04:39:51.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.426 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.426 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:51.427 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.758969+0000", 2024-09-18T04:39:51.427 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3497000000000004e-05 2024-09-18T04:39:51.427 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.427 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.427 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:51.427 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.758988+0000", 2024-09-18T04:39:51.427 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8765000000000001e-05 2024-09-18T04:39:51.427 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.427 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.427 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:51.427 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.760476+0000", 2024-09-18T04:39:51.427 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001488828 2024-09-18T04:39:51.427 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.428 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:51.428 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.428 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.428 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.428 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.5 87/86 [PushOp(2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:7, version: 79'59, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:7@79'59, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:39:51.428 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.762725+0000", 2024-09-18T04:39:51.428 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.6122022500000002, 2024-09-18T04:39:51.428 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028321890000000001, 2024-09-18T04:39:51.428 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:51.428 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:51.428 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:51.428 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.429 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:51.429 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.762725+0000", 2024-09-18T04:39:51.429 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.429 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.429 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.429 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:51.429 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.762725+0000", 2024-09-18T04:39:51.429 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.429 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.429 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.429 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:51.429 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.762724+0000", 2024-09-18T04:39:51.429 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:51.430 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.430 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.430 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:51.430 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.763395+0000", 2024-09-18T04:39:51.430 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00067064800000000003 2024-09-18T04:39:51.430 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.430 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.430 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:51.430 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.763416+0000", 2024-09-18T04:39:51.430 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0984000000000001e-05 2024-09-18T04:39:51.430 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.430 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.430 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:51.431 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.763442+0000", 2024-09-18T04:39:51.431 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6635000000000002e-05 2024-09-18T04:39:51.431 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.431 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.431 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:51.431 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.763584+0000", 2024-09-18T04:39:51.431 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000142037 2024-09-18T04:39:51.431 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.431 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.431 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:51.431 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.763610+0000", 2024-09-18T04:39:51.431 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6027999999999999e-05 2024-09-18T04:39:51.431 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.431 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.432 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:51.432 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.765557+0000", 2024-09-18T04:39:51.432 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0019470939999999999 2024-09-18T04:39:51.432 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.432 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:51.432 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.432 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.432 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.432 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:7, version: 79'64, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:7@79'64, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:39:51.432 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.763323+0000", 2024-09-18T04:39:51.432 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.6116040649999999, 2024-09-18T04:39:51.432 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003042215, 2024-09-18T04:39:51.432 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:51.433 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:51.433 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:51.433 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.433 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:51.433 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.763323+0000", 2024-09-18T04:39:51.433 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.433 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.433 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.433 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:51.433 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.763323+0000", 2024-09-18T04:39:51.433 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.433 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.433 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.434 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:51.434 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.763322+0000", 2024-09-18T04:39:51.434 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:51.434 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.434 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.434 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:51.434 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.764567+0000", 2024-09-18T04:39:51.434 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001244312 2024-09-18T04:39:51.434 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.434 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.434 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:51.434 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.764600+0000", 2024-09-18T04:39:51.434 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2787000000000003e-05 2024-09-18T04:39:51.434 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.435 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.435 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:51.435 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.764640+0000", 2024-09-18T04:39:51.435 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0825000000000003e-05 2024-09-18T04:39:51.435 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.435 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.435 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:51.435 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.764789+0000", 2024-09-18T04:39:51.436 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000148598 2024-09-18T04:39:51.436 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.436 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.436 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:51.436 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.764812+0000", 2024-09-18T04:39:51.436 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2702e-05 2024-09-18T04:39:51.437 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.437 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.437 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:51.437 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.766366+0000", 2024-09-18T04:39:51.437 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0015540110000000001 2024-09-18T04:39:51.437 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.437 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:51.437 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.437 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.437 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.437 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.5 87/86 [PushOp(2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:7, version: 79'59, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:7@79'59, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:39:51.438 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.768078+0000", 2024-09-18T04:39:51.438 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.6068492870000002, 2024-09-18T04:39:51.438 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028938509999999998, 2024-09-18T04:39:51.438 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:51.438 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:51.438 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:51.438 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.439 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:51.439 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.768078+0000", 2024-09-18T04:39:51.439 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.439 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.439 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.439 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:51.439 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.768078+0000", 2024-09-18T04:39:51.439 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.439 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.440 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.440 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:51.440 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.768076+0000", 2024-09-18T04:39:51.440 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:51.440 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.440 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.440 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:51.440 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.768951+0000", 2024-09-18T04:39:51.440 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000874455 2024-09-18T04:39:51.440 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.440 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.440 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:51.441 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.768972+0000", 2024-09-18T04:39:51.441 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.154e-05 2024-09-18T04:39:51.441 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.441 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.441 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:51.441 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.768992+0000", 2024-09-18T04:39:51.441 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9822000000000001e-05 2024-09-18T04:39:51.441 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.441 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.441 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:51.441 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.769110+0000", 2024-09-18T04:39:51.441 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000117762 2024-09-18T04:39:51.441 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.442 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.442 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:51.442 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.769123+0000", 2024-09-18T04:39:51.442 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2473999999999999e-05 2024-09-18T04:39:51.442 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.442 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.442 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:51.442 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.770972+0000", 2024-09-18T04:39:51.442 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001849613 2024-09-18T04:39:51.442 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.442 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:51.442 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.442 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.442 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.443 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:7, version: 79'64, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:7@79'64, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:39:51.443 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.769830+0000", 2024-09-18T04:39:51.443 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.605098023, 2024-09-18T04:39:51.443 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0040223419999999999, 2024-09-18T04:39:51.443 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:51.443 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:51.443 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:51.443 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.443 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:51.443 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.769830+0000", 2024-09-18T04:39:51.443 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.443 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.443 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.444 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:51.444 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.769830+0000", 2024-09-18T04:39:51.444 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.444 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.444 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.444 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:51.444 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.769829+0000", 2024-09-18T04:39:51.444 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:51.445 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.445 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.445 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:51.445 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.770812+0000", 2024-09-18T04:39:51.445 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00098347800000000009 2024-09-18T04:39:51.446 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.446 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.446 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:51.446 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.770834+0000", 2024-09-18T04:39:51.446 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1648999999999999e-05 2024-09-18T04:39:51.446 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.446 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.446 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:51.446 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.770847+0000", 2024-09-18T04:39:51.447 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3169999999999999e-05 2024-09-18T04:39:51.447 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.447 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.447 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:51.447 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.770969+0000", 2024-09-18T04:39:51.447 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000121721 2024-09-18T04:39:51.447 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.447 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.447 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:51.447 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.770989+0000", 2024-09-18T04:39:51.447 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9944000000000001e-05 2024-09-18T04:39:51.447 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.447 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.447 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:51.448 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.773852+0000", 2024-09-18T04:39:51.448 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028632319999999998 2024-09-18T04:39:51.448 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.448 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:51.448 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.448 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.448 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.448 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.5 87/86 [PushOp(2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head, version: 79'62, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head@79'62, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-18T04:39:51.448 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.772378+0000", 2024-09-18T04:39:51.448 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.6025498999999996, 2024-09-18T04:39:51.448 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002661535, 2024-09-18T04:39:51.448 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:51.448 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:51.449 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:51.449 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.449 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:51.449 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.772378+0000", 2024-09-18T04:39:51.449 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.449 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.450 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.450 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:51.450 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.772378+0000", 2024-09-18T04:39:51.450 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.450 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.450 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.450 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:51.450 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.772377+0000", 2024-09-18T04:39:51.450 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:51.450 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.451 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.451 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:51.451 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.772393+0000", 2024-09-18T04:39:51.451 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6495e-05 2024-09-18T04:39:51.451 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.451 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.451 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:51.451 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.772417+0000", 2024-09-18T04:39:51.451 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4304e-05 2024-09-18T04:39:51.451 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.451 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.452 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:51.452 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.772431+0000", 2024-09-18T04:39:51.452 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3165e-05 2024-09-18T04:39:51.452 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.452 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.452 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:51.452 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.772499+0000", 2024-09-18T04:39:51.452 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8627000000000005e-05 2024-09-18T04:39:51.452 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.453 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.453 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:51.453 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.772508+0000", 2024-09-18T04:39:51.453 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5609999999999999e-06 2024-09-18T04:39:51.453 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.453 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.453 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:51.453 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.775039+0000", 2024-09-18T04:39:51.453 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002531387 2024-09-18T04:39:51.453 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.453 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:51.453 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.453 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.454 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.454 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head, version: 79'65, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head@79'65, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-18T04:39:51.454 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.775934+0000", 2024-09-18T04:39:51.454 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.5989931469999998, 2024-09-18T04:39:51.454 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023666939999999999, 2024-09-18T04:39:51.454 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:51.454 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:51.454 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:51.454 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.454 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:51.454 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.775934+0000", 2024-09-18T04:39:51.454 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.454 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.455 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.455 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:51.455 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.775934+0000", 2024-09-18T04:39:51.455 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.455 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.455 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.455 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:51.455 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.775933+0000", 2024-09-18T04:39:51.455 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:51.455 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.455 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.455 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:51.455 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.775955+0000", 2024-09-18T04:39:51.456 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1512e-05 2024-09-18T04:39:51.456 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.456 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.456 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:51.456 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.775975+0000", 2024-09-18T04:39:51.456 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9653999999999999e-05 2024-09-18T04:39:51.456 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.456 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.456 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:51.456 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.775992+0000", 2024-09-18T04:39:51.457 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7326e-05 2024-09-18T04:39:51.457 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.457 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.457 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:51.457 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.776078+0000", 2024-09-18T04:39:51.457 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.6599000000000002e-05 2024-09-18T04:39:51.457 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.457 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.457 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:51.457 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.776088+0000", 2024-09-18T04:39:51.457 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.8120000000000002e-06 2024-09-18T04:39:51.457 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.457 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.458 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:51.458 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.778301+0000", 2024-09-18T04:39:51.458 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002212805 2024-09-18T04:39:51.458 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.458 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:51.458 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.458 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.458 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.458 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.5 87/86 [PushOp(2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:7, version: 79'61, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:7@79'61, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:39:51.458 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.781808+0000", 2024-09-18T04:39:51.458 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.5931199019999998, 2024-09-18T04:39:51.458 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027222169999999999, 2024-09-18T04:39:51.458 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:51.459 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:51.459 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:51.459 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.459 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:51.459 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.781808+0000", 2024-09-18T04:39:51.459 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.459 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.459 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.460 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:51.460 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.781808+0000", 2024-09-18T04:39:51.460 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.460 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.460 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.460 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:51.460 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.781807+0000", 2024-09-18T04:39:51.460 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:51.460 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.461 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.461 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:51.461 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.782242+0000", 2024-09-18T04:39:51.461 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00043528300000000001 2024-09-18T04:39:51.461 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.461 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.462 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:51.462 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.782265+0000", 2024-09-18T04:39:51.462 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3227e-05 2024-09-18T04:39:51.462 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.462 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.462 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:51.462 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.782283+0000", 2024-09-18T04:39:51.462 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7535999999999999e-05 2024-09-18T04:39:51.462 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.462 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.462 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:51.462 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.782418+0000", 2024-09-18T04:39:51.462 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000135298 2024-09-18T04:39:51.463 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.463 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.463 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:51.463 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.782444+0000", 2024-09-18T04:39:51.463 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6020000000000002e-05 2024-09-18T04:39:51.463 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.463 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.463 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:51.463 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.784530+0000", 2024-09-18T04:39:51.463 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020858890000000001 2024-09-18T04:39:51.463 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.463 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:51.463 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.464 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.464 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.464 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head, version: 81'75, data_included: [0~49152,1097728~999424], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head@81'75, size: 2146304, copy_subset: [0~2146304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-18T04:39:51.464 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.782052+0000", 2024-09-18T04:39:51.464 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.5928755380000004, 2024-09-18T04:39:51.464 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0037389630000000001, 2024-09-18T04:39:51.464 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:51.464 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:51.465 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:51.465 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.465 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:51.465 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.782052+0000", 2024-09-18T04:39:51.465 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.466 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.466 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.466 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:51.466 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.782052+0000", 2024-09-18T04:39:51.466 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.466 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.467 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.467 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:51.467 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.782051+0000", 2024-09-18T04:39:51.467 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:51.467 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.468 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.468 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:51.468 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.783658+0000", 2024-09-18T04:39:51.468 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0016072149999999999 2024-09-18T04:39:51.468 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.468 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.469 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:51.469 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.783698+0000", 2024-09-18T04:39:51.469 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9394000000000003e-05 2024-09-18T04:39:51.469 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.469 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.470 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:51.470 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.783733+0000", 2024-09-18T04:39:51.470 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5428e-05 2024-09-18T04:39:51.470 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.470 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.470 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:51.471 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.783854+0000", 2024-09-18T04:39:51.471 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012067599999999999 2024-09-18T04:39:51.471 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.471 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.471 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:51.472 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.783869+0000", 2024-09-18T04:39:51.472 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5554999999999999e-05 2024-09-18T04:39:51.472 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.472 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.472 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:51.472 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.785791+0000", 2024-09-18T04:39:51.472 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001921629 2024-09-18T04:39:51.473 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.473 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:51.473 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.473 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.473 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.474 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head, version: 81'75, data_included: [2097152~49152], data_size: 49152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head@81'75, size: 2146304, copy_subset: [0~2146304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2146304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:39:51.474 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.787740+0000", 2024-09-18T04:39:51.474 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.5871880239999996, 2024-09-18T04:39:51.474 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030127370000000001, 2024-09-18T04:39:51.474 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:51.474 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:51.474 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:51.474 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.474 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:51.474 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.787740+0000", 2024-09-18T04:39:51.474 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.474 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.474 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.475 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:51.475 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.787740+0000", 2024-09-18T04:39:51.475 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.475 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.475 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.475 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:51.475 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.787738+0000", 2024-09-18T04:39:51.475 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:51.475 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.475 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.475 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:51.475 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.787795+0000", 2024-09-18T04:39:51.475 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6753999999999999e-05 2024-09-18T04:39:51.476 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.476 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.476 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:51.476 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.787826+0000", 2024-09-18T04:39:51.476 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0617000000000002e-05 2024-09-18T04:39:51.476 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.476 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.476 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:51.476 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.787853+0000", 2024-09-18T04:39:51.476 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7538999999999998e-05 2024-09-18T04:39:51.476 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.476 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.476 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:51.476 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.787963+0000", 2024-09-18T04:39:51.477 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000109387 2024-09-18T04:39:51.477 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.477 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.477 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:51.477 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.787973+0000", 2024-09-18T04:39:51.477 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0781999999999999e-05 2024-09-18T04:39:51.477 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.477 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.477 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:51.477 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.790752+0000", 2024-09-18T04:39:51.478 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027788520000000001 2024-09-18T04:39:51.478 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.478 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:51.478 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.478 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.478 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.478 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.5 87/86 [PushOp(2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:7, version: 79'61, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:7@79'61, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:39:51.478 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.791668+0000", 2024-09-18T04:39:51.478 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.5832593309999998, 2024-09-18T04:39:51.478 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025634299999999998, 2024-09-18T04:39:51.478 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:51.479 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:51.479 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:51.479 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.479 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:51.479 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.791668+0000", 2024-09-18T04:39:51.479 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.479 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.479 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.479 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:51.479 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.791668+0000", 2024-09-18T04:39:51.479 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.480 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.480 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.480 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:51.480 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.791667+0000", 2024-09-18T04:39:51.480 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:51.480 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.480 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.480 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:51.480 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.792071+0000", 2024-09-18T04:39:51.480 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00040348599999999999 2024-09-18T04:39:51.480 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.480 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.481 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:51.481 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.792094+0000", 2024-09-18T04:39:51.481 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3105e-05 2024-09-18T04:39:51.481 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.481 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.481 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:51.481 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.792112+0000", 2024-09-18T04:39:51.481 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8502999999999999e-05 2024-09-18T04:39:51.481 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.481 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.481 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:51.481 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.792187+0000", 2024-09-18T04:39:51.482 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5074000000000007e-05 2024-09-18T04:39:51.482 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.482 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.482 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:51.482 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.792197+0000", 2024-09-18T04:39:51.482 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.2979999999999997e-06 2024-09-18T04:39:51.482 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.483 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.483 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:51.483 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.794232+0000", 2024-09-18T04:39:51.483 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002035149 2024-09-18T04:39:51.483 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.483 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:51.483 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.484 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.484 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.484 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.5 87/86 [PushOp(2:b227e6b1:::rbd_data.125fc09df6d1.000000000000000e:7, version: 79'63, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:b227e6b1:::rbd_data.125fc09df6d1.000000000000000e:7@79'63, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:39:51.484 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.813128+0000", 2024-09-18T04:39:51.484 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.5617994700000004, 2024-09-18T04:39:51.484 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025359520000000002, 2024-09-18T04:39:51.484 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:51.484 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:51.484 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:51.484 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.484 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:51.484 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.813128+0000", 2024-09-18T04:39:51.484 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.485 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.485 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.485 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:51.485 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.813128+0000", 2024-09-18T04:39:51.485 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.486 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.486 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.486 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:51.486 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.813127+0000", 2024-09-18T04:39:51.486 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:51.486 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.486 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.486 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:51.486 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.813544+0000", 2024-09-18T04:39:51.486 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000417248 2024-09-18T04:39:51.486 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.486 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.487 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:51.487 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.813571+0000", 2024-09-18T04:39:51.487 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6789000000000001e-05 2024-09-18T04:39:51.487 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.487 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.487 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:51.487 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.813589+0000", 2024-09-18T04:39:51.487 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8247e-05 2024-09-18T04:39:51.487 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.487 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.487 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:51.487 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.813698+0000", 2024-09-18T04:39:51.487 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000108937 2024-09-18T04:39:51.488 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.488 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.488 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:51.488 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.813715+0000", 2024-09-18T04:39:51.488 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7110000000000001e-05 2024-09-18T04:39:51.488 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.488 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.488 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:51.488 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.815664+0000", 2024-09-18T04:39:51.488 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001948946 2024-09-18T04:39:51.488 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.488 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:51.488 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.488 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.489 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:51.489 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:51.490 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:51.484+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:39:51.491 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:39:51.591 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:51.588+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:51.644 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:51.645 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:51.645 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:39:51.645 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:51.654 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:39:51.700 INFO:teuthology.orchestra.run.smithi028.stdout:moved 0 / 34 (0%) 2024-09-18T04:39:51.701 INFO:teuthology.orchestra.run.smithi028.stdout:avg 6.8 2024-09-18T04:39:51.701 INFO:teuthology.orchestra.run.smithi028.stdout:stddev 1.46969 -> 1.46969 (expected baseline 2.33238) 2024-09-18T04:39:51.701 INFO:teuthology.orchestra.run.smithi028.stdout:min osd.3 with 5 -> 5 pgs (0.735294 -> 0.735294 * mean) 2024-09-18T04:39:51.701 INFO:teuthology.orchestra.run.smithi028.stdout:max osd.1 with 9 -> 9 pgs (1.32353 -> 1.32353 * mean) 2024-09-18T04:39:51.701 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:39:51.701 INFO:teuthology.orchestra.run.smithi028.stdout:oload 1000 2024-09-18T04:39:51.701 INFO:teuthology.orchestra.run.smithi028.stdout:max_change 0.05 2024-09-18T04:39:51.701 INFO:teuthology.orchestra.run.smithi028.stdout:max_change_osds 4 2024-09-18T04:39:51.701 INFO:teuthology.orchestra.run.smithi028.stdout:average_utilization 0.0009 2024-09-18T04:39:51.702 INFO:teuthology.orchestra.run.smithi028.stdout:overload_utilization 0.0093 2024-09-18T04:39:51.702 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:51.696+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:39:51.704 INFO:teuthology.orchestra.run.smithi028.stderr:no change 2024-09-18T04:39:51.720 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-09-18T04:39:51.720 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 1000 0.05 2024-09-18T04:39:51.757 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-09-18T04:39:51.762 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:51.763 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:51.763 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:39:51.763 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:39:51.763 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:51.772 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:39:51.792 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:51.784+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:51.893 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:51.884+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:51.929 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:51.929 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:39:51.929 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:39:51.929 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:39:51.929 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.930 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:37 2.5 e51/28 2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:head v 51'1, mlcod=0'0)", 2024-09-18T04:39:51.930 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:39:51.930 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 69.632239190000007, 2024-09-18T04:39:51.930 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012279973, 2024-09-18T04:39:51.930 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:51.930 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:51.930 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:51.930 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.930 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:51.930 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:39:51.930 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.930 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.930 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.931 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:51.931 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:39:51.931 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.931 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.931 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.931 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:51.931 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292468+0000", 2024-09-18T04:39:51.931 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:51.931 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.931 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.931 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:51.931 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295837+0000", 2024-09-18T04:39:51.931 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0033691699999999999 2024-09-18T04:39:51.932 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.932 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.932 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:51.932 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295860+0000", 2024-09-18T04:39:51.932 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2756999999999999e-05 2024-09-18T04:39:51.932 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.932 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.932 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:51.932 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295872+0000", 2024-09-18T04:39:51.932 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2442e-05 2024-09-18T04:39:51.932 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.932 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.932 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:51.933 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295934+0000", 2024-09-18T04:39:51.933 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1187999999999998e-05 2024-09-18T04:39:51.933 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.933 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.933 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:51.933 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295967+0000", 2024-09-18T04:39:51.933 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3185999999999997e-05 2024-09-18T04:39:51.933 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.933 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.933 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:51.933 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304697+0000", 2024-09-18T04:39:51.933 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0087298059999999997 2024-09-18T04:39:51.934 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.934 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.934 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:51.934 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304749+0000", 2024-09-18T04:39:51.934 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1922000000000002e-05 2024-09-18T04:39:51.934 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.934 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:51.934 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.934 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.934 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.934 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:49 2.5 e51/28 2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head v 51'2, mlcod=0'0)", 2024-09-18T04:39:51.934 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:39:51.934 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 69.598912351999999, 2024-09-18T04:39:51.935 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018033041999999999, 2024-09-18T04:39:51.935 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:51.935 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:51.935 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:51.935 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.935 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:51.935 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:39:51.935 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.935 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.935 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.935 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:51.935 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:39:51.935 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.936 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.936 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.936 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:51.936 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325793+0000", 2024-09-18T04:39:51.936 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:51.936 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.936 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.936 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:51.936 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329038+0000", 2024-09-18T04:39:51.936 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0032444010000000001 2024-09-18T04:39:51.936 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.936 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.936 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:51.937 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329065+0000", 2024-09-18T04:39:51.937 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7500000000000001e-05 2024-09-18T04:39:51.937 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.937 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.937 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:51.937 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329077+0000", 2024-09-18T04:39:51.937 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1919999999999999e-05 2024-09-18T04:39:51.937 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.937 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.937 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:51.937 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329149+0000", 2024-09-18T04:39:51.937 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2349999999999997e-05 2024-09-18T04:39:51.937 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.937 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.938 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:51.938 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329192+0000", 2024-09-18T04:39:51.938 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2744000000000003e-05 2024-09-18T04:39:51.938 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.938 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.938 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:51.938 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343773+0000", 2024-09-18T04:39:51.938 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014581246000000001 2024-09-18T04:39:51.938 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.938 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.938 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:51.938 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343829+0000", 2024-09-18T04:39:51.938 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5143999999999999e-05 2024-09-18T04:39:51.938 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.939 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:51.939 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.939 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.939 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.939 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:47 2.4 e51/17 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 51'3, mlcod=51'1)", 2024-09-18T04:39:51.939 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:39:51.939 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 69.598332087000003, 2024-09-18T04:39:51.939 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018960186, 2024-09-18T04:39:51.939 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:51.939 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:51.939 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:51.939 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.939 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:51.939 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:39:51.940 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.940 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.940 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.940 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:51.940 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:39:51.940 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.940 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.940 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.940 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:51.940 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326374+0000", 2024-09-18T04:39:51.940 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:51.941 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.941 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.941 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:51.941 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330883+0000", 2024-09-18T04:39:51.941 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0045084740000000002 2024-09-18T04:39:51.941 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.941 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.941 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:51.941 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330901+0000", 2024-09-18T04:39:51.941 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7890999999999999e-05 2024-09-18T04:39:51.941 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.941 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.941 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:51.941 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330912+0000", 2024-09-18T04:39:51.942 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1041e-05 2024-09-18T04:39:51.942 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.942 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.942 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:51.942 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.331003+0000", 2024-09-18T04:39:51.942 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1323999999999995e-05 2024-09-18T04:39:51.942 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.942 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.942 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:51.942 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.331030+0000", 2024-09-18T04:39:51.943 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7351999999999998e-05 2024-09-18T04:39:51.943 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.943 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.943 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:51.943 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345287+0000", 2024-09-18T04:39:51.943 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014256332 2024-09-18T04:39:51.943 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.943 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.943 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:51.943 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345336+0000", 2024-09-18T04:39:51.943 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9356e-05 2024-09-18T04:39:51.943 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.943 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:51.943 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.944 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.944 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.944 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:58 2.5 e51/28 2:b227e6b1:::rbd_data.125fc09df6d1.000000000000000e:head v 51'3, mlcod=51'1)", 2024-09-18T04:39:51.944 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:39:51.944 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 69.557905460000001, 2024-09-18T04:39:51.944 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016513678, 2024-09-18T04:39:51.944 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:51.944 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:51.944 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:51.944 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.945 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:51.945 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:39:51.945 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.945 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.945 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.945 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:51.945 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:39:51.945 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.945 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.945 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.945 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:51.945 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366800+0000", 2024-09-18T04:39:51.945 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:51.946 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.946 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.946 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:51.946 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368644+0000", 2024-09-18T04:39:51.946 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001844399 2024-09-18T04:39:51.946 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.946 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.946 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:51.946 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368681+0000", 2024-09-18T04:39:51.947 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6526999999999998e-05 2024-09-18T04:39:51.947 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.947 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.947 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:51.947 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368698+0000", 2024-09-18T04:39:51.947 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7272999999999999e-05 2024-09-18T04:39:51.947 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.947 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.947 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:51.947 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368773+0000", 2024-09-18T04:39:51.948 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4862000000000003e-05 2024-09-18T04:39:51.948 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.948 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.948 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:51.948 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368801+0000", 2024-09-18T04:39:51.948 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7925000000000001e-05 2024-09-18T04:39:51.948 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.948 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.949 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:51.949 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383261+0000", 2024-09-18T04:39:51.949 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014460377999999999 2024-09-18T04:39:51.949 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.949 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.949 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:51.949 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383316+0000", 2024-09-18T04:39:51.949 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4623999999999997e-05 2024-09-18T04:39:51.949 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.950 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:51.950 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.950 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.950 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.950 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:56 2.4 e51/17 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 51'4, mlcod=51'2)", 2024-09-18T04:39:51.950 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:39:51.950 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 69.555901434000006, 2024-09-18T04:39:51.950 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015105221, 2024-09-18T04:39:51.950 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:51.950 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:51.950 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:51.951 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.951 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:51.951 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:39:51.951 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.951 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.951 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.951 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:51.951 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:39:51.951 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.951 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.951 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.951 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:51.951 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368805+0000", 2024-09-18T04:39:51.952 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:51.952 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.952 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.952 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:51.952 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371603+0000", 2024-09-18T04:39:51.952 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027982660000000002 2024-09-18T04:39:51.952 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.952 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.952 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:51.952 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371633+0000", 2024-09-18T04:39:51.952 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0073000000000002e-05 2024-09-18T04:39:51.952 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.952 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.953 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:51.953 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371652+0000", 2024-09-18T04:39:51.953 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8597000000000002e-05 2024-09-18T04:39:51.953 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.953 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.953 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:51.953 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371752+0000", 2024-09-18T04:39:51.953 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9772999999999995e-05 2024-09-18T04:39:51.953 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.953 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.953 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:51.953 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371793+0000", 2024-09-18T04:39:51.953 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1173e-05 2024-09-18T04:39:51.953 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.954 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.954 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:51.954 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383875+0000", 2024-09-18T04:39:51.954 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012081852000000001 2024-09-18T04:39:51.954 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.954 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.954 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:51.954 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383912+0000", 2024-09-18T04:39:51.954 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7045000000000002e-05 2024-09-18T04:39:51.954 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.954 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:51.954 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.954 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.955 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.955 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:36 2.5 e51/28 2:bd5b8dd8:ns::rbd_data.133e813ab5bc.0000000000000002:head v 51'4, mlcod=51'2)", 2024-09-18T04:39:51.955 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:39:51.955 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 69.387182443, 2024-09-18T04:39:51.955 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020758161000000001, 2024-09-18T04:39:51.955 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:51.955 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:51.955 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:51.955 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.955 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:51.955 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:39:51.955 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.956 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.956 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.956 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:51.956 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:39:51.956 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.956 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.956 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.956 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:51.956 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537523+0000", 2024-09-18T04:39:51.956 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:51.956 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.956 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.956 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:51.956 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541578+0000", 2024-09-18T04:39:51.957 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0040549710000000001 2024-09-18T04:39:51.957 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.957 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.957 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:51.957 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541608+0000", 2024-09-18T04:39:51.957 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9349e-05 2024-09-18T04:39:51.957 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.957 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.957 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:51.957 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541621+0000", 2024-09-18T04:39:51.957 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2736e-05 2024-09-18T04:39:51.957 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.957 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.958 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:51.958 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541715+0000", 2024-09-18T04:39:51.958 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4197999999999995e-05 2024-09-18T04:39:51.958 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.958 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.958 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:51.958 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541755+0000", 2024-09-18T04:39:51.958 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9818999999999999e-05 2024-09-18T04:39:51.958 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.958 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.958 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:51.958 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558217+0000", 2024-09-18T04:39:51.958 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016462234999999999 2024-09-18T04:39:51.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.959 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.959 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:51.959 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558284+0000", 2024-09-18T04:39:51.959 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6774999999999997e-05 2024-09-18T04:39:51.959 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.959 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:51.959 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.959 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.959 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:39 2.4 e51/17 2:304364d1:ns::rbd_data.133e813ab5bc.0000000000000004:head v 51'5, mlcod=51'3)", 2024-09-18T04:39:51.959 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:39:51.959 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 69.379043315999994, 2024-09-18T04:39:51.960 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016777558000000001, 2024-09-18T04:39:51.960 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:51.960 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:51.960 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:51.960 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.960 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:51.960 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:39:51.960 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.960 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.960 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.960 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:51.960 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:39:51.960 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.960 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.961 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.961 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:51.961 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545663+0000", 2024-09-18T04:39:51.961 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:51.961 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.961 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.961 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:51.961 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548286+0000", 2024-09-18T04:39:51.961 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026233390000000001 2024-09-18T04:39:51.961 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.961 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.961 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:51.961 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548305+0000", 2024-09-18T04:39:51.961 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8425000000000001e-05 2024-09-18T04:39:51.962 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.962 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.962 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:51.962 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548316+0000", 2024-09-18T04:39:51.962 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1163e-05 2024-09-18T04:39:51.962 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.962 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.962 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:51.962 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548379+0000", 2024-09-18T04:39:51.962 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2780000000000005e-05 2024-09-18T04:39:51.962 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.962 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.962 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:51.962 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548399+0000", 2024-09-18T04:39:51.962 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0080000000000001e-05 2024-09-18T04:39:51.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.963 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.963 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:51.963 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562383+0000", 2024-09-18T04:39:51.963 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013984671000000001 2024-09-18T04:39:51.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.963 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.963 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:51.963 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562442+0000", 2024-09-18T04:39:51.963 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8700999999999999e-05 2024-09-18T04:39:51.963 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.963 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:51.963 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.964 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.964 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.964 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:37 2.5 e51/28 2:bf39b8e4:ns::rbd_data.133e813ab5bc.0000000000000003:head v 51'5, mlcod=51'2)", 2024-09-18T04:39:51.964 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:39:51.964 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 69.376175047000004, 2024-09-18T04:39:51.964 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014264492, 2024-09-18T04:39:51.964 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:51.964 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:51.964 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:51.964 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.964 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:51.964 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:39:51.964 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.965 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.965 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.965 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:51.965 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:39:51.965 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.965 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.965 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.965 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:51.965 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548531+0000", 2024-09-18T04:39:51.965 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:51.965 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.965 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.965 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:51.966 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551500+0000", 2024-09-18T04:39:51.966 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0029688129999999998 2024-09-18T04:39:51.966 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.966 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.966 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:51.966 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551516+0000", 2024-09-18T04:39:51.966 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5994e-05 2024-09-18T04:39:51.966 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.966 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.966 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:51.966 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551526+0000", 2024-09-18T04:39:51.966 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9299999999999998e-06 2024-09-18T04:39:51.966 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.966 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.967 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:51.967 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551577+0000", 2024-09-18T04:39:51.967 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0871000000000001e-05 2024-09-18T04:39:51.967 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.967 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.967 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:51.967 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551602+0000", 2024-09-18T04:39:51.967 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4813000000000001e-05 2024-09-18T04:39:51.967 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.967 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.967 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:51.967 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562741+0000", 2024-09-18T04:39:51.967 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011139428999999999 2024-09-18T04:39:51.968 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.968 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.968 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:51.968 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562797+0000", 2024-09-18T04:39:51.968 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6329000000000002e-05 2024-09-18T04:39:51.968 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.968 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:51.968 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.968 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.968 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.968 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:49 2.4 e51/17 2:20f0b0c9:ns::rbd_data.133e813ab5bc.0000000000000009:head v 51'6, mlcod=51'4)", 2024-09-18T04:39:51.968 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:39:51.968 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 69.344740489000003, 2024-09-18T04:39:51.969 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013849046, 2024-09-18T04:39:51.969 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:51.969 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:51.969 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:51.969 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.969 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:51.969 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:39:51.969 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.969 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.969 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.969 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:51.969 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:39:51.969 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.970 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.970 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.970 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:51.970 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579966+0000", 2024-09-18T04:39:51.970 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:51.970 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.970 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.970 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:51.970 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582324+0000", 2024-09-18T04:39:51.970 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023578890000000002 2024-09-18T04:39:51.970 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.970 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.970 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:51.970 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582355+0000", 2024-09-18T04:39:51.971 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1485000000000001e-05 2024-09-18T04:39:51.971 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.971 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.971 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:51.971 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582371+0000", 2024-09-18T04:39:51.971 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6147999999999999e-05 2024-09-18T04:39:51.971 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.971 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.971 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:51.971 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582467+0000", 2024-09-18T04:39:51.972 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6059999999999998e-05 2024-09-18T04:39:51.972 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.972 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.972 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:51.972 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582505+0000", 2024-09-18T04:39:51.972 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7679000000000003e-05 2024-09-18T04:39:51.972 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.972 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.972 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:51.972 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593743+0000", 2024-09-18T04:39:51.972 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011238356 2024-09-18T04:39:51.972 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.972 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.973 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:51.973 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593816+0000", 2024-09-18T04:39:51.973 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2935000000000006e-05 2024-09-18T04:39:51.973 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.973 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:51.973 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.973 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.973 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.973 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:53 2.4 e51/17 2:39f7f174:ns::rbd_data.133e813ab5bc.000000000000000b:head v 51'7, mlcod=51'4)", 2024-09-18T04:39:51.973 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:39:51.973 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 69.330039334000006, 2024-09-18T04:39:51.974 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0079914159999999995, 2024-09-18T04:39:51.974 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:51.974 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:51.974 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:51.974 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.974 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:51.974 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:39:51.974 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.974 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.974 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.974 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:51.975 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:39:51.975 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.975 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.975 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:51.975 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594667+0000", 2024-09-18T04:39:51.975 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:51.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.975 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.975 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:51.975 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596446+0000", 2024-09-18T04:39:51.975 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001779033 2024-09-18T04:39:51.976 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.976 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.976 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:51.976 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596472+0000", 2024-09-18T04:39:51.976 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6120000000000001e-05 2024-09-18T04:39:51.976 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.976 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.976 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:51.976 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596500+0000", 2024-09-18T04:39:51.976 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7518e-05 2024-09-18T04:39:51.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.977 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.977 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:51.977 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596564+0000", 2024-09-18T04:39:51.977 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3880999999999994e-05 2024-09-18T04:39:51.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.977 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.977 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:51.977 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596596+0000", 2024-09-18T04:39:51.977 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2638999999999997e-05 2024-09-18T04:39:51.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.978 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.978 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:51.978 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602605+0000", 2024-09-18T04:39:51.978 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0060080699999999999 2024-09-18T04:39:51.978 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.978 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.978 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:51.978 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602660+0000", 2024-09-18T04:39:51.978 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5396000000000003e-05 2024-09-18T04:39:51.978 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.978 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:51.978 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.979 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.979 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.979 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:60 2.5 e51/28 2:a06f51e4:ns::rbd_data.133e813ab5bc.000000000000000f:head v 51'6, mlcod=51'4)", 2024-09-18T04:39:51.979 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:39:51.979 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 69.315222348999995, 2024-09-18T04:39:51.979 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0093954989999999999, 2024-09-18T04:39:51.979 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:51.979 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:51.979 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:51.980 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.980 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:51.980 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:39:51.980 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.980 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.980 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:51.980 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:39:51.980 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.980 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.980 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:51.980 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609483+0000", 2024-09-18T04:39:51.981 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:51.981 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.981 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.981 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:51.981 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611767+0000", 2024-09-18T04:39:51.981 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002283724 2024-09-18T04:39:51.981 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.981 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.981 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:51.981 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611791+0000", 2024-09-18T04:39:51.981 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3547e-05 2024-09-18T04:39:51.981 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.981 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.981 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:51.982 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611800+0000", 2024-09-18T04:39:51.982 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4979999999999995e-06 2024-09-18T04:39:51.982 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.982 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.982 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:51.982 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611864+0000", 2024-09-18T04:39:51.982 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3974999999999996e-05 2024-09-18T04:39:51.982 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.982 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.982 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:51.982 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611887+0000", 2024-09-18T04:39:51.982 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2633000000000002e-05 2024-09-18T04:39:51.982 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.982 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.983 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:51.983 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.618777+0000", 2024-09-18T04:39:51.983 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0068897979999999999 2024-09-18T04:39:51.983 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.983 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.983 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:51.983 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.618881+0000", 2024-09-18T04:39:51.983 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010437699999999999 2024-09-18T04:39:51.983 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.983 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:51.983 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.983 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.983 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.984 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:663 2.4 e57/17 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 57'13, mlcod=57'11)", 2024-09-18T04:39:51.984 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.355300+0000", 2024-09-18T04:39:51.984 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 60.569407413999997, 2024-09-18T04:39:51.984 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0049192760000000002, 2024-09-18T04:39:51.984 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:51.984 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:51.984 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:51.984 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.984 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:51.984 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355300+0000", 2024-09-18T04:39:51.984 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.984 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.984 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.984 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:51.985 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355300+0000", 2024-09-18T04:39:51.985 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.985 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.985 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.985 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:51.985 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355299+0000", 2024-09-18T04:39:51.985 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:51.985 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.985 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.985 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:51.985 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355876+0000", 2024-09-18T04:39:51.985 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00057730900000000003 2024-09-18T04:39:51.985 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.986 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.986 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:51.986 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355897+0000", 2024-09-18T04:39:51.986 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0477000000000001e-05 2024-09-18T04:39:51.986 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.986 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.986 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:51.986 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355905+0000", 2024-09-18T04:39:51.986 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5739999999999996e-06 2024-09-18T04:39:51.986 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.986 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.986 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:51.986 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355989+0000", 2024-09-18T04:39:51.986 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4002e-05 2024-09-18T04:39:51.987 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.987 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.987 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:51.987 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.356041+0000", 2024-09-18T04:39:51.987 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1347000000000002e-05 2024-09-18T04:39:51.987 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.987 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.987 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:51.987 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.360172+0000", 2024-09-18T04:39:51.987 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0041315090000000002 2024-09-18T04:39:51.987 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.987 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.987 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:51.987 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.360220+0000", 2024-09-18T04:39:51.988 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7584999999999999e-05 2024-09-18T04:39:51.988 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.988 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:51.988 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.988 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.988 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.988 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:679 2.5 e57/28 2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head v 57'20, mlcod=57'18)", 2024-09-18T04:39:51.988 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.445316+0000", 2024-09-18T04:39:51.988 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 60.479391937000003, 2024-09-18T04:39:51.988 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0047016760000000001, 2024-09-18T04:39:51.988 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:51.988 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:51.988 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:51.989 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.989 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:51.989 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445316+0000", 2024-09-18T04:39:51.989 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.989 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.989 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.989 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:51.989 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445316+0000", 2024-09-18T04:39:51.989 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.989 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.989 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.989 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:51.989 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445314+0000", 2024-09-18T04:39:51.989 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:51.989 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.990 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.990 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:51.990 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445947+0000", 2024-09-18T04:39:51.990 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00063298900000000003 2024-09-18T04:39:51.990 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.990 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.990 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:51.990 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445968+0000", 2024-09-18T04:39:51.990 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.054e-05 2024-09-18T04:39:51.990 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.990 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.990 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:51.990 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445981+0000", 2024-09-18T04:39:51.990 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2697000000000001e-05 2024-09-18T04:39:51.991 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.991 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.991 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:51.991 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.446102+0000", 2024-09-18T04:39:51.991 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012189300000000001 2024-09-18T04:39:51.991 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.991 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.991 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:51.991 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.446151+0000", 2024-09-18T04:39:51.991 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8279000000000003e-05 2024-09-18T04:39:51.991 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.991 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.991 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:51.992 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.449953+0000", 2024-09-18T04:39:51.992 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003801979 2024-09-18T04:39:51.992 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.992 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.992 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:51.992 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.450018+0000", 2024-09-18T04:39:51.992 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4873e-05 2024-09-18T04:39:51.992 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.992 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:51.992 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.992 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.992 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.992 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:699 2.4 e57/17 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 57'17, mlcod=57'15)", 2024-09-18T04:39:51.992 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.566073+0000", 2024-09-18T04:39:51.993 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 60.358634424000002, 2024-09-18T04:39:51.993 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055398380000000001, 2024-09-18T04:39:51.993 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:51.993 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:51.993 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:51.993 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.993 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:51.993 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566073+0000", 2024-09-18T04:39:51.993 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.993 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.993 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.993 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:51.993 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566073+0000", 2024-09-18T04:39:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566071+0000", 2024-09-18T04:39:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566637+0000", 2024-09-18T04:39:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00056516699999999999 2024-09-18T04:39:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566668+0000", 2024-09-18T04:39:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1544999999999997e-05 2024-09-18T04:39:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566685+0000", 2024-09-18T04:39:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6857000000000001e-05 2024-09-18T04:39:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566816+0000", 2024-09-18T04:39:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013092799999999999 2024-09-18T04:39:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.996 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.996 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:51.996 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566875+0000", 2024-09-18T04:39:51.996 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9583999999999997e-05 2024-09-18T04:39:51.996 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.996 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.996 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:51.996 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.571567+0000", 2024-09-18T04:39:51.996 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.004691682 2024-09-18T04:39:51.996 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.996 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.996 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:51.996 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.571613+0000", 2024-09-18T04:39:51.997 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6125000000000003e-05 2024-09-18T04:39:51.997 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.997 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:51.997 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:51.997 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.997 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.997 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:98 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'27, mlcod=60'25)", 2024-09-18T04:39:51.997 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.847978+0000", 2024-09-18T04:39:51.997 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 51.076729864999997, 2024-09-18T04:39:51.997 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0044617709999999998, 2024-09-18T04:39:51.997 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:51.997 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:51.997 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:51.997 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.998 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:51.998 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.847978+0000", 2024-09-18T04:39:51.998 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.998 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.998 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.998 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:51.998 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.847978+0000", 2024-09-18T04:39:51.998 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:51.998 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.998 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.998 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:51.998 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.847977+0000", 2024-09-18T04:39:51.998 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:51.998 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.999 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.999 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:51.999 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849038+0000", 2024-09-18T04:39:51.999 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0010614820000000001 2024-09-18T04:39:51.999 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.999 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.999 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:51.999 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849065+0000", 2024-09-18T04:39:51.999 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7127999999999998e-05 2024-09-18T04:39:51.999 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:51.999 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:51.999 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:51.999 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849082+0000", 2024-09-18T04:39:52.000 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7170999999999999e-05 2024-09-18T04:39:52.000 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:52.000 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.000 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:52.000 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849188+0000", 2024-09-18T04:39:52.000 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010605800000000001 2024-09-18T04:39:52.000 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:52.000 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.000 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:52.000 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849233+0000", 2024-09-18T04:39:52.000 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4579000000000001e-05 2024-09-18T04:39:52.000 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:52.000 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.000 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:52.001 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.852370+0000", 2024-09-18T04:39:52.001 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00313751 2024-09-18T04:39:52.001 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:52.001 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.001 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:52.001 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.852440+0000", 2024-09-18T04:39:52.001 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9238000000000004e-05 2024-09-18T04:39:52.001 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:52.001 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:52.001 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:52.001 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:52.001 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.001 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:100 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'29, mlcod=64'27)", 2024-09-18T04:39:52.002 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.858766+0000", 2024-09-18T04:39:52.002 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 51.065941535999997, 2024-09-18T04:39:52.002 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0061964840000000004, 2024-09-18T04:39:52.002 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:52.002 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:52.002 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:52.002 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.002 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:52.002 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.858766+0000", 2024-09-18T04:39:52.002 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:52.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:52.002 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.002 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:52.002 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.858766+0000", 2024-09-18T04:39:52.003 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:52.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:52.003 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.003 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:52.003 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.858765+0000", 2024-09-18T04:39:52.003 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:52.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:52.003 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.003 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:52.003 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861612+0000", 2024-09-18T04:39:52.003 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028473169999999998 2024-09-18T04:39:52.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:52.003 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.003 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:52.004 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861643+0000", 2024-09-18T04:39:52.004 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1356000000000003e-05 2024-09-18T04:39:52.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:52.004 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.004 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:52.004 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861656+0000", 2024-09-18T04:39:52.004 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2899e-05 2024-09-18T04:39:52.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:52.004 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.004 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:52.004 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861773+0000", 2024-09-18T04:39:52.004 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011637999999999999 2024-09-18T04:39:52.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:52.005 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.005 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:52.005 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861808+0000", 2024-09-18T04:39:52.005 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5002999999999997e-05 2024-09-18T04:39:52.005 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:52.005 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.005 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:52.005 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.864876+0000", 2024-09-18T04:39:52.005 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030680719999999998 2024-09-18T04:39:52.005 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:52.005 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.005 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:52.005 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.864963+0000", 2024-09-18T04:39:52.006 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7114000000000007e-05 2024-09-18T04:39:52.006 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:52.006 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:52.006 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:52.006 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:52.006 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.006 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:102 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'31, mlcod=64'29)", 2024-09-18T04:39:52.006 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.872098+0000", 2024-09-18T04:39:52.006 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 51.052610279, 2024-09-18T04:39:52.006 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0064676450000000002, 2024-09-18T04:39:52.006 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:52.006 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:52.006 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:52.007 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.007 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:52.007 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.872098+0000", 2024-09-18T04:39:52.007 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:52.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:52.007 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.007 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:52.007 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.872098+0000", 2024-09-18T04:39:52.007 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:52.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:52.007 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.007 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:52.007 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.872096+0000", 2024-09-18T04:39:52.007 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:52.008 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:52.008 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.008 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:52.008 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875176+0000", 2024-09-18T04:39:52.008 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030799809999999999 2024-09-18T04:39:52.008 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:52.008 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.008 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:52.008 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875204+0000", 2024-09-18T04:39:52.008 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8025999999999999e-05 2024-09-18T04:39:52.008 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:52.008 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.008 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:52.009 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875216+0000", 2024-09-18T04:39:52.009 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2072999999999999e-05 2024-09-18T04:39:52.009 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:52.009 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.009 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:52.009 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875307+0000", 2024-09-18T04:39:52.009 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0373999999999999e-05 2024-09-18T04:39:52.009 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:52.009 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.009 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:52.009 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875349+0000", 2024-09-18T04:39:52.009 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2299000000000003e-05 2024-09-18T04:39:52.009 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:52.010 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.010 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:52.010 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.878496+0000", 2024-09-18T04:39:52.010 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0031468960000000002 2024-09-18T04:39:52.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:52.010 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.010 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:52.010 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.878565+0000", 2024-09-18T04:39:52.010 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9465e-05 2024-09-18T04:39:52.010 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:52.010 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:52.010 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:52.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:52.010 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.011 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:104 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'33, mlcod=64'31)", 2024-09-18T04:39:52.011 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.885100+0000", 2024-09-18T04:39:52.011 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 51.039607744000001, 2024-09-18T04:39:52.011 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0051888159999999997, 2024-09-18T04:39:52.011 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:52.011 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:52.011 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:52.011 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.011 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:52.011 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.885100+0000", 2024-09-18T04:39:52.011 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:52.011 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:52.011 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.012 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:52.012 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.885100+0000", 2024-09-18T04:39:52.012 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:52.012 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:52.012 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.012 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:52.012 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.885098+0000", 2024-09-18T04:39:52.012 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:52.012 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:52.012 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.012 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:52.012 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886809+0000", 2024-09-18T04:39:52.012 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017110459999999999 2024-09-18T04:39:52.013 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:52.013 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.013 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:52.013 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886842+0000", 2024-09-18T04:39:52.013 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2638999999999997e-05 2024-09-18T04:39:52.013 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:52.013 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.013 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:52.013 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886855+0000", 2024-09-18T04:39:52.013 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3380000000000001e-05 2024-09-18T04:39:52.013 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:52.013 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.013 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:52.014 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886939+0000", 2024-09-18T04:39:52.014 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3733999999999995e-05 2024-09-18T04:39:52.014 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:52.014 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.014 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:52.014 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886981+0000", 2024-09-18T04:39:52.014 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1956000000000003e-05 2024-09-18T04:39:52.014 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:52.014 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.014 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:52.014 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.890201+0000", 2024-09-18T04:39:52.014 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003219927 2024-09-18T04:39:52.014 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:52.015 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.015 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:52.015 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.890289+0000", 2024-09-18T04:39:52.015 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7997999999999993e-05 2024-09-18T04:39:52.015 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:52.015 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:52.015 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:52.015 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:52.015 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.015 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:145 2.4 e64/62 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 64'37, mlcod=64'35)", 2024-09-18T04:39:52.015 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:01.099398+0000", 2024-09-18T04:39:52.015 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 50.825310256000002, 2024-09-18T04:39:52.015 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0050078459999999998, 2024-09-18T04:39:52.016 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:52.016 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:52.016 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:52.016 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.016 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:52.016 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.099398+0000", 2024-09-18T04:39:52.016 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:52.016 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:52.016 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.016 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:52.016 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.099398+0000", 2024-09-18T04:39:52.016 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:52.016 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:52.016 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.017 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:52.017 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.099396+0000", 2024-09-18T04:39:52.017 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:52.017 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:52.017 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.017 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:52.017 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100199+0000", 2024-09-18T04:39:52.017 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00080338000000000002 2024-09-18T04:39:52.017 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:52.017 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.017 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:52.017 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100230+0000", 2024-09-18T04:39:52.017 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0716000000000002e-05 2024-09-18T04:39:52.017 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:52.018 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.018 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:52.018 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100243+0000", 2024-09-18T04:39:52.018 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3359e-05 2024-09-18T04:39:52.018 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:52.018 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.018 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:52.018 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100337+0000", 2024-09-18T04:39:52.019 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4387000000000003e-05 2024-09-18T04:39:52.019 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:52.020 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.020 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:52.020 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100396+0000", 2024-09-18T04:39:52.020 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8164000000000001e-05 2024-09-18T04:39:52.020 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:52.020 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.020 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:52.020 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.104309+0000", 2024-09-18T04:39:52.020 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0039137800000000004 2024-09-18T04:39:52.020 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:52.020 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.020 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:52.021 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.104405+0000", 2024-09-18T04:39:52.021 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6039999999999995e-05 2024-09-18T04:39:52.021 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:52.021 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:52.021 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:52.021 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:52.021 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.021 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:147 2.4 e64/62 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 64'39, mlcod=64'37)", 2024-09-18T04:39:52.022 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:01.110658+0000", 2024-09-18T04:39:52.022 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 50.814050166999998, 2024-09-18T04:39:52.022 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0046316910000000003, 2024-09-18T04:39:52.022 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:52.022 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:52.022 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:52.022 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.022 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:52.022 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.110658+0000", 2024-09-18T04:39:52.022 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:52.022 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:52.023 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.023 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:52.023 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.110658+0000", 2024-09-18T04:39:52.023 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:52.023 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:52.023 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.023 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:52.023 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.110656+0000", 2024-09-18T04:39:52.023 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:52.023 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:52.023 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.023 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:52.024 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111444+0000", 2024-09-18T04:39:52.024 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00078762400000000001 2024-09-18T04:39:52.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:52.024 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.024 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:52.024 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111474+0000", 2024-09-18T04:39:52.024 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0085000000000001e-05 2024-09-18T04:39:52.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:52.024 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.024 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:52.024 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111486+0000", 2024-09-18T04:39:52.024 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2645e-05 2024-09-18T04:39:52.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:52.025 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.025 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:52.025 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111581+0000", 2024-09-18T04:39:52.025 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4432000000000007e-05 2024-09-18T04:39:52.025 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:52.025 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.025 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:52.025 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111631+0000", 2024-09-18T04:39:52.025 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9901000000000002e-05 2024-09-18T04:39:52.025 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:52.025 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.025 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:52.025 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.115209+0000", 2024-09-18T04:39:52.026 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003577916 2024-09-18T04:39:52.026 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:52.026 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:52.026 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:52.026 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.115289+0000", 2024-09-18T04:39:52.026 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0704999999999994e-05 2024-09-18T04:39:52.026 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:52.026 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:52.026 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:52.026 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:52.026 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:52.026 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:52.028 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:39:52.029 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:52.023+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:52.037 INFO:teuthology.orchestra.run.smithi028.stdout:moved 0 / 34 (0%) 2024-09-18T04:39:52.037 INFO:teuthology.orchestra.run.smithi028.stdout:avg 6.8 2024-09-18T04:39:52.037 INFO:teuthology.orchestra.run.smithi028.stdout:stddev 1.46969 -> 1.46969 (expected baseline 2.33238) 2024-09-18T04:39:52.037 INFO:teuthology.orchestra.run.smithi028.stdout:min osd.3 with 5 -> 5 pgs (0.735294 -> 0.735294 * mean) 2024-09-18T04:39:52.037 INFO:teuthology.orchestra.run.smithi028.stdout:max osd.1 with 9 -> 9 pgs (1.32353 -> 1.32353 * mean) 2024-09-18T04:39:52.037 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:39:52.037 INFO:teuthology.orchestra.run.smithi028.stdout:oload 1000 2024-09-18T04:39:52.037 INFO:teuthology.orchestra.run.smithi028.stdout:max_change 0.05 2024-09-18T04:39:52.038 INFO:teuthology.orchestra.run.smithi028.stdout:max_change_osds 4 2024-09-18T04:39:52.038 INFO:teuthology.orchestra.run.smithi028.stdout:average_utilization 0.0009 2024-09-18T04:39:52.038 INFO:teuthology.orchestra.run.smithi028.stdout:overload_utilization 0.0093 2024-09-18T04:39:52.041 INFO:teuthology.orchestra.run.smithi028.stderr:no change 2024-09-18T04:39:52.128 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:52.123+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:52.129 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T04:39:52.137 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:52.137 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:52.137 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:39:52.137 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:52.146 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:39:52.229 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:52.220+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:52.253 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:39:52.274 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:39:52.291 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:39:52.297 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:52.297 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:52.307 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:39:52.314 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:39:52.330 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:52.327+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:52.335 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:39:52.356 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:39:52.358 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:39:52.358 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:39:52.358 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:39:52.358 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:39:52.359 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:39:52.359 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:39:52.406 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:52.407 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:52.416 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:39:52.466 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:52.460+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:52.532 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:52.532 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:52.532 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:52.527+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:52.541 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:39:52.636 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:52.624+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:52.647 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:52.647 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:52.657 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:39:52.734 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:52.727+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:52.807 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:52.807 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:52.816 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:39:52.834 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:52.831+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:52.921 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:52.921 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:52.931 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:39:52.935 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:52.928+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:53.036 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:53.028+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:53.037 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:53.037 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:53.047 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:39:53.138 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:53.131+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:39:53.201 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:53.201 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:53.211 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:39:53.238 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:53.235+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:53.319 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:53.319 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:53.329 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:39:53.339 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:53.335+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:53.440 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:53.432+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:53.441 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:53.441 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:53.450 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:39:53.542 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:53.535+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:39:53.557 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:53.557 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:53.567 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:39:53.642 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:53.639+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:39:53.687 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:53.687 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:53.698 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:39:53.744 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:53.736+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:53.767 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-18T04:39:53.831 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:53.831 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:53.841 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:39:53.844 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:53.839+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:39:53.944 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:53.939+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:53.953 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:53.953 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:53.963 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:39:54.045 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:54.040+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:54.068 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:54.068 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:54.078 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:39:54.146 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:54.143+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:54.182 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:54.182 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:54.191 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:39:54.247 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:54.243+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:39:54.296 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:54.296 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:54.305 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:39:54.350 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:54.347+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:54.372 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T04:39:54.432 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:54.432 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:54.441 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:39:54.449 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:54.444+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:54.513 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:39:54.532 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:39:54.550 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:54.547+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:54.551 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:39:54.553 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:54.553 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:54.553 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:39:54.553 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:39:54.554 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:54.563 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:39:54.576 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:39:54.599 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:39:54.623 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:39:54.625 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:39:54.625 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:39:54.625 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:39:54.625 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:39:54.625 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:39:54.625 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:39:54.651 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:54.644+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:54.667 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:54.668 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:39:54.668 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:39:54.668 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:39:54.668 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.668 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:33 2.1 e51/24 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head v 51'4, mlcod=51'2)", 2024-09-18T04:39:54.668 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:39:54.668 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 72.383683821000005, 2024-09-18T04:39:54.668 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018702106, 2024-09-18T04:39:54.668 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:54.668 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:54.668 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:54.669 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.669 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:54.669 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:39:54.669 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:54.669 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.669 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.669 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:54.669 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:39:54.669 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:54.669 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.669 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.669 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:54.669 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279197+0000", 2024-09-18T04:39:54.669 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:54.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.670 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.670 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:54.670 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282682+0000", 2024-09-18T04:39:54.670 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0034848689999999998 2024-09-18T04:39:54.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.670 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.670 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:54.670 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282709+0000", 2024-09-18T04:39:54.670 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7599000000000001e-05 2024-09-18T04:39:54.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.670 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.670 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:54.670 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282720+0000", 2024-09-18T04:39:54.670 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.092e-05 2024-09-18T04:39:54.671 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.671 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.671 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:54.671 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282791+0000", 2024-09-18T04:39:54.671 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1268999999999998e-05 2024-09-18T04:39:54.671 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.671 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.671 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:54.671 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282843+0000", 2024-09-18T04:39:54.671 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1771999999999998e-05 2024-09-18T04:39:54.671 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.671 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.671 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:54.672 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.297830+0000", 2024-09-18T04:39:54.672 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014987228999999999 2024-09-18T04:39:54.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.672 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.672 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:54.672 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.297901+0000", 2024-09-18T04:39:54.672 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0209999999999994e-05 2024-09-18T04:39:54.672 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:54.672 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:54.672 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:54.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.672 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.672 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:38 2.0 e51/17 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head v 51'10, mlcod=51'8)", 2024-09-18T04:39:54.672 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:39:54.673 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 72.382265391000004, 2024-09-18T04:39:54.673 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.023984472, 2024-09-18T04:39:54.673 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:54.673 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:54.673 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:54.673 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.673 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:54.673 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:39:54.673 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:54.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.674 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.674 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:54.674 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:39:54.674 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:54.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.674 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.674 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:54.674 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280614+0000", 2024-09-18T04:39:54.674 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:54.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.674 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.675 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:54.675 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293094+0000", 2024-09-18T04:39:54.675 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012479571 2024-09-18T04:39:54.675 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.675 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.675 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:54.675 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293111+0000", 2024-09-18T04:39:54.675 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7354999999999999e-05 2024-09-18T04:39:54.675 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.675 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.675 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:54.675 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293121+0000", 2024-09-18T04:39:54.675 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6979999999999993e-06 2024-09-18T04:39:54.675 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.675 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.676 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:54.676 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293180+0000", 2024-09-18T04:39:54.676 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9333000000000003e-05 2024-09-18T04:39:54.676 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.676 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.676 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:54.676 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293203+0000", 2024-09-18T04:39:54.676 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2633000000000002e-05 2024-09-18T04:39:54.676 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.677 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.677 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:54.677 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304534+0000", 2024-09-18T04:39:54.677 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011330761 2024-09-18T04:39:54.677 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.677 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.677 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:54.677 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304601+0000", 2024-09-18T04:39:54.677 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7657000000000006e-05 2024-09-18T04:39:54.677 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:54.677 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:54.678 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:54.678 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.678 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.678 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:46 2.6 2:7ba25245:::rbd_data.125fc09df6d1.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:54.678 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:39:54.678 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 72.354722590999998, 2024-09-18T04:39:54.679 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.035604595000000003, 2024-09-18T04:39:54.680 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:54.680 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:54.680 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:54.680 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:54.680 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:54.680 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 46 2024-09-18T04:39:54.680 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.680 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:54.680 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.680 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:54.680 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:39:54.680 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:54.681 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.681 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.681 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:54.681 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:39:54.681 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:54.681 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.681 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.681 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:54.681 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308158+0000", 2024-09-18T04:39:54.681 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:54.681 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.681 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.681 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:54.681 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313191+0000", 2024-09-18T04:39:54.682 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0050323929999999996 2024-09-18T04:39:54.682 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.682 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.682 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:54.682 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313210+0000", 2024-09-18T04:39:54.682 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9673e-05 2024-09-18T04:39:54.682 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.682 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.682 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:54.682 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313221+0000", 2024-09-18T04:39:54.682 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0444e-05 2024-09-18T04:39:54.682 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.682 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.682 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:54.683 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313294+0000", 2024-09-18T04:39:54.683 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2650000000000004e-05 2024-09-18T04:39:54.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.683 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.683 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:54.683 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313493+0000", 2024-09-18T04:39:54.683 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019968300000000001 2024-09-18T04:39:54.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.683 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.683 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:54.683 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313666+0000", 2024-09-18T04:39:54.683 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00017248599999999999 2024-09-18T04:39:54.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.684 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.684 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:54.684 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.323175+0000", 2024-09-18T04:39:54.684 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0095088819999999998 2024-09-18T04:39:54.684 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.684 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.684 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:54.684 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343651+0000", 2024-09-18T04:39:54.684 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020475892999999998 2024-09-18T04:39:54.684 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.684 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.684 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:54.684 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343715+0000", 2024-09-18T04:39:54.685 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4443999999999995e-05 2024-09-18T04:39:54.685 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.685 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.685 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:54.685 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343764+0000", 2024-09-18T04:39:54.685 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9280000000000003e-05 2024-09-18T04:39:54.685 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:54.685 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:54.685 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:54.685 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.685 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.685 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:47 2.4 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:54.685 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:39:54.685 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 72.349640554000004, 2024-09-18T04:39:54.686 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.032570693999999997, 2024-09-18T04:39:54.686 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:54.686 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:54.686 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:54.686 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:54.686 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:54.686 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 47 2024-09-18T04:39:54.686 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.686 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:54.686 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.686 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:54.686 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:39:54.687 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:54.687 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.687 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.687 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:54.687 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:39:54.687 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:54.687 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.687 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.688 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:54.688 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313240+0000", 2024-09-18T04:39:54.688 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:54.688 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.688 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.688 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:54.688 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325825+0000", 2024-09-18T04:39:54.688 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012584425999999999 2024-09-18T04:39:54.688 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.688 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.689 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:54.689 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325848+0000", 2024-09-18T04:39:54.689 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3025999999999999e-05 2024-09-18T04:39:54.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.689 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.689 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:54.689 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325863+0000", 2024-09-18T04:39:54.689 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5152e-05 2024-09-18T04:39:54.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.689 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.689 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:54.689 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325938+0000", 2024-09-18T04:39:54.689 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4982999999999997e-05 2024-09-18T04:39:54.690 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.690 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.690 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:54.690 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326090+0000", 2024-09-18T04:39:54.690 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015207900000000001 2024-09-18T04:39:54.690 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.690 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.690 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:54.690 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326242+0000", 2024-09-18T04:39:54.690 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015163500000000001 2024-09-18T04:39:54.690 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.690 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.690 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:54.691 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343015+0000", 2024-09-18T04:39:54.691 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016773414 2024-09-18T04:39:54.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.691 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.691 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:54.691 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345664+0000", 2024-09-18T04:39:54.691 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026487870000000001 2024-09-18T04:39:54.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.691 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.691 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:54.691 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345739+0000", 2024-09-18T04:39:54.691 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5018999999999994e-05 2024-09-18T04:39:54.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.691 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.692 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:54.692 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345812+0000", 2024-09-18T04:39:54.692 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3547000000000006e-05 2024-09-18T04:39:54.692 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:54.692 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:54.692 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:54.692 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.692 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.692 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:48 2.6 2:61db7a3a:::rbd_data.125fc09df6d1.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:54.692 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:39:54.692 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 72.336983881999998, 2024-09-18T04:39:54.692 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.025064388, 2024-09-18T04:39:54.692 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:54.692 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:54.693 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:54.693 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:54.693 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:54.693 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 48 2024-09-18T04:39:54.693 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.693 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:54.693 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.693 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:54.693 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:39:54.693 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:54.693 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.693 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.693 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:54.693 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:39:54.694 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:54.694 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.694 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.694 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:54.694 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325897+0000", 2024-09-18T04:39:54.694 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:54.694 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.694 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.694 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:54.694 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333697+0000", 2024-09-18T04:39:54.694 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0078002649999999998 2024-09-18T04:39:54.694 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.694 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.694 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:54.694 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333716+0000", 2024-09-18T04:39:54.695 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8910999999999999e-05 2024-09-18T04:39:54.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.695 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.695 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:54.695 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333728+0000", 2024-09-18T04:39:54.695 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1936000000000001e-05 2024-09-18T04:39:54.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.695 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.695 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:54.695 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333792+0000", 2024-09-18T04:39:54.695 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4010999999999994e-05 2024-09-18T04:39:54.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.695 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.695 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:54.696 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.334082+0000", 2024-09-18T04:39:54.696 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00028955799999999998 2024-09-18T04:39:54.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.696 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.696 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:54.696 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.334326+0000", 2024-09-18T04:39:54.696 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00024399099999999999 2024-09-18T04:39:54.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.696 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.696 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:54.696 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350745+0000", 2024-09-18T04:39:54.696 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016419474999999999 2024-09-18T04:39:54.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.696 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.697 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:54.697 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350808+0000", 2024-09-18T04:39:54.697 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3244999999999995e-05 2024-09-18T04:39:54.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.697 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.697 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:54.697 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350892+0000", 2024-09-18T04:39:54.697 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4074999999999997e-05 2024-09-18T04:39:54.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.697 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.697 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:54.697 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350963+0000", 2024-09-18T04:39:54.697 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0412000000000003e-05 2024-09-18T04:39:54.697 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:54.698 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:54.698 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:54.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.698 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.698 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:51 2.6 2:775d7b11:::rbd_data.125fc09df6d1.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:54.698 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:39:54.698 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 72.329131746000002, 2024-09-18T04:39:54.698 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.023824221, 2024-09-18T04:39:54.698 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:54.698 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:54.698 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:54.698 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:54.698 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:54.698 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 51 2024-09-18T04:39:54.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.699 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:54.699 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.699 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:54.699 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:39:54.699 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:54.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.699 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.699 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:54.699 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:39:54.699 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:54.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.699 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.699 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:54.700 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333749+0000", 2024-09-18T04:39:54.700 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:54.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.700 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.700 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:54.700 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340899+0000", 2024-09-18T04:39:54.700 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0071500340000000004 2024-09-18T04:39:54.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.700 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.700 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:54.700 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340929+0000", 2024-09-18T04:39:54.700 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0093000000000001e-05 2024-09-18T04:39:54.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.700 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.701 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:54.701 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340943+0000", 2024-09-18T04:39:54.701 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4019000000000001e-05 2024-09-18T04:39:54.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.701 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.701 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:54.701 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343834+0000", 2024-09-18T04:39:54.701 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028918120000000001 2024-09-18T04:39:54.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.701 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.701 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:54.701 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344020+0000", 2024-09-18T04:39:54.701 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000185892 2024-09-18T04:39:54.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.702 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.702 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:54.702 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344190+0000", 2024-09-18T04:39:54.702 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016946800000000001 2024-09-18T04:39:54.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.702 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.702 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:54.702 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.351291+0000", 2024-09-18T04:39:54.702 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.007100805 2024-09-18T04:39:54.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.702 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.702 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:54.702 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357470+0000", 2024-09-18T04:39:54.702 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0061797570000000001 2024-09-18T04:39:54.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.703 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.703 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:54.703 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357533+0000", 2024-09-18T04:39:54.703 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2906999999999999e-05 2024-09-18T04:39:54.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.703 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.703 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:54.703 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357575+0000", 2024-09-18T04:39:54.703 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1403999999999999e-05 2024-09-18T04:39:54.703 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:54.703 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:54.703 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:54.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.704 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.704 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:56 2.4 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:54.704 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:39:54.704 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 72.313086652999999, 2024-09-18T04:39:54.704 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.034295870999999999, 2024-09-18T04:39:54.704 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:54.704 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:54.704 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:54.704 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:54.704 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:54.704 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 56 2024-09-18T04:39:54.704 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.704 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:54.704 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.705 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:54.705 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:39:54.705 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:54.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.705 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.705 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:54.705 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:39:54.705 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:54.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.705 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.705 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:54.705 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349794+0000", 2024-09-18T04:39:54.705 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:54.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.706 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.706 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:54.706 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366632+0000", 2024-09-18T04:39:54.706 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01683755 2024-09-18T04:39:54.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.706 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.706 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:54.706 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366669+0000", 2024-09-18T04:39:54.706 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7351000000000002e-05 2024-09-18T04:39:54.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.707 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.707 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:54.707 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366686+0000", 2024-09-18T04:39:54.707 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6507999999999998e-05 2024-09-18T04:39:54.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.707 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.707 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:54.707 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366962+0000", 2024-09-18T04:39:54.707 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00027655900000000001 2024-09-18T04:39:54.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.707 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.707 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:54.707 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.367452+0000", 2024-09-18T04:39:54.708 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00049025299999999996 2024-09-18T04:39:54.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.708 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.708 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:54.708 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368113+0000", 2024-09-18T04:39:54.708 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00066093399999999996 2024-09-18T04:39:54.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.708 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.708 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:54.708 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383719+0000", 2024-09-18T04:39:54.708 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015606106999999999 2024-09-18T04:39:54.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.708 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.709 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:54.709 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384035+0000", 2024-09-18T04:39:54.709 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00031590399999999999 2024-09-18T04:39:54.709 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.709 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.709 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:54.709 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384067+0000", 2024-09-18T04:39:54.709 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.129e-05 2024-09-18T04:39:54.709 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.709 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.709 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:54.709 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384091+0000", 2024-09-18T04:39:54.709 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4788000000000001e-05 2024-09-18T04:39:54.709 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:54.710 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:54.710 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:54.710 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.710 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.710 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:57 2.0 e51/17 2:0cdd070e:::rbd_data.125fc09df6d1.000000000000000d:head v 51'21, mlcod=51'19)", 2024-09-18T04:39:54.710 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:39:54.710 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 72.305259165999999, 2024-09-18T04:39:54.710 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.025216311000000002, 2024-09-18T04:39:54.710 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:54.710 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:54.711 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:54.711 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.711 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:54.711 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:39:54.711 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:54.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.711 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.711 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:54.711 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:39:54.711 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:54.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.711 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.711 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:54.711 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:39:54.712 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:54.712 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.712 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.712 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:54.712 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363844+0000", 2024-09-18T04:39:54.712 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0062218450000000002 2024-09-18T04:39:54.712 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.712 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.712 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:54.712 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363879+0000", 2024-09-18T04:39:54.712 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4382999999999999e-05 2024-09-18T04:39:54.712 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.712 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.713 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:54.713 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363893+0000", 2024-09-18T04:39:54.713 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4348999999999999e-05 2024-09-18T04:39:54.713 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.713 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.713 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:54.713 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363977+0000", 2024-09-18T04:39:54.713 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3499999999999997e-05 2024-09-18T04:39:54.713 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.713 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.713 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:54.713 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364009+0000", 2024-09-18T04:39:54.713 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2292000000000002e-05 2024-09-18T04:39:54.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.714 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.714 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:54.714 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.382781+0000", 2024-09-18T04:39:54.714 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018771626 2024-09-18T04:39:54.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.714 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.714 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:54.714 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.382839+0000", 2024-09-18T04:39:54.714 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8758000000000002e-05 2024-09-18T04:39:54.714 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:54.714 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:54.714 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:54.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.715 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.715 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:59 2.0 e51/17 2:19c52cde:::rbd_data.125fc09df6d1.000000000000000f:head v 51'22, mlcod=51'19)", 2024-09-18T04:39:54.715 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:39:54.715 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 72.298599240000001, 2024-09-18T04:39:54.715 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.019497112, 2024-09-18T04:39:54.715 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:54.715 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:54.715 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:54.715 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.715 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:54.715 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:39:54.715 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:54.715 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.715 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.715 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:54.716 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:39:54.716 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:54.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.716 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.716 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:54.716 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364282+0000", 2024-09-18T04:39:54.716 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:54.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.716 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.716 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:54.716 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371823+0000", 2024-09-18T04:39:54.716 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0075411829999999999 2024-09-18T04:39:54.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.717 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.717 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:54.717 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371837+0000", 2024-09-18T04:39:54.717 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4336e-05 2024-09-18T04:39:54.717 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.717 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.717 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:54.717 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371846+0000", 2024-09-18T04:39:54.717 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9770000000000006e-06 2024-09-18T04:39:54.717 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.717 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.717 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:54.717 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.373566+0000", 2024-09-18T04:39:54.717 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017192539999999999 2024-09-18T04:39:54.718 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.718 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.718 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:54.718 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.373595+0000", 2024-09-18T04:39:54.718 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8975999999999998e-05 2024-09-18T04:39:54.718 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.718 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.718 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:54.718 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383740+0000", 2024-09-18T04:39:54.719 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01014525 2024-09-18T04:39:54.719 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.719 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.719 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:54.719 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383780+0000", 2024-09-18T04:39:54.719 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0275999999999999e-05 2024-09-18T04:39:54.719 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:54.719 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:54.719 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:54.720 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.720 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.720 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:34 2.6 2:66289171:ns::rbd_data.133e813ab5bc.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:54.720 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.524015+0000", 2024-09-18T04:39:54.720 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 72.138867630999997, 2024-09-18T04:39:54.720 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027582111999999999, 2024-09-18T04:39:54.720 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:54.720 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:54.720 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:54.721 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:54.721 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:54.721 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 34 2024-09-18T04:39:54.721 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.721 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:54.721 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.721 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:54.721 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524015+0000", 2024-09-18T04:39:54.721 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:54.721 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.721 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.721 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:54.722 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524015+0000", 2024-09-18T04:39:54.722 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:54.722 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.722 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.722 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:54.722 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524012+0000", 2024-09-18T04:39:54.722 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:54.722 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.722 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.722 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:54.722 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534956+0000", 2024-09-18T04:39:54.722 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010943403000000001 2024-09-18T04:39:54.722 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.722 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.723 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:54.723 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534978+0000", 2024-09-18T04:39:54.723 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1965999999999999e-05 2024-09-18T04:39:54.723 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.723 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.723 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:54.723 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534990+0000", 2024-09-18T04:39:54.723 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2065999999999999e-05 2024-09-18T04:39:54.723 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.723 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.723 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:54.723 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535065+0000", 2024-09-18T04:39:54.723 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5523000000000002e-05 2024-09-18T04:39:54.724 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.724 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.724 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:54.724 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535208+0000", 2024-09-18T04:39:54.724 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014309500000000001 2024-09-18T04:39:54.724 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.724 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.724 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:54.724 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535343+0000", 2024-09-18T04:39:54.724 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013437400000000001 2024-09-18T04:39:54.724 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.724 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.724 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:54.725 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.546576+0000", 2024-09-18T04:39:54.725 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011233425999999999 2024-09-18T04:39:54.725 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.725 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.726 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:54.726 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551498+0000", 2024-09-18T04:39:54.726 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0049219349999999997 2024-09-18T04:39:54.726 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.727 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.727 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:54.727 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551554+0000", 2024-09-18T04:39:54.727 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6400999999999998e-05 2024-09-18T04:39:54.728 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.728 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.728 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:54.728 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551597+0000", 2024-09-18T04:39:54.729 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2283000000000002e-05 2024-09-18T04:39:54.729 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:54.729 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:54.729 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:54.730 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.730 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.730 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:33 2.1 e51/24 2:8b40f1c5:ns::rbd_data.133e813ab5bc.0000000000000000:head v 51'7, mlcod=51'5)", 2024-09-18T04:39:54.730 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.531767+0000", 2024-09-18T04:39:54.731 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 72.131115714000003, 2024-09-18T04:39:54.731 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015370295000000001, 2024-09-18T04:39:54.731 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:54.731 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:54.731 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:54.731 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.731 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:54.732 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531767+0000", 2024-09-18T04:39:54.732 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:54.732 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.732 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.732 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:54.733 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531767+0000", 2024-09-18T04:39:54.733 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:54.733 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.733 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.734 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:54.734 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531765+0000", 2024-09-18T04:39:54.734 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:54.734 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.734 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.735 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:54.735 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534258+0000", 2024-09-18T04:39:54.735 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002493561 2024-09-18T04:39:54.735 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.735 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.735 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:54.736 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534288+0000", 2024-09-18T04:39:54.736 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.976e-05 2024-09-18T04:39:54.736 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.736 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.736 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:54.736 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534300+0000", 2024-09-18T04:39:54.737 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1928e-05 2024-09-18T04:39:54.737 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.737 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.737 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:54.737 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534367+0000", 2024-09-18T04:39:54.738 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7045999999999995e-05 2024-09-18T04:39:54.738 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.738 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.738 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:54.738 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534394+0000", 2024-09-18T04:39:54.738 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6903999999999999e-05 2024-09-18T04:39:54.739 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.739 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.739 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:54.739 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.547073+0000", 2024-09-18T04:39:54.739 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012679365 2024-09-18T04:39:54.740 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.740 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.740 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:54.740 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.547137+0000", 2024-09-18T04:39:54.740 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3691999999999999e-05 2024-09-18T04:39:54.740 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:54.741 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:54.741 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:54.741 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.741 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.741 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:39 2.4 2:304364d1:ns::rbd_data.133e813ab5bc.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:54.742 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.535013+0000", 2024-09-18T04:39:54.742 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 72.127868778999996, 2024-09-18T04:39:54.742 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027764487000000001, 2024-09-18T04:39:54.742 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:54.742 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:54.742 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:54.742 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:54.743 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:54.743 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 39 2024-09-18T04:39:54.743 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.743 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:54.743 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.743 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:54.744 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535013+0000", 2024-09-18T04:39:54.744 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:54.744 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.744 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.744 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:54.745 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535013+0000", 2024-09-18T04:39:54.745 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:54.745 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.745 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.745 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:54.745 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535012+0000", 2024-09-18T04:39:54.746 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:54.746 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.746 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.746 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:54.746 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545016+0000", 2024-09-18T04:39:54.747 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01000322 2024-09-18T04:39:54.747 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.747 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.747 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:54.747 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545035+0000", 2024-09-18T04:39:54.747 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9845999999999999e-05 2024-09-18T04:39:54.748 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.748 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.748 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:54.748 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545045+0000", 2024-09-18T04:39:54.748 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3640000000000005e-06 2024-09-18T04:39:54.748 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.749 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.749 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:54.749 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545152+0000", 2024-09-18T04:39:54.749 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000107025 2024-09-18T04:39:54.749 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.749 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.749 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:54.750 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545338+0000", 2024-09-18T04:39:54.750 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018643500000000001 2024-09-18T04:39:54.750 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.750 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.750 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:54.751 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545492+0000", 2024-09-18T04:39:54.751 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015393199999999999 2024-09-18T04:39:54.751 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.751 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.751 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:54.751 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.553549+0000", 2024-09-18T04:39:54.752 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0080569170000000002 2024-09-18T04:39:54.752 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.752 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.752 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:54.752 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562693+0000", 2024-09-18T04:39:54.752 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0091441609999999996 2024-09-18T04:39:54.753 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.753 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.753 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:54.753 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562748+0000", 2024-09-18T04:39:54.753 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5047000000000003e-05 2024-09-18T04:39:54.753 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.754 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.754 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:54.754 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562778+0000", 2024-09-18T04:39:54.754 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9652e-05 2024-09-18T04:39:54.754 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:54.755 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:54.755 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:54.755 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.755 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.755 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:40 2.6 2:784d6cd1:ns::rbd_data.133e813ab5bc.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:54.755 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.545062+0000", 2024-09-18T04:39:54.756 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 72.117819728000001, 2024-09-18T04:39:54.756 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.032771623, 2024-09-18T04:39:54.756 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:54.756 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:54.756 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:54.757 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:54.757 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:54.757 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 40 2024-09-18T04:39:54.757 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.757 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:54.757 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.758 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:54.758 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545062+0000", 2024-09-18T04:39:54.758 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:54.758 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.758 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.759 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:54.759 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545062+0000", 2024-09-18T04:39:54.759 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:54.759 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.759 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.760 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:54.760 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545061+0000", 2024-09-18T04:39:54.760 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:54.760 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.761 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.761 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:54.761 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552477+0000", 2024-09-18T04:39:54.761 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.007415914 2024-09-18T04:39:54.761 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.761 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.762 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:54.762 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552500+0000", 2024-09-18T04:39:54.762 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3546000000000001e-05 2024-09-18T04:39:54.762 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.762 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.762 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:54.763 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552515+0000", 2024-09-18T04:39:54.763 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4606999999999999e-05 2024-09-18T04:39:54.763 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.763 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.763 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:54.763 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552661+0000", 2024-09-18T04:39:54.763 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000146007 2024-09-18T04:39:54.764 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.764 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.764 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:54.764 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552876+0000", 2024-09-18T04:39:54.764 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000214618 2024-09-18T04:39:54.764 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.764 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.765 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:54.765 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.553061+0000", 2024-09-18T04:39:54.765 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018564100000000001 2024-09-18T04:39:54.765 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.765 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.766 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:54.766 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.565612+0000", 2024-09-18T04:39:54.766 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012550301999999999 2024-09-18T04:39:54.766 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.767 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.767 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:54.767 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577695+0000", 2024-09-18T04:39:54.767 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012083493000000001 2024-09-18T04:39:54.767 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.768 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.768 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:54.768 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577772+0000", 2024-09-18T04:39:54.768 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7158999999999998e-05 2024-09-18T04:39:54.769 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.769 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.769 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:54.769 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577834+0000", 2024-09-18T04:39:54.769 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1851000000000001e-05 2024-09-18T04:39:54.770 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:54.770 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:54.770 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:54.770 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.770 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.771 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:43 2.6 2:759f27b6:ns::rbd_data.133e813ab5bc.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:54.771 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.555868+0000", 2024-09-18T04:39:54.771 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 72.107014234000005, 2024-09-18T04:39:54.771 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.022332497999999999, 2024-09-18T04:39:54.771 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:54.772 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:54.772 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:54.772 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:54.772 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:54.773 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 43 2024-09-18T04:39:54.773 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.773 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:54.773 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.773 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:54.774 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555868+0000", 2024-09-18T04:39:54.774 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:54.774 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.774 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.774 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:54.775 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555868+0000", 2024-09-18T04:39:54.775 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:54.775 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.775 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.775 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:54.776 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555865+0000", 2024-09-18T04:39:54.776 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999971 2024-09-18T04:39:54.776 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.776 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.777 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:54.777 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561031+0000", 2024-09-18T04:39:54.777 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0051658770000000001 2024-09-18T04:39:54.777 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.778 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.778 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:54.778 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561048+0000", 2024-09-18T04:39:54.778 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6569e-05 2024-09-18T04:39:54.778 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.779 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.779 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:54.779 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561055+0000", 2024-09-18T04:39:54.779 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.328e-06 2024-09-18T04:39:54.780 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.780 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.780 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:54.780 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561114+0000", 2024-09-18T04:39:54.780 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9540000000000003e-05 2024-09-18T04:39:54.781 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.781 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.781 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:54.781 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561314+0000", 2024-09-18T04:39:54.781 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019965799999999999 2024-09-18T04:39:54.782 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.782 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.782 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:54.782 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561473+0000", 2024-09-18T04:39:54.783 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000158904 2024-09-18T04:39:54.783 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.783 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.783 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:54.783 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.567015+0000", 2024-09-18T04:39:54.784 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055420859999999999 2024-09-18T04:39:54.784 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.784 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.784 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:54.784 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577982+0000", 2024-09-18T04:39:54.785 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010966872000000001 2024-09-18T04:39:54.785 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.785 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.785 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:54.785 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578162+0000", 2024-09-18T04:39:54.785 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018050999999999999 2024-09-18T04:39:54.786 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.786 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.786 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:54.786 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578200+0000", 2024-09-18T04:39:54.786 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8000999999999998e-05 2024-09-18T04:39:54.786 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:54.787 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:54.787 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:54.787 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.787 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.787 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:48 2.6 2:7c225365:ns::rbd_data.133e813ab5bc.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:54.787 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:39:54.788 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 72.099215010999998, 2024-09-18T04:39:54.788 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.024576377, 2024-09-18T04:39:54.788 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:54.788 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:54.788 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:54.788 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:54.789 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:54.789 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 48 2024-09-18T04:39:54.789 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.789 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:54.789 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.790 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:54.790 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:39:54.790 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:54.790 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.790 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.790 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:54.791 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:39:54.791 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:54.791 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.791 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.791 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:54.791 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.563666+0000", 2024-09-18T04:39:54.792 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:54.792 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.792 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.792 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:54.792 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570331+0000", 2024-09-18T04:39:54.792 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0066653329999999998 2024-09-18T04:39:54.793 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.793 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.793 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:54.793 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570348+0000", 2024-09-18T04:39:54.793 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6507e-05 2024-09-18T04:39:54.793 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.794 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.794 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:54.794 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570355+0000", 2024-09-18T04:39:54.794 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2749999999999998e-06 2024-09-18T04:39:54.794 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.794 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.794 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:54.795 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570396+0000", 2024-09-18T04:39:54.795 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0809000000000002e-05 2024-09-18T04:39:54.795 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.795 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.795 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:54.795 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570571+0000", 2024-09-18T04:39:54.795 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000174823 2024-09-18T04:39:54.795 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.795 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.795 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:54.795 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570722+0000", 2024-09-18T04:39:54.795 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015154599999999999 2024-09-18T04:39:54.795 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.795 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.795 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:54.796 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579756+0000", 2024-09-18T04:39:54.796 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0090337000000000004 2024-09-18T04:39:54.796 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.796 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.796 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:54.796 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588093+0000", 2024-09-18T04:39:54.796 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.008336553 2024-09-18T04:39:54.796 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.796 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.796 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:54.796 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588173+0000", 2024-09-18T04:39:54.796 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0333999999999994e-05 2024-09-18T04:39:54.796 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.796 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.797 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:54.797 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588244+0000", 2024-09-18T04:39:54.797 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0573000000000003e-05 2024-09-18T04:39:54.797 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:54.797 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:54.797 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:54.797 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.797 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.797 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:49 2.4 2:20f0b0c9:ns::rbd_data.133e813ab5bc.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:54.797 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.570370+0000", 2024-09-18T04:39:54.797 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 72.092512683999999, 2024-09-18T04:39:54.797 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.029633981, 2024-09-18T04:39:54.797 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:54.797 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:54.798 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:54.798 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:54.798 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:54.798 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 49 2024-09-18T04:39:54.798 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.798 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:54.798 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.798 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:54.798 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570370+0000", 2024-09-18T04:39:54.798 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:54.798 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.798 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.798 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:54.798 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570370+0000", 2024-09-18T04:39:54.799 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:54.799 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.799 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.799 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:54.799 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570369+0000", 2024-09-18T04:39:54.799 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:54.799 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.799 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.799 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:54.799 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579090+0000", 2024-09-18T04:39:54.799 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0087212339999999996 2024-09-18T04:39:54.799 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.799 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.799 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:54.800 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579109+0000", 2024-09-18T04:39:54.800 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9120999999999999e-05 2024-09-18T04:39:54.800 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.800 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.800 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:54.800 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579123+0000", 2024-09-18T04:39:54.800 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4382000000000001e-05 2024-09-18T04:39:54.800 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.800 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.800 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:54.800 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579221+0000", 2024-09-18T04:39:54.800 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.7503999999999993e-05 2024-09-18T04:39:54.800 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.800 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.801 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:54.801 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579553+0000", 2024-09-18T04:39:54.801 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00033254200000000002 2024-09-18T04:39:54.801 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.801 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.801 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:54.801 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579817+0000", 2024-09-18T04:39:54.801 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00026317099999999998 2024-09-18T04:39:54.801 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.801 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.801 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:54.801 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.599905+0000", 2024-09-18T04:39:54.801 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020088575000000001 2024-09-18T04:39:54.801 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.801 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.802 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:54.802 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.599923+0000", 2024-09-18T04:39:54.802 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7425999999999999e-05 2024-09-18T04:39:54.802 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.802 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.802 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:54.802 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.599973+0000", 2024-09-18T04:39:54.802 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0649999999999998e-05 2024-09-18T04:39:54.802 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.802 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.802 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:54.802 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.600004+0000", 2024-09-18T04:39:54.802 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0361999999999998e-05 2024-09-18T04:39:54.802 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:54.803 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:54.803 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:54.803 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.803 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.803 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:50 2.1 e51/24 2:8c706374:ns::rbd_data.133e813ab5bc.000000000000000a:head v 51'8, mlcod=51'6)", 2024-09-18T04:39:54.803 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.577029+0000", 2024-09-18T04:39:54.803 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 72.085853313000001, 2024-09-18T04:39:54.803 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.017362855, 2024-09-18T04:39:54.803 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:54.803 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:54.803 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:54.803 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.803 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:54.803 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577029+0000", 2024-09-18T04:39:54.804 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:54.804 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.804 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.804 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:54.804 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577029+0000", 2024-09-18T04:39:54.804 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:54.804 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.804 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.804 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:54.804 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577027+0000", 2024-09-18T04:39:54.804 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:54.804 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.804 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.804 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:54.805 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578880+0000", 2024-09-18T04:39:54.805 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0018532080000000001 2024-09-18T04:39:54.805 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.805 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.805 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:54.805 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578906+0000", 2024-09-18T04:39:54.805 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5573999999999999e-05 2024-09-18T04:39:54.805 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.805 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.805 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:54.805 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578920+0000", 2024-09-18T04:39:54.805 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4483e-05 2024-09-18T04:39:54.805 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.806 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.806 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:54.806 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578983+0000", 2024-09-18T04:39:54.806 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2248000000000004e-05 2024-09-18T04:39:54.806 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.806 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.806 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:54.806 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579016+0000", 2024-09-18T04:39:54.806 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3364000000000001e-05 2024-09-18T04:39:54.806 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.806 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.806 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:54.806 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594342+0000", 2024-09-18T04:39:54.806 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01532564 2024-09-18T04:39:54.806 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.807 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.807 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:54.807 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594392+0000", 2024-09-18T04:39:54.807 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0068999999999998e-05 2024-09-18T04:39:54.807 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:54.807 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:54.807 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:54.807 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.807 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.807 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:45 2.0 e51/17 2:0fbfa3dc:ns::rbd_data.133e813ab5bc.0000000000000007:head v 51'24, mlcod=51'22)", 2024-09-18T04:39:54.807 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.581892+0000", 2024-09-18T04:39:54.807 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 72.080990641, 2024-09-18T04:39:54.807 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015646006, 2024-09-18T04:39:54.807 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:54.808 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:54.808 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:54.808 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.808 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:54.808 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581892+0000", 2024-09-18T04:39:54.808 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:54.808 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.808 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.808 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:54.808 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581892+0000", 2024-09-18T04:39:54.808 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:54.808 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.808 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.809 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:54.809 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581890+0000", 2024-09-18T04:39:54.809 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:54.809 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.809 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.809 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:54.809 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586188+0000", 2024-09-18T04:39:54.809 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0042985769999999996 2024-09-18T04:39:54.809 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.809 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.809 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:54.809 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586210+0000", 2024-09-18T04:39:54.809 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1069000000000001e-05 2024-09-18T04:39:54.809 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.810 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.810 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:54.810 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586221+0000", 2024-09-18T04:39:54.810 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1287000000000001e-05 2024-09-18T04:39:54.810 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.810 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.810 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:54.810 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586313+0000", 2024-09-18T04:39:54.810 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1935999999999995e-05 2024-09-18T04:39:54.810 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.810 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.810 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:54.810 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586368+0000", 2024-09-18T04:39:54.810 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5578000000000002e-05 2024-09-18T04:39:54.811 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.811 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.811 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:54.811 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.597473+0000", 2024-09-18T04:39:54.811 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011105139999999999 2024-09-18T04:39:54.811 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.811 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.811 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:54.811 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.597538+0000", 2024-09-18T04:39:54.811 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4133000000000004e-05 2024-09-18T04:39:54.811 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:54.811 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:54.811 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:54.811 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.811 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.812 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:53 2.4 2:39f7f174:ns::rbd_data.133e813ab5bc.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:54.812 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.584263+0000", 2024-09-18T04:39:54.812 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 72.078618766999995, 2024-09-18T04:39:54.812 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018701180000000001, 2024-09-18T04:39:54.812 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:54.812 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:54.812 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:54.812 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:54.812 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:54.812 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 53 2024-09-18T04:39:54.812 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.812 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:54.812 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.813 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:54.813 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584263+0000", 2024-09-18T04:39:54.813 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:54.813 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.813 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.813 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:54.813 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584263+0000", 2024-09-18T04:39:54.813 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:54.813 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.813 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.813 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:54.813 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584262+0000", 2024-09-18T04:39:54.813 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:54.813 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.814 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.814 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:54.814 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.591617+0000", 2024-09-18T04:39:54.814 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0073550600000000001 2024-09-18T04:39:54.814 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.814 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.814 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:54.814 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.591637+0000", 2024-09-18T04:39:54.814 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0432999999999999e-05 2024-09-18T04:39:54.814 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.814 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.814 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:54.814 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.591646+0000", 2024-09-18T04:39:54.814 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7849999999999997e-06 2024-09-18T04:39:54.815 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.815 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.815 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:54.815 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593478+0000", 2024-09-18T04:39:54.815 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001831495 2024-09-18T04:39:54.815 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.815 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.815 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:54.815 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593661+0000", 2024-09-18T04:39:54.815 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018364999999999999 2024-09-18T04:39:54.815 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.815 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.815 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:54.815 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593834+0000", 2024-09-18T04:39:54.816 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000173065 2024-09-18T04:39:54.816 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.816 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.816 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:54.816 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.600670+0000", 2024-09-18T04:39:54.816 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0068354970000000003 2024-09-18T04:39:54.816 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.816 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.816 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:54.816 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602851+0000", 2024-09-18T04:39:54.816 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002181653 2024-09-18T04:39:54.816 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.816 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.816 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:54.817 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602930+0000", 2024-09-18T04:39:54.817 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8045000000000002e-05 2024-09-18T04:39:54.817 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.817 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.817 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:54.817 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602965+0000", 2024-09-18T04:39:54.817 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5102999999999999e-05 2024-09-18T04:39:54.817 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:54.817 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:54.817 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:54.817 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.817 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.817 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:56 2.6 2:72453c3e:ns::rbd_data.133e813ab5bc.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:54.817 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.595988+0000", 2024-09-18T04:39:54.818 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 72.066894364999996, 2024-09-18T04:39:54.818 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.030296172, 2024-09-18T04:39:54.818 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:54.818 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:54.818 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:54.818 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:54.818 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:54.818 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 56 2024-09-18T04:39:54.818 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.818 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:54.818 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.818 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:54.818 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.595988+0000", 2024-09-18T04:39:54.818 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:54.818 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.819 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.819 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:54.819 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.595988+0000", 2024-09-18T04:39:54.819 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:54.819 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.819 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.819 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:54.819 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.595986+0000", 2024-09-18T04:39:54.819 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:54.819 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.819 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.819 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:54.819 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605262+0000", 2024-09-18T04:39:54.819 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0092767240000000001 2024-09-18T04:39:54.819 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.820 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.820 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:54.820 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605282+0000", 2024-09-18T04:39:54.820 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9307e-05 2024-09-18T04:39:54.820 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.820 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.820 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:54.820 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605292+0000", 2024-09-18T04:39:54.820 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9510000000000001e-06 2024-09-18T04:39:54.820 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.820 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.820 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:54.820 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605369+0000", 2024-09-18T04:39:54.820 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7846e-05 2024-09-18T04:39:54.821 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.821 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.821 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:54.821 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605668+0000", 2024-09-18T04:39:54.821 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00029896099999999999 2024-09-18T04:39:54.821 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.821 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.821 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:54.821 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605906+0000", 2024-09-18T04:39:54.821 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000237758 2024-09-18T04:39:54.821 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.821 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.821 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:54.821 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.613532+0000", 2024-09-18T04:39:54.821 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0076260590000000001 2024-09-18T04:39:54.822 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.822 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.822 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:54.822 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.626116+0000", 2024-09-18T04:39:54.822 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012584092999999999 2024-09-18T04:39:54.822 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.822 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.822 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:54.822 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.626216+0000", 2024-09-18T04:39:54.822 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9630999999999995e-05 2024-09-18T04:39:54.822 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:54.822 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:54.822 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:54.822 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.626284+0000", 2024-09-18T04:39:54.823 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8034999999999995e-05 2024-09-18T04:39:54.823 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:54.823 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:54.823 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:54.823 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:54.823 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:54.823 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:54.825 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:39:54.825 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-09-18T04:39:54.826 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:54.816+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:54.926 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:54.919+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:54.943 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:54.943 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:54.943 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:39:54.943 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:54.954 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:39:55.026 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:55.020+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:55.116 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:55.116 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:55.116 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:39:55.116 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:39:55.116 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:55.126 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:39:55.128 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:55.123+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:39:55.228 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:55.223+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:55.231 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:55.231 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:39:55.231 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:39:55.231 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:39:55.231 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.231 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:39:55.231 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:39:55.231 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 147.43887192299999, 2024-09-18T04:39:55.232 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017399218000000001, 2024-09-18T04:39:55.232 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:55.232 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:55.232 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:55.232 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:39:55.232 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:39:55.232 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 114 2024-09-18T04:39:55.232 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.232 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:55.232 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.232 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:55.232 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:39:55.232 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:55.232 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.233 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.233 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:55.233 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:39:55.233 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:55.233 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.233 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.233 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:55.233 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786900+0000", 2024-09-18T04:39:55.233 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:55.233 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.233 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.233 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:55.233 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786908+0000", 2024-09-18T04:39:55.234 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8199999999999997e-06 2024-09-18T04:39:55.234 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.234 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.234 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:55.234 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786925+0000", 2024-09-18T04:39:55.234 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6849000000000001e-05 2024-09-18T04:39:55.234 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.234 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.234 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:55.234 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786934+0000", 2024-09-18T04:39:55.234 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6299999999999993e-06 2024-09-18T04:39:55.234 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.234 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.235 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:55.235 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800926+0000", 2024-09-18T04:39:55.235 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013991632 2024-09-18T04:39:55.235 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.235 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.235 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:55.235 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801060+0000", 2024-09-18T04:39:55.235 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013378199999999999 2024-09-18T04:39:55.235 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.235 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.235 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:55.235 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801166+0000", 2024-09-18T04:39:55.235 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010607099999999999 2024-09-18T04:39:55.235 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.235 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.236 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:55.236 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803026+0000", 2024-09-18T04:39:55.236 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001860469 2024-09-18T04:39:55.236 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.236 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.236 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:55.236 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804245+0000", 2024-09-18T04:39:55.236 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0012185900000000001 2024-09-18T04:39:55.236 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.236 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.237 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:55.237 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804278+0000", 2024-09-18T04:39:55.237 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3272000000000003e-05 2024-09-18T04:39:55.237 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.237 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.237 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:55.237 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804300+0000", 2024-09-18T04:39:55.237 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2266e-05 2024-09-18T04:39:55.237 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:55.237 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:55.237 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:55.237 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.238 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.238 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:39:55.238 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:39:55.238 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 147.437959025, 2024-09-18T04:39:55.238 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017521662, 2024-09-18T04:39:55.238 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:55.238 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:55.238 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:55.238 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:39:55.238 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:39:55.238 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 116 2024-09-18T04:39:55.238 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.238 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:55.239 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.239 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:55.239 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:39:55.239 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:55.239 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.239 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.239 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:55.239 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:39:55.239 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:55.239 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.239 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.239 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:55.239 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787813+0000", 2024-09-18T04:39:55.240 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:55.240 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.240 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.240 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:55.240 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787824+0000", 2024-09-18T04:39:55.240 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1157e-05 2024-09-18T04:39:55.240 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.240 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.240 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:55.240 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787840+0000", 2024-09-18T04:39:55.240 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6269e-05 2024-09-18T04:39:55.240 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.240 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.240 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:55.241 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787853+0000", 2024-09-18T04:39:55.241 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3156000000000001e-05 2024-09-18T04:39:55.241 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.241 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.241 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:55.241 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803110+0000", 2024-09-18T04:39:55.241 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015256736 2024-09-18T04:39:55.241 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.241 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.241 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:55.241 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803197+0000", 2024-09-18T04:39:55.241 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7195999999999997e-05 2024-09-18T04:39:55.241 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.241 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.242 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:55.242 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803308+0000", 2024-09-18T04:39:55.242 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111005 2024-09-18T04:39:55.242 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.242 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.242 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:55.242 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804316+0000", 2024-09-18T04:39:55.242 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010073949999999999 2024-09-18T04:39:55.242 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.242 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.242 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:55.242 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805232+0000", 2024-09-18T04:39:55.242 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091636300000000001 2024-09-18T04:39:55.242 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.242 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.243 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:55.243 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805277+0000", 2024-09-18T04:39:55.243 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4864000000000003e-05 2024-09-18T04:39:55.243 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.243 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.243 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:55.243 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805336+0000", 2024-09-18T04:39:55.243 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8968000000000002e-05 2024-09-18T04:39:55.243 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:55.243 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:55.243 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:55.243 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.243 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.243 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:39:55.244 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:39:55.244 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 147.43790697, 2024-09-18T04:39:55.244 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017983453, 2024-09-18T04:39:55.244 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:55.244 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:55.244 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:55.244 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:39:55.244 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:39:55.244 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 117 2024-09-18T04:39:55.244 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.244 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:55.244 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.244 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:55.244 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:39:55.244 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:55.245 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.245 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.245 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:55.245 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:39:55.245 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:55.245 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.245 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.245 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:55.245 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787865+0000", 2024-09-18T04:39:55.245 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:55.245 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.245 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.245 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:55.246 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787889+0000", 2024-09-18T04:39:55.246 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4244999999999999e-05 2024-09-18T04:39:55.246 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.246 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.246 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:55.246 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787905+0000", 2024-09-18T04:39:55.246 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5739999999999998e-05 2024-09-18T04:39:55.246 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.246 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.246 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:55.246 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787918+0000", 2024-09-18T04:39:55.246 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.252e-05 2024-09-18T04:39:55.246 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.246 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.247 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:55.247 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803693+0000", 2024-09-18T04:39:55.247 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015774872999999998 2024-09-18T04:39:55.247 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.247 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.247 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:55.247 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803775+0000", 2024-09-18T04:39:55.247 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2761000000000003e-05 2024-09-18T04:39:55.247 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.247 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.247 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:55.247 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803887+0000", 2024-09-18T04:39:55.247 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111708 2024-09-18T04:39:55.247 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.248 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.248 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:55.248 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804823+0000", 2024-09-18T04:39:55.248 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00093625100000000003 2024-09-18T04:39:55.248 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.248 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.248 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:55.248 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805804+0000", 2024-09-18T04:39:55.248 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00098114699999999992 2024-09-18T04:39:55.248 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.248 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.248 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:55.248 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805831+0000", 2024-09-18T04:39:55.248 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6401999999999999e-05 2024-09-18T04:39:55.248 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.249 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.249 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:55.249 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805850+0000", 2024-09-18T04:39:55.249 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8862e-05 2024-09-18T04:39:55.249 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:55.249 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:55.249 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:55.249 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.249 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.249 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:36 2.7 2:fe1b02ad:::rbd_data.125fc09df6d1.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:55.249 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:39:55.249 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 72.960050859999996, 2024-09-18T04:39:55.249 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.040387804999999999, 2024-09-18T04:39:55.250 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:55.250 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:55.250 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:55.250 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:55.250 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:55.250 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 36 2024-09-18T04:39:55.250 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.250 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:55.250 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.250 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:55.250 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:39:55.250 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:55.250 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.250 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.250 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:55.250 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:39:55.251 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:55.251 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.251 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.251 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:55.251 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265720+0000", 2024-09-18T04:39:55.251 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:55.251 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.251 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.251 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:55.251 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272590+0000", 2024-09-18T04:39:55.251 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.006870508 2024-09-18T04:39:55.251 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.251 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.251 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:55.251 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272623+0000", 2024-09-18T04:39:55.252 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2716000000000003e-05 2024-09-18T04:39:55.252 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.252 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.252 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:55.252 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272637+0000", 2024-09-18T04:39:55.252 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4175999999999999e-05 2024-09-18T04:39:55.252 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.252 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.252 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:55.252 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272779+0000", 2024-09-18T04:39:55.252 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000141392 2024-09-18T04:39:55.252 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.252 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.253 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:55.253 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273031+0000", 2024-09-18T04:39:55.253 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00025249099999999998 2024-09-18T04:39:55.253 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.253 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.253 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:39:55.253 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273219+0000", 2024-09-18T04:39:55.253 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000187531 2024-09-18T04:39:55.253 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.253 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.253 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:55.253 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.302806+0000", 2024-09-18T04:39:55.253 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.029587553999999999 2024-09-18T04:39:55.253 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.254 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.254 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:55.254 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306004+0000", 2024-09-18T04:39:55.254 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0031982629999999998 2024-09-18T04:39:55.254 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.254 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.254 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:55.254 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306071+0000", 2024-09-18T04:39:55.254 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6829999999999995e-05 2024-09-18T04:39:55.254 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.254 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.254 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:55.254 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306110+0000", 2024-09-18T04:39:55.254 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8874000000000001e-05 2024-09-18T04:39:55.254 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:55.255 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:55.255 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:55.255 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.255 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.255 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:38 2.0 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:55.255 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:39:55.255 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 72.953097322999994, 2024-09-18T04:39:55.255 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.033754348000000003, 2024-09-18T04:39:55.255 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:55.255 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:55.255 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:55.255 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:55.255 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:55.255 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 38 2024-09-18T04:39:55.256 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.256 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:55.256 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.256 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:55.256 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:39:55.256 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:55.256 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.256 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.256 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:55.256 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:39:55.256 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:55.256 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.256 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.256 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:55.256 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272673+0000", 2024-09-18T04:39:55.257 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:55.257 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.257 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.257 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:55.257 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277166+0000", 2024-09-18T04:39:55.257 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044922249999999999 2024-09-18T04:39:55.257 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.257 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.257 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:55.257 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277189+0000", 2024-09-18T04:39:55.257 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3283000000000001e-05 2024-09-18T04:39:55.257 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.257 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.257 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:55.258 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277205+0000", 2024-09-18T04:39:55.258 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6305000000000001e-05 2024-09-18T04:39:55.258 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.258 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.258 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:55.258 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277290+0000", 2024-09-18T04:39:55.258 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5135000000000004e-05 2024-09-18T04:39:55.258 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.258 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.258 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:55.258 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277531+0000", 2024-09-18T04:39:55.258 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00024014699999999999 2024-09-18T04:39:55.258 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.259 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.259 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:55.259 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277793+0000", 2024-09-18T04:39:55.259 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00026224499999999998 2024-09-18T04:39:55.259 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.259 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.259 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:55.259 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303990+0000", 2024-09-18T04:39:55.259 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.026197295999999998 2024-09-18T04:39:55.259 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.259 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.259 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:55.259 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306340+0000", 2024-09-18T04:39:55.259 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.002349524 2024-09-18T04:39:55.260 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.260 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.260 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:55.260 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306394+0000", 2024-09-18T04:39:55.260 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3934999999999998e-05 2024-09-18T04:39:55.260 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.260 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.260 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:55.260 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306430+0000", 2024-09-18T04:39:55.260 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6690000000000003e-05 2024-09-18T04:39:55.260 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:55.260 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:55.260 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:55.260 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.261 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.261 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:39 2.7 2:e9c909c5:::rbd_data.125fc09df6d1.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:55.261 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:39:55.261 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 72.948537717999997, 2024-09-18T04:39:55.261 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.029110673, 2024-09-18T04:39:55.261 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:55.261 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:55.261 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:55.261 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:55.261 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:55.261 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 39 2024-09-18T04:39:55.261 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.261 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:55.262 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.262 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:55.262 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:39:55.262 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:55.262 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.262 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.262 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:55.262 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:39:55.262 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:55.262 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.262 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.262 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:55.262 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277234+0000", 2024-09-18T04:39:55.262 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:55.263 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.263 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.263 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:55.263 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281351+0000", 2024-09-18T04:39:55.263 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0041170920000000001 2024-09-18T04:39:55.263 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.263 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.263 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:55.263 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281375+0000", 2024-09-18T04:39:55.263 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.446e-05 2024-09-18T04:39:55.263 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.263 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.263 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:55.263 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281390+0000", 2024-09-18T04:39:55.263 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4309e-05 2024-09-18T04:39:55.264 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.264 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.264 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:55.264 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.282306+0000", 2024-09-18T04:39:55.264 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091662000000000004 2024-09-18T04:39:55.264 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.264 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.264 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:55.264 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.282613+0000", 2024-09-18T04:39:55.264 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00030663799999999997 2024-09-18T04:39:55.264 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.264 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.264 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:39:55.264 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283001+0000", 2024-09-18T04:39:55.265 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00038801599999999998 2024-09-18T04:39:55.265 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.265 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.265 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:55.265 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306239+0000", 2024-09-18T04:39:55.265 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023238466999999999 2024-09-18T04:39:55.265 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.265 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.265 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:55.265 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306259+0000", 2024-09-18T04:39:55.265 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9837999999999999e-05 2024-09-18T04:39:55.265 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.265 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.265 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:55.266 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306299+0000", 2024-09-18T04:39:55.266 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0194000000000002e-05 2024-09-18T04:39:55.266 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.266 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.266 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:55.266 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306346+0000", 2024-09-18T04:39:55.266 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6845000000000003e-05 2024-09-18T04:39:55.266 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:55.266 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:55.266 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:55.266 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.266 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.266 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:46 2.6 e51/17 2:7ba25245:::rbd_data.125fc09df6d1.0000000000000006:head v 51'2, mlcod=0'0)", 2024-09-18T04:39:55.266 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:39:55.266 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 72.912616585999999, 2024-09-18T04:39:55.267 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019843098, 2024-09-18T04:39:55.267 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:55.267 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:55.267 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:55.267 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.267 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:55.267 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:39:55.267 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:55.267 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.267 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.267 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:55.267 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:39:55.267 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:55.267 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.268 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.268 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:55.268 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313156+0000", 2024-09-18T04:39:55.268 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:55.268 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.268 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.268 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:55.268 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320144+0000", 2024-09-18T04:39:55.268 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.006988345 2024-09-18T04:39:55.268 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.268 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.268 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:55.268 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320175+0000", 2024-09-18T04:39:55.268 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.0861999999999997e-05 2024-09-18T04:39:55.269 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.269 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.269 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:55.269 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320188+0000", 2024-09-18T04:39:55.269 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2714999999999999e-05 2024-09-18T04:39:55.269 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.269 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.269 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:55.269 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320267+0000", 2024-09-18T04:39:55.269 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9586999999999995e-05 2024-09-18T04:39:55.269 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.269 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.269 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:55.269 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320311+0000", 2024-09-18T04:39:55.269 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3529000000000003e-05 2024-09-18T04:39:55.270 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.270 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.270 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:55.270 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.332949+0000", 2024-09-18T04:39:55.270 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012638267999999999 2024-09-18T04:39:55.270 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.270 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.270 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:55.270 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.333000+0000", 2024-09-18T04:39:55.270 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0736999999999999e-05 2024-09-18T04:39:55.270 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:55.270 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:55.270 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:55.270 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.271 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.271 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:57 2.0 2:0cdd070e:::rbd_data.125fc09df6d1.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:55.271 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:39:55.271 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 72.876759011000004, 2024-09-18T04:39:55.271 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.033329974999999998, 2024-09-18T04:39:55.271 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:55.271 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:55.271 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:55.271 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:55.271 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:55.271 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 57 2024-09-18T04:39:55.271 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.271 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:55.271 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.271 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:55.272 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:39:55.272 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:55.272 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.272 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.272 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:55.272 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:39:55.272 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:55.272 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.272 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.272 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:55.272 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349013+0000", 2024-09-18T04:39:55.272 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:55.272 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.272 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.273 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:55.273 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356138+0000", 2024-09-18T04:39:55.273 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071258010000000002 2024-09-18T04:39:55.273 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.273 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.273 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:55.273 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356168+0000", 2024-09-18T04:39:55.273 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9679000000000002e-05 2024-09-18T04:39:55.273 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.273 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.273 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:55.273 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356181+0000", 2024-09-18T04:39:55.273 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3164e-05 2024-09-18T04:39:55.273 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.274 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.274 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:55.274 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356254+0000", 2024-09-18T04:39:55.274 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3182999999999994e-05 2024-09-18T04:39:55.274 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.274 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.274 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:55.274 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356453+0000", 2024-09-18T04:39:55.274 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019876599999999999 2024-09-18T04:39:55.274 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.274 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.274 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:55.274 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356627+0000", 2024-09-18T04:39:55.274 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017340199999999999 2024-09-18T04:39:55.274 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.275 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.275 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:55.275 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.369200+0000", 2024-09-18T04:39:55.275 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012573302 2024-09-18T04:39:55.275 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.275 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.275 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:55.275 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382262+0000", 2024-09-18T04:39:55.275 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013062549 2024-09-18T04:39:55.275 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.275 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.275 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:55.275 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382310+0000", 2024-09-18T04:39:55.275 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7366e-05 2024-09-18T04:39:55.276 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.276 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.276 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:55.276 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382344+0000", 2024-09-18T04:39:55.276 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4319999999999997e-05 2024-09-18T04:39:55.276 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:55.276 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:55.276 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:55.276 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.276 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.276 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:59 2.0 2:19c52cde:::rbd_data.125fc09df6d1.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:55.276 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:39:55.276 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 72.869567845999995, 2024-09-18T04:39:55.276 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.027137247999999999, 2024-09-18T04:39:55.277 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:55.277 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:55.277 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:55.277 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:55.277 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:55.277 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 59 2024-09-18T04:39:55.277 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.277 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:55.277 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.277 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:55.277 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:39:55.277 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:55.277 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.277 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.277 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:55.278 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:39:55.278 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:55.278 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.278 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.278 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:55.278 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356204+0000", 2024-09-18T04:39:55.278 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:55.278 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.278 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.278 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:55.278 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362661+0000", 2024-09-18T04:39:55.278 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0064565129999999997 2024-09-18T04:39:55.278 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.278 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.279 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:55.279 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362684+0000", 2024-09-18T04:39:55.279 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3688000000000001e-05 2024-09-18T04:39:55.279 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.279 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.279 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:55.279 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362699+0000", 2024-09-18T04:39:55.279 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4798e-05 2024-09-18T04:39:55.279 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.279 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.279 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:55.279 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363016+0000", 2024-09-18T04:39:55.279 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031737300000000003 2024-09-18T04:39:55.279 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.280 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.280 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:55.280 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363201+0000", 2024-09-18T04:39:55.280 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000184131 2024-09-18T04:39:55.280 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.280 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.280 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:55.280 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363343+0000", 2024-09-18T04:39:55.280 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014236699999999999 2024-09-18T04:39:55.280 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.280 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.280 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:55.280 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.372189+0000", 2024-09-18T04:39:55.280 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0088464819999999993 2024-09-18T04:39:55.281 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.281 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.281 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:55.281 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383269+0000", 2024-09-18T04:39:55.281 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011079594 2024-09-18T04:39:55.281 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.281 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.281 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:55.281 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383314+0000", 2024-09-18T04:39:55.281 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4932000000000003e-05 2024-09-18T04:39:55.281 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.281 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.281 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:55.281 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383343+0000", 2024-09-18T04:39:55.282 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8600999999999999e-05 2024-09-18T04:39:55.282 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:55.282 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:55.282 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:55.282 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.282 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.282 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:40 2.6 e51/17 2:784d6cd1:ns::rbd_data.133e813ab5bc.0000000000000005:head v 51'7, mlcod=51'5)", 2024-09-18T04:39:55.282 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:39:55.282 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 72.673318112000004, 2024-09-18T04:39:55.282 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020497496, 2024-09-18T04:39:55.282 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:55.282 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:55.283 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:55.283 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.283 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:55.283 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:39:55.283 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:55.283 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.283 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.283 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:55.283 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:39:55.283 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:55.283 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.283 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.283 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:55.283 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552454+0000", 2024-09-18T04:39:55.283 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:55.284 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.284 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.284 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:55.284 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556448+0000", 2024-09-18T04:39:55.284 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0039940990000000001 2024-09-18T04:39:55.284 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.284 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.284 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:55.284 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556481+0000", 2024-09-18T04:39:55.284 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3778000000000001e-05 2024-09-18T04:39:55.284 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.284 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.284 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:55.285 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556493+0000", 2024-09-18T04:39:55.285 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1622e-05 2024-09-18T04:39:55.285 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.285 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.285 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:55.285 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556556+0000", 2024-09-18T04:39:55.285 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2582999999999994e-05 2024-09-18T04:39:55.285 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.285 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.286 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:55.287 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556592+0000", 2024-09-18T04:39:55.287 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5880999999999997e-05 2024-09-18T04:39:55.287 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.287 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.287 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:55.287 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.572786+0000", 2024-09-18T04:39:55.287 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016194571000000001 2024-09-18T04:39:55.287 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.287 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.287 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:55.287 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.572953+0000", 2024-09-18T04:39:55.287 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000166436 2024-09-18T04:39:55.287 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:55.287 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:55.288 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:55.288 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.288 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.288 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:45 2.0 2:0fbfa3dc:ns::rbd_data.133e813ab5bc.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:55.288 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:39:55.288 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 72.662103813000002, 2024-09-18T04:39:55.288 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.036558168000000002, 2024-09-18T04:39:55.288 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:55.288 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:55.288 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:55.288 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:55.288 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:55.288 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 45 2024-09-18T04:39:55.288 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.289 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:55.289 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.289 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:55.289 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:39:55.289 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:55.289 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.289 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.289 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:55.289 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:39:55.289 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:55.289 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.289 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.289 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:55.289 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:39:55.290 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:55.290 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.290 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.290 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:55.290 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578594+0000", 2024-09-18T04:39:55.290 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014927147999999999 2024-09-18T04:39:55.290 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.290 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.290 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:55.290 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578612+0000", 2024-09-18T04:39:55.290 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7623999999999999e-05 2024-09-18T04:39:55.290 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.290 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.291 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:55.291 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578620+0000", 2024-09-18T04:39:55.291 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9070000000000007e-06 2024-09-18T04:39:55.291 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.291 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.291 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:55.291 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578702+0000", 2024-09-18T04:39:55.291 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2349999999999996e-05 2024-09-18T04:39:55.291 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.291 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.291 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:55.291 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.579028+0000", 2024-09-18T04:39:55.291 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032622899999999998 2024-09-18T04:39:55.292 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.292 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.292 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:55.292 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.579343+0000", 2024-09-18T04:39:55.292 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031448899999999999 2024-09-18T04:39:55.292 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.292 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.292 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:55.292 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.590078+0000", 2024-09-18T04:39:55.292 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010734946 2024-09-18T04:39:55.292 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.292 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.292 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:55.293 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600089+0000", 2024-09-18T04:39:55.293 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010011195000000001 2024-09-18T04:39:55.293 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.293 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.293 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:55.293 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600177+0000", 2024-09-18T04:39:55.293 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8425999999999997e-05 2024-09-18T04:39:55.293 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.293 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.293 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:55.293 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600228+0000", 2024-09-18T04:39:55.293 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0216999999999997e-05 2024-09-18T04:39:55.293 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:55.293 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:55.293 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:55.294 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.294 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.294 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:54 2.0 2:04f17ef6:ns::rbd_data.133e813ab5bc.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:55.294 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:39:55.294 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 72.640531976000005, 2024-09-18T04:39:55.294 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.024612999, 2024-09-18T04:39:55.294 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:55.294 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:55.294 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:55.294 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:55.294 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:55.294 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 54 2024-09-18T04:39:55.294 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.295 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:55.295 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.295 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:55.295 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:39:55.295 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:55.295 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.295 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.295 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:55.295 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:39:55.295 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:55.295 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.295 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.295 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:55.295 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585239+0000", 2024-09-18T04:39:55.297 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:55.297 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.297 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.297 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:55.297 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593702+0000", 2024-09-18T04:39:55.297 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0084632990000000005 2024-09-18T04:39:55.297 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.297 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.297 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:55.297 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593719+0000", 2024-09-18T04:39:55.297 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.664e-05 2024-09-18T04:39:55.297 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.297 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.298 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:55.298 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593727+0000", 2024-09-18T04:39:55.298 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6839999999999996e-06 2024-09-18T04:39:55.298 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.298 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.298 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:55.298 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593767+0000", 2024-09-18T04:39:55.298 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0330999999999997e-05 2024-09-18T04:39:55.298 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.298 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.298 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:55.298 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593905+0000", 2024-09-18T04:39:55.298 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013754800000000001 2024-09-18T04:39:55.298 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.298 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.298 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:55.299 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.594013+0000", 2024-09-18T04:39:55.299 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000108805 2024-09-18T04:39:55.299 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.299 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.299 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:55.299 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604678+0000", 2024-09-18T04:39:55.299 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010664391 2024-09-18T04:39:55.299 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.299 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.299 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:55.299 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609662+0000", 2024-09-18T04:39:55.299 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0049846100000000004 2024-09-18T04:39:55.299 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.299 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.299 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:55.300 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609788+0000", 2024-09-18T04:39:55.300 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012518800000000001 2024-09-18T04:39:55.300 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.300 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.300 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:55.300 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609854+0000", 2024-09-18T04:39:55.300 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6556000000000004e-05 2024-09-18T04:39:55.300 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:55.300 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:55.300 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:55.300 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.300 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.300 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:58 2.7 2:fc7c6cd3:ns::rbd_data.133e813ab5bc.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:55.300 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:39:55.300 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 72.627689547000003, 2024-09-18T04:39:55.301 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.027690566999999999, 2024-09-18T04:39:55.301 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:55.301 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:55.301 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:55.301 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:55.301 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:55.301 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 58 2024-09-18T04:39:55.301 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.301 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:55.301 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.301 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:55.301 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:39:55.301 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:55.301 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.302 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.302 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:55.302 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:39:55.302 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:55.302 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.302 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.302 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:55.302 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598082+0000", 2024-09-18T04:39:55.302 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:55.302 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.302 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.302 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:55.302 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604832+0000", 2024-09-18T04:39:55.302 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0067504510000000002 2024-09-18T04:39:55.303 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.303 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.303 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:55.303 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604852+0000", 2024-09-18T04:39:55.303 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0228999999999999e-05 2024-09-18T04:39:55.303 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.303 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.303 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:55.303 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604870+0000", 2024-09-18T04:39:55.303 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7952e-05 2024-09-18T04:39:55.303 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.303 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.303 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:55.303 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604936+0000", 2024-09-18T04:39:55.303 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5431000000000004e-05 2024-09-18T04:39:55.304 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.304 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.304 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:55.304 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605085+0000", 2024-09-18T04:39:55.304 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014910300000000001 2024-09-18T04:39:55.304 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.304 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.304 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:39:55.304 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605205+0000", 2024-09-18T04:39:55.304 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000119887 2024-09-18T04:39:55.305 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.305 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.305 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:55.305 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625586+0000", 2024-09-18T04:39:55.305 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020381399000000001 2024-09-18T04:39:55.305 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.305 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.305 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:55.305 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625628+0000", 2024-09-18T04:39:55.305 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1873999999999999e-05 2024-09-18T04:39:55.305 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.305 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.305 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:55.306 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625722+0000", 2024-09-18T04:39:55.306 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3787000000000002e-05 2024-09-18T04:39:55.306 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.306 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.306 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:55.306 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625774+0000", 2024-09-18T04:39:55.306 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2503999999999997e-05 2024-09-18T04:39:55.306 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:55.306 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:55.306 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:55.306 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.306 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.306 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:56 2.6 e51/17 2:72453c3e:ns::rbd_data.133e813ab5bc.000000000000000d:head v 51'10, mlcod=51'8)", 2024-09-18T04:39:55.306 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:39:55.307 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 72.620492439000003, 2024-09-18T04:39:55.307 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019766552999999999, 2024-09-18T04:39:55.307 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:55.307 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:55.307 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:55.307 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.307 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:55.307 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:39:55.307 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:55.307 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.307 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.307 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:55.307 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:39:55.307 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:55.308 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.308 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.308 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:55.308 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605279+0000", 2024-09-18T04:39:55.308 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:55.308 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.308 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.308 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:55.308 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609417+0000", 2024-09-18T04:39:55.308 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0041378059999999999 2024-09-18T04:39:55.308 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.308 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.308 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:55.308 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609443+0000", 2024-09-18T04:39:55.309 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6098e-05 2024-09-18T04:39:55.309 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.309 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.309 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:55.309 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609458+0000", 2024-09-18T04:39:55.309 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4671000000000001e-05 2024-09-18T04:39:55.309 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.309 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.309 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:55.309 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609553+0000", 2024-09-18T04:39:55.309 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5265000000000004e-05 2024-09-18T04:39:55.309 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.309 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.309 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:55.310 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609583+0000", 2024-09-18T04:39:55.310 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9839e-05 2024-09-18T04:39:55.310 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.310 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.310 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:55.310 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625004+0000", 2024-09-18T04:39:55.310 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015420639999999999 2024-09-18T04:39:55.310 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.310 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.310 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:55.310 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625047+0000", 2024-09-18T04:39:55.310 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3665999999999998e-05 2024-09-18T04:39:55.310 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:55.310 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:55.310 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:55.311 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.311 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.311 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:99 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:55.311 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.582718+0000", 2024-09-18T04:39:55.311 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 58.643055529999998, 2024-09-18T04:39:55.311 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019362575, 2024-09-18T04:39:55.311 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:55.311 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:55.311 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:55.311 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:55.311 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:55.311 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 99 2024-09-18T04:39:55.311 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.311 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:55.312 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.312 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:55.312 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582718+0000", 2024-09-18T04:39:55.312 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:55.312 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.312 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.312 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:55.312 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582718+0000", 2024-09-18T04:39:55.312 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:55.312 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.312 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.312 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:55.312 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582717+0000", 2024-09-18T04:39:55.312 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:55.313 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.313 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.313 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:55.313 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582720+0000", 2024-09-18T04:39:55.313 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2239999999999998e-06 2024-09-18T04:39:55.313 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.313 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.313 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:55.313 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582723+0000", 2024-09-18T04:39:55.313 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.258e-06 2024-09-18T04:39:55.313 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.313 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.313 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:55.313 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582726+0000", 2024-09-18T04:39:55.314 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8870000000000001e-06 2024-09-18T04:39:55.314 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.314 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.314 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:55.314 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.592018+0000", 2024-09-18T04:39:55.314 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0092926420000000003 2024-09-18T04:39:55.314 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.314 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.314 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:55.314 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.592214+0000", 2024-09-18T04:39:55.314 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000195267 2024-09-18T04:39:55.314 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.314 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.314 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:55.314 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.592913+0000", 2024-09-18T04:39:55.314 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00069924600000000005 2024-09-18T04:39:55.315 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.315 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.315 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:55.315 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597493+0000", 2024-09-18T04:39:55.315 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045797629999999997 2024-09-18T04:39:55.315 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.315 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.315 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:55.315 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.602009+0000", 2024-09-18T04:39:55.315 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045159559999999998 2024-09-18T04:39:55.315 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.315 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.315 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:55.315 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.602042+0000", 2024-09-18T04:39:55.316 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3084999999999999e-05 2024-09-18T04:39:55.316 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.316 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.316 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:55.316 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.602080+0000", 2024-09-18T04:39:55.316 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8600000000000003e-05 2024-09-18T04:39:55.316 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:55.316 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:55.316 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:55.316 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.316 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.316 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:104 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:55.316 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.583165+0000", 2024-09-18T04:39:55.316 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 58.642608232000001, 2024-09-18T04:39:55.316 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023679603, 2024-09-18T04:39:55.317 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:55.317 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:55.317 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:55.317 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:55.317 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:55.317 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 104 2024-09-18T04:39:55.317 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.317 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:55.317 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.317 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:55.317 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583165+0000", 2024-09-18T04:39:55.317 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:55.317 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.317 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.318 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:55.318 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583165+0000", 2024-09-18T04:39:55.318 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:55.318 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.318 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.318 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:55.318 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583164+0000", 2024-09-18T04:39:55.318 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:55.318 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.318 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.318 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:55.318 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583167+0000", 2024-09-18T04:39:55.318 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9210000000000001e-06 2024-09-18T04:39:55.318 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.318 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.319 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:55.319 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583171+0000", 2024-09-18T04:39:55.319 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9759999999999997e-06 2024-09-18T04:39:55.319 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.319 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.319 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:55.319 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583174+0000", 2024-09-18T04:39:55.319 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9960000000000002e-06 2024-09-18T04:39:55.319 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.319 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.319 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:55.319 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.594373+0000", 2024-09-18T04:39:55.319 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011198665999999999 2024-09-18T04:39:55.319 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.319 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.320 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:55.320 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.594434+0000", 2024-09-18T04:39:55.320 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.1446999999999996e-05 2024-09-18T04:39:55.320 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.320 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.320 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:55.320 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.594692+0000", 2024-09-18T04:39:55.320 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00025756599999999998 2024-09-18T04:39:55.320 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.320 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.320 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:55.320 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599302+0000", 2024-09-18T04:39:55.320 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0046099640000000002 2024-09-18T04:39:55.320 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.321 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.321 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:55.321 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606710+0000", 2024-09-18T04:39:55.321 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0074077739999999998 2024-09-18T04:39:55.321 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.321 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.321 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:55.321 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606798+0000", 2024-09-18T04:39:55.321 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8133999999999993e-05 2024-09-18T04:39:55.321 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.321 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.321 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:55.321 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606845+0000", 2024-09-18T04:39:55.321 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6653000000000002e-05 2024-09-18T04:39:55.322 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:55.322 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:55.322 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:55.322 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.322 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.322 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:105 2.0 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:55.322 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.583193+0000", 2024-09-18T04:39:55.322 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 58.642580613, 2024-09-18T04:39:55.322 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023713649, 2024-09-18T04:39:55.322 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:55.322 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:55.322 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:55.322 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:55.322 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:55.323 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 105 2024-09-18T04:39:55.323 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.323 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:55.323 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.323 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:55.323 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583193+0000", 2024-09-18T04:39:55.323 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:55.323 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.323 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.323 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:55.323 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583193+0000", 2024-09-18T04:39:55.323 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:55.323 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.323 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.323 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:55.324 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583192+0000", 2024-09-18T04:39:55.324 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:55.324 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.324 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.324 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:55.324 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583194+0000", 2024-09-18T04:39:55.324 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.305e-06 2024-09-18T04:39:55.324 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.324 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.324 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:55.324 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583198+0000", 2024-09-18T04:39:55.324 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9369999999999997e-06 2024-09-18T04:39:55.324 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.324 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.325 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:55.325 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583202+0000", 2024-09-18T04:39:55.325 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6480000000000001e-06 2024-09-18T04:39:55.325 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.325 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.325 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:55.325 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.595304+0000", 2024-09-18T04:39:55.325 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012102063999999999 2024-09-18T04:39:55.325 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.325 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.325 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:55.325 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.595390+0000", 2024-09-18T04:39:55.325 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5493000000000003e-05 2024-09-18T04:39:55.326 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.326 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.326 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:55.326 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.595509+0000", 2024-09-18T04:39:55.326 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011966199999999999 2024-09-18T04:39:55.326 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.326 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.326 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:55.326 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599358+0000", 2024-09-18T04:39:55.326 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0038486689999999999 2024-09-18T04:39:55.326 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.326 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.326 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:55.326 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606859+0000", 2024-09-18T04:39:55.326 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0075010789999999999 2024-09-18T04:39:55.327 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.327 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.327 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:55.327 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606887+0000", 2024-09-18T04:39:55.327 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7773e-05 2024-09-18T04:39:55.327 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.327 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.327 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:55.327 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606906+0000", 2024-09-18T04:39:55.327 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9417000000000001e-05 2024-09-18T04:39:55.327 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:55.327 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:55.327 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:55.327 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.327 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.328 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:106 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:55.328 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.584425+0000", 2024-09-18T04:39:55.328 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 58.641348317000002, 2024-09-18T04:39:55.328 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.022529293999999998, 2024-09-18T04:39:55.328 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:55.328 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:55.328 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:55.328 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:55.328 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:55.328 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 106 2024-09-18T04:39:55.328 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.328 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:55.328 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.329 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:55.329 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584425+0000", 2024-09-18T04:39:55.329 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:55.329 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.329 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.329 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:55.329 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584425+0000", 2024-09-18T04:39:55.329 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:55.329 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.329 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.329 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:55.329 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584424+0000", 2024-09-18T04:39:55.329 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:55.329 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.330 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.330 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:55.330 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584430+0000", 2024-09-18T04:39:55.330 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6389999999999997e-06 2024-09-18T04:39:55.330 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.330 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.330 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:55.330 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584456+0000", 2024-09-18T04:39:55.330 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6188999999999999e-05 2024-09-18T04:39:55.330 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.330 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.330 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:55.330 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584463+0000", 2024-09-18T04:39:55.330 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7340000000000002e-06 2024-09-18T04:39:55.331 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.331 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.331 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:55.331 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597537+0000", 2024-09-18T04:39:55.331 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013074161000000001 2024-09-18T04:39:55.331 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.331 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.331 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:55.331 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597598+0000", 2024-09-18T04:39:55.331 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0930000000000001e-05 2024-09-18T04:39:55.331 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.331 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.331 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:55.331 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597922+0000", 2024-09-18T04:39:55.332 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032473000000000002 2024-09-18T04:39:55.332 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.332 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.332 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:55.332 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.601103+0000", 2024-09-18T04:39:55.332 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0031809849999999999 2024-09-18T04:39:55.332 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.332 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.332 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:55.332 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606912+0000", 2024-09-18T04:39:55.332 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058083550000000003 2024-09-18T04:39:55.332 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.332 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.332 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:55.333 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606935+0000", 2024-09-18T04:39:55.333 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2759999999999999e-05 2024-09-18T04:39:55.333 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.333 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.333 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:55.333 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606954+0000", 2024-09-18T04:39:55.333 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9602999999999998e-05 2024-09-18T04:39:55.333 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:55.333 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:55.333 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:55.333 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.333 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.333 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:109 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:55.333 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.585231+0000", 2024-09-18T04:39:55.334 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 58.640541697000003, 2024-09-18T04:39:55.334 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.021767496000000001, 2024-09-18T04:39:55.334 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:55.334 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:55.334 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:55.334 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:55.334 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:55.334 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 109 2024-09-18T04:39:55.334 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.334 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:55.334 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.334 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:55.334 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585231+0000", 2024-09-18T04:39:55.334 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:55.334 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.335 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.335 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:55.335 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585231+0000", 2024-09-18T04:39:55.335 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:55.335 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.335 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.335 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:55.335 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585230+0000", 2024-09-18T04:39:55.335 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:55.335 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.335 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.335 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:55.335 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585237+0000", 2024-09-18T04:39:55.335 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.618e-06 2024-09-18T04:39:55.336 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.336 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.336 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:55.336 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585256+0000", 2024-09-18T04:39:55.336 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9048000000000001e-05 2024-09-18T04:39:55.336 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.336 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.336 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:55.336 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585274+0000", 2024-09-18T04:39:55.336 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8153000000000001e-05 2024-09-18T04:39:55.336 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.336 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.336 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:55.337 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.598284+0000", 2024-09-18T04:39:55.337 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013009682999999999 2024-09-18T04:39:55.337 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.337 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.337 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:55.337 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.598383+0000", 2024-09-18T04:39:55.337 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.8722999999999997e-05 2024-09-18T04:39:55.337 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.337 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.337 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:55.337 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.598829+0000", 2024-09-18T04:39:55.337 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00044623999999999998 2024-09-18T04:39:55.337 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.337 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.338 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:55.338 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.601149+0000", 2024-09-18T04:39:55.338 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0023204150000000002 2024-09-18T04:39:55.338 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.338 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.338 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:55.338 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606960+0000", 2024-09-18T04:39:55.338 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058106390000000003 2024-09-18T04:39:55.338 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.338 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.338 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:55.338 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606982+0000", 2024-09-18T04:39:55.338 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2252e-05 2024-09-18T04:39:55.338 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.338 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.339 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:55.339 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606999+0000", 2024-09-18T04:39:55.339 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6871000000000002e-05 2024-09-18T04:39:55.339 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:55.339 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:55.339 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:55.339 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.339 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.339 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:110 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:55.339 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.586640+0000", 2024-09-18T04:39:55.339 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 58.639133377999997, 2024-09-18T04:39:55.339 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020400911000000001, 2024-09-18T04:39:55.339 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:55.339 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:55.339 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:55.340 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:55.340 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:55.340 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 110 2024-09-18T04:39:55.340 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.340 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:55.340 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.340 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:55.340 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586640+0000", 2024-09-18T04:39:55.340 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:55.340 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.340 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.340 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:55.340 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586640+0000", 2024-09-18T04:39:55.341 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:55.341 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.341 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.341 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:55.341 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586639+0000", 2024-09-18T04:39:55.341 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:55.341 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.341 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.341 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:55.341 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586652+0000", 2024-09-18T04:39:55.341 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2482e-05 2024-09-18T04:39:55.341 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.341 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.341 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:55.342 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586660+0000", 2024-09-18T04:39:55.342 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1659999999999994e-06 2024-09-18T04:39:55.342 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.342 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.342 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:55.342 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586666+0000", 2024-09-18T04:39:55.342 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6900000000000003e-06 2024-09-18T04:39:55.342 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.342 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.342 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:55.342 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599381+0000", 2024-09-18T04:39:55.342 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012714328 2024-09-18T04:39:55.342 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.343 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.343 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:55.343 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599436+0000", 2024-09-18T04:39:55.343 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4802999999999997e-05 2024-09-18T04:39:55.343 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.343 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.343 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:55.343 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599629+0000", 2024-09-18T04:39:55.343 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000193921 2024-09-18T04:39:55.343 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.343 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.343 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:55.343 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.601174+0000", 2024-09-18T04:39:55.343 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001544857 2024-09-18T04:39:55.344 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.344 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.344 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:55.344 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.607004+0000", 2024-09-18T04:39:55.344 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058300820000000003 2024-09-18T04:39:55.344 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.344 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.344 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:55.344 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.607024+0000", 2024-09-18T04:39:55.344 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9888e-05 2024-09-18T04:39:55.344 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.344 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.344 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:55.345 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.607041+0000", 2024-09-18T04:39:55.345 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6450999999999999e-05 2024-09-18T04:39:55.345 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:55.345 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:55.345 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:55.345 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:55.345 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:55.345 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:55.345 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:39:55.347 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:55.344+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:55.448 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:55.440+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:55.457 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:55.457 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:55.458 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:39:55.458 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:55.467 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:39:55.549 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:55.540+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:55.569 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:55.569 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:55.569 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:39:55.570 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:39:55.570 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:55.579 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:39:55.650 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:55.647+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:55.731 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:55.731 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:39:55.731 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:39:55.731 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:39:55.731 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.731 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.5431.0:1 3.3 3:d80af3cb:::rbd_id.testrbdnbd17342:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-18T04:39:55.732 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:13.542818+0000", 2024-09-18T04:39:55.732 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 42.186161095999999, 2024-09-18T04:39:55.732 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048493400000000002, 2024-09-18T04:39:55.732 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:55.732 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:39:55.732 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:55.732 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.5431", 2024-09-18T04:39:55.732 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3147136083", 2024-09-18T04:39:55.732 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 1 2024-09-18T04:39:55.732 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.732 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:55.732 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.732 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:55.733 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542818+0000", 2024-09-18T04:39:55.733 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:55.733 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.733 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.733 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:55.733 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542818+0000", 2024-09-18T04:39:55.733 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:55.733 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.733 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.733 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:55.733 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542816+0000", 2024-09-18T04:39:55.734 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:55.734 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.734 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.734 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:55.734 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542828+0000", 2024-09-18T04:39:55.734 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2699e-05 2024-09-18T04:39:55.734 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.734 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.734 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:55.735 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542852+0000", 2024-09-18T04:39:55.735 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3176e-05 2024-09-18T04:39:55.735 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.735 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.735 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:55.735 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542874+0000", 2024-09-18T04:39:55.735 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1967000000000002e-05 2024-09-18T04:39:55.735 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.735 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.735 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:55.735 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542984+0000", 2024-09-18T04:39:55.735 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011026099999999999 2024-09-18T04:39:55.735 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.736 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.736 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:55.736 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543161+0000", 2024-09-18T04:39:55.736 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017720299999999999 2024-09-18T04:39:55.736 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.736 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.736 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:55.736 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543303+0000", 2024-09-18T04:39:55.736 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014174700000000001 2024-09-18T04:39:55.736 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:55.736 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:55.736 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:55.736 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.737 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.737 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.5431.0:2 3.3 3:d80af3cb:::rbd_id.testrbdnbd17342:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-18T04:39:55.737 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:13.543564+0000", 2024-09-18T04:39:55.737 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 42.185414610000002, 2024-09-18T04:39:55.737 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010399929999999999, 2024-09-18T04:39:55.737 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:55.737 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:39:55.737 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:55.737 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.5431", 2024-09-18T04:39:55.737 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3147136083", 2024-09-18T04:39:55.737 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 2 2024-09-18T04:39:55.738 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.738 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:55.738 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.738 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:55.738 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543564+0000", 2024-09-18T04:39:55.738 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:55.738 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.738 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.738 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:55.738 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543564+0000", 2024-09-18T04:39:55.738 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:55.738 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.738 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.738 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:55.739 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543563+0000", 2024-09-18T04:39:55.739 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:55.739 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.739 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.739 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:55.739 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543571+0000", 2024-09-18T04:39:55.739 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4270000000000008e-06 2024-09-18T04:39:55.739 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.739 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.739 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:55.739 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543591+0000", 2024-09-18T04:39:55.739 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.929e-05 2024-09-18T04:39:55.740 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.740 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.740 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:55.740 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543600+0000", 2024-09-18T04:39:55.740 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.1390000000000001e-06 2024-09-18T04:39:55.740 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.740 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.740 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:55.740 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543724+0000", 2024-09-18T04:39:55.740 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012458400000000001 2024-09-18T04:39:55.740 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.741 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.741 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:55.741 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543968+0000", 2024-09-18T04:39:55.741 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00024323199999999999 2024-09-18T04:39:55.741 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:55.741 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:55.741 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:55.741 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.544604+0000", 2024-09-18T04:39:55.741 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00063675799999999996 2024-09-18T04:39:55.742 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:55.742 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:55.742 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:55.742 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:55.742 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:55.742 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:55.743 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:39:55.751 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:55.747+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:39:55.850 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:55.850 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:55.850 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:39:55.850 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:55.852 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:55.847+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:39:55.859 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:39:55.953 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:55.944+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:55.966 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:55.966 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:39:55.966 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:39:55.966 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:39:55.967 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:55.976 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:39:56.053 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:56.047+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:56.129 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:39:56.129 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:39:56.129 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:39:56.129 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:39:56.129 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.129 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-18T04:39:56.129 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:39:56.129 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 141.65345287, 2024-09-18T04:39:56.129 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0095781189999999995, 2024-09-18T04:39:56.129 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:56.130 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:56.130 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:56.130 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:39:56.130 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:39:56.130 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 1 2024-09-18T04:39:56.130 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.130 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:56.130 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.130 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:56.130 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:39:56.130 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:56.130 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.130 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.130 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:56.131 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:39:56.131 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:56.131 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.131 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.131 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:56.131 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469862+0000", 2024-09-18T04:39:56.131 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:56.131 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.131 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.131 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:56.131 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469879+0000", 2024-09-18T04:39:56.131 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7431999999999998e-05 2024-09-18T04:39:56.132 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.132 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.132 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:56.132 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469896+0000", 2024-09-18T04:39:56.132 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6742999999999999e-05 2024-09-18T04:39:56.132 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.132 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.132 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:56.132 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469915+0000", 2024-09-18T04:39:56.132 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8991000000000002e-05 2024-09-18T04:39:56.132 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.132 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.132 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:56.133 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470397+0000", 2024-09-18T04:39:56.133 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048167799999999998 2024-09-18T04:39:56.133 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.133 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.133 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-09-18T04:39:56.133 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470412+0000", 2024-09-18T04:39:56.133 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5526e-05 2024-09-18T04:39:56.133 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.133 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.133 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:56.133 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473279+0000", 2024-09-18T04:39:56.133 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0028667469999999998 2024-09-18T04:39:56.133 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.134 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.134 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:56.134 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473441+0000", 2024-09-18T04:39:56.134 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000162082 2024-09-18T04:39:56.134 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.134 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.134 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:56.134 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473573+0000", 2024-09-18T04:39:56.134 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000132032 2024-09-18T04:39:56.134 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.134 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.134 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:56.134 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474826+0000", 2024-09-18T04:39:56.135 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001252717 2024-09-18T04:39:56.135 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.135 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.135 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:56.135 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479307+0000", 2024-09-18T04:39:56.135 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044813220000000003 2024-09-18T04:39:56.135 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.135 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.135 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:56.135 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479366+0000", 2024-09-18T04:39:56.135 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9024999999999998e-05 2024-09-18T04:39:56.135 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.136 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.136 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:56.136 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479443+0000", 2024-09-18T04:39:56.136 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6339000000000002e-05 2024-09-18T04:39:56.136 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:56.136 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:56.136 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:56.136 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.136 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.137 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-18T04:39:56.137 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:39:56.137 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 139.64044444999999, 2024-09-18T04:39:56.137 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082699560000000002, 2024-09-18T04:39:56.137 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:56.137 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:56.137 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:56.137 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:39:56.137 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:39:56.137 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 7 2024-09-18T04:39:56.137 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.138 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:56.138 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.138 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:56.138 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:39:56.138 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:56.138 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.138 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.138 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:56.138 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:39:56.138 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:56.138 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.138 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.139 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:56.139 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482870+0000", 2024-09-18T04:39:56.139 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999967 2024-09-18T04:39:56.139 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.139 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.139 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:56.139 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482889+0000", 2024-09-18T04:39:56.139 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.965e-05 2024-09-18T04:39:56.139 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.139 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.139 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:56.140 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482938+0000", 2024-09-18T04:39:56.140 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8548999999999999e-05 2024-09-18T04:39:56.140 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.140 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.140 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:56.140 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482963+0000", 2024-09-18T04:39:56.140 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5075999999999998e-05 2024-09-18T04:39:56.140 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.140 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.140 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:56.140 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483121+0000", 2024-09-18T04:39:56.140 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015819400000000001 2024-09-18T04:39:56.141 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.141 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.141 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op must wait for map", 2024-09-18T04:39:56.141 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483138+0000", 2024-09-18T04:39:56.141 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7068999999999999e-05 2024-09-18T04:39:56.141 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.141 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.141 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:56.141 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488666+0000", 2024-09-18T04:39:56.141 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0055282049999999996 2024-09-18T04:39:56.141 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.141 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.141 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:56.141 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488783+0000", 2024-09-18T04:39:56.141 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000116737 2024-09-18T04:39:56.142 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.142 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.142 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:39:56.142 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488991+0000", 2024-09-18T04:39:56.142 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020781400000000001 2024-09-18T04:39:56.142 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.142 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.142 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:56.142 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490554+0000", 2024-09-18T04:39:56.142 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0015628370000000001 2024-09-18T04:39:56.142 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.142 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.142 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:56.142 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491019+0000", 2024-09-18T04:39:56.143 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000465608 2024-09-18T04:39:56.143 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.143 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.143 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:56.143 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491087+0000", 2024-09-18T04:39:56.143 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7988e-05 2024-09-18T04:39:56.143 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.143 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.143 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:56.143 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491143+0000", 2024-09-18T04:39:56.143 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5538999999999998e-05 2024-09-18T04:39:56.143 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:56.143 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:56.143 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:56.144 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.144 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.144 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:34 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:56.144 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.261758+0000", 2024-09-18T04:39:56.144 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 73.861559299000007, 2024-09-18T04:39:56.144 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.042024191000000002, 2024-09-18T04:39:56.144 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:56.144 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:56.144 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:56.144 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:56.144 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:56.144 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 34 2024-09-18T04:39:56.145 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.145 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:56.145 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.145 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:56.145 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.261758+0000", 2024-09-18T04:39:56.145 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:56.145 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.145 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.145 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:56.145 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.261758+0000", 2024-09-18T04:39:56.146 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:56.146 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.146 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.146 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:56.146 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.261756+0000", 2024-09-18T04:39:56.146 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:56.146 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.146 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.146 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:56.146 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.270785+0000", 2024-09-18T04:39:56.146 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0090292340000000006 2024-09-18T04:39:56.146 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.146 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.146 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:56.147 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.270890+0000", 2024-09-18T04:39:56.147 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000104541 2024-09-18T04:39:56.147 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.147 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.147 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:56.147 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.270991+0000", 2024-09-18T04:39:56.147 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000100778 2024-09-18T04:39:56.147 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.147 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.147 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:56.147 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.271157+0000", 2024-09-18T04:39:56.147 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016653999999999999 2024-09-18T04:39:56.147 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.147 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.148 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:56.148 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.271591+0000", 2024-09-18T04:39:56.148 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00043409500000000001 2024-09-18T04:39:56.148 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.148 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.148 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:56.148 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272017+0000", 2024-09-18T04:39:56.148 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000425769 2024-09-18T04:39:56.149 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.149 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.149 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:56.149 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.294246+0000", 2024-09-18T04:39:56.149 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.022229237999999998 2024-09-18T04:39:56.149 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.149 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.149 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:56.149 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303571+0000", 2024-09-18T04:39:56.150 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0093250239999999995 2024-09-18T04:39:56.150 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.150 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.150 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:56.150 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303705+0000", 2024-09-18T04:39:56.150 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000133831 2024-09-18T04:39:56.150 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.150 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.150 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:56.150 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303782+0000", 2024-09-18T04:39:56.150 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7156000000000005e-05 2024-09-18T04:39:56.151 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:56.151 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:56.151 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:56.151 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.151 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.151 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:36 2.7 e51/32 2:fe1b02ad:::rbd_data.125fc09df6d1.0000000000000002:head v 51'1, mlcod=0'0)", 2024-09-18T04:39:56.151 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.273441+0000", 2024-09-18T04:39:56.151 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 73.849876318, 2024-09-18T04:39:56.151 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.032400768000000003, 2024-09-18T04:39:56.151 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:56.152 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:56.152 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:56.152 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.152 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:56.152 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273441+0000", 2024-09-18T04:39:56.152 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:56.152 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.152 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.152 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:56.152 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273441+0000", 2024-09-18T04:39:56.152 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:56.153 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.153 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.153 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:56.153 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273440+0000", 2024-09-18T04:39:56.153 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:56.153 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.153 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.153 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:56.153 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278723+0000", 2024-09-18T04:39:56.153 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0052829909999999999 2024-09-18T04:39:56.153 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.153 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.154 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:56.154 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278752+0000", 2024-09-18T04:39:56.154 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9178000000000001e-05 2024-09-18T04:39:56.154 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.154 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.154 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:56.154 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278761+0000", 2024-09-18T04:39:56.154 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7900000000000005e-06 2024-09-18T04:39:56.154 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.154 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.154 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:56.154 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278864+0000", 2024-09-18T04:39:56.155 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000102699 2024-09-18T04:39:56.155 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.155 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.155 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:56.155 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278900+0000", 2024-09-18T04:39:56.155 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6001999999999999e-05 2024-09-18T04:39:56.155 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.155 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.155 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:56.155 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.305793+0000", 2024-09-18T04:39:56.155 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.026893344 2024-09-18T04:39:56.155 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.155 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.155 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:56.156 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.305842+0000", 2024-09-18T04:39:56.156 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8720000000000001e-05 2024-09-18T04:39:56.156 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:56.156 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:56.156 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:56.156 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.156 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.156 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:39 2.7 e51/32 2:e9c909c5:::rbd_data.125fc09df6d1.0000000000000005:head v 51'2, mlcod=0'0)", 2024-09-18T04:39:56.156 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.283175+0000", 2024-09-18T04:39:56.156 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 73.840142628999999, 2024-09-18T04:39:56.156 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.022864102000000001, 2024-09-18T04:39:56.156 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:56.156 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:56.156 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:56.157 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.157 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:56.157 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283175+0000", 2024-09-18T04:39:56.157 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:56.157 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.157 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.157 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:56.157 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283175+0000", 2024-09-18T04:39:56.157 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:56.157 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.157 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.157 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:56.157 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283173+0000", 2024-09-18T04:39:56.157 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:56.158 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.158 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.158 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:56.158 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.285951+0000", 2024-09-18T04:39:56.158 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0027784540000000001 2024-09-18T04:39:56.158 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.158 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.158 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:56.158 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.285970+0000", 2024-09-18T04:39:56.158 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8855000000000002e-05 2024-09-18T04:39:56.158 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.158 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.159 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:56.159 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.285984+0000", 2024-09-18T04:39:56.159 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4252e-05 2024-09-18T04:39:56.159 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.159 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.159 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:56.159 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.286041+0000", 2024-09-18T04:39:56.159 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6431999999999998e-05 2024-09-18T04:39:56.159 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.159 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.159 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:56.159 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.286086+0000", 2024-09-18T04:39:56.160 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5423000000000002e-05 2024-09-18T04:39:56.160 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.160 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.160 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:56.160 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.305998+0000", 2024-09-18T04:39:56.160 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019911697999999999 2024-09-18T04:39:56.160 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.160 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.160 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:56.160 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306039+0000", 2024-09-18T04:39:56.160 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0973999999999998e-05 2024-09-18T04:39:56.160 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:56.161 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:56.161 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:56.161 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.161 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.161 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:50 2.3 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:56.161 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.312563+0000", 2024-09-18T04:39:56.161 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 73.810754602000003, 2024-09-18T04:39:56.161 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.032636001999999997, 2024-09-18T04:39:56.161 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:56.161 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:56.161 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:56.161 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:39:56.161 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:56.162 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 50 2024-09-18T04:39:56.162 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.162 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:56.162 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.162 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:56.162 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.312563+0000", 2024-09-18T04:39:56.162 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:56.162 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.162 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.162 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:56.162 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.312563+0000", 2024-09-18T04:39:56.162 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:56.162 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.162 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.163 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:56.163 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.312561+0000", 2024-09-18T04:39:56.163 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:56.163 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.163 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.163 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:56.163 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321709+0000", 2024-09-18T04:39:56.163 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0091486650000000003 2024-09-18T04:39:56.163 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.163 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.163 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:56.163 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321739+0000", 2024-09-18T04:39:56.163 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9666e-05 2024-09-18T04:39:56.164 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.164 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.164 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:56.164 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321751+0000", 2024-09-18T04:39:56.164 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2086000000000001e-05 2024-09-18T04:39:56.164 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.164 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.164 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:56.164 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321833+0000", 2024-09-18T04:39:56.164 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1842999999999996e-05 2024-09-18T04:39:56.164 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.164 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.164 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:56.165 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321981+0000", 2024-09-18T04:39:56.165 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000147441 2024-09-18T04:39:56.165 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.165 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.165 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:56.165 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.322109+0000", 2024-09-18T04:39:56.165 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012851200000000001 2024-09-18T04:39:56.165 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.165 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.165 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:56.165 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.332829+0000", 2024-09-18T04:39:56.165 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010720225999999999 2024-09-18T04:39:56.165 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.165 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.166 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:56.166 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.345039+0000", 2024-09-18T04:39:56.166 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012209831000000001 2024-09-18T04:39:56.166 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.166 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.166 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:56.166 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.345120+0000", 2024-09-18T04:39:56.166 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1145999999999999e-05 2024-09-18T04:39:56.166 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.166 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.166 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:56.166 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.345199+0000", 2024-09-18T04:39:56.166 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8546000000000003e-05 2024-09-18T04:39:56.166 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:56.167 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:56.167 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:56.167 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.167 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.167 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:30 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:56.167 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.516122+0000", 2024-09-18T04:39:56.167 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 73.607195439999998, 2024-09-18T04:39:56.167 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0091865030000000004, 2024-09-18T04:39:56.167 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:56.167 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:56.167 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:56.167 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:56.167 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:56.167 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 30 2024-09-18T04:39:56.168 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.168 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:56.168 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.168 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:56.168 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516122+0000", 2024-09-18T04:39:56.168 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:56.168 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.168 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.168 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:56.168 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516122+0000", 2024-09-18T04:39:56.168 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:56.168 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.168 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.169 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:56.169 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516120+0000", 2024-09-18T04:39:56.169 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:56.169 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.169 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.169 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:56.169 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516129+0000", 2024-09-18T04:39:56.169 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.85e-06 2024-09-18T04:39:56.169 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.169 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.169 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:56.169 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516155+0000", 2024-09-18T04:39:56.169 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5882000000000001e-05 2024-09-18T04:39:56.169 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.170 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.170 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:56.170 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516175+0000", 2024-09-18T04:39:56.170 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.022e-05 2024-09-18T04:39:56.170 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.170 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.170 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:56.170 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519780+0000", 2024-09-18T04:39:56.170 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003604187 2024-09-18T04:39:56.170 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.170 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.170 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:56.170 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519964+0000", 2024-09-18T04:39:56.170 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00018391999999999999 2024-09-18T04:39:56.171 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.171 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.171 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:56.171 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.520794+0000", 2024-09-18T04:39:56.171 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00083078100000000005 2024-09-18T04:39:56.171 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.171 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.171 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:56.171 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.524129+0000", 2024-09-18T04:39:56.171 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033342419999999998 2024-09-18T04:39:56.171 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.171 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.172 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:56.172 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525239+0000", 2024-09-18T04:39:56.172 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001110778 2024-09-18T04:39:56.172 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.172 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.172 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:56.172 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525277+0000", 2024-09-18T04:39:56.172 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7596999999999999e-05 2024-09-18T04:39:56.172 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.172 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.172 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:56.172 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525308+0000", 2024-09-18T04:39:56.172 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.1535999999999998e-05 2024-09-18T04:39:56.172 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:56.173 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:56.173 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:56.173 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.173 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.173 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:32 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:56.173 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.519217+0000", 2024-09-18T04:39:56.173 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 73.604100119999998, 2024-09-18T04:39:56.173 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082266370000000002, 2024-09-18T04:39:56.173 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:56.173 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:56.173 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:56.173 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:56.173 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:56.174 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 32 2024-09-18T04:39:56.174 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.174 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:56.174 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.174 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:56.174 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519217+0000", 2024-09-18T04:39:56.174 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:56.174 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.174 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.174 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:56.174 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519217+0000", 2024-09-18T04:39:56.174 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:56.174 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.174 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.175 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:56.175 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519215+0000", 2024-09-18T04:39:56.175 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:56.175 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.175 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.175 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:56.175 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519229+0000", 2024-09-18T04:39:56.175 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3329999999999999e-05 2024-09-18T04:39:56.175 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.175 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.175 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:56.175 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519268+0000", 2024-09-18T04:39:56.175 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9533000000000002e-05 2024-09-18T04:39:56.176 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.176 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.176 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:56.176 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519285+0000", 2024-09-18T04:39:56.176 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6446000000000001e-05 2024-09-18T04:39:56.176 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.176 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.176 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:56.176 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.522358+0000", 2024-09-18T04:39:56.176 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030734310000000002 2024-09-18T04:39:56.176 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.176 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.176 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:56.177 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.522551+0000", 2024-09-18T04:39:56.177 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019317600000000001 2024-09-18T04:39:56.177 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.177 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.177 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:56.177 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.523262+0000", 2024-09-18T04:39:56.177 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00071069999999999998 2024-09-18T04:39:56.177 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.177 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.177 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:56.177 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525318+0000", 2024-09-18T04:39:56.177 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0020561469999999999 2024-09-18T04:39:56.177 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.177 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.178 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:56.178 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.527376+0000", 2024-09-18T04:39:56.178 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0020584969999999998 2024-09-18T04:39:56.178 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.178 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.178 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:56.178 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.527416+0000", 2024-09-18T04:39:56.178 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9054000000000003e-05 2024-09-18T04:39:56.178 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.178 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.178 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:56.178 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.527444+0000", 2024-09-18T04:39:56.178 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8347000000000002e-05 2024-09-18T04:39:56.178 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:56.179 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:56.179 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:56.179 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.179 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.179 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:42 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:56.179 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.551160+0000", 2024-09-18T04:39:56.179 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 73.572157727999993, 2024-09-18T04:39:56.179 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010692757000000001, 2024-09-18T04:39:56.179 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:56.179 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:56.179 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:56.179 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:39:56.180 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:56.180 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 42 2024-09-18T04:39:56.180 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.180 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:56.180 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.180 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:56.180 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551160+0000", 2024-09-18T04:39:56.180 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:56.180 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.180 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.180 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:56.180 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551160+0000", 2024-09-18T04:39:56.180 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:56.181 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.181 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.181 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:56.181 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551159+0000", 2024-09-18T04:39:56.181 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:56.181 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.181 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.181 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:56.181 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551166+0000", 2024-09-18T04:39:56.181 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1029999999999996e-06 2024-09-18T04:39:56.181 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.181 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.181 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:56.181 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551177+0000", 2024-09-18T04:39:56.182 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1384e-05 2024-09-18T04:39:56.182 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.182 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.182 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:56.182 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551185+0000", 2024-09-18T04:39:56.182 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1990000000000006e-06 2024-09-18T04:39:56.182 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.182 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.182 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:56.182 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552021+0000", 2024-09-18T04:39:56.183 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00083574400000000003 2024-09-18T04:39:56.183 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.183 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.183 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:56.183 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552339+0000", 2024-09-18T04:39:56.183 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031771399999999997 2024-09-18T04:39:56.183 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.183 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.183 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:56.183 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.553140+0000", 2024-09-18T04:39:56.183 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00080088100000000003 2024-09-18T04:39:56.183 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.183 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.183 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:56.184 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.554585+0000", 2024-09-18T04:39:56.184 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001445801 2024-09-18T04:39:56.184 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.184 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.184 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:56.184 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.561711+0000", 2024-09-18T04:39:56.184 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071260109999999998 2024-09-18T04:39:56.184 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.184 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.184 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:56.184 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.561790+0000", 2024-09-18T04:39:56.184 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8967000000000005e-05 2024-09-18T04:39:56.185 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.185 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.185 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:56.185 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.561852+0000", 2024-09-18T04:39:56.185 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2045999999999995e-05 2024-09-18T04:39:56.185 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:56.185 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:56.185 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:56.185 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.185 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.185 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:58 2.7 e51/32 2:fc7c6cd3:ns::rbd_data.133e813ab5bc.000000000000000e:head v 51'7, mlcod=51'5)", 2024-09-18T04:39:56.185 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.605333+0000", 2024-09-18T04:39:56.185 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 73.517984522999996, 2024-09-18T04:39:56.186 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020085902999999999, 2024-09-18T04:39:56.186 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:56.186 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:56.186 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:56.186 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.186 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:56.186 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605333+0000", 2024-09-18T04:39:56.186 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:56.186 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.186 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.186 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:56.186 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605333+0000", 2024-09-18T04:39:56.186 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:56.186 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.187 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.187 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:56.187 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605332+0000", 2024-09-18T04:39:56.187 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:56.187 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.187 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.187 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:56.187 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608667+0000", 2024-09-18T04:39:56.187 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033350300000000001 2024-09-18T04:39:56.187 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.187 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.187 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:56.187 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608706+0000", 2024-09-18T04:39:56.187 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8951e-05 2024-09-18T04:39:56.188 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.188 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.188 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:56.188 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608725+0000", 2024-09-18T04:39:56.188 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9675999999999999e-05 2024-09-18T04:39:56.188 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.188 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.188 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:56.188 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608798+0000", 2024-09-18T04:39:56.188 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3135999999999999e-05 2024-09-18T04:39:56.188 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.188 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.188 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:56.188 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608824+0000", 2024-09-18T04:39:56.189 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5284000000000001e-05 2024-09-18T04:39:56.189 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.189 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.189 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:56.189 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625370+0000", 2024-09-18T04:39:56.189 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016545975000000001 2024-09-18T04:39:56.189 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.189 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.189 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:56.189 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625419+0000", 2024-09-18T04:39:56.189 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9076999999999998e-05 2024-09-18T04:39:56.189 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:56.189 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:56.189 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:56.190 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.190 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.190 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:611 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:56.190 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.069561+0000", 2024-09-18T04:39:56.190 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 65.053755968999994, 2024-09-18T04:39:56.190 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0072330299999999997, 2024-09-18T04:39:56.190 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:56.190 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:56.190 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:56.190 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:39:56.190 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:56.190 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 611 2024-09-18T04:39:56.191 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.191 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:56.191 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.191 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:56.191 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.069561+0000", 2024-09-18T04:39:56.191 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:56.191 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.191 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.191 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:56.191 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.069561+0000", 2024-09-18T04:39:56.191 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:56.191 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.191 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.192 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:56.192 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.069559+0000", 2024-09-18T04:39:56.192 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:56.192 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.192 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.192 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:56.192 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070801+0000", 2024-09-18T04:39:56.192 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0012423359999999999 2024-09-18T04:39:56.192 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.192 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.192 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:56.192 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070828+0000", 2024-09-18T04:39:56.192 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6387e-05 2024-09-18T04:39:56.193 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.193 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.193 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:56.193 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070840+0000", 2024-09-18T04:39:56.193 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2689e-05 2024-09-18T04:39:56.193 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.193 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.193 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:56.193 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070990+0000", 2024-09-18T04:39:56.193 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014946800000000001 2024-09-18T04:39:56.193 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.193 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.193 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:56.194 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.071128+0000", 2024-09-18T04:39:56.194 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013796699999999999 2024-09-18T04:39:56.194 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.194 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.194 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:56.194 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.071318+0000", 2024-09-18T04:39:56.194 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000190106 2024-09-18T04:39:56.194 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.194 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.194 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:56.194 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.075711+0000", 2024-09-18T04:39:56.194 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.004393002 2024-09-18T04:39:56.194 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.194 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.195 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:56.195 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.076665+0000", 2024-09-18T04:39:56.195 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00095390399999999999 2024-09-18T04:39:56.195 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.195 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.195 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:56.195 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.076742+0000", 2024-09-18T04:39:56.195 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7112000000000003e-05 2024-09-18T04:39:56.195 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.195 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.195 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:56.195 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.076794+0000", 2024-09-18T04:39:56.195 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2475999999999997e-05 2024-09-18T04:39:56.195 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:56.196 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:56.196 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:56.196 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.196 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.196 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:687 2.3 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:56.196 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.487994+0000", 2024-09-18T04:39:56.196 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 64.635322986000006, 2024-09-18T04:39:56.196 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0074167690000000001, 2024-09-18T04:39:56.196 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:56.196 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:56.196 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:56.196 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:39:56.197 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:56.197 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 687 2024-09-18T04:39:56.197 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.197 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:56.197 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.197 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:56.197 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.487994+0000", 2024-09-18T04:39:56.197 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:56.197 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.197 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.197 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:56.197 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.487994+0000", 2024-09-18T04:39:56.197 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:56.198 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.198 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.198 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:56.198 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.487993+0000", 2024-09-18T04:39:56.198 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:56.198 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.198 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.198 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:56.198 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.488979+0000", 2024-09-18T04:39:56.198 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00098620700000000001 2024-09-18T04:39:56.198 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.198 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.198 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:56.198 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489002+0000", 2024-09-18T04:39:56.198 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2500000000000001e-05 2024-09-18T04:39:56.199 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.199 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.199 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:56.199 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489020+0000", 2024-09-18T04:39:56.199 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7691000000000001e-05 2024-09-18T04:39:56.199 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.199 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.199 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:56.199 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489091+0000", 2024-09-18T04:39:56.199 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1307e-05 2024-09-18T04:39:56.199 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.199 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.199 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:56.199 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489219+0000", 2024-09-18T04:39:56.200 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012766000000000001 2024-09-18T04:39:56.200 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.200 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.200 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:56.200 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489387+0000", 2024-09-18T04:39:56.200 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016855400000000001 2024-09-18T04:39:56.200 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.200 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.200 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:56.200 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495243+0000", 2024-09-18T04:39:56.200 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058563189999999996 2024-09-18T04:39:56.200 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.201 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.201 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:56.201 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495266+0000", 2024-09-18T04:39:56.201 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2810999999999999e-05 2024-09-18T04:39:56.201 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.201 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.201 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:56.201 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495332+0000", 2024-09-18T04:39:56.201 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5469999999999995e-05 2024-09-18T04:39:56.201 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.201 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.201 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:56.201 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495411+0000", 2024-09-18T04:39:56.201 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9395999999999997e-05 2024-09-18T04:39:56.202 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:56.202 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:56.202 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:56.202 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.202 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.202 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:689 2.3 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:56.202 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.500070+0000", 2024-09-18T04:39:56.202 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 64.623247169999999, 2024-09-18T04:39:56.202 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071096650000000003, 2024-09-18T04:39:56.202 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:56.202 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:56.202 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:56.202 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:39:56.202 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:39:56.202 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 689 2024-09-18T04:39:56.203 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.203 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:56.203 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.203 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:56.203 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.500070+0000", 2024-09-18T04:39:56.203 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:56.203 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.203 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.203 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:56.203 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.500070+0000", 2024-09-18T04:39:56.203 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:56.203 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.203 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.204 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:56.204 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.500069+0000", 2024-09-18T04:39:56.204 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:56.204 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.204 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.204 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:56.204 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501082+0000", 2024-09-18T04:39:56.204 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010136570000000001 2024-09-18T04:39:56.204 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.204 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.204 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:56.204 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501104+0000", 2024-09-18T04:39:56.204 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1883e-05 2024-09-18T04:39:56.204 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.205 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.205 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:56.205 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501115+0000", 2024-09-18T04:39:56.205 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0414000000000001e-05 2024-09-18T04:39:56.205 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.205 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.205 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:56.205 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501178+0000", 2024-09-18T04:39:56.205 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3128999999999998e-05 2024-09-18T04:39:56.205 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.205 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.205 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:56.205 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501257+0000", 2024-09-18T04:39:56.205 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9592999999999995e-05 2024-09-18T04:39:56.206 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.206 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.206 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:56.206 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501366+0000", 2024-09-18T04:39:56.206 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000108643 2024-09-18T04:39:56.206 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.206 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.206 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:56.206 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.504041+0000", 2024-09-18T04:39:56.206 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0026750889999999999 2024-09-18T04:39:56.206 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.206 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.206 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:56.206 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.507097+0000", 2024-09-18T04:39:56.207 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030556070000000001 2024-09-18T04:39:56.207 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.207 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.207 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:56.207 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.507140+0000", 2024-09-18T04:39:56.207 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3745000000000002e-05 2024-09-18T04:39:56.207 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.207 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.207 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:56.207 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.507180+0000", 2024-09-18T04:39:56.207 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9475999999999999e-05 2024-09-18T04:39:56.207 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:56.207 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:56.207 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:56.208 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.208 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.208 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4972.0:13 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:39:56.208 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.948263+0000", 2024-09-18T04:39:56.208 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 64.175054379000002, 2024-09-18T04:39:56.208 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0091358970000000005, 2024-09-18T04:39:56.208 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:56.208 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:39:56.208 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:56.208 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4972", 2024-09-18T04:39:56.208 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2280251472", 2024-09-18T04:39:56.208 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 13 2024-09-18T04:39:56.208 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.208 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:56.209 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.209 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:56.209 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948263+0000", 2024-09-18T04:39:56.209 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:56.209 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.209 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.209 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:56.209 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948263+0000", 2024-09-18T04:39:56.209 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:56.209 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.209 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.209 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:56.209 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948261+0000", 2024-09-18T04:39:56.210 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:56.210 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.210 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.210 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:56.210 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948273+0000", 2024-09-18T04:39:56.210 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1576999999999999e-05 2024-09-18T04:39:56.210 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.210 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.210 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:56.210 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948305+0000", 2024-09-18T04:39:56.210 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2209000000000003e-05 2024-09-18T04:39:56.210 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.210 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.211 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:56.211 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948325+0000", 2024-09-18T04:39:56.211 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.022e-05 2024-09-18T04:39:56.211 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.211 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.211 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:56.211 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948455+0000", 2024-09-18T04:39:56.211 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000129624 2024-09-18T04:39:56.211 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.211 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.211 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:56.211 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948598+0000", 2024-09-18T04:39:56.211 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014376699999999999 2024-09-18T04:39:56.211 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.212 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.212 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:56.212 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.957399+0000", 2024-09-18T04:39:56.212 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0088004299999999997 2024-09-18T04:39:56.212 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:56.212 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:56.212 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:56.212 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.212 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.212 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:101 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:39:56.212 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.576367+0000", 2024-09-18T04:39:56.212 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 59.546950670999998, 2024-09-18T04:39:56.212 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0078671020000000008, 2024-09-18T04:39:56.213 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:56.213 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:56.213 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:56.213 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:39:56.213 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:39:56.213 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 101 2024-09-18T04:39:56.213 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.213 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:56.213 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.213 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:56.213 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576367+0000", 2024-09-18T04:39:56.213 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:56.213 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.213 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.213 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:56.214 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576367+0000", 2024-09-18T04:39:56.214 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:56.214 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.214 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.214 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:56.214 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576364+0000", 2024-09-18T04:39:56.214 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999971 2024-09-18T04:39:56.214 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.214 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.214 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:56.214 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576378+0000", 2024-09-18T04:39:56.214 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3879999999999999e-05 2024-09-18T04:39:56.214 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.215 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.215 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:56.215 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576410+0000", 2024-09-18T04:39:56.215 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2286000000000002e-05 2024-09-18T04:39:56.215 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.215 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.215 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:56.215 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576425+0000", 2024-09-18T04:39:56.215 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5290000000000001e-05 2024-09-18T04:39:56.215 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.215 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.215 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:56.216 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576551+0000", 2024-09-18T04:39:56.216 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012555199999999999 2024-09-18T04:39:56.216 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.216 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.216 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:56.216 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576719+0000", 2024-09-18T04:39:56.216 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016787400000000001 2024-09-18T04:39:56.216 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.217 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.217 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:56.217 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576951+0000", 2024-09-18T04:39:56.217 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00023252400000000001 2024-09-18T04:39:56.217 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.217 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.217 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:56.217 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.581459+0000", 2024-09-18T04:39:56.217 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045074410000000001 2024-09-18T04:39:56.217 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.217 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.217 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:56.218 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584076+0000", 2024-09-18T04:39:56.218 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0026173860000000002 2024-09-18T04:39:56.218 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.218 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.218 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:56.218 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584160+0000", 2024-09-18T04:39:56.218 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4191999999999996e-05 2024-09-18T04:39:56.218 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.218 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.218 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:56.218 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584234+0000", 2024-09-18T04:39:56.218 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.339e-05 2024-09-18T04:39:56.218 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:56.218 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:56.219 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:56.219 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.219 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.219 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:44 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:56.219 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.550448+0000", 2024-09-18T04:39:56.219 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 55.572869482000002, 2024-09-18T04:39:56.219 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082117590000000008, 2024-09-18T04:39:56.219 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:56.219 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:56.219 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:56.219 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:39:56.219 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:56.219 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 44 2024-09-18T04:39:56.220 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.220 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:56.220 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.220 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:56.220 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.550448+0000", 2024-09-18T04:39:56.220 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:56.220 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.220 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.220 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:56.220 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.550448+0000", 2024-09-18T04:39:56.220 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:56.220 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.220 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.221 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:56.221 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.550445+0000", 2024-09-18T04:39:56.221 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999971 2024-09-18T04:39:56.221 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.221 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.221 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:56.221 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.551904+0000", 2024-09-18T04:39:56.221 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001458322 2024-09-18T04:39:56.221 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.221 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.221 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:56.221 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.551938+0000", 2024-09-18T04:39:56.221 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4359000000000001e-05 2024-09-18T04:39:56.222 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.222 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.222 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:56.222 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.551959+0000", 2024-09-18T04:39:56.222 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1403000000000002e-05 2024-09-18T04:39:56.222 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.222 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.222 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:56.222 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.552049+0000", 2024-09-18T04:39:56.222 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9970999999999996e-05 2024-09-18T04:39:56.222 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.222 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.222 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:56.223 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.552190+0000", 2024-09-18T04:39:56.223 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000140271 2024-09-18T04:39:56.223 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.223 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.223 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:56.223 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.552385+0000", 2024-09-18T04:39:56.223 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019498700000000001 2024-09-18T04:39:56.223 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.223 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.223 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:56.223 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.556308+0000", 2024-09-18T04:39:56.224 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00392333 2024-09-18T04:39:56.224 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.224 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.224 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:56.224 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.558479+0000", 2024-09-18T04:39:56.224 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0021715889999999998 2024-09-18T04:39:56.224 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.224 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.224 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:56.225 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.558616+0000", 2024-09-18T04:39:56.225 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013613399999999999 2024-09-18T04:39:56.225 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.225 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.225 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:56.225 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.558660+0000", 2024-09-18T04:39:56.225 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4053e-05 2024-09-18T04:39:56.225 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:56.225 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:56.225 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:56.225 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.225 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.225 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:46 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:56.225 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.564089+0000", 2024-09-18T04:39:56.225 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 55.559228353000002, 2024-09-18T04:39:56.226 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0069437090000000002, 2024-09-18T04:39:56.226 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:56.226 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:56.226 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:56.226 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:39:56.226 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:56.226 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 46 2024-09-18T04:39:56.226 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.226 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:56.226 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.226 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:56.226 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.564089+0000", 2024-09-18T04:39:56.226 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:56.227 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.227 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.227 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:56.227 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.564089+0000", 2024-09-18T04:39:56.227 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:56.227 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.227 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.227 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:56.227 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.564087+0000", 2024-09-18T04:39:56.227 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:56.227 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.227 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.227 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:56.227 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565126+0000", 2024-09-18T04:39:56.228 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001038902 2024-09-18T04:39:56.228 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.228 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.228 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:56.228 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565148+0000", 2024-09-18T04:39:56.228 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1911000000000001e-05 2024-09-18T04:39:56.228 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.228 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.228 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:56.228 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565157+0000", 2024-09-18T04:39:56.228 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6220000000000004e-06 2024-09-18T04:39:56.228 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.228 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.228 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:56.229 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565261+0000", 2024-09-18T04:39:56.229 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000103063 2024-09-18T04:39:56.229 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.229 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.229 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:56.229 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565394+0000", 2024-09-18T04:39:56.229 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000133032 2024-09-18T04:39:56.229 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.229 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.229 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:39:56.229 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565564+0000", 2024-09-18T04:39:56.229 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017068000000000001 2024-09-18T04:39:56.229 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.229 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.229 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:56.230 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.570556+0000", 2024-09-18T04:39:56.230 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0049920620000000002 2024-09-18T04:39:56.230 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.230 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.230 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:56.230 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.570885+0000", 2024-09-18T04:39:56.230 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032827900000000001 2024-09-18T04:39:56.230 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.230 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.230 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:56.230 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.570959+0000", 2024-09-18T04:39:56.230 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3955999999999995e-05 2024-09-18T04:39:56.230 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.230 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.231 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:56.231 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.571033+0000", 2024-09-18T04:39:56.231 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4191999999999997e-05 2024-09-18T04:39:56.231 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:56.231 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:56.231 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:56.231 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.231 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.231 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:100 2.4 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:56.231 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.856389+0000", 2024-09-18T04:39:56.231 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 55.266928241000002, 2024-09-18T04:39:56.231 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0081480800000000003, 2024-09-18T04:39:56.231 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:56.231 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:56.232 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:56.232 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:39:56.232 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:56.232 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 100 2024-09-18T04:39:56.232 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.232 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:56.232 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.232 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:56.232 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.856389+0000", 2024-09-18T04:39:56.232 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:56.232 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.232 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.232 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:56.232 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.856389+0000", 2024-09-18T04:39:56.233 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:56.233 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.233 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.233 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:56.233 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.856388+0000", 2024-09-18T04:39:56.233 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:39:56.233 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.233 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.233 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:56.233 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857295+0000", 2024-09-18T04:39:56.233 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00090725599999999995 2024-09-18T04:39:56.233 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.233 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.233 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:56.234 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857311+0000", 2024-09-18T04:39:56.234 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6092000000000001e-05 2024-09-18T04:39:56.234 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.234 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.234 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:56.234 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857319+0000", 2024-09-18T04:39:56.234 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9200000000000004e-06 2024-09-18T04:39:56.234 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.234 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.234 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:56.234 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857388+0000", 2024-09-18T04:39:56.234 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8591999999999997e-05 2024-09-18T04:39:56.234 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.234 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.234 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:56.235 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857535+0000", 2024-09-18T04:39:56.235 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014684299999999999 2024-09-18T04:39:56.235 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.235 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.235 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:56.235 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857747+0000", 2024-09-18T04:39:56.235 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000212361 2024-09-18T04:39:56.235 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.235 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.235 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:56.235 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.860831+0000", 2024-09-18T04:39:56.235 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030841789999999998 2024-09-18T04:39:56.235 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.235 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.236 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:56.236 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.864361+0000", 2024-09-18T04:39:56.236 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0035298780000000002 2024-09-18T04:39:56.236 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.236 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.236 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:56.236 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.864456+0000", 2024-09-18T04:39:56.236 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4557999999999998e-05 2024-09-18T04:39:56.236 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.236 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.236 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:56.236 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.864537+0000", 2024-09-18T04:39:56.237 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1577999999999997e-05 2024-09-18T04:39:56.237 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:56.237 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:56.237 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:56.237 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.237 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.237 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:102 2.4 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:56.237 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.869773+0000", 2024-09-18T04:39:56.237 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 55.253544583999997, 2024-09-18T04:39:56.237 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0084430359999999992, 2024-09-18T04:39:56.237 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:56.237 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:56.237 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:56.237 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:39:56.238 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:56.238 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 102 2024-09-18T04:39:56.238 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.238 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:56.238 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.238 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:56.238 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.869773+0000", 2024-09-18T04:39:56.238 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:56.238 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.238 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.238 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:56.238 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.869773+0000", 2024-09-18T04:39:56.238 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:56.239 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.239 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.239 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:56.239 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.869771+0000", 2024-09-18T04:39:56.239 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:56.239 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.239 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.239 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:56.239 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870629+0000", 2024-09-18T04:39:56.239 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00085760999999999997 2024-09-18T04:39:56.239 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.239 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.239 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:56.239 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870647+0000", 2024-09-18T04:39:56.240 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8363999999999999e-05 2024-09-18T04:39:56.240 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.240 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.240 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:56.240 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870655+0000", 2024-09-18T04:39:56.240 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0099999999999995e-06 2024-09-18T04:39:56.240 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.240 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.240 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:56.240 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870803+0000", 2024-09-18T04:39:56.240 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014798 2024-09-18T04:39:56.240 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.241 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.241 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:56.241 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870951+0000", 2024-09-18T04:39:56.241 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014715999999999999 2024-09-18T04:39:56.241 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.241 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.241 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:56.241 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.871145+0000", 2024-09-18T04:39:56.241 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000194253 2024-09-18T04:39:56.241 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.241 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.241 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:56.241 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.877033+0000", 2024-09-18T04:39:56.241 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058883099999999999 2024-09-18T04:39:56.242 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.242 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.242 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:56.242 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.878078+0000", 2024-09-18T04:39:56.242 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001044747 2024-09-18T04:39:56.242 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.242 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.242 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:56.242 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.878160+0000", 2024-09-18T04:39:56.242 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1904999999999996e-05 2024-09-18T04:39:56.242 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.242 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.242 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:56.242 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.878216+0000", 2024-09-18T04:39:56.242 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6082000000000003e-05 2024-09-18T04:39:56.243 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:56.243 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:56.243 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:56.243 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.243 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.243 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:145 2.4 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:39:56.243 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:01.096935+0000", 2024-09-18T04:39:56.243 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 55.026382804000001, 2024-09-18T04:39:56.243 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0069937869999999996, 2024-09-18T04:39:56.243 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:39:56.243 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:56.243 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:39:56.243 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:39:56.243 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:39:56.244 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 145 2024-09-18T04:39:56.244 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.244 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:39:56.244 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.244 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:39:56.244 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.096935+0000", 2024-09-18T04:39:56.244 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:56.244 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.244 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.244 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:39:56.244 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.096935+0000", 2024-09-18T04:39:56.244 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:39:56.244 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.245 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.245 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:39:56.245 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.096932+0000", 2024-09-18T04:39:56.245 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:56.245 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.245 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.245 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:39:56.245 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097815+0000", 2024-09-18T04:39:56.245 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00088276000000000001 2024-09-18T04:39:56.245 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.245 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.245 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:39:56.245 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097842+0000", 2024-09-18T04:39:56.245 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6806999999999999e-05 2024-09-18T04:39:56.246 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.246 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.246 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:39:56.246 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097852+0000", 2024-09-18T04:39:56.246 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0548e-05 2024-09-18T04:39:56.246 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.246 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.246 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:39:56.246 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097931+0000", 2024-09-18T04:39:56.246 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8417000000000005e-05 2024-09-18T04:39:56.246 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.246 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.246 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:39:56.246 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.098047+0000", 2024-09-18T04:39:56.247 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011611 2024-09-18T04:39:56.247 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.247 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.247 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:56.247 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.098172+0000", 2024-09-18T04:39:56.247 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000125482 2024-09-18T04:39:56.247 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.247 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.247 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:39:56.247 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.100888+0000", 2024-09-18T04:39:56.247 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0027154430000000001 2024-09-18T04:39:56.247 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.248 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.248 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:56.248 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.103814+0000", 2024-09-18T04:39:56.248 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0029259619999999998 2024-09-18T04:39:56.248 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.248 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.248 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:39:56.248 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.103891+0000", 2024-09-18T04:39:56.248 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7619000000000003e-05 2024-09-18T04:39:56.248 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:39:56.248 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:39:56.248 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:39:56.248 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.103928+0000", 2024-09-18T04:39:56.248 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6900999999999998e-05 2024-09-18T04:39:56.249 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:56.249 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:56.249 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:56.249 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:39:56.249 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:39:56.249 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:39:56.251 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:39:56.251 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:56.247+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:56.352 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:56.347+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:56.370 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:56.370 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:56.370 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:39:56.370 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:56.380 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:39:56.453 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:56.447+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:56.492 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:56.492 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:56.492 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:39:56.492 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:39:56.492 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:56.502 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:39:56.554 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:56.551+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:39:56.611 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:56.611 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:39:56.611 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:39:56.611 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:39:56.611 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.611 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:7, version: 79'62, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:7@79'62, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 8=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-18T04:39:56.611 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.724969+0000", 2024-09-18T04:39:56.611 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 9.8825683160000004, 2024-09-18T04:39:56.611 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0035525930000000002, 2024-09-18T04:39:56.612 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:56.612 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:56.612 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:56.612 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.612 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:56.612 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.724969+0000", 2024-09-18T04:39:56.612 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:56.612 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.612 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.612 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:56.612 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.724969+0000", 2024-09-18T04:39:56.612 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:56.612 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.612 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.613 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:56.613 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.724968+0000", 2024-09-18T04:39:56.613 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:56.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.613 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.613 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:56.613 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.726245+0000", 2024-09-18T04:39:56.613 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001276395 2024-09-18T04:39:56.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.613 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.613 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:56.613 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.726262+0000", 2024-09-18T04:39:56.613 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6973000000000002e-05 2024-09-18T04:39:56.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.614 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.614 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:56.614 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.726293+0000", 2024-09-18T04:39:56.614 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0886999999999998e-05 2024-09-18T04:39:56.614 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.614 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.614 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:56.614 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.726403+0000", 2024-09-18T04:39:56.614 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000110288 2024-09-18T04:39:56.614 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.614 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.614 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:56.614 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.726418+0000", 2024-09-18T04:39:56.615 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5503999999999999e-05 2024-09-18T04:39:56.615 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.615 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.615 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:56.615 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.728522+0000", 2024-09-18T04:39:56.615 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021032630000000002 2024-09-18T04:39:56.615 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:56.615 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:56.615 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:56.615 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.615 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.615 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.1 87/86 [PushOp(2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head, version: 81'220, data_included: [4182016~12288], data_size: 12288, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head@81'220, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:4182016, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:39:56.615 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.730366+0000", 2024-09-18T04:39:56.615 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 9.8771719190000002, 2024-09-18T04:39:56.616 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0032522580000000001, 2024-09-18T04:39:56.616 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:56.616 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:56.616 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:56.616 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.616 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:56.616 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.730366+0000", 2024-09-18T04:39:56.616 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:56.616 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.616 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.616 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:56.616 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.730366+0000", 2024-09-18T04:39:56.616 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:56.616 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.616 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.617 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:56.617 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.730365+0000", 2024-09-18T04:39:56.617 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:56.617 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.617 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.617 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:56.617 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.730385+0000", 2024-09-18T04:39:56.617 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0766000000000001e-05 2024-09-18T04:39:56.617 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.617 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.617 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:56.617 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.730406+0000", 2024-09-18T04:39:56.617 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0706000000000001e-05 2024-09-18T04:39:56.618 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.618 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.618 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:56.618 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.730423+0000", 2024-09-18T04:39:56.618 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7411e-05 2024-09-18T04:39:56.618 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.618 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.618 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:56.618 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.730507+0000", 2024-09-18T04:39:56.619 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3388999999999997e-05 2024-09-18T04:39:56.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.619 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.619 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:56.619 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.730523+0000", 2024-09-18T04:39:56.619 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5843999999999999e-05 2024-09-18T04:39:56.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.619 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.619 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:56.619 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.733618+0000", 2024-09-18T04:39:56.619 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030951540000000001 2024-09-18T04:39:56.619 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:56.619 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:56.619 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:56.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.620 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.620 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:7, version: 79'62, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:7@79'62, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 8=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:39:56.620 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.732222+0000", 2024-09-18T04:39:56.620 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 9.8753159589999999, 2024-09-18T04:39:56.620 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030084410000000002, 2024-09-18T04:39:56.620 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:56.620 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:56.620 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:56.620 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.620 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:56.620 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.732222+0000", 2024-09-18T04:39:56.620 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:56.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.620 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.621 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:56.621 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.732222+0000", 2024-09-18T04:39:56.621 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:56.621 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.621 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.621 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:56.621 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.732221+0000", 2024-09-18T04:39:56.621 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:56.621 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.621 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.621 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:56.621 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.733174+0000", 2024-09-18T04:39:56.622 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00095350299999999997 2024-09-18T04:39:56.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.622 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.622 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:56.622 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.733197+0000", 2024-09-18T04:39:56.622 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2925000000000002e-05 2024-09-18T04:39:56.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.622 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.622 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:56.622 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.733218+0000", 2024-09-18T04:39:56.622 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0477999999999999e-05 2024-09-18T04:39:56.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.622 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.623 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:56.623 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.733323+0000", 2024-09-18T04:39:56.623 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000105717 2024-09-18T04:39:56.623 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.623 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.623 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:56.623 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.733358+0000", 2024-09-18T04:39:56.623 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4406000000000002e-05 2024-09-18T04:39:56.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.624 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.624 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:56.624 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.735230+0000", 2024-09-18T04:39:56.624 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0018720659999999999 2024-09-18T04:39:56.624 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:56.625 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:56.625 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:56.625 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.625 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.625 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:7, version: 79'62, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:7@79'62, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 8=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:39:56.626 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.743433+0000", 2024-09-18T04:39:56.626 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 9.8641044779999998, 2024-09-18T04:39:56.626 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003400237, 2024-09-18T04:39:56.626 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:56.626 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:56.626 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:56.626 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.626 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:56.627 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.743433+0000", 2024-09-18T04:39:56.627 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:56.627 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.627 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.627 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:56.627 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.743433+0000", 2024-09-18T04:39:56.627 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:56.627 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.627 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.627 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:56.627 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.743432+0000", 2024-09-18T04:39:56.627 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:56.627 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.627 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.628 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:56.628 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.744392+0000", 2024-09-18T04:39:56.628 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00095993499999999998 2024-09-18T04:39:56.628 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.628 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.628 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:56.628 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.744417+0000", 2024-09-18T04:39:56.628 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5249000000000002e-05 2024-09-18T04:39:56.628 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.628 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.628 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:56.629 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.744434+0000", 2024-09-18T04:39:56.629 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6597999999999999e-05 2024-09-18T04:39:56.629 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.629 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.629 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:56.629 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.744499+0000", 2024-09-18T04:39:56.629 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4987000000000006e-05 2024-09-18T04:39:56.629 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.629 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.629 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:56.629 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.744506+0000", 2024-09-18T04:39:56.629 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2679999999999997e-06 2024-09-18T04:39:56.629 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.630 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.630 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:56.630 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.746833+0000", 2024-09-18T04:39:56.630 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023270529999999999 2024-09-18T04:39:56.630 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:56.630 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:56.630 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:56.630 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.630 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.630 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.5 87/86 [PushOp(2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:head, version: 79'60, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:head@79'60, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-18T04:39:56.630 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.747807+0000", 2024-09-18T04:39:56.630 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 9.8597300039999993, 2024-09-18T04:39:56.630 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00244378, 2024-09-18T04:39:56.630 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:56.631 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:56.631 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:56.631 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.631 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:56.631 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.747807+0000", 2024-09-18T04:39:56.631 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:56.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.631 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.631 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:56.631 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.747807+0000", 2024-09-18T04:39:56.631 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:56.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.631 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.631 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:56.632 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.747806+0000", 2024-09-18T04:39:56.632 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:56.632 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.632 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.632 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:56.632 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.747825+0000", 2024-09-18T04:39:56.632 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.925e-05 2024-09-18T04:39:56.632 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.632 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.632 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:56.632 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.747853+0000", 2024-09-18T04:39:56.632 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7174000000000001e-05 2024-09-18T04:39:56.632 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.633 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.633 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:56.633 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.747887+0000", 2024-09-18T04:39:56.633 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4731999999999999e-05 2024-09-18T04:39:56.633 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.633 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.633 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:56.633 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.747983+0000", 2024-09-18T04:39:56.634 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5964999999999993e-05 2024-09-18T04:39:56.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.634 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.634 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:56.634 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.747994+0000", 2024-09-18T04:39:56.634 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0526999999999999e-05 2024-09-18T04:39:56.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.634 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.634 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:56.635 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.750251+0000", 2024-09-18T04:39:56.635 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0022573519999999998 2024-09-18T04:39:56.635 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:56.635 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:56.635 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:56.635 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.635 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.635 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:7, version: 79'64, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:7@79'64, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-18T04:39:56.635 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.750570+0000", 2024-09-18T04:39:56.636 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 9.8569676010000009, 2024-09-18T04:39:56.636 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0029248949999999998, 2024-09-18T04:39:56.636 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:56.636 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:56.636 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:56.636 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.636 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:56.636 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.750570+0000", 2024-09-18T04:39:56.637 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:56.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.637 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.637 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:56.637 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.750570+0000", 2024-09-18T04:39:56.637 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:56.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.637 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.637 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:56.638 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.750569+0000", 2024-09-18T04:39:56.638 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:56.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.638 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.638 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:56.638 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.751724+0000", 2024-09-18T04:39:56.638 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001155037 2024-09-18T04:39:56.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.638 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.639 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:56.639 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.751747+0000", 2024-09-18T04:39:56.639 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2679000000000001e-05 2024-09-18T04:39:56.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.639 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.639 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:56.639 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.751761+0000", 2024-09-18T04:39:56.639 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.467e-05 2024-09-18T04:39:56.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.639 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.639 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:56.639 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.751839+0000", 2024-09-18T04:39:56.640 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7824000000000006e-05 2024-09-18T04:39:56.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.640 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.640 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:56.640 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.751849+0000", 2024-09-18T04:39:56.640 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4159999999999993e-06 2024-09-18T04:39:56.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.640 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.640 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:56.640 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.753495+0000", 2024-09-18T04:39:56.640 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0016461830000000001 2024-09-18T04:39:56.640 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:56.640 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:56.640 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:56.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.641 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.641 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.5 87/86 [PushOp(2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:7, version: 79'59, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:7@79'59, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-18T04:39:56.641 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.752479+0000", 2024-09-18T04:39:56.641 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 9.8550582339999995, 2024-09-18T04:39:56.641 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027271040000000002, 2024-09-18T04:39:56.641 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:56.641 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:56.641 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:56.641 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.641 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:56.641 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.752479+0000", 2024-09-18T04:39:56.641 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:56.642 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.642 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.642 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:56.642 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.752479+0000", 2024-09-18T04:39:56.642 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:56.642 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.642 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.643 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:56.643 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.752478+0000", 2024-09-18T04:39:56.643 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:56.643 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.643 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.643 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:56.644 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.753784+0000", 2024-09-18T04:39:56.644 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0013058869999999999 2024-09-18T04:39:56.644 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.644 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.644 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:56.644 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.753821+0000", 2024-09-18T04:39:56.644 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7268999999999998e-05 2024-09-18T04:39:56.644 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.644 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.644 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:56.644 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.753873+0000", 2024-09-18T04:39:56.644 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1564000000000003e-05 2024-09-18T04:39:56.644 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.644 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.645 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:56.645 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.753963+0000", 2024-09-18T04:39:56.645 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0636000000000004e-05 2024-09-18T04:39:56.645 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.645 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.646 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:56.646 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.753972+0000", 2024-09-18T04:39:56.646 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0529999999999996e-06 2024-09-18T04:39:56.646 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.646 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.646 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:56.646 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.755206+0000", 2024-09-18T04:39:56.646 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001234113 2024-09-18T04:39:56.646 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:56.647 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:56.647 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:56.647 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.647 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.647 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:7, version: 79'64, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:7@79'64, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:39:56.647 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.756672+0000", 2024-09-18T04:39:56.647 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 9.8508651569999994, 2024-09-18T04:39:56.647 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028688770000000001, 2024-09-18T04:39:56.647 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:56.648 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:56.648 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:56.648 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.648 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:56.648 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.756672+0000", 2024-09-18T04:39:56.648 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:56.648 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.648 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.648 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:56.648 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.756672+0000", 2024-09-18T04:39:56.648 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:56.648 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.648 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.648 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:56.648 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.756671+0000", 2024-09-18T04:39:56.649 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:56.649 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.649 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.649 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:56.649 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.757623+0000", 2024-09-18T04:39:56.649 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00095182199999999995 2024-09-18T04:39:56.649 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.649 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.649 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:56.649 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.757644+0000", 2024-09-18T04:39:56.649 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0996999999999999e-05 2024-09-18T04:39:56.649 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.649 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.649 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:56.650 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.757657+0000", 2024-09-18T04:39:56.650 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3264999999999999e-05 2024-09-18T04:39:56.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.650 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.650 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:56.650 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.757779+0000", 2024-09-18T04:39:56.650 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000121655 2024-09-18T04:39:56.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.650 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.650 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:56.650 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.757801+0000", 2024-09-18T04:39:56.650 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2340999999999998e-05 2024-09-18T04:39:56.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.650 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.650 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:56.651 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.759541+0000", 2024-09-18T04:39:56.651 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017398120000000001 2024-09-18T04:39:56.651 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:56.651 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:56.651 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:56.651 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.651 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.651 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.5 87/86 [PushOp(2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:7, version: 79'59, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:7@79'59, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:39:56.651 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.758140+0000", 2024-09-18T04:39:56.651 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 9.8493972519999993, 2024-09-18T04:39:56.651 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023362539999999998, 2024-09-18T04:39:56.651 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:56.651 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:56.651 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:56.651 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.652 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:56.652 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.758140+0000", 2024-09-18T04:39:56.652 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:56.652 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.652 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.652 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:56.652 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.758140+0000", 2024-09-18T04:39:56.652 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:56.652 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.652 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.652 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:56.652 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.758139+0000", 2024-09-18T04:39:56.652 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:56.653 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.653 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.653 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:56.653 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.758850+0000", 2024-09-18T04:39:56.653 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00071123099999999999 2024-09-18T04:39:56.653 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.653 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.653 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:56.653 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.758872+0000", 2024-09-18T04:39:56.653 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2195999999999999e-05 2024-09-18T04:39:56.653 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.653 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.653 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:56.654 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.758885+0000", 2024-09-18T04:39:56.654 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.327e-05 2024-09-18T04:39:56.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.654 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.654 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:56.654 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.758969+0000", 2024-09-18T04:39:56.654 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3497000000000004e-05 2024-09-18T04:39:56.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.654 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.654 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:56.654 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.758988+0000", 2024-09-18T04:39:56.654 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8765000000000001e-05 2024-09-18T04:39:56.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.654 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.654 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:56.654 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.760476+0000", 2024-09-18T04:39:56.655 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001488828 2024-09-18T04:39:56.655 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:56.655 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:56.655 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:56.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.655 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.655 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.5 87/86 [PushOp(2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:7, version: 79'59, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:7@79'59, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:39:56.655 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.762725+0000", 2024-09-18T04:39:56.655 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 9.8448121509999993, 2024-09-18T04:39:56.655 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028321890000000001, 2024-09-18T04:39:56.655 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:56.655 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:56.655 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:56.655 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.655 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:56.656 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.762725+0000", 2024-09-18T04:39:56.656 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:56.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.656 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.656 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:56.656 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.762725+0000", 2024-09-18T04:39:56.656 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:56.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.656 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.656 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:56.656 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.762724+0000", 2024-09-18T04:39:56.656 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:56.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.656 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.656 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:56.657 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.763395+0000", 2024-09-18T04:39:56.657 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00067064800000000003 2024-09-18T04:39:56.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.657 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.657 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:56.657 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.763416+0000", 2024-09-18T04:39:56.657 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0984000000000001e-05 2024-09-18T04:39:56.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.657 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.657 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:56.657 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.763442+0000", 2024-09-18T04:39:56.657 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6635000000000002e-05 2024-09-18T04:39:56.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.657 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.658 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:56.658 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.763584+0000", 2024-09-18T04:39:56.658 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000142037 2024-09-18T04:39:56.658 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.658 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.658 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:56.658 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.763610+0000", 2024-09-18T04:39:56.658 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6027999999999999e-05 2024-09-18T04:39:56.658 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.658 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.658 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:56.658 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.765557+0000", 2024-09-18T04:39:56.658 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0019470939999999999 2024-09-18T04:39:56.658 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:56.658 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:56.659 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:56.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.659 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.659 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:7, version: 79'64, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:7@79'64, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:39:56.659 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.763323+0000", 2024-09-18T04:39:56.659 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 9.8442139659999999, 2024-09-18T04:39:56.659 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003042215, 2024-09-18T04:39:56.659 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:56.659 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:56.659 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:56.659 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.659 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:56.659 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.763323+0000", 2024-09-18T04:39:56.659 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:56.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.660 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.660 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:56.660 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.763323+0000", 2024-09-18T04:39:56.660 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:56.660 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.660 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.660 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:56.660 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.763322+0000", 2024-09-18T04:39:56.660 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:56.660 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.660 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.660 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:56.660 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.764567+0000", 2024-09-18T04:39:56.660 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001244312 2024-09-18T04:39:56.660 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.661 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.661 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:56.661 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.764600+0000", 2024-09-18T04:39:56.661 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2787000000000003e-05 2024-09-18T04:39:56.661 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.661 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.661 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:56.661 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.764640+0000", 2024-09-18T04:39:56.661 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0825000000000003e-05 2024-09-18T04:39:56.661 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.661 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.661 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:56.661 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.764789+0000", 2024-09-18T04:39:56.661 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000148598 2024-09-18T04:39:56.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.662 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.662 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:56.662 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.764812+0000", 2024-09-18T04:39:56.662 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2702e-05 2024-09-18T04:39:56.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.662 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.662 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:56.662 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.766366+0000", 2024-09-18T04:39:56.662 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0015540110000000001 2024-09-18T04:39:56.662 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:56.662 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:56.662 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:56.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.662 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.663 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.5 87/86 [PushOp(2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:7, version: 79'59, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:7@79'59, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:39:56.663 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.768078+0000", 2024-09-18T04:39:56.663 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 9.8394591879999993, 2024-09-18T04:39:56.663 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028938509999999998, 2024-09-18T04:39:56.663 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:56.663 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:56.663 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:56.663 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.663 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:56.663 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.768078+0000", 2024-09-18T04:39:56.663 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:56.663 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.663 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.663 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:56.663 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.768078+0000", 2024-09-18T04:39:56.664 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:56.664 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.664 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.664 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:56.664 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.768076+0000", 2024-09-18T04:39:56.664 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:56.664 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.664 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.664 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:56.664 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.768951+0000", 2024-09-18T04:39:56.664 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000874455 2024-09-18T04:39:56.664 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.664 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.664 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:56.665 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.768972+0000", 2024-09-18T04:39:56.665 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.154e-05 2024-09-18T04:39:56.665 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.665 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.665 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:56.665 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.768992+0000", 2024-09-18T04:39:56.665 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9822000000000001e-05 2024-09-18T04:39:56.665 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.665 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.665 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:56.666 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.769110+0000", 2024-09-18T04:39:56.666 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000117762 2024-09-18T04:39:56.666 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.666 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.666 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:56.666 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.769123+0000", 2024-09-18T04:39:56.666 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2473999999999999e-05 2024-09-18T04:39:56.667 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.667 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.667 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:56.667 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.770972+0000", 2024-09-18T04:39:56.667 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001849613 2024-09-18T04:39:56.667 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:56.667 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:56.667 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:56.667 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.667 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.667 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:7, version: 79'64, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:7@79'64, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:39:56.668 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.769830+0000", 2024-09-18T04:39:56.668 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 9.837707924, 2024-09-18T04:39:56.668 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0040223419999999999, 2024-09-18T04:39:56.668 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:56.668 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:56.668 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:56.669 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.669 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:56.669 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.769830+0000", 2024-09-18T04:39:56.669 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:56.669 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.669 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.670 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:56.670 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.769830+0000", 2024-09-18T04:39:56.670 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:56.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.670 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.670 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:56.670 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.769829+0000", 2024-09-18T04:39:56.670 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:56.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.670 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.670 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:56.670 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.770812+0000", 2024-09-18T04:39:56.670 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00098347800000000009 2024-09-18T04:39:56.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.671 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.671 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:56.671 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.770834+0000", 2024-09-18T04:39:56.671 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1648999999999999e-05 2024-09-18T04:39:56.671 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.671 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.671 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:56.671 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.770847+0000", 2024-09-18T04:39:56.671 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3169999999999999e-05 2024-09-18T04:39:56.671 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.671 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.671 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:56.671 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.770969+0000", 2024-09-18T04:39:56.671 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000121721 2024-09-18T04:39:56.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.672 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.672 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:56.672 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.770989+0000", 2024-09-18T04:39:56.672 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9944000000000001e-05 2024-09-18T04:39:56.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.672 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.672 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:56.672 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.773852+0000", 2024-09-18T04:39:56.672 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028632319999999998 2024-09-18T04:39:56.672 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:56.673 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:56.673 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:56.673 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.673 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.673 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.5 87/86 [PushOp(2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head, version: 79'62, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head@79'62, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-18T04:39:56.673 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.772378+0000", 2024-09-18T04:39:56.673 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 9.8351598009999996, 2024-09-18T04:39:56.673 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002661535, 2024-09-18T04:39:56.673 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:56.673 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:56.673 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:56.673 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.673 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:56.673 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.772378+0000", 2024-09-18T04:39:56.674 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:56.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.674 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.674 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:56.674 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.772378+0000", 2024-09-18T04:39:56.674 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:56.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.674 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.674 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:56.674 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.772377+0000", 2024-09-18T04:39:56.674 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:56.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.674 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.674 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:56.674 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.772393+0000", 2024-09-18T04:39:56.675 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6495e-05 2024-09-18T04:39:56.675 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.675 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.675 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:56.675 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.772417+0000", 2024-09-18T04:39:56.675 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4304e-05 2024-09-18T04:39:56.675 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.675 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.675 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:56.675 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.772431+0000", 2024-09-18T04:39:56.675 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3165e-05 2024-09-18T04:39:56.675 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.675 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.675 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:56.675 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.772499+0000", 2024-09-18T04:39:56.675 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8627000000000005e-05 2024-09-18T04:39:56.676 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.676 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.676 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:56.676 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.772508+0000", 2024-09-18T04:39:56.676 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5609999999999999e-06 2024-09-18T04:39:56.676 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.676 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.676 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:56.676 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.775039+0000", 2024-09-18T04:39:56.676 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002531387 2024-09-18T04:39:56.676 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:56.676 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:56.676 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:56.676 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.676 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.677 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head, version: 79'65, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head@79'65, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-18T04:39:56.677 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.775934+0000", 2024-09-18T04:39:56.677 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 9.8316030479999998, 2024-09-18T04:39:56.677 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023666939999999999, 2024-09-18T04:39:56.677 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:56.677 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:56.677 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:56.677 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.677 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:56.677 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.775934+0000", 2024-09-18T04:39:56.677 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:56.677 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.677 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.678 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:56.678 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.775934+0000", 2024-09-18T04:39:56.678 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:56.678 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.678 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.678 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:56.678 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.775933+0000", 2024-09-18T04:39:56.678 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:56.678 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.678 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.678 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:56.678 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.775955+0000", 2024-09-18T04:39:56.678 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1512e-05 2024-09-18T04:39:56.678 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.678 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.678 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:56.679 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.775975+0000", 2024-09-18T04:39:56.679 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9653999999999999e-05 2024-09-18T04:39:56.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.679 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.679 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:56.679 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.775992+0000", 2024-09-18T04:39:56.679 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7326e-05 2024-09-18T04:39:56.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.679 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.679 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:56.679 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.776078+0000", 2024-09-18T04:39:56.679 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.6599000000000002e-05 2024-09-18T04:39:56.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.680 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.680 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:56.680 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.776088+0000", 2024-09-18T04:39:56.680 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.8120000000000002e-06 2024-09-18T04:39:56.680 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.680 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.680 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:56.680 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.778301+0000", 2024-09-18T04:39:56.680 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002212805 2024-09-18T04:39:56.680 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:56.680 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:56.680 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:56.680 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.680 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.681 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.5 87/86 [PushOp(2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:7, version: 79'61, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:7@79'61, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:39:56.681 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.781808+0000", 2024-09-18T04:39:56.681 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 9.8257298029999998, 2024-09-18T04:39:56.681 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027222169999999999, 2024-09-18T04:39:56.681 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:56.681 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:56.681 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:56.681 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.681 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:56.681 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.781808+0000", 2024-09-18T04:39:56.681 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:56.682 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.682 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.682 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:56.682 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.781808+0000", 2024-09-18T04:39:56.682 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:56.682 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.682 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.682 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:56.682 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.781807+0000", 2024-09-18T04:39:56.682 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:56.682 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.682 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.682 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:56.683 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.782242+0000", 2024-09-18T04:39:56.683 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00043528300000000001 2024-09-18T04:39:56.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.683 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.683 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:56.683 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.782265+0000", 2024-09-18T04:39:56.683 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3227e-05 2024-09-18T04:39:56.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.683 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.683 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:56.683 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.782283+0000", 2024-09-18T04:39:56.683 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7535999999999999e-05 2024-09-18T04:39:56.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.683 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.683 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:56.684 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.782418+0000", 2024-09-18T04:39:56.684 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000135298 2024-09-18T04:39:56.684 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.684 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.684 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:56.684 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.782444+0000", 2024-09-18T04:39:56.684 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6020000000000002e-05 2024-09-18T04:39:56.684 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.684 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.684 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:56.684 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.784530+0000", 2024-09-18T04:39:56.684 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020858890000000001 2024-09-18T04:39:56.684 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:56.684 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:56.685 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:56.685 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.685 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.685 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head, version: 81'75, data_included: [0~49152,1097728~999424], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head@81'75, size: 2146304, copy_subset: [0~2146304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-18T04:39:56.685 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.782052+0000", 2024-09-18T04:39:56.685 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 9.8254854389999995, 2024-09-18T04:39:56.685 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0037389630000000001, 2024-09-18T04:39:56.685 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:56.685 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:56.685 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:56.685 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.685 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:56.685 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.782052+0000", 2024-09-18T04:39:56.685 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:56.685 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.686 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.686 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:56.686 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.782052+0000", 2024-09-18T04:39:56.686 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:56.686 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.686 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.686 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:56.686 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.782051+0000", 2024-09-18T04:39:56.686 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:56.686 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.686 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.686 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:56.686 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.783658+0000", 2024-09-18T04:39:56.686 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0016072149999999999 2024-09-18T04:39:56.686 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.687 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.687 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:56.687 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.783698+0000", 2024-09-18T04:39:56.687 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9394000000000003e-05 2024-09-18T04:39:56.687 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.687 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.687 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:56.687 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.783733+0000", 2024-09-18T04:39:56.687 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5428e-05 2024-09-18T04:39:56.687 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.687 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.687 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:56.687 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.783854+0000", 2024-09-18T04:39:56.687 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012067599999999999 2024-09-18T04:39:56.687 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.687 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.688 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:56.688 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.783869+0000", 2024-09-18T04:39:56.688 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5554999999999999e-05 2024-09-18T04:39:56.688 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.688 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.688 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:56.688 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.785791+0000", 2024-09-18T04:39:56.688 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001921629 2024-09-18T04:39:56.688 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:56.688 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:56.689 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:56.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.689 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.689 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head, version: 81'75, data_included: [2097152~49152], data_size: 49152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head@81'75, size: 2146304, copy_subset: [0~2146304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2146304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:39:56.689 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.787740+0000", 2024-09-18T04:39:56.689 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 9.8197979249999996, 2024-09-18T04:39:56.690 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030127370000000001, 2024-09-18T04:39:56.690 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:56.690 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:56.690 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:56.690 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.690 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:56.691 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.787740+0000", 2024-09-18T04:39:56.691 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:56.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.691 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.691 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:56.691 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.787740+0000", 2024-09-18T04:39:56.691 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:56.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.691 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.691 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:56.691 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.787738+0000", 2024-09-18T04:39:56.691 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:56.692 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.692 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.692 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:56.692 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.787795+0000", 2024-09-18T04:39:56.692 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6753999999999999e-05 2024-09-18T04:39:56.692 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.692 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.692 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:56.692 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.787826+0000", 2024-09-18T04:39:56.692 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0617000000000002e-05 2024-09-18T04:39:56.692 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.692 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.692 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:56.693 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.787853+0000", 2024-09-18T04:39:56.693 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7538999999999998e-05 2024-09-18T04:39:56.693 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.693 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.693 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:56.693 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.787963+0000", 2024-09-18T04:39:56.693 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000109387 2024-09-18T04:39:56.693 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.693 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.693 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:56.693 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.787973+0000", 2024-09-18T04:39:56.693 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0781999999999999e-05 2024-09-18T04:39:56.693 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.693 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.694 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:56.694 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.790752+0000", 2024-09-18T04:39:56.694 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027788520000000001 2024-09-18T04:39:56.694 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:56.694 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:56.694 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:56.694 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.694 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.695 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.5 87/86 [PushOp(2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:7, version: 79'61, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:7@79'61, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:39:56.695 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.791668+0000", 2024-09-18T04:39:56.695 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 9.8158692320000007, 2024-09-18T04:39:56.695 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025634299999999998, 2024-09-18T04:39:56.695 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:56.695 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:56.695 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:56.695 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.695 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:56.695 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.791668+0000", 2024-09-18T04:39:56.696 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:56.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.696 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.696 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:56.696 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.791668+0000", 2024-09-18T04:39:56.696 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:56.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.696 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.696 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:56.696 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.791667+0000", 2024-09-18T04:39:56.696 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:56.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.696 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.696 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:56.697 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.792071+0000", 2024-09-18T04:39:56.697 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00040348599999999999 2024-09-18T04:39:56.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.697 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.697 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:56.697 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.792094+0000", 2024-09-18T04:39:56.697 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3105e-05 2024-09-18T04:39:56.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.697 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.697 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:56.697 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.792112+0000", 2024-09-18T04:39:56.697 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8502999999999999e-05 2024-09-18T04:39:56.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.697 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.697 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:56.698 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.792187+0000", 2024-09-18T04:39:56.698 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5074000000000007e-05 2024-09-18T04:39:56.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.698 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.698 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:56.698 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.792197+0000", 2024-09-18T04:39:56.698 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.2979999999999997e-06 2024-09-18T04:39:56.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.698 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.698 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:56.698 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.794232+0000", 2024-09-18T04:39:56.698 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002035149 2024-09-18T04:39:56.698 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:56.698 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:56.698 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:56.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.699 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.699 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.5 87/86 [PushOp(2:b227e6b1:::rbd_data.125fc09df6d1.000000000000000e:7, version: 79'63, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:b227e6b1:::rbd_data.125fc09df6d1.000000000000000e:7@79'63, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:39:56.699 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.813128+0000", 2024-09-18T04:39:56.699 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 9.7944093710000004, 2024-09-18T04:39:56.699 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025359520000000002, 2024-09-18T04:39:56.699 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:56.699 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:39:56.699 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:56.699 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.699 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:56.699 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.813128+0000", 2024-09-18T04:39:56.699 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:56.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.700 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.700 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:56.700 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.813128+0000", 2024-09-18T04:39:56.700 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:56.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.700 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.700 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:56.700 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.813127+0000", 2024-09-18T04:39:56.700 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:56.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.700 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.700 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:56.701 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.813544+0000", 2024-09-18T04:39:56.701 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000417248 2024-09-18T04:39:56.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.701 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.701 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:56.701 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.813571+0000", 2024-09-18T04:39:56.701 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6789000000000001e-05 2024-09-18T04:39:56.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.701 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.701 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:56.701 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.813589+0000", 2024-09-18T04:39:56.701 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8247e-05 2024-09-18T04:39:56.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.701 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.702 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:56.702 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.813698+0000", 2024-09-18T04:39:56.702 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000108937 2024-09-18T04:39:56.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.702 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.702 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:56.702 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.813715+0000", 2024-09-18T04:39:56.702 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7110000000000001e-05 2024-09-18T04:39:56.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:56.702 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:56.702 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:56.702 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.815664+0000", 2024-09-18T04:39:56.702 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001948946 2024-09-18T04:39:56.702 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:56.702 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:56.703 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:56.703 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:56.703 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:56.703 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:56.703 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T04:39:56.705 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:39:56.705 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:56.696+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:56.806 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:56.799+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:56.813 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:56.813 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:56.813 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:39:56.813 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:56.824 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:39:56.826 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-18T04:39:56.848 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:39:56.868 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:39:56.888 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:39:56.906 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:56.904+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:56.911 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:39:56.924 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:56.924 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:56.934 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:39:56.935 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:39:56.975 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:39:56.976 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:39:56.977 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:39:56.977 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:39:56.977 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:39:56.977 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:39:56.977 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:39:57.007 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:57.004+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:57.043 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:57.043 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:57.051 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 2] out_osds: [4] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 2] 2024-09-18T04:39:57.051 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T04:39:57.052 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-18T04:39:57.052 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-18T04:39:57.053 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:39:57.108 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:57.103+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:57.172 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:57.172 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:57.182 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:39:57.209 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:57.204+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:57.310 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:57.307+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:57.331 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:57.331 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:57.341 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:39:57.346 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:39:57.346 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:57.347 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 90, 2024-09-18T04:39:57.347 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "be6146ba-a991-4f6e-b359-99cbedc9f9b1", 2024-09-18T04:39:57.347 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-09-18T04:37:15.676634+0000", 2024-09-18T04:39:57.347 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-09-18T04:39:54.747372+0000", 2024-09-18T04:39:57.347 INFO:teuthology.orchestra.run.smithi028.stdout: "last_up_change": "2024-09-18T04:39:45.699693+0000", 2024-09-18T04:39:57.347 INFO:teuthology.orchestra.run.smithi028.stdout: "last_in_change": "2024-09-18T04:39:17.550632+0000", 2024-09-18T04:39:57.347 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-18T04:39:57.347 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_num": 5799936, 2024-09-18T04:39:57.347 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_set": [ 2024-09-18T04:39:57.347 INFO:teuthology.orchestra.run.smithi028.stdout: "pglog_hardlimit", 2024-09-18T04:39:57.347 INFO:teuthology.orchestra.run.smithi028.stdout: "purged_snapdirs", 2024-09-18T04:39:57.348 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_deletes", 2024-09-18T04:39:57.348 INFO:teuthology.orchestra.run.smithi028.stdout: "sortbitwise" 2024-09-18T04:39:57.348 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-18T04:39:57.348 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_version": 6, 2024-09-18T04:39:57.348 INFO:teuthology.orchestra.run.smithi028.stdout: "full_ratio": 0.94999998807907104, 2024-09-18T04:39:57.348 INFO:teuthology.orchestra.run.smithi028.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-18T04:39:57.348 INFO:teuthology.orchestra.run.smithi028.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-18T04:39:57.348 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_snapshot": "", 2024-09-18T04:39:57.348 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_max": 3, 2024-09-18T04:39:57.348 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd": 6, 2024-09-18T04:39:57.348 INFO:teuthology.orchestra.run.smithi028.stdout: "require_min_compat_client": "luminous", 2024-09-18T04:39:57.349 INFO:teuthology.orchestra.run.smithi028.stdout: "min_compat_client": "luminous", 2024-09-18T04:39:57.349 INFO:teuthology.orchestra.run.smithi028.stdout: "require_osd_release": "reef", 2024-09-18T04:39:57.349 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_crimson": false, 2024-09-18T04:39:57.349 INFO:teuthology.orchestra.run.smithi028.stdout: "pools": [ 2024-09-18T04:39:57.349 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.349 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 1, 2024-09-18T04:39:57.349 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": ".mgr", 2024-09-18T04:39:57.349 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-18T04:37:25.654950+0000", 2024-09-18T04:39:57.349 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-18T04:39:57.350 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-09-18T04:39:57.350 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-18T04:39:57.350 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 2, 2024-09-18T04:39:57.350 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 1, 2024-09-18T04:39:57.350 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-18T04:39:57.350 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-18T04:39:57.350 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-18T04:39:57.350 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T04:39:57.350 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T04:39:57.350 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-18T04:39:57.350 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-18T04:39:57.351 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 1, 2024-09-18T04:39:57.351 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 1, 2024-09-18T04:39:57.351 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 1, 2024-09-18T04:39:57.351 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 1, 2024-09-18T04:39:57.351 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 1, 2024-09-18T04:39:57.351 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-18T04:39:57.351 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-18T04:39:57.351 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-18T04:39:57.351 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-18T04:39:57.352 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-18T04:39:57.352 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-18T04:39:57.352 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-18T04:39:57.352 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.352 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "16", 2024-09-18T04:39:57.352 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-18T04:39:57.352 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T04:39:57.352 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T04:39:57.352 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-18T04:39:57.352 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-18T04:39:57.353 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-09-18T04:39:57.353 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-09-18T04:39:57.353 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-18T04:39:57.353 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-18T04:39:57.353 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-18T04:39:57.353 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-18T04:39:57.353 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-18T04:39:57.353 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-18T04:39:57.353 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-18T04:39:57.354 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-18T04:39:57.354 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-18T04:39:57.354 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-18T04:39:57.355 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-18T04:39:57.355 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T04:39:57.355 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T04:39:57.355 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T04:39:57.355 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-18T04:39:57.355 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-18T04:39:57.355 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-18T04:39:57.355 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-18T04:39:57.355 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-18T04:39:57.355 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.355 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-18T04:39:57.355 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-18T04:39:57.355 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-18T04:39:57.356 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-18T04:39:57.356 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-18T04:39:57.356 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T04:39:57.356 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-18T04:39:57.356 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-18T04:39:57.356 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-18T04:39:57.356 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-18T04:39:57.356 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-18T04:39:57.356 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-09-18T04:39:57.356 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_max": 32, 2024-09-18T04:39:57.356 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_min": 1 2024-09-18T04:39:57.356 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.357 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-18T04:39:57.357 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": {} 2024-09-18T04:39:57.357 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.357 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-09-18T04:39:57.357 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 6.059999942779541, 2024-09-18T04:39:57.357 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 6.059999942779541, 2024-09-18T04:39:57.357 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.33000001311302185, 2024-09-18T04:39:57.357 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2, 2024-09-18T04:39:57.357 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2, 2024-09-18T04:39:57.357 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 1, 2024-09-18T04:39:57.357 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 1, 2024-09-18T04:39:57.357 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 1 2024-09-18T04:39:57.358 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:57.358 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.358 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.358 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 2, 2024-09-18T04:39:57.358 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "rbd", 2024-09-18T04:39:57.358 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-18T04:37:32.672462+0000", 2024-09-18T04:39:57.358 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 8193, 2024-09-18T04:39:57.358 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-18T04:39:57.358 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-18T04:39:57.358 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 2, 2024-09-18T04:39:57.358 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 1, 2024-09-18T04:39:57.359 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-18T04:39:57.359 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-18T04:39:57.359 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-18T04:39:57.359 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T04:39:57.359 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T04:39:57.359 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-18T04:39:57.359 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-18T04:39:57.359 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 8, 2024-09-18T04:39:57.359 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 8, 2024-09-18T04:39:57.359 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 8, 2024-09-18T04:39:57.359 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 8, 2024-09-18T04:39:57.359 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 8, 2024-09-18T04:39:57.359 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-18T04:39:57.360 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-18T04:39:57.360 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-18T04:39:57.360 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-18T04:39:57.360 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-18T04:39:57.360 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-18T04:39:57.360 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-18T04:39:57.360 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.360 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "80", 2024-09-18T04:39:57.360 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-18T04:39:57.360 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T04:39:57.360 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T04:39:57.360 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-18T04:39:57.360 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-18T04:39:57.361 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 8, 2024-09-18T04:39:57.361 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 80, 2024-09-18T04:39:57.361 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-18T04:39:57.361 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-18T04:39:57.361 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-18T04:39:57.361 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-18T04:39:57.361 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-18T04:39:57.361 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-18T04:39:57.361 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-18T04:39:57.361 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-18T04:39:57.361 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-18T04:39:57.361 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-18T04:39:57.361 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-18T04:39:57.362 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T04:39:57.362 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T04:39:57.362 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T04:39:57.362 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-18T04:39:57.362 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-18T04:39:57.362 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-18T04:39:57.362 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-18T04:39:57.362 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-18T04:39:57.362 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.362 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-18T04:39:57.362 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-18T04:39:57.362 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-18T04:39:57.363 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-18T04:39:57.363 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-18T04:39:57.363 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T04:39:57.363 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-18T04:39:57.363 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-18T04:39:57.363 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-18T04:39:57.363 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-18T04:39:57.363 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-18T04:39:57.363 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-09-18T04:39:57.363 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-18T04:39:57.363 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd": {} 2024-09-18T04:39:57.363 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.364 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-09-18T04:39:57.364 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2.2699999809265137, 2024-09-18T04:39:57.364 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2.2699999809265137, 2024-09-18T04:39:57.364 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.82999998331069946, 2024-09-18T04:39:57.364 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.8799999952316284, 2024-09-18T04:39:57.364 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.8799999952316284, 2024-09-18T04:39:57.364 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 1, 2024-09-18T04:39:57.364 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 1, 2024-09-18T04:39:57.364 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 1 2024-09-18T04:39:57.364 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:57.364 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.364 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.365 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 3, 2024-09-18T04:39:57.365 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "new_default_pool17342", 2024-09-18T04:39:57.365 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-09-18T04:38:42.986132+0000", 2024-09-18T04:39:57.365 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 8193, 2024-09-18T04:39:57.365 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-18T04:39:57.365 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-09-18T04:39:57.365 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 2, 2024-09-18T04:39:57.365 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 1, 2024-09-18T04:39:57.365 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-09-18T04:39:57.365 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-09-18T04:39:57.365 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-09-18T04:39:57.365 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T04:39:57.366 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T04:39:57.366 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-09-18T04:39:57.366 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-09-18T04:39:57.366 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 8, 2024-09-18T04:39:57.366 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 8, 2024-09-18T04:39:57.366 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 8, 2024-09-18T04:39:57.366 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 8, 2024-09-18T04:39:57.366 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 8, 2024-09-18T04:39:57.366 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-09-18T04:39:57.366 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-09-18T04:39:57.366 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-09-18T04:39:57.366 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-09-18T04:39:57.366 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-09-18T04:39:57.367 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-09-18T04:39:57.367 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-09-18T04:39:57.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.367 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "55", 2024-09-18T04:39:57.367 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-09-18T04:39:57.367 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T04:39:57.367 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T04:39:57.367 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-09-18T04:39:57.367 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-09-18T04:39:57.367 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 3, 2024-09-18T04:39:57.367 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 55, 2024-09-18T04:39:57.367 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-09-18T04:39:57.368 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-09-18T04:39:57.368 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-09-18T04:39:57.368 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-09-18T04:39:57.368 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-09-18T04:39:57.368 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-09-18T04:39:57.368 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-09-18T04:39:57.368 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-09-18T04:39:57.368 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-09-18T04:39:57.368 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-09-18T04:39:57.368 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-09-18T04:39:57.368 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T04:39:57.368 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T04:39:57.369 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T04:39:57.369 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-09-18T04:39:57.369 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-09-18T04:39:57.369 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-09-18T04:39:57.369 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-09-18T04:39:57.369 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-09-18T04:39:57.369 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.369 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-09-18T04:39:57.369 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-09-18T04:39:57.369 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-09-18T04:39:57.369 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-09-18T04:39:57.369 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-09-18T04:39:57.369 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T04:39:57.369 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-09-18T04:39:57.370 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-09-18T04:39:57.370 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-09-18T04:39:57.370 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-09-18T04:39:57.370 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-09-18T04:39:57.370 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-09-18T04:39:57.370 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-09-18T04:39:57.370 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd": {} 2024-09-18T04:39:57.370 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.370 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-09-18T04:39:57.370 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2.2699999809265137, 2024-09-18T04:39:57.370 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2.2699999809265137, 2024-09-18T04:39:57.370 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.82999998331069946, 2024-09-18T04:39:57.370 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.8799999952316284, 2024-09-18T04:39:57.370 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.8799999952316284, 2024-09-18T04:39:57.370 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 1, 2024-09-18T04:39:57.371 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 1, 2024-09-18T04:39:57.371 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 1 2024-09-18T04:39:57.371 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:57.371 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:57.371 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-18T04:39:57.371 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-09-18T04:39:57.371 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.371 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-09-18T04:39:57.371 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "6177002d-72e6-4888-bcde-29b6a92d9dc4", 2024-09-18T04:39:57.371 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-18T04:39:57.371 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-18T04:39:57.371 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-18T04:39:57.372 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-18T04:39:57.372 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-18T04:39:57.372 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-18T04:39:57.372 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 13, 2024-09-18T04:39:57.372 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 86, 2024-09-18T04:39:57.372 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-18T04:39:57.372 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-18T04:39:57.372 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-18T04:39:57.372 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:39:57.372 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.372 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:39:57.372 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6801", 2024-09-18T04:39:57.372 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1554407851 2024-09-18T04:39:57.372 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.373 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.373 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:39:57.373 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6803", 2024-09-18T04:39:57.373 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1554407851 2024-09-18T04:39:57.373 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:57.373 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:57.373 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.373 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-18T04:39:57.373 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:39:57.373 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.373 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:39:57.373 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6805", 2024-09-18T04:39:57.373 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1554407851 2024-09-18T04:39:57.374 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.374 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.374 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:39:57.374 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6807", 2024-09-18T04:39:57.374 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1554407851 2024-09-18T04:39:57.374 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:57.374 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:57.374 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.374 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-18T04:39:57.374 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:39:57.374 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.374 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:39:57.375 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6813", 2024-09-18T04:39:57.375 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1554407851 2024-09-18T04:39:57.375 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.375 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.375 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:39:57.375 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6815", 2024-09-18T04:39:57.375 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1554407851 2024-09-18T04:39:57.375 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:57.375 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:57.375 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.375 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-18T04:39:57.375 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:39:57.375 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.375 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:39:57.376 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6809", 2024-09-18T04:39:57.376 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1554407851 2024-09-18T04:39:57.376 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.376 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.376 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:39:57.376 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6811", 2024-09-18T04:39:57.376 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1554407851 2024-09-18T04:39:57.376 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:57.376 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:57.376 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.376 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6803/1554407851", 2024-09-18T04:39:57.376 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6807/1554407851", 2024-09-18T04:39:57.376 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6815/1554407851", 2024-09-18T04:39:57.377 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6811/1554407851", 2024-09-18T04:39:57.377 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-18T04:39:57.377 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-18T04:39:57.377 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-18T04:39:57.377 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:57.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.377 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.377 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-09-18T04:39:57.377 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "4f62cb09-3457-418c-bb69-491d9b70914c", 2024-09-18T04:39:57.377 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-18T04:39:57.377 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-18T04:39:57.377 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-18T04:39:57.377 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-18T04:39:57.377 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-18T04:39:57.378 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-18T04:39:57.378 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 12, 2024-09-18T04:39:57.378 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 86, 2024-09-18T04:39:57.378 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-18T04:39:57.378 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-18T04:39:57.378 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-18T04:39:57.378 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:39:57.378 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.378 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:39:57.378 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6800", 2024-09-18T04:39:57.378 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 511141401 2024-09-18T04:39:57.378 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.378 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.378 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:39:57.379 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6802", 2024-09-18T04:39:57.379 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 511141401 2024-09-18T04:39:57.379 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:57.379 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:57.379 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.379 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-18T04:39:57.379 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:39:57.379 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.379 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:39:57.379 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6804", 2024-09-18T04:39:57.379 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 511141401 2024-09-18T04:39:57.379 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.379 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.379 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:39:57.380 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6806", 2024-09-18T04:39:57.380 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 511141401 2024-09-18T04:39:57.380 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:57.380 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:57.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.380 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-18T04:39:57.380 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:39:57.380 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.380 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:39:57.380 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6812", 2024-09-18T04:39:57.380 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 511141401 2024-09-18T04:39:57.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.380 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.380 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:39:57.381 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6814", 2024-09-18T04:39:57.381 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 511141401 2024-09-18T04:39:57.381 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:57.381 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:57.381 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.381 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-18T04:39:57.381 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:39:57.381 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.381 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:39:57.381 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6808", 2024-09-18T04:39:57.381 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 511141401 2024-09-18T04:39:57.381 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.381 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.381 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:39:57.381 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6810", 2024-09-18T04:39:57.382 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 511141401 2024-09-18T04:39:57.382 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:57.382 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:57.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.382 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6802/511141401", 2024-09-18T04:39:57.382 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6806/511141401", 2024-09-18T04:39:57.382 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6814/511141401", 2024-09-18T04:39:57.382 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6810/511141401", 2024-09-18T04:39:57.382 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-18T04:39:57.382 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-18T04:39:57.382 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-18T04:39:57.382 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:57.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.383 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.383 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-09-18T04:39:57.383 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "854fe031-3b87-4290-9c2f-3fb196fb806c", 2024-09-18T04:39:57.383 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-18T04:39:57.383 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-18T04:39:57.383 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-18T04:39:57.383 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-18T04:39:57.383 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 12, 2024-09-18T04:39:57.383 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 76, 2024-09-18T04:39:57.383 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 86, 2024-09-18T04:39:57.383 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 86, 2024-09-18T04:39:57.383 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 77, 2024-09-18T04:39:57.383 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-18T04:39:57.384 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-18T04:39:57.384 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:39:57.384 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.384 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:39:57.384 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6816", 2024-09-18T04:39:57.384 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3849236161 2024-09-18T04:39:57.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.384 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.384 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:39:57.384 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6817", 2024-09-18T04:39:57.384 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3849236161 2024-09-18T04:39:57.384 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:57.384 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:57.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.385 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-18T04:39:57.385 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:39:57.385 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.385 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:39:57.385 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6818", 2024-09-18T04:39:57.385 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3849236161 2024-09-18T04:39:57.385 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.385 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.385 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:39:57.385 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6819", 2024-09-18T04:39:57.385 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3849236161 2024-09-18T04:39:57.385 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:57.385 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:57.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.386 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-18T04:39:57.386 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:39:57.386 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.386 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:39:57.386 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6822", 2024-09-18T04:39:57.386 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3849236161 2024-09-18T04:39:57.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.386 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.386 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:39:57.386 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6823", 2024-09-18T04:39:57.386 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3849236161 2024-09-18T04:39:57.386 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:57.386 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:57.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.387 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-18T04:39:57.387 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:39:57.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.387 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:39:57.387 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6820", 2024-09-18T04:39:57.387 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3849236161 2024-09-18T04:39:57.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.387 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:39:57.387 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6821", 2024-09-18T04:39:57.387 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3849236161 2024-09-18T04:39:57.387 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:57.388 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:57.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.388 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6817/3849236161", 2024-09-18T04:39:57.388 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6819/3849236161", 2024-09-18T04:39:57.388 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6823/3849236161", 2024-09-18T04:39:57.388 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6821/3849236161", 2024-09-18T04:39:57.388 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-18T04:39:57.388 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-18T04:39:57.388 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-18T04:39:57.388 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:57.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.388 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.388 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-09-18T04:39:57.388 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "c2625f98-780b-4f86-bf36-a48d2a227756", 2024-09-18T04:39:57.389 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-18T04:39:57.389 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-18T04:39:57.389 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-18T04:39:57.389 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-18T04:39:57.389 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-18T04:39:57.389 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-18T04:39:57.389 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 12, 2024-09-18T04:39:57.389 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 32, 2024-09-18T04:39:57.389 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-18T04:39:57.389 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-18T04:39:57.389 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-18T04:39:57.389 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:39:57.389 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.389 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:39:57.390 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6808", 2024-09-18T04:39:57.390 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2746051132 2024-09-18T04:39:57.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.390 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.390 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:39:57.390 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6809", 2024-09-18T04:39:57.390 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2746051132 2024-09-18T04:39:57.390 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:57.390 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:57.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.390 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-18T04:39:57.390 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:39:57.390 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.391 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:39:57.391 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6810", 2024-09-18T04:39:57.391 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2746051132 2024-09-18T04:39:57.391 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.391 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.391 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:39:57.391 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6811", 2024-09-18T04:39:57.391 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2746051132 2024-09-18T04:39:57.391 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:57.391 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:57.391 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.391 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-18T04:39:57.391 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:39:57.392 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.392 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:39:57.392 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6814", 2024-09-18T04:39:57.392 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2746051132 2024-09-18T04:39:57.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.392 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.392 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:39:57.392 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6815", 2024-09-18T04:39:57.392 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2746051132 2024-09-18T04:39:57.392 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:57.392 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:57.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.393 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-18T04:39:57.393 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:39:57.393 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.393 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:39:57.393 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6812", 2024-09-18T04:39:57.393 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2746051132 2024-09-18T04:39:57.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.393 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.393 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:39:57.393 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6813", 2024-09-18T04:39:57.393 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2746051132 2024-09-18T04:39:57.393 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:57.393 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:57.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.394 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.82:6809/2746051132", 2024-09-18T04:39:57.394 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.82:6811/2746051132", 2024-09-18T04:39:57.394 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.82:6815/2746051132", 2024-09-18T04:39:57.394 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.82:6813/2746051132", 2024-09-18T04:39:57.394 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-18T04:39:57.394 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-18T04:39:57.394 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-18T04:39:57.394 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:57.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.394 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.394 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-09-18T04:39:57.395 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "deb22d86-4081-484d-96eb-9cb9d96b60bc", 2024-09-18T04:39:57.395 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-18T04:39:57.395 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 0, 2024-09-18T04:39:57.395 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-18T04:39:57.395 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-18T04:39:57.395 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-18T04:39:57.395 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-18T04:39:57.395 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 13, 2024-09-18T04:39:57.395 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 52, 2024-09-18T04:39:57.395 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-18T04:39:57.395 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-18T04:39:57.395 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-18T04:39:57.395 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:39:57.396 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.396 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:39:57.396 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6800", 2024-09-18T04:39:57.396 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2005712460 2024-09-18T04:39:57.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.396 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.396 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:39:57.396 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6801", 2024-09-18T04:39:57.396 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2005712460 2024-09-18T04:39:57.396 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:57.396 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:57.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.397 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-18T04:39:57.397 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:39:57.397 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.397 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:39:57.397 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6802", 2024-09-18T04:39:57.397 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2005712460 2024-09-18T04:39:57.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.397 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.397 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:39:57.397 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6803", 2024-09-18T04:39:57.397 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2005712460 2024-09-18T04:39:57.397 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:57.397 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:57.398 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.398 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-18T04:39:57.398 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:39:57.398 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.398 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:39:57.398 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6806", 2024-09-18T04:39:57.398 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2005712460 2024-09-18T04:39:57.398 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.398 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.398 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:39:57.398 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6807", 2024-09-18T04:39:57.398 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2005712460 2024-09-18T04:39:57.398 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:57.398 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:57.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.399 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-18T04:39:57.399 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:39:57.399 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.399 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:39:57.399 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6804", 2024-09-18T04:39:57.399 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2005712460 2024-09-18T04:39:57.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.399 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.399 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:39:57.399 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6805", 2024-09-18T04:39:57.399 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2005712460 2024-09-18T04:39:57.399 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:57.399 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:57.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.400 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.82:6801/2005712460", 2024-09-18T04:39:57.400 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.82:6803/2005712460", 2024-09-18T04:39:57.400 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.82:6807/2005712460", 2024-09-18T04:39:57.400 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.82:6805/2005712460", 2024-09-18T04:39:57.400 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-18T04:39:57.400 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-18T04:39:57.400 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-18T04:39:57.400 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:57.400 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.400 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.400 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-09-18T04:39:57.400 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "061d332c-f51b-403d-bb18-eba415c97e7f", 2024-09-18T04:39:57.401 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-09-18T04:39:57.401 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-09-18T04:39:57.401 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-09-18T04:39:57.401 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-09-18T04:39:57.401 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-09-18T04:39:57.401 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-09-18T04:39:57.401 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 13, 2024-09-18T04:39:57.401 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 86, 2024-09-18T04:39:57.401 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-09-18T04:39:57.401 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-09-18T04:39:57.401 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-18T04:39:57.401 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:39:57.401 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.401 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:39:57.402 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6816", 2024-09-18T04:39:57.402 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1267644232 2024-09-18T04:39:57.402 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.402 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.402 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:39:57.402 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6817", 2024-09-18T04:39:57.402 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1267644232 2024-09-18T04:39:57.402 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:57.402 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:57.402 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.402 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-09-18T04:39:57.402 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:39:57.402 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.402 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:39:57.402 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6818", 2024-09-18T04:39:57.403 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1267644232 2024-09-18T04:39:57.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.403 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.403 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:39:57.403 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6819", 2024-09-18T04:39:57.403 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1267644232 2024-09-18T04:39:57.403 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:57.403 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:57.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.403 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-09-18T04:39:57.403 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:39:57.403 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.404 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:39:57.404 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6822", 2024-09-18T04:39:57.404 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1267644232 2024-09-18T04:39:57.404 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.404 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.404 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:39:57.404 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6823", 2024-09-18T04:39:57.404 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1267644232 2024-09-18T04:39:57.404 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:57.404 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:57.404 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.404 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-09-18T04:39:57.404 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-18T04:39:57.405 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.405 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-18T04:39:57.405 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6820", 2024-09-18T04:39:57.405 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1267644232 2024-09-18T04:39:57.405 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.405 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.405 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-18T04:39:57.405 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.82:6821", 2024-09-18T04:39:57.405 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1267644232 2024-09-18T04:39:57.405 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:57.405 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:57.405 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.405 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.82:6817/1267644232", 2024-09-18T04:39:57.405 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.82:6819/1267644232", 2024-09-18T04:39:57.406 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.82:6823/1267644232", 2024-09-18T04:39:57.406 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.82:6821/1267644232", 2024-09-18T04:39:57.406 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-09-18T04:39:57.406 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-09-18T04:39:57.406 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-09-18T04:39:57.406 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:57.406 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:57.406 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-18T04:39:57.406 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_xinfo": [ 2024-09-18T04:39:57.406 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.406 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-09-18T04:39:57.406 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-18T04:39:57.406 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-18T04:39:57.406 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-18T04:39:57.407 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138322906710015, 2024-09-18T04:39:57.407 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-18T04:39:57.407 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-18T04:37:22.192087+0000", 2024-09-18T04:39:57.407 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-18T04:39:57.407 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.407 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.407 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-09-18T04:39:57.407 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-18T04:39:57.407 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-18T04:39:57.407 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-18T04:39:57.407 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138322906710015, 2024-09-18T04:39:57.407 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-18T04:39:57.407 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-18T04:37:22.017833+0000", 2024-09-18T04:39:57.408 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-18T04:39:57.408 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.408 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.408 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-09-18T04:39:57.408 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-18T04:39:57.408 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-18T04:39:57.408 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-18T04:39:57.408 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138322906710015, 2024-09-18T04:39:57.408 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-18T04:39:57.408 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-18T04:37:22.755774+0000", 2024-09-18T04:39:57.408 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 77 2024-09-18T04:39:57.408 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.408 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.409 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-09-18T04:39:57.409 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-18T04:39:57.409 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-18T04:39:57.409 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-18T04:39:57.409 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138322906710015, 2024-09-18T04:39:57.409 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-18T04:39:57.409 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-18T04:37:22.663785+0000", 2024-09-18T04:39:57.409 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-18T04:39:57.409 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.409 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.409 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-09-18T04:39:57.409 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-18T04:39:57.409 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-18T04:39:57.409 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-18T04:39:57.410 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138322906710015, 2024-09-18T04:39:57.410 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 65536, 2024-09-18T04:39:57.410 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-18T04:37:22.531758+0000", 2024-09-18T04:39:57.410 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-18T04:39:57.410 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.410 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.410 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-09-18T04:39:57.410 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-09-18T04:39:57.410 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-09-18T04:39:57.410 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-09-18T04:39:57.410 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540138322906710015, 2024-09-18T04:39:57.410 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-09-18T04:39:57.410 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-09-18T04:37:22.400774+0000", 2024-09-18T04:39:57.411 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-09-18T04:39:57.411 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:57.411 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-18T04:39:57.411 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap": [ 2024-09-18T04:39:57.411 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.411 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "2.5", 2024-09-18T04:39:57.411 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-09-18T04:39:57.411 INFO:teuthology.orchestra.run.smithi028.stdout: 2, 2024-09-18T04:39:57.411 INFO:teuthology.orchestra.run.smithi028.stdout: 0 2024-09-18T04:39:57.411 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:57.411 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.411 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.411 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "2.7", 2024-09-18T04:39:57.411 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-09-18T04:39:57.412 INFO:teuthology.orchestra.run.smithi028.stdout: 3, 2024-09-18T04:39:57.412 INFO:teuthology.orchestra.run.smithi028.stdout: 5 2024-09-18T04:39:57.412 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:57.412 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:57.412 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-18T04:39:57.412 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap_items": [ 2024-09-18T04:39:57.412 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.412 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "2.1", 2024-09-18T04:39:57.412 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-09-18T04:39:57.412 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.412 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 2, 2024-09-18T04:39:57.412 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 1 2024-09-18T04:39:57.412 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:57.412 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:57.413 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.413 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.413 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "2.4", 2024-09-18T04:39:57.413 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-09-18T04:39:57.413 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:57.413 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-09-18T04:39:57.413 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 5 2024-09-18T04:39:57.413 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:57.413 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:57.413 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:57.413 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-18T04:39:57.413 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap_primaries": [], 2024-09-18T04:39:57.413 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_temp": [], 2024-09-18T04:39:57.413 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_temp": [], 2024-09-18T04:39:57.413 INFO:teuthology.orchestra.run.smithi028.stdout: "blocklist": {}, 2024-09-18T04:39:57.414 INFO:teuthology.orchestra.run.smithi028.stdout: "range_blocklist": {}, 2024-09-18T04:39:57.414 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profiles": { 2024-09-18T04:39:57.414 INFO:teuthology.orchestra.run.smithi028.stdout: "default": { 2024-09-18T04:39:57.414 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-failure-domain": "osd", 2024-09-18T04:39:57.414 INFO:teuthology.orchestra.run.smithi028.stdout: "k": "2", 2024-09-18T04:39:57.414 INFO:teuthology.orchestra.run.smithi028.stdout: "m": "1", 2024-09-18T04:39:57.414 INFO:teuthology.orchestra.run.smithi028.stdout: "plugin": "jerasure", 2024-09-18T04:39:57.414 INFO:teuthology.orchestra.run.smithi028.stdout: "technique": "reed_sol_van" 2024-09-18T04:39:57.414 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:57.414 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:57.414 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps_queue": [], 2024-09-18T04:39:57.414 INFO:teuthology.orchestra.run.smithi028.stdout: "new_removed_snaps": [], 2024-09-18T04:39:57.414 INFO:teuthology.orchestra.run.smithi028.stdout: "new_purged_snaps": [], 2024-09-18T04:39:57.414 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_node_flags": {}, 2024-09-18T04:39:57.415 INFO:teuthology.orchestra.run.smithi028.stdout: "device_class_flags": {}, 2024-09-18T04:39:57.415 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": { 2024-09-18T04:39:57.415 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_enabled": false, 2024-09-18T04:39:57.415 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_bucket_count": 0, 2024-09-18T04:39:57.415 INFO:teuthology.orchestra.run.smithi028.stdout: "degraded_stretch_mode": 0, 2024-09-18T04:39:57.415 INFO:teuthology.orchestra.run.smithi028.stdout: "recovering_stretch_mode": 0, 2024-09-18T04:39:57.415 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_bucket": 0 2024-09-18T04:39:57.415 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:57.415 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:57.417 INFO:tasks.thrashosds.thrasher:j is {'epoch': 90, 'fsid': 'be6146ba-a991-4f6e-b359-99cbedc9f9b1', 'created': '2024-09-18T04:37:15.676634+0000', 'modified': '2024-09-18T04:39:54.747372+0000', 'last_up_change': '2024-09-18T04:39:45.699693+0000', 'last_in_change': '2024-09-18T04:39:17.550632+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 6, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-18T04:37:25.654950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-18T04:37:32.672462+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 8, 'snap_epoch': 80, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2.2699999809265137, 'score_stable': 2.2699999809265137, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'new_default_pool17342', 'create_time': '2024-09-18T04:38:42.986132+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 55, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2.2699999809265137, 'score_stable': 2.2699999809265137, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '6177002d-72e6-4888-bcde-29b6a92d9dc4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 86, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6801', 'nonce': 1554407851}, {'type': 'v1', 'addr': '172.21.15.28:6803', 'nonce': 1554407851}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6805', 'nonce': 1554407851}, {'type': 'v1', 'addr': '172.21.15.28:6807', 'nonce': 1554407851}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6813', 'nonce': 1554407851}, {'type': 'v1', 'addr': '172.21.15.28:6815', 'nonce': 1554407851}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6809', 'nonce': 1554407851}, {'type': 'v1', 'addr': '172.21.15.28:6811', 'nonce': 1554407851}]}, 'public_addr': '172.21.15.28:6803/1554407851', 'cluster_addr': '172.21.15.28:6807/1554407851', 'heartbeat_back_addr': '172.21.15.28:6815/1554407851', 'heartbeat_front_addr': '172.21.15.28:6811/1554407851', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '4f62cb09-3457-418c-bb69-491d9b70914c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 86, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6800', 'nonce': 511141401}, {'type': 'v1', 'addr': '172.21.15.28:6802', 'nonce': 511141401}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6804', 'nonce': 511141401}, {'type': 'v1', 'addr': '172.21.15.28:6806', 'nonce': 511141401}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6812', 'nonce': 511141401}, {'type': 'v1', 'addr': '172.21.15.28:6814', 'nonce': 511141401}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6808', 'nonce': 511141401}, {'type': 'v1', 'addr': '172.21.15.28:6810', 'nonce': 511141401}]}, 'public_addr': '172.21.15.28:6802/511141401', 'cluster_addr': '172.21.15.28:6806/511141401', 'heartbeat_back_addr': '172.21.15.28:6814/511141401', 'heartbeat_front_addr': '172.21.15.28:6810/511141401', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '854fe031-3b87-4290-9c2f-3fb196fb806c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 12, 'last_clean_end': 76, 'up_from': 86, 'up_thru': 86, 'down_at': 77, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6816', 'nonce': 3849236161}, {'type': 'v1', 'addr': '172.21.15.28:6817', 'nonce': 3849236161}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6818', 'nonce': 3849236161}, {'type': 'v1', 'addr': '172.21.15.28:6819', 'nonce': 3849236161}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6822', 'nonce': 3849236161}, {'type': 'v1', 'addr': '172.21.15.28:6823', 'nonce': 3849236161}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6820', 'nonce': 3849236161}, {'type': 'v1', 'addr': '172.21.15.28:6821', 'nonce': 3849236161}]}, 'public_addr': '172.21.15.28:6817/3849236161', 'cluster_addr': '172.21.15.28:6819/3849236161', 'heartbeat_back_addr': '172.21.15.28:6823/3849236161', 'heartbeat_front_addr': '172.21.15.28:6821/3849236161', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'c2625f98-780b-4f86-bf36-a48d2a227756', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 32, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6808', 'nonce': 2746051132}, {'type': 'v1', 'addr': '172.21.15.82:6809', 'nonce': 2746051132}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6810', 'nonce': 2746051132}, {'type': 'v1', 'addr': '172.21.15.82:6811', 'nonce': 2746051132}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6814', 'nonce': 2746051132}, {'type': 'v1', 'addr': '172.21.15.82:6815', 'nonce': 2746051132}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6812', 'nonce': 2746051132}, {'type': 'v1', 'addr': '172.21.15.82:6813', 'nonce': 2746051132}]}, 'public_addr': '172.21.15.82:6809/2746051132', 'cluster_addr': '172.21.15.82:6811/2746051132', 'heartbeat_back_addr': '172.21.15.82:6815/2746051132', 'heartbeat_front_addr': '172.21.15.82:6813/2746051132', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'deb22d86-4081-484d-96eb-9cb9d96b60bc', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 52, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6800', 'nonce': 2005712460}, {'type': 'v1', 'addr': '172.21.15.82:6801', 'nonce': 2005712460}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6802', 'nonce': 2005712460}, {'type': 'v1', 'addr': '172.21.15.82:6803', 'nonce': 2005712460}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6806', 'nonce': 2005712460}, {'type': 'v1', 'addr': '172.21.15.82:6807', 'nonce': 2005712460}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6804', 'nonce': 2005712460}, {'type': 'v1', 'addr': '172.21.15.82:6805', 'nonce': 2005712460}]}, 'public_addr': '172.21.15.82:6801/2005712460', 'cluster_addr': '172.21.15.82:6803/2005712460', 'heartbeat_back_addr': '172.21.15.82:6807/2005712460', 'heartbeat_front_addr': '172.21.15.82:6805/2005712460', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '061d332c-f51b-403d-bb18-eba415c97e7f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 86, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6816', 'nonce': 1267644232}, {'type': 'v1', 'addr': '172.21.15.82:6817', 'nonce': 1267644232}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6818', 'nonce': 1267644232}, {'type': 'v1', 'addr': '172.21.15.82:6819', 'nonce': 1267644232}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6822', 'nonce': 1267644232}, {'type': 'v1', 'addr': '172.21.15.82:6823', 'nonce': 1267644232}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6820', 'nonce': 1267644232}, {'type': 'v1', 'addr': '172.21.15.82:6821', 'nonce': 1267644232}]}, 'public_addr': '172.21.15.82:6817/1267644232', 'cluster_addr': '172.21.15.82:6819/1267644232', 'heartbeat_back_addr': '172.21.15.82:6823/1267644232', 'heartbeat_front_addr': '172.21.15.82:6821/1267644232', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T04:37:22.192087+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T04:37:22.017833+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T04:37:22.755774+0000', 'dead_epoch': 77}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T04:37:22.663785+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-18T04:37:22.531758+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T04:37:22.400774+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.5', 'osds': [2, 0]}, {'pgid': '2.7', 'osds': [3, 5]}], 'pg_upmap_items': [{'pgid': '2.1', 'mappings': [{'from': 2, 'to': 1}]}, {'pgid': '2.4', 'mappings': [{'from': 1, 'to': 5}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-18T04:39:57.417 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T04:39:57.418 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:57.412+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:57.447 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:57.447 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:57.458 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:39:57.518 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:57.512+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:57.576 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:57.576 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:57.586 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:39:57.619 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:57.616+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:57.694 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:57.695 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:57.705 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:39:57.744 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:39:57.746 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:57.744+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:57.747 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-18T04:39:57.759 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":161,"stamp":"2024-09-18T04:39:57.631956+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":147465123,"num_objects":77,"num_object_clones":19,"num_object_copies":154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77,"num_whiteouts":13,"num_read":6021,"num_read_kb":175182,"num_write":934,"num_write_kb":273251,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":39960592,"num_keys_recovered":25,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":888,"ondisk_log_size":888,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":471859200,"kb_used":437360,"kb_used_data":276080,"kb_used_omap":16,"kb_used_meta":161199,"kb_avail":471421840,"statfs":{"total":483183820800,"available":482735964160,"internally_reserved":0,"allocated":282705920,"data_stored":301354659,"data_compressed":482441,"data_compressed_allocated":516096,"data_compressed_original":9871360,"omap_allocated":16507,"internal_metadata":165068677},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-19,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":27377664,"num_keys_recovered":22,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002244"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":233,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-18T04:39:48.735308+0000","last_change":"2024-09-18T04:37:27.423069+0000","last_active":"2024-09-18T04:39:48.735308+0000","last_peered":"2024-09-18T04:39:48.735308+0000","last_clean":"2024-09-18T04:39:48.735308+0000","last_became_active":"2024-09-18T04:37:27.422350+0000","last_became_peered":"2024-09-18T04:37:27.422350+0000","last_unstale":"2024-09-18T04:39:48.735308+0000","last_undegraded":"2024-09-18T04:39:48.735308+0000","last_fullsized":"2024-09-18T04:39:48.735308+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:26.406292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:26.406292+0000","last_clean_scrub_stamp":"2024-09-18T04:37:26.406292+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:31:45.852149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"81'78","reported_seq":421,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T04:39:53.557350+0000","last_change":"2024-09-18T04:39:46.795376+0000","last_active":"2024-09-18T04:39:53.557350+0000","last_peered":"2024-09-18T04:39:53.557350+0000","last_clean":"2024-09-18T04:39:53.557350+0000","last_became_active":"2024-09-18T04:39:46.718066+0000","last_became_peered":"2024-09-18T04:39:46.718066+0000","last_unstale":"2024-09-18T04:39:53.557350+0000","last_undegraded":"2024-09-18T04:39:53.557350+0000","last_fullsized":"2024-09-18T04:39:53.557350+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T07:09:22.599338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018293420000000001,"stat_sum":{"num_bytes":10535004,"num_objects":9,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":378,"num_read_kb":16943,"num_write":99,"num_write_kb":25695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":9486336,"num_keys_recovered":9,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"69'3","reported_seq":57,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-18T04:39:48.754410+0000","last_change":"2024-09-18T04:38:46.966799+0000","last_active":"2024-09-18T04:39:48.754410+0000","last_peered":"2024-09-18T04:39:48.754410+0000","last_clean":"2024-09-18T04:39:48.754410+0000","last_became_active":"2024-09-18T04:38:44.953092+0000","last_became_peered":"2024-09-18T04:38:44.953092+0000","last_unstale":"2024-09-18T04:39:48.754410+0000","last_undegraded":"2024-09-18T04:39:48.754410+0000","last_fullsized":"2024-09-18T04:39:48.754410+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_clean_scrub_stamp":"2024-09-18T04:38:43.928423+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:04:46.888906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010424099999999999,"stat_sum":{"num_bytes":54,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"81'205","reported_seq":1151,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-18T04:39:48.735311+0000","last_change":"2024-09-18T04:37:36.484950+0000","last_active":"2024-09-18T04:39:48.735311+0000","last_peered":"2024-09-18T04:39:48.735311+0000","last_clean":"2024-09-18T04:39:48.735311+0000","last_became_active":"2024-09-18T04:37:34.470841+0000","last_became_peered":"2024-09-18T04:37:34.470841+0000","last_unstale":"2024-09-18T04:39:48.735311+0000","last_undegraded":"2024-09-18T04:39:48.735311+0000","last_fullsized":"2024-09-18T04:39:48.735311+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:16:34.528783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090513900000000001,"stat_sum":{"num_bytes":25165900,"num_objects":11,"num_object_clones":3,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":2,"num_read":1557,"num_read_kb":37910,"num_write":304,"num_write_kb":45334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":70,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T04:39:53.557785+0000","last_change":"2024-09-18T04:39:46.719677+0000","last_active":"2024-09-18T04:39:53.557785+0000","last_peered":"2024-09-18T04:39:53.557785+0000","last_clean":"2024-09-18T04:39:53.557785+0000","last_became_active":"2024-09-18T04:39:46.719326+0000","last_became_peered":"2024-09-18T04:39:46.719326+0000","last_unstale":"2024-09-18T04:39:53.557785+0000","last_undegraded":"2024-09-18T04:39:53.557785+0000","last_fullsized":"2024-09-18T04:39:53.557785+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_clean_scrub_stamp":"2024-09-18T04:38:43.928423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:17:29.396859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075989799999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"87'238","reported_seq":1646,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T04:39:51.751128+0000","last_change":"2024-09-18T04:39:46.746657+0000","last_active":"2024-09-18T04:39:51.751128+0000","last_peered":"2024-09-18T04:39:51.751128+0000","last_clean":"2024-09-18T04:39:51.751128+0000","last_became_active":"2024-09-18T04:39:46.716998+0000","last_became_peered":"2024-09-18T04:39:46.716998+0000","last_unstale":"2024-09-18T04:39:51.751128+0000","last_undegraded":"2024-09-18T04:39:51.751128+0000","last_fullsized":"2024-09-18T04:39:51.751128+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":238,"log_dups_size":0,"ondisk_log_size":238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:37:53.810382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18898982,"num_objects":9,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3053,"num_read_kb":26737,"num_write":148,"num_write_kb":27756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":5308416,"num_keys_recovered":13,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":63,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-18T04:39:48.754282+0000","last_change":"2024-09-18T04:39:46.719007+0000","last_active":"2024-09-18T04:39:48.754282+0000","last_peered":"2024-09-18T04:39:48.754282+0000","last_clean":"2024-09-18T04:39:48.754282+0000","last_became_active":"2024-09-18T04:39:46.718167+0000","last_became_peered":"2024-09-18T04:39:46.718167+0000","last_unstale":"2024-09-18T04:39:48.754282+0000","last_undegraded":"2024-09-18T04:39:48.754282+0000","last_fullsized":"2024-09-18T04:39:48.754282+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_clean_scrub_stamp":"2024-09-18T04:38:43.928423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T08:34:24.603108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011574549999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"20'2","reported_seq":186,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T04:39:53.557369+0000","last_change":"2024-09-18T04:37:36.490461+0000","last_active":"2024-09-18T04:39:53.557369+0000","last_peered":"2024-09-18T04:39:53.557369+0000","last_clean":"2024-09-18T04:39:53.557369+0000","last_became_active":"2024-09-18T04:37:34.473088+0000","last_became_peered":"2024-09-18T04:37:34.473088+0000","last_unstale":"2024-09-18T04:39:53.557369+0000","last_undegraded":"2024-09-18T04:39:53.557369+0000","last_fullsized":"2024-09-18T04:39:53.557369+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:17:57.352397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001879932,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":99,"num_read_kb":99,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"69'2","reported_seq":64,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T04:39:53.557875+0000","last_change":"2024-09-18T04:39:18.581995+0000","last_active":"2024-09-18T04:39:53.557875+0000","last_peered":"2024-09-18T04:39:53.557875+0000","last_clean":"2024-09-18T04:39:53.557875+0000","last_became_active":"2024-09-18T04:39:18.576236+0000","last_became_peered":"2024-09-18T04:39:18.576236+0000","last_unstale":"2024-09-18T04:39:53.557875+0000","last_undegraded":"2024-09-18T04:39:53.557875+0000","last_fullsized":"2024-09-18T04:39:53.557875+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_clean_scrub_stamp":"2024-09-18T04:38:43.928423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:29:16.861010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010091900000000001,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":16,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"69'3","reported_seq":76,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T04:39:53.557410+0000","last_change":"2024-09-18T04:39:46.720085+0000","last_active":"2024-09-18T04:39:53.557410+0000","last_peered":"2024-09-18T04:39:53.557410+0000","last_clean":"2024-09-18T04:39:53.557410+0000","last_became_active":"2024-09-18T04:39:46.719365+0000","last_became_peered":"2024-09-18T04:39:46.719365+0000","last_unstale":"2024-09-18T04:39:53.557410+0000","last_undegraded":"2024-09-18T04:39:53.557410+0000","last_fullsized":"2024-09-18T04:39:53.557410+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_clean_scrub_stamp":"2024-09-18T04:38:43.928423+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:38:16.629282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065509499999999996,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"79'45","reported_seq":71,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T04:39:53.557370+0000","last_change":"2024-09-18T04:38:59.528402+0000","last_active":"2024-09-18T04:39:53.557370+0000","last_peered":"2024-09-18T04:39:53.557370+0000","last_clean":"2024-09-18T04:39:53.557370+0000","last_became_active":"2024-09-18T04:38:59.462279+0000","last_became_peered":"2024-09-18T04:38:59.462279+0000","last_unstale":"2024-09-18T04:39:53.557370+0000","last_undegraded":"2024-09-18T04:39:53.557370+0000","last_fullsized":"2024-09-18T04:39:53.557370+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:32:58.894411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081514499999999998,"stat_sum":{"num_bytes":20971520,"num_objects":9,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":2,"num_read":83,"num_read_kb":16649,"num_write":43,"num_write_kb":36866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":12582912,"num_keys_recovered":3,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"69'1","reported_seq":65,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-18T04:39:48.754258+0000","last_change":"2024-09-18T04:39:46.718266+0000","last_active":"2024-09-18T04:39:48.754258+0000","last_peered":"2024-09-18T04:39:48.754258+0000","last_clean":"2024-09-18T04:39:48.754258+0000","last_became_active":"2024-09-18T04:39:46.717950+0000","last_became_peered":"2024-09-18T04:39:46.717950+0000","last_unstale":"2024-09-18T04:39:48.754258+0000","last_undegraded":"2024-09-18T04:39:48.754258+0000","last_fullsized":"2024-09-18T04:39:48.754258+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_clean_scrub_stamp":"2024-09-18T04:38:43.928423+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:17:31.749220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016757619999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"79'64","reported_seq":381,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-18T04:39:48.755304+0000","last_change":"2024-09-18T04:39:46.817083+0000","last_active":"2024-09-18T04:39:48.755304+0000","last_peered":"2024-09-18T04:39:48.755304+0000","last_clean":"2024-09-18T04:39:48.755304+0000","last_became_active":"2024-09-18T04:39:46.717423+0000","last_became_peered":"2024-09-18T04:39:46.717423+0000","last_unstale":"2024-09-18T04:39:48.755304+0000","last_undegraded":"2024-09-18T04:39:48.755304+0000","last_fullsized":"2024-09-18T04:39:48.755304+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T09:52:29.555572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165862,"num_objects":10,"num_object_clones":3,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":3,"num_read":244,"num_read_kb":31404,"num_write":61,"num_write_kb":49153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-18T04:39:48.755506+0000","last_change":"2024-09-18T04:39:46.717470+0000","last_active":"2024-09-18T04:39:48.755506+0000","last_peered":"2024-09-18T04:39:48.755506+0000","last_clean":"2024-09-18T04:39:48.755506+0000","last_became_active":"2024-09-18T04:39:46.717099+0000","last_became_peered":"2024-09-18T04:39:46.717099+0000","last_unstale":"2024-09-18T04:39:48.755506+0000","last_undegraded":"2024-09-18T04:39:48.755506+0000","last_fullsized":"2024-09-18T04:39:48.755506+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_clean_scrub_stamp":"2024-09-18T04:38:43.928423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:08:02.208525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"80'68","reported_seq":259,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-18T04:39:48.754522+0000","last_change":"2024-09-18T04:37:36.494193+0000","last_active":"2024-09-18T04:39:48.754522+0000","last_peered":"2024-09-18T04:39:48.754522+0000","last_clean":"2024-09-18T04:39:48.754522+0000","last_became_active":"2024-09-18T04:37:34.472940+0000","last_became_peered":"2024-09-18T04:37:34.472940+0000","last_unstale":"2024-09-18T04:39:48.754522+0000","last_undegraded":"2024-09-18T04:39:48.754522+0000","last_fullsized":"2024-09-18T04:39:48.754522+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:18:29.266815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070553900000000004,"stat_sum":{"num_bytes":33554470,"num_objects":13,"num_object_clones":3,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":3,"num_read":112,"num_read_kb":28698,"num_write":65,"num_write_kb":57347,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":65,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-18T04:39:53.557892+0000","last_change":"2024-09-18T04:39:18.577075+0000","last_active":"2024-09-18T04:39:53.557892+0000","last_peered":"2024-09-18T04:39:53.557892+0000","last_clean":"2024-09-18T04:39:53.557892+0000","last_became_active":"2024-09-18T04:39:18.576593+0000","last_became_peered":"2024-09-18T04:39:18.576593+0000","last_unstale":"2024-09-18T04:39:53.557892+0000","last_undegraded":"2024-09-18T04:39:53.557892+0000","last_fullsized":"2024-09-18T04:39:53.557892+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_clean_scrub_stamp":"2024-09-18T04:38:43.928423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:27:02.258820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058611500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"79'62","reported_seq":326,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-18T04:39:48.735423+0000","last_change":"2024-09-18T04:38:05.754820+0000","last_active":"2024-09-18T04:39:48.735423+0000","last_peered":"2024-09-18T04:39:48.735423+0000","last_clean":"2024-09-18T04:39:48.735423+0000","last_became_active":"2024-09-18T04:38:05.753962+0000","last_became_peered":"2024-09-18T04:38:05.753962+0000","last_unstale":"2024-09-18T04:39:48.735423+0000","last_undegraded":"2024-09-18T04:39:48.735423+0000","last_fullsized":"2024-09-18T04:39:48.735423+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:53:52.224098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047343899999999998,"stat_sum":{"num_bytes":12582928,"num_objects":7,"num_object_clones":2,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":2,"num_read":344,"num_read_kb":16613,"num_write":47,"num_write_kb":28676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":70,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":16,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1835,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":146874685,"num_objects":69,"num_object_clones":19,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":13,"num_read":5870,"num_read_kb":175053,"num_write":769,"num_write_kb":270829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":39960576,"num_keys_recovered":25,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":277356544,"data_stored":298156666,"data_compressed":416401,"data_compressed_allocated":425984,"data_compressed_original":8724480,"omap_allocated":3627,"internal_metadata":0},"log_size":762,"ondisk_log_size":762,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":66040,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":13,"seq":55834574882,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74204,"kb_used_data":43084,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94297636,"statfs":{"total":96636764160,"available":96560779264,"internally_reserved":0,"allocated":44118016,"data_stored":43448418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:39:28 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.85099999999999998,"15min":0.86499999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.0469999999999999,"5min":1.5249999999999999,"15min":1.5249999999999999},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.79700000000000004,"15min":0.80800000000000005},"min":{"1min":0.53500000000000003,"5min":0.502,"15min":0.502},"max":{"1min":0.996,"5min":1.369,"15min":1.369},"last":0.61799999999999999}]},{"osd":1,"last update":"Wed Sep 18 04:39:28 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.83999999999999997,"15min":0.85199999999999998},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.091,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.79500000000000004,"15min":0.79900000000000004},"min":{"1min":0.59999999999999998,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.71599999999999997}]},{"osd":2,"last update":"Wed Sep 18 04:38:26 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.4930000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.82499999999999996}]},{"osd":3,"last update":"Wed Sep 18 04:39:28 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.75700000000000001,"15min":0.77000000000000002},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.0169999999999999,"5min":1.22,"15min":1.22},"last":0.753},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.78900000000000003,"15min":0.79700000000000004},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.98199999999999998}]},{"osd":4,"last update":"Wed Sep 18 04:39:28 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.77800000000000002,"15min":0.78500000000000003},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.998,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.89900000000000002},{"interface":"front","average":{"1min":0.751,"5min":0.82499999999999996,"15min":0.83699999999999997},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.075,"5min":1.121,"15min":1.121},"last":0.86399999999999999}]}]},{"osd":4,"up_from":0,"seq":55834574882,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":13,"seq":55834574883,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81352,"kb_used_data":46140,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94290488,"statfs":{"total":96636764160,"available":96553459712,"internally_reserved":0,"allocated":47247360,"data_stored":47130178,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 04:39:28 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.59999999999999998,"15min":0.60899999999999999},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.78000000000000003,"5min":1.113,"15min":1.113},"last":0.501},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.63,"15min":0.64200000000000002},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.89900000000000002,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.41399999999999998}]},{"osd":2,"last update":"Wed Sep 18 04:38:28 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.433},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.8180000000000001,"5min":1.8180000000000001,"15min":1.8180000000000001},"last":0.34899999999999998}]},{"osd":3,"last update":"Wed Sep 18 04:39:28 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.81499999999999995,"15min":0.81799999999999995},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":0.99399999999999999,"5min":1.169,"15min":1.169},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.81899999999999995,"15min":0.82599999999999996},"min":{"1min":0.49199999999999999,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.97699999999999998,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.90300000000000002}]},{"osd":4,"last update":"Wed Sep 18 04:39:28 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.84599999999999997,"15min":0.85399999999999998},"min":{"1min":0.61699999999999999,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.0600000000000001,"5min":1.073,"15min":1.073},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.89600000000000002,"15min":0.90800000000000003},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.84099999999999997}]},{"osd":5,"last update":"Wed Sep 18 04:39:28 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.91600000000000004,"15min":0.93100000000000005},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.1479999999999999,"5min":1.8919999999999999,"15min":1.8919999999999999},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.95399999999999996,"15min":0.97199999999999998},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.3140000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":0.73299999999999998}]}]},{"osd":3,"up_from":12,"seq":51539607586,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":97752,"kb_used_data":66636,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94274088,"statfs":{"total":96636764160,"available":96536666112,"internally_reserved":0,"allocated":68235264,"data_stored":68167326,"data_compressed":36109,"data_compressed_allocated":49152,"data_compressed_original":638976,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.93200000000000005,"15min":0.95299999999999996},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.264,"5min":2.665,"15min":2.665},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.86799999999999999,"15min":0.878},"min":{"1min":0.55700000000000005,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.0649999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.68200000000000005}]},{"osd":1,"last update":"Wed Sep 18 04:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.86299999999999999,"15min":0.88300000000000001},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.121,"5min":2.4510000000000001,"15min":2.4510000000000001},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.77200000000000002,"15min":0.78600000000000003},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.046,"5min":1.077,"15min":1.077},"last":0.57799999999999996}]},{"osd":2,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":2.577,"5min":2.577,"15min":2.577},"last":0.505},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.64300000000000002}]},{"osd":4,"last update":"Wed Sep 18 04:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.72499999999999998,"15min":0.72699999999999998},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.97399999999999998,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.751,"15min":0.749},"min":{"1min":0.49199999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.0209999999999999,"5min":1.044,"15min":1.044},"last":0.53800000000000003}]},{"osd":5,"last update":"Wed Sep 18 04:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.73599999999999999,"15min":0.73899999999999999},"min":{"1min":0.56399999999999995,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.069,"5min":1.173,"15min":1.173},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.877,"5min":0.86899999999999999,"15min":0.86799999999999999},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.087,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.76600000000000001}]}]},{"osd":2,"up_from":86,"seq":369367187459,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79072,"kb_used_data":50452,"kb_used_omap":9,"kb_used_meta":28598,"kb_avail":94292768,"statfs":{"total":96636764160,"available":96555794432,"internally_reserved":0,"allocated":51662848,"data_stored":61334676,"data_compressed":205860,"data_compressed_allocated":274432,"data_compressed_original":4308992,"omap_allocated":10134,"internal_metadata":29284458},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81000000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86699999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.244}]}]},{"osd":1,"up_from":12,"seq":51539607586,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":104980,"kb_used_data":69768,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94266860,"statfs":{"total":96636764160,"available":96529264640,"internally_reserved":0,"allocated":71442432,"data_stored":81274061,"data_compressed":207452,"data_compressed_allocated":147456,"data_compressed_original":4349952,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:39:27 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.60299999999999998,"15min":0.59499999999999997},"min":{"1min":0.433,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.625,"15min":0.627},"min":{"1min":0.42599999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.871,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.48899999999999999}]},{"osd":2,"last update":"Wed Sep 18 04:39:27 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.66200000000000003,"15min":0.66300000000000003},"min":{"1min":0.46700000000000003,"5min":0.436,"15min":0.436},"max":{"1min":1.05,"5min":1.327,"15min":1.327},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.66400000000000003,"15min":0.66700000000000004},"min":{"1min":0.43099999999999999,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.872,"5min":1.002,"15min":1.002},"last":0.41099999999999998}]},{"osd":3,"last update":"Wed Sep 18 04:39:27 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.79900000000000004,"15min":0.80100000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.22,"5min":1.22,"15min":1.22},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.86799999999999999,"15min":0.871},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.80900000000000005}]},{"osd":4,"last update":"Wed Sep 18 04:39:27 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"min":{"1min":0.63600000000000001,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.018,"5min":1.236,"15min":1.236},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.93000000000000005,"15min":0.93100000000000005},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.268,"5min":1.268,"15min":1.268},"last":0.71599999999999997}]},{"osd":5,"last update":"Wed Sep 18 04:39:27 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.90700000000000003,"15min":0.91100000000000003},"min":{"1min":0.71899999999999997,"5min":0.628,"15min":0.628},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.159,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.626}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":46141440,"data_stored":46137382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":70397952,"data_stored":80871595,"data_compressed":207452,"data_compressed_allocated":147456,"data_compressed_original":4349952,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":50626560,"data_stored":60932264,"data_compressed":205860,"data_compressed_allocated":274432,"data_compressed_original":4308992,"omap_allocated":3627,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":67129344,"data_stored":67174530,"data_compressed":3089,"data_compressed_allocated":4096,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":43061248,"data_stored":43040895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":16,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":54,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1835,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":16,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":16,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":54,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T04:39:57.761 INFO:tasks.thrashosds.thrasher:Setting 3.0 to [2, 1] 2024-09-18T04:39:57.761 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '3.0', '2', '1'] 2024-09-18T04:39:57.761 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 3.0 2 1 2024-09-18T04:39:57.777 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-09-18T04:39:57.821 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:57.815+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:39:57.856 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:57.857 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:57.867 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:39:57.922 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:57.915+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:57.973 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:57.973 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:57.983 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:39:58.023 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:58.019+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:39:58.088 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:58.088 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:58.098 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:39:58.124 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:58.119+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:39:58.201 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:58.201 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:58.210 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:39:58.225 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:58.219+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:39:58.314 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:58.315 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:58.325 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:39:58.326 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:58.323+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:58.428 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:39:58.424+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:39:58.434 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:58.435 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:58.444 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:39:58.529 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:58.523+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:58.549 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:58.549 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:58.559 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:39:58.630 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:58.627+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:58.678 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:58.678 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:58.688 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:39:58.731 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:58.728+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:58.780 INFO:teuthology.orchestra.run.smithi028.stderr:set 3.0 pg_upmap mapping to [2,1] 2024-09-18T04:39:58.803 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:58.803 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:58.813 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:39:58.831 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:58.827+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:58.918 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:58.918 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:58.928 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:39:58.971 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:58.967+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:39:58.992 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T04:39:59.033 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:59.028+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:59.085 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:39:59.085 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:39:59.095 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:39:59.134 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:39:59.134 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:59.131+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:59.155 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:39:59.176 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:39:59.195 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:39:59.200 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:59.200 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:59.200 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:39:59.200 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:39:59.200 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:59.210 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:39:59.226 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:39:59.251 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:39:59.251 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:59.247+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:39:59.253 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:39:59.253 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:39:59.253 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:39:59.254 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:39:59.254 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:39:59.254 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:39:59.336 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:59.337 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:39:59.337 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:39:59.337 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:39:59.337 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.337 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:37 2.5 e51/28 2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:head v 51'1, mlcod=0'0)", 2024-09-18T04:39:59.337 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:39:59.337 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 77.039874600000005, 2024-09-18T04:39:59.337 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012279973, 2024-09-18T04:39:59.337 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:59.337 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:59.337 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:59.337 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.338 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:59.338 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:39:59.338 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.338 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.338 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.338 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:59.338 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:39:59.338 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.338 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.338 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.338 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:59.338 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292468+0000", 2024-09-18T04:39:59.338 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:59.339 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.339 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.339 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:59.339 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295837+0000", 2024-09-18T04:39:59.339 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0033691699999999999 2024-09-18T04:39:59.339 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.339 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.339 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:59.339 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295860+0000", 2024-09-18T04:39:59.339 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2756999999999999e-05 2024-09-18T04:39:59.339 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.339 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.339 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:59.339 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295872+0000", 2024-09-18T04:39:59.340 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2442e-05 2024-09-18T04:39:59.340 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.340 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.340 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:59.340 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295934+0000", 2024-09-18T04:39:59.340 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1187999999999998e-05 2024-09-18T04:39:59.340 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.340 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.340 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:59.340 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295967+0000", 2024-09-18T04:39:59.340 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3185999999999997e-05 2024-09-18T04:39:59.340 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.340 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.341 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:59.341 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304697+0000", 2024-09-18T04:39:59.341 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0087298059999999997 2024-09-18T04:39:59.341 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.341 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.341 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:59.341 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304749+0000", 2024-09-18T04:39:59.341 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1922000000000002e-05 2024-09-18T04:39:59.341 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.341 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:59.341 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.341 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.341 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.342 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:49 2.5 e51/28 2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head v 51'2, mlcod=0'0)", 2024-09-18T04:39:59.342 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:39:59.342 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 77.006547761999997, 2024-09-18T04:39:59.342 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018033041999999999, 2024-09-18T04:39:59.342 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:59.342 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:59.342 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:59.342 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.342 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:59.342 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:39:59.342 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.342 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.343 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.343 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:59.343 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:39:59.343 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.343 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.343 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.343 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:59.343 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325793+0000", 2024-09-18T04:39:59.343 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:59.343 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.343 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.343 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:59.343 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329038+0000", 2024-09-18T04:39:59.343 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0032444010000000001 2024-09-18T04:39:59.343 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.344 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.344 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:59.344 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329065+0000", 2024-09-18T04:39:59.344 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7500000000000001e-05 2024-09-18T04:39:59.344 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.344 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.344 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:59.344 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329077+0000", 2024-09-18T04:39:59.344 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1919999999999999e-05 2024-09-18T04:39:59.344 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.344 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.344 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:59.344 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329149+0000", 2024-09-18T04:39:59.345 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2349999999999997e-05 2024-09-18T04:39:59.345 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.345 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.345 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:59.345 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329192+0000", 2024-09-18T04:39:59.345 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2744000000000003e-05 2024-09-18T04:39:59.345 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.345 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.345 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:59.345 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343773+0000", 2024-09-18T04:39:59.345 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014581246000000001 2024-09-18T04:39:59.345 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.345 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.345 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:59.346 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343829+0000", 2024-09-18T04:39:59.346 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5143999999999999e-05 2024-09-18T04:39:59.346 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.346 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:59.346 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.346 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.346 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.346 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:47 2.4 e51/17 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 51'3, mlcod=51'1)", 2024-09-18T04:39:59.346 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:39:59.346 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 77.005967497, 2024-09-18T04:39:59.346 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018960186, 2024-09-18T04:39:59.346 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:59.346 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:59.346 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:59.347 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.347 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:59.347 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:39:59.347 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.347 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.347 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.347 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:59.347 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:39:59.347 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.347 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.347 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.347 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:59.347 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326374+0000", 2024-09-18T04:39:59.347 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:59.348 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.348 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.348 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:59.348 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330883+0000", 2024-09-18T04:39:59.348 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0045084740000000002 2024-09-18T04:39:59.348 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.348 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.348 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:59.348 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330901+0000", 2024-09-18T04:39:59.348 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7890999999999999e-05 2024-09-18T04:39:59.348 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.348 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.348 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:59.348 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330912+0000", 2024-09-18T04:39:59.349 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1041e-05 2024-09-18T04:39:59.349 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.349 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.349 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:59.349 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.331003+0000", 2024-09-18T04:39:59.349 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1323999999999995e-05 2024-09-18T04:39:59.349 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.349 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.349 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:59.349 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.331030+0000", 2024-09-18T04:39:59.349 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7351999999999998e-05 2024-09-18T04:39:59.349 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.349 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.350 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:59.350 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345287+0000", 2024-09-18T04:39:59.350 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014256332 2024-09-18T04:39:59.350 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.350 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.350 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:59.350 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345336+0000", 2024-09-18T04:39:59.350 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9356e-05 2024-09-18T04:39:59.350 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.350 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:59.350 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.350 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.350 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.350 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:58 2.5 e51/28 2:b227e6b1:::rbd_data.125fc09df6d1.000000000000000e:head v 51'3, mlcod=51'1)", 2024-09-18T04:39:59.351 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:39:59.351 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 76.965540869999998, 2024-09-18T04:39:59.351 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016513678, 2024-09-18T04:39:59.351 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:59.351 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:59.351 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:59.351 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.351 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:59.351 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:39:59.351 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.351 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.351 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.351 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:59.352 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:39:59.352 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.352 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.352 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.352 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:59.352 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366800+0000", 2024-09-18T04:39:59.352 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:59.352 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.352 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.352 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:59.352 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368644+0000", 2024-09-18T04:39:59.352 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001844399 2024-09-18T04:39:59.352 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.352 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.353 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:59.353 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368681+0000", 2024-09-18T04:39:59.353 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6526999999999998e-05 2024-09-18T04:39:59.353 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.353 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.353 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:59.353 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368698+0000", 2024-09-18T04:39:59.353 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7272999999999999e-05 2024-09-18T04:39:59.353 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.353 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.353 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:59.354 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368773+0000", 2024-09-18T04:39:59.354 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4862000000000003e-05 2024-09-18T04:39:59.354 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.354 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.354 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:59.354 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368801+0000", 2024-09-18T04:39:59.354 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7925000000000001e-05 2024-09-18T04:39:59.354 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.354 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.354 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:59.354 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383261+0000", 2024-09-18T04:39:59.354 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014460377999999999 2024-09-18T04:39:59.354 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.355 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.355 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:59.355 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383316+0000", 2024-09-18T04:39:59.355 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4623999999999997e-05 2024-09-18T04:39:59.355 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.355 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:59.355 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.355 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.355 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.355 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:56 2.4 e51/17 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 51'4, mlcod=51'2)", 2024-09-18T04:39:59.355 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:39:59.355 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 76.963536844000004, 2024-09-18T04:39:59.356 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015105221, 2024-09-18T04:39:59.356 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:59.356 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:59.356 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:59.356 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.356 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:59.356 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:39:59.356 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.357 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.357 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.357 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:59.357 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:39:59.357 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.357 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.357 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.357 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:59.357 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368805+0000", 2024-09-18T04:39:59.358 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:59.358 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.358 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.358 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:59.358 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371603+0000", 2024-09-18T04:39:59.358 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027982660000000002 2024-09-18T04:39:59.358 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.358 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.358 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:59.358 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371633+0000", 2024-09-18T04:39:59.358 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0073000000000002e-05 2024-09-18T04:39:59.358 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.358 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.359 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:59.359 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371652+0000", 2024-09-18T04:39:59.359 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8597000000000002e-05 2024-09-18T04:39:59.359 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.359 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.359 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:59.359 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371752+0000", 2024-09-18T04:39:59.359 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9772999999999995e-05 2024-09-18T04:39:59.359 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.359 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.359 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:59.359 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371793+0000", 2024-09-18T04:39:59.359 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1173e-05 2024-09-18T04:39:59.359 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.360 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.360 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:59.360 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383875+0000", 2024-09-18T04:39:59.360 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012081852000000001 2024-09-18T04:39:59.360 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.360 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.360 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:59.360 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383912+0000", 2024-09-18T04:39:59.360 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7045000000000002e-05 2024-09-18T04:39:59.360 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.360 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:59.360 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.360 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.361 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.361 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:36 2.5 e51/28 2:bd5b8dd8:ns::rbd_data.133e813ab5bc.0000000000000002:head v 51'4, mlcod=51'2)", 2024-09-18T04:39:59.361 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:39:59.361 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 76.794817852999998, 2024-09-18T04:39:59.361 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020758161000000001, 2024-09-18T04:39:59.361 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:59.361 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:59.361 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:59.361 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.361 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:59.361 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:39:59.361 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.361 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.362 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.362 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:59.362 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:39:59.362 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.362 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.362 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.362 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:59.362 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537523+0000", 2024-09-18T04:39:59.362 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:59.362 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.362 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.362 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:59.362 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541578+0000", 2024-09-18T04:39:59.362 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0040549710000000001 2024-09-18T04:39:59.362 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.363 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.363 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:59.363 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541608+0000", 2024-09-18T04:39:59.363 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9349e-05 2024-09-18T04:39:59.363 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.363 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.363 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:59.363 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541621+0000", 2024-09-18T04:39:59.363 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2736e-05 2024-09-18T04:39:59.363 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.363 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.363 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:59.364 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541715+0000", 2024-09-18T04:39:59.364 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4197999999999995e-05 2024-09-18T04:39:59.364 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.364 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.364 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:59.364 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541755+0000", 2024-09-18T04:39:59.364 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9818999999999999e-05 2024-09-18T04:39:59.364 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.364 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.364 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:59.364 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558217+0000", 2024-09-18T04:39:59.364 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016462234999999999 2024-09-18T04:39:59.364 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.364 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.365 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:59.365 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558284+0000", 2024-09-18T04:39:59.365 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6774999999999997e-05 2024-09-18T04:39:59.365 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.365 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:59.365 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.365 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.365 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:39 2.4 e51/17 2:304364d1:ns::rbd_data.133e813ab5bc.0000000000000004:head v 51'5, mlcod=51'3)", 2024-09-18T04:39:59.365 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:39:59.365 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 76.786678726000005, 2024-09-18T04:39:59.365 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016777558000000001, 2024-09-18T04:39:59.365 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:59.365 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:59.366 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:59.366 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.366 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:59.366 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:39:59.366 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.366 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.366 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.366 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:59.366 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:39:59.366 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.366 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.366 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.366 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:59.366 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545663+0000", 2024-09-18T04:39:59.367 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:59.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.367 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.367 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:59.367 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548286+0000", 2024-09-18T04:39:59.367 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026233390000000001 2024-09-18T04:39:59.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.367 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.367 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:59.367 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548305+0000", 2024-09-18T04:39:59.367 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8425000000000001e-05 2024-09-18T04:39:59.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.367 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.368 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:59.368 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548316+0000", 2024-09-18T04:39:59.368 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1163e-05 2024-09-18T04:39:59.368 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.368 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.368 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:59.368 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548379+0000", 2024-09-18T04:39:59.368 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2780000000000005e-05 2024-09-18T04:39:59.368 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.368 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.368 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:59.368 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548399+0000", 2024-09-18T04:39:59.368 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0080000000000001e-05 2024-09-18T04:39:59.369 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.369 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.369 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:59.369 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562383+0000", 2024-09-18T04:39:59.369 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013984671000000001 2024-09-18T04:39:59.369 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.369 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.369 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:59.369 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562442+0000", 2024-09-18T04:39:59.369 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8700999999999999e-05 2024-09-18T04:39:59.369 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.369 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:59.369 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.369 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.370 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.370 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:37 2.5 e51/28 2:bf39b8e4:ns::rbd_data.133e813ab5bc.0000000000000003:head v 51'5, mlcod=51'2)", 2024-09-18T04:39:59.370 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:39:59.370 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 76.783810457000001, 2024-09-18T04:39:59.370 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014264492, 2024-09-18T04:39:59.370 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:59.370 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:59.370 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:59.370 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.370 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:59.370 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:39:59.370 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.370 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.371 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.371 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:59.371 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:39:59.371 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.371 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.371 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:59.371 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548531+0000", 2024-09-18T04:39:59.371 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:59.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.371 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.371 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:59.371 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551500+0000", 2024-09-18T04:39:59.371 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0029688129999999998 2024-09-18T04:39:59.372 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.372 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.372 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:59.372 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551516+0000", 2024-09-18T04:39:59.372 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5994e-05 2024-09-18T04:39:59.372 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.372 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.372 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:59.372 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551526+0000", 2024-09-18T04:39:59.372 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9299999999999998e-06 2024-09-18T04:39:59.372 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.372 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.372 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:59.373 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551577+0000", 2024-09-18T04:39:59.373 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0871000000000001e-05 2024-09-18T04:39:59.373 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.373 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.373 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:59.373 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551602+0000", 2024-09-18T04:39:59.373 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4813000000000001e-05 2024-09-18T04:39:59.373 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.373 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.373 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:59.373 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562741+0000", 2024-09-18T04:39:59.373 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011139428999999999 2024-09-18T04:39:59.373 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.374 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.374 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:59.374 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562797+0000", 2024-09-18T04:39:59.374 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6329000000000002e-05 2024-09-18T04:39:59.374 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.374 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:59.374 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.374 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.374 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.374 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:49 2.4 e51/17 2:20f0b0c9:ns::rbd_data.133e813ab5bc.0000000000000009:head v 51'6, mlcod=51'4)", 2024-09-18T04:39:59.374 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:39:59.374 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 76.752375899, 2024-09-18T04:39:59.374 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013849046, 2024-09-18T04:39:59.375 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:59.375 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:59.375 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:59.375 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.375 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:59.375 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:39:59.375 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.375 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.375 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.375 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:59.375 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:39:59.375 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.375 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.375 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.376 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:59.376 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579966+0000", 2024-09-18T04:39:59.376 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:59.376 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.376 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.376 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:59.376 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582324+0000", 2024-09-18T04:39:59.376 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023578890000000002 2024-09-18T04:39:59.376 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.376 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.376 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:59.376 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582355+0000", 2024-09-18T04:39:59.376 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1485000000000001e-05 2024-09-18T04:39:59.376 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.376 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.377 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:59.377 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582371+0000", 2024-09-18T04:39:59.377 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6147999999999999e-05 2024-09-18T04:39:59.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.377 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.377 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:59.377 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582467+0000", 2024-09-18T04:39:59.377 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6059999999999998e-05 2024-09-18T04:39:59.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.377 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.377 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:59.377 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582505+0000", 2024-09-18T04:39:59.377 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7679000000000003e-05 2024-09-18T04:39:59.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.378 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.378 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:59.378 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593743+0000", 2024-09-18T04:39:59.378 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011238356 2024-09-18T04:39:59.378 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.378 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.378 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:59.378 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593816+0000", 2024-09-18T04:39:59.378 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2935000000000006e-05 2024-09-18T04:39:59.378 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.378 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:59.378 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.378 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.378 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.379 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:53 2.4 e51/17 2:39f7f174:ns::rbd_data.133e813ab5bc.000000000000000b:head v 51'7, mlcod=51'4)", 2024-09-18T04:39:59.379 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:39:59.379 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 76.737674744000003, 2024-09-18T04:39:59.379 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0079914159999999995, 2024-09-18T04:39:59.379 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:59.379 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:59.379 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:59.379 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.379 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:59.379 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:39:59.379 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.379 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.379 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.379 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:59.380 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:39:59.380 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.380 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.380 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:59.380 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594667+0000", 2024-09-18T04:39:59.380 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:59.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.380 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.380 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:59.380 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596446+0000", 2024-09-18T04:39:59.380 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001779033 2024-09-18T04:39:59.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.380 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.381 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:59.381 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596472+0000", 2024-09-18T04:39:59.381 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6120000000000001e-05 2024-09-18T04:39:59.381 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.381 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.381 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:59.381 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596500+0000", 2024-09-18T04:39:59.381 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7518e-05 2024-09-18T04:39:59.381 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.381 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.381 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:59.381 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596564+0000", 2024-09-18T04:39:59.382 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3880999999999994e-05 2024-09-18T04:39:59.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.382 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.382 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:59.382 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596596+0000", 2024-09-18T04:39:59.382 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2638999999999997e-05 2024-09-18T04:39:59.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.382 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.382 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:59.382 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602605+0000", 2024-09-18T04:39:59.382 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0060080699999999999 2024-09-18T04:39:59.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.382 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.382 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:59.383 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602660+0000", 2024-09-18T04:39:59.383 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5396000000000003e-05 2024-09-18T04:39:59.383 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.383 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:59.383 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.383 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.383 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:60 2.5 e51/28 2:a06f51e4:ns::rbd_data.133e813ab5bc.000000000000000f:head v 51'6, mlcod=51'4)", 2024-09-18T04:39:59.383 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:39:59.383 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 76.722857758999993, 2024-09-18T04:39:59.383 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0093954989999999999, 2024-09-18T04:39:59.383 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:59.383 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:59.384 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:59.384 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.384 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:59.384 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:39:59.384 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.384 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.384 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:59.384 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:39:59.384 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.384 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.384 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:59.384 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609483+0000", 2024-09-18T04:39:59.385 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:59.385 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.385 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.385 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:59.385 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611767+0000", 2024-09-18T04:39:59.385 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002283724 2024-09-18T04:39:59.385 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.385 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.385 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:59.385 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611791+0000", 2024-09-18T04:39:59.385 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3547e-05 2024-09-18T04:39:59.385 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.385 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.386 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:59.386 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611800+0000", 2024-09-18T04:39:59.386 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4979999999999995e-06 2024-09-18T04:39:59.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.386 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.386 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:59.386 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611864+0000", 2024-09-18T04:39:59.386 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3974999999999996e-05 2024-09-18T04:39:59.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.386 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.386 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:59.386 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611887+0000", 2024-09-18T04:39:59.386 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2633000000000002e-05 2024-09-18T04:39:59.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.387 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:59.387 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.618777+0000", 2024-09-18T04:39:59.387 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0068897979999999999 2024-09-18T04:39:59.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.387 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:59.387 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.618881+0000", 2024-09-18T04:39:59.387 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010437699999999999 2024-09-18T04:39:59.387 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.387 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:59.387 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.388 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:663 2.4 e57/17 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 57'13, mlcod=57'11)", 2024-09-18T04:39:59.388 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.355300+0000", 2024-09-18T04:39:59.388 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 67.977042823999994, 2024-09-18T04:39:59.388 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0049192760000000002, 2024-09-18T04:39:59.388 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:59.388 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:59.388 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:59.388 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.388 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:59.388 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355300+0000", 2024-09-18T04:39:59.388 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.388 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.389 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:59.389 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355300+0000", 2024-09-18T04:39:59.389 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.389 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.389 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.389 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:59.389 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355299+0000", 2024-09-18T04:39:59.389 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:59.389 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.389 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.389 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:59.390 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355876+0000", 2024-09-18T04:39:59.390 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00057730900000000003 2024-09-18T04:39:59.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.390 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.390 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:59.390 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355897+0000", 2024-09-18T04:39:59.390 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0477000000000001e-05 2024-09-18T04:39:59.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.390 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.390 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:59.390 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355905+0000", 2024-09-18T04:39:59.390 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5739999999999996e-06 2024-09-18T04:39:59.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.390 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.391 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:59.391 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355989+0000", 2024-09-18T04:39:59.391 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4002e-05 2024-09-18T04:39:59.391 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.391 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.391 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:59.391 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.356041+0000", 2024-09-18T04:39:59.391 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1347000000000002e-05 2024-09-18T04:39:59.391 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.391 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.391 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:59.391 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.360172+0000", 2024-09-18T04:39:59.391 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0041315090000000002 2024-09-18T04:39:59.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.392 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.392 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:59.392 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.360220+0000", 2024-09-18T04:39:59.392 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7584999999999999e-05 2024-09-18T04:39:59.392 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.392 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:59.392 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.392 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.392 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:679 2.5 e57/28 2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head v 57'20, mlcod=57'18)", 2024-09-18T04:39:59.392 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.445316+0000", 2024-09-18T04:39:59.392 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 67.887027347, 2024-09-18T04:39:59.393 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0047016760000000001, 2024-09-18T04:39:59.393 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:59.393 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:59.393 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:59.393 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.393 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:59.393 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445316+0000", 2024-09-18T04:39:59.393 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.393 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.393 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:59.393 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445316+0000", 2024-09-18T04:39:59.393 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.394 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.394 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:59.394 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445314+0000", 2024-09-18T04:39:59.394 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:59.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.394 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.394 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:59.394 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445947+0000", 2024-09-18T04:39:59.394 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00063298900000000003 2024-09-18T04:39:59.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.394 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.394 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:59.394 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445968+0000", 2024-09-18T04:39:59.395 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.054e-05 2024-09-18T04:39:59.395 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.395 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.395 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:59.395 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445981+0000", 2024-09-18T04:39:59.395 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2697000000000001e-05 2024-09-18T04:39:59.395 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.395 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.395 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:59.395 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.446102+0000", 2024-09-18T04:39:59.395 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012189300000000001 2024-09-18T04:39:59.395 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.395 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.396 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:59.396 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.446151+0000", 2024-09-18T04:39:59.396 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8279000000000003e-05 2024-09-18T04:39:59.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.396 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.396 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:59.396 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.449953+0000", 2024-09-18T04:39:59.396 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003801979 2024-09-18T04:39:59.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.396 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.396 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:59.396 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.450018+0000", 2024-09-18T04:39:59.396 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4873e-05 2024-09-18T04:39:59.397 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.397 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:59.397 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.397 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.397 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:699 2.4 e57/17 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 57'17, mlcod=57'15)", 2024-09-18T04:39:59.397 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.566073+0000", 2024-09-18T04:39:59.397 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 67.766269833999999, 2024-09-18T04:39:59.397 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055398380000000001, 2024-09-18T04:39:59.397 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:59.397 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:59.397 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:59.397 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.398 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:59.398 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566073+0000", 2024-09-18T04:39:59.398 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.398 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.398 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.398 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:59.398 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566073+0000", 2024-09-18T04:39:59.398 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.398 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.398 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.398 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:59.398 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566071+0000", 2024-09-18T04:39:59.398 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:59.398 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.399 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.399 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:59.399 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566637+0000", 2024-09-18T04:39:59.399 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00056516699999999999 2024-09-18T04:39:59.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.399 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.399 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:59.399 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566668+0000", 2024-09-18T04:39:59.399 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1544999999999997e-05 2024-09-18T04:39:59.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.399 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.399 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:59.399 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566685+0000", 2024-09-18T04:39:59.399 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6857000000000001e-05 2024-09-18T04:39:59.400 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.400 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.400 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:59.400 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566816+0000", 2024-09-18T04:39:59.400 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013092799999999999 2024-09-18T04:39:59.400 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.400 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.400 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:59.400 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566875+0000", 2024-09-18T04:39:59.400 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9583999999999997e-05 2024-09-18T04:39:59.400 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.400 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.400 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:59.400 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.571567+0000", 2024-09-18T04:39:59.401 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.004691682 2024-09-18T04:39:59.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.401 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.401 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:59.401 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.571613+0000", 2024-09-18T04:39:59.401 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6125000000000003e-05 2024-09-18T04:39:59.401 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.401 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:59.401 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.401 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.401 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:98 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'27, mlcod=60'25)", 2024-09-18T04:39:59.401 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.847978+0000", 2024-09-18T04:39:59.402 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 58.484365275000002, 2024-09-18T04:39:59.402 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0044617709999999998, 2024-09-18T04:39:59.402 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:59.402 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:59.402 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:59.402 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.402 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:59.402 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.847978+0000", 2024-09-18T04:39:59.402 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.402 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.402 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.402 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:59.402 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.847978+0000", 2024-09-18T04:39:59.402 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.403 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.403 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:59.403 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.847977+0000", 2024-09-18T04:39:59.403 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:59.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.403 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.403 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:59.403 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849038+0000", 2024-09-18T04:39:59.403 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0010614820000000001 2024-09-18T04:39:59.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.403 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.403 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:59.404 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849065+0000", 2024-09-18T04:39:59.404 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7127999999999998e-05 2024-09-18T04:39:59.404 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.404 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.404 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:59.404 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849082+0000", 2024-09-18T04:39:59.404 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7170999999999999e-05 2024-09-18T04:39:59.404 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.404 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.404 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:59.404 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849188+0000", 2024-09-18T04:39:59.404 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010605800000000001 2024-09-18T04:39:59.404 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.404 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.405 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:59.405 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849233+0000", 2024-09-18T04:39:59.405 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4579000000000001e-05 2024-09-18T04:39:59.405 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.405 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.405 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:59.405 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.852370+0000", 2024-09-18T04:39:59.405 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00313751 2024-09-18T04:39:59.405 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.405 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.405 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:59.405 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.852440+0000", 2024-09-18T04:39:59.405 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9238000000000004e-05 2024-09-18T04:39:59.405 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.406 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:59.406 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.406 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.406 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.406 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:100 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'29, mlcod=64'27)", 2024-09-18T04:39:59.406 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.858766+0000", 2024-09-18T04:39:59.406 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 58.473576946000001, 2024-09-18T04:39:59.406 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0061964840000000004, 2024-09-18T04:39:59.406 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:59.406 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:59.406 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:59.406 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.406 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:59.406 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.858766+0000", 2024-09-18T04:39:59.407 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.407 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.407 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.407 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:59.407 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.858766+0000", 2024-09-18T04:39:59.407 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.407 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.407 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.407 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:59.407 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.858765+0000", 2024-09-18T04:39:59.407 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:59.407 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.407 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.407 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:59.407 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861612+0000", 2024-09-18T04:39:59.408 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028473169999999998 2024-09-18T04:39:59.408 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.408 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.408 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:59.408 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861643+0000", 2024-09-18T04:39:59.408 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1356000000000003e-05 2024-09-18T04:39:59.408 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.408 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.408 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:59.408 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861656+0000", 2024-09-18T04:39:59.408 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2899e-05 2024-09-18T04:39:59.408 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.408 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.408 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:59.409 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861773+0000", 2024-09-18T04:39:59.409 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011637999999999999 2024-09-18T04:39:59.409 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.409 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.409 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:59.409 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861808+0000", 2024-09-18T04:39:59.409 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5002999999999997e-05 2024-09-18T04:39:59.409 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.409 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.409 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:59.409 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.864876+0000", 2024-09-18T04:39:59.409 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030680719999999998 2024-09-18T04:39:59.409 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.409 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.410 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:59.410 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.864963+0000", 2024-09-18T04:39:59.410 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7114000000000007e-05 2024-09-18T04:39:59.410 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.410 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:59.410 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.410 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.410 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.410 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:102 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'31, mlcod=64'29)", 2024-09-18T04:39:59.410 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.872098+0000", 2024-09-18T04:39:59.410 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 58.460245688999997, 2024-09-18T04:39:59.410 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0064676450000000002, 2024-09-18T04:39:59.410 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:59.411 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:59.411 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:59.411 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.411 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:59.411 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.872098+0000", 2024-09-18T04:39:59.411 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.411 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.411 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.411 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:59.411 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.872098+0000", 2024-09-18T04:39:59.411 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.411 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.411 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.411 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:59.411 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.872096+0000", 2024-09-18T04:39:59.412 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:59.412 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.412 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.412 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:59.412 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875176+0000", 2024-09-18T04:39:59.412 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030799809999999999 2024-09-18T04:39:59.412 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.412 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.412 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:59.412 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875204+0000", 2024-09-18T04:39:59.412 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8025999999999999e-05 2024-09-18T04:39:59.412 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.412 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.412 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:59.413 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875216+0000", 2024-09-18T04:39:59.413 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2072999999999999e-05 2024-09-18T04:39:59.413 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.413 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.413 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:59.413 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875307+0000", 2024-09-18T04:39:59.413 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0373999999999999e-05 2024-09-18T04:39:59.413 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.413 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.413 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:59.413 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875349+0000", 2024-09-18T04:39:59.413 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2299000000000003e-05 2024-09-18T04:39:59.413 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.414 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.414 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:59.414 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.878496+0000", 2024-09-18T04:39:59.414 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0031468960000000002 2024-09-18T04:39:59.414 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.414 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.414 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:59.414 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.878565+0000", 2024-09-18T04:39:59.414 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9465e-05 2024-09-18T04:39:59.414 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.414 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:59.414 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.414 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.414 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.414 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:104 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'33, mlcod=64'31)", 2024-09-18T04:39:59.414 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.885100+0000", 2024-09-18T04:39:59.415 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 58.447243153999999, 2024-09-18T04:39:59.415 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0051888159999999997, 2024-09-18T04:39:59.415 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:59.415 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:59.415 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:59.415 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.415 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:59.415 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.885100+0000", 2024-09-18T04:39:59.415 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.415 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.415 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.415 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:59.415 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.885100+0000", 2024-09-18T04:39:59.415 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.415 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.416 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.416 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:59.416 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.885098+0000", 2024-09-18T04:39:59.416 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:59.416 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.416 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.416 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:59.416 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886809+0000", 2024-09-18T04:39:59.416 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017110459999999999 2024-09-18T04:39:59.416 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.416 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.416 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:59.416 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886842+0000", 2024-09-18T04:39:59.416 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2638999999999997e-05 2024-09-18T04:39:59.417 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.417 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.417 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:59.417 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886855+0000", 2024-09-18T04:39:59.417 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3380000000000001e-05 2024-09-18T04:39:59.417 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.417 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.417 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:59.417 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886939+0000", 2024-09-18T04:39:59.417 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3733999999999995e-05 2024-09-18T04:39:59.417 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.417 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.417 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:59.417 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886981+0000", 2024-09-18T04:39:59.418 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1956000000000003e-05 2024-09-18T04:39:59.418 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.418 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.418 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:59.418 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.890201+0000", 2024-09-18T04:39:59.418 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003219927 2024-09-18T04:39:59.418 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.418 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.418 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:59.418 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.890289+0000", 2024-09-18T04:39:59.418 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7997999999999993e-05 2024-09-18T04:39:59.418 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.418 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:59.418 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.419 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.419 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.419 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:145 2.4 e64/62 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 64'37, mlcod=64'35)", 2024-09-18T04:39:59.419 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:01.099398+0000", 2024-09-18T04:39:59.419 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 58.232945665999999, 2024-09-18T04:39:59.419 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0050078459999999998, 2024-09-18T04:39:59.419 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:59.419 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:59.419 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:59.419 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.419 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:59.419 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.099398+0000", 2024-09-18T04:39:59.419 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.419 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.420 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.420 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:59.420 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.099398+0000", 2024-09-18T04:39:59.420 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.420 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.420 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.420 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:59.420 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.099396+0000", 2024-09-18T04:39:59.420 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:59.420 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.420 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.420 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:59.420 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100199+0000", 2024-09-18T04:39:59.421 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00080338000000000002 2024-09-18T04:39:59.421 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.421 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.421 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:59.421 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100230+0000", 2024-09-18T04:39:59.421 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0716000000000002e-05 2024-09-18T04:39:59.421 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.421 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.421 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:59.421 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100243+0000", 2024-09-18T04:39:59.421 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3359e-05 2024-09-18T04:39:59.421 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.421 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.421 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:59.422 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100337+0000", 2024-09-18T04:39:59.422 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4387000000000003e-05 2024-09-18T04:39:59.422 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.422 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.422 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:59.422 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100396+0000", 2024-09-18T04:39:59.422 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8164000000000001e-05 2024-09-18T04:39:59.422 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.422 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.422 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:59.422 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.104309+0000", 2024-09-18T04:39:59.422 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0039137800000000004 2024-09-18T04:39:59.422 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.422 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.423 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:59.423 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.104405+0000", 2024-09-18T04:39:59.423 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6039999999999995e-05 2024-09-18T04:39:59.423 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.423 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:59.423 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.423 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.423 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:147 2.4 e64/62 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 64'39, mlcod=64'37)", 2024-09-18T04:39:59.423 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:01.110658+0000", 2024-09-18T04:39:59.423 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 58.221685577000002, 2024-09-18T04:39:59.423 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0046316910000000003, 2024-09-18T04:39:59.424 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:59.424 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:59.424 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:59.424 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.424 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:59.424 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.110658+0000", 2024-09-18T04:39:59.424 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.424 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.424 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.424 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:59.424 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.110658+0000", 2024-09-18T04:39:59.424 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.424 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.424 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.425 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:59.425 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.110656+0000", 2024-09-18T04:39:59.425 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:59.425 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.425 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.425 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:59.425 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111444+0000", 2024-09-18T04:39:59.425 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00078762400000000001 2024-09-18T04:39:59.425 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.425 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.425 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:59.425 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111474+0000", 2024-09-18T04:39:59.425 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0085000000000001e-05 2024-09-18T04:39:59.425 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.426 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.426 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:59.426 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111486+0000", 2024-09-18T04:39:59.426 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2645e-05 2024-09-18T04:39:59.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.426 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.426 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:59.426 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111581+0000", 2024-09-18T04:39:59.426 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4432000000000007e-05 2024-09-18T04:39:59.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.427 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.427 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:59.427 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111631+0000", 2024-09-18T04:39:59.427 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9901000000000002e-05 2024-09-18T04:39:59.427 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.427 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.427 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:59.427 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.115209+0000", 2024-09-18T04:39:59.427 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003577916 2024-09-18T04:39:59.427 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.427 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.427 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:59.427 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.115289+0000", 2024-09-18T04:39:59.428 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0704999999999994e-05 2024-09-18T04:39:59.428 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.428 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:59.428 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.428 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.428 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:59.428 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:59.429 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:59.332+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:59.429 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:39:59.437 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:59.431+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:39:59.538 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:39:59.532+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:39:59.542 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:59.542 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:59.542 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:39:59.542 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:59.552 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:39:59.639 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:39:59.636+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:39:59.663 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:59.663 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:39:59.663 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:39:59.663 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:39:59.663 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:59.673 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:39:59.740 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:39:59.735+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:39:59.790 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:39:59.790 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:39:59.790 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:39:59.790 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:39:59.790 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.790 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:33 2.1 e51/24 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head v 51'4, mlcod=51'2)", 2024-09-18T04:39:59.790 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:39:59.790 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 77.507250025999994, 2024-09-18T04:39:59.790 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018702106, 2024-09-18T04:39:59.791 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:59.791 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:59.791 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:59.791 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.791 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:59.791 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:39:59.791 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.791 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.791 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.791 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:59.791 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:39:59.791 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.791 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.792 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.792 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:59.792 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279197+0000", 2024-09-18T04:39:59.792 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:59.792 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.792 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.792 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:59.792 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282682+0000", 2024-09-18T04:39:59.792 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0034848689999999998 2024-09-18T04:39:59.792 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.792 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.792 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:59.792 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282709+0000", 2024-09-18T04:39:59.792 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7599000000000001e-05 2024-09-18T04:39:59.793 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.793 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.793 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:59.793 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282720+0000", 2024-09-18T04:39:59.793 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.092e-05 2024-09-18T04:39:59.793 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.793 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.793 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:59.793 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282791+0000", 2024-09-18T04:39:59.793 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1268999999999998e-05 2024-09-18T04:39:59.793 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.793 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.793 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:59.793 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282843+0000", 2024-09-18T04:39:59.793 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1771999999999998e-05 2024-09-18T04:39:59.794 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.794 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.794 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:59.794 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.297830+0000", 2024-09-18T04:39:59.794 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014987228999999999 2024-09-18T04:39:59.794 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.794 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.794 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:59.794 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.297901+0000", 2024-09-18T04:39:59.794 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0209999999999994e-05 2024-09-18T04:39:59.794 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.794 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:59.794 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.794 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.795 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.795 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:38 2.0 e51/17 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head v 51'10, mlcod=51'8)", 2024-09-18T04:39:59.795 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:39:59.795 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 77.505831595999993, 2024-09-18T04:39:59.795 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.023984472, 2024-09-18T04:39:59.795 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:59.795 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:59.795 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:59.795 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.795 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:59.795 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:39:59.795 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.795 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.795 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.796 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:59.796 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:39:59.796 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.796 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.796 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.796 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:59.796 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280614+0000", 2024-09-18T04:39:59.796 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:59.796 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.796 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.796 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:59.796 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293094+0000", 2024-09-18T04:39:59.796 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012479571 2024-09-18T04:39:59.796 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.797 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.797 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:59.797 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293111+0000", 2024-09-18T04:39:59.797 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7354999999999999e-05 2024-09-18T04:39:59.797 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.797 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.797 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:59.797 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293121+0000", 2024-09-18T04:39:59.797 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6979999999999993e-06 2024-09-18T04:39:59.797 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.797 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.797 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:59.797 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293180+0000", 2024-09-18T04:39:59.797 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9333000000000003e-05 2024-09-18T04:39:59.797 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.798 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.798 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:59.798 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293203+0000", 2024-09-18T04:39:59.798 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2633000000000002e-05 2024-09-18T04:39:59.798 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.798 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.798 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:59.798 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304534+0000", 2024-09-18T04:39:59.798 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011330761 2024-09-18T04:39:59.798 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.798 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.798 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:59.798 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304601+0000", 2024-09-18T04:39:59.798 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7657000000000006e-05 2024-09-18T04:39:59.799 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.799 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:59.799 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.799 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.799 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.799 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:46 2.6 2:7ba25245:::rbd_data.125fc09df6d1.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:59.799 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:39:59.799 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 77.478288796000001, 2024-09-18T04:39:59.799 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.035604595000000003, 2024-09-18T04:39:59.799 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:59.799 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:59.799 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:59.799 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:59.799 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:59.799 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 46 2024-09-18T04:39:59.800 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.800 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:59.800 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.800 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:59.800 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:39:59.800 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.800 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.800 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.800 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:59.800 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:39:59.800 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.800 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.800 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.800 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:59.801 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308158+0000", 2024-09-18T04:39:59.801 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:59.801 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.801 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.801 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:59.801 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313191+0000", 2024-09-18T04:39:59.801 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0050323929999999996 2024-09-18T04:39:59.801 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.801 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.801 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:59.801 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313210+0000", 2024-09-18T04:39:59.801 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9673e-05 2024-09-18T04:39:59.801 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.801 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.801 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:59.802 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313221+0000", 2024-09-18T04:39:59.802 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0444e-05 2024-09-18T04:39:59.802 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.802 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.802 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:59.802 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313294+0000", 2024-09-18T04:39:59.802 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2650000000000004e-05 2024-09-18T04:39:59.802 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.802 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.802 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:59.802 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313493+0000", 2024-09-18T04:39:59.802 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019968300000000001 2024-09-18T04:39:59.802 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.802 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.803 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:59.803 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313666+0000", 2024-09-18T04:39:59.803 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00017248599999999999 2024-09-18T04:39:59.803 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.803 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.803 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:59.803 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.323175+0000", 2024-09-18T04:39:59.803 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0095088819999999998 2024-09-18T04:39:59.803 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.803 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.803 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:59.803 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343651+0000", 2024-09-18T04:39:59.803 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020475892999999998 2024-09-18T04:39:59.803 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.803 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.804 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:59.804 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343715+0000", 2024-09-18T04:39:59.804 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4443999999999995e-05 2024-09-18T04:39:59.804 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.804 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.804 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:59.804 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343764+0000", 2024-09-18T04:39:59.804 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9280000000000003e-05 2024-09-18T04:39:59.804 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.804 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:59.804 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.804 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.804 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.805 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:47 2.4 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:59.805 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:39:59.805 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 77.473206758999993, 2024-09-18T04:39:59.805 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.032570693999999997, 2024-09-18T04:39:59.805 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:59.805 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:59.805 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:59.805 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:59.805 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:59.805 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 47 2024-09-18T04:39:59.805 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.805 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:59.805 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.805 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:59.805 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:39:59.806 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.806 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.806 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.806 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:59.806 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:39:59.806 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.806 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.806 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.806 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:59.806 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313240+0000", 2024-09-18T04:39:59.806 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:59.806 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.806 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.807 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:59.807 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325825+0000", 2024-09-18T04:39:59.807 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012584425999999999 2024-09-18T04:39:59.807 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.807 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.807 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:59.807 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325848+0000", 2024-09-18T04:39:59.807 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3025999999999999e-05 2024-09-18T04:39:59.807 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.807 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.807 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:59.807 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325863+0000", 2024-09-18T04:39:59.807 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5152e-05 2024-09-18T04:39:59.807 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.808 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.808 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:59.808 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325938+0000", 2024-09-18T04:39:59.808 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4982999999999997e-05 2024-09-18T04:39:59.808 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.808 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.808 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:59.808 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326090+0000", 2024-09-18T04:39:59.808 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015207900000000001 2024-09-18T04:39:59.808 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.808 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.808 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:59.808 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326242+0000", 2024-09-18T04:39:59.808 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015163500000000001 2024-09-18T04:39:59.809 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.809 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.809 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:59.809 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343015+0000", 2024-09-18T04:39:59.809 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016773414 2024-09-18T04:39:59.809 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.809 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.809 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:59.809 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345664+0000", 2024-09-18T04:39:59.809 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026487870000000001 2024-09-18T04:39:59.809 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.809 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.809 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:59.809 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345739+0000", 2024-09-18T04:39:59.810 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5018999999999994e-05 2024-09-18T04:39:59.810 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.810 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.810 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:59.810 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345812+0000", 2024-09-18T04:39:59.810 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3547000000000006e-05 2024-09-18T04:39:59.810 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.810 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:59.810 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.810 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.810 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.810 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:48 2.6 2:61db7a3a:::rbd_data.125fc09df6d1.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:59.810 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:39:59.810 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 77.460550087000001, 2024-09-18T04:39:59.810 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.025064388, 2024-09-18T04:39:59.811 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:59.811 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:59.811 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:59.811 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:59.811 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:59.811 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 48 2024-09-18T04:39:59.811 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.811 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:59.811 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.811 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:59.811 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:39:59.811 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.811 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.811 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.812 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:59.812 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:39:59.812 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.812 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.812 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.812 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:59.812 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325897+0000", 2024-09-18T04:39:59.812 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:59.812 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.812 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.812 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:59.812 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333697+0000", 2024-09-18T04:39:59.812 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0078002649999999998 2024-09-18T04:39:59.812 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.813 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.813 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:59.813 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333716+0000", 2024-09-18T04:39:59.813 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8910999999999999e-05 2024-09-18T04:39:59.813 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.813 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.813 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:59.813 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333728+0000", 2024-09-18T04:39:59.813 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1936000000000001e-05 2024-09-18T04:39:59.813 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.813 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.813 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:59.813 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333792+0000", 2024-09-18T04:39:59.814 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4010999999999994e-05 2024-09-18T04:39:59.814 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.814 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.814 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:59.814 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.334082+0000", 2024-09-18T04:39:59.814 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00028955799999999998 2024-09-18T04:39:59.814 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.814 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.814 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:59.814 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.334326+0000", 2024-09-18T04:39:59.814 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00024399099999999999 2024-09-18T04:39:59.814 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.814 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.814 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:59.815 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350745+0000", 2024-09-18T04:39:59.815 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016419474999999999 2024-09-18T04:39:59.815 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.815 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.815 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:59.815 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350808+0000", 2024-09-18T04:39:59.815 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3244999999999995e-05 2024-09-18T04:39:59.815 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.815 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.815 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:59.815 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350892+0000", 2024-09-18T04:39:59.815 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4074999999999997e-05 2024-09-18T04:39:59.815 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.815 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.815 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:59.816 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350963+0000", 2024-09-18T04:39:59.816 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0412000000000003e-05 2024-09-18T04:39:59.816 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.816 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:59.816 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.816 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.816 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.816 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:51 2.6 2:775d7b11:::rbd_data.125fc09df6d1.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:59.816 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:39:59.816 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 77.452697951000005, 2024-09-18T04:39:59.816 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.023824221, 2024-09-18T04:39:59.816 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:59.816 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:59.817 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:59.817 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:59.817 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:59.817 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 51 2024-09-18T04:39:59.817 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.817 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:59.817 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.817 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:59.817 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:39:59.817 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.817 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.817 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.817 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:59.817 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:39:59.818 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.818 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.818 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.818 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:59.818 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333749+0000", 2024-09-18T04:39:59.818 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:59.818 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.818 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.818 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:59.818 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340899+0000", 2024-09-18T04:39:59.818 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0071500340000000004 2024-09-18T04:39:59.818 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.818 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.818 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:59.819 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340929+0000", 2024-09-18T04:39:59.819 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0093000000000001e-05 2024-09-18T04:39:59.819 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.819 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.819 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:59.819 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340943+0000", 2024-09-18T04:39:59.819 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4019000000000001e-05 2024-09-18T04:39:59.819 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.819 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.819 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:59.819 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343834+0000", 2024-09-18T04:39:59.819 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028918120000000001 2024-09-18T04:39:59.819 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.819 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.820 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:59.820 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344020+0000", 2024-09-18T04:39:59.820 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000185892 2024-09-18T04:39:59.820 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.820 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.820 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:59.820 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344190+0000", 2024-09-18T04:39:59.820 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016946800000000001 2024-09-18T04:39:59.820 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.820 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.820 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:59.820 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.351291+0000", 2024-09-18T04:39:59.820 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.007100805 2024-09-18T04:39:59.821 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.821 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.821 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:59.821 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357470+0000", 2024-09-18T04:39:59.821 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0061797570000000001 2024-09-18T04:39:59.821 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.821 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.821 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:59.821 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357533+0000", 2024-09-18T04:39:59.821 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2906999999999999e-05 2024-09-18T04:39:59.821 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.821 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.821 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:59.821 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357575+0000", 2024-09-18T04:39:59.822 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1403999999999999e-05 2024-09-18T04:39:59.822 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.822 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:59.822 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.822 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.822 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.822 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:56 2.4 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:59.822 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:39:59.822 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 77.436652858000002, 2024-09-18T04:39:59.822 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.034295870999999999, 2024-09-18T04:39:59.822 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:59.822 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:59.822 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:59.822 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:39:59.823 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:39:59.823 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 56 2024-09-18T04:39:59.823 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.823 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:59.823 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.823 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:59.823 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:39:59.823 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.823 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.823 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.823 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:59.823 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:39:59.823 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.823 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.824 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.824 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:59.824 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349794+0000", 2024-09-18T04:39:59.824 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:59.824 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.824 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.824 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:59.824 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366632+0000", 2024-09-18T04:39:59.824 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01683755 2024-09-18T04:39:59.824 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.824 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.824 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:59.824 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366669+0000", 2024-09-18T04:39:59.825 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7351000000000002e-05 2024-09-18T04:39:59.825 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.825 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.825 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:59.825 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366686+0000", 2024-09-18T04:39:59.825 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6507999999999998e-05 2024-09-18T04:39:59.825 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.825 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.825 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:59.825 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366962+0000", 2024-09-18T04:39:59.825 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00027655900000000001 2024-09-18T04:39:59.825 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.825 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.825 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:59.826 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.367452+0000", 2024-09-18T04:39:59.826 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00049025299999999996 2024-09-18T04:39:59.826 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.826 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.826 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:59.826 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368113+0000", 2024-09-18T04:39:59.826 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00066093399999999996 2024-09-18T04:39:59.826 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.826 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.826 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:59.826 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383719+0000", 2024-09-18T04:39:59.826 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015606106999999999 2024-09-18T04:39:59.826 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.826 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.826 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:59.827 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384035+0000", 2024-09-18T04:39:59.827 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00031590399999999999 2024-09-18T04:39:59.827 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.827 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.827 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:59.827 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384067+0000", 2024-09-18T04:39:59.827 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.129e-05 2024-09-18T04:39:59.827 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.827 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.827 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:59.827 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384091+0000", 2024-09-18T04:39:59.827 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4788000000000001e-05 2024-09-18T04:39:59.827 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.827 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:59.827 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.828 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.828 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.828 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:57 2.0 e51/17 2:0cdd070e:::rbd_data.125fc09df6d1.000000000000000d:head v 51'21, mlcod=51'19)", 2024-09-18T04:39:59.828 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:39:59.828 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 77.428825371000002, 2024-09-18T04:39:59.828 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.025216311000000002, 2024-09-18T04:39:59.828 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:59.828 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:59.828 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:59.828 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.828 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:59.828 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:39:59.828 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.828 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.829 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.829 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:59.829 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:39:59.829 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.829 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.829 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.829 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:59.829 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:39:59.829 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:39:59.829 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.829 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.829 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:59.829 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363844+0000", 2024-09-18T04:39:59.829 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0062218450000000002 2024-09-18T04:39:59.829 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.830 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.830 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:59.830 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363879+0000", 2024-09-18T04:39:59.830 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4382999999999999e-05 2024-09-18T04:39:59.830 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.830 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.830 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:59.830 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363893+0000", 2024-09-18T04:39:59.830 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4348999999999999e-05 2024-09-18T04:39:59.830 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.830 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.830 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:59.830 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363977+0000", 2024-09-18T04:39:59.831 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3499999999999997e-05 2024-09-18T04:39:59.831 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.831 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.831 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:59.831 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364009+0000", 2024-09-18T04:39:59.831 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2292000000000002e-05 2024-09-18T04:39:59.831 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.831 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.831 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:59.831 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.382781+0000", 2024-09-18T04:39:59.831 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018771626 2024-09-18T04:39:59.831 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.831 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.831 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:59.832 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.382839+0000", 2024-09-18T04:39:59.832 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8758000000000002e-05 2024-09-18T04:39:59.832 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.832 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:59.832 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.832 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.832 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.832 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:59 2.0 e51/17 2:19c52cde:::rbd_data.125fc09df6d1.000000000000000f:head v 51'22, mlcod=51'19)", 2024-09-18T04:39:59.832 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:39:59.832 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 77.422165445000005, 2024-09-18T04:39:59.832 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.019497112, 2024-09-18T04:39:59.832 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:59.832 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:59.833 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:59.833 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.833 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:59.833 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:39:59.833 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.833 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.833 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.833 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:59.833 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:39:59.833 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.833 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.833 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.833 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:59.833 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364282+0000", 2024-09-18T04:39:59.834 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:59.834 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.834 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.834 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:59.834 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371823+0000", 2024-09-18T04:39:59.834 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0075411829999999999 2024-09-18T04:39:59.834 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.834 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.834 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:59.834 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371837+0000", 2024-09-18T04:39:59.834 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4336e-05 2024-09-18T04:39:59.834 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.834 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.834 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:59.834 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371846+0000", 2024-09-18T04:39:59.835 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9770000000000006e-06 2024-09-18T04:39:59.835 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.835 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.835 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:59.835 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.373566+0000", 2024-09-18T04:39:59.835 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017192539999999999 2024-09-18T04:39:59.835 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.835 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.835 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:59.835 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.373595+0000", 2024-09-18T04:39:59.835 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8975999999999998e-05 2024-09-18T04:39:59.835 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.835 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.835 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:59.836 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383740+0000", 2024-09-18T04:39:59.836 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01014525 2024-09-18T04:39:59.836 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.836 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.836 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:59.836 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383780+0000", 2024-09-18T04:39:59.836 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0275999999999999e-05 2024-09-18T04:39:59.836 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.836 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:59.836 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.836 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.836 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.836 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:34 2.6 2:66289171:ns::rbd_data.133e813ab5bc.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:59.836 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.524015+0000", 2024-09-18T04:39:59.837 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 77.262433836, 2024-09-18T04:39:59.837 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027582111999999999, 2024-09-18T04:39:59.837 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:59.837 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:59.837 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:59.837 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:59.837 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:59.837 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 34 2024-09-18T04:39:59.837 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.837 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:59.837 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.837 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:59.837 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524015+0000", 2024-09-18T04:39:59.837 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.838 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.838 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.838 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:59.838 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524015+0000", 2024-09-18T04:39:59.838 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.838 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.838 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.838 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:59.838 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524012+0000", 2024-09-18T04:39:59.838 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:59.838 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.838 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.838 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:59.838 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534956+0000", 2024-09-18T04:39:59.839 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010943403000000001 2024-09-18T04:39:59.839 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.839 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.839 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:59.839 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534978+0000", 2024-09-18T04:39:59.839 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1965999999999999e-05 2024-09-18T04:39:59.839 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.839 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.839 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:59.839 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534990+0000", 2024-09-18T04:39:59.839 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2065999999999999e-05 2024-09-18T04:39:59.839 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.839 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.840 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:59.840 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535065+0000", 2024-09-18T04:39:59.840 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5523000000000002e-05 2024-09-18T04:39:59.840 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.840 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.840 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:59.840 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535208+0000", 2024-09-18T04:39:59.840 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014309500000000001 2024-09-18T04:39:59.840 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.840 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.840 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:59.840 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535343+0000", 2024-09-18T04:39:59.840 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013437400000000001 2024-09-18T04:39:59.840 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.841 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.841 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:59.841 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.546576+0000", 2024-09-18T04:39:59.841 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011233425999999999 2024-09-18T04:39:59.841 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.841 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.841 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:59.841 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551498+0000", 2024-09-18T04:39:59.841 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0049219349999999997 2024-09-18T04:39:59.841 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.841 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.841 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:59.841 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551554+0000", 2024-09-18T04:39:59.841 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6400999999999998e-05 2024-09-18T04:39:59.842 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.842 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.842 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:59.842 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551597+0000", 2024-09-18T04:39:59.842 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2283000000000002e-05 2024-09-18T04:39:59.842 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.842 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:59.842 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.842 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.842 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.842 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:33 2.1 e51/24 2:8b40f1c5:ns::rbd_data.133e813ab5bc.0000000000000000:head v 51'7, mlcod=51'5)", 2024-09-18T04:39:59.842 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.531767+0000", 2024-09-18T04:39:59.842 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 77.254681919000006, 2024-09-18T04:39:59.842 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015370295000000001, 2024-09-18T04:39:59.843 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:59.843 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:59.843 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:59.843 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.843 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:59.843 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531767+0000", 2024-09-18T04:39:59.843 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.843 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.843 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.843 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:59.843 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531767+0000", 2024-09-18T04:39:59.843 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.843 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.843 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.844 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:59.844 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531765+0000", 2024-09-18T04:39:59.844 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:59.844 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.844 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.844 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:59.844 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534258+0000", 2024-09-18T04:39:59.844 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002493561 2024-09-18T04:39:59.844 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.844 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.844 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:59.844 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534288+0000", 2024-09-18T04:39:59.845 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.976e-05 2024-09-18T04:39:59.845 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.845 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.845 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:59.845 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534300+0000", 2024-09-18T04:39:59.845 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1928e-05 2024-09-18T04:39:59.845 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.845 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.845 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:59.845 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534367+0000", 2024-09-18T04:39:59.845 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7045999999999995e-05 2024-09-18T04:39:59.845 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.846 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.846 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:59.846 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534394+0000", 2024-09-18T04:39:59.846 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6903999999999999e-05 2024-09-18T04:39:59.846 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.846 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.846 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:59.846 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.547073+0000", 2024-09-18T04:39:59.846 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012679365 2024-09-18T04:39:59.846 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.846 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.846 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:59.847 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.547137+0000", 2024-09-18T04:39:59.847 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3691999999999999e-05 2024-09-18T04:39:59.847 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.847 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:59.847 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.847 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.847 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.847 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:39 2.4 2:304364d1:ns::rbd_data.133e813ab5bc.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:59.847 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.535013+0000", 2024-09-18T04:39:59.847 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 77.251434983999999, 2024-09-18T04:39:59.847 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027764487000000001, 2024-09-18T04:39:59.847 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:59.847 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:59.848 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:59.848 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:59.848 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:59.848 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 39 2024-09-18T04:39:59.848 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.848 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:59.848 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.848 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:59.848 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535013+0000", 2024-09-18T04:39:59.848 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.848 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.848 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.848 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:59.849 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535013+0000", 2024-09-18T04:39:59.849 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.849 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.849 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.849 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:59.849 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535012+0000", 2024-09-18T04:39:59.849 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:59.849 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.849 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.849 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:59.849 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545016+0000", 2024-09-18T04:39:59.849 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01000322 2024-09-18T04:39:59.849 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.849 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.850 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:59.850 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545035+0000", 2024-09-18T04:39:59.850 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9845999999999999e-05 2024-09-18T04:39:59.850 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.850 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.850 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:59.850 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545045+0000", 2024-09-18T04:39:59.850 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3640000000000005e-06 2024-09-18T04:39:59.850 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.850 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.850 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:59.850 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545152+0000", 2024-09-18T04:39:59.850 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000107025 2024-09-18T04:39:59.850 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.851 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.851 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:59.851 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545338+0000", 2024-09-18T04:39:59.851 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018643500000000001 2024-09-18T04:39:59.851 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.851 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.851 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:59.851 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545492+0000", 2024-09-18T04:39:59.851 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015393199999999999 2024-09-18T04:39:59.851 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.851 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.851 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:59.851 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.553549+0000", 2024-09-18T04:39:59.851 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0080569170000000002 2024-09-18T04:39:59.851 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.852 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.852 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:59.852 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562693+0000", 2024-09-18T04:39:59.852 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0091441609999999996 2024-09-18T04:39:59.852 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.852 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.852 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:59.852 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562748+0000", 2024-09-18T04:39:59.852 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5047000000000003e-05 2024-09-18T04:39:59.852 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.852 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.852 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:59.852 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562778+0000", 2024-09-18T04:39:59.852 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9652e-05 2024-09-18T04:39:59.853 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.853 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:59.853 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.853 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.853 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.853 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:40 2.6 2:784d6cd1:ns::rbd_data.133e813ab5bc.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:59.853 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.545062+0000", 2024-09-18T04:39:59.853 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 77.241385933000004, 2024-09-18T04:39:59.853 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.032771623, 2024-09-18T04:39:59.853 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:59.853 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:59.853 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:59.853 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:59.853 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:59.853 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 40 2024-09-18T04:39:59.854 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.854 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:59.854 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.854 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:59.854 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545062+0000", 2024-09-18T04:39:59.854 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.854 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.854 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.854 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:59.854 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545062+0000", 2024-09-18T04:39:59.854 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.854 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.854 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.855 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:59.855 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545061+0000", 2024-09-18T04:39:59.855 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:59.855 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.855 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.855 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:59.855 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552477+0000", 2024-09-18T04:39:59.855 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.007415914 2024-09-18T04:39:59.855 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.855 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.855 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:59.855 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552500+0000", 2024-09-18T04:39:59.855 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3546000000000001e-05 2024-09-18T04:39:59.855 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.856 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.856 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:59.856 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552515+0000", 2024-09-18T04:39:59.856 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4606999999999999e-05 2024-09-18T04:39:59.856 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.856 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.856 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:59.856 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552661+0000", 2024-09-18T04:39:59.856 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000146007 2024-09-18T04:39:59.856 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.856 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.856 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:59.856 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552876+0000", 2024-09-18T04:39:59.856 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000214618 2024-09-18T04:39:59.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.857 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.857 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:59.857 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.553061+0000", 2024-09-18T04:39:59.857 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018564100000000001 2024-09-18T04:39:59.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.857 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.857 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:59.857 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.565612+0000", 2024-09-18T04:39:59.857 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012550301999999999 2024-09-18T04:39:59.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.857 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.857 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:59.857 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577695+0000", 2024-09-18T04:39:59.858 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012083493000000001 2024-09-18T04:39:59.858 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.858 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.858 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:59.858 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577772+0000", 2024-09-18T04:39:59.858 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7158999999999998e-05 2024-09-18T04:39:59.858 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.858 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.858 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:59.858 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577834+0000", 2024-09-18T04:39:59.858 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1851000000000001e-05 2024-09-18T04:39:59.858 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.858 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:59.858 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.859 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.859 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.859 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:43 2.6 2:759f27b6:ns::rbd_data.133e813ab5bc.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:59.859 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.555868+0000", 2024-09-18T04:39:59.859 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 77.230580438999993, 2024-09-18T04:39:59.859 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.022332497999999999, 2024-09-18T04:39:59.859 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:59.859 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:59.859 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:59.859 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:59.859 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:59.859 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 43 2024-09-18T04:39:59.859 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.859 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:59.860 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.860 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:59.860 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555868+0000", 2024-09-18T04:39:59.860 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.860 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.860 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.860 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:59.860 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555868+0000", 2024-09-18T04:39:59.860 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.860 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.860 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.860 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:59.860 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555865+0000", 2024-09-18T04:39:59.861 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999971 2024-09-18T04:39:59.861 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.861 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.861 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:59.861 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561031+0000", 2024-09-18T04:39:59.861 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0051658770000000001 2024-09-18T04:39:59.861 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.861 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.861 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:59.861 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561048+0000", 2024-09-18T04:39:59.861 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6569e-05 2024-09-18T04:39:59.861 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.861 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.861 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:59.862 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561055+0000", 2024-09-18T04:39:59.862 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.328e-06 2024-09-18T04:39:59.862 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.862 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.862 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:59.862 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561114+0000", 2024-09-18T04:39:59.862 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9540000000000003e-05 2024-09-18T04:39:59.862 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.862 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.862 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:59.862 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561314+0000", 2024-09-18T04:39:59.862 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019965799999999999 2024-09-18T04:39:59.862 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.862 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.862 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:59.863 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561473+0000", 2024-09-18T04:39:59.863 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000158904 2024-09-18T04:39:59.863 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.863 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.863 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:59.863 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.567015+0000", 2024-09-18T04:39:59.863 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055420859999999999 2024-09-18T04:39:59.863 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.863 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.863 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:59.863 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577982+0000", 2024-09-18T04:39:59.863 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010966872000000001 2024-09-18T04:39:59.863 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.864 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.864 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:59.864 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578162+0000", 2024-09-18T04:39:59.864 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018050999999999999 2024-09-18T04:39:59.864 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.864 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.864 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:59.864 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578200+0000", 2024-09-18T04:39:59.864 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8000999999999998e-05 2024-09-18T04:39:59.864 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.864 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:59.864 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.864 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.864 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.865 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:48 2.6 2:7c225365:ns::rbd_data.133e813ab5bc.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:59.865 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:39:59.865 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 77.222781216000001, 2024-09-18T04:39:59.865 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.024576377, 2024-09-18T04:39:59.865 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:59.865 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:59.865 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:59.865 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:59.865 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:59.865 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 48 2024-09-18T04:39:59.865 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.865 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:59.865 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.865 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:59.866 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:39:59.866 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.866 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.866 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.866 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:59.866 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:39:59.866 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.866 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.866 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.866 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:59.866 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.563666+0000", 2024-09-18T04:39:59.866 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:59.866 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.866 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.866 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:59.867 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570331+0000", 2024-09-18T04:39:59.867 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0066653329999999998 2024-09-18T04:39:59.867 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.867 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.867 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:59.867 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570348+0000", 2024-09-18T04:39:59.867 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6507e-05 2024-09-18T04:39:59.867 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.867 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.867 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:59.867 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570355+0000", 2024-09-18T04:39:59.867 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2749999999999998e-06 2024-09-18T04:39:59.867 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.867 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.867 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:59.868 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570396+0000", 2024-09-18T04:39:59.868 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0809000000000002e-05 2024-09-18T04:39:59.868 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.868 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.868 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:59.868 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570571+0000", 2024-09-18T04:39:59.868 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000174823 2024-09-18T04:39:59.868 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.868 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.868 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:59.868 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570722+0000", 2024-09-18T04:39:59.868 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015154599999999999 2024-09-18T04:39:59.868 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.868 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.869 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:59.869 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579756+0000", 2024-09-18T04:39:59.869 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0090337000000000004 2024-09-18T04:39:59.869 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.869 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.869 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:59.869 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588093+0000", 2024-09-18T04:39:59.869 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.008336553 2024-09-18T04:39:59.869 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.869 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.869 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:59.869 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588173+0000", 2024-09-18T04:39:59.869 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0333999999999994e-05 2024-09-18T04:39:59.869 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.869 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.870 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:59.870 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588244+0000", 2024-09-18T04:39:59.870 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0573000000000003e-05 2024-09-18T04:39:59.870 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.870 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:59.870 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.870 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.870 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.870 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:49 2.4 2:20f0b0c9:ns::rbd_data.133e813ab5bc.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:59.870 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.570370+0000", 2024-09-18T04:39:59.870 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 77.216078889000002, 2024-09-18T04:39:59.870 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.029633981, 2024-09-18T04:39:59.870 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:59.870 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:59.871 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:59.871 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:59.871 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:59.871 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 49 2024-09-18T04:39:59.871 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.871 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:59.871 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.871 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:59.871 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570370+0000", 2024-09-18T04:39:59.871 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.871 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.871 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.871 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:59.871 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570370+0000", 2024-09-18T04:39:59.872 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.872 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.872 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.872 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:59.872 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570369+0000", 2024-09-18T04:39:59.872 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:39:59.872 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.872 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.872 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:59.872 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579090+0000", 2024-09-18T04:39:59.872 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0087212339999999996 2024-09-18T04:39:59.872 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.872 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.872 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:59.872 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579109+0000", 2024-09-18T04:39:59.873 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9120999999999999e-05 2024-09-18T04:39:59.873 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.873 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.873 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:59.873 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579123+0000", 2024-09-18T04:39:59.873 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4382000000000001e-05 2024-09-18T04:39:59.873 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.873 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.873 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:59.873 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579221+0000", 2024-09-18T04:39:59.873 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.7503999999999993e-05 2024-09-18T04:39:59.873 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.873 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.873 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:59.873 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579553+0000", 2024-09-18T04:39:59.874 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00033254200000000002 2024-09-18T04:39:59.874 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.874 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.874 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:59.874 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579817+0000", 2024-09-18T04:39:59.874 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00026317099999999998 2024-09-18T04:39:59.874 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.874 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.874 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:59.874 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.599905+0000", 2024-09-18T04:39:59.874 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020088575000000001 2024-09-18T04:39:59.874 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.874 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.875 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:59.875 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.599923+0000", 2024-09-18T04:39:59.875 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7425999999999999e-05 2024-09-18T04:39:59.875 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.875 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.875 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:59.875 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.599973+0000", 2024-09-18T04:39:59.875 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0649999999999998e-05 2024-09-18T04:39:59.875 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.875 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.875 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:59.875 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.600004+0000", 2024-09-18T04:39:59.875 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0361999999999998e-05 2024-09-18T04:39:59.875 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.875 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:59.876 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.876 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.876 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.876 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:50 2.1 e51/24 2:8c706374:ns::rbd_data.133e813ab5bc.000000000000000a:head v 51'8, mlcod=51'6)", 2024-09-18T04:39:59.876 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.577029+0000", 2024-09-18T04:39:59.876 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 77.209419518000004, 2024-09-18T04:39:59.876 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.017362855, 2024-09-18T04:39:59.876 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:59.876 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:59.876 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:59.876 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.876 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:59.876 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577029+0000", 2024-09-18T04:39:59.877 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.877 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.877 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.877 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:59.877 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577029+0000", 2024-09-18T04:39:59.877 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.877 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.877 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.877 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:59.877 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577027+0000", 2024-09-18T04:39:59.877 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:59.877 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.878 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.878 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:59.878 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578880+0000", 2024-09-18T04:39:59.878 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0018532080000000001 2024-09-18T04:39:59.878 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.878 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.878 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:59.878 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578906+0000", 2024-09-18T04:39:59.878 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5573999999999999e-05 2024-09-18T04:39:59.878 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.878 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.878 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:59.878 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578920+0000", 2024-09-18T04:39:59.878 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4483e-05 2024-09-18T04:39:59.878 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.879 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.879 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:59.879 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578983+0000", 2024-09-18T04:39:59.879 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2248000000000004e-05 2024-09-18T04:39:59.879 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.879 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.879 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:59.879 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579016+0000", 2024-09-18T04:39:59.879 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3364000000000001e-05 2024-09-18T04:39:59.879 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.879 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.879 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:59.879 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594342+0000", 2024-09-18T04:39:59.879 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01532564 2024-09-18T04:39:59.879 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.880 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.880 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:59.880 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594392+0000", 2024-09-18T04:39:59.880 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0068999999999998e-05 2024-09-18T04:39:59.880 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.880 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:59.880 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.880 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.880 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.880 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:45 2.0 e51/17 2:0fbfa3dc:ns::rbd_data.133e813ab5bc.0000000000000007:head v 51'24, mlcod=51'22)", 2024-09-18T04:39:59.880 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.581892+0000", 2024-09-18T04:39:59.880 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 77.204556846000003, 2024-09-18T04:39:59.880 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015646006, 2024-09-18T04:39:59.880 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:59.881 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:59.881 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:59.881 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.881 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:59.881 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581892+0000", 2024-09-18T04:39:59.881 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.881 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.881 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.881 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:59.881 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581892+0000", 2024-09-18T04:39:59.881 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.881 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.881 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.882 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:59.882 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581890+0000", 2024-09-18T04:39:59.882 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:39:59.882 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.882 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.882 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:59.882 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586188+0000", 2024-09-18T04:39:59.882 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0042985769999999996 2024-09-18T04:39:59.882 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.882 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.882 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:59.882 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586210+0000", 2024-09-18T04:39:59.882 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1069000000000001e-05 2024-09-18T04:39:59.882 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.882 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.883 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:59.883 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586221+0000", 2024-09-18T04:39:59.883 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1287000000000001e-05 2024-09-18T04:39:59.883 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.883 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.883 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:59.883 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586313+0000", 2024-09-18T04:39:59.883 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1935999999999995e-05 2024-09-18T04:39:59.883 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.883 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.883 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:59.883 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586368+0000", 2024-09-18T04:39:59.883 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5578000000000002e-05 2024-09-18T04:39:59.883 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.884 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.884 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:59.884 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.597473+0000", 2024-09-18T04:39:59.884 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011105139999999999 2024-09-18T04:39:59.884 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.884 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.884 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:59.884 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.597538+0000", 2024-09-18T04:39:59.884 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4133000000000004e-05 2024-09-18T04:39:59.884 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.884 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:59.884 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.884 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.884 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.885 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:53 2.4 2:39f7f174:ns::rbd_data.133e813ab5bc.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:59.885 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.584263+0000", 2024-09-18T04:39:59.885 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 77.202184971999998, 2024-09-18T04:39:59.885 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018701180000000001, 2024-09-18T04:39:59.885 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:59.885 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:59.885 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:59.885 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:59.885 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:59.885 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 53 2024-09-18T04:39:59.885 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.885 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:59.885 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.885 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:59.886 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584263+0000", 2024-09-18T04:39:59.886 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.886 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.886 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.886 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:59.886 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584263+0000", 2024-09-18T04:39:59.886 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.886 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.886 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.886 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:59.886 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584262+0000", 2024-09-18T04:39:59.886 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:39:59.886 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.886 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.887 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:59.887 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.591617+0000", 2024-09-18T04:39:59.887 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0073550600000000001 2024-09-18T04:39:59.887 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.887 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.887 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:59.887 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.591637+0000", 2024-09-18T04:39:59.887 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0432999999999999e-05 2024-09-18T04:39:59.887 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.887 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.887 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:59.887 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.591646+0000", 2024-09-18T04:39:59.887 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7849999999999997e-06 2024-09-18T04:39:59.887 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.888 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.888 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:59.888 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593478+0000", 2024-09-18T04:39:59.888 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001831495 2024-09-18T04:39:59.888 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.888 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.888 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:59.888 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593661+0000", 2024-09-18T04:39:59.888 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018364999999999999 2024-09-18T04:39:59.888 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.888 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.888 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:39:59.888 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593834+0000", 2024-09-18T04:39:59.888 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000173065 2024-09-18T04:39:59.889 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.889 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.889 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:59.889 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.600670+0000", 2024-09-18T04:39:59.889 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0068354970000000003 2024-09-18T04:39:59.889 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.889 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.889 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:59.889 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602851+0000", 2024-09-18T04:39:59.889 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002181653 2024-09-18T04:39:59.889 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.889 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.889 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:59.889 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602930+0000", 2024-09-18T04:39:59.890 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8045000000000002e-05 2024-09-18T04:39:59.890 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.890 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.890 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:59.890 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602965+0000", 2024-09-18T04:39:59.890 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5102999999999999e-05 2024-09-18T04:39:59.890 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.890 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:59.890 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.890 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.890 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.890 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:56 2.6 2:72453c3e:ns::rbd_data.133e813ab5bc.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:39:59.890 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.595988+0000", 2024-09-18T04:39:59.890 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 77.190460569999999, 2024-09-18T04:39:59.891 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.030296172, 2024-09-18T04:39:59.891 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:39:59.891 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:39:59.891 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:39:59.891 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:39:59.891 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:39:59.891 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 56 2024-09-18T04:39:59.891 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.891 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:39:59.891 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.891 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:39:59.891 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.595988+0000", 2024-09-18T04:39:59.891 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.891 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.892 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.892 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:39:59.892 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.595988+0000", 2024-09-18T04:39:59.892 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:39:59.892 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.892 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.892 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:39:59.892 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.595986+0000", 2024-09-18T04:39:59.892 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:39:59.892 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.892 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.892 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:39:59.892 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605262+0000", 2024-09-18T04:39:59.892 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0092767240000000001 2024-09-18T04:39:59.893 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.893 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.893 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:39:59.893 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605282+0000", 2024-09-18T04:39:59.893 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9307e-05 2024-09-18T04:39:59.893 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.893 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.893 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:39:59.893 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605292+0000", 2024-09-18T04:39:59.893 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9510000000000001e-06 2024-09-18T04:39:59.893 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.893 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.893 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:39:59.893 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605369+0000", 2024-09-18T04:39:59.893 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7846e-05 2024-09-18T04:39:59.894 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.894 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.894 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:39:59.894 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605668+0000", 2024-09-18T04:39:59.894 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00029896099999999999 2024-09-18T04:39:59.894 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.894 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.894 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:39:59.894 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605906+0000", 2024-09-18T04:39:59.894 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000237758 2024-09-18T04:39:59.894 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.894 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.894 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:39:59.894 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.613532+0000", 2024-09-18T04:39:59.895 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0076260590000000001 2024-09-18T04:39:59.895 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.895 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.895 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:39:59.895 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.626116+0000", 2024-09-18T04:39:59.895 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012584092999999999 2024-09-18T04:39:59.895 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.895 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.895 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:39:59.895 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.626216+0000", 2024-09-18T04:39:59.895 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9630999999999995e-05 2024-09-18T04:39:59.895 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:39:59.895 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:39:59.895 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:39:59.895 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.626284+0000", 2024-09-18T04:39:59.896 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8034999999999995e-05 2024-09-18T04:39:59.896 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.896 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:59.896 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.896 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:39:59.896 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:39:59.896 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:39:59.896 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-18T04:39:59.898 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:39:59.898 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:39:59.891+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:39:59.998 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:39:59.995+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:40:00.016 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:40:00.016 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:40:00.016 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:40:00.016 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:40:00.026 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:40:00.099 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:40:00.096+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:40:00.130 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:40:00.130 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:40:00.130 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:40:00.130 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:40:00.130 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:40:00.140 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:40:00.200 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:40:00.195+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:40:00.246 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:40:00.246 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:40:00.246 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:40:00.247 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:40:00.247 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.247 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:40:00.247 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:40:00.247 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 152.45553790299999, 2024-09-18T04:40:00.247 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017399218000000001, 2024-09-18T04:40:00.247 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:00.247 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:00.247 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:00.247 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:40:00.247 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:40:00.247 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 114 2024-09-18T04:40:00.247 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.248 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:00.248 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.248 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:00.248 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:40:00.248 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:00.248 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.248 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.248 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:00.248 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:40:00.248 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:00.248 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.248 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.248 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:00.248 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786900+0000", 2024-09-18T04:40:00.249 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:40:00.249 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.249 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.249 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:00.249 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786908+0000", 2024-09-18T04:40:00.249 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8199999999999997e-06 2024-09-18T04:40:00.249 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.249 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.249 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:00.249 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786925+0000", 2024-09-18T04:40:00.249 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6849000000000001e-05 2024-09-18T04:40:00.249 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.249 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.249 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:00.250 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786934+0000", 2024-09-18T04:40:00.250 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6299999999999993e-06 2024-09-18T04:40:00.250 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.250 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.250 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:00.250 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800926+0000", 2024-09-18T04:40:00.250 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013991632 2024-09-18T04:40:00.250 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.250 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.250 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:00.250 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801060+0000", 2024-09-18T04:40:00.250 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013378199999999999 2024-09-18T04:40:00.250 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.251 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.251 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:40:00.251 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801166+0000", 2024-09-18T04:40:00.251 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010607099999999999 2024-09-18T04:40:00.251 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.251 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.251 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:00.251 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803026+0000", 2024-09-18T04:40:00.251 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001860469 2024-09-18T04:40:00.251 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.251 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.251 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:00.251 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804245+0000", 2024-09-18T04:40:00.252 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0012185900000000001 2024-09-18T04:40:00.252 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.252 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.252 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:00.252 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804278+0000", 2024-09-18T04:40:00.252 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3272000000000003e-05 2024-09-18T04:40:00.252 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.252 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.252 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:00.252 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804300+0000", 2024-09-18T04:40:00.252 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2266e-05 2024-09-18T04:40:00.252 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:00.252 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:00.252 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:00.253 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.253 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.253 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:40:00.253 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:40:00.253 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 152.454625005, 2024-09-18T04:40:00.253 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017521662, 2024-09-18T04:40:00.253 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:00.253 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:00.253 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:00.253 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:40:00.253 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:40:00.253 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 116 2024-09-18T04:40:00.253 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.253 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:00.254 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.254 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:00.254 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:40:00.254 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:00.254 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.254 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.254 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:00.254 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:40:00.254 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:00.254 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.254 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.254 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:00.254 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787813+0000", 2024-09-18T04:40:00.254 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:00.255 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.255 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.255 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:00.255 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787824+0000", 2024-09-18T04:40:00.255 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1157e-05 2024-09-18T04:40:00.255 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.255 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.255 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:00.255 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787840+0000", 2024-09-18T04:40:00.255 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6269e-05 2024-09-18T04:40:00.255 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.255 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.255 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:00.256 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787853+0000", 2024-09-18T04:40:00.256 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3156000000000001e-05 2024-09-18T04:40:00.256 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.256 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.256 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:00.256 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803110+0000", 2024-09-18T04:40:00.256 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015256736 2024-09-18T04:40:00.256 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.256 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.256 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:00.256 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803197+0000", 2024-09-18T04:40:00.256 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7195999999999997e-05 2024-09-18T04:40:00.256 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.256 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.257 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:40:00.257 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803308+0000", 2024-09-18T04:40:00.257 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111005 2024-09-18T04:40:00.257 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.257 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.257 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:00.257 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804316+0000", 2024-09-18T04:40:00.257 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010073949999999999 2024-09-18T04:40:00.257 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.257 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.257 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:00.257 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805232+0000", 2024-09-18T04:40:00.257 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091636300000000001 2024-09-18T04:40:00.257 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.258 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.258 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:00.258 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805277+0000", 2024-09-18T04:40:00.258 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4864000000000003e-05 2024-09-18T04:40:00.258 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.258 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.258 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:00.258 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805336+0000", 2024-09-18T04:40:00.258 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8968000000000002e-05 2024-09-18T04:40:00.258 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:00.258 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:00.258 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:00.258 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.258 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.258 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:40:00.259 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:40:00.259 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 152.45457295, 2024-09-18T04:40:00.259 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017983453, 2024-09-18T04:40:00.259 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:00.259 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:00.259 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:00.259 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:40:00.259 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:40:00.259 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 117 2024-09-18T04:40:00.259 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.259 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:00.259 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.259 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:00.259 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:40:00.260 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:00.260 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.260 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.260 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:00.260 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:40:00.260 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:00.260 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.260 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.260 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:00.260 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787865+0000", 2024-09-18T04:40:00.260 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:40:00.260 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.260 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.261 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:00.261 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787889+0000", 2024-09-18T04:40:00.261 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4244999999999999e-05 2024-09-18T04:40:00.261 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.261 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.261 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:00.261 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787905+0000", 2024-09-18T04:40:00.261 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5739999999999998e-05 2024-09-18T04:40:00.261 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.261 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.261 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:00.262 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787918+0000", 2024-09-18T04:40:00.262 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.252e-05 2024-09-18T04:40:00.262 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.262 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.262 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:00.262 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803693+0000", 2024-09-18T04:40:00.262 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015774872999999998 2024-09-18T04:40:00.262 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.262 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.262 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:00.262 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803775+0000", 2024-09-18T04:40:00.262 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2761000000000003e-05 2024-09-18T04:40:00.262 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.263 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.263 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:40:00.263 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803887+0000", 2024-09-18T04:40:00.263 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111708 2024-09-18T04:40:00.263 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.263 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.263 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:00.263 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804823+0000", 2024-09-18T04:40:00.263 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00093625100000000003 2024-09-18T04:40:00.263 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.263 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.263 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:00.263 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805804+0000", 2024-09-18T04:40:00.263 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00098114699999999992 2024-09-18T04:40:00.264 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.264 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.264 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:00.264 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805831+0000", 2024-09-18T04:40:00.264 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6401999999999999e-05 2024-09-18T04:40:00.264 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.264 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.264 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:00.264 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805850+0000", 2024-09-18T04:40:00.264 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8862e-05 2024-09-18T04:40:00.264 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:00.264 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:00.265 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:00.265 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.265 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.265 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:36 2.7 2:fe1b02ad:::rbd_data.125fc09df6d1.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:00.265 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:40:00.265 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 77.976716839999995, 2024-09-18T04:40:00.265 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.040387804999999999, 2024-09-18T04:40:00.265 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:00.265 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:00.265 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:00.265 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:40:00.265 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:40:00.266 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 36 2024-09-18T04:40:00.266 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.266 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:00.266 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.266 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:00.266 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:40:00.266 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:00.266 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.266 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.266 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:00.266 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:40:00.266 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:00.266 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.267 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.267 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:00.267 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265720+0000", 2024-09-18T04:40:00.267 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:40:00.267 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.267 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.267 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:00.267 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272590+0000", 2024-09-18T04:40:00.267 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.006870508 2024-09-18T04:40:00.267 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.267 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.267 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:00.267 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272623+0000", 2024-09-18T04:40:00.267 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2716000000000003e-05 2024-09-18T04:40:00.268 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.268 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.268 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:00.268 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272637+0000", 2024-09-18T04:40:00.268 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4175999999999999e-05 2024-09-18T04:40:00.268 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.268 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.268 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:00.268 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272779+0000", 2024-09-18T04:40:00.268 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000141392 2024-09-18T04:40:00.268 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.268 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.268 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:00.269 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273031+0000", 2024-09-18T04:40:00.269 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00025249099999999998 2024-09-18T04:40:00.269 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.269 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.269 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:40:00.269 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273219+0000", 2024-09-18T04:40:00.269 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000187531 2024-09-18T04:40:00.269 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.269 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.269 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:00.269 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.302806+0000", 2024-09-18T04:40:00.269 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.029587553999999999 2024-09-18T04:40:00.269 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.269 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.270 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:00.270 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306004+0000", 2024-09-18T04:40:00.270 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0031982629999999998 2024-09-18T04:40:00.270 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.270 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.270 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:00.270 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306071+0000", 2024-09-18T04:40:00.270 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6829999999999995e-05 2024-09-18T04:40:00.270 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.270 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.270 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:00.270 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306110+0000", 2024-09-18T04:40:00.270 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8874000000000001e-05 2024-09-18T04:40:00.270 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:00.270 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:00.271 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:00.271 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.271 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.271 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:38 2.0 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:00.271 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:40:00.271 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 77.969763302999993, 2024-09-18T04:40:00.271 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.033754348000000003, 2024-09-18T04:40:00.271 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:00.271 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:00.271 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:00.271 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:40:00.271 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:40:00.271 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 38 2024-09-18T04:40:00.272 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.272 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:00.272 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.272 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:00.272 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:40:00.272 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:00.272 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.272 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.272 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:00.272 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:40:00.272 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:00.272 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.272 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.272 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:00.272 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272673+0000", 2024-09-18T04:40:00.273 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:40:00.273 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.273 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.273 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:00.273 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277166+0000", 2024-09-18T04:40:00.273 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044922249999999999 2024-09-18T04:40:00.273 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.273 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.273 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:00.273 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277189+0000", 2024-09-18T04:40:00.273 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3283000000000001e-05 2024-09-18T04:40:00.273 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.273 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.274 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:00.274 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277205+0000", 2024-09-18T04:40:00.274 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6305000000000001e-05 2024-09-18T04:40:00.274 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.274 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.274 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:00.274 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277290+0000", 2024-09-18T04:40:00.274 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5135000000000004e-05 2024-09-18T04:40:00.274 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.274 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.274 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:00.274 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277531+0000", 2024-09-18T04:40:00.274 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00024014699999999999 2024-09-18T04:40:00.274 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.275 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.275 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:00.275 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277793+0000", 2024-09-18T04:40:00.275 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00026224499999999998 2024-09-18T04:40:00.275 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.275 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.275 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:00.275 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303990+0000", 2024-09-18T04:40:00.275 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.026197295999999998 2024-09-18T04:40:00.275 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.275 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.275 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:00.275 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306340+0000", 2024-09-18T04:40:00.275 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.002349524 2024-09-18T04:40:00.275 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.276 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.276 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:00.276 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306394+0000", 2024-09-18T04:40:00.276 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3934999999999998e-05 2024-09-18T04:40:00.276 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.276 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.276 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:00.276 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306430+0000", 2024-09-18T04:40:00.276 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6690000000000003e-05 2024-09-18T04:40:00.276 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:00.276 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:00.276 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:00.276 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.277 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.277 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:39 2.7 2:e9c909c5:::rbd_data.125fc09df6d1.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:00.277 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:40:00.277 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 77.965203697999996, 2024-09-18T04:40:00.277 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.029110673, 2024-09-18T04:40:00.277 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:00.277 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:00.277 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:00.277 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:40:00.277 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:40:00.277 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 39 2024-09-18T04:40:00.277 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.277 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:00.278 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.278 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:00.278 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:40:00.278 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:00.278 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.278 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.278 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:00.278 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:40:00.278 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:00.278 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.278 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.279 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:00.279 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277234+0000", 2024-09-18T04:40:00.279 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:00.279 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.279 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.279 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:00.279 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281351+0000", 2024-09-18T04:40:00.279 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0041170920000000001 2024-09-18T04:40:00.279 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.279 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.279 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:00.279 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281375+0000", 2024-09-18T04:40:00.279 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.446e-05 2024-09-18T04:40:00.280 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.280 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.280 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:00.280 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281390+0000", 2024-09-18T04:40:00.280 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4309e-05 2024-09-18T04:40:00.280 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.280 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.280 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:00.280 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.282306+0000", 2024-09-18T04:40:00.280 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091662000000000004 2024-09-18T04:40:00.280 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.280 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.280 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:00.280 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.282613+0000", 2024-09-18T04:40:00.281 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00030663799999999997 2024-09-18T04:40:00.281 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.281 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.281 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:40:00.281 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283001+0000", 2024-09-18T04:40:00.281 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00038801599999999998 2024-09-18T04:40:00.281 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.281 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.281 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:00.281 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306239+0000", 2024-09-18T04:40:00.281 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023238466999999999 2024-09-18T04:40:00.281 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.281 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.282 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:00.282 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306259+0000", 2024-09-18T04:40:00.282 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9837999999999999e-05 2024-09-18T04:40:00.282 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.282 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.282 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:00.282 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306299+0000", 2024-09-18T04:40:00.282 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0194000000000002e-05 2024-09-18T04:40:00.282 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.282 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.282 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:00.282 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306346+0000", 2024-09-18T04:40:00.282 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6845000000000003e-05 2024-09-18T04:40:00.282 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:00.282 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:00.283 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:00.283 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.283 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.283 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:46 2.6 e51/17 2:7ba25245:::rbd_data.125fc09df6d1.0000000000000006:head v 51'2, mlcod=0'0)", 2024-09-18T04:40:00.283 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:40:00.283 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 77.929282565999998, 2024-09-18T04:40:00.283 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019843098, 2024-09-18T04:40:00.283 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:00.283 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:00.283 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:00.283 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.283 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:00.283 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:40:00.283 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:00.284 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.284 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.284 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:00.284 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:40:00.284 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:00.284 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.284 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.284 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:00.284 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313156+0000", 2024-09-18T04:40:00.284 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:40:00.284 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.284 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.284 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:00.284 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320144+0000", 2024-09-18T04:40:00.284 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.006988345 2024-09-18T04:40:00.285 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.285 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.285 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:00.285 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320175+0000", 2024-09-18T04:40:00.285 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.0861999999999997e-05 2024-09-18T04:40:00.285 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.285 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.285 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:00.285 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320188+0000", 2024-09-18T04:40:00.285 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2714999999999999e-05 2024-09-18T04:40:00.285 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.285 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.286 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:00.286 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320267+0000", 2024-09-18T04:40:00.286 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9586999999999995e-05 2024-09-18T04:40:00.286 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.286 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.286 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:00.286 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320311+0000", 2024-09-18T04:40:00.286 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3529000000000003e-05 2024-09-18T04:40:00.286 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.286 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.286 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:00.286 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.332949+0000", 2024-09-18T04:40:00.286 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012638267999999999 2024-09-18T04:40:00.286 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.287 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.287 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:00.287 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.333000+0000", 2024-09-18T04:40:00.287 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0736999999999999e-05 2024-09-18T04:40:00.287 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:00.287 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:00.287 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:00.287 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.287 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.287 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:57 2.0 2:0cdd070e:::rbd_data.125fc09df6d1.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:00.287 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:40:00.287 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 77.893424991000003, 2024-09-18T04:40:00.287 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.033329974999999998, 2024-09-18T04:40:00.288 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:00.288 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:00.288 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:00.288 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:40:00.288 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:40:00.288 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 57 2024-09-18T04:40:00.288 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.288 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:00.288 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.288 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:00.288 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:40:00.288 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:00.288 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.289 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.289 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:00.289 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:40:00.289 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:00.289 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.289 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.289 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:00.289 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349013+0000", 2024-09-18T04:40:00.289 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:00.289 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.289 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.289 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:00.290 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356138+0000", 2024-09-18T04:40:00.290 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071258010000000002 2024-09-18T04:40:00.290 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.290 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.290 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:00.290 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356168+0000", 2024-09-18T04:40:00.290 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9679000000000002e-05 2024-09-18T04:40:00.290 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.290 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.290 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:00.290 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356181+0000", 2024-09-18T04:40:00.290 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3164e-05 2024-09-18T04:40:00.290 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.290 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.291 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:00.291 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356254+0000", 2024-09-18T04:40:00.291 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3182999999999994e-05 2024-09-18T04:40:00.291 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.291 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.291 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:00.291 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356453+0000", 2024-09-18T04:40:00.291 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019876599999999999 2024-09-18T04:40:00.291 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.291 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.291 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:00.291 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356627+0000", 2024-09-18T04:40:00.291 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017340199999999999 2024-09-18T04:40:00.292 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.292 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.292 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:00.292 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.369200+0000", 2024-09-18T04:40:00.292 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012573302 2024-09-18T04:40:00.292 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.292 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.292 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:00.292 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382262+0000", 2024-09-18T04:40:00.292 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013062549 2024-09-18T04:40:00.292 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.292 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.292 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:00.293 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382310+0000", 2024-09-18T04:40:00.293 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7366e-05 2024-09-18T04:40:00.293 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.293 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.293 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:00.293 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382344+0000", 2024-09-18T04:40:00.293 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4319999999999997e-05 2024-09-18T04:40:00.293 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:00.293 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:00.293 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:00.293 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.293 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.293 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:59 2.0 2:19c52cde:::rbd_data.125fc09df6d1.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:00.293 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:40:00.294 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 77.886233825999994, 2024-09-18T04:40:00.294 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.027137247999999999, 2024-09-18T04:40:00.294 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:00.294 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:00.294 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:00.294 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:40:00.294 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:40:00.294 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 59 2024-09-18T04:40:00.294 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.294 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:00.294 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.295 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:00.295 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:40:00.295 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:00.295 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.295 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.295 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:00.295 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:40:00.295 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:00.295 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.295 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.295 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:00.295 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356204+0000", 2024-09-18T04:40:00.295 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:00.295 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.296 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.296 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:00.296 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362661+0000", 2024-09-18T04:40:00.296 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0064565129999999997 2024-09-18T04:40:00.296 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.296 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.296 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:00.296 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362684+0000", 2024-09-18T04:40:00.296 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3688000000000001e-05 2024-09-18T04:40:00.296 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.296 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.296 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:00.296 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362699+0000", 2024-09-18T04:40:00.296 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4798e-05 2024-09-18T04:40:00.297 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.297 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.297 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:00.297 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363016+0000", 2024-09-18T04:40:00.297 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031737300000000003 2024-09-18T04:40:00.297 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.297 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.297 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:00.297 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363201+0000", 2024-09-18T04:40:00.297 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000184131 2024-09-18T04:40:00.297 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.297 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.297 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:00.298 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363343+0000", 2024-09-18T04:40:00.298 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014236699999999999 2024-09-18T04:40:00.298 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.298 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.298 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:00.298 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.372189+0000", 2024-09-18T04:40:00.298 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0088464819999999993 2024-09-18T04:40:00.298 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.298 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.298 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:00.298 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383269+0000", 2024-09-18T04:40:00.298 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011079594 2024-09-18T04:40:00.299 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.299 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.299 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:00.299 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383314+0000", 2024-09-18T04:40:00.299 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4932000000000003e-05 2024-09-18T04:40:00.299 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.299 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.299 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:00.299 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383343+0000", 2024-09-18T04:40:00.299 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8600999999999999e-05 2024-09-18T04:40:00.299 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:00.299 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:00.300 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:00.300 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.300 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.300 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:40 2.6 e51/17 2:784d6cd1:ns::rbd_data.133e813ab5bc.0000000000000005:head v 51'7, mlcod=51'5)", 2024-09-18T04:40:00.300 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:40:00.300 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 77.689984092000003, 2024-09-18T04:40:00.300 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020497496, 2024-09-18T04:40:00.300 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:00.300 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:00.300 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:00.300 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.300 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:00.300 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:40:00.300 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:00.300 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.301 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.301 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:00.301 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:40:00.301 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:00.301 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.301 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.301 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:00.301 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552454+0000", 2024-09-18T04:40:00.301 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:00.301 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.301 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.301 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:00.301 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556448+0000", 2024-09-18T04:40:00.301 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0039940990000000001 2024-09-18T04:40:00.302 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.302 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.302 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:00.302 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556481+0000", 2024-09-18T04:40:00.302 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3778000000000001e-05 2024-09-18T04:40:00.302 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.302 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.302 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:00.302 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556493+0000", 2024-09-18T04:40:00.302 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1622e-05 2024-09-18T04:40:00.302 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.302 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.302 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:00.303 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556556+0000", 2024-09-18T04:40:00.303 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2582999999999994e-05 2024-09-18T04:40:00.303 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.303 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.303 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:00.303 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556592+0000", 2024-09-18T04:40:00.303 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5880999999999997e-05 2024-09-18T04:40:00.303 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.303 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.303 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:00.303 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.572786+0000", 2024-09-18T04:40:00.303 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016194571000000001 2024-09-18T04:40:00.303 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.304 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.304 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:00.304 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.572953+0000", 2024-09-18T04:40:00.304 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000166436 2024-09-18T04:40:00.304 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:00.304 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:00.304 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:00.304 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.304 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.304 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:45 2.0 2:0fbfa3dc:ns::rbd_data.133e813ab5bc.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:00.304 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:40:00.304 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 77.678769793000001, 2024-09-18T04:40:00.304 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.036558168000000002, 2024-09-18T04:40:00.305 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:00.305 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:00.305 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:00.305 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:40:00.305 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:00.305 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 45 2024-09-18T04:40:00.305 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.305 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:00.305 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.305 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:00.305 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:40:00.305 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:00.305 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.305 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.306 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:00.306 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:40:00.306 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:00.306 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.306 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.306 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:00.306 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:40:00.306 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:40:00.306 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.306 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.306 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:00.306 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578594+0000", 2024-09-18T04:40:00.306 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014927147999999999 2024-09-18T04:40:00.306 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.307 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.307 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:00.307 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578612+0000", 2024-09-18T04:40:00.307 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7623999999999999e-05 2024-09-18T04:40:00.307 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.307 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.307 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:00.307 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578620+0000", 2024-09-18T04:40:00.307 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9070000000000007e-06 2024-09-18T04:40:00.307 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.307 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.307 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:00.307 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578702+0000", 2024-09-18T04:40:00.307 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2349999999999996e-05 2024-09-18T04:40:00.308 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.308 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.308 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:00.308 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.579028+0000", 2024-09-18T04:40:00.308 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032622899999999998 2024-09-18T04:40:00.308 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.308 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.308 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:00.308 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.579343+0000", 2024-09-18T04:40:00.308 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031448899999999999 2024-09-18T04:40:00.308 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.308 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.308 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:00.309 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.590078+0000", 2024-09-18T04:40:00.309 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010734946 2024-09-18T04:40:00.309 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.309 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.309 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:00.309 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600089+0000", 2024-09-18T04:40:00.309 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010011195000000001 2024-09-18T04:40:00.309 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.309 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.309 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:00.309 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600177+0000", 2024-09-18T04:40:00.309 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8425999999999997e-05 2024-09-18T04:40:00.309 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.309 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.310 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:00.310 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600228+0000", 2024-09-18T04:40:00.310 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0216999999999997e-05 2024-09-18T04:40:00.310 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:00.310 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:00.310 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:00.310 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.310 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.310 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:54 2.0 2:04f17ef6:ns::rbd_data.133e813ab5bc.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:00.310 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:40:00.310 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 77.657197956000005, 2024-09-18T04:40:00.310 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.024612999, 2024-09-18T04:40:00.310 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:00.310 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:00.311 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:00.311 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:40:00.311 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:00.311 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 54 2024-09-18T04:40:00.311 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.311 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:00.311 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.311 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:00.311 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:40:00.311 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:00.311 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.311 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.311 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:00.311 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:40:00.312 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:00.312 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.312 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.312 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:00.312 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585239+0000", 2024-09-18T04:40:00.312 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:00.312 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.312 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.312 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:00.312 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593702+0000", 2024-09-18T04:40:00.312 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0084632990000000005 2024-09-18T04:40:00.312 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.312 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.313 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:00.313 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593719+0000", 2024-09-18T04:40:00.313 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.664e-05 2024-09-18T04:40:00.313 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.313 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.313 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:00.313 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593727+0000", 2024-09-18T04:40:00.313 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6839999999999996e-06 2024-09-18T04:40:00.313 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.313 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.313 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:00.313 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593767+0000", 2024-09-18T04:40:00.313 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0330999999999997e-05 2024-09-18T04:40:00.313 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.314 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.314 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:00.314 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593905+0000", 2024-09-18T04:40:00.314 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013754800000000001 2024-09-18T04:40:00.314 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.314 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.314 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:00.314 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.594013+0000", 2024-09-18T04:40:00.314 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000108805 2024-09-18T04:40:00.314 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.314 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.314 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:00.314 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604678+0000", 2024-09-18T04:40:00.315 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010664391 2024-09-18T04:40:00.315 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.315 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.315 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:00.315 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609662+0000", 2024-09-18T04:40:00.315 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0049846100000000004 2024-09-18T04:40:00.315 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.315 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.315 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:00.315 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609788+0000", 2024-09-18T04:40:00.315 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012518800000000001 2024-09-18T04:40:00.315 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.315 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.315 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:00.315 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609854+0000", 2024-09-18T04:40:00.316 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6556000000000004e-05 2024-09-18T04:40:00.316 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:00.316 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:00.316 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:00.316 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.316 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.316 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:58 2.7 2:fc7c6cd3:ns::rbd_data.133e813ab5bc.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:00.316 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:40:00.316 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 77.644355527000002, 2024-09-18T04:40:00.316 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.027690566999999999, 2024-09-18T04:40:00.316 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:00.316 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:00.316 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:00.317 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:40:00.317 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:00.317 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 58 2024-09-18T04:40:00.317 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.317 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:00.317 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.317 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:00.317 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:40:00.317 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:00.317 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.317 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.317 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:00.317 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:40:00.318 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:00.318 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.318 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.318 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:00.318 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598082+0000", 2024-09-18T04:40:00.318 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:00.318 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.318 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.318 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:00.318 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604832+0000", 2024-09-18T04:40:00.318 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0067504510000000002 2024-09-18T04:40:00.318 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.318 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.318 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:00.319 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604852+0000", 2024-09-18T04:40:00.319 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0228999999999999e-05 2024-09-18T04:40:00.319 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.319 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.319 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:00.319 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604870+0000", 2024-09-18T04:40:00.319 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7952e-05 2024-09-18T04:40:00.319 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.319 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.319 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:00.319 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604936+0000", 2024-09-18T04:40:00.319 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5431000000000004e-05 2024-09-18T04:40:00.319 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.319 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.320 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:00.320 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605085+0000", 2024-09-18T04:40:00.320 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014910300000000001 2024-09-18T04:40:00.320 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.320 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.320 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:40:00.320 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605205+0000", 2024-09-18T04:40:00.320 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000119887 2024-09-18T04:40:00.320 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.320 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.320 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:00.320 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625586+0000", 2024-09-18T04:40:00.320 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020381399000000001 2024-09-18T04:40:00.321 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.321 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.321 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:00.321 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625628+0000", 2024-09-18T04:40:00.321 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1873999999999999e-05 2024-09-18T04:40:00.321 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.321 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.321 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:00.321 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625722+0000", 2024-09-18T04:40:00.321 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3787000000000002e-05 2024-09-18T04:40:00.321 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.321 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.321 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:00.322 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625774+0000", 2024-09-18T04:40:00.322 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2503999999999997e-05 2024-09-18T04:40:00.322 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:00.322 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:00.322 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:00.322 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.322 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.322 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:56 2.6 e51/17 2:72453c3e:ns::rbd_data.133e813ab5bc.000000000000000d:head v 51'10, mlcod=51'8)", 2024-09-18T04:40:00.322 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:40:00.322 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 77.637158419000002, 2024-09-18T04:40:00.322 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019766552999999999, 2024-09-18T04:40:00.322 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:00.322 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:00.323 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:00.323 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.323 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:00.323 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:40:00.323 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:00.323 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.323 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.323 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:00.323 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:40:00.323 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:00.323 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.323 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.323 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:00.324 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605279+0000", 2024-09-18T04:40:00.324 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:00.324 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.324 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.324 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:00.324 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609417+0000", 2024-09-18T04:40:00.324 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0041378059999999999 2024-09-18T04:40:00.324 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.324 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.324 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:00.324 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609443+0000", 2024-09-18T04:40:00.324 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6098e-05 2024-09-18T04:40:00.324 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.325 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.325 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:00.325 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609458+0000", 2024-09-18T04:40:00.325 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4671000000000001e-05 2024-09-18T04:40:00.325 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.325 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.325 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:00.325 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609553+0000", 2024-09-18T04:40:00.325 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5265000000000004e-05 2024-09-18T04:40:00.325 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.325 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.325 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:00.325 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609583+0000", 2024-09-18T04:40:00.326 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9839e-05 2024-09-18T04:40:00.326 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.326 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.326 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:00.326 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625004+0000", 2024-09-18T04:40:00.326 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015420639999999999 2024-09-18T04:40:00.326 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.326 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.326 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:00.326 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625047+0000", 2024-09-18T04:40:00.326 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3665999999999998e-05 2024-09-18T04:40:00.326 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:00.326 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:00.326 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:00.327 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.327 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.327 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:99 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:40:00.327 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.582718+0000", 2024-09-18T04:40:00.327 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 63.659721509999997, 2024-09-18T04:40:00.327 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019362575, 2024-09-18T04:40:00.327 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:00.327 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:00.327 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:00.327 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:40:00.327 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:40:00.327 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 99 2024-09-18T04:40:00.327 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.327 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:00.328 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.328 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:00.328 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582718+0000", 2024-09-18T04:40:00.328 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:00.328 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.328 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.328 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:00.328 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582718+0000", 2024-09-18T04:40:00.328 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:00.328 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.328 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.328 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:00.328 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582717+0000", 2024-09-18T04:40:00.329 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:40:00.329 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.329 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.329 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:00.329 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582720+0000", 2024-09-18T04:40:00.329 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2239999999999998e-06 2024-09-18T04:40:00.329 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.329 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.329 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:00.329 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582723+0000", 2024-09-18T04:40:00.329 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.258e-06 2024-09-18T04:40:00.329 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.329 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.330 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:00.330 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582726+0000", 2024-09-18T04:40:00.330 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8870000000000001e-06 2024-09-18T04:40:00.330 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.330 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.330 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:00.330 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.592018+0000", 2024-09-18T04:40:00.330 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0092926420000000003 2024-09-18T04:40:00.330 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.330 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.330 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:00.330 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.592214+0000", 2024-09-18T04:40:00.331 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000195267 2024-09-18T04:40:00.331 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.331 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.331 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:00.331 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.592913+0000", 2024-09-18T04:40:00.331 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00069924600000000005 2024-09-18T04:40:00.331 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.331 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.331 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:00.331 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597493+0000", 2024-09-18T04:40:00.331 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045797629999999997 2024-09-18T04:40:00.331 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.331 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.331 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:00.332 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.602009+0000", 2024-09-18T04:40:00.332 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045159559999999998 2024-09-18T04:40:00.332 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.332 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.332 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:00.332 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.602042+0000", 2024-09-18T04:40:00.332 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3084999999999999e-05 2024-09-18T04:40:00.332 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.332 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.332 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:00.332 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.602080+0000", 2024-09-18T04:40:00.332 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8600000000000003e-05 2024-09-18T04:40:00.332 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:00.332 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:00.333 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:00.333 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.333 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.333 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:104 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:40:00.333 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.583165+0000", 2024-09-18T04:40:00.333 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 63.659274212, 2024-09-18T04:40:00.333 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023679603, 2024-09-18T04:40:00.333 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:00.333 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:00.333 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:00.333 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:40:00.333 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:40:00.333 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 104 2024-09-18T04:40:00.333 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.334 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:00.334 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.334 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:00.334 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583165+0000", 2024-09-18T04:40:00.334 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:00.334 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.334 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.334 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:00.334 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583165+0000", 2024-09-18T04:40:00.334 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:00.334 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.334 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.334 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:00.335 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583164+0000", 2024-09-18T04:40:00.335 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:40:00.335 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.335 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.335 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:00.335 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583167+0000", 2024-09-18T04:40:00.335 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9210000000000001e-06 2024-09-18T04:40:00.335 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.335 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.335 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:00.335 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583171+0000", 2024-09-18T04:40:00.335 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9759999999999997e-06 2024-09-18T04:40:00.335 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.335 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.335 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:00.336 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583174+0000", 2024-09-18T04:40:00.336 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9960000000000002e-06 2024-09-18T04:40:00.336 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.336 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.336 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:00.336 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.594373+0000", 2024-09-18T04:40:00.336 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011198665999999999 2024-09-18T04:40:00.336 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.336 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.336 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:00.336 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.594434+0000", 2024-09-18T04:40:00.336 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.1446999999999996e-05 2024-09-18T04:40:00.336 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.337 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.337 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:00.337 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.594692+0000", 2024-09-18T04:40:00.337 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00025756599999999998 2024-09-18T04:40:00.337 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.337 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.337 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:00.337 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599302+0000", 2024-09-18T04:40:00.337 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0046099640000000002 2024-09-18T04:40:00.337 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.337 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.337 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:00.337 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606710+0000", 2024-09-18T04:40:00.337 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0074077739999999998 2024-09-18T04:40:00.338 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.338 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.338 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:00.338 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606798+0000", 2024-09-18T04:40:00.338 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8133999999999993e-05 2024-09-18T04:40:00.338 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.338 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.338 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:00.338 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606845+0000", 2024-09-18T04:40:00.338 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6653000000000002e-05 2024-09-18T04:40:00.338 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:00.338 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:00.338 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:00.339 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.339 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.339 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:105 2.0 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:40:00.339 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.583193+0000", 2024-09-18T04:40:00.339 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 63.659246592999999, 2024-09-18T04:40:00.339 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023713649, 2024-09-18T04:40:00.339 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:00.339 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:00.339 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:00.339 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:40:00.339 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:40:00.339 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 105 2024-09-18T04:40:00.339 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.340 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:00.340 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.340 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:00.340 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583193+0000", 2024-09-18T04:40:00.340 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:00.340 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.340 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.340 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:00.340 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583193+0000", 2024-09-18T04:40:00.340 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:00.340 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.340 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.340 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:00.340 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583192+0000", 2024-09-18T04:40:00.341 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:40:00.341 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.341 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.341 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:00.341 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583194+0000", 2024-09-18T04:40:00.341 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.305e-06 2024-09-18T04:40:00.341 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.341 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.341 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:00.341 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583198+0000", 2024-09-18T04:40:00.341 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9369999999999997e-06 2024-09-18T04:40:00.341 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.342 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.342 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:00.342 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583202+0000", 2024-09-18T04:40:00.342 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6480000000000001e-06 2024-09-18T04:40:00.342 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.342 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.342 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:00.342 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.595304+0000", 2024-09-18T04:40:00.342 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012102063999999999 2024-09-18T04:40:00.342 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.342 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.342 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:00.342 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.595390+0000", 2024-09-18T04:40:00.342 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5493000000000003e-05 2024-09-18T04:40:00.342 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.343 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.343 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:00.343 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.595509+0000", 2024-09-18T04:40:00.343 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011966199999999999 2024-09-18T04:40:00.343 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.343 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.343 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:00.343 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599358+0000", 2024-09-18T04:40:00.343 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0038486689999999999 2024-09-18T04:40:00.343 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.343 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.343 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:00.343 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606859+0000", 2024-09-18T04:40:00.344 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0075010789999999999 2024-09-18T04:40:00.344 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.344 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.344 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:00.344 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606887+0000", 2024-09-18T04:40:00.344 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7773e-05 2024-09-18T04:40:00.344 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.344 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.344 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:00.344 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606906+0000", 2024-09-18T04:40:00.344 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9417000000000001e-05 2024-09-18T04:40:00.344 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:00.344 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:00.344 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:00.345 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.345 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.345 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:106 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:40:00.345 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.584425+0000", 2024-09-18T04:40:00.345 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 63.658014297000001, 2024-09-18T04:40:00.345 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.022529293999999998, 2024-09-18T04:40:00.345 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:00.345 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:00.345 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:00.345 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:40:00.345 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:40:00.345 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 106 2024-09-18T04:40:00.345 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.346 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:00.346 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.346 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:00.346 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584425+0000", 2024-09-18T04:40:00.346 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:00.346 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.346 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.346 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:00.346 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584425+0000", 2024-09-18T04:40:00.346 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:00.346 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.346 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.346 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:00.346 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584424+0000", 2024-09-18T04:40:00.347 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:40:00.347 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.347 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.347 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:00.347 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584430+0000", 2024-09-18T04:40:00.347 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6389999999999997e-06 2024-09-18T04:40:00.347 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.347 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.347 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:00.347 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584456+0000", 2024-09-18T04:40:00.347 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6188999999999999e-05 2024-09-18T04:40:00.347 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.347 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.347 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:00.348 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584463+0000", 2024-09-18T04:40:00.348 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7340000000000002e-06 2024-09-18T04:40:00.348 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.348 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.348 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:00.348 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597537+0000", 2024-09-18T04:40:00.348 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013074161000000001 2024-09-18T04:40:00.348 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.348 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.348 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:00.348 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597598+0000", 2024-09-18T04:40:00.348 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0930000000000001e-05 2024-09-18T04:40:00.348 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.349 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.349 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:00.349 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597922+0000", 2024-09-18T04:40:00.349 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032473000000000002 2024-09-18T04:40:00.349 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.349 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.349 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:00.349 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.601103+0000", 2024-09-18T04:40:00.349 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0031809849999999999 2024-09-18T04:40:00.349 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.349 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.349 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:00.349 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606912+0000", 2024-09-18T04:40:00.349 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058083550000000003 2024-09-18T04:40:00.350 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.350 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.350 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:00.350 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606935+0000", 2024-09-18T04:40:00.350 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2759999999999999e-05 2024-09-18T04:40:00.350 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.350 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.350 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:00.350 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606954+0000", 2024-09-18T04:40:00.350 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9602999999999998e-05 2024-09-18T04:40:00.350 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:00.350 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:00.350 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:00.351 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.351 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.351 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:109 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:40:00.351 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.585231+0000", 2024-09-18T04:40:00.351 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 63.657207677000002, 2024-09-18T04:40:00.351 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.021767496000000001, 2024-09-18T04:40:00.351 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:00.351 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:00.351 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:00.351 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:40:00.351 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:40:00.351 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 109 2024-09-18T04:40:00.351 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.351 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:00.352 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.352 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:00.352 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585231+0000", 2024-09-18T04:40:00.352 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:00.352 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.352 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.352 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:00.352 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585231+0000", 2024-09-18T04:40:00.352 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:00.352 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.352 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.352 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:00.352 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585230+0000", 2024-09-18T04:40:00.353 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:40:00.353 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.353 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.353 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:00.353 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585237+0000", 2024-09-18T04:40:00.353 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.618e-06 2024-09-18T04:40:00.353 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.353 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.353 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:00.353 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585256+0000", 2024-09-18T04:40:00.353 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9048000000000001e-05 2024-09-18T04:40:00.353 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.353 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.353 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:00.354 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585274+0000", 2024-09-18T04:40:00.354 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8153000000000001e-05 2024-09-18T04:40:00.354 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.354 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.354 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:00.354 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.598284+0000", 2024-09-18T04:40:00.354 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013009682999999999 2024-09-18T04:40:00.354 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.354 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.354 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:00.354 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.598383+0000", 2024-09-18T04:40:00.354 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.8722999999999997e-05 2024-09-18T04:40:00.354 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.354 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.355 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:00.355 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.598829+0000", 2024-09-18T04:40:00.355 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00044623999999999998 2024-09-18T04:40:00.355 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.355 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.355 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:00.355 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.601149+0000", 2024-09-18T04:40:00.355 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0023204150000000002 2024-09-18T04:40:00.355 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.355 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.355 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:00.355 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606960+0000", 2024-09-18T04:40:00.355 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058106390000000003 2024-09-18T04:40:00.355 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.356 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.356 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:00.356 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606982+0000", 2024-09-18T04:40:00.356 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2252e-05 2024-09-18T04:40:00.356 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.356 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.356 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:00.356 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606999+0000", 2024-09-18T04:40:00.356 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6871000000000002e-05 2024-09-18T04:40:00.356 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:00.356 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:00.356 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:00.356 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.357 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.357 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:110 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:40:00.357 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.586640+0000", 2024-09-18T04:40:00.357 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 63.655799358000003, 2024-09-18T04:40:00.357 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020400911000000001, 2024-09-18T04:40:00.357 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:00.357 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:00.357 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:00.357 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:40:00.357 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:40:00.357 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 110 2024-09-18T04:40:00.357 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.357 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:00.357 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.357 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:00.358 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586640+0000", 2024-09-18T04:40:00.358 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:00.358 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.358 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.358 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:00.358 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586640+0000", 2024-09-18T04:40:00.358 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:00.358 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.358 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.358 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:00.358 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586639+0000", 2024-09-18T04:40:00.358 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:40:00.358 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.358 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.359 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:00.359 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586652+0000", 2024-09-18T04:40:00.359 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2482e-05 2024-09-18T04:40:00.359 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.359 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.359 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:00.359 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586660+0000", 2024-09-18T04:40:00.359 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1659999999999994e-06 2024-09-18T04:40:00.359 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.359 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.359 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:00.359 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586666+0000", 2024-09-18T04:40:00.359 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6900000000000003e-06 2024-09-18T04:40:00.359 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.360 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.360 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:00.360 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599381+0000", 2024-09-18T04:40:00.360 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012714328 2024-09-18T04:40:00.360 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.360 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.360 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:00.360 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599436+0000", 2024-09-18T04:40:00.360 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4802999999999997e-05 2024-09-18T04:40:00.360 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.360 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.360 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:00.360 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599629+0000", 2024-09-18T04:40:00.361 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000193921 2024-09-18T04:40:00.361 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.361 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.361 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:00.361 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.601174+0000", 2024-09-18T04:40:00.361 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001544857 2024-09-18T04:40:00.361 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.361 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.361 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:00.361 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.607004+0000", 2024-09-18T04:40:00.361 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058300820000000003 2024-09-18T04:40:00.361 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.361 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.361 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:00.362 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.607024+0000", 2024-09-18T04:40:00.362 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9888e-05 2024-09-18T04:40:00.362 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.362 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.362 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:00.362 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.607041+0000", 2024-09-18T04:40:00.362 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6450999999999999e-05 2024-09-18T04:40:00.362 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:00.362 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:00.362 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:00.362 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:00.362 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:00.362 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:40:00.364 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:40:00.365 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:40:00.359+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:40:00.465 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:40:00.459+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:40:00.479 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:40:00.479 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:40:00.479 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:40:00.479 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:40:00.490 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:40:00.566 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:40:00.563+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:40:00.592 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:40:00.592 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:40:00.592 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:40:00.592 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:40:00.592 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:40:00.601 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:40:00.667 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:40:00.664+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:40:00.709 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:40:00.709 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:40:00.709 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:40:00.709 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:40:00.709 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.709 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.5431.0:1 3.3 3:d80af3cb:::rbd_id.testrbdnbd17342:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-18T04:40:00.709 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:13.542818+0000", 2024-09-18T04:40:00.709 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 47.163991121000002, 2024-09-18T04:40:00.709 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048493400000000002, 2024-09-18T04:40:00.709 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:00.709 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:40:00.709 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:00.710 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.5431", 2024-09-18T04:40:00.710 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3147136083", 2024-09-18T04:40:00.710 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 1 2024-09-18T04:40:00.710 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.710 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:00.710 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.710 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:00.710 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542818+0000", 2024-09-18T04:40:00.710 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:00.710 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.710 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.710 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:00.710 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542818+0000", 2024-09-18T04:40:00.711 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:00.711 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.711 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.711 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:00.711 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542816+0000", 2024-09-18T04:40:00.711 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:00.711 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.711 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.711 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:00.711 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542828+0000", 2024-09-18T04:40:00.711 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2699e-05 2024-09-18T04:40:00.711 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.711 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.711 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:00.712 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542852+0000", 2024-09-18T04:40:00.712 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3176e-05 2024-09-18T04:40:00.712 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.712 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.712 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:00.712 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542874+0000", 2024-09-18T04:40:00.712 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1967000000000002e-05 2024-09-18T04:40:00.712 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.712 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.712 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:00.712 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542984+0000", 2024-09-18T04:40:00.712 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011026099999999999 2024-09-18T04:40:00.712 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.713 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.713 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:00.713 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543161+0000", 2024-09-18T04:40:00.713 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017720299999999999 2024-09-18T04:40:00.713 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.713 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.713 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:00.713 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543303+0000", 2024-09-18T04:40:00.713 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014174700000000001 2024-09-18T04:40:00.713 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:00.713 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:00.713 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:00.713 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.713 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.713 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.5431.0:2 3.3 3:d80af3cb:::rbd_id.testrbdnbd17342:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-18T04:40:00.714 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:13.543564+0000", 2024-09-18T04:40:00.714 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 47.163244634999998, 2024-09-18T04:40:00.714 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010399929999999999, 2024-09-18T04:40:00.714 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:00.714 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:40:00.714 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:00.714 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.5431", 2024-09-18T04:40:00.714 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3147136083", 2024-09-18T04:40:00.714 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 2 2024-09-18T04:40:00.714 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.714 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:00.714 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.714 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:00.715 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543564+0000", 2024-09-18T04:40:00.715 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:00.715 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.715 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.715 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:00.715 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543564+0000", 2024-09-18T04:40:00.715 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:00.715 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.715 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.715 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:00.715 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543563+0000", 2024-09-18T04:40:00.715 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:00.715 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.715 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.716 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:00.716 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543571+0000", 2024-09-18T04:40:00.716 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4270000000000008e-06 2024-09-18T04:40:00.716 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.716 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.716 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:00.716 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543591+0000", 2024-09-18T04:40:00.716 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.929e-05 2024-09-18T04:40:00.716 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.716 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.716 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:00.716 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543600+0000", 2024-09-18T04:40:00.716 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.1390000000000001e-06 2024-09-18T04:40:00.716 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.717 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.717 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:00.717 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543724+0000", 2024-09-18T04:40:00.717 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012458400000000001 2024-09-18T04:40:00.717 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.717 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.717 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:00.717 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543968+0000", 2024-09-18T04:40:00.717 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00024323199999999999 2024-09-18T04:40:00.717 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:00.717 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:00.717 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:00.717 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.544604+0000", 2024-09-18T04:40:00.717 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00063675799999999996 2024-09-18T04:40:00.718 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:00.718 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:00.718 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:00.718 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:00.718 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:00.718 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:40:00.719 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:40:00.767 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:40:00.763+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:40:00.800 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-09-18T04:40:00.869 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:40:00.863+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:40:00.871 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:40:00.872 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:40:00.872 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:40:00.872 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:40:00.881 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:40:00.969 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:40:00.963+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:40:00.986 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:40:00.986 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:40:00.986 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:40:00.986 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:40:00.986 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:40:00.996 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:40:01.070 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:40:01.067+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:40:01.100 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:40:01.100 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:40:01.100 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:40:01.100 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:40:01.100 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.100 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-18T04:40:01.100 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:40:01.100 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 146.624532937, 2024-09-18T04:40:01.100 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0095781189999999995, 2024-09-18T04:40:01.101 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:01.101 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:01.101 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:01.101 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:40:01.101 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:40:01.101 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 1 2024-09-18T04:40:01.101 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.101 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:01.101 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.101 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:01.101 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:40:01.101 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:01.101 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.102 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.102 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:01.102 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:40:01.102 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:01.102 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.102 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.102 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:01.102 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469862+0000", 2024-09-18T04:40:01.102 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:40:01.102 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.102 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.102 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:01.102 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469879+0000", 2024-09-18T04:40:01.102 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7431999999999998e-05 2024-09-18T04:40:01.103 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.103 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.103 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:01.103 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469896+0000", 2024-09-18T04:40:01.103 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6742999999999999e-05 2024-09-18T04:40:01.103 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.103 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.103 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:01.103 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469915+0000", 2024-09-18T04:40:01.103 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8991000000000002e-05 2024-09-18T04:40:01.103 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.103 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.103 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:01.103 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470397+0000", 2024-09-18T04:40:01.104 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048167799999999998 2024-09-18T04:40:01.104 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.104 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.104 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-09-18T04:40:01.104 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470412+0000", 2024-09-18T04:40:01.104 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5526e-05 2024-09-18T04:40:01.104 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.104 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.104 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:01.104 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473279+0000", 2024-09-18T04:40:01.104 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0028667469999999998 2024-09-18T04:40:01.104 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.104 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.105 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:01.105 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473441+0000", 2024-09-18T04:40:01.105 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000162082 2024-09-18T04:40:01.105 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.105 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.105 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:40:01.105 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473573+0000", 2024-09-18T04:40:01.105 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000132032 2024-09-18T04:40:01.105 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.105 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.105 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:01.105 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474826+0000", 2024-09-18T04:40:01.105 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001252717 2024-09-18T04:40:01.105 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.106 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.106 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:01.106 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479307+0000", 2024-09-18T04:40:01.106 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044813220000000003 2024-09-18T04:40:01.106 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.106 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.106 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:01.106 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479366+0000", 2024-09-18T04:40:01.106 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9024999999999998e-05 2024-09-18T04:40:01.106 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.106 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.106 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:01.106 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479443+0000", 2024-09-18T04:40:01.106 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6339000000000002e-05 2024-09-18T04:40:01.107 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:01.107 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:01.107 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:01.107 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.107 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.107 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-18T04:40:01.107 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:40:01.107 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 144.61152451699999, 2024-09-18T04:40:01.107 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082699560000000002, 2024-09-18T04:40:01.107 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:01.107 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:01.107 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:01.107 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:40:01.107 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:40:01.108 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 7 2024-09-18T04:40:01.108 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.108 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:01.108 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.108 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:01.108 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:40:01.108 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:01.108 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.108 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.108 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:01.108 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:40:01.108 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:01.108 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.109 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.109 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:01.109 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482870+0000", 2024-09-18T04:40:01.109 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999967 2024-09-18T04:40:01.109 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.109 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.109 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:01.109 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482889+0000", 2024-09-18T04:40:01.109 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.965e-05 2024-09-18T04:40:01.109 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.109 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.109 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:01.109 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482938+0000", 2024-09-18T04:40:01.110 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8548999999999999e-05 2024-09-18T04:40:01.110 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.110 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.110 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:01.110 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482963+0000", 2024-09-18T04:40:01.110 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5075999999999998e-05 2024-09-18T04:40:01.110 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.110 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.110 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:01.110 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483121+0000", 2024-09-18T04:40:01.110 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015819400000000001 2024-09-18T04:40:01.110 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.110 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.110 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op must wait for map", 2024-09-18T04:40:01.111 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483138+0000", 2024-09-18T04:40:01.111 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7068999999999999e-05 2024-09-18T04:40:01.111 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.111 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.111 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:01.111 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488666+0000", 2024-09-18T04:40:01.111 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0055282049999999996 2024-09-18T04:40:01.111 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.111 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.111 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:01.111 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488783+0000", 2024-09-18T04:40:01.111 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000116737 2024-09-18T04:40:01.111 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.111 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.112 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:01.112 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488991+0000", 2024-09-18T04:40:01.112 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020781400000000001 2024-09-18T04:40:01.112 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.112 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.112 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:01.112 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490554+0000", 2024-09-18T04:40:01.112 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0015628370000000001 2024-09-18T04:40:01.112 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.112 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.112 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:01.112 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491019+0000", 2024-09-18T04:40:01.112 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000465608 2024-09-18T04:40:01.112 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.113 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.113 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:01.113 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491087+0000", 2024-09-18T04:40:01.113 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7988e-05 2024-09-18T04:40:01.113 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.113 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.113 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:01.113 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491143+0000", 2024-09-18T04:40:01.113 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5538999999999998e-05 2024-09-18T04:40:01.113 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:01.113 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:01.113 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:01.113 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.114 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.114 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:34 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:01.114 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.261758+0000", 2024-09-18T04:40:01.114 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 78.832639365999995, 2024-09-18T04:40:01.114 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.042024191000000002, 2024-09-18T04:40:01.114 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:01.114 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:01.114 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:01.114 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:40:01.114 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:40:01.114 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 34 2024-09-18T04:40:01.114 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.114 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:01.115 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.115 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:01.115 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.261758+0000", 2024-09-18T04:40:01.115 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:01.115 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.115 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.115 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:01.115 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.261758+0000", 2024-09-18T04:40:01.115 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:01.115 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.115 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.115 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:01.115 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.261756+0000", 2024-09-18T04:40:01.116 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:01.116 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.116 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.116 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:01.116 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.270785+0000", 2024-09-18T04:40:01.116 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0090292340000000006 2024-09-18T04:40:01.116 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.116 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.116 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:01.116 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.270890+0000", 2024-09-18T04:40:01.116 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000104541 2024-09-18T04:40:01.116 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.116 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.117 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:01.117 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.270991+0000", 2024-09-18T04:40:01.117 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000100778 2024-09-18T04:40:01.117 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.117 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.117 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:01.117 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.271157+0000", 2024-09-18T04:40:01.117 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016653999999999999 2024-09-18T04:40:01.117 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.117 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.117 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:01.117 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.271591+0000", 2024-09-18T04:40:01.117 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00043409500000000001 2024-09-18T04:40:01.117 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.118 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.118 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:40:01.118 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272017+0000", 2024-09-18T04:40:01.118 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000425769 2024-09-18T04:40:01.118 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.118 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.118 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:01.118 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.294246+0000", 2024-09-18T04:40:01.118 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.022229237999999998 2024-09-18T04:40:01.118 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.118 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.118 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:01.118 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303571+0000", 2024-09-18T04:40:01.118 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0093250239999999995 2024-09-18T04:40:01.118 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.119 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.119 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:01.119 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303705+0000", 2024-09-18T04:40:01.119 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000133831 2024-09-18T04:40:01.119 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.119 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.119 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:01.119 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303782+0000", 2024-09-18T04:40:01.119 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7156000000000005e-05 2024-09-18T04:40:01.119 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:01.119 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:01.119 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:01.119 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.119 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.120 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:36 2.7 e51/32 2:fe1b02ad:::rbd_data.125fc09df6d1.0000000000000002:head v 51'1, mlcod=0'0)", 2024-09-18T04:40:01.120 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.273441+0000", 2024-09-18T04:40:01.120 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 78.820956385000002, 2024-09-18T04:40:01.120 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.032400768000000003, 2024-09-18T04:40:01.120 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:01.120 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:01.120 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:01.120 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.120 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:01.120 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273441+0000", 2024-09-18T04:40:01.120 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:01.120 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.120 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.120 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:01.120 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273441+0000", 2024-09-18T04:40:01.121 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:01.121 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.121 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.121 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:01.121 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273440+0000", 2024-09-18T04:40:01.121 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:40:01.121 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.121 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.121 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:01.121 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278723+0000", 2024-09-18T04:40:01.121 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0052829909999999999 2024-09-18T04:40:01.121 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.121 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.121 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:01.122 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278752+0000", 2024-09-18T04:40:01.122 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9178000000000001e-05 2024-09-18T04:40:01.122 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.122 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.122 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:01.122 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278761+0000", 2024-09-18T04:40:01.122 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7900000000000005e-06 2024-09-18T04:40:01.122 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.122 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.122 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:01.122 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278864+0000", 2024-09-18T04:40:01.122 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000102699 2024-09-18T04:40:01.122 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.122 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.122 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:01.123 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278900+0000", 2024-09-18T04:40:01.123 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6001999999999999e-05 2024-09-18T04:40:01.123 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.123 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.123 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:01.123 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.305793+0000", 2024-09-18T04:40:01.123 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.026893344 2024-09-18T04:40:01.123 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.123 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.123 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:01.123 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.305842+0000", 2024-09-18T04:40:01.123 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8720000000000001e-05 2024-09-18T04:40:01.123 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:01.123 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:01.124 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:01.124 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.124 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.124 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:39 2.7 e51/32 2:e9c909c5:::rbd_data.125fc09df6d1.0000000000000005:head v 51'2, mlcod=0'0)", 2024-09-18T04:40:01.124 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.283175+0000", 2024-09-18T04:40:01.124 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 78.811222696000002, 2024-09-18T04:40:01.124 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.022864102000000001, 2024-09-18T04:40:01.124 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:01.124 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:01.124 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:01.124 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.124 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:01.124 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283175+0000", 2024-09-18T04:40:01.125 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:01.125 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.125 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.125 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:01.125 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283175+0000", 2024-09-18T04:40:01.125 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:01.125 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.125 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.125 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:01.125 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283173+0000", 2024-09-18T04:40:01.125 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:01.125 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.125 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.125 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:01.125 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.285951+0000", 2024-09-18T04:40:01.126 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0027784540000000001 2024-09-18T04:40:01.126 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.126 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.126 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:01.126 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.285970+0000", 2024-09-18T04:40:01.126 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8855000000000002e-05 2024-09-18T04:40:01.126 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.126 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.126 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:01.126 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.285984+0000", 2024-09-18T04:40:01.126 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4252e-05 2024-09-18T04:40:01.126 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.126 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.127 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:01.127 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.286041+0000", 2024-09-18T04:40:01.127 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6431999999999998e-05 2024-09-18T04:40:01.127 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.127 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.127 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:01.127 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.286086+0000", 2024-09-18T04:40:01.127 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5423000000000002e-05 2024-09-18T04:40:01.127 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.127 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.127 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:01.127 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.305998+0000", 2024-09-18T04:40:01.127 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019911697999999999 2024-09-18T04:40:01.128 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.128 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.128 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:01.128 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306039+0000", 2024-09-18T04:40:01.128 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0973999999999998e-05 2024-09-18T04:40:01.128 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:01.128 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:01.128 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:01.128 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.128 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.128 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:50 2.3 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:01.128 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.312563+0000", 2024-09-18T04:40:01.128 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 78.781834669000006, 2024-09-18T04:40:01.129 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.032636001999999997, 2024-09-18T04:40:01.129 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:01.129 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:01.129 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:01.129 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:40:01.129 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:40:01.129 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 50 2024-09-18T04:40:01.129 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.129 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:01.129 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.129 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:01.129 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.312563+0000", 2024-09-18T04:40:01.129 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:01.129 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.130 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.130 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:01.130 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.312563+0000", 2024-09-18T04:40:01.130 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:01.130 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.130 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.130 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:01.130 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.312561+0000", 2024-09-18T04:40:01.130 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:01.130 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.130 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.130 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:01.130 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321709+0000", 2024-09-18T04:40:01.130 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0091486650000000003 2024-09-18T04:40:01.131 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.131 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.131 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:01.131 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321739+0000", 2024-09-18T04:40:01.131 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9666e-05 2024-09-18T04:40:01.131 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.131 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.131 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:01.131 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321751+0000", 2024-09-18T04:40:01.131 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2086000000000001e-05 2024-09-18T04:40:01.131 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.131 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.131 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:01.132 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321833+0000", 2024-09-18T04:40:01.132 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1842999999999996e-05 2024-09-18T04:40:01.132 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.132 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.132 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:01.132 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321981+0000", 2024-09-18T04:40:01.132 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000147441 2024-09-18T04:40:01.132 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.132 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.132 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:40:01.132 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.322109+0000", 2024-09-18T04:40:01.132 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012851200000000001 2024-09-18T04:40:01.132 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.132 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.133 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:01.133 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.332829+0000", 2024-09-18T04:40:01.133 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010720225999999999 2024-09-18T04:40:01.133 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.133 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.133 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:01.133 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.345039+0000", 2024-09-18T04:40:01.133 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012209831000000001 2024-09-18T04:40:01.133 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.133 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.133 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:01.133 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.345120+0000", 2024-09-18T04:40:01.133 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1145999999999999e-05 2024-09-18T04:40:01.134 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.134 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.134 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:01.134 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.345199+0000", 2024-09-18T04:40:01.134 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8546000000000003e-05 2024-09-18T04:40:01.134 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:01.134 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:01.134 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:01.134 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.134 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.134 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:30 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:01.134 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.516122+0000", 2024-09-18T04:40:01.134 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 78.578275507000001, 2024-09-18T04:40:01.134 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0091865030000000004, 2024-09-18T04:40:01.135 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:01.135 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:01.135 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:01.135 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:40:01.135 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:01.135 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 30 2024-09-18T04:40:01.135 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.135 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:01.135 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.135 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:01.135 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516122+0000", 2024-09-18T04:40:01.135 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:01.135 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.135 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.136 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:01.136 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516122+0000", 2024-09-18T04:40:01.136 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:01.136 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.136 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.136 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:01.136 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516120+0000", 2024-09-18T04:40:01.136 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:01.136 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.136 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.136 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:01.136 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516129+0000", 2024-09-18T04:40:01.137 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.85e-06 2024-09-18T04:40:01.137 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.137 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.137 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:01.137 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516155+0000", 2024-09-18T04:40:01.137 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5882000000000001e-05 2024-09-18T04:40:01.137 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.137 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.137 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:01.137 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516175+0000", 2024-09-18T04:40:01.137 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.022e-05 2024-09-18T04:40:01.137 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.137 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.137 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:01.138 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519780+0000", 2024-09-18T04:40:01.138 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003604187 2024-09-18T04:40:01.138 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.138 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.138 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:01.138 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519964+0000", 2024-09-18T04:40:01.138 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00018391999999999999 2024-09-18T04:40:01.138 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.138 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.138 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:40:01.138 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.520794+0000", 2024-09-18T04:40:01.138 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00083078100000000005 2024-09-18T04:40:01.138 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.138 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.139 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:01.139 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.524129+0000", 2024-09-18T04:40:01.139 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033342419999999998 2024-09-18T04:40:01.139 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.139 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.139 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:01.139 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525239+0000", 2024-09-18T04:40:01.139 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001110778 2024-09-18T04:40:01.139 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.139 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.139 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:01.139 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525277+0000", 2024-09-18T04:40:01.139 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7596999999999999e-05 2024-09-18T04:40:01.139 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.140 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.140 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:01.140 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525308+0000", 2024-09-18T04:40:01.140 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.1535999999999998e-05 2024-09-18T04:40:01.140 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:01.140 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:01.140 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:01.140 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.140 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.140 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:32 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:01.140 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.519217+0000", 2024-09-18T04:40:01.140 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 78.575180187000001, 2024-09-18T04:40:01.140 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082266370000000002, 2024-09-18T04:40:01.141 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:01.141 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:01.141 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:01.141 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:40:01.141 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:01.141 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 32 2024-09-18T04:40:01.141 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.141 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:01.141 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.141 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:01.141 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519217+0000", 2024-09-18T04:40:01.141 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:01.141 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.141 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.142 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:01.142 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519217+0000", 2024-09-18T04:40:01.142 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:01.142 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.142 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.142 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:01.142 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519215+0000", 2024-09-18T04:40:01.142 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:01.142 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.142 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.142 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:01.142 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519229+0000", 2024-09-18T04:40:01.142 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3329999999999999e-05 2024-09-18T04:40:01.143 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.143 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.143 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:01.143 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519268+0000", 2024-09-18T04:40:01.143 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9533000000000002e-05 2024-09-18T04:40:01.143 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.143 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.143 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:01.143 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519285+0000", 2024-09-18T04:40:01.143 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6446000000000001e-05 2024-09-18T04:40:01.143 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.143 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.143 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:01.143 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.522358+0000", 2024-09-18T04:40:01.144 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030734310000000002 2024-09-18T04:40:01.144 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.144 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.144 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:01.144 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.522551+0000", 2024-09-18T04:40:01.144 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019317600000000001 2024-09-18T04:40:01.144 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.144 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.144 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:40:01.144 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.523262+0000", 2024-09-18T04:40:01.144 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00071069999999999998 2024-09-18T04:40:01.144 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.144 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.145 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:01.145 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525318+0000", 2024-09-18T04:40:01.145 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0020561469999999999 2024-09-18T04:40:01.145 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.145 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.145 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:01.145 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.527376+0000", 2024-09-18T04:40:01.145 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0020584969999999998 2024-09-18T04:40:01.145 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.145 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.145 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:01.145 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.527416+0000", 2024-09-18T04:40:01.145 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9054000000000003e-05 2024-09-18T04:40:01.146 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.146 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.146 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:01.146 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.527444+0000", 2024-09-18T04:40:01.146 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8347000000000002e-05 2024-09-18T04:40:01.146 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:01.146 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:01.146 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:01.146 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.146 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.146 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:42 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:01.146 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.551160+0000", 2024-09-18T04:40:01.146 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 78.543237794999996, 2024-09-18T04:40:01.146 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010692757000000001, 2024-09-18T04:40:01.147 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:01.147 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:01.147 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:01.147 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:40:01.147 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:01.147 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 42 2024-09-18T04:40:01.147 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.147 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:01.147 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.147 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:01.147 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551160+0000", 2024-09-18T04:40:01.147 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:01.147 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.148 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.148 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:01.148 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551160+0000", 2024-09-18T04:40:01.148 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:01.148 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.148 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.148 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:01.148 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551159+0000", 2024-09-18T04:40:01.148 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:40:01.148 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.148 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.148 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:01.148 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551166+0000", 2024-09-18T04:40:01.149 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1029999999999996e-06 2024-09-18T04:40:01.149 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.149 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.149 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:01.149 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551177+0000", 2024-09-18T04:40:01.149 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1384e-05 2024-09-18T04:40:01.149 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.149 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.149 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:01.149 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551185+0000", 2024-09-18T04:40:01.149 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1990000000000006e-06 2024-09-18T04:40:01.149 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.149 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.150 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:01.150 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552021+0000", 2024-09-18T04:40:01.150 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00083574400000000003 2024-09-18T04:40:01.150 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.150 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.150 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:01.150 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552339+0000", 2024-09-18T04:40:01.150 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031771399999999997 2024-09-18T04:40:01.150 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.150 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.150 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:40:01.150 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.553140+0000", 2024-09-18T04:40:01.150 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00080088100000000003 2024-09-18T04:40:01.151 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.151 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.151 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:01.151 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.554585+0000", 2024-09-18T04:40:01.151 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001445801 2024-09-18T04:40:01.151 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.151 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.151 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:01.151 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.561711+0000", 2024-09-18T04:40:01.151 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071260109999999998 2024-09-18T04:40:01.151 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.151 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.151 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:01.152 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.561790+0000", 2024-09-18T04:40:01.152 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8967000000000005e-05 2024-09-18T04:40:01.152 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.152 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.152 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:01.152 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.561852+0000", 2024-09-18T04:40:01.152 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2045999999999995e-05 2024-09-18T04:40:01.152 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:01.152 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:01.152 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:01.152 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.152 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.152 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:58 2.7 e51/32 2:fc7c6cd3:ns::rbd_data.133e813ab5bc.000000000000000e:head v 51'7, mlcod=51'5)", 2024-09-18T04:40:01.152 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.605333+0000", 2024-09-18T04:40:01.153 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 78.489064589999998, 2024-09-18T04:40:01.153 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020085902999999999, 2024-09-18T04:40:01.153 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:01.153 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:01.153 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:01.153 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.153 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:01.153 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605333+0000", 2024-09-18T04:40:01.153 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:01.153 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.153 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.153 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:01.153 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605333+0000", 2024-09-18T04:40:01.154 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:01.154 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.154 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.154 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:01.154 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605332+0000", 2024-09-18T04:40:01.154 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:01.154 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.154 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.154 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:01.154 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608667+0000", 2024-09-18T04:40:01.154 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033350300000000001 2024-09-18T04:40:01.154 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.154 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.155 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:01.155 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608706+0000", 2024-09-18T04:40:01.155 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8951e-05 2024-09-18T04:40:01.155 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.155 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.155 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:01.155 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608725+0000", 2024-09-18T04:40:01.155 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9675999999999999e-05 2024-09-18T04:40:01.155 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.155 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.155 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:01.155 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608798+0000", 2024-09-18T04:40:01.155 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3135999999999999e-05 2024-09-18T04:40:01.156 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.156 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.156 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:01.156 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608824+0000", 2024-09-18T04:40:01.156 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5284000000000001e-05 2024-09-18T04:40:01.156 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.156 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.156 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:01.156 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625370+0000", 2024-09-18T04:40:01.156 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016545975000000001 2024-09-18T04:40:01.156 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.156 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.157 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:01.157 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625419+0000", 2024-09-18T04:40:01.157 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9076999999999998e-05 2024-09-18T04:40:01.157 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:01.157 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:01.157 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:01.157 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.157 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.157 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:611 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:40:01.157 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.069561+0000", 2024-09-18T04:40:01.157 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 70.024836035999996, 2024-09-18T04:40:01.157 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0072330299999999997, 2024-09-18T04:40:01.157 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:01.158 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:01.158 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:01.158 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:40:01.158 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:40:01.158 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 611 2024-09-18T04:40:01.158 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.158 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:01.158 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.158 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:01.158 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.069561+0000", 2024-09-18T04:40:01.158 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:01.158 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.158 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.159 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:01.159 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.069561+0000", 2024-09-18T04:40:01.159 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:01.159 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.159 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.159 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:01.159 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.069559+0000", 2024-09-18T04:40:01.159 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:40:01.159 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.159 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.159 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:01.159 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070801+0000", 2024-09-18T04:40:01.159 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0012423359999999999 2024-09-18T04:40:01.159 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.160 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.160 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:01.160 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070828+0000", 2024-09-18T04:40:01.160 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6387e-05 2024-09-18T04:40:01.160 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.160 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.160 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:01.160 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070840+0000", 2024-09-18T04:40:01.160 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2689e-05 2024-09-18T04:40:01.160 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.160 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.160 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:01.160 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070990+0000", 2024-09-18T04:40:01.161 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014946800000000001 2024-09-18T04:40:01.161 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.161 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.161 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:01.161 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.071128+0000", 2024-09-18T04:40:01.161 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013796699999999999 2024-09-18T04:40:01.161 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.161 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.161 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:40:01.161 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.071318+0000", 2024-09-18T04:40:01.161 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000190106 2024-09-18T04:40:01.161 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.161 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.162 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:01.162 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.075711+0000", 2024-09-18T04:40:01.162 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.004393002 2024-09-18T04:40:01.162 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.162 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.162 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:01.162 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.076665+0000", 2024-09-18T04:40:01.162 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00095390399999999999 2024-09-18T04:40:01.162 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.162 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.162 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:01.162 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.076742+0000", 2024-09-18T04:40:01.162 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7112000000000003e-05 2024-09-18T04:40:01.162 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.162 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.163 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:01.163 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.076794+0000", 2024-09-18T04:40:01.163 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2475999999999997e-05 2024-09-18T04:40:01.163 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:01.163 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:01.163 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:01.163 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.163 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.163 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:687 2.3 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:40:01.163 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.487994+0000", 2024-09-18T04:40:01.163 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 69.606403052999994, 2024-09-18T04:40:01.163 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0074167690000000001, 2024-09-18T04:40:01.164 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:01.164 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:01.164 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:01.164 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:40:01.164 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:40:01.164 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 687 2024-09-18T04:40:01.164 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.164 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:01.164 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.164 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:01.164 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.487994+0000", 2024-09-18T04:40:01.164 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:01.164 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.164 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.165 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:01.165 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.487994+0000", 2024-09-18T04:40:01.165 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:01.165 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.165 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.165 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:01.165 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.487993+0000", 2024-09-18T04:40:01.165 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:40:01.165 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.165 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.165 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:01.165 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.488979+0000", 2024-09-18T04:40:01.165 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00098620700000000001 2024-09-18T04:40:01.166 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.166 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.166 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:01.166 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489002+0000", 2024-09-18T04:40:01.166 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2500000000000001e-05 2024-09-18T04:40:01.166 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.166 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.166 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:01.166 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489020+0000", 2024-09-18T04:40:01.166 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7691000000000001e-05 2024-09-18T04:40:01.166 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.166 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.166 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:01.166 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489091+0000", 2024-09-18T04:40:01.167 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1307e-05 2024-09-18T04:40:01.167 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.167 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.167 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:01.167 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489219+0000", 2024-09-18T04:40:01.167 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012766000000000001 2024-09-18T04:40:01.167 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.167 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.167 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:40:01.167 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489387+0000", 2024-09-18T04:40:01.167 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016855400000000001 2024-09-18T04:40:01.167 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.167 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.167 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:01.168 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495243+0000", 2024-09-18T04:40:01.168 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058563189999999996 2024-09-18T04:40:01.168 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.168 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.168 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:01.168 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495266+0000", 2024-09-18T04:40:01.168 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2810999999999999e-05 2024-09-18T04:40:01.168 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.168 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.168 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:01.168 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495332+0000", 2024-09-18T04:40:01.168 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5469999999999995e-05 2024-09-18T04:40:01.168 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.169 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.169 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:01.169 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495411+0000", 2024-09-18T04:40:01.169 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9395999999999997e-05 2024-09-18T04:40:01.169 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:01.169 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:01.169 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:01.169 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.169 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.169 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:689 2.3 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:40:01.169 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.500070+0000", 2024-09-18T04:40:01.169 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 69.594327237000002, 2024-09-18T04:40:01.169 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071096650000000003, 2024-09-18T04:40:01.170 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:01.170 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:01.170 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:01.170 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:40:01.170 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:40:01.170 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 689 2024-09-18T04:40:01.170 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.170 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:01.170 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.170 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:01.170 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.500070+0000", 2024-09-18T04:40:01.170 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:01.170 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.170 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.171 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:01.171 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.500070+0000", 2024-09-18T04:40:01.171 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:01.171 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.171 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.171 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:01.171 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.500069+0000", 2024-09-18T04:40:01.171 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:01.171 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.171 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.171 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:01.171 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501082+0000", 2024-09-18T04:40:01.171 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010136570000000001 2024-09-18T04:40:01.172 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.172 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.172 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:01.172 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501104+0000", 2024-09-18T04:40:01.172 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1883e-05 2024-09-18T04:40:01.172 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.172 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.172 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:01.172 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501115+0000", 2024-09-18T04:40:01.172 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0414000000000001e-05 2024-09-18T04:40:01.172 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.172 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.172 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:01.173 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501178+0000", 2024-09-18T04:40:01.173 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3128999999999998e-05 2024-09-18T04:40:01.173 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.173 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.173 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:01.173 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501257+0000", 2024-09-18T04:40:01.173 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9592999999999995e-05 2024-09-18T04:40:01.173 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.173 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.173 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:40:01.173 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501366+0000", 2024-09-18T04:40:01.173 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000108643 2024-09-18T04:40:01.173 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.173 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.174 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:01.174 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.504041+0000", 2024-09-18T04:40:01.174 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0026750889999999999 2024-09-18T04:40:01.174 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.174 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.174 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:01.174 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.507097+0000", 2024-09-18T04:40:01.174 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030556070000000001 2024-09-18T04:40:01.174 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.174 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.174 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:01.174 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.507140+0000", 2024-09-18T04:40:01.174 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3745000000000002e-05 2024-09-18T04:40:01.174 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.175 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.175 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:01.175 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.507180+0000", 2024-09-18T04:40:01.175 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9475999999999999e-05 2024-09-18T04:40:01.175 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:01.175 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:01.175 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:01.175 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.175 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.175 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4972.0:13 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:40:01.175 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.948263+0000", 2024-09-18T04:40:01.175 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 69.146134446000005, 2024-09-18T04:40:01.175 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0091358970000000005, 2024-09-18T04:40:01.176 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:01.176 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:40:01.176 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:01.176 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4972", 2024-09-18T04:40:01.176 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2280251472", 2024-09-18T04:40:01.176 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 13 2024-09-18T04:40:01.176 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.176 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:01.176 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.176 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:01.176 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948263+0000", 2024-09-18T04:40:01.176 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:01.176 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.176 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.177 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:01.177 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948263+0000", 2024-09-18T04:40:01.177 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:01.177 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.177 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.177 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:01.177 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948261+0000", 2024-09-18T04:40:01.177 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:01.177 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.177 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.177 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:01.177 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948273+0000", 2024-09-18T04:40:01.178 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1576999999999999e-05 2024-09-18T04:40:01.178 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.178 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.178 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:01.178 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948305+0000", 2024-09-18T04:40:01.178 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2209000000000003e-05 2024-09-18T04:40:01.178 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.178 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.178 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:01.178 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948325+0000", 2024-09-18T04:40:01.178 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.022e-05 2024-09-18T04:40:01.178 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.178 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.179 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:01.179 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948455+0000", 2024-09-18T04:40:01.179 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000129624 2024-09-18T04:40:01.179 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.179 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.179 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:01.179 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948598+0000", 2024-09-18T04:40:01.179 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014376699999999999 2024-09-18T04:40:01.179 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.179 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.179 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:01.179 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.957399+0000", 2024-09-18T04:40:01.179 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0088004299999999997 2024-09-18T04:40:01.179 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:01.180 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:01.180 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:01.180 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.180 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.180 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:101 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:40:01.180 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.576367+0000", 2024-09-18T04:40:01.180 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 64.518030737999993, 2024-09-18T04:40:01.180 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0078671020000000008, 2024-09-18T04:40:01.180 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:01.180 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:01.180 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:01.180 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:40:01.180 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:40:01.180 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 101 2024-09-18T04:40:01.181 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.181 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:01.181 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.181 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:01.181 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576367+0000", 2024-09-18T04:40:01.181 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:01.181 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.181 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.181 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:01.181 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576367+0000", 2024-09-18T04:40:01.181 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:01.181 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.181 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.181 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:01.182 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576364+0000", 2024-09-18T04:40:01.182 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999971 2024-09-18T04:40:01.182 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.182 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.182 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:01.182 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576378+0000", 2024-09-18T04:40:01.182 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3879999999999999e-05 2024-09-18T04:40:01.182 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.182 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.182 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:01.182 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576410+0000", 2024-09-18T04:40:01.182 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2286000000000002e-05 2024-09-18T04:40:01.182 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.182 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.182 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:01.183 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576425+0000", 2024-09-18T04:40:01.183 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5290000000000001e-05 2024-09-18T04:40:01.183 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.183 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.183 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:01.183 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576551+0000", 2024-09-18T04:40:01.183 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012555199999999999 2024-09-18T04:40:01.183 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.183 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.183 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:01.183 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576719+0000", 2024-09-18T04:40:01.183 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016787400000000001 2024-09-18T04:40:01.183 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.184 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.184 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:40:01.184 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576951+0000", 2024-09-18T04:40:01.184 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00023252400000000001 2024-09-18T04:40:01.184 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.184 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.184 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:01.184 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.581459+0000", 2024-09-18T04:40:01.184 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045074410000000001 2024-09-18T04:40:01.184 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.184 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.184 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:01.184 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584076+0000", 2024-09-18T04:40:01.184 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0026173860000000002 2024-09-18T04:40:01.185 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.185 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.185 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:01.185 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584160+0000", 2024-09-18T04:40:01.185 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4191999999999996e-05 2024-09-18T04:40:01.185 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.185 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.185 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:01.185 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584234+0000", 2024-09-18T04:40:01.185 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.339e-05 2024-09-18T04:40:01.185 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:01.185 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:01.185 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:01.185 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.186 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.186 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:44 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:40:01.186 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.550448+0000", 2024-09-18T04:40:01.186 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 60.543949548999997, 2024-09-18T04:40:01.186 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082117590000000008, 2024-09-18T04:40:01.186 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:01.186 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:01.186 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:01.186 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:40:01.186 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:40:01.186 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 44 2024-09-18T04:40:01.186 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.186 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:01.187 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.187 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:01.187 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.550448+0000", 2024-09-18T04:40:01.187 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:01.187 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.187 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.187 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:01.187 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.550448+0000", 2024-09-18T04:40:01.187 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:01.187 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.187 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.187 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:01.187 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.550445+0000", 2024-09-18T04:40:01.188 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999971 2024-09-18T04:40:01.188 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.188 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.188 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:01.188 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.551904+0000", 2024-09-18T04:40:01.188 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001458322 2024-09-18T04:40:01.188 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.188 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.188 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:01.188 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.551938+0000", 2024-09-18T04:40:01.188 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4359000000000001e-05 2024-09-18T04:40:01.188 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.189 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.189 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:01.189 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.551959+0000", 2024-09-18T04:40:01.189 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1403000000000002e-05 2024-09-18T04:40:01.189 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.189 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.189 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:01.189 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.552049+0000", 2024-09-18T04:40:01.189 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9970999999999996e-05 2024-09-18T04:40:01.189 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.189 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.189 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:01.189 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.552190+0000", 2024-09-18T04:40:01.189 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000140271 2024-09-18T04:40:01.190 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.190 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.190 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:40:01.190 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.552385+0000", 2024-09-18T04:40:01.190 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019498700000000001 2024-09-18T04:40:01.190 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.190 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.190 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:01.190 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.556308+0000", 2024-09-18T04:40:01.190 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00392333 2024-09-18T04:40:01.190 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.190 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.191 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:01.191 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.558479+0000", 2024-09-18T04:40:01.191 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0021715889999999998 2024-09-18T04:40:01.191 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.191 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.191 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:01.191 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.558616+0000", 2024-09-18T04:40:01.191 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013613399999999999 2024-09-18T04:40:01.191 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.191 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.191 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:01.191 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.558660+0000", 2024-09-18T04:40:01.192 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4053e-05 2024-09-18T04:40:01.192 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:01.192 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:01.192 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:01.192 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.192 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.192 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:46 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:40:01.192 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.564089+0000", 2024-09-18T04:40:01.192 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 60.530308419999997, 2024-09-18T04:40:01.192 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0069437090000000002, 2024-09-18T04:40:01.192 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:01.192 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:01.192 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:01.192 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:40:01.193 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:40:01.193 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 46 2024-09-18T04:40:01.193 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.193 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:01.193 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.193 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:01.193 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.564089+0000", 2024-09-18T04:40:01.193 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:01.193 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.193 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.193 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:01.193 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.564089+0000", 2024-09-18T04:40:01.193 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:01.194 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.194 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.194 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:01.194 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.564087+0000", 2024-09-18T04:40:01.194 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:01.194 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.194 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.194 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:01.194 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565126+0000", 2024-09-18T04:40:01.194 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001038902 2024-09-18T04:40:01.194 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.194 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.194 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:01.194 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565148+0000", 2024-09-18T04:40:01.195 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1911000000000001e-05 2024-09-18T04:40:01.195 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.195 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.195 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:01.195 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565157+0000", 2024-09-18T04:40:01.195 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6220000000000004e-06 2024-09-18T04:40:01.195 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.195 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.195 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:01.195 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565261+0000", 2024-09-18T04:40:01.195 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000103063 2024-09-18T04:40:01.195 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.195 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.196 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:01.196 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565394+0000", 2024-09-18T04:40:01.196 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000133032 2024-09-18T04:40:01.196 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.196 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.196 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:40:01.196 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565564+0000", 2024-09-18T04:40:01.196 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017068000000000001 2024-09-18T04:40:01.196 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.196 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.196 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:01.196 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.570556+0000", 2024-09-18T04:40:01.196 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0049920620000000002 2024-09-18T04:40:01.197 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.197 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.197 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:01.197 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.570885+0000", 2024-09-18T04:40:01.197 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032827900000000001 2024-09-18T04:40:01.197 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.197 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.197 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:01.197 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.570959+0000", 2024-09-18T04:40:01.197 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3955999999999995e-05 2024-09-18T04:40:01.197 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.197 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.197 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:01.197 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.571033+0000", 2024-09-18T04:40:01.198 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4191999999999997e-05 2024-09-18T04:40:01.198 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:01.198 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:01.198 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:01.198 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.198 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.198 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:100 2.4 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:40:01.198 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.856389+0000", 2024-09-18T04:40:01.198 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 60.238008307999998, 2024-09-18T04:40:01.198 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0081480800000000003, 2024-09-18T04:40:01.198 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:01.198 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:01.198 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:01.199 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:40:01.199 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:40:01.199 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 100 2024-09-18T04:40:01.199 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.199 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:01.199 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.199 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:01.199 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.856389+0000", 2024-09-18T04:40:01.199 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:01.199 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.199 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.199 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:01.199 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.856389+0000", 2024-09-18T04:40:01.199 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:01.200 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.200 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.200 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:01.200 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.856388+0000", 2024-09-18T04:40:01.200 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:40:01.200 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.200 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.200 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:01.200 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857295+0000", 2024-09-18T04:40:01.200 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00090725599999999995 2024-09-18T04:40:01.200 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.200 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.201 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:01.201 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857311+0000", 2024-09-18T04:40:01.201 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6092000000000001e-05 2024-09-18T04:40:01.201 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.201 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.201 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:01.201 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857319+0000", 2024-09-18T04:40:01.201 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9200000000000004e-06 2024-09-18T04:40:01.201 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.201 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.201 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:01.201 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857388+0000", 2024-09-18T04:40:01.201 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8591999999999997e-05 2024-09-18T04:40:01.202 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.202 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.202 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:01.202 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857535+0000", 2024-09-18T04:40:01.202 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014684299999999999 2024-09-18T04:40:01.202 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.202 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.202 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:40:01.202 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857747+0000", 2024-09-18T04:40:01.202 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000212361 2024-09-18T04:40:01.202 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.202 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.202 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:01.202 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.860831+0000", 2024-09-18T04:40:01.203 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030841789999999998 2024-09-18T04:40:01.203 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.203 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.203 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:01.203 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.864361+0000", 2024-09-18T04:40:01.203 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0035298780000000002 2024-09-18T04:40:01.203 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.203 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.203 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:01.203 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.864456+0000", 2024-09-18T04:40:01.203 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4557999999999998e-05 2024-09-18T04:40:01.203 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.203 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.204 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:01.204 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.864537+0000", 2024-09-18T04:40:01.204 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1577999999999997e-05 2024-09-18T04:40:01.204 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:01.204 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:01.204 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:01.204 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.204 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.204 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:102 2.4 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:40:01.204 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.869773+0000", 2024-09-18T04:40:01.204 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 60.224624650999999, 2024-09-18T04:40:01.204 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0084430359999999992, 2024-09-18T04:40:01.204 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:01.204 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:01.205 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:01.205 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:40:01.205 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:40:01.205 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 102 2024-09-18T04:40:01.205 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.205 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:01.205 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.205 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:01.205 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.869773+0000", 2024-09-18T04:40:01.205 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:01.205 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.205 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.205 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:01.205 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.869773+0000", 2024-09-18T04:40:01.206 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:01.206 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.206 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.206 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:01.206 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.869771+0000", 2024-09-18T04:40:01.206 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:01.206 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.206 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.206 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:01.206 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870629+0000", 2024-09-18T04:40:01.206 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00085760999999999997 2024-09-18T04:40:01.206 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.206 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.206 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:01.207 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870647+0000", 2024-09-18T04:40:01.207 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8363999999999999e-05 2024-09-18T04:40:01.207 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.207 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.207 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:01.207 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870655+0000", 2024-09-18T04:40:01.207 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0099999999999995e-06 2024-09-18T04:40:01.207 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.207 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.207 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:01.207 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870803+0000", 2024-09-18T04:40:01.207 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014798 2024-09-18T04:40:01.207 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.208 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.208 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:01.208 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870951+0000", 2024-09-18T04:40:01.208 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014715999999999999 2024-09-18T04:40:01.208 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.208 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.208 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:40:01.208 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.871145+0000", 2024-09-18T04:40:01.208 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000194253 2024-09-18T04:40:01.208 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.208 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.208 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:01.208 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.877033+0000", 2024-09-18T04:40:01.208 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058883099999999999 2024-09-18T04:40:01.209 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.209 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.209 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:01.209 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.878078+0000", 2024-09-18T04:40:01.209 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001044747 2024-09-18T04:40:01.209 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.209 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.209 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:01.209 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.878160+0000", 2024-09-18T04:40:01.209 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1904999999999996e-05 2024-09-18T04:40:01.209 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.209 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.209 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:01.210 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.878216+0000", 2024-09-18T04:40:01.210 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6082000000000003e-05 2024-09-18T04:40:01.210 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:01.210 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:01.210 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:01.210 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.210 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.210 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:145 2.4 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:40:01.210 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:01.096935+0000", 2024-09-18T04:40:01.210 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 59.997462871000003, 2024-09-18T04:40:01.210 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0069937869999999996, 2024-09-18T04:40:01.210 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:01.210 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:01.210 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:01.211 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:40:01.211 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:40:01.211 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 145 2024-09-18T04:40:01.211 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.211 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:01.211 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.211 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:01.211 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.096935+0000", 2024-09-18T04:40:01.211 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:01.211 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.211 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.211 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:01.211 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.096935+0000", 2024-09-18T04:40:01.211 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:01.211 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.212 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.212 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:01.212 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.096932+0000", 2024-09-18T04:40:01.212 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:40:01.212 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.212 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.212 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:01.212 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097815+0000", 2024-09-18T04:40:01.212 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00088276000000000001 2024-09-18T04:40:01.212 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.212 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.212 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:01.212 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097842+0000", 2024-09-18T04:40:01.212 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6806999999999999e-05 2024-09-18T04:40:01.213 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.213 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.213 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:01.213 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097852+0000", 2024-09-18T04:40:01.213 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0548e-05 2024-09-18T04:40:01.213 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.213 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.213 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:01.213 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097931+0000", 2024-09-18T04:40:01.213 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8417000000000005e-05 2024-09-18T04:40:01.213 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.213 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.213 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:01.214 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.098047+0000", 2024-09-18T04:40:01.214 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011611 2024-09-18T04:40:01.214 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.214 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.214 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:40:01.214 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.098172+0000", 2024-09-18T04:40:01.214 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000125482 2024-09-18T04:40:01.214 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.214 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.214 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:01.214 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.100888+0000", 2024-09-18T04:40:01.214 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0027154430000000001 2024-09-18T04:40:01.214 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.214 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.215 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:01.215 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.103814+0000", 2024-09-18T04:40:01.215 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0029259619999999998 2024-09-18T04:40:01.215 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.215 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.215 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:01.215 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.103891+0000", 2024-09-18T04:40:01.215 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7619000000000003e-05 2024-09-18T04:40:01.215 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:01.215 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:01.215 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:01.215 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.103928+0000", 2024-09-18T04:40:01.215 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6900999999999998e-05 2024-09-18T04:40:01.216 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:01.216 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:01.216 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:01.216 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:01.216 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:01.216 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:40:01.217 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:40:01.218 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:40:01.211+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:40:01.268 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T04:40:01.318 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:40:01.316+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:40:01.335 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:40:01.335 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:40:01.335 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:40:01.335 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:40:01.345 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:40:01.414 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:40:01.419 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:40:01.415+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:40:01.434 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:40:01.446 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:40:01.446 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:40:01.446 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:40:01.446 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:40:01.447 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:40:01.455 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:40:01.456 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:40:01.480 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:40:01.503 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:40:01.520 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:40:01.515+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:40:01.526 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:40:01.529 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:40:01.529 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:40:01.529 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:40:01.529 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:40:01.529 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:40:01.529 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:40:01.574 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:01.574 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:01.584 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:40:01.621 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:40:01.616+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:40:01.722 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:40:01.720+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:40:01.741 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:01.741 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:01.752 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:40:01.823 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:40:01.820+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:40:01.859 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:01.859 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:01.869 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:40:01.924 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:40:01.920+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:40:01.975 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:01.975 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:01.985 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:40:02.025 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:40:02.020+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:40:02.095 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:02.096 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:02.105 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:40:02.150 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:40:02.147+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:40:02.214 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:02.214 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:02.224 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:40:02.226 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:40:02.223+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:40:02.327 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:40:02.324+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:40:02.331 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:02.331 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:02.341 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:40:02.428 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:40:02.424+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:40:02.456 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:02.456 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:02.467 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:40:02.529 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:40:02.523+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:40:02.571 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:02.571 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:02.581 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:40:02.630 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:40:02.624+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:40:02.686 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:02.686 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:02.696 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:40:02.731 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:40:02.727+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:40:02.803 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:02.804 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:02.814 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:40:02.818 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-18T04:40:02.866 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:40:02.863+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:40:02.917 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:02.918 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:02.927 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:40:02.933 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:40:02.927+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:40:03.034 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:40:03.032+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:40:03.035 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:03.036 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:03.045 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:40:03.135 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:40:03.132+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:40:03.198 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:03.198 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:03.208 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:40:03.236 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:40:03.231+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:40:03.311 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:03.311 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:03.321 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:40:03.337 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:40:03.335+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:40:03.431 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:03.431 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:03.438 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:40:03.431+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:40:03.441 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:40:03.539 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:40:03.535+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:40:03.540 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T04:40:03.559 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:03.559 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:03.573 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:40:03.641 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:40:03.635+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:40:03.702 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:40:03.704 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:03.704 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:03.714 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:40:03.728 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:40:03.741 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:40:03.735+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:40:03.749 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:40:03.771 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:40:03.793 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 2] out_osds: [4] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 2] 2024-09-18T04:40:03.793 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T04:40:03.793 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-09-18T04:40:03.793 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 4 2024-09-18T04:40:03.796 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:40:03.823 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:40:03.824 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:40:03.824 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:40:03.824 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:40:03.824 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.824 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:7, version: 79'62, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:7@79'62, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 8=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-18T04:40:03.824 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.724969+0000", 2024-09-18T04:40:03.824 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.095248991999998, 2024-09-18T04:40:03.824 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0035525930000000002, 2024-09-18T04:40:03.824 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:03.824 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:03.824 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:03.824 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.825 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:03.825 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.724969+0000", 2024-09-18T04:40:03.825 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:03.825 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.825 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.825 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:03.825 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.724969+0000", 2024-09-18T04:40:03.825 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:03.825 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.825 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.825 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:03.825 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.724968+0000", 2024-09-18T04:40:03.825 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:03.826 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.826 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.826 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:03.826 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.726245+0000", 2024-09-18T04:40:03.826 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001276395 2024-09-18T04:40:03.826 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.826 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.826 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:03.826 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.726262+0000", 2024-09-18T04:40:03.826 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6973000000000002e-05 2024-09-18T04:40:03.826 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.826 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.826 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:03.827 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.726293+0000", 2024-09-18T04:40:03.827 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0886999999999998e-05 2024-09-18T04:40:03.827 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.827 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.827 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:03.827 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.726403+0000", 2024-09-18T04:40:03.827 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000110288 2024-09-18T04:40:03.827 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.827 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.827 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:03.827 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.726418+0000", 2024-09-18T04:40:03.827 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5503999999999999e-05 2024-09-18T04:40:03.827 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.828 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.828 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:03.828 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.728522+0000", 2024-09-18T04:40:03.828 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021032630000000002 2024-09-18T04:40:03.828 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:03.828 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:03.828 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:03.828 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.828 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.828 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.1 87/86 [PushOp(2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head, version: 81'220, data_included: [4182016~12288], data_size: 12288, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head@81'220, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:4182016, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:40:03.828 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.730366+0000", 2024-09-18T04:40:03.828 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.089852595, 2024-09-18T04:40:03.829 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0032522580000000001, 2024-09-18T04:40:03.829 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:03.829 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:03.829 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:03.829 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.829 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:03.829 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.730366+0000", 2024-09-18T04:40:03.829 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:03.829 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.829 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.829 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:03.829 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.730366+0000", 2024-09-18T04:40:03.829 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:03.830 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.830 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.830 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:03.830 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.730365+0000", 2024-09-18T04:40:03.830 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:03.830 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.830 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.830 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:03.830 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.730385+0000", 2024-09-18T04:40:03.830 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0766000000000001e-05 2024-09-18T04:40:03.830 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.830 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.830 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:03.831 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.730406+0000", 2024-09-18T04:40:03.831 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0706000000000001e-05 2024-09-18T04:40:03.831 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.831 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.831 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:03.831 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.730423+0000", 2024-09-18T04:40:03.831 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7411e-05 2024-09-18T04:40:03.831 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.831 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.831 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:03.831 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.730507+0000", 2024-09-18T04:40:03.831 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3388999999999997e-05 2024-09-18T04:40:03.831 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.832 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.832 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:03.832 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.730523+0000", 2024-09-18T04:40:03.832 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5843999999999999e-05 2024-09-18T04:40:03.832 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.832 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.832 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:03.832 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.733618+0000", 2024-09-18T04:40:03.832 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030951540000000001 2024-09-18T04:40:03.832 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:03.832 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:03.833 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:03.833 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.833 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.833 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:7, version: 79'62, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:7@79'62, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 8=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:40:03.833 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.732222+0000", 2024-09-18T04:40:03.833 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.087996635, 2024-09-18T04:40:03.833 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030084410000000002, 2024-09-18T04:40:03.833 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:03.833 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:03.833 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:03.833 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.833 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:03.833 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.732222+0000", 2024-09-18T04:40:03.834 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:03.834 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.834 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.834 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:03.834 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.732222+0000", 2024-09-18T04:40:03.834 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:03.834 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.834 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.834 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:03.834 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.732221+0000", 2024-09-18T04:40:03.834 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:40:03.835 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.835 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.835 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:03.835 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.733174+0000", 2024-09-18T04:40:03.835 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00095350299999999997 2024-09-18T04:40:03.835 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.835 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.835 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:03.835 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.733197+0000", 2024-09-18T04:40:03.835 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2925000000000002e-05 2024-09-18T04:40:03.835 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.835 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.835 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:03.836 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.733218+0000", 2024-09-18T04:40:03.836 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0477999999999999e-05 2024-09-18T04:40:03.836 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.836 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.836 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:03.836 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.733323+0000", 2024-09-18T04:40:03.836 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000105717 2024-09-18T04:40:03.836 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.836 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.836 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:03.836 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.733358+0000", 2024-09-18T04:40:03.836 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4406000000000002e-05 2024-09-18T04:40:03.837 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.837 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.837 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:03.837 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.735230+0000", 2024-09-18T04:40:03.837 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0018720659999999999 2024-09-18T04:40:03.837 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:03.837 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:03.837 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:03.837 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.837 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.837 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:7, version: 79'62, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:7@79'62, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 8=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:40:03.837 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.743433+0000", 2024-09-18T04:40:03.837 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.076785154, 2024-09-18T04:40:03.838 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003400237, 2024-09-18T04:40:03.838 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:03.838 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:03.838 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:03.838 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.838 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:03.838 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.743433+0000", 2024-09-18T04:40:03.838 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:03.838 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.838 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.838 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:03.838 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.743433+0000", 2024-09-18T04:40:03.838 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:03.839 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.839 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.839 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:03.839 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.743432+0000", 2024-09-18T04:40:03.839 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:03.839 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.839 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.839 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:03.839 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.744392+0000", 2024-09-18T04:40:03.839 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00095993499999999998 2024-09-18T04:40:03.839 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.839 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.840 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:03.840 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.744417+0000", 2024-09-18T04:40:03.840 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5249000000000002e-05 2024-09-18T04:40:03.840 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.840 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.840 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:03.840 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.744434+0000", 2024-09-18T04:40:03.840 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6597999999999999e-05 2024-09-18T04:40:03.840 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.840 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.840 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:03.840 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.744499+0000", 2024-09-18T04:40:03.840 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4987000000000006e-05 2024-09-18T04:40:03.841 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.841 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.841 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:03.841 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.744506+0000", 2024-09-18T04:40:03.841 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2679999999999997e-06 2024-09-18T04:40:03.841 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.841 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.841 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:03.841 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.746833+0000", 2024-09-18T04:40:03.841 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023270529999999999 2024-09-18T04:40:03.841 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:03.841 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:03.842 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:03.842 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.842 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.842 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.5 87/86 [PushOp(2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:head, version: 79'60, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:head@79'60, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-18T04:40:03.842 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.747807+0000", 2024-09-18T04:40:03.842 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.072410680000001, 2024-09-18T04:40:03.842 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00244378, 2024-09-18T04:40:03.842 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:03.842 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:03.842 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:03.842 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.842 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:03.842 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.747807+0000", 2024-09-18T04:40:03.843 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:03.843 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.843 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.843 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:03.843 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.747807+0000", 2024-09-18T04:40:03.843 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:03.843 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.843 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.843 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:03.843 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.747806+0000", 2024-09-18T04:40:03.843 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:03.843 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.843 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.844 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:03.844 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.747825+0000", 2024-09-18T04:40:03.844 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.925e-05 2024-09-18T04:40:03.844 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.844 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.844 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:03.844 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.747853+0000", 2024-09-18T04:40:03.844 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7174000000000001e-05 2024-09-18T04:40:03.844 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.844 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.844 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:03.844 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.747887+0000", 2024-09-18T04:40:03.845 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4731999999999999e-05 2024-09-18T04:40:03.845 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.845 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.845 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:03.845 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.747983+0000", 2024-09-18T04:40:03.845 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5964999999999993e-05 2024-09-18T04:40:03.845 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.845 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.845 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:03.845 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.747994+0000", 2024-09-18T04:40:03.845 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0526999999999999e-05 2024-09-18T04:40:03.845 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.845 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.846 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:03.846 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.750251+0000", 2024-09-18T04:40:03.846 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0022573519999999998 2024-09-18T04:40:03.846 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:03.846 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:03.846 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:03.846 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.846 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.846 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:7, version: 79'64, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:7@79'64, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-18T04:40:03.846 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.750570+0000", 2024-09-18T04:40:03.846 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.069648276999999, 2024-09-18T04:40:03.846 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0029248949999999998, 2024-09-18T04:40:03.846 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:03.847 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:03.847 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:03.847 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.847 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:03.847 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.750570+0000", 2024-09-18T04:40:03.847 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:03.847 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.847 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.847 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:03.847 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.750570+0000", 2024-09-18T04:40:03.847 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:03.847 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.847 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.847 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:03.848 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.750569+0000", 2024-09-18T04:40:03.848 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:03.848 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.848 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.848 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:03.848 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.751724+0000", 2024-09-18T04:40:03.848 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001155037 2024-09-18T04:40:03.848 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.848 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.848 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:03.848 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.751747+0000", 2024-09-18T04:40:03.848 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2679000000000001e-05 2024-09-18T04:40:03.849 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.849 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.849 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:03.849 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.751761+0000", 2024-09-18T04:40:03.849 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.467e-05 2024-09-18T04:40:03.849 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.849 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.849 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:03.849 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.751839+0000", 2024-09-18T04:40:03.849 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7824000000000006e-05 2024-09-18T04:40:03.849 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.849 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.849 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:03.849 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.751849+0000", 2024-09-18T04:40:03.850 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4159999999999993e-06 2024-09-18T04:40:03.850 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.850 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.850 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:03.850 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.753495+0000", 2024-09-18T04:40:03.850 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0016461830000000001 2024-09-18T04:40:03.850 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:03.850 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:03.850 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:03.850 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.851 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.851 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.5 87/86 [PushOp(2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:7, version: 79'59, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:7@79'59, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-18T04:40:03.851 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.752479+0000", 2024-09-18T04:40:03.851 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.067738909999999, 2024-09-18T04:40:03.851 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027271040000000002, 2024-09-18T04:40:03.851 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:03.851 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:03.851 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:03.851 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.851 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:03.851 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.752479+0000", 2024-09-18T04:40:03.851 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:03.851 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.852 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.852 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:03.852 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.752479+0000", 2024-09-18T04:40:03.852 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:03.852 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.852 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.852 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:03.852 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.752478+0000", 2024-09-18T04:40:03.852 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:03.852 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.852 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.852 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:03.853 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.753784+0000", 2024-09-18T04:40:03.853 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0013058869999999999 2024-09-18T04:40:03.853 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.853 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.853 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:03.853 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.753821+0000", 2024-09-18T04:40:03.853 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7268999999999998e-05 2024-09-18T04:40:03.853 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.853 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.853 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:03.853 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.753873+0000", 2024-09-18T04:40:03.853 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1564000000000003e-05 2024-09-18T04:40:03.854 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.854 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.854 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:03.854 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.753963+0000", 2024-09-18T04:40:03.854 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0636000000000004e-05 2024-09-18T04:40:03.854 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.854 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.854 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:03.854 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.753972+0000", 2024-09-18T04:40:03.854 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0529999999999996e-06 2024-09-18T04:40:03.854 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.854 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.854 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:03.854 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.755206+0000", 2024-09-18T04:40:03.855 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001234113 2024-09-18T04:40:03.855 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:03.855 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:03.855 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:03.855 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.855 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.855 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:7, version: 79'64, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:7@79'64, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:40:03.855 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.756672+0000", 2024-09-18T04:40:03.855 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.063545832999999, 2024-09-18T04:40:03.855 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028688770000000001, 2024-09-18T04:40:03.855 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:03.855 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:03.855 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:03.856 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.856 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:03.856 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.756672+0000", 2024-09-18T04:40:03.856 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:03.856 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.856 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.856 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:03.856 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.756672+0000", 2024-09-18T04:40:03.856 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:03.856 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.856 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.856 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:03.856 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.756671+0000", 2024-09-18T04:40:03.857 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:03.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.857 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.857 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:03.857 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.757623+0000", 2024-09-18T04:40:03.857 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00095182199999999995 2024-09-18T04:40:03.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.857 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.857 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:03.857 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.757644+0000", 2024-09-18T04:40:03.857 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0996999999999999e-05 2024-09-18T04:40:03.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.857 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.857 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:03.858 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.757657+0000", 2024-09-18T04:40:03.858 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3264999999999999e-05 2024-09-18T04:40:03.858 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.858 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.858 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:03.858 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.757779+0000", 2024-09-18T04:40:03.858 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000121655 2024-09-18T04:40:03.858 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.858 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.858 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:03.858 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.757801+0000", 2024-09-18T04:40:03.858 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2340999999999998e-05 2024-09-18T04:40:03.858 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.858 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.859 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:03.859 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.759541+0000", 2024-09-18T04:40:03.859 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017398120000000001 2024-09-18T04:40:03.859 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:03.859 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:03.859 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:03.859 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.859 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.859 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.5 87/86 [PushOp(2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:7, version: 79'59, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:7@79'59, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:40:03.859 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.758140+0000", 2024-09-18T04:40:03.859 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.062077928000001, 2024-09-18T04:40:03.859 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023362539999999998, 2024-09-18T04:40:03.860 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:03.860 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:03.860 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:03.860 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.860 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:03.860 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.758140+0000", 2024-09-18T04:40:03.860 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:03.860 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.860 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.860 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:03.861 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.758140+0000", 2024-09-18T04:40:03.861 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:03.861 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.861 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.861 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:03.861 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.758139+0000", 2024-09-18T04:40:03.861 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:03.861 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.861 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.861 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:03.861 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.758850+0000", 2024-09-18T04:40:03.861 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00071123099999999999 2024-09-18T04:40:03.862 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.862 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.862 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:03.862 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.758872+0000", 2024-09-18T04:40:03.862 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2195999999999999e-05 2024-09-18T04:40:03.862 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.862 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.862 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:03.862 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.758885+0000", 2024-09-18T04:40:03.862 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.327e-05 2024-09-18T04:40:03.862 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.862 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.862 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:03.862 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.758969+0000", 2024-09-18T04:40:03.863 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3497000000000004e-05 2024-09-18T04:40:03.863 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.863 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.863 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:03.863 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.758988+0000", 2024-09-18T04:40:03.863 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8765000000000001e-05 2024-09-18T04:40:03.863 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.863 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.863 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:03.864 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.760476+0000", 2024-09-18T04:40:03.864 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001488828 2024-09-18T04:40:03.864 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:03.864 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:03.864 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:03.864 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.864 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.864 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.5 87/86 [PushOp(2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:7, version: 79'59, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:7@79'59, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:40:03.864 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.762725+0000", 2024-09-18T04:40:03.864 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.057492827000001, 2024-09-18T04:40:03.864 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028321890000000001, 2024-09-18T04:40:03.864 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:03.865 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:03.865 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:03.865 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.865 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:03.865 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.762725+0000", 2024-09-18T04:40:03.865 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:03.865 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.865 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.865 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:03.865 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.762725+0000", 2024-09-18T04:40:03.865 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:03.865 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.865 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.866 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:03.866 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.762724+0000", 2024-09-18T04:40:03.866 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:03.866 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.866 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.866 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:03.866 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.763395+0000", 2024-09-18T04:40:03.866 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00067064800000000003 2024-09-18T04:40:03.866 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.866 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.866 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:03.866 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.763416+0000", 2024-09-18T04:40:03.867 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0984000000000001e-05 2024-09-18T04:40:03.867 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.867 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.867 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:03.867 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.763442+0000", 2024-09-18T04:40:03.867 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6635000000000002e-05 2024-09-18T04:40:03.867 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.867 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.867 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:03.867 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.763584+0000", 2024-09-18T04:40:03.867 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000142037 2024-09-18T04:40:03.868 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.868 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.868 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:03.868 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.763610+0000", 2024-09-18T04:40:03.868 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6027999999999999e-05 2024-09-18T04:40:03.868 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.868 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.868 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:03.868 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.765557+0000", 2024-09-18T04:40:03.868 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0019470939999999999 2024-09-18T04:40:03.868 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:03.869 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:03.869 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:03.869 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.869 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.869 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:7, version: 79'64, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:7@79'64, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:40:03.869 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.763323+0000", 2024-09-18T04:40:03.869 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.056894642, 2024-09-18T04:40:03.869 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003042215, 2024-09-18T04:40:03.869 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:03.869 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:03.869 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:03.870 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.870 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:03.870 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.763323+0000", 2024-09-18T04:40:03.870 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:03.870 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.870 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.870 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:03.870 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.763323+0000", 2024-09-18T04:40:03.870 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:03.870 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.870 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.870 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:03.870 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.763322+0000", 2024-09-18T04:40:03.871 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:03.871 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.871 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.871 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:03.871 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.764567+0000", 2024-09-18T04:40:03.871 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001244312 2024-09-18T04:40:03.871 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.871 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.871 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:03.871 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.764600+0000", 2024-09-18T04:40:03.871 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2787000000000003e-05 2024-09-18T04:40:03.871 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.871 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.872 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:03.872 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.764640+0000", 2024-09-18T04:40:03.872 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0825000000000003e-05 2024-09-18T04:40:03.872 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.872 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.872 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:03.872 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.764789+0000", 2024-09-18T04:40:03.872 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000148598 2024-09-18T04:40:03.872 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.872 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.872 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:03.872 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.764812+0000", 2024-09-18T04:40:03.872 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2702e-05 2024-09-18T04:40:03.873 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.873 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.873 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:03.873 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.766366+0000", 2024-09-18T04:40:03.873 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0015540110000000001 2024-09-18T04:40:03.873 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:03.873 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:03.873 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:03.873 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.873 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.873 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.5 87/86 [PushOp(2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:7, version: 79'59, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:7@79'59, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:40:03.873 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.768078+0000", 2024-09-18T04:40:03.873 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.052139864000001, 2024-09-18T04:40:03.874 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028938509999999998, 2024-09-18T04:40:03.874 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:03.874 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:03.874 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:03.874 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.874 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:03.874 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.768078+0000", 2024-09-18T04:40:03.874 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:03.874 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.874 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.874 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:03.874 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.768078+0000", 2024-09-18T04:40:03.874 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:03.874 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.875 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.875 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:03.875 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.768076+0000", 2024-09-18T04:40:03.875 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:03.875 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.875 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.875 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:03.875 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.768951+0000", 2024-09-18T04:40:03.875 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000874455 2024-09-18T04:40:03.875 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.875 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.875 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:03.875 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.768972+0000", 2024-09-18T04:40:03.876 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.154e-05 2024-09-18T04:40:03.876 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.876 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.876 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:03.876 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.768992+0000", 2024-09-18T04:40:03.876 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9822000000000001e-05 2024-09-18T04:40:03.876 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.876 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.876 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:03.876 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.769110+0000", 2024-09-18T04:40:03.876 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000117762 2024-09-18T04:40:03.876 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.877 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.877 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:03.877 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.769123+0000", 2024-09-18T04:40:03.877 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2473999999999999e-05 2024-09-18T04:40:03.877 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.877 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.877 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:03.877 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.770972+0000", 2024-09-18T04:40:03.877 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001849613 2024-09-18T04:40:03.877 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:03.877 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:03.877 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:03.877 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.878 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.878 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:7, version: 79'64, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:7@79'64, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:40:03.878 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.769830+0000", 2024-09-18T04:40:03.878 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.050388600000002, 2024-09-18T04:40:03.878 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0040223419999999999, 2024-09-18T04:40:03.878 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:03.878 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:03.878 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:03.878 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.878 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:03.878 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.769830+0000", 2024-09-18T04:40:03.878 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:03.878 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.879 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.879 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:03.879 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.769830+0000", 2024-09-18T04:40:03.879 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:03.879 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.879 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.879 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:03.879 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.769829+0000", 2024-09-18T04:40:03.879 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:03.879 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.879 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.879 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:03.879 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.770812+0000", 2024-09-18T04:40:03.879 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00098347800000000009 2024-09-18T04:40:03.880 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.880 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.880 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:03.880 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.770834+0000", 2024-09-18T04:40:03.880 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1648999999999999e-05 2024-09-18T04:40:03.880 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.880 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.880 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:03.880 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.770847+0000", 2024-09-18T04:40:03.880 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3169999999999999e-05 2024-09-18T04:40:03.880 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.880 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.881 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:03.881 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.770969+0000", 2024-09-18T04:40:03.881 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000121721 2024-09-18T04:40:03.881 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.881 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.881 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:03.881 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.770989+0000", 2024-09-18T04:40:03.881 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9944000000000001e-05 2024-09-18T04:40:03.881 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.881 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.881 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:03.881 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.773852+0000", 2024-09-18T04:40:03.881 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028632319999999998 2024-09-18T04:40:03.881 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:03.882 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:03.882 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:03.882 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.882 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.882 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.5 87/86 [PushOp(2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head, version: 79'62, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head@79'62, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-18T04:40:03.882 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.772378+0000", 2024-09-18T04:40:03.882 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.047840477000001, 2024-09-18T04:40:03.882 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002661535, 2024-09-18T04:40:03.882 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:03.882 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:03.882 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:03.882 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.882 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:03.883 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.772378+0000", 2024-09-18T04:40:03.883 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:03.883 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.883 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.883 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:03.883 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.772378+0000", 2024-09-18T04:40:03.883 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:03.883 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.883 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.883 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:03.883 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.772377+0000", 2024-09-18T04:40:03.883 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:03.883 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.884 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.884 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:03.884 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.772393+0000", 2024-09-18T04:40:03.884 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6495e-05 2024-09-18T04:40:03.884 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.884 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.884 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:03.884 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.772417+0000", 2024-09-18T04:40:03.884 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4304e-05 2024-09-18T04:40:03.884 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.884 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.884 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:03.885 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.772431+0000", 2024-09-18T04:40:03.885 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3165e-05 2024-09-18T04:40:03.885 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.885 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.885 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:03.885 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.772499+0000", 2024-09-18T04:40:03.885 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8627000000000005e-05 2024-09-18T04:40:03.885 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.885 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.885 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:03.885 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.772508+0000", 2024-09-18T04:40:03.885 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5609999999999999e-06 2024-09-18T04:40:03.885 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.885 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.886 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:03.886 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.775039+0000", 2024-09-18T04:40:03.886 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002531387 2024-09-18T04:40:03.886 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:03.886 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:03.886 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:03.886 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.886 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.886 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head, version: 79'65, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head@79'65, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-18T04:40:03.886 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.775934+0000", 2024-09-18T04:40:03.886 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.044283724, 2024-09-18T04:40:03.886 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023666939999999999, 2024-09-18T04:40:03.887 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:03.887 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:03.887 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:03.887 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.887 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:03.887 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.775934+0000", 2024-09-18T04:40:03.887 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:03.887 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.887 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.887 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:03.887 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.775934+0000", 2024-09-18T04:40:03.887 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:03.887 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.888 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.888 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:03.888 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.775933+0000", 2024-09-18T04:40:03.888 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:03.888 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.888 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.888 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:03.888 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.775955+0000", 2024-09-18T04:40:03.888 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1512e-05 2024-09-18T04:40:03.888 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.888 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.888 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:03.889 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.775975+0000", 2024-09-18T04:40:03.889 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9653999999999999e-05 2024-09-18T04:40:03.889 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.889 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.889 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:03.889 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.775992+0000", 2024-09-18T04:40:03.889 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7326e-05 2024-09-18T04:40:03.889 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.889 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.889 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:03.889 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.776078+0000", 2024-09-18T04:40:03.889 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.6599000000000002e-05 2024-09-18T04:40:03.889 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.890 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.890 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:03.890 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.776088+0000", 2024-09-18T04:40:03.890 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.8120000000000002e-06 2024-09-18T04:40:03.890 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.890 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.890 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:03.890 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.778301+0000", 2024-09-18T04:40:03.890 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002212805 2024-09-18T04:40:03.890 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:03.890 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:03.890 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:03.890 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.890 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.891 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.5 87/86 [PushOp(2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:7, version: 79'61, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:7@79'61, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:40:03.891 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.781808+0000", 2024-09-18T04:40:03.891 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.038410478999999, 2024-09-18T04:40:03.891 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027222169999999999, 2024-09-18T04:40:03.891 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:03.891 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:03.891 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:03.891 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.891 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:03.891 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.781808+0000", 2024-09-18T04:40:03.891 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:03.891 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.891 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.891 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:03.892 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.781808+0000", 2024-09-18T04:40:03.892 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:03.892 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.892 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.892 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:03.892 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.781807+0000", 2024-09-18T04:40:03.892 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:03.892 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.892 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.892 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:03.892 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.782242+0000", 2024-09-18T04:40:03.892 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00043528300000000001 2024-09-18T04:40:03.892 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.893 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.893 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:03.893 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.782265+0000", 2024-09-18T04:40:03.893 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3227e-05 2024-09-18T04:40:03.893 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.893 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.893 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:03.893 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.782283+0000", 2024-09-18T04:40:03.893 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7535999999999999e-05 2024-09-18T04:40:03.893 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.893 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.894 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:03.894 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.782418+0000", 2024-09-18T04:40:03.894 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000135298 2024-09-18T04:40:03.894 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.894 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.894 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:03.894 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.782444+0000", 2024-09-18T04:40:03.894 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6020000000000002e-05 2024-09-18T04:40:03.894 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.894 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.894 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:03.894 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.784530+0000", 2024-09-18T04:40:03.894 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020858890000000001 2024-09-18T04:40:03.895 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:03.895 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:03.895 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:03.895 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.895 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.895 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head, version: 81'75, data_included: [0~49152,1097728~999424], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head@81'75, size: 2146304, copy_subset: [0~2146304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-18T04:40:03.895 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.782052+0000", 2024-09-18T04:40:03.895 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.038166114999999, 2024-09-18T04:40:03.895 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0037389630000000001, 2024-09-18T04:40:03.895 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:03.895 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:03.895 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:03.895 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.895 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:03.896 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.782052+0000", 2024-09-18T04:40:03.896 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:03.896 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.896 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.896 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:03.896 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.782052+0000", 2024-09-18T04:40:03.896 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:03.896 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.896 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.896 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:03.896 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.782051+0000", 2024-09-18T04:40:03.896 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:03.896 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.896 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.897 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:03.897 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.783658+0000", 2024-09-18T04:40:03.897 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0016072149999999999 2024-09-18T04:40:03.897 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.897 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.897 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:03.897 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.783698+0000", 2024-09-18T04:40:03.897 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9394000000000003e-05 2024-09-18T04:40:03.897 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.897 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.897 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:03.897 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.783733+0000", 2024-09-18T04:40:03.897 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5428e-05 2024-09-18T04:40:03.897 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.897 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.898 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:03.898 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.783854+0000", 2024-09-18T04:40:03.898 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012067599999999999 2024-09-18T04:40:03.898 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.898 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.898 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:03.898 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.783869+0000", 2024-09-18T04:40:03.898 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5554999999999999e-05 2024-09-18T04:40:03.898 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.898 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.898 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:03.898 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.785791+0000", 2024-09-18T04:40:03.898 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001921629 2024-09-18T04:40:03.899 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:03.899 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:03.899 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:03.899 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.899 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.899 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head, version: 81'75, data_included: [2097152~49152], data_size: 49152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head@81'75, size: 2146304, copy_subset: [0~2146304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2146304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:40:03.899 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.787740+0000", 2024-09-18T04:40:03.899 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.032478601000001, 2024-09-18T04:40:03.899 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030127370000000001, 2024-09-18T04:40:03.899 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:03.899 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:03.899 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:03.899 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.899 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:03.899 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.787740+0000", 2024-09-18T04:40:03.900 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:03.900 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.900 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.900 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:03.900 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.787740+0000", 2024-09-18T04:40:03.900 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:03.900 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.900 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.900 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:03.900 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.787738+0000", 2024-09-18T04:40:03.900 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:03.900 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.900 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.900 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:03.901 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.787795+0000", 2024-09-18T04:40:03.901 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6753999999999999e-05 2024-09-18T04:40:03.901 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.901 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.901 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:03.901 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.787826+0000", 2024-09-18T04:40:03.901 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0617000000000002e-05 2024-09-18T04:40:03.901 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.901 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.901 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:03.901 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.787853+0000", 2024-09-18T04:40:03.901 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7538999999999998e-05 2024-09-18T04:40:03.901 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.902 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.902 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:03.902 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.787963+0000", 2024-09-18T04:40:03.902 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000109387 2024-09-18T04:40:03.902 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.902 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.902 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:03.902 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.787973+0000", 2024-09-18T04:40:03.902 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0781999999999999e-05 2024-09-18T04:40:03.902 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.902 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.902 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:03.902 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.790752+0000", 2024-09-18T04:40:03.902 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027788520000000001 2024-09-18T04:40:03.903 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:03.903 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:03.903 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:03.903 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.903 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.903 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.5 87/86 [PushOp(2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:7, version: 79'61, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:7@79'61, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:40:03.903 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.791668+0000", 2024-09-18T04:40:03.903 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.028549907999999, 2024-09-18T04:40:03.903 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025634299999999998, 2024-09-18T04:40:03.903 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:03.903 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:03.903 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:03.903 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.903 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:03.904 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.791668+0000", 2024-09-18T04:40:03.904 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:03.904 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.904 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.904 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:03.904 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.791668+0000", 2024-09-18T04:40:03.904 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:03.904 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.904 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.904 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:03.904 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.791667+0000", 2024-09-18T04:40:03.904 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:03.904 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.904 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.905 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:03.905 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.792071+0000", 2024-09-18T04:40:03.905 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00040348599999999999 2024-09-18T04:40:03.905 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.905 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.905 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:03.905 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.792094+0000", 2024-09-18T04:40:03.905 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3105e-05 2024-09-18T04:40:03.905 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.905 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.905 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:03.905 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.792112+0000", 2024-09-18T04:40:03.905 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8502999999999999e-05 2024-09-18T04:40:03.906 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.906 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.906 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:03.906 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.792187+0000", 2024-09-18T04:40:03.906 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5074000000000007e-05 2024-09-18T04:40:03.906 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.906 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.906 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:03.906 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.792197+0000", 2024-09-18T04:40:03.906 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.2979999999999997e-06 2024-09-18T04:40:03.906 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.906 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.906 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:03.906 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.794232+0000", 2024-09-18T04:40:03.906 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002035149 2024-09-18T04:40:03.907 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:03.907 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:03.907 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:03.907 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.907 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.907 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.5 87/86 [PushOp(2:b227e6b1:::rbd_data.125fc09df6d1.000000000000000e:7, version: 79'63, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:b227e6b1:::rbd_data.125fc09df6d1.000000000000000e:7@79'63, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:40:03.907 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.813128+0000", 2024-09-18T04:40:03.907 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 17.007090046999998, 2024-09-18T04:40:03.907 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025359520000000002, 2024-09-18T04:40:03.907 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:03.907 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:03.907 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:03.907 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.907 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:03.908 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.813128+0000", 2024-09-18T04:40:03.908 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:03.908 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.908 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.908 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:03.908 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.813128+0000", 2024-09-18T04:40:03.908 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:03.908 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.908 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.908 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:03.908 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.813127+0000", 2024-09-18T04:40:03.908 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:03.908 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.909 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.909 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:03.909 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.813544+0000", 2024-09-18T04:40:03.909 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000417248 2024-09-18T04:40:03.909 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.909 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.909 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:03.909 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.813571+0000", 2024-09-18T04:40:03.909 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6789000000000001e-05 2024-09-18T04:40:03.909 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.909 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.909 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:03.909 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.813589+0000", 2024-09-18T04:40:03.910 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8247e-05 2024-09-18T04:40:03.910 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.910 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.910 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:03.910 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.813698+0000", 2024-09-18T04:40:03.910 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000108937 2024-09-18T04:40:03.910 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.910 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.910 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:03.910 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.813715+0000", 2024-09-18T04:40:03.910 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7110000000000001e-05 2024-09-18T04:40:03.910 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:03.910 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:03.910 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:03.911 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.815664+0000", 2024-09-18T04:40:03.911 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001948946 2024-09-18T04:40:03.911 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:03.911 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:03.911 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:03.911 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:03.911 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:03.911 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:40:03.911 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-09-18T04:40:03.912 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:40:03.912 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:40:03.912 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:40:03.912 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:40:03.912 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:40:03.912 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:40:03.912 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:40:03.914 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:40:03.955 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:40:03.951+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:40:04.017 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:40:04.011+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:40:04.064 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:40:04.064 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:40:04.064 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:40:04.064 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:40:04.074 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:40:04.101 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-09-18T04:40:04.095+0000 7f8dc00db700 -1 mon.a@0(leader).osd e95 definitely_dead 0 2024-09-18T04:40:04.118 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:40:04.112+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:40:04.179 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:40:04.179 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:40:04.179 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:40:04.179 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:40:04.179 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:40:04.189 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:40:04.231 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:40:04.227+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:40:04.319 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:40:04.316+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:40:04.344 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:40:04.344 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:40:04.344 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:40:04.344 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:40:04.344 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.344 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:37 2.5 e51/28 2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:head v 51'1, mlcod=0'0)", 2024-09-18T04:40:04.345 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:40:04.345 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 82.046947901999999, 2024-09-18T04:40:04.345 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012279973, 2024-09-18T04:40:04.345 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:04.345 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:04.345 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:04.345 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.345 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:04.345 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:40:04.345 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.345 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.345 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.345 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:04.346 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:40:04.346 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.346 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.346 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.346 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:04.346 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292468+0000", 2024-09-18T04:40:04.346 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:40:04.346 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.346 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.346 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:04.346 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295837+0000", 2024-09-18T04:40:04.346 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0033691699999999999 2024-09-18T04:40:04.346 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.346 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.347 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:04.347 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295860+0000", 2024-09-18T04:40:04.347 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2756999999999999e-05 2024-09-18T04:40:04.347 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.347 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.347 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:04.347 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295872+0000", 2024-09-18T04:40:04.347 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2442e-05 2024-09-18T04:40:04.347 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.347 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.347 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:04.347 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295934+0000", 2024-09-18T04:40:04.347 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1187999999999998e-05 2024-09-18T04:40:04.347 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.348 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.348 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:04.348 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295967+0000", 2024-09-18T04:40:04.348 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3185999999999997e-05 2024-09-18T04:40:04.348 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.348 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.348 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:04.348 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304697+0000", 2024-09-18T04:40:04.348 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0087298059999999997 2024-09-18T04:40:04.348 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.348 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.348 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:04.348 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304749+0000", 2024-09-18T04:40:04.348 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1922000000000002e-05 2024-09-18T04:40:04.349 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.349 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:04.349 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.349 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.349 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.349 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:49 2.5 e51/28 2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head v 51'2, mlcod=0'0)", 2024-09-18T04:40:04.349 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:40:04.349 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 82.013621064000006, 2024-09-18T04:40:04.349 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018033041999999999, 2024-09-18T04:40:04.349 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:04.349 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:04.349 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:04.349 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.350 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:04.350 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:40:04.350 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.350 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.350 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.350 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:04.350 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:40:04.350 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.350 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.350 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.350 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:04.350 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325793+0000", 2024-09-18T04:40:04.350 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:40:04.350 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.351 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.351 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:04.351 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329038+0000", 2024-09-18T04:40:04.351 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0032444010000000001 2024-09-18T04:40:04.351 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.351 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.351 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:04.351 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329065+0000", 2024-09-18T04:40:04.351 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7500000000000001e-05 2024-09-18T04:40:04.351 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.351 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.351 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:04.351 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329077+0000", 2024-09-18T04:40:04.351 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1919999999999999e-05 2024-09-18T04:40:04.351 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.352 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.352 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:04.352 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329149+0000", 2024-09-18T04:40:04.352 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2349999999999997e-05 2024-09-18T04:40:04.352 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.352 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.352 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:04.352 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329192+0000", 2024-09-18T04:40:04.352 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2744000000000003e-05 2024-09-18T04:40:04.352 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.352 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.352 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:04.352 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343773+0000", 2024-09-18T04:40:04.353 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014581246000000001 2024-09-18T04:40:04.353 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.353 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.353 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:04.353 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343829+0000", 2024-09-18T04:40:04.353 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5143999999999999e-05 2024-09-18T04:40:04.353 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.353 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:04.353 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.353 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.353 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.353 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:47 2.4 e51/17 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 51'3, mlcod=51'1)", 2024-09-18T04:40:04.353 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:40:04.353 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 82.013040798999995, 2024-09-18T04:40:04.353 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018960186, 2024-09-18T04:40:04.354 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:04.354 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:04.354 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:04.354 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.354 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:04.354 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:40:04.354 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.354 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.354 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.354 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:04.354 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:40:04.354 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.354 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.354 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.355 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:04.355 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326374+0000", 2024-09-18T04:40:04.355 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:04.355 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.355 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.355 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:04.355 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330883+0000", 2024-09-18T04:40:04.355 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0045084740000000002 2024-09-18T04:40:04.355 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.355 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.355 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:04.355 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330901+0000", 2024-09-18T04:40:04.355 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7890999999999999e-05 2024-09-18T04:40:04.355 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.356 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.356 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:04.356 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330912+0000", 2024-09-18T04:40:04.356 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1041e-05 2024-09-18T04:40:04.356 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.356 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.356 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:04.356 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.331003+0000", 2024-09-18T04:40:04.356 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1323999999999995e-05 2024-09-18T04:40:04.356 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.356 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.356 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:04.356 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.331030+0000", 2024-09-18T04:40:04.357 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7351999999999998e-05 2024-09-18T04:40:04.357 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.357 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.357 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:04.357 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345287+0000", 2024-09-18T04:40:04.357 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014256332 2024-09-18T04:40:04.357 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.357 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.357 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:04.357 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345336+0000", 2024-09-18T04:40:04.357 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9356e-05 2024-09-18T04:40:04.357 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.357 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:04.358 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.358 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.358 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.358 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:58 2.5 e51/28 2:b227e6b1:::rbd_data.125fc09df6d1.000000000000000e:head v 51'3, mlcod=51'1)", 2024-09-18T04:40:04.358 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:40:04.358 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 81.972614171999993, 2024-09-18T04:40:04.358 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016513678, 2024-09-18T04:40:04.358 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:04.358 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:04.358 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:04.358 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.358 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:04.358 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:40:04.359 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.359 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.359 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.359 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:04.359 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:40:04.359 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.359 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.359 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.359 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:04.359 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366800+0000", 2024-09-18T04:40:04.359 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:40:04.359 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.359 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.360 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:04.360 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368644+0000", 2024-09-18T04:40:04.360 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001844399 2024-09-18T04:40:04.360 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.360 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.360 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:04.360 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368681+0000", 2024-09-18T04:40:04.360 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6526999999999998e-05 2024-09-18T04:40:04.360 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.360 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.360 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:04.360 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368698+0000", 2024-09-18T04:40:04.361 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7272999999999999e-05 2024-09-18T04:40:04.361 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.361 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.361 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:04.361 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368773+0000", 2024-09-18T04:40:04.361 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4862000000000003e-05 2024-09-18T04:40:04.361 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.361 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.361 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:04.361 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368801+0000", 2024-09-18T04:40:04.361 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7925000000000001e-05 2024-09-18T04:40:04.362 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.362 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.362 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:04.362 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383261+0000", 2024-09-18T04:40:04.362 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014460377999999999 2024-09-18T04:40:04.362 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.362 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.362 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:04.362 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383316+0000", 2024-09-18T04:40:04.362 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4623999999999997e-05 2024-09-18T04:40:04.363 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.363 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:04.363 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.363 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.363 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.363 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:56 2.4 e51/17 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 51'4, mlcod=51'2)", 2024-09-18T04:40:04.363 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:40:04.363 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 81.970610145999999, 2024-09-18T04:40:04.363 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015105221, 2024-09-18T04:40:04.363 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:04.364 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:04.364 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:04.364 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.364 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:04.364 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:40:04.364 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.364 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.364 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.364 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:04.364 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:40:04.364 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.364 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.364 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.364 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:04.365 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368805+0000", 2024-09-18T04:40:04.365 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:04.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.365 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.365 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:04.365 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371603+0000", 2024-09-18T04:40:04.365 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027982660000000002 2024-09-18T04:40:04.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.365 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.365 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:04.365 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371633+0000", 2024-09-18T04:40:04.365 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0073000000000002e-05 2024-09-18T04:40:04.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.366 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.366 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:04.366 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371652+0000", 2024-09-18T04:40:04.366 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8597000000000002e-05 2024-09-18T04:40:04.366 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.366 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.366 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:04.366 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371752+0000", 2024-09-18T04:40:04.366 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9772999999999995e-05 2024-09-18T04:40:04.366 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.366 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.366 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:04.366 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371793+0000", 2024-09-18T04:40:04.366 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1173e-05 2024-09-18T04:40:04.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.367 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.367 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:04.367 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383875+0000", 2024-09-18T04:40:04.367 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012081852000000001 2024-09-18T04:40:04.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.367 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.367 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:04.367 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383912+0000", 2024-09-18T04:40:04.367 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7045000000000002e-05 2024-09-18T04:40:04.367 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.367 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:04.367 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.368 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.368 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.368 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:36 2.5 e51/28 2:bd5b8dd8:ns::rbd_data.133e813ab5bc.0000000000000002:head v 51'4, mlcod=51'2)", 2024-09-18T04:40:04.368 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:40:04.368 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 81.801891155000007, 2024-09-18T04:40:04.368 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020758161000000001, 2024-09-18T04:40:04.368 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:04.368 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:04.368 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:04.368 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.368 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:04.368 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:40:04.368 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.369 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.369 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.369 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:04.369 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:40:04.369 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.369 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.369 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.369 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:04.369 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537523+0000", 2024-09-18T04:40:04.369 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:04.369 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.369 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.369 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:04.369 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541578+0000", 2024-09-18T04:40:04.370 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0040549710000000001 2024-09-18T04:40:04.370 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.370 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.370 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:04.370 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541608+0000", 2024-09-18T04:40:04.370 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9349e-05 2024-09-18T04:40:04.370 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.370 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.370 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:04.370 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541621+0000", 2024-09-18T04:40:04.370 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2736e-05 2024-09-18T04:40:04.370 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.370 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.370 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:04.371 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541715+0000", 2024-09-18T04:40:04.371 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4197999999999995e-05 2024-09-18T04:40:04.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.371 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.371 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:04.371 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541755+0000", 2024-09-18T04:40:04.371 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9818999999999999e-05 2024-09-18T04:40:04.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.371 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.371 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:04.371 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558217+0000", 2024-09-18T04:40:04.371 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016462234999999999 2024-09-18T04:40:04.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.371 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.372 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:04.372 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558284+0000", 2024-09-18T04:40:04.372 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6774999999999997e-05 2024-09-18T04:40:04.372 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.372 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:04.372 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.372 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.372 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.372 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:39 2.4 e51/17 2:304364d1:ns::rbd_data.133e813ab5bc.0000000000000004:head v 51'5, mlcod=51'3)", 2024-09-18T04:40:04.372 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:40:04.372 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 81.793752028, 2024-09-18T04:40:04.372 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016777558000000001, 2024-09-18T04:40:04.372 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:04.372 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:04.373 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:04.373 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.373 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:04.373 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:40:04.373 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.373 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.373 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.373 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:04.373 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:40:04.373 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.373 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.373 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.373 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:04.373 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545663+0000", 2024-09-18T04:40:04.374 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:04.374 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.374 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.374 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:04.374 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548286+0000", 2024-09-18T04:40:04.374 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026233390000000001 2024-09-18T04:40:04.374 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.374 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.374 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:04.374 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548305+0000", 2024-09-18T04:40:04.374 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8425000000000001e-05 2024-09-18T04:40:04.374 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.374 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.374 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:04.375 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548316+0000", 2024-09-18T04:40:04.375 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1163e-05 2024-09-18T04:40:04.375 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.375 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.375 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:04.375 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548379+0000", 2024-09-18T04:40:04.375 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2780000000000005e-05 2024-09-18T04:40:04.375 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.375 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.375 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:04.375 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548399+0000", 2024-09-18T04:40:04.375 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0080000000000001e-05 2024-09-18T04:40:04.375 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.375 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.376 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:04.376 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562383+0000", 2024-09-18T04:40:04.376 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013984671000000001 2024-09-18T04:40:04.376 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.376 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.376 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:04.376 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562442+0000", 2024-09-18T04:40:04.376 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8700999999999999e-05 2024-09-18T04:40:04.376 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.376 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:04.376 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.376 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.376 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.377 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:37 2.5 e51/28 2:bf39b8e4:ns::rbd_data.133e813ab5bc.0000000000000003:head v 51'5, mlcod=51'2)", 2024-09-18T04:40:04.377 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:40:04.377 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 81.790883758999996, 2024-09-18T04:40:04.377 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014264492, 2024-09-18T04:40:04.377 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:04.377 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:04.377 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:04.377 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.377 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:04.377 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:40:04.377 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.377 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.378 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:04.378 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:40:04.378 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.378 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.378 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.378 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:04.378 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548531+0000", 2024-09-18T04:40:04.378 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:04.378 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.378 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.378 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:04.378 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551500+0000", 2024-09-18T04:40:04.378 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0029688129999999998 2024-09-18T04:40:04.378 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.379 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.379 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:04.379 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551516+0000", 2024-09-18T04:40:04.379 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5994e-05 2024-09-18T04:40:04.379 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.379 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.379 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:04.379 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551526+0000", 2024-09-18T04:40:04.379 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9299999999999998e-06 2024-09-18T04:40:04.379 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.379 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.379 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:04.379 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551577+0000", 2024-09-18T04:40:04.379 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0871000000000001e-05 2024-09-18T04:40:04.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.380 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.380 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:04.380 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551602+0000", 2024-09-18T04:40:04.380 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4813000000000001e-05 2024-09-18T04:40:04.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.380 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.380 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:04.380 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562741+0000", 2024-09-18T04:40:04.380 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011139428999999999 2024-09-18T04:40:04.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.380 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.380 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:04.380 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562797+0000", 2024-09-18T04:40:04.381 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6329000000000002e-05 2024-09-18T04:40:04.381 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.381 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:04.381 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.381 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.381 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.381 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:49 2.4 e51/17 2:20f0b0c9:ns::rbd_data.133e813ab5bc.0000000000000009:head v 51'6, mlcod=51'4)", 2024-09-18T04:40:04.381 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:40:04.381 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 81.759449200999995, 2024-09-18T04:40:04.381 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013849046, 2024-09-18T04:40:04.381 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:04.381 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:04.381 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:04.382 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.382 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:04.382 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:40:04.382 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.382 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.382 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:04.382 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:40:04.382 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.382 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.382 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:04.382 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579966+0000", 2024-09-18T04:40:04.382 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:04.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.383 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.383 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:04.383 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582324+0000", 2024-09-18T04:40:04.383 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023578890000000002 2024-09-18T04:40:04.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.383 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.383 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:04.383 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582355+0000", 2024-09-18T04:40:04.383 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1485000000000001e-05 2024-09-18T04:40:04.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.383 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.383 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:04.383 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582371+0000", 2024-09-18T04:40:04.383 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6147999999999999e-05 2024-09-18T04:40:04.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.384 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.384 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:04.384 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582467+0000", 2024-09-18T04:40:04.384 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6059999999999998e-05 2024-09-18T04:40:04.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.384 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.384 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:04.384 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582505+0000", 2024-09-18T04:40:04.384 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7679000000000003e-05 2024-09-18T04:40:04.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.384 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.384 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:04.384 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593743+0000", 2024-09-18T04:40:04.385 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011238356 2024-09-18T04:40:04.385 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.385 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.385 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:04.385 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593816+0000", 2024-09-18T04:40:04.385 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2935000000000006e-05 2024-09-18T04:40:04.385 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.385 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:04.385 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.385 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.385 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.385 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:53 2.4 e51/17 2:39f7f174:ns::rbd_data.133e813ab5bc.000000000000000b:head v 51'7, mlcod=51'4)", 2024-09-18T04:40:04.385 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:40:04.385 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 81.744748045999998, 2024-09-18T04:40:04.386 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0079914159999999995, 2024-09-18T04:40:04.386 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:04.386 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:04.386 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:04.386 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.386 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:04.386 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:40:04.386 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.386 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.386 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:04.386 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:40:04.386 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.387 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:04.387 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594667+0000", 2024-09-18T04:40:04.387 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:04.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.387 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:04.387 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596446+0000", 2024-09-18T04:40:04.387 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001779033 2024-09-18T04:40:04.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.387 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:04.387 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596472+0000", 2024-09-18T04:40:04.387 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6120000000000001e-05 2024-09-18T04:40:04.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.388 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.388 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:04.388 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596500+0000", 2024-09-18T04:40:04.388 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7518e-05 2024-09-18T04:40:04.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.388 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.388 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:04.388 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596564+0000", 2024-09-18T04:40:04.388 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3880999999999994e-05 2024-09-18T04:40:04.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.388 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.388 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:04.388 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596596+0000", 2024-09-18T04:40:04.388 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2638999999999997e-05 2024-09-18T04:40:04.389 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.389 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.389 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:04.389 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602605+0000", 2024-09-18T04:40:04.389 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0060080699999999999 2024-09-18T04:40:04.389 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.389 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.389 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:04.389 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602660+0000", 2024-09-18T04:40:04.389 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5396000000000003e-05 2024-09-18T04:40:04.389 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.389 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:04.389 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.390 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.390 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:60 2.5 e51/28 2:a06f51e4:ns::rbd_data.133e813ab5bc.000000000000000f:head v 51'6, mlcod=51'4)", 2024-09-18T04:40:04.390 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:40:04.390 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 81.729931061000002, 2024-09-18T04:40:04.390 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0093954989999999999, 2024-09-18T04:40:04.390 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:04.390 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:04.390 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:04.390 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.390 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:04.390 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:40:04.390 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.391 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.391 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:04.391 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:40:04.391 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.391 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.391 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.391 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:04.391 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609483+0000", 2024-09-18T04:40:04.391 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:04.391 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.391 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.391 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:04.391 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611767+0000", 2024-09-18T04:40:04.391 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002283724 2024-09-18T04:40:04.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.392 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.392 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:04.392 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611791+0000", 2024-09-18T04:40:04.392 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3547e-05 2024-09-18T04:40:04.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.392 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.392 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:04.392 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611800+0000", 2024-09-18T04:40:04.392 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4979999999999995e-06 2024-09-18T04:40:04.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.392 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.392 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:04.392 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611864+0000", 2024-09-18T04:40:04.393 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3974999999999996e-05 2024-09-18T04:40:04.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.393 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.393 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:04.393 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611887+0000", 2024-09-18T04:40:04.393 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2633000000000002e-05 2024-09-18T04:40:04.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.393 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.393 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:04.393 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.618777+0000", 2024-09-18T04:40:04.393 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0068897979999999999 2024-09-18T04:40:04.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.393 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.393 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:04.394 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.618881+0000", 2024-09-18T04:40:04.394 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010437699999999999 2024-09-18T04:40:04.394 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.394 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:04.394 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.394 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.394 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:663 2.4 e57/17 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 57'13, mlcod=57'11)", 2024-09-18T04:40:04.394 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.355300+0000", 2024-09-18T04:40:04.394 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 72.984116126000004, 2024-09-18T04:40:04.394 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0049192760000000002, 2024-09-18T04:40:04.394 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:04.394 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:04.394 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:04.395 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.395 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:04.395 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355300+0000", 2024-09-18T04:40:04.395 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.395 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.395 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.395 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:04.395 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355300+0000", 2024-09-18T04:40:04.395 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.395 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.395 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.395 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:04.395 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355299+0000", 2024-09-18T04:40:04.396 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:04.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.396 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.396 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:04.396 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355876+0000", 2024-09-18T04:40:04.396 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00057730900000000003 2024-09-18T04:40:04.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.396 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.396 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:04.396 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355897+0000", 2024-09-18T04:40:04.396 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0477000000000001e-05 2024-09-18T04:40:04.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.396 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.396 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:04.397 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355905+0000", 2024-09-18T04:40:04.397 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5739999999999996e-06 2024-09-18T04:40:04.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.397 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.397 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:04.397 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355989+0000", 2024-09-18T04:40:04.397 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4002e-05 2024-09-18T04:40:04.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.397 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.397 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:04.397 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.356041+0000", 2024-09-18T04:40:04.397 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1347000000000002e-05 2024-09-18T04:40:04.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.397 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.398 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:04.398 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.360172+0000", 2024-09-18T04:40:04.398 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0041315090000000002 2024-09-18T04:40:04.398 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.398 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.398 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:04.398 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.360220+0000", 2024-09-18T04:40:04.398 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7584999999999999e-05 2024-09-18T04:40:04.398 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.398 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:04.398 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.398 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.398 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.399 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:679 2.5 e57/28 2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head v 57'20, mlcod=57'18)", 2024-09-18T04:40:04.399 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.445316+0000", 2024-09-18T04:40:04.399 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 72.894100648999995, 2024-09-18T04:40:04.399 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0047016760000000001, 2024-09-18T04:40:04.399 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:04.399 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:04.399 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:04.399 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.399 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:04.399 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445316+0000", 2024-09-18T04:40:04.399 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.399 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.400 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:04.400 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445316+0000", 2024-09-18T04:40:04.400 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.400 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.400 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.400 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:04.400 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445314+0000", 2024-09-18T04:40:04.400 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:04.400 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.400 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.400 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:04.400 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445947+0000", 2024-09-18T04:40:04.400 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00063298900000000003 2024-09-18T04:40:04.400 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.401 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.401 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:04.401 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445968+0000", 2024-09-18T04:40:04.401 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.054e-05 2024-09-18T04:40:04.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.401 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.401 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:04.401 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445981+0000", 2024-09-18T04:40:04.401 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2697000000000001e-05 2024-09-18T04:40:04.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.401 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.401 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:04.401 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.446102+0000", 2024-09-18T04:40:04.401 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012189300000000001 2024-09-18T04:40:04.402 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.402 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.402 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:04.402 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.446151+0000", 2024-09-18T04:40:04.402 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8279000000000003e-05 2024-09-18T04:40:04.402 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.402 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.402 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:04.402 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.449953+0000", 2024-09-18T04:40:04.402 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003801979 2024-09-18T04:40:04.402 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.402 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.402 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:04.402 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.450018+0000", 2024-09-18T04:40:04.403 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4873e-05 2024-09-18T04:40:04.403 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.403 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:04.403 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.403 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.403 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:699 2.4 e57/17 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 57'17, mlcod=57'15)", 2024-09-18T04:40:04.403 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.566073+0000", 2024-09-18T04:40:04.403 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 72.773343135999994, 2024-09-18T04:40:04.403 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055398380000000001, 2024-09-18T04:40:04.403 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:04.403 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:04.403 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:04.403 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.404 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:04.404 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566073+0000", 2024-09-18T04:40:04.404 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.404 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.404 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.404 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:04.404 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566073+0000", 2024-09-18T04:40:04.404 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.404 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.404 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.404 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:04.404 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566071+0000", 2024-09-18T04:40:04.404 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:04.405 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.405 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.405 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:04.405 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566637+0000", 2024-09-18T04:40:04.405 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00056516699999999999 2024-09-18T04:40:04.405 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.405 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.405 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:04.405 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566668+0000", 2024-09-18T04:40:04.405 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1544999999999997e-05 2024-09-18T04:40:04.405 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.405 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.405 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:04.405 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566685+0000", 2024-09-18T04:40:04.406 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6857000000000001e-05 2024-09-18T04:40:04.406 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.406 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.406 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:04.406 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566816+0000", 2024-09-18T04:40:04.406 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013092799999999999 2024-09-18T04:40:04.406 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.406 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.406 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:04.406 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566875+0000", 2024-09-18T04:40:04.406 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9583999999999997e-05 2024-09-18T04:40:04.406 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.406 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.407 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:04.407 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.571567+0000", 2024-09-18T04:40:04.407 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.004691682 2024-09-18T04:40:04.407 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.407 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.407 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:04.407 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.571613+0000", 2024-09-18T04:40:04.407 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6125000000000003e-05 2024-09-18T04:40:04.407 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.407 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:04.407 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.407 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.407 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.407 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:98 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'27, mlcod=60'25)", 2024-09-18T04:40:04.408 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.847978+0000", 2024-09-18T04:40:04.408 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 63.491438576999997, 2024-09-18T04:40:04.408 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0044617709999999998, 2024-09-18T04:40:04.408 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:04.408 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:04.408 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:04.408 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.408 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:04.408 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.847978+0000", 2024-09-18T04:40:04.408 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.408 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.408 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.408 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:04.408 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.847978+0000", 2024-09-18T04:40:04.409 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.409 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.409 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.409 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:04.409 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.847977+0000", 2024-09-18T04:40:04.409 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:04.409 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.409 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.409 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:04.409 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849038+0000", 2024-09-18T04:40:04.409 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0010614820000000001 2024-09-18T04:40:04.409 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.409 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.410 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:04.410 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849065+0000", 2024-09-18T04:40:04.410 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7127999999999998e-05 2024-09-18T04:40:04.410 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.410 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.410 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:04.410 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849082+0000", 2024-09-18T04:40:04.410 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7170999999999999e-05 2024-09-18T04:40:04.410 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.410 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.410 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:04.410 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849188+0000", 2024-09-18T04:40:04.410 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010605800000000001 2024-09-18T04:40:04.411 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.411 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.411 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:04.411 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849233+0000", 2024-09-18T04:40:04.411 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4579000000000001e-05 2024-09-18T04:40:04.411 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.411 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.411 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:04.411 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.852370+0000", 2024-09-18T04:40:04.411 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00313751 2024-09-18T04:40:04.411 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.411 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.411 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:04.411 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.852440+0000", 2024-09-18T04:40:04.411 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9238000000000004e-05 2024-09-18T04:40:04.412 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.412 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:04.412 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.412 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.412 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.412 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:100 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'29, mlcod=64'27)", 2024-09-18T04:40:04.412 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.858766+0000", 2024-09-18T04:40:04.412 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 63.480650248000003, 2024-09-18T04:40:04.412 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0061964840000000004, 2024-09-18T04:40:04.412 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:04.412 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:04.412 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:04.412 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.412 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:04.413 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.858766+0000", 2024-09-18T04:40:04.413 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.413 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.413 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.413 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:04.413 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.858766+0000", 2024-09-18T04:40:04.413 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.413 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.413 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.413 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:04.413 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.858765+0000", 2024-09-18T04:40:04.413 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:04.413 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.413 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.414 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:04.414 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861612+0000", 2024-09-18T04:40:04.414 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028473169999999998 2024-09-18T04:40:04.414 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.414 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.414 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:04.414 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861643+0000", 2024-09-18T04:40:04.414 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1356000000000003e-05 2024-09-18T04:40:04.414 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.414 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.414 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:04.414 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861656+0000", 2024-09-18T04:40:04.414 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2899e-05 2024-09-18T04:40:04.414 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.414 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.415 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:04.415 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861773+0000", 2024-09-18T04:40:04.415 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011637999999999999 2024-09-18T04:40:04.415 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.415 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.415 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:04.415 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861808+0000", 2024-09-18T04:40:04.415 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5002999999999997e-05 2024-09-18T04:40:04.415 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.415 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.415 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:04.415 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.864876+0000", 2024-09-18T04:40:04.415 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030680719999999998 2024-09-18T04:40:04.415 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.415 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.416 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:04.416 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.864963+0000", 2024-09-18T04:40:04.416 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7114000000000007e-05 2024-09-18T04:40:04.416 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.416 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:04.416 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.416 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.416 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.416 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:102 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'31, mlcod=64'29)", 2024-09-18T04:40:04.416 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.872098+0000", 2024-09-18T04:40:04.416 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 63.467318990999999, 2024-09-18T04:40:04.416 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0064676450000000002, 2024-09-18T04:40:04.416 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:04.416 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:04.417 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:04.417 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.417 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:04.417 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.872098+0000", 2024-09-18T04:40:04.417 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.417 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.417 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.417 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:04.417 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.872098+0000", 2024-09-18T04:40:04.417 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.417 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.417 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.417 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:04.417 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.872096+0000", 2024-09-18T04:40:04.418 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:04.418 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.418 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.418 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:04.418 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875176+0000", 2024-09-18T04:40:04.418 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030799809999999999 2024-09-18T04:40:04.418 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.418 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.418 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:04.418 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875204+0000", 2024-09-18T04:40:04.418 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8025999999999999e-05 2024-09-18T04:40:04.418 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.418 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.418 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:04.418 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875216+0000", 2024-09-18T04:40:04.419 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2072999999999999e-05 2024-09-18T04:40:04.419 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.419 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.419 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:04.419 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875307+0000", 2024-09-18T04:40:04.419 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0373999999999999e-05 2024-09-18T04:40:04.419 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.419 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.419 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:04.419 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875349+0000", 2024-09-18T04:40:04.419 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2299000000000003e-05 2024-09-18T04:40:04.419 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.419 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.419 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:04.419 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.878496+0000", 2024-09-18T04:40:04.420 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0031468960000000002 2024-09-18T04:40:04.420 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.420 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.420 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:04.420 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.878565+0000", 2024-09-18T04:40:04.420 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9465e-05 2024-09-18T04:40:04.420 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.420 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:04.420 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.420 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.420 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.420 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:104 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'33, mlcod=64'31)", 2024-09-18T04:40:04.420 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.885100+0000", 2024-09-18T04:40:04.420 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 63.454316456000001, 2024-09-18T04:40:04.421 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0051888159999999997, 2024-09-18T04:40:04.421 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:04.421 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:04.421 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:04.421 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.421 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:04.421 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.885100+0000", 2024-09-18T04:40:04.421 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.421 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.421 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.421 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:04.421 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.885100+0000", 2024-09-18T04:40:04.421 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.421 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.422 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.422 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:04.422 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.885098+0000", 2024-09-18T04:40:04.422 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:04.422 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.422 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.422 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:04.422 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886809+0000", 2024-09-18T04:40:04.422 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017110459999999999 2024-09-18T04:40:04.422 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.422 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.422 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:04.422 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886842+0000", 2024-09-18T04:40:04.422 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2638999999999997e-05 2024-09-18T04:40:04.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.423 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.423 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:04.423 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886855+0000", 2024-09-18T04:40:04.423 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3380000000000001e-05 2024-09-18T04:40:04.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.423 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.423 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:04.423 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886939+0000", 2024-09-18T04:40:04.423 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3733999999999995e-05 2024-09-18T04:40:04.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.423 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.424 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:04.424 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886981+0000", 2024-09-18T04:40:04.424 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1956000000000003e-05 2024-09-18T04:40:04.424 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.424 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.424 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:04.424 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.890201+0000", 2024-09-18T04:40:04.424 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003219927 2024-09-18T04:40:04.424 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.424 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.424 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:04.425 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.890289+0000", 2024-09-18T04:40:04.425 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7997999999999993e-05 2024-09-18T04:40:04.425 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.425 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:04.425 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.425 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.425 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.425 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:145 2.4 e64/62 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 64'37, mlcod=64'35)", 2024-09-18T04:40:04.425 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:01.099398+0000", 2024-09-18T04:40:04.425 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 63.240018968000001, 2024-09-18T04:40:04.425 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0050078459999999998, 2024-09-18T04:40:04.425 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:04.425 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:04.425 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:04.426 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.426 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:04.426 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.099398+0000", 2024-09-18T04:40:04.426 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.426 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.426 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:04.426 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.099398+0000", 2024-09-18T04:40:04.426 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.427 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.427 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:04.427 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.099396+0000", 2024-09-18T04:40:04.427 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:04.427 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.427 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.427 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:04.427 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100199+0000", 2024-09-18T04:40:04.427 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00080338000000000002 2024-09-18T04:40:04.427 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.427 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.427 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:04.427 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100230+0000", 2024-09-18T04:40:04.427 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0716000000000002e-05 2024-09-18T04:40:04.428 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.428 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.428 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:04.428 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100243+0000", 2024-09-18T04:40:04.428 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3359e-05 2024-09-18T04:40:04.428 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.428 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.428 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:04.428 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100337+0000", 2024-09-18T04:40:04.428 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4387000000000003e-05 2024-09-18T04:40:04.428 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.428 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.428 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:04.428 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100396+0000", 2024-09-18T04:40:04.429 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8164000000000001e-05 2024-09-18T04:40:04.429 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.429 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.429 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:04.429 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.104309+0000", 2024-09-18T04:40:04.429 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0039137800000000004 2024-09-18T04:40:04.429 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.429 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.429 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:04.429 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.104405+0000", 2024-09-18T04:40:04.429 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6039999999999995e-05 2024-09-18T04:40:04.429 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.429 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:04.429 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.430 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.430 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.430 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:147 2.4 e64/62 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 64'39, mlcod=64'37)", 2024-09-18T04:40:04.430 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:01.110658+0000", 2024-09-18T04:40:04.430 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 63.228758878999997, 2024-09-18T04:40:04.430 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0046316910000000003, 2024-09-18T04:40:04.430 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:04.430 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:04.430 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:04.430 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.430 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:04.430 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.110658+0000", 2024-09-18T04:40:04.430 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.430 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.431 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.431 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:04.431 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.110658+0000", 2024-09-18T04:40:04.431 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.431 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.431 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.431 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:04.431 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.110656+0000", 2024-09-18T04:40:04.431 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:04.431 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.431 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.431 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:04.431 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111444+0000", 2024-09-18T04:40:04.431 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00078762400000000001 2024-09-18T04:40:04.432 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.432 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.432 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:04.432 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111474+0000", 2024-09-18T04:40:04.432 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0085000000000001e-05 2024-09-18T04:40:04.432 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.432 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.432 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:04.432 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111486+0000", 2024-09-18T04:40:04.432 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2645e-05 2024-09-18T04:40:04.432 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.432 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.432 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:04.432 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111581+0000", 2024-09-18T04:40:04.433 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4432000000000007e-05 2024-09-18T04:40:04.433 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.433 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.433 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:04.433 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111631+0000", 2024-09-18T04:40:04.433 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9901000000000002e-05 2024-09-18T04:40:04.433 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.433 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.433 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:04.433 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.115209+0000", 2024-09-18T04:40:04.433 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003577916 2024-09-18T04:40:04.433 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.433 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.433 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:04.433 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.115289+0000", 2024-09-18T04:40:04.434 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0704999999999994e-05 2024-09-18T04:40:04.434 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.434 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:04.434 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.434 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.434 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:04.434 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:40:04.436 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:40:04.436 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:40:04.431+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:40:04.536 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:40:04.532+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:40:04.547 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:40:04.547 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:40:04.547 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:40:04.547 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:40:04.557 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:40:04.637 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:40:04.632+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:40:04.665 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:40:04.665 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:40:04.665 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:40:04.665 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:40:04.665 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:40:04.675 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:40:04.738 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:40:04.731+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:40:04.787 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:40:04.788 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:40:04.788 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:40:04.788 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:40:04.788 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.788 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:33 2.1 e51/24 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head v 51'4, mlcod=51'2)", 2024-09-18T04:40:04.788 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:40:04.788 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 82.504915140999998, 2024-09-18T04:40:04.788 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018702106, 2024-09-18T04:40:04.788 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:04.788 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:04.788 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:04.788 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.788 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:04.789 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:40:04.789 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.789 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.789 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.789 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:04.789 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:40:04.789 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.789 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.789 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.789 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:04.789 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279197+0000", 2024-09-18T04:40:04.789 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:04.789 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.789 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.790 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:04.790 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282682+0000", 2024-09-18T04:40:04.790 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0034848689999999998 2024-09-18T04:40:04.790 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.790 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.790 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:04.790 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282709+0000", 2024-09-18T04:40:04.790 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7599000000000001e-05 2024-09-18T04:40:04.790 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.790 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.790 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:04.790 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282720+0000", 2024-09-18T04:40:04.790 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.092e-05 2024-09-18T04:40:04.791 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.791 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.791 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:04.791 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282791+0000", 2024-09-18T04:40:04.791 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1268999999999998e-05 2024-09-18T04:40:04.791 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.791 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.791 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:04.791 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282843+0000", 2024-09-18T04:40:04.791 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1771999999999998e-05 2024-09-18T04:40:04.791 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.791 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.791 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:04.791 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.297830+0000", 2024-09-18T04:40:04.792 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014987228999999999 2024-09-18T04:40:04.792 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.792 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.792 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:04.792 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.297901+0000", 2024-09-18T04:40:04.792 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0209999999999994e-05 2024-09-18T04:40:04.792 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.792 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:04.792 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.792 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.792 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.792 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:38 2.0 e51/17 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head v 51'10, mlcod=51'8)", 2024-09-18T04:40:04.792 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:40:04.793 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 82.503496710999997, 2024-09-18T04:40:04.793 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.023984472, 2024-09-18T04:40:04.793 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:04.793 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:04.793 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:04.793 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.793 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:04.793 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:40:04.793 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.793 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.793 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.793 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:04.793 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:40:04.794 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.794 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.794 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.794 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:04.794 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280614+0000", 2024-09-18T04:40:04.794 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:40:04.794 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.794 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.794 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:04.794 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293094+0000", 2024-09-18T04:40:04.794 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012479571 2024-09-18T04:40:04.794 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.794 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.795 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:04.795 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293111+0000", 2024-09-18T04:40:04.795 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7354999999999999e-05 2024-09-18T04:40:04.795 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.795 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.795 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:04.795 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293121+0000", 2024-09-18T04:40:04.795 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6979999999999993e-06 2024-09-18T04:40:04.795 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.795 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.795 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:04.795 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293180+0000", 2024-09-18T04:40:04.795 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9333000000000003e-05 2024-09-18T04:40:04.796 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.796 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.796 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:04.796 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293203+0000", 2024-09-18T04:40:04.796 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2633000000000002e-05 2024-09-18T04:40:04.796 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.796 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.796 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:04.796 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304534+0000", 2024-09-18T04:40:04.796 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011330761 2024-09-18T04:40:04.796 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.796 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.797 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:04.797 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304601+0000", 2024-09-18T04:40:04.797 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7657000000000006e-05 2024-09-18T04:40:04.797 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.797 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:04.797 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.797 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.797 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.797 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:46 2.6 2:7ba25245:::rbd_data.125fc09df6d1.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:04.797 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:40:04.797 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 82.475953911000005, 2024-09-18T04:40:04.797 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.035604595000000003, 2024-09-18T04:40:04.797 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:04.797 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:04.798 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:04.798 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:40:04.798 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:40:04.798 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 46 2024-09-18T04:40:04.798 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.798 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:04.798 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.798 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:04.798 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:40:04.798 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.798 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.798 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.798 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:04.798 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:40:04.799 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.799 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.799 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.799 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:04.799 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308158+0000", 2024-09-18T04:40:04.799 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:04.799 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.799 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.799 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:04.799 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313191+0000", 2024-09-18T04:40:04.799 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0050323929999999996 2024-09-18T04:40:04.799 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.799 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.800 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:04.800 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313210+0000", 2024-09-18T04:40:04.800 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9673e-05 2024-09-18T04:40:04.800 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.800 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.800 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:04.800 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313221+0000", 2024-09-18T04:40:04.800 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0444e-05 2024-09-18T04:40:04.800 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.800 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.800 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:04.800 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313294+0000", 2024-09-18T04:40:04.800 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2650000000000004e-05 2024-09-18T04:40:04.801 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.801 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.801 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:04.801 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313493+0000", 2024-09-18T04:40:04.801 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019968300000000001 2024-09-18T04:40:04.801 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.801 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.801 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:40:04.801 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313666+0000", 2024-09-18T04:40:04.801 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00017248599999999999 2024-09-18T04:40:04.802 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.802 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.802 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:04.802 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.323175+0000", 2024-09-18T04:40:04.802 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0095088819999999998 2024-09-18T04:40:04.802 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.802 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.802 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:04.802 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343651+0000", 2024-09-18T04:40:04.802 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020475892999999998 2024-09-18T04:40:04.802 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.802 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.802 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:04.803 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343715+0000", 2024-09-18T04:40:04.803 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4443999999999995e-05 2024-09-18T04:40:04.803 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.803 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.803 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:04.803 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343764+0000", 2024-09-18T04:40:04.803 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9280000000000003e-05 2024-09-18T04:40:04.803 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.803 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:04.803 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.803 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.803 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.803 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:47 2.4 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:04.803 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:40:04.805 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 82.470871873999997, 2024-09-18T04:40:04.805 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.032570693999999997, 2024-09-18T04:40:04.805 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:04.805 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:04.805 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:04.805 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:40:04.805 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:40:04.805 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 47 2024-09-18T04:40:04.805 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.805 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:04.805 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.806 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:04.806 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:40:04.806 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.806 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.806 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.806 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:04.806 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:40:04.806 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.806 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.806 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.806 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:04.806 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313240+0000", 2024-09-18T04:40:04.806 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:04.806 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.807 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.807 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:04.807 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325825+0000", 2024-09-18T04:40:04.807 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012584425999999999 2024-09-18T04:40:04.807 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.807 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.807 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:04.807 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325848+0000", 2024-09-18T04:40:04.807 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3025999999999999e-05 2024-09-18T04:40:04.807 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.807 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.807 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:04.807 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325863+0000", 2024-09-18T04:40:04.807 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5152e-05 2024-09-18T04:40:04.807 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.808 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.808 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:04.808 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325938+0000", 2024-09-18T04:40:04.808 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4982999999999997e-05 2024-09-18T04:40:04.808 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.808 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.808 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:04.808 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326090+0000", 2024-09-18T04:40:04.808 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015207900000000001 2024-09-18T04:40:04.808 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.808 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.808 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:40:04.808 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326242+0000", 2024-09-18T04:40:04.808 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015163500000000001 2024-09-18T04:40:04.809 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.809 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.809 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:04.809 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343015+0000", 2024-09-18T04:40:04.809 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016773414 2024-09-18T04:40:04.809 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.809 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.809 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:04.809 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345664+0000", 2024-09-18T04:40:04.809 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026487870000000001 2024-09-18T04:40:04.809 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.809 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.809 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:04.809 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345739+0000", 2024-09-18T04:40:04.810 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5018999999999994e-05 2024-09-18T04:40:04.810 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.810 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.810 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:04.810 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345812+0000", 2024-09-18T04:40:04.810 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3547000000000006e-05 2024-09-18T04:40:04.810 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.810 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:04.810 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.810 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.810 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.811 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:48 2.6 2:61db7a3a:::rbd_data.125fc09df6d1.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:04.811 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:40:04.811 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 82.458215202000005, 2024-09-18T04:40:04.811 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.025064388, 2024-09-18T04:40:04.811 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:04.811 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:04.811 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:04.811 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:40:04.811 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:40:04.811 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 48 2024-09-18T04:40:04.812 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.812 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:04.812 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.812 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:04.812 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:40:04.812 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.812 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.812 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.812 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:04.812 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:40:04.812 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.812 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.812 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.813 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:04.813 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325897+0000", 2024-09-18T04:40:04.813 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:04.813 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.813 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.813 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:04.813 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333697+0000", 2024-09-18T04:40:04.813 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0078002649999999998 2024-09-18T04:40:04.813 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.813 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.813 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:04.813 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333716+0000", 2024-09-18T04:40:04.813 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8910999999999999e-05 2024-09-18T04:40:04.813 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.813 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.814 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:04.814 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333728+0000", 2024-09-18T04:40:04.814 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1936000000000001e-05 2024-09-18T04:40:04.814 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.814 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.814 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:04.814 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333792+0000", 2024-09-18T04:40:04.814 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4010999999999994e-05 2024-09-18T04:40:04.814 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.814 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.814 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:04.814 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.334082+0000", 2024-09-18T04:40:04.814 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00028955799999999998 2024-09-18T04:40:04.814 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.815 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.815 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:40:04.815 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.334326+0000", 2024-09-18T04:40:04.815 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00024399099999999999 2024-09-18T04:40:04.815 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.815 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.815 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:04.815 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350745+0000", 2024-09-18T04:40:04.815 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016419474999999999 2024-09-18T04:40:04.815 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.815 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.815 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:04.815 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350808+0000", 2024-09-18T04:40:04.815 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3244999999999995e-05 2024-09-18T04:40:04.816 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.816 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.816 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:04.816 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350892+0000", 2024-09-18T04:40:04.816 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4074999999999997e-05 2024-09-18T04:40:04.816 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.816 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.816 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:04.816 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350963+0000", 2024-09-18T04:40:04.816 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0412000000000003e-05 2024-09-18T04:40:04.816 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.816 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:04.816 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.816 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.816 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.817 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:51 2.6 2:775d7b11:::rbd_data.125fc09df6d1.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:04.817 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:40:04.817 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 82.450363065999994, 2024-09-18T04:40:04.817 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.023824221, 2024-09-18T04:40:04.817 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:04.817 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:04.817 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:04.817 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:40:04.817 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:40:04.817 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 51 2024-09-18T04:40:04.817 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.817 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:04.817 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.817 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:04.818 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:40:04.818 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.818 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.818 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.818 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:04.818 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:40:04.818 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.818 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.818 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.818 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:04.818 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333749+0000", 2024-09-18T04:40:04.818 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:04.818 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.818 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.819 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:04.819 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340899+0000", 2024-09-18T04:40:04.819 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0071500340000000004 2024-09-18T04:40:04.819 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.819 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.819 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:04.819 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340929+0000", 2024-09-18T04:40:04.819 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0093000000000001e-05 2024-09-18T04:40:04.819 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.819 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.819 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:04.819 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340943+0000", 2024-09-18T04:40:04.819 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4019000000000001e-05 2024-09-18T04:40:04.819 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.819 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.820 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:04.820 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343834+0000", 2024-09-18T04:40:04.820 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028918120000000001 2024-09-18T04:40:04.820 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.820 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.820 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:04.820 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344020+0000", 2024-09-18T04:40:04.820 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000185892 2024-09-18T04:40:04.820 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.820 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.820 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:40:04.820 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344190+0000", 2024-09-18T04:40:04.820 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016946800000000001 2024-09-18T04:40:04.820 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.821 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.821 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:04.821 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.351291+0000", 2024-09-18T04:40:04.821 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.007100805 2024-09-18T04:40:04.821 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.821 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.821 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:04.821 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357470+0000", 2024-09-18T04:40:04.821 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0061797570000000001 2024-09-18T04:40:04.821 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.821 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.821 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:04.821 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357533+0000", 2024-09-18T04:40:04.821 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2906999999999999e-05 2024-09-18T04:40:04.822 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.822 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.822 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:04.822 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357575+0000", 2024-09-18T04:40:04.822 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1403999999999999e-05 2024-09-18T04:40:04.822 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.822 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:04.822 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.822 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.822 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.822 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:56 2.4 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:04.822 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:40:04.822 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 82.434317973000006, 2024-09-18T04:40:04.822 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.034295870999999999, 2024-09-18T04:40:04.822 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:04.823 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:04.823 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:04.823 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:40:04.823 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:40:04.823 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 56 2024-09-18T04:40:04.823 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.823 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:04.823 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.823 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:04.823 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:40:04.823 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.823 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.823 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.823 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:04.823 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:40:04.824 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.824 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.824 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.824 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:04.824 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349794+0000", 2024-09-18T04:40:04.824 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:04.824 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.824 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.824 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:04.824 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366632+0000", 2024-09-18T04:40:04.824 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01683755 2024-09-18T04:40:04.824 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.824 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.824 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:04.825 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366669+0000", 2024-09-18T04:40:04.825 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7351000000000002e-05 2024-09-18T04:40:04.825 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.826 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.826 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:04.826 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366686+0000", 2024-09-18T04:40:04.826 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6507999999999998e-05 2024-09-18T04:40:04.826 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.826 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.826 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:04.826 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366962+0000", 2024-09-18T04:40:04.826 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00027655900000000001 2024-09-18T04:40:04.826 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.826 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.826 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:04.826 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.367452+0000", 2024-09-18T04:40:04.826 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00049025299999999996 2024-09-18T04:40:04.827 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.827 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.827 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:40:04.827 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368113+0000", 2024-09-18T04:40:04.827 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00066093399999999996 2024-09-18T04:40:04.827 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.827 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.827 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:04.827 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383719+0000", 2024-09-18T04:40:04.827 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015606106999999999 2024-09-18T04:40:04.827 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.827 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.827 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:04.827 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384035+0000", 2024-09-18T04:40:04.828 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00031590399999999999 2024-09-18T04:40:04.828 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.828 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.828 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:04.828 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384067+0000", 2024-09-18T04:40:04.828 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.129e-05 2024-09-18T04:40:04.828 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.828 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.828 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:04.828 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384091+0000", 2024-09-18T04:40:04.828 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4788000000000001e-05 2024-09-18T04:40:04.828 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.828 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:04.828 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.828 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.829 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.829 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:57 2.0 e51/17 2:0cdd070e:::rbd_data.125fc09df6d1.000000000000000d:head v 51'21, mlcod=51'19)", 2024-09-18T04:40:04.829 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:40:04.829 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 82.426490486000006, 2024-09-18T04:40:04.829 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.025216311000000002, 2024-09-18T04:40:04.829 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:04.829 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:04.829 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:04.829 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.829 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:04.829 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:40:04.829 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.829 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.829 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.830 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:04.830 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:40:04.830 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.830 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.830 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.830 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:04.830 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:40:04.830 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:40:04.830 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.830 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.830 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:04.830 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363844+0000", 2024-09-18T04:40:04.830 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0062218450000000002 2024-09-18T04:40:04.830 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.830 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.831 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:04.831 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363879+0000", 2024-09-18T04:40:04.831 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4382999999999999e-05 2024-09-18T04:40:04.831 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.831 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.831 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:04.831 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363893+0000", 2024-09-18T04:40:04.831 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4348999999999999e-05 2024-09-18T04:40:04.831 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.831 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.831 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:04.831 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363977+0000", 2024-09-18T04:40:04.831 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3499999999999997e-05 2024-09-18T04:40:04.831 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.832 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.832 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:04.832 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364009+0000", 2024-09-18T04:40:04.832 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2292000000000002e-05 2024-09-18T04:40:04.832 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.832 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.832 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:04.832 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.382781+0000", 2024-09-18T04:40:04.832 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018771626 2024-09-18T04:40:04.832 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.832 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.832 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:04.832 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.382839+0000", 2024-09-18T04:40:04.832 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8758000000000002e-05 2024-09-18T04:40:04.832 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.833 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:04.833 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.833 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.833 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.833 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:59 2.0 e51/17 2:19c52cde:::rbd_data.125fc09df6d1.000000000000000f:head v 51'22, mlcod=51'19)", 2024-09-18T04:40:04.833 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:40:04.833 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 82.419830559999994, 2024-09-18T04:40:04.833 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.019497112, 2024-09-18T04:40:04.833 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:04.833 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:04.833 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:04.833 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.833 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:04.833 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:40:04.834 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.834 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.834 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.834 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:04.834 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:40:04.834 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.834 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.834 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.834 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:04.834 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364282+0000", 2024-09-18T04:40:04.834 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:04.834 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.834 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.834 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:04.834 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371823+0000", 2024-09-18T04:40:04.835 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0075411829999999999 2024-09-18T04:40:04.835 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.835 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.835 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:04.835 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371837+0000", 2024-09-18T04:40:04.835 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4336e-05 2024-09-18T04:40:04.835 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.835 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.835 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:04.835 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371846+0000", 2024-09-18T04:40:04.835 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9770000000000006e-06 2024-09-18T04:40:04.835 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.835 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.835 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:04.836 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.373566+0000", 2024-09-18T04:40:04.836 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017192539999999999 2024-09-18T04:40:04.836 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.836 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.836 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:04.836 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.373595+0000", 2024-09-18T04:40:04.836 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8975999999999998e-05 2024-09-18T04:40:04.836 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.836 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.836 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:04.836 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383740+0000", 2024-09-18T04:40:04.836 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01014525 2024-09-18T04:40:04.836 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.836 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.836 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:04.837 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383780+0000", 2024-09-18T04:40:04.837 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0275999999999999e-05 2024-09-18T04:40:04.837 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.837 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:04.837 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.837 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.837 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.837 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:34 2.6 2:66289171:ns::rbd_data.133e813ab5bc.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:04.837 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.524015+0000", 2024-09-18T04:40:04.837 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 82.260098951000003, 2024-09-18T04:40:04.837 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027582111999999999, 2024-09-18T04:40:04.837 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:04.837 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:04.837 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:04.838 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:40:04.838 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:04.838 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 34 2024-09-18T04:40:04.838 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.838 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:04.838 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.838 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:04.838 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524015+0000", 2024-09-18T04:40:04.838 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.838 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.838 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.838 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:04.838 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524015+0000", 2024-09-18T04:40:04.838 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.838 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.839 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.839 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:04.839 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524012+0000", 2024-09-18T04:40:04.839 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:40:04.839 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.839 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.839 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:04.839 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534956+0000", 2024-09-18T04:40:04.839 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010943403000000001 2024-09-18T04:40:04.839 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.839 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.839 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:04.839 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534978+0000", 2024-09-18T04:40:04.839 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1965999999999999e-05 2024-09-18T04:40:04.839 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.840 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.840 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:04.840 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534990+0000", 2024-09-18T04:40:04.840 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2065999999999999e-05 2024-09-18T04:40:04.840 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.840 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.840 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:04.840 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535065+0000", 2024-09-18T04:40:04.840 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5523000000000002e-05 2024-09-18T04:40:04.840 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.840 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.840 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:04.840 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535208+0000", 2024-09-18T04:40:04.840 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014309500000000001 2024-09-18T04:40:04.841 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.841 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.841 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:40:04.841 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535343+0000", 2024-09-18T04:40:04.841 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013437400000000001 2024-09-18T04:40:04.841 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.841 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.841 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:04.841 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.546576+0000", 2024-09-18T04:40:04.841 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011233425999999999 2024-09-18T04:40:04.841 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.841 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.841 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:04.841 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551498+0000", 2024-09-18T04:40:04.841 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0049219349999999997 2024-09-18T04:40:04.842 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.842 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.842 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:04.842 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551554+0000", 2024-09-18T04:40:04.842 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6400999999999998e-05 2024-09-18T04:40:04.842 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.842 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.842 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:04.842 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551597+0000", 2024-09-18T04:40:04.842 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2283000000000002e-05 2024-09-18T04:40:04.842 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.842 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:04.842 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.842 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.843 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.843 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:33 2.1 e51/24 2:8b40f1c5:ns::rbd_data.133e813ab5bc.0000000000000000:head v 51'7, mlcod=51'5)", 2024-09-18T04:40:04.843 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.531767+0000", 2024-09-18T04:40:04.843 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 82.252347033999996, 2024-09-18T04:40:04.843 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015370295000000001, 2024-09-18T04:40:04.843 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:04.843 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:04.843 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:04.843 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.843 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:04.843 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531767+0000", 2024-09-18T04:40:04.843 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.843 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.843 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.843 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:04.844 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531767+0000", 2024-09-18T04:40:04.844 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.844 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.844 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.844 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:04.844 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531765+0000", 2024-09-18T04:40:04.844 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:04.844 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.844 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.844 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:04.844 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534258+0000", 2024-09-18T04:40:04.844 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002493561 2024-09-18T04:40:04.844 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.844 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.845 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:04.845 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534288+0000", 2024-09-18T04:40:04.845 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.976e-05 2024-09-18T04:40:04.845 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.845 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.845 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:04.845 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534300+0000", 2024-09-18T04:40:04.845 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1928e-05 2024-09-18T04:40:04.845 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.845 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.845 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:04.845 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534367+0000", 2024-09-18T04:40:04.845 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7045999999999995e-05 2024-09-18T04:40:04.845 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.845 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.846 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:04.846 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534394+0000", 2024-09-18T04:40:04.846 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6903999999999999e-05 2024-09-18T04:40:04.846 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.846 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.846 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:04.846 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.547073+0000", 2024-09-18T04:40:04.846 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012679365 2024-09-18T04:40:04.846 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.846 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.846 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:04.846 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.547137+0000", 2024-09-18T04:40:04.846 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3691999999999999e-05 2024-09-18T04:40:04.846 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.846 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:04.847 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.847 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.847 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.847 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:39 2.4 2:304364d1:ns::rbd_data.133e813ab5bc.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:04.847 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.535013+0000", 2024-09-18T04:40:04.847 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 82.249100099000003, 2024-09-18T04:40:04.847 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027764487000000001, 2024-09-18T04:40:04.847 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:04.847 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:04.847 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:04.847 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:40:04.847 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:04.847 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 39 2024-09-18T04:40:04.847 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.848 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:04.848 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.848 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:04.848 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535013+0000", 2024-09-18T04:40:04.848 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.848 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.848 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.848 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:04.848 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535013+0000", 2024-09-18T04:40:04.848 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.848 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.848 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.848 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:04.848 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535012+0000", 2024-09-18T04:40:04.849 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:04.849 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.849 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.849 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:04.849 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545016+0000", 2024-09-18T04:40:04.849 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01000322 2024-09-18T04:40:04.849 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.849 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.849 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:04.849 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545035+0000", 2024-09-18T04:40:04.849 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9845999999999999e-05 2024-09-18T04:40:04.849 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.849 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.849 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:04.849 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545045+0000", 2024-09-18T04:40:04.850 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3640000000000005e-06 2024-09-18T04:40:04.850 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.850 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.850 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:04.850 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545152+0000", 2024-09-18T04:40:04.850 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000107025 2024-09-18T04:40:04.850 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.850 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.850 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:04.850 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545338+0000", 2024-09-18T04:40:04.850 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018643500000000001 2024-09-18T04:40:04.850 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.850 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.850 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:40:04.850 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545492+0000", 2024-09-18T04:40:04.851 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015393199999999999 2024-09-18T04:40:04.851 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.851 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.851 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:04.851 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.553549+0000", 2024-09-18T04:40:04.851 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0080569170000000002 2024-09-18T04:40:04.851 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.851 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.851 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:04.851 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562693+0000", 2024-09-18T04:40:04.851 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0091441609999999996 2024-09-18T04:40:04.851 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.851 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.851 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:04.851 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562748+0000", 2024-09-18T04:40:04.852 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5047000000000003e-05 2024-09-18T04:40:04.852 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.852 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.852 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:04.852 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562778+0000", 2024-09-18T04:40:04.852 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9652e-05 2024-09-18T04:40:04.852 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.852 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:04.852 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.852 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.852 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.852 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:40 2.6 2:784d6cd1:ns::rbd_data.133e813ab5bc.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:04.852 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.545062+0000", 2024-09-18T04:40:04.852 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 82.239051047999993, 2024-09-18T04:40:04.853 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.032771623, 2024-09-18T04:40:04.853 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:04.853 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:04.853 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:04.853 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:40:04.853 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:04.853 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 40 2024-09-18T04:40:04.853 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.853 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:04.853 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.853 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:04.853 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545062+0000", 2024-09-18T04:40:04.853 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.853 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.854 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.854 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:04.854 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545062+0000", 2024-09-18T04:40:04.854 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.854 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.854 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.854 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:04.854 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545061+0000", 2024-09-18T04:40:04.854 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:04.854 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.854 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.854 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:04.854 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552477+0000", 2024-09-18T04:40:04.854 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.007415914 2024-09-18T04:40:04.855 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.855 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.855 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:04.855 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552500+0000", 2024-09-18T04:40:04.855 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3546000000000001e-05 2024-09-18T04:40:04.855 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.855 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.855 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:04.855 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552515+0000", 2024-09-18T04:40:04.855 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4606999999999999e-05 2024-09-18T04:40:04.855 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.855 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.855 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:04.855 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552661+0000", 2024-09-18T04:40:04.855 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000146007 2024-09-18T04:40:04.856 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.856 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.856 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:04.856 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552876+0000", 2024-09-18T04:40:04.856 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000214618 2024-09-18T04:40:04.856 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.856 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.856 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:40:04.856 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.553061+0000", 2024-09-18T04:40:04.856 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018564100000000001 2024-09-18T04:40:04.856 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.856 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.856 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:04.856 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.565612+0000", 2024-09-18T04:40:04.856 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012550301999999999 2024-09-18T04:40:04.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.857 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.857 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:04.857 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577695+0000", 2024-09-18T04:40:04.857 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012083493000000001 2024-09-18T04:40:04.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.857 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.857 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:04.857 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577772+0000", 2024-09-18T04:40:04.857 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7158999999999998e-05 2024-09-18T04:40:04.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.857 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.857 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:04.857 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577834+0000", 2024-09-18T04:40:04.858 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1851000000000001e-05 2024-09-18T04:40:04.858 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.858 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:04.858 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.858 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.858 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.858 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:43 2.6 2:759f27b6:ns::rbd_data.133e813ab5bc.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:04.858 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.555868+0000", 2024-09-18T04:40:04.858 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 82.228245553999997, 2024-09-18T04:40:04.858 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.022332497999999999, 2024-09-18T04:40:04.858 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:04.858 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:04.858 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:04.859 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:40:04.859 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:04.859 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 43 2024-09-18T04:40:04.859 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.859 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:04.859 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.859 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:04.859 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555868+0000", 2024-09-18T04:40:04.859 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.859 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.859 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.859 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:04.859 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555868+0000", 2024-09-18T04:40:04.859 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.859 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.860 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.860 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:04.860 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555865+0000", 2024-09-18T04:40:04.860 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999971 2024-09-18T04:40:04.860 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.860 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.860 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:04.860 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561031+0000", 2024-09-18T04:40:04.860 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0051658770000000001 2024-09-18T04:40:04.860 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.860 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.860 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:04.860 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561048+0000", 2024-09-18T04:40:04.860 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6569e-05 2024-09-18T04:40:04.861 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.861 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.861 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:04.861 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561055+0000", 2024-09-18T04:40:04.861 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.328e-06 2024-09-18T04:40:04.861 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.861 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.861 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:04.861 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561114+0000", 2024-09-18T04:40:04.861 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9540000000000003e-05 2024-09-18T04:40:04.861 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.861 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.861 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:04.861 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561314+0000", 2024-09-18T04:40:04.862 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019965799999999999 2024-09-18T04:40:04.862 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.862 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.862 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:40:04.862 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561473+0000", 2024-09-18T04:40:04.862 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000158904 2024-09-18T04:40:04.862 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.862 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.862 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:04.862 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.567015+0000", 2024-09-18T04:40:04.862 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055420859999999999 2024-09-18T04:40:04.862 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.862 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.862 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:04.863 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577982+0000", 2024-09-18T04:40:04.863 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010966872000000001 2024-09-18T04:40:04.863 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.863 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.863 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:04.863 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578162+0000", 2024-09-18T04:40:04.863 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018050999999999999 2024-09-18T04:40:04.863 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.863 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.863 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:04.863 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578200+0000", 2024-09-18T04:40:04.863 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8000999999999998e-05 2024-09-18T04:40:04.863 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.863 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:04.863 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.864 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.864 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.864 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:48 2.6 2:7c225365:ns::rbd_data.133e813ab5bc.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:04.864 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:40:04.864 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 82.220446331000005, 2024-09-18T04:40:04.864 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.024576377, 2024-09-18T04:40:04.864 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:04.864 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:04.864 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:04.864 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:40:04.864 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:04.864 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 48 2024-09-18T04:40:04.864 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.865 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:04.865 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.865 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:04.865 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:40:04.865 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.865 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.865 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.865 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:04.865 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:40:04.865 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.865 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.865 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.865 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:04.865 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.563666+0000", 2024-09-18T04:40:04.866 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:04.866 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.866 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.866 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:04.866 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570331+0000", 2024-09-18T04:40:04.866 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0066653329999999998 2024-09-18T04:40:04.866 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.866 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.866 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:04.866 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570348+0000", 2024-09-18T04:40:04.866 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6507e-05 2024-09-18T04:40:04.866 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.866 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.866 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:04.866 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570355+0000", 2024-09-18T04:40:04.867 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2749999999999998e-06 2024-09-18T04:40:04.867 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.867 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.867 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:04.867 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570396+0000", 2024-09-18T04:40:04.867 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0809000000000002e-05 2024-09-18T04:40:04.867 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.867 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.867 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:04.867 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570571+0000", 2024-09-18T04:40:04.867 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000174823 2024-09-18T04:40:04.867 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.867 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.867 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:40:04.867 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570722+0000", 2024-09-18T04:40:04.868 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015154599999999999 2024-09-18T04:40:04.868 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.868 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.868 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:04.868 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579756+0000", 2024-09-18T04:40:04.868 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0090337000000000004 2024-09-18T04:40:04.868 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.868 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.868 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:04.868 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588093+0000", 2024-09-18T04:40:04.868 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.008336553 2024-09-18T04:40:04.868 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.868 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.868 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:04.869 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588173+0000", 2024-09-18T04:40:04.869 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0333999999999994e-05 2024-09-18T04:40:04.869 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.869 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.869 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:04.869 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588244+0000", 2024-09-18T04:40:04.869 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0573000000000003e-05 2024-09-18T04:40:04.869 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.869 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:04.869 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.869 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.869 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.869 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:49 2.4 2:20f0b0c9:ns::rbd_data.133e813ab5bc.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:04.869 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.570370+0000", 2024-09-18T04:40:04.869 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 82.213744004000006, 2024-09-18T04:40:04.870 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.029633981, 2024-09-18T04:40:04.870 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:04.870 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:04.870 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:04.870 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:40:04.870 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:04.870 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 49 2024-09-18T04:40:04.870 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.870 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:04.870 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.870 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:04.870 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570370+0000", 2024-09-18T04:40:04.870 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.870 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.871 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.871 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:04.871 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570370+0000", 2024-09-18T04:40:04.871 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.871 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.871 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.871 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:04.871 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570369+0000", 2024-09-18T04:40:04.871 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:04.871 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.871 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.871 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:04.871 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579090+0000", 2024-09-18T04:40:04.871 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0087212339999999996 2024-09-18T04:40:04.871 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.872 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.872 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:04.872 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579109+0000", 2024-09-18T04:40:04.872 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9120999999999999e-05 2024-09-18T04:40:04.872 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.872 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.872 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:04.872 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579123+0000", 2024-09-18T04:40:04.872 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4382000000000001e-05 2024-09-18T04:40:04.872 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.872 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.872 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:04.872 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579221+0000", 2024-09-18T04:40:04.872 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.7503999999999993e-05 2024-09-18T04:40:04.872 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.873 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.873 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:04.873 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579553+0000", 2024-09-18T04:40:04.873 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00033254200000000002 2024-09-18T04:40:04.873 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.873 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.873 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:40:04.873 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579817+0000", 2024-09-18T04:40:04.873 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00026317099999999998 2024-09-18T04:40:04.873 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.873 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.873 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:04.873 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.599905+0000", 2024-09-18T04:40:04.873 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020088575000000001 2024-09-18T04:40:04.874 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.874 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.874 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:04.874 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.599923+0000", 2024-09-18T04:40:04.874 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7425999999999999e-05 2024-09-18T04:40:04.874 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.874 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.874 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:04.874 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.599973+0000", 2024-09-18T04:40:04.874 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0649999999999998e-05 2024-09-18T04:40:04.874 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.874 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.874 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:04.874 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.600004+0000", 2024-09-18T04:40:04.874 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0361999999999998e-05 2024-09-18T04:40:04.875 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.875 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:04.875 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.875 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.875 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.875 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:50 2.1 e51/24 2:8c706374:ns::rbd_data.133e813ab5bc.000000000000000a:head v 51'8, mlcod=51'6)", 2024-09-18T04:40:04.875 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.577029+0000", 2024-09-18T04:40:04.875 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 82.207084632999994, 2024-09-18T04:40:04.875 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.017362855, 2024-09-18T04:40:04.875 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:04.875 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:04.875 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:04.875 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.875 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:04.876 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577029+0000", 2024-09-18T04:40:04.876 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.876 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.876 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.876 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:04.876 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577029+0000", 2024-09-18T04:40:04.876 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.876 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.876 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.876 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:04.876 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577027+0000", 2024-09-18T04:40:04.876 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:04.876 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.876 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.876 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:04.877 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578880+0000", 2024-09-18T04:40:04.877 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0018532080000000001 2024-09-18T04:40:04.877 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.877 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.877 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:04.877 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578906+0000", 2024-09-18T04:40:04.877 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5573999999999999e-05 2024-09-18T04:40:04.877 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.877 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.877 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:04.877 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578920+0000", 2024-09-18T04:40:04.877 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4483e-05 2024-09-18T04:40:04.877 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.877 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.878 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:04.878 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578983+0000", 2024-09-18T04:40:04.878 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2248000000000004e-05 2024-09-18T04:40:04.878 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.878 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.878 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:04.878 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579016+0000", 2024-09-18T04:40:04.878 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3364000000000001e-05 2024-09-18T04:40:04.878 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.878 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.878 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:04.878 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594342+0000", 2024-09-18T04:40:04.878 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01532564 2024-09-18T04:40:04.878 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.879 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.879 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:04.879 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594392+0000", 2024-09-18T04:40:04.879 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0068999999999998e-05 2024-09-18T04:40:04.879 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.879 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:04.879 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.879 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.879 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.879 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:45 2.0 e51/17 2:0fbfa3dc:ns::rbd_data.133e813ab5bc.0000000000000007:head v 51'24, mlcod=51'22)", 2024-09-18T04:40:04.879 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.581892+0000", 2024-09-18T04:40:04.879 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 82.202221961000006, 2024-09-18T04:40:04.879 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015646006, 2024-09-18T04:40:04.879 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:04.879 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:04.880 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:04.880 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.880 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:04.880 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581892+0000", 2024-09-18T04:40:04.880 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.880 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.880 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.880 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:04.880 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581892+0000", 2024-09-18T04:40:04.880 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.880 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.880 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.880 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:04.880 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581890+0000", 2024-09-18T04:40:04.881 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:04.881 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.881 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.881 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:04.881 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586188+0000", 2024-09-18T04:40:04.881 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0042985769999999996 2024-09-18T04:40:04.881 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.881 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.881 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:04.881 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586210+0000", 2024-09-18T04:40:04.881 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1069000000000001e-05 2024-09-18T04:40:04.881 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.881 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.881 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:04.882 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586221+0000", 2024-09-18T04:40:04.882 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1287000000000001e-05 2024-09-18T04:40:04.882 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.882 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.882 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:04.882 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586313+0000", 2024-09-18T04:40:04.882 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1935999999999995e-05 2024-09-18T04:40:04.882 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.882 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.882 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:04.882 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586368+0000", 2024-09-18T04:40:04.882 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5578000000000002e-05 2024-09-18T04:40:04.882 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.882 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.883 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:04.883 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.597473+0000", 2024-09-18T04:40:04.883 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011105139999999999 2024-09-18T04:40:04.883 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.883 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.883 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:04.883 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.597538+0000", 2024-09-18T04:40:04.883 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4133000000000004e-05 2024-09-18T04:40:04.883 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.883 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:04.883 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.883 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.883 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.883 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:53 2.4 2:39f7f174:ns::rbd_data.133e813ab5bc.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:04.884 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.584263+0000", 2024-09-18T04:40:04.884 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 82.199850087000002, 2024-09-18T04:40:04.884 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018701180000000001, 2024-09-18T04:40:04.884 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:04.884 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:04.884 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:04.884 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:40:04.884 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:04.884 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 53 2024-09-18T04:40:04.884 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.884 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:04.884 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.884 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:04.884 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584263+0000", 2024-09-18T04:40:04.885 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.885 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.885 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.885 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:04.885 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584263+0000", 2024-09-18T04:40:04.885 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.885 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.885 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.885 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:04.885 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584262+0000", 2024-09-18T04:40:04.885 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:04.885 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.885 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.885 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:04.886 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.591617+0000", 2024-09-18T04:40:04.886 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0073550600000000001 2024-09-18T04:40:04.886 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.886 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.886 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:04.886 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.591637+0000", 2024-09-18T04:40:04.886 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0432999999999999e-05 2024-09-18T04:40:04.886 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.886 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.886 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:04.886 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.591646+0000", 2024-09-18T04:40:04.886 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7849999999999997e-06 2024-09-18T04:40:04.886 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.886 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.886 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:04.887 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593478+0000", 2024-09-18T04:40:04.887 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001831495 2024-09-18T04:40:04.887 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.887 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.887 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:04.887 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593661+0000", 2024-09-18T04:40:04.887 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018364999999999999 2024-09-18T04:40:04.887 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.887 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.887 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:40:04.887 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593834+0000", 2024-09-18T04:40:04.887 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000173065 2024-09-18T04:40:04.887 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.887 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.888 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:04.888 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.600670+0000", 2024-09-18T04:40:04.888 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0068354970000000003 2024-09-18T04:40:04.888 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.888 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.888 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:04.888 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602851+0000", 2024-09-18T04:40:04.888 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002181653 2024-09-18T04:40:04.888 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.888 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.888 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:04.888 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602930+0000", 2024-09-18T04:40:04.888 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8045000000000002e-05 2024-09-18T04:40:04.888 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.888 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.889 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:04.889 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602965+0000", 2024-09-18T04:40:04.889 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5102999999999999e-05 2024-09-18T04:40:04.889 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.889 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:04.889 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.889 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.889 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.889 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:56 2.6 2:72453c3e:ns::rbd_data.133e813ab5bc.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:04.889 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.595988+0000", 2024-09-18T04:40:04.889 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 82.188125685000003, 2024-09-18T04:40:04.889 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.030296172, 2024-09-18T04:40:04.889 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:04.889 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:04.890 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:04.890 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:40:04.890 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:04.890 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 56 2024-09-18T04:40:04.890 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.890 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:04.890 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.890 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:04.890 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.595988+0000", 2024-09-18T04:40:04.890 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.890 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.890 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.890 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:04.890 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.595988+0000", 2024-09-18T04:40:04.890 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:04.891 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.891 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.891 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:04.891 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.595986+0000", 2024-09-18T04:40:04.891 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:40:04.891 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.891 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.891 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:04.891 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605262+0000", 2024-09-18T04:40:04.891 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0092767240000000001 2024-09-18T04:40:04.891 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.891 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.891 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:04.891 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605282+0000", 2024-09-18T04:40:04.892 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9307e-05 2024-09-18T04:40:04.892 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.892 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.892 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:04.892 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605292+0000", 2024-09-18T04:40:04.892 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9510000000000001e-06 2024-09-18T04:40:04.892 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.892 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.892 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:04.892 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605369+0000", 2024-09-18T04:40:04.892 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7846e-05 2024-09-18T04:40:04.892 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.892 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.892 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:04.893 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605668+0000", 2024-09-18T04:40:04.893 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00029896099999999999 2024-09-18T04:40:04.893 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.893 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.893 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:40:04.893 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605906+0000", 2024-09-18T04:40:04.893 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000237758 2024-09-18T04:40:04.893 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.893 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.893 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:04.893 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.613532+0000", 2024-09-18T04:40:04.893 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0076260590000000001 2024-09-18T04:40:04.893 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.893 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.894 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:04.894 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.626116+0000", 2024-09-18T04:40:04.894 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012584092999999999 2024-09-18T04:40:04.894 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.894 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.894 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:04.894 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.626216+0000", 2024-09-18T04:40:04.894 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9630999999999995e-05 2024-09-18T04:40:04.894 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:04.894 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:04.894 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:04.894 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.626284+0000", 2024-09-18T04:40:04.894 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8034999999999995e-05 2024-09-18T04:40:04.894 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.894 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:04.895 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.895 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:04.895 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:04.895 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:40:04.896 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:40:04.897 INFO:teuthology.orchestra.run.smithi028.stderr:marked in osd.4. 2024-09-18T04:40:04.897 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-09-18T04:40:04.898 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:40:04.891+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:40:04.998 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:40:04.992+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:40:05.013 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:40:05.013 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:40:05.013 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:40:05.013 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:40:05.024 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:40:05.099 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:40:05.096+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:40:05.129 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:40:05.129 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:40:05.129 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:40:05.129 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:40:05.129 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:40:05.139 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:40:05.200 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:40:05.196+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:40:05.247 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:40:05.248 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:40:05.248 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:40:05.248 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:40:05.248 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.248 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:40:05.248 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:40:05.248 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 157.45431323599999, 2024-09-18T04:40:05.248 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017399218000000001, 2024-09-18T04:40:05.249 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:05.249 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:05.249 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:05.249 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:40:05.249 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:40:05.249 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 114 2024-09-18T04:40:05.249 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.249 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:05.249 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.249 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:05.249 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:40:05.249 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:05.249 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.250 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.250 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:05.250 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:40:05.250 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:05.250 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.250 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.250 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:05.250 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786900+0000", 2024-09-18T04:40:05.250 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:40:05.250 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.250 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.250 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:05.250 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786908+0000", 2024-09-18T04:40:05.250 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8199999999999997e-06 2024-09-18T04:40:05.251 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.251 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.251 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:05.251 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786925+0000", 2024-09-18T04:40:05.251 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6849000000000001e-05 2024-09-18T04:40:05.251 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.251 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.251 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:05.251 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786934+0000", 2024-09-18T04:40:05.251 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6299999999999993e-06 2024-09-18T04:40:05.251 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.251 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.252 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:05.252 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800926+0000", 2024-09-18T04:40:05.252 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013991632 2024-09-18T04:40:05.252 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.252 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.253 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:05.253 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801060+0000", 2024-09-18T04:40:05.253 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013378199999999999 2024-09-18T04:40:05.253 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.253 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.253 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:40:05.253 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801166+0000", 2024-09-18T04:40:05.253 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010607099999999999 2024-09-18T04:40:05.253 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.253 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.253 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:05.253 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803026+0000", 2024-09-18T04:40:05.254 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001860469 2024-09-18T04:40:05.254 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.254 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.254 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:05.254 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804245+0000", 2024-09-18T04:40:05.254 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0012185900000000001 2024-09-18T04:40:05.254 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.254 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.254 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:05.254 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804278+0000", 2024-09-18T04:40:05.254 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3272000000000003e-05 2024-09-18T04:40:05.254 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.255 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.255 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:05.255 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804300+0000", 2024-09-18T04:40:05.255 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2266e-05 2024-09-18T04:40:05.255 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:05.255 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:05.255 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:05.255 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.255 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.255 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:40:05.255 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:40:05.255 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 157.45340033799999, 2024-09-18T04:40:05.256 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017521662, 2024-09-18T04:40:05.256 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:05.256 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:05.256 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:05.256 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:40:05.256 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:40:05.256 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 116 2024-09-18T04:40:05.256 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.256 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:05.256 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.256 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:05.256 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:40:05.256 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:05.257 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.257 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.257 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:05.257 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:40:05.257 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:05.257 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.257 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.257 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:05.257 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787813+0000", 2024-09-18T04:40:05.257 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:05.257 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.257 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.257 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:05.257 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787824+0000", 2024-09-18T04:40:05.258 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1157e-05 2024-09-18T04:40:05.258 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.258 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.258 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:05.258 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787840+0000", 2024-09-18T04:40:05.258 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6269e-05 2024-09-18T04:40:05.258 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.258 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.258 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:05.258 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787853+0000", 2024-09-18T04:40:05.258 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3156000000000001e-05 2024-09-18T04:40:05.259 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.259 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.259 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:05.259 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803110+0000", 2024-09-18T04:40:05.259 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015256736 2024-09-18T04:40:05.259 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.259 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.259 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:05.259 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803197+0000", 2024-09-18T04:40:05.260 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7195999999999997e-05 2024-09-18T04:40:05.260 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.260 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.260 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:40:05.260 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803308+0000", 2024-09-18T04:40:05.260 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111005 2024-09-18T04:40:05.260 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.260 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.260 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:05.260 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804316+0000", 2024-09-18T04:40:05.260 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010073949999999999 2024-09-18T04:40:05.261 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.261 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.261 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:05.261 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805232+0000", 2024-09-18T04:40:05.261 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091636300000000001 2024-09-18T04:40:05.261 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.261 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.261 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:05.262 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805277+0000", 2024-09-18T04:40:05.262 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4864000000000003e-05 2024-09-18T04:40:05.262 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.262 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.262 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:05.262 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805336+0000", 2024-09-18T04:40:05.262 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8968000000000002e-05 2024-09-18T04:40:05.262 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:05.262 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:05.262 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:05.262 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.262 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.262 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:40:05.263 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:40:05.263 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 157.453348283, 2024-09-18T04:40:05.263 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017983453, 2024-09-18T04:40:05.263 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:05.263 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:05.263 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:05.263 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:40:05.263 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:40:05.263 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 117 2024-09-18T04:40:05.263 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.263 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:05.263 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.264 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:05.264 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:40:05.265 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:05.265 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.265 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.265 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:05.265 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:40:05.265 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:05.265 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.265 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.265 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:05.266 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787865+0000", 2024-09-18T04:40:05.266 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:40:05.266 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.266 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.266 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:05.266 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787889+0000", 2024-09-18T04:40:05.266 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4244999999999999e-05 2024-09-18T04:40:05.266 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.266 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.266 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:05.266 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787905+0000", 2024-09-18T04:40:05.266 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5739999999999998e-05 2024-09-18T04:40:05.266 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.266 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.267 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:05.267 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787918+0000", 2024-09-18T04:40:05.267 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.252e-05 2024-09-18T04:40:05.267 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.267 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.267 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:05.267 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803693+0000", 2024-09-18T04:40:05.267 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015774872999999998 2024-09-18T04:40:05.267 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.267 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.267 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:05.268 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803775+0000", 2024-09-18T04:40:05.268 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2761000000000003e-05 2024-09-18T04:40:05.268 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.268 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.268 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:40:05.269 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803887+0000", 2024-09-18T04:40:05.269 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111708 2024-09-18T04:40:05.269 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.269 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.269 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:05.269 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804823+0000", 2024-09-18T04:40:05.269 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00093625100000000003 2024-09-18T04:40:05.269 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.269 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.269 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:05.269 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805804+0000", 2024-09-18T04:40:05.269 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00098114699999999992 2024-09-18T04:40:05.269 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.269 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.270 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:05.270 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805831+0000", 2024-09-18T04:40:05.270 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6401999999999999e-05 2024-09-18T04:40:05.270 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.270 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.270 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:05.270 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805850+0000", 2024-09-18T04:40:05.270 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8862e-05 2024-09-18T04:40:05.270 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:05.270 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:05.270 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:05.270 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.270 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.271 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:36 2.7 2:fe1b02ad:::rbd_data.125fc09df6d1.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:05.271 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:40:05.271 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 82.975492173000006, 2024-09-18T04:40:05.271 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.040387804999999999, 2024-09-18T04:40:05.271 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:05.271 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:05.271 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:05.271 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:40:05.271 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:40:05.271 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 36 2024-09-18T04:40:05.271 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.271 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:05.271 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.272 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:05.272 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:40:05.272 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:05.272 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.272 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.272 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:05.272 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:40:05.272 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:05.272 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.272 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.272 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:05.272 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265720+0000", 2024-09-18T04:40:05.272 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:40:05.272 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.273 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.273 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:05.273 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272590+0000", 2024-09-18T04:40:05.273 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.006870508 2024-09-18T04:40:05.273 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.273 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.273 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:05.273 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272623+0000", 2024-09-18T04:40:05.273 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2716000000000003e-05 2024-09-18T04:40:05.273 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.273 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.273 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:05.273 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272637+0000", 2024-09-18T04:40:05.274 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4175999999999999e-05 2024-09-18T04:40:05.274 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.274 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.274 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:05.274 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272779+0000", 2024-09-18T04:40:05.274 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000141392 2024-09-18T04:40:05.274 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.274 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.274 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:05.274 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273031+0000", 2024-09-18T04:40:05.274 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00025249099999999998 2024-09-18T04:40:05.275 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.275 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.275 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:40:05.275 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273219+0000", 2024-09-18T04:40:05.275 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000187531 2024-09-18T04:40:05.275 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.275 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.275 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:05.275 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.302806+0000", 2024-09-18T04:40:05.275 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.029587553999999999 2024-09-18T04:40:05.276 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.276 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.276 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:05.276 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306004+0000", 2024-09-18T04:40:05.276 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0031982629999999998 2024-09-18T04:40:05.276 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.276 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.276 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:05.276 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306071+0000", 2024-09-18T04:40:05.276 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6829999999999995e-05 2024-09-18T04:40:05.277 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.277 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.277 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:05.277 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306110+0000", 2024-09-18T04:40:05.277 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8874000000000001e-05 2024-09-18T04:40:05.277 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:05.277 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:05.277 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:05.277 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.278 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.278 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:38 2.0 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:05.278 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:40:05.278 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 82.968538636000005, 2024-09-18T04:40:05.278 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.033754348000000003, 2024-09-18T04:40:05.278 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:05.278 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:05.280 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:05.280 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:40:05.280 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:40:05.280 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 38 2024-09-18T04:40:05.280 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.280 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:05.281 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.281 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:05.281 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:40:05.281 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:05.281 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.281 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.281 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:05.281 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:40:05.281 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:05.281 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.281 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.282 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:05.282 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272673+0000", 2024-09-18T04:40:05.282 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:40:05.282 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.282 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.282 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:05.282 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277166+0000", 2024-09-18T04:40:05.282 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044922249999999999 2024-09-18T04:40:05.282 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.282 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.282 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:05.282 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277189+0000", 2024-09-18T04:40:05.283 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3283000000000001e-05 2024-09-18T04:40:05.283 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.283 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.283 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:05.283 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277205+0000", 2024-09-18T04:40:05.283 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6305000000000001e-05 2024-09-18T04:40:05.283 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.283 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.283 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:05.283 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277290+0000", 2024-09-18T04:40:05.283 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5135000000000004e-05 2024-09-18T04:40:05.283 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.284 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.284 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:05.284 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277531+0000", 2024-09-18T04:40:05.284 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00024014699999999999 2024-09-18T04:40:05.284 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.284 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.284 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:05.284 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277793+0000", 2024-09-18T04:40:05.284 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00026224499999999998 2024-09-18T04:40:05.284 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.284 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.285 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:05.285 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303990+0000", 2024-09-18T04:40:05.285 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.026197295999999998 2024-09-18T04:40:05.285 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.285 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.285 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:05.285 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306340+0000", 2024-09-18T04:40:05.285 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.002349524 2024-09-18T04:40:05.285 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.285 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.285 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:05.286 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306394+0000", 2024-09-18T04:40:05.286 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3934999999999998e-05 2024-09-18T04:40:05.286 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.286 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.286 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:05.286 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306430+0000", 2024-09-18T04:40:05.286 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6690000000000003e-05 2024-09-18T04:40:05.286 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:05.286 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:05.286 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:05.286 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.287 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.287 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:39 2.7 2:e9c909c5:::rbd_data.125fc09df6d1.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:05.287 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:40:05.287 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 82.963979030999994, 2024-09-18T04:40:05.287 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.029110673, 2024-09-18T04:40:05.287 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:05.287 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:05.287 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:05.288 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:40:05.289 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:40:05.289 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 39 2024-09-18T04:40:05.289 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.289 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:05.289 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.289 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:05.289 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:40:05.289 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:05.289 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.290 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.290 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:05.290 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:40:05.290 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:05.290 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.290 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.290 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:05.290 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277234+0000", 2024-09-18T04:40:05.290 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:05.290 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.290 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.291 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:05.291 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281351+0000", 2024-09-18T04:40:05.291 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0041170920000000001 2024-09-18T04:40:05.291 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.291 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.291 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:05.291 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281375+0000", 2024-09-18T04:40:05.291 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.446e-05 2024-09-18T04:40:05.291 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.291 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.291 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:05.292 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281390+0000", 2024-09-18T04:40:05.292 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4309e-05 2024-09-18T04:40:05.292 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.292 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.292 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:05.292 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.282306+0000", 2024-09-18T04:40:05.292 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091662000000000004 2024-09-18T04:40:05.292 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.292 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.292 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:05.292 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.282613+0000", 2024-09-18T04:40:05.292 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00030663799999999997 2024-09-18T04:40:05.292 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.293 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.293 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:40:05.293 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283001+0000", 2024-09-18T04:40:05.293 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00038801599999999998 2024-09-18T04:40:05.293 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.293 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.293 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:05.293 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306239+0000", 2024-09-18T04:40:05.293 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023238466999999999 2024-09-18T04:40:05.293 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.293 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.294 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:05.294 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306259+0000", 2024-09-18T04:40:05.294 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9837999999999999e-05 2024-09-18T04:40:05.294 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.294 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.294 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:05.294 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306299+0000", 2024-09-18T04:40:05.294 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0194000000000002e-05 2024-09-18T04:40:05.294 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.294 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.295 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:05.295 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306346+0000", 2024-09-18T04:40:05.295 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6845000000000003e-05 2024-09-18T04:40:05.295 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:05.295 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:05.295 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:05.295 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.295 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.295 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:46 2.6 e51/17 2:7ba25245:::rbd_data.125fc09df6d1.0000000000000006:head v 51'2, mlcod=0'0)", 2024-09-18T04:40:05.295 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:40:05.295 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 82.928057898999995, 2024-09-18T04:40:05.295 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019843098, 2024-09-18T04:40:05.295 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:05.295 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:05.296 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:05.296 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.296 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:05.296 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:40:05.296 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:05.296 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.296 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.296 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:05.296 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:40:05.296 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:05.296 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.296 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.296 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:05.297 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313156+0000", 2024-09-18T04:40:05.297 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:40:05.297 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.297 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.297 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:05.297 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320144+0000", 2024-09-18T04:40:05.297 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.006988345 2024-09-18T04:40:05.297 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.297 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.297 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:05.297 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320175+0000", 2024-09-18T04:40:05.297 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.0861999999999997e-05 2024-09-18T04:40:05.297 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.298 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.298 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:05.298 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320188+0000", 2024-09-18T04:40:05.298 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2714999999999999e-05 2024-09-18T04:40:05.298 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.298 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.298 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:05.298 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320267+0000", 2024-09-18T04:40:05.298 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9586999999999995e-05 2024-09-18T04:40:05.298 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.298 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.298 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:05.298 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320311+0000", 2024-09-18T04:40:05.298 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3529000000000003e-05 2024-09-18T04:40:05.298 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.299 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.299 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:05.299 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.332949+0000", 2024-09-18T04:40:05.299 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012638267999999999 2024-09-18T04:40:05.299 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.299 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.299 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:05.299 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.333000+0000", 2024-09-18T04:40:05.299 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0736999999999999e-05 2024-09-18T04:40:05.299 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:05.299 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:05.299 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:05.299 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.300 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.300 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:57 2.0 2:0cdd070e:::rbd_data.125fc09df6d1.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:05.300 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:40:05.300 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 82.892200324000001, 2024-09-18T04:40:05.300 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.033329974999999998, 2024-09-18T04:40:05.300 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:05.300 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:05.300 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:05.300 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:40:05.300 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:40:05.300 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 57 2024-09-18T04:40:05.300 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.300 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:05.301 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.301 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:05.301 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:40:05.301 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:05.301 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.301 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.301 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:05.301 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:40:05.301 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:05.301 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.301 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.301 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:05.302 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349013+0000", 2024-09-18T04:40:05.302 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:05.302 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.302 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.302 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:05.302 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356138+0000", 2024-09-18T04:40:05.302 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071258010000000002 2024-09-18T04:40:05.302 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.302 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.302 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:05.302 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356168+0000", 2024-09-18T04:40:05.302 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9679000000000002e-05 2024-09-18T04:40:05.302 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.303 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.303 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:05.303 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356181+0000", 2024-09-18T04:40:05.303 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3164e-05 2024-09-18T04:40:05.303 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.303 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.303 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:05.303 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356254+0000", 2024-09-18T04:40:05.303 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3182999999999994e-05 2024-09-18T04:40:05.303 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.303 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.303 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:05.303 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356453+0000", 2024-09-18T04:40:05.303 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019876599999999999 2024-09-18T04:40:05.304 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.304 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.304 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:05.304 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356627+0000", 2024-09-18T04:40:05.304 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017340199999999999 2024-09-18T04:40:05.304 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.304 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.304 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:05.304 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.369200+0000", 2024-09-18T04:40:05.304 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012573302 2024-09-18T04:40:05.304 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.304 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.304 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:05.304 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382262+0000", 2024-09-18T04:40:05.305 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013062549 2024-09-18T04:40:05.305 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.305 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.305 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:05.305 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382310+0000", 2024-09-18T04:40:05.305 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7366e-05 2024-09-18T04:40:05.305 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.305 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.305 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:05.305 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382344+0000", 2024-09-18T04:40:05.305 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4319999999999997e-05 2024-09-18T04:40:05.305 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:05.305 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:05.306 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:05.306 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.306 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.306 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:59 2.0 2:19c52cde:::rbd_data.125fc09df6d1.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:05.306 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:40:05.306 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 82.885009159000006, 2024-09-18T04:40:05.306 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.027137247999999999, 2024-09-18T04:40:05.306 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:05.306 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:05.306 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:05.306 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:40:05.306 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:40:05.306 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 59 2024-09-18T04:40:05.306 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.307 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:05.307 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.307 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:05.307 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:40:05.307 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:05.307 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.307 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.307 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:05.307 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:40:05.307 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:05.307 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.307 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.308 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:05.308 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356204+0000", 2024-09-18T04:40:05.308 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:05.308 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.308 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.308 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:05.308 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362661+0000", 2024-09-18T04:40:05.308 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0064565129999999997 2024-09-18T04:40:05.308 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.308 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.309 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:05.309 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362684+0000", 2024-09-18T04:40:05.309 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3688000000000001e-05 2024-09-18T04:40:05.309 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.309 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.309 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:05.309 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362699+0000", 2024-09-18T04:40:05.309 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4798e-05 2024-09-18T04:40:05.309 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.309 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.309 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:05.310 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363016+0000", 2024-09-18T04:40:05.310 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031737300000000003 2024-09-18T04:40:05.310 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.310 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.310 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:05.310 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363201+0000", 2024-09-18T04:40:05.310 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000184131 2024-09-18T04:40:05.310 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.310 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.310 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:05.310 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363343+0000", 2024-09-18T04:40:05.311 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014236699999999999 2024-09-18T04:40:05.311 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.311 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.311 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:05.311 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.372189+0000", 2024-09-18T04:40:05.311 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0088464819999999993 2024-09-18T04:40:05.311 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.311 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.311 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:05.311 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383269+0000", 2024-09-18T04:40:05.311 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011079594 2024-09-18T04:40:05.311 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.312 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.312 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:05.312 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383314+0000", 2024-09-18T04:40:05.312 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4932000000000003e-05 2024-09-18T04:40:05.312 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.312 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.312 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:05.312 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383343+0000", 2024-09-18T04:40:05.312 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8600999999999999e-05 2024-09-18T04:40:05.312 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:05.313 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:05.313 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:05.313 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.313 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.313 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:40 2.6 e51/17 2:784d6cd1:ns::rbd_data.133e813ab5bc.0000000000000005:head v 51'7, mlcod=51'5)", 2024-09-18T04:40:05.313 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:40:05.313 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 82.688759425000001, 2024-09-18T04:40:05.313 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020497496, 2024-09-18T04:40:05.313 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:05.313 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:05.313 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:05.313 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.314 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:05.314 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:40:05.314 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:05.314 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.314 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.314 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:05.314 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:40:05.314 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:05.314 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.314 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.314 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:05.314 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552454+0000", 2024-09-18T04:40:05.314 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:05.315 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.315 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.315 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:05.315 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556448+0000", 2024-09-18T04:40:05.315 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0039940990000000001 2024-09-18T04:40:05.315 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.315 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.315 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:05.315 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556481+0000", 2024-09-18T04:40:05.315 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3778000000000001e-05 2024-09-18T04:40:05.315 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.315 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.316 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:05.316 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556493+0000", 2024-09-18T04:40:05.316 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1622e-05 2024-09-18T04:40:05.316 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.316 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.316 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:05.316 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556556+0000", 2024-09-18T04:40:05.316 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2582999999999994e-05 2024-09-18T04:40:05.316 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.316 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.317 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:05.317 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556592+0000", 2024-09-18T04:40:05.317 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5880999999999997e-05 2024-09-18T04:40:05.317 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.317 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.317 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:05.317 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.572786+0000", 2024-09-18T04:40:05.317 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016194571000000001 2024-09-18T04:40:05.317 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.318 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.318 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:05.318 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.572953+0000", 2024-09-18T04:40:05.318 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000166436 2024-09-18T04:40:05.318 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:05.318 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:05.318 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:05.318 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.318 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.319 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:45 2.0 2:0fbfa3dc:ns::rbd_data.133e813ab5bc.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:05.319 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:40:05.319 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 82.677545125999998, 2024-09-18T04:40:05.319 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.036558168000000002, 2024-09-18T04:40:05.319 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:05.319 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:05.319 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:05.319 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:40:05.319 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:05.319 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 45 2024-09-18T04:40:05.319 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.319 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:05.320 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.320 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:05.320 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:40:05.320 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:05.320 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.320 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.320 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:05.320 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:40:05.320 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:05.320 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.321 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.321 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:05.321 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:40:05.321 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:40:05.321 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.321 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.321 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:05.322 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578594+0000", 2024-09-18T04:40:05.322 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014927147999999999 2024-09-18T04:40:05.322 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.322 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.322 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:05.322 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578612+0000", 2024-09-18T04:40:05.322 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7623999999999999e-05 2024-09-18T04:40:05.322 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.322 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.322 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:05.322 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578620+0000", 2024-09-18T04:40:05.323 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9070000000000007e-06 2024-09-18T04:40:05.323 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.323 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.323 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:05.323 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578702+0000", 2024-09-18T04:40:05.323 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2349999999999996e-05 2024-09-18T04:40:05.323 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.323 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.323 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:05.323 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.579028+0000", 2024-09-18T04:40:05.323 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032622899999999998 2024-09-18T04:40:05.323 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.323 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.324 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:05.324 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.579343+0000", 2024-09-18T04:40:05.324 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031448899999999999 2024-09-18T04:40:05.324 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.324 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.324 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:05.324 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.590078+0000", 2024-09-18T04:40:05.324 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010734946 2024-09-18T04:40:05.324 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.324 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.324 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:05.325 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600089+0000", 2024-09-18T04:40:05.325 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010011195000000001 2024-09-18T04:40:05.325 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.325 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.325 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:05.325 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600177+0000", 2024-09-18T04:40:05.325 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8425999999999997e-05 2024-09-18T04:40:05.325 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.325 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.325 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:05.325 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600228+0000", 2024-09-18T04:40:05.326 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0216999999999997e-05 2024-09-18T04:40:05.326 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:05.326 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:05.328 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:05.328 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.328 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.328 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:54 2.0 2:04f17ef6:ns::rbd_data.133e813ab5bc.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:05.329 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:40:05.329 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 82.655973289000002, 2024-09-18T04:40:05.329 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.024612999, 2024-09-18T04:40:05.329 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:05.329 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:05.329 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:05.329 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:40:05.329 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:05.329 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 54 2024-09-18T04:40:05.329 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.329 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:05.329 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.330 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:05.330 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:40:05.330 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:05.330 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.330 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.330 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:05.330 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:40:05.330 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:05.330 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.330 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.330 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:05.330 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585239+0000", 2024-09-18T04:40:05.330 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:05.330 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.331 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.331 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:05.331 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593702+0000", 2024-09-18T04:40:05.331 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0084632990000000005 2024-09-18T04:40:05.331 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.331 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.331 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:05.331 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593719+0000", 2024-09-18T04:40:05.331 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.664e-05 2024-09-18T04:40:05.331 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.331 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.331 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:05.331 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593727+0000", 2024-09-18T04:40:05.331 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6839999999999996e-06 2024-09-18T04:40:05.331 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.332 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.332 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:05.332 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593767+0000", 2024-09-18T04:40:05.332 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0330999999999997e-05 2024-09-18T04:40:05.332 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.332 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.332 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:05.332 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593905+0000", 2024-09-18T04:40:05.332 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013754800000000001 2024-09-18T04:40:05.332 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.332 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.332 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:05.332 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.594013+0000", 2024-09-18T04:40:05.333 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000108805 2024-09-18T04:40:05.333 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.333 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.333 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:05.333 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604678+0000", 2024-09-18T04:40:05.333 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010664391 2024-09-18T04:40:05.333 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.333 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.333 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:05.333 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609662+0000", 2024-09-18T04:40:05.333 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0049846100000000004 2024-09-18T04:40:05.333 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.334 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.334 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:05.334 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609788+0000", 2024-09-18T04:40:05.334 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012518800000000001 2024-09-18T04:40:05.334 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.334 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.334 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:05.334 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609854+0000", 2024-09-18T04:40:05.334 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6556000000000004e-05 2024-09-18T04:40:05.334 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:05.334 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:05.334 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:05.334 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.334 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.335 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:58 2.7 2:fc7c6cd3:ns::rbd_data.133e813ab5bc.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:05.335 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:40:05.335 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 82.643130859999999, 2024-09-18T04:40:05.335 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.027690566999999999, 2024-09-18T04:40:05.335 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:05.335 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:05.335 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:05.335 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:40:05.335 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:05.335 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 58 2024-09-18T04:40:05.335 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.335 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:05.336 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.336 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:05.336 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:40:05.336 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:05.336 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.336 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.336 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:05.336 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:40:05.336 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:05.336 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.336 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.336 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:05.336 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598082+0000", 2024-09-18T04:40:05.336 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:05.337 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.337 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.337 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:05.337 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604832+0000", 2024-09-18T04:40:05.337 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0067504510000000002 2024-09-18T04:40:05.337 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.337 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.337 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:05.337 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604852+0000", 2024-09-18T04:40:05.337 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0228999999999999e-05 2024-09-18T04:40:05.337 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.337 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.338 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:05.338 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604870+0000", 2024-09-18T04:40:05.338 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7952e-05 2024-09-18T04:40:05.338 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.338 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.338 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:05.338 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604936+0000", 2024-09-18T04:40:05.338 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5431000000000004e-05 2024-09-18T04:40:05.338 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.338 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.338 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:05.338 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605085+0000", 2024-09-18T04:40:05.338 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014910300000000001 2024-09-18T04:40:05.339 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.339 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.339 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:40:05.339 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605205+0000", 2024-09-18T04:40:05.339 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000119887 2024-09-18T04:40:05.339 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.339 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.339 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:05.339 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625586+0000", 2024-09-18T04:40:05.339 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020381399000000001 2024-09-18T04:40:05.339 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.340 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.340 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:05.340 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625628+0000", 2024-09-18T04:40:05.340 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1873999999999999e-05 2024-09-18T04:40:05.340 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.340 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.340 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:05.340 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625722+0000", 2024-09-18T04:40:05.340 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3787000000000002e-05 2024-09-18T04:40:05.340 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.340 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.340 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:05.340 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625774+0000", 2024-09-18T04:40:05.341 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2503999999999997e-05 2024-09-18T04:40:05.341 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:05.341 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:05.341 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:05.341 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.341 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.341 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:56 2.6 e51/17 2:72453c3e:ns::rbd_data.133e813ab5bc.000000000000000d:head v 51'10, mlcod=51'8)", 2024-09-18T04:40:05.341 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:40:05.341 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 82.635933752, 2024-09-18T04:40:05.341 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019766552999999999, 2024-09-18T04:40:05.341 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:05.341 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:05.342 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:05.342 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.342 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:05.342 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:40:05.342 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:05.342 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.342 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.342 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:05.342 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:40:05.342 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:05.342 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.342 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.343 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:05.343 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605279+0000", 2024-09-18T04:40:05.343 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:05.343 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.343 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.343 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:05.343 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609417+0000", 2024-09-18T04:40:05.343 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0041378059999999999 2024-09-18T04:40:05.343 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.343 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.343 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:05.343 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609443+0000", 2024-09-18T04:40:05.344 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6098e-05 2024-09-18T04:40:05.344 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.344 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.344 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:05.344 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609458+0000", 2024-09-18T04:40:05.344 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4671000000000001e-05 2024-09-18T04:40:05.344 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.344 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.344 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:05.344 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609553+0000", 2024-09-18T04:40:05.344 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5265000000000004e-05 2024-09-18T04:40:05.345 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.345 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.345 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:05.345 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609583+0000", 2024-09-18T04:40:05.345 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9839e-05 2024-09-18T04:40:05.345 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.345 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.345 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:05.345 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625004+0000", 2024-09-18T04:40:05.345 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015420639999999999 2024-09-18T04:40:05.345 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.345 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.346 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:05.346 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625047+0000", 2024-09-18T04:40:05.346 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3665999999999998e-05 2024-09-18T04:40:05.346 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:05.346 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:05.346 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:05.346 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.346 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.346 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:99 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:40:05.346 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.582718+0000", 2024-09-18T04:40:05.346 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 68.658496842999995, 2024-09-18T04:40:05.346 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019362575, 2024-09-18T04:40:05.347 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:05.347 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:05.347 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:05.347 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:40:05.347 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:40:05.347 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 99 2024-09-18T04:40:05.347 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.347 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:05.348 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.348 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:05.348 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582718+0000", 2024-09-18T04:40:05.348 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:05.348 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.348 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.348 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:05.349 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582718+0000", 2024-09-18T04:40:05.349 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:05.349 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.349 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.349 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:05.349 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582717+0000", 2024-09-18T04:40:05.349 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:40:05.349 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.349 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.349 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:05.349 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582720+0000", 2024-09-18T04:40:05.349 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2239999999999998e-06 2024-09-18T04:40:05.350 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.350 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.350 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:05.350 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582723+0000", 2024-09-18T04:40:05.350 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.258e-06 2024-09-18T04:40:05.350 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.350 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.350 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:05.350 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582726+0000", 2024-09-18T04:40:05.350 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8870000000000001e-06 2024-09-18T04:40:05.351 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.351 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.351 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:05.351 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.592018+0000", 2024-09-18T04:40:05.351 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0092926420000000003 2024-09-18T04:40:05.351 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.351 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.351 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:05.351 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.592214+0000", 2024-09-18T04:40:05.351 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000195267 2024-09-18T04:40:05.351 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.352 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.352 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:05.352 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.592913+0000", 2024-09-18T04:40:05.352 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00069924600000000005 2024-09-18T04:40:05.352 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.352 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.352 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:05.352 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597493+0000", 2024-09-18T04:40:05.352 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045797629999999997 2024-09-18T04:40:05.352 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.352 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.353 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:05.353 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.602009+0000", 2024-09-18T04:40:05.353 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045159559999999998 2024-09-18T04:40:05.353 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.353 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.353 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:05.353 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.602042+0000", 2024-09-18T04:40:05.353 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3084999999999999e-05 2024-09-18T04:40:05.353 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.353 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.353 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:05.353 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.602080+0000", 2024-09-18T04:40:05.354 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8600000000000003e-05 2024-09-18T04:40:05.354 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:05.354 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:05.354 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:05.354 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.354 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.354 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:104 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:40:05.354 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.583165+0000", 2024-09-18T04:40:05.354 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 68.658049544999997, 2024-09-18T04:40:05.354 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023679603, 2024-09-18T04:40:05.354 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:05.354 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:05.355 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:05.355 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:40:05.355 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:40:05.355 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 104 2024-09-18T04:40:05.355 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.355 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:05.355 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.355 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:05.355 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583165+0000", 2024-09-18T04:40:05.355 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:05.355 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.355 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.356 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:05.356 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583165+0000", 2024-09-18T04:40:05.356 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:05.356 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.356 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.356 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:05.356 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583164+0000", 2024-09-18T04:40:05.357 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:40:05.357 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.357 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.357 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:05.357 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583167+0000", 2024-09-18T04:40:05.357 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9210000000000001e-06 2024-09-18T04:40:05.357 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.357 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.357 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:05.357 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583171+0000", 2024-09-18T04:40:05.357 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9759999999999997e-06 2024-09-18T04:40:05.357 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.358 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.358 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:05.358 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583174+0000", 2024-09-18T04:40:05.358 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9960000000000002e-06 2024-09-18T04:40:05.358 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.358 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.358 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:05.358 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.594373+0000", 2024-09-18T04:40:05.358 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011198665999999999 2024-09-18T04:40:05.358 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.358 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.358 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:05.358 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.594434+0000", 2024-09-18T04:40:05.358 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.1446999999999996e-05 2024-09-18T04:40:05.359 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.359 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.359 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:05.359 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.594692+0000", 2024-09-18T04:40:05.359 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00025756599999999998 2024-09-18T04:40:05.359 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.359 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.359 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:05.359 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599302+0000", 2024-09-18T04:40:05.359 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0046099640000000002 2024-09-18T04:40:05.359 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.359 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.359 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:05.359 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606710+0000", 2024-09-18T04:40:05.359 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0074077739999999998 2024-09-18T04:40:05.360 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.360 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.360 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:05.360 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606798+0000", 2024-09-18T04:40:05.360 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8133999999999993e-05 2024-09-18T04:40:05.360 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.360 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.360 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:05.360 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606845+0000", 2024-09-18T04:40:05.360 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6653000000000002e-05 2024-09-18T04:40:05.360 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:05.360 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:05.360 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:05.360 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.361 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.361 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:105 2.0 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:40:05.361 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.583193+0000", 2024-09-18T04:40:05.361 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 68.658021926000004, 2024-09-18T04:40:05.361 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023713649, 2024-09-18T04:40:05.361 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:05.361 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:05.361 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:05.361 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:40:05.361 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:40:05.361 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 105 2024-09-18T04:40:05.361 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.361 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:05.361 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.362 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:05.362 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583193+0000", 2024-09-18T04:40:05.362 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:05.362 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.362 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.362 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:05.362 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583193+0000", 2024-09-18T04:40:05.362 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:05.362 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.362 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.362 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:05.362 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583192+0000", 2024-09-18T04:40:05.363 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:40:05.363 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.363 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.363 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:05.363 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583194+0000", 2024-09-18T04:40:05.363 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.305e-06 2024-09-18T04:40:05.363 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.363 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.363 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:05.363 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583198+0000", 2024-09-18T04:40:05.363 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9369999999999997e-06 2024-09-18T04:40:05.363 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.363 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.363 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:05.364 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583202+0000", 2024-09-18T04:40:05.364 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6480000000000001e-06 2024-09-18T04:40:05.364 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.364 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.364 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:05.364 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.595304+0000", 2024-09-18T04:40:05.364 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012102063999999999 2024-09-18T04:40:05.364 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.364 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.364 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:05.364 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.595390+0000", 2024-09-18T04:40:05.364 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5493000000000003e-05 2024-09-18T04:40:05.364 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.364 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.364 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:05.365 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.595509+0000", 2024-09-18T04:40:05.365 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011966199999999999 2024-09-18T04:40:05.365 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.365 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.365 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:05.365 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599358+0000", 2024-09-18T04:40:05.365 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0038486689999999999 2024-09-18T04:40:05.365 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.365 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.365 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:05.365 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606859+0000", 2024-09-18T04:40:05.365 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0075010789999999999 2024-09-18T04:40:05.365 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.365 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.366 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:05.366 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606887+0000", 2024-09-18T04:40:05.366 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7773e-05 2024-09-18T04:40:05.366 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.366 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.366 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:05.366 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606906+0000", 2024-09-18T04:40:05.366 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9417000000000001e-05 2024-09-18T04:40:05.366 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:05.366 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:05.366 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:05.366 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.366 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.366 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:106 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:40:05.366 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.584425+0000", 2024-09-18T04:40:05.367 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 68.656789630000006, 2024-09-18T04:40:05.367 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.022529293999999998, 2024-09-18T04:40:05.367 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:05.367 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:05.367 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:05.367 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:40:05.367 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:40:05.367 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 106 2024-09-18T04:40:05.367 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.367 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:05.368 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.368 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:05.368 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584425+0000", 2024-09-18T04:40:05.368 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:05.368 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.368 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.368 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:05.368 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584425+0000", 2024-09-18T04:40:05.368 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:05.368 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.369 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.369 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:05.369 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584424+0000", 2024-09-18T04:40:05.369 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:40:05.369 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.369 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.369 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:05.369 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584430+0000", 2024-09-18T04:40:05.369 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6389999999999997e-06 2024-09-18T04:40:05.370 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.370 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.370 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:05.370 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584456+0000", 2024-09-18T04:40:05.370 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6188999999999999e-05 2024-09-18T04:40:05.370 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.370 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.370 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:05.370 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584463+0000", 2024-09-18T04:40:05.370 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7340000000000002e-06 2024-09-18T04:40:05.370 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.370 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.371 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:05.371 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597537+0000", 2024-09-18T04:40:05.371 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013074161000000001 2024-09-18T04:40:05.371 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.371 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.371 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:05.371 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597598+0000", 2024-09-18T04:40:05.371 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0930000000000001e-05 2024-09-18T04:40:05.371 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.371 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.371 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:05.371 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597922+0000", 2024-09-18T04:40:05.371 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032473000000000002 2024-09-18T04:40:05.372 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.372 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.372 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:05.372 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.601103+0000", 2024-09-18T04:40:05.372 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0031809849999999999 2024-09-18T04:40:05.372 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.372 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.372 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:05.372 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606912+0000", 2024-09-18T04:40:05.372 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058083550000000003 2024-09-18T04:40:05.372 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.372 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.372 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:05.373 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606935+0000", 2024-09-18T04:40:05.373 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2759999999999999e-05 2024-09-18T04:40:05.373 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.373 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.373 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:05.373 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606954+0000", 2024-09-18T04:40:05.373 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9602999999999998e-05 2024-09-18T04:40:05.373 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:05.373 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:05.373 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:05.373 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.373 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.373 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:109 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:40:05.373 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.585231+0000", 2024-09-18T04:40:05.374 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 68.65598301, 2024-09-18T04:40:05.374 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.021767496000000001, 2024-09-18T04:40:05.374 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:05.374 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:05.374 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:05.374 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:40:05.374 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:40:05.374 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 109 2024-09-18T04:40:05.374 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.374 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:05.374 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.374 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:05.375 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585231+0000", 2024-09-18T04:40:05.375 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:05.375 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.375 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.375 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:05.375 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585231+0000", 2024-09-18T04:40:05.375 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:05.375 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.375 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.375 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:05.375 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585230+0000", 2024-09-18T04:40:05.375 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:40:05.376 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.376 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.376 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:05.376 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585237+0000", 2024-09-18T04:40:05.376 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.618e-06 2024-09-18T04:40:05.376 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.376 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.376 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:05.376 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585256+0000", 2024-09-18T04:40:05.376 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9048000000000001e-05 2024-09-18T04:40:05.376 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.376 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.377 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:05.377 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585274+0000", 2024-09-18T04:40:05.377 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8153000000000001e-05 2024-09-18T04:40:05.377 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.377 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.377 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:05.378 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.598284+0000", 2024-09-18T04:40:05.378 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013009682999999999 2024-09-18T04:40:05.378 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.378 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.379 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:05.379 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.598383+0000", 2024-09-18T04:40:05.379 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.8722999999999997e-05 2024-09-18T04:40:05.379 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.379 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.379 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:05.379 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.598829+0000", 2024-09-18T04:40:05.379 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00044623999999999998 2024-09-18T04:40:05.379 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.379 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.379 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:05.379 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.601149+0000", 2024-09-18T04:40:05.379 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0023204150000000002 2024-09-18T04:40:05.379 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.380 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.380 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:05.380 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606960+0000", 2024-09-18T04:40:05.380 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058106390000000003 2024-09-18T04:40:05.380 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.380 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.380 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:05.380 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606982+0000", 2024-09-18T04:40:05.380 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2252e-05 2024-09-18T04:40:05.380 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.380 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.380 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:05.380 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606999+0000", 2024-09-18T04:40:05.381 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6871000000000002e-05 2024-09-18T04:40:05.381 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:05.381 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:05.381 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:05.381 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.381 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.381 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:110 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:40:05.381 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.586640+0000", 2024-09-18T04:40:05.381 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 68.654574690999993, 2024-09-18T04:40:05.381 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020400911000000001, 2024-09-18T04:40:05.381 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:05.381 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:05.381 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:05.381 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:40:05.382 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:40:05.382 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 110 2024-09-18T04:40:05.382 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.382 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:05.382 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.382 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:05.382 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586640+0000", 2024-09-18T04:40:05.382 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:05.382 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.382 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.382 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:05.382 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586640+0000", 2024-09-18T04:40:05.382 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:05.382 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.382 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.383 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:05.383 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586639+0000", 2024-09-18T04:40:05.383 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:40:05.383 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.383 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.383 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:05.383 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586652+0000", 2024-09-18T04:40:05.383 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2482e-05 2024-09-18T04:40:05.383 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.383 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.383 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:05.383 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586660+0000", 2024-09-18T04:40:05.383 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1659999999999994e-06 2024-09-18T04:40:05.384 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.384 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.384 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:05.384 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586666+0000", 2024-09-18T04:40:05.384 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6900000000000003e-06 2024-09-18T04:40:05.384 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.384 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.384 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:05.384 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599381+0000", 2024-09-18T04:40:05.384 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012714328 2024-09-18T04:40:05.384 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.384 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.384 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:05.384 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599436+0000", 2024-09-18T04:40:05.385 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4802999999999997e-05 2024-09-18T04:40:05.385 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.385 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.385 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:05.385 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599629+0000", 2024-09-18T04:40:05.385 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000193921 2024-09-18T04:40:05.385 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.385 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.385 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:05.386 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.601174+0000", 2024-09-18T04:40:05.386 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001544857 2024-09-18T04:40:05.386 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.386 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.386 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:05.386 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.607004+0000", 2024-09-18T04:40:05.386 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058300820000000003 2024-09-18T04:40:05.386 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.386 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.386 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:05.386 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.607024+0000", 2024-09-18T04:40:05.386 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9888e-05 2024-09-18T04:40:05.386 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.386 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.387 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:05.387 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.607041+0000", 2024-09-18T04:40:05.387 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6450999999999999e-05 2024-09-18T04:40:05.387 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:05.387 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:05.387 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:05.387 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:05.387 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:05.387 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:40:05.389 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:40:05.389 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:40:05.383+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:40:05.489 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:40:05.484+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:40:05.546 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:40:05.546 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:40:05.546 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:40:05.546 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:40:05.555 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:40:05.629 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:40:05.620+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:40:05.674 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:40:05.674 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:40:05.675 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:40:05.675 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:40:05.675 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:40:05.684 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:40:05.696 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:40:05.691+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:40:05.793 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:40:05.788+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:40:05.806 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:40:05.806 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:40:05.806 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:40:05.806 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:40:05.806 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.806 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.5431.0:1 3.3 3:d80af3cb:::rbd_id.testrbdnbd17342:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-18T04:40:05.806 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:13.542818+0000", 2024-09-18T04:40:05.806 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 52.261327352000002, 2024-09-18T04:40:05.807 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048493400000000002, 2024-09-18T04:40:05.807 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:05.807 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:40:05.807 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:05.807 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.5431", 2024-09-18T04:40:05.807 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3147136083", 2024-09-18T04:40:05.807 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 1 2024-09-18T04:40:05.807 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.807 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:05.807 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.807 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:05.807 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542818+0000", 2024-09-18T04:40:05.807 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:05.807 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.808 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.808 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:05.808 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542818+0000", 2024-09-18T04:40:05.808 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:05.808 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.808 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.808 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:05.808 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542816+0000", 2024-09-18T04:40:05.808 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:05.808 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.808 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.808 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:05.808 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542828+0000", 2024-09-18T04:40:05.808 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2699e-05 2024-09-18T04:40:05.809 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.809 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.809 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:05.809 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542852+0000", 2024-09-18T04:40:05.809 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3176e-05 2024-09-18T04:40:05.809 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.809 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.809 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:05.809 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542874+0000", 2024-09-18T04:40:05.809 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1967000000000002e-05 2024-09-18T04:40:05.809 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.809 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.809 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:05.809 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542984+0000", 2024-09-18T04:40:05.809 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011026099999999999 2024-09-18T04:40:05.810 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.810 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.810 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:05.810 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543161+0000", 2024-09-18T04:40:05.810 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017720299999999999 2024-09-18T04:40:05.810 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.810 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.810 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:05.810 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543303+0000", 2024-09-18T04:40:05.810 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014174700000000001 2024-09-18T04:40:05.810 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:05.810 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:05.810 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:05.810 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.811 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.811 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.5431.0:2 3.3 3:d80af3cb:::rbd_id.testrbdnbd17342:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-18T04:40:05.811 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:13.543564+0000", 2024-09-18T04:40:05.811 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 52.260580865999998, 2024-09-18T04:40:05.811 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010399929999999999, 2024-09-18T04:40:05.811 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:05.811 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:40:05.811 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:05.811 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.5431", 2024-09-18T04:40:05.811 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3147136083", 2024-09-18T04:40:05.811 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 2 2024-09-18T04:40:05.812 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.812 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:05.812 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.812 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:05.812 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543564+0000", 2024-09-18T04:40:05.812 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:05.812 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.812 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.812 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:05.812 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543564+0000", 2024-09-18T04:40:05.812 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:05.813 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.813 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.813 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:05.813 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543563+0000", 2024-09-18T04:40:05.813 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:05.813 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.813 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.813 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:05.813 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543571+0000", 2024-09-18T04:40:05.813 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4270000000000008e-06 2024-09-18T04:40:05.813 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.813 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.814 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:05.814 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543591+0000", 2024-09-18T04:40:05.814 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.929e-05 2024-09-18T04:40:05.814 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.814 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.814 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:05.814 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543600+0000", 2024-09-18T04:40:05.814 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.1390000000000001e-06 2024-09-18T04:40:05.814 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.815 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.815 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:05.815 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543724+0000", 2024-09-18T04:40:05.815 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012458400000000001 2024-09-18T04:40:05.815 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.815 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.815 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:05.815 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543968+0000", 2024-09-18T04:40:05.815 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00024323199999999999 2024-09-18T04:40:05.815 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:05.815 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:05.815 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:05.815 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.544604+0000", 2024-09-18T04:40:05.815 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00063675799999999996 2024-09-18T04:40:05.816 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:05.816 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:05.816 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:05.816 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:05.816 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:05.816 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:40:05.817 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:40:05.895 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:40:05.892+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:40:05.915 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-18T04:40:05.916 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T04:40:05.923 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:40:05.923 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:40:05.923 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:40:05.923 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:40:05.933 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:40:05.996 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:40:05.991+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:40:06.071 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:40:06.084 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:40:06.084 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:40:06.084 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:40:06.084 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:40:06.084 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:40:06.091 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:40:06.095 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:40:06.097 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:40:06.092+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:40:06.113 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:40:06.137 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:40:06.158 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:40:06.179 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:40:06.181 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:40:06.181 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:40:06.181 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:40:06.182 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:40:06.182 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:40:06.182 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:40:06.197 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:40:06.192+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:40:06.211 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:06.211 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:06.221 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:40:06.299 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:40:06.296+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:40:06.325 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:06.325 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:06.334 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:40:06.400 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:40:06.396+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:40:06.439 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:06.439 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:06.449 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:40:06.500 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:40:06.496+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:40:06.559 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:06.559 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:06.570 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:40:06.614 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:40:06.611+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:40:06.675 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:06.676 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:06.686 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:40:06.702 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:40:06.699+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:40:06.791 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:06.792 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:06.802 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:40:06.842 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:40:06.839+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:40:06.844 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-09-18T04:40:06.904 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:40:06.899+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:40:06.948 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:06.949 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:06.958 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:40:07.006 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:40:07.000+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:40:07.065 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:07.065 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:07.075 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:40:07.107 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:40:07.103+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:40:07.182 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:07.182 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:07.192 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:40:07.207 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:40:07.203+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:40:07.299 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:07.299 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:07.309 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:40:07.309 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:40:07.303+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:40:07.410 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:40:07.403+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:40:07.427 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:07.427 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:07.437 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:40:07.511 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:40:07.507+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:40:07.543 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:07.543 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:07.553 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:40:07.559 INFO:tasks.workunit.client.0.smithi115.stderr:+ DEV=/dev/nbd1 2024-09-18T04:40:07.559 INFO:tasks.workunit.client.0.smithi115.stderr:+ get_pid rbd 2024-09-18T04:40:07.559 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pool=rbd 2024-09-18T04:40:07.559 INFO:tasks.workunit.client.0.smithi115.stderr:+ local ns= 2024-09-18T04:40:07.560 INFO:tasks.workunit.client.0.smithi115.stderr:++ rbd device --device-type nbd --format xml list 2024-09-18T04:40:07.560 INFO:tasks.workunit.client.0.smithi115.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd17342'\''][device='\''/dev/nbd1'\'']/id' 2024-09-18T04:40:07.612 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:40:07.608+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:40:07.661 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:07.661 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:07.663 INFO:tasks.workunit.client.0.smithi115.stderr:+ PID=20186 2024-09-18T04:40:07.663 INFO:tasks.workunit.client.0.smithi115.stderr:+ test -n 20186 2024-09-18T04:40:07.663 INFO:tasks.workunit.client.0.smithi115.stderr:+ ps -p 20186 -C rbd-nbd 2024-09-18T04:40:07.669 INFO:tasks.workunit.client.0.smithi115.stdout: PID TTY TIME CMD 2024-09-18T04:40:07.669 INFO:tasks.workunit.client.0.smithi115.stdout: 20186 ? 00:00:00 rbd-nbd 2024-09-18T04:40:07.669 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo blkdiscard --offset 0 --length 3584 /dev/nbd1 2024-09-18T04:40:07.669 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:40:07.670 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:40:07.673 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:40:07.673 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:40:07.674 INFO:tasks.workunit.client.0.smithi115.stderr:++ which blkdiscard 2024-09-18T04:40:07.674 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/sbin/blkdiscard 2024-09-18T04:40:07.674 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:40:07.674 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 0 --length 3584 /dev/nbd1 2024-09-18T04:40:07.681 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo blkdiscard --offset 0 --length 4096 /dev/nbd1 2024-09-18T04:40:07.681 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:40:07.681 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:40:07.682 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:40:07.682 INFO:tasks.workunit.client.0.smithi115.stderr:++ which blkdiscard 2024-09-18T04:40:07.684 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/sbin/blkdiscard 2024-09-18T04:40:07.684 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:40:07.684 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 0 --length 4096 /dev/nbd1 2024-09-18T04:40:07.700 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo blkdiscard --offset 0 --length 4608 /dev/nbd1 2024-09-18T04:40:07.700 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:40:07.700 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:40:07.702 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:40:07.702 INFO:tasks.workunit.client.0.smithi115.stderr:++ which blkdiscard 2024-09-18T04:40:07.705 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/sbin/blkdiscard 2024-09-18T04:40:07.705 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:40:07.705 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 0 --length 4608 /dev/nbd1 2024-09-18T04:40:07.712 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:40:07.707+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:40:07.722 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo blkdiscard --offset 512 --length 7168 /dev/nbd1 2024-09-18T04:40:07.723 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:40:07.723 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:40:07.725 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:40:07.725 INFO:tasks.workunit.client.0.smithi115.stderr:++ which blkdiscard 2024-09-18T04:40:07.727 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/sbin/blkdiscard 2024-09-18T04:40:07.727 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:40:07.728 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 512 --length 7168 /dev/nbd1 2024-09-18T04:40:07.737 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo blkdiscard --offset 512 --length 7680 /dev/nbd1 2024-09-18T04:40:07.737 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:40:07.737 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:40:07.739 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:40:07.740 INFO:tasks.workunit.client.0.smithi115.stderr:++ which blkdiscard 2024-09-18T04:40:07.742 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/sbin/blkdiscard 2024-09-18T04:40:07.742 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:40:07.742 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 512 --length 7680 /dev/nbd1 2024-09-18T04:40:07.756 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo blkdiscard --offset 512 --length 8192 /dev/nbd1 2024-09-18T04:40:07.756 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:40:07.756 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:40:07.757 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:40:07.757 INFO:tasks.workunit.client.0.smithi115.stderr:++ which blkdiscard 2024-09-18T04:40:07.759 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/sbin/blkdiscard 2024-09-18T04:40:07.759 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:40:07.759 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 512 --length 8192 /dev/nbd1 2024-09-18T04:40:07.774 INFO:tasks.workunit.client.0.smithi115.stderr:+ tries=0 2024-09-18T04:40:07.775 INFO:tasks.workunit.client.0.smithi115.stderr:++ rbd journal inspect --pool rbd --image testrbdnbd17342 2024-09-18T04:40:07.776 INFO:tasks.workunit.client.0.smithi115.stderr:++ awk '/entries inspected/ {print $1}' 2024-09-18T04:40:07.813 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:40:07.807+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:40:07.843 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:07.843 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:07.854 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:40:07.874 INFO:tasks.workunit.client.0.smithi115.stderr:2024-09-18T04:40:07.867+0000 7f85d9d78700 -1 JournalMetadata: 0x558c738c8bd0 failed to locate client: INSPECT 2024-09-18T04:40:07.882 INFO:tasks.workunit.client.0.smithi115.stderr:+ queue_length=4 2024-09-18T04:40:07.882 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 0 -lt 10 ']' 2024-09-18T04:40:07.882 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 4 -gt 0 ']' 2024-09-18T04:40:07.882 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd journal inspect --pool rbd --image testrbdnbd17342 --verbose 2024-09-18T04:40:07.914 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:40:07.911+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:40:07.957 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:07.957 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:07.968 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:40:07.976 INFO:tasks.workunit.client.0.smithi115.stdout:Entry: tag_id=1, commit_tid=1 2024-09-18T04:40:07.976 INFO:tasks.workunit.client.0.smithi115.stdout:{ 2024-09-18T04:40:07.976 INFO:tasks.workunit.client.0.smithi115.stdout: "event_type": "AioDiscard", 2024-09-18T04:40:07.976 INFO:tasks.workunit.client.0.smithi115.stdout: "offset": 0, 2024-09-18T04:40:07.977 INFO:tasks.workunit.client.0.smithi115.stdout: "length": 4096, 2024-09-18T04:40:07.977 INFO:tasks.workunit.client.0.smithi115.stdout: "discard_granularity_bytes": 4096, 2024-09-18T04:40:07.977 INFO:tasks.workunit.client.0.smithi115.stdout: "timestamp": "2024-09-18T04:40:07.691875+0000" 2024-09-18T04:40:07.977 INFO:tasks.workunit.client.0.smithi115.stdout:} 2024-09-18T04:40:07.977 INFO:tasks.workunit.client.0.smithi115.stdout:Entry: tag_id=1, commit_tid=2 2024-09-18T04:40:07.977 INFO:tasks.workunit.client.0.smithi115.stdout:{ 2024-09-18T04:40:07.977 INFO:tasks.workunit.client.0.smithi115.stdout: "event_type": "AioDiscard", 2024-09-18T04:40:07.977 INFO:tasks.workunit.client.0.smithi115.stdout: "offset": 0, 2024-09-18T04:40:07.977 INFO:tasks.workunit.client.0.smithi115.stdout: "length": 4096, 2024-09-18T04:40:07.978 INFO:tasks.workunit.client.0.smithi115.stdout: "discard_granularity_bytes": 4096, 2024-09-18T04:40:07.978 INFO:tasks.workunit.client.0.smithi115.stdout: "timestamp": "2024-09-18T04:40:07.714307+0000" 2024-09-18T04:40:07.978 INFO:tasks.workunit.client.0.smithi115.stdout:} 2024-09-18T04:40:07.978 INFO:tasks.workunit.client.0.smithi115.stdout:Entry: tag_id=1, commit_tid=3 2024-09-18T04:40:07.978 INFO:tasks.workunit.client.0.smithi115.stdout:{ 2024-09-18T04:40:07.978 INFO:tasks.workunit.client.0.smithi115.stdout: "event_type": "AioDiscard", 2024-09-18T04:40:07.978 INFO:tasks.workunit.client.0.smithi115.stdout: "offset": 4096, 2024-09-18T04:40:07.978 INFO:tasks.workunit.client.0.smithi115.stdout: "length": 4096, 2024-09-18T04:40:07.978 INFO:tasks.workunit.client.0.smithi115.stdout: "discard_granularity_bytes": 4096, 2024-09-18T04:40:07.979 INFO:tasks.workunit.client.0.smithi115.stdout: "timestamp": "2024-09-18T04:40:07.745921+0000" 2024-09-18T04:40:07.979 INFO:tasks.workunit.client.0.smithi115.stdout:} 2024-09-18T04:40:07.979 INFO:tasks.workunit.client.0.smithi115.stdout:Entry: tag_id=1, commit_tid=4 2024-09-18T04:40:07.979 INFO:tasks.workunit.client.0.smithi115.stdout:{ 2024-09-18T04:40:07.979 INFO:tasks.workunit.client.0.smithi115.stdout: "event_type": "AioDiscard", 2024-09-18T04:40:07.979 INFO:tasks.workunit.client.0.smithi115.stdout: "offset": 4096, 2024-09-18T04:40:07.980 INFO:tasks.workunit.client.0.smithi115.stdout: "length": 4096, 2024-09-18T04:40:07.980 INFO:tasks.workunit.client.0.smithi115.stdout: "discard_granularity_bytes": 4096, 2024-09-18T04:40:07.980 INFO:tasks.workunit.client.0.smithi115.stdout: "timestamp": "2024-09-18T04:40:07.765789+0000" 2024-09-18T04:40:07.980 INFO:tasks.workunit.client.0.smithi115.stdout:} 2024-09-18T04:40:07.980 INFO:tasks.workunit.client.0.smithi115.stdout:Summary: 2024-09-18T04:40:07.980 INFO:tasks.workunit.client.0.smithi115.stdout: 4 entries inspected, 0 errors 2024-09-18T04:40:07.986 INFO:tasks.workunit.client.0.smithi115.stderr:2024-09-18T04:40:07.979+0000 7fb4ab7fe700 -1 JournalMetadata: 0x5595ea5d78c0 failed to locate client: INSPECT 2024-09-18T04:40:07.992 INFO:tasks.workunit.client.0.smithi115.stderr:+ sleep 1 2024-09-18T04:40:08.015 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:40:08.011+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:40:08.071 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:08.071 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:08.080 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:40:08.116 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:40:08.111+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:40:08.189 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:08.189 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:08.193 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T04:40:08.198 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:40:08.246 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:40:08.244+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:40:08.299 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:08.299 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:08.309 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:40:08.317 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:40:08.311+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:40:08.345 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:40:08.365 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:40:08.385 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:40:08.409 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:40:08.418 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:40:08.416+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:40:08.418 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:08.419 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:08.428 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:40:08.429 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:40:08.450 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:40:08.452 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:40:08.452 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:40:08.452 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:40:08.452 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:40:08.452 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:40:08.452 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:40:08.520 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:40:08.516+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:40:08.541 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:40:08.541 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:40:08.541 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:40:08.541 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:40:08.551 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:40:08.621 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:40:08.615+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:40:08.654 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:40:08.654 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:40:08.655 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:40:08.655 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:40:08.655 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:40:08.664 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:40:08.722 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:40:08.716+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:40:08.820 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:40:08.820 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:40:08.820 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:40:08.820 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:40:08.820 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.821 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:7, version: 79'62, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:7@79'62, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 8=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-18T04:40:08.821 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.724969+0000", 2024-09-18T04:40:08.821 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.089685587000002, 2024-09-18T04:40:08.821 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0035525930000000002, 2024-09-18T04:40:08.821 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:08.821 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:08.821 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:08.821 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.821 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:08.822 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.724969+0000", 2024-09-18T04:40:08.822 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:08.822 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.822 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.822 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:08.823 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.724969+0000", 2024-09-18T04:40:08.823 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:08.823 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.823 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.824 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:08.824 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.724968+0000", 2024-09-18T04:40:08.824 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:08.824 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.824 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.824 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:08.824 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.726245+0000", 2024-09-18T04:40:08.824 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001276395 2024-09-18T04:40:08.824 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.824 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.824 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:08.824 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.726262+0000", 2024-09-18T04:40:08.824 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6973000000000002e-05 2024-09-18T04:40:08.825 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.825 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.825 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:08.825 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.726293+0000", 2024-09-18T04:40:08.825 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0886999999999998e-05 2024-09-18T04:40:08.825 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.825 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.825 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:08.825 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.726403+0000", 2024-09-18T04:40:08.825 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000110288 2024-09-18T04:40:08.825 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.825 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.825 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:08.826 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.726418+0000", 2024-09-18T04:40:08.826 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5503999999999999e-05 2024-09-18T04:40:08.826 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.826 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.826 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:08.826 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.728522+0000", 2024-09-18T04:40:08.826 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021032630000000002 2024-09-18T04:40:08.826 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:08.826 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:08.826 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:08.826 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.826 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.826 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.1 87/86 [PushOp(2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head, version: 81'220, data_included: [4182016~12288], data_size: 12288, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head@81'220, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:4182016, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:40:08.826 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.730366+0000", 2024-09-18T04:40:08.827 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.08428919, 2024-09-18T04:40:08.827 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0032522580000000001, 2024-09-18T04:40:08.827 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:08.827 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:08.827 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:08.827 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.827 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:08.827 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.730366+0000", 2024-09-18T04:40:08.827 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:08.827 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.827 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.827 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:08.828 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.730366+0000", 2024-09-18T04:40:08.828 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:08.828 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.828 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.828 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:08.828 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.730365+0000", 2024-09-18T04:40:08.828 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:08.828 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.828 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.828 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:08.828 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.730385+0000", 2024-09-18T04:40:08.828 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0766000000000001e-05 2024-09-18T04:40:08.828 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.829 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.829 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:08.829 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.730406+0000", 2024-09-18T04:40:08.829 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0706000000000001e-05 2024-09-18T04:40:08.829 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.829 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.829 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:08.829 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.730423+0000", 2024-09-18T04:40:08.829 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7411e-05 2024-09-18T04:40:08.829 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.829 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.829 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:08.829 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.730507+0000", 2024-09-18T04:40:08.830 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3388999999999997e-05 2024-09-18T04:40:08.830 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.830 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.830 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:08.830 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.730523+0000", 2024-09-18T04:40:08.830 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5843999999999999e-05 2024-09-18T04:40:08.830 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.830 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.830 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:08.830 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.733618+0000", 2024-09-18T04:40:08.830 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030951540000000001 2024-09-18T04:40:08.830 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:08.830 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:08.831 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:08.831 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.831 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.831 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:7, version: 79'62, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:7@79'62, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 8=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:40:08.831 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.732222+0000", 2024-09-18T04:40:08.831 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.082433229999999, 2024-09-18T04:40:08.831 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030084410000000002, 2024-09-18T04:40:08.831 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:08.831 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:08.831 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:08.831 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.831 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:08.831 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.732222+0000", 2024-09-18T04:40:08.831 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:08.832 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.832 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.832 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:08.832 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.732222+0000", 2024-09-18T04:40:08.832 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:08.832 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.832 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.832 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:08.832 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.732221+0000", 2024-09-18T04:40:08.832 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:40:08.832 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.832 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.832 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:08.832 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.733174+0000", 2024-09-18T04:40:08.833 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00095350299999999997 2024-09-18T04:40:08.833 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.833 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.833 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:08.833 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.733197+0000", 2024-09-18T04:40:08.833 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2925000000000002e-05 2024-09-18T04:40:08.833 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.833 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.833 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:08.833 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.733218+0000", 2024-09-18T04:40:08.833 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0477999999999999e-05 2024-09-18T04:40:08.833 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.833 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.834 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:08.834 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.733323+0000", 2024-09-18T04:40:08.834 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000105717 2024-09-18T04:40:08.834 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.834 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.834 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:08.834 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.733358+0000", 2024-09-18T04:40:08.834 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4406000000000002e-05 2024-09-18T04:40:08.834 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.834 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.834 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:08.834 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.735230+0000", 2024-09-18T04:40:08.834 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0018720659999999999 2024-09-18T04:40:08.835 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:08.835 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:08.835 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:08.835 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.835 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.835 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:7, version: 79'62, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:7@79'62, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 8=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:40:08.835 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.743433+0000", 2024-09-18T04:40:08.835 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.071221748999999, 2024-09-18T04:40:08.835 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003400237, 2024-09-18T04:40:08.835 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:08.835 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:08.835 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:08.835 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.835 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:08.836 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.743433+0000", 2024-09-18T04:40:08.836 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:08.836 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.836 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.836 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:08.836 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.743433+0000", 2024-09-18T04:40:08.836 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:08.836 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.836 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.836 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:08.836 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.743432+0000", 2024-09-18T04:40:08.836 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:08.836 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.836 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.837 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:08.837 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.744392+0000", 2024-09-18T04:40:08.837 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00095993499999999998 2024-09-18T04:40:08.837 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.837 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.837 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:08.837 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.744417+0000", 2024-09-18T04:40:08.837 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5249000000000002e-05 2024-09-18T04:40:08.837 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.837 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.837 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:08.837 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.744434+0000", 2024-09-18T04:40:08.837 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6597999999999999e-05 2024-09-18T04:40:08.837 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.837 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.838 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:08.838 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.744499+0000", 2024-09-18T04:40:08.838 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4987000000000006e-05 2024-09-18T04:40:08.838 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.838 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.838 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:08.838 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.744506+0000", 2024-09-18T04:40:08.838 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2679999999999997e-06 2024-09-18T04:40:08.838 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.838 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.838 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:08.838 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.746833+0000", 2024-09-18T04:40:08.838 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023270529999999999 2024-09-18T04:40:08.839 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:08.839 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:08.839 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:08.839 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.839 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.839 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.5 87/86 [PushOp(2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:head, version: 79'60, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:head@79'60, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-18T04:40:08.839 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.747807+0000", 2024-09-18T04:40:08.839 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.066847275000001, 2024-09-18T04:40:08.839 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00244378, 2024-09-18T04:40:08.839 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:08.839 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:08.839 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:08.839 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.840 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:08.840 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.747807+0000", 2024-09-18T04:40:08.840 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:08.840 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.840 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.840 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:08.840 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.747807+0000", 2024-09-18T04:40:08.840 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:08.840 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.840 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.840 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:08.840 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.747806+0000", 2024-09-18T04:40:08.841 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:08.841 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.841 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.841 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:08.841 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.747825+0000", 2024-09-18T04:40:08.841 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.925e-05 2024-09-18T04:40:08.841 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.841 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.841 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:08.841 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.747853+0000", 2024-09-18T04:40:08.841 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7174000000000001e-05 2024-09-18T04:40:08.841 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.841 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.841 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:08.842 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.747887+0000", 2024-09-18T04:40:08.842 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4731999999999999e-05 2024-09-18T04:40:08.842 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.842 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.842 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:08.842 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.747983+0000", 2024-09-18T04:40:08.842 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5964999999999993e-05 2024-09-18T04:40:08.842 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.842 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.842 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:08.842 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.747994+0000", 2024-09-18T04:40:08.842 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0526999999999999e-05 2024-09-18T04:40:08.842 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.843 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.843 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:08.843 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.750251+0000", 2024-09-18T04:40:08.843 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0022573519999999998 2024-09-18T04:40:08.843 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:08.843 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:08.843 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:08.843 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.843 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.843 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:7, version: 79'64, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:7@79'64, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-18T04:40:08.843 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.750570+0000", 2024-09-18T04:40:08.843 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.064084871999999, 2024-09-18T04:40:08.843 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0029248949999999998, 2024-09-18T04:40:08.844 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:08.844 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:08.844 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:08.844 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.844 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:08.844 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.750570+0000", 2024-09-18T04:40:08.844 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:08.844 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.844 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.844 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:08.844 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.750570+0000", 2024-09-18T04:40:08.844 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:08.845 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.845 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.845 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:08.845 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.750569+0000", 2024-09-18T04:40:08.845 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:08.845 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.845 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.845 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:08.845 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.751724+0000", 2024-09-18T04:40:08.845 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001155037 2024-09-18T04:40:08.845 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.845 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.845 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:08.846 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.751747+0000", 2024-09-18T04:40:08.846 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2679000000000001e-05 2024-09-18T04:40:08.846 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.846 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.846 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:08.846 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.751761+0000", 2024-09-18T04:40:08.846 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.467e-05 2024-09-18T04:40:08.846 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.846 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.846 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:08.846 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.751839+0000", 2024-09-18T04:40:08.846 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7824000000000006e-05 2024-09-18T04:40:08.846 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.846 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.846 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:08.847 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.751849+0000", 2024-09-18T04:40:08.847 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4159999999999993e-06 2024-09-18T04:40:08.847 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.847 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.847 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:08.847 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.753495+0000", 2024-09-18T04:40:08.847 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0016461830000000001 2024-09-18T04:40:08.847 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:08.847 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:08.847 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:08.847 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.847 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.847 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.5 87/86 [PushOp(2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:7, version: 79'59, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:7@79'59, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-18T04:40:08.847 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.752479+0000", 2024-09-18T04:40:08.848 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.062175504999999, 2024-09-18T04:40:08.848 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027271040000000002, 2024-09-18T04:40:08.848 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:08.848 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:08.848 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:08.848 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.848 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:08.848 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.752479+0000", 2024-09-18T04:40:08.848 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:08.848 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.849 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.849 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:08.849 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.752479+0000", 2024-09-18T04:40:08.849 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:08.849 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.849 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.849 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:08.849 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.752478+0000", 2024-09-18T04:40:08.849 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:08.849 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.849 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.849 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:08.849 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.753784+0000", 2024-09-18T04:40:08.849 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0013058869999999999 2024-09-18T04:40:08.850 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.850 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.850 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:08.850 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.753821+0000", 2024-09-18T04:40:08.850 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7268999999999998e-05 2024-09-18T04:40:08.850 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.850 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.850 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:08.850 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.753873+0000", 2024-09-18T04:40:08.850 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1564000000000003e-05 2024-09-18T04:40:08.850 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.850 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.850 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:08.850 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.753963+0000", 2024-09-18T04:40:08.851 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0636000000000004e-05 2024-09-18T04:40:08.851 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.851 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.851 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:08.851 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.753972+0000", 2024-09-18T04:40:08.851 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0529999999999996e-06 2024-09-18T04:40:08.851 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.851 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.851 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:08.851 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.755206+0000", 2024-09-18T04:40:08.851 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001234113 2024-09-18T04:40:08.851 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:08.851 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:08.851 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:08.852 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.852 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.852 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:7, version: 79'64, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:7@79'64, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:40:08.852 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.756672+0000", 2024-09-18T04:40:08.852 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.057982427999999, 2024-09-18T04:40:08.852 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028688770000000001, 2024-09-18T04:40:08.852 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:08.852 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:08.852 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:08.852 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.852 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:08.852 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.756672+0000", 2024-09-18T04:40:08.852 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:08.852 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.853 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.853 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:08.853 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.756672+0000", 2024-09-18T04:40:08.853 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:08.853 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.853 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.853 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:08.853 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.756671+0000", 2024-09-18T04:40:08.853 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:08.853 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.853 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.853 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:08.853 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.757623+0000", 2024-09-18T04:40:08.853 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00095182199999999995 2024-09-18T04:40:08.854 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.854 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.854 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:08.854 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.757644+0000", 2024-09-18T04:40:08.854 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0996999999999999e-05 2024-09-18T04:40:08.854 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.854 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.854 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:08.854 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.757657+0000", 2024-09-18T04:40:08.854 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3264999999999999e-05 2024-09-18T04:40:08.854 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.854 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.854 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:08.854 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.757779+0000", 2024-09-18T04:40:08.854 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000121655 2024-09-18T04:40:08.855 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.855 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.855 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:08.855 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.757801+0000", 2024-09-18T04:40:08.855 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2340999999999998e-05 2024-09-18T04:40:08.855 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.855 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.855 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:08.855 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.759541+0000", 2024-09-18T04:40:08.855 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017398120000000001 2024-09-18T04:40:08.855 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:08.855 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:08.855 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:08.855 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.855 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.856 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.5 87/86 [PushOp(2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:7, version: 79'59, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:7@79'59, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:40:08.856 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.758140+0000", 2024-09-18T04:40:08.856 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.056514523000001, 2024-09-18T04:40:08.856 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023362539999999998, 2024-09-18T04:40:08.856 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:08.856 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:08.856 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:08.856 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.856 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:08.856 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.758140+0000", 2024-09-18T04:40:08.856 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:08.856 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.856 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.857 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:08.857 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.758140+0000", 2024-09-18T04:40:08.857 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:08.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.857 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.857 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:08.857 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.758139+0000", 2024-09-18T04:40:08.857 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:08.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.857 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.857 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:08.857 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.758850+0000", 2024-09-18T04:40:08.857 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00071123099999999999 2024-09-18T04:40:08.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.858 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.858 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:08.858 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.758872+0000", 2024-09-18T04:40:08.858 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2195999999999999e-05 2024-09-18T04:40:08.858 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.858 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.858 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:08.858 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.758885+0000", 2024-09-18T04:40:08.858 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.327e-05 2024-09-18T04:40:08.858 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.858 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.858 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:08.858 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.758969+0000", 2024-09-18T04:40:08.858 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3497000000000004e-05 2024-09-18T04:40:08.858 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.859 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.859 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:08.859 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.758988+0000", 2024-09-18T04:40:08.859 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8765000000000001e-05 2024-09-18T04:40:08.859 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.859 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.859 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:08.859 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.760476+0000", 2024-09-18T04:40:08.859 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001488828 2024-09-18T04:40:08.859 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:08.859 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:08.859 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:08.859 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.859 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.860 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.5 87/86 [PushOp(2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:7, version: 79'59, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:7@79'59, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:40:08.860 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.762725+0000", 2024-09-18T04:40:08.860 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.051929422000001, 2024-09-18T04:40:08.860 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028321890000000001, 2024-09-18T04:40:08.860 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:08.860 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:08.860 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:08.860 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.860 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:08.860 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.762725+0000", 2024-09-18T04:40:08.860 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:08.860 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.860 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.860 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:08.860 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.762725+0000", 2024-09-18T04:40:08.861 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:08.861 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.861 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.861 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:08.861 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.762724+0000", 2024-09-18T04:40:08.861 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:08.861 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.861 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.861 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:08.861 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.763395+0000", 2024-09-18T04:40:08.862 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00067064800000000003 2024-09-18T04:40:08.862 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.862 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.862 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:08.862 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.763416+0000", 2024-09-18T04:40:08.862 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0984000000000001e-05 2024-09-18T04:40:08.862 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.862 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.862 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:08.862 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.763442+0000", 2024-09-18T04:40:08.862 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6635000000000002e-05 2024-09-18T04:40:08.862 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.862 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.863 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:08.863 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.763584+0000", 2024-09-18T04:40:08.863 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000142037 2024-09-18T04:40:08.863 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.863 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.863 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:08.863 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.763610+0000", 2024-09-18T04:40:08.863 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6027999999999999e-05 2024-09-18T04:40:08.863 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.863 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.863 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:08.863 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.765557+0000", 2024-09-18T04:40:08.863 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0019470939999999999 2024-09-18T04:40:08.863 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:08.863 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:08.864 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:08.864 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.864 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.864 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:7, version: 79'64, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:7@79'64, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:40:08.864 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.763323+0000", 2024-09-18T04:40:08.864 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.051331236999999, 2024-09-18T04:40:08.864 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003042215, 2024-09-18T04:40:08.864 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:08.864 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:08.864 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:08.864 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.864 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:08.865 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.763323+0000", 2024-09-18T04:40:08.865 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:08.865 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.865 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.865 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:08.865 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.763323+0000", 2024-09-18T04:40:08.865 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:08.865 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.865 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.865 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:08.865 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.763322+0000", 2024-09-18T04:40:08.866 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:08.866 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.866 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.866 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:08.866 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.764567+0000", 2024-09-18T04:40:08.866 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001244312 2024-09-18T04:40:08.866 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.866 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.866 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:08.866 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.764600+0000", 2024-09-18T04:40:08.866 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2787000000000003e-05 2024-09-18T04:40:08.866 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.866 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.866 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:08.867 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.764640+0000", 2024-09-18T04:40:08.867 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0825000000000003e-05 2024-09-18T04:40:08.867 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.867 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.867 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:08.867 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.764789+0000", 2024-09-18T04:40:08.867 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000148598 2024-09-18T04:40:08.867 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.867 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.867 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:08.867 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.764812+0000", 2024-09-18T04:40:08.867 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2702e-05 2024-09-18T04:40:08.867 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.867 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.867 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:08.868 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.766366+0000", 2024-09-18T04:40:08.868 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0015540110000000001 2024-09-18T04:40:08.868 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:08.868 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:08.868 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:08.868 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.868 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.868 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.5 87/86 [PushOp(2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:7, version: 79'59, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:7@79'59, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:40:08.868 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.768078+0000", 2024-09-18T04:40:08.868 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.046576459000001, 2024-09-18T04:40:08.868 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028938509999999998, 2024-09-18T04:40:08.868 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:08.868 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:08.869 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:08.869 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.869 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:08.869 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.768078+0000", 2024-09-18T04:40:08.869 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:08.869 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.869 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.869 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:08.869 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.768078+0000", 2024-09-18T04:40:08.869 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:08.869 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.869 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.869 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:08.870 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.768076+0000", 2024-09-18T04:40:08.870 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:08.870 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.870 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.870 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:08.870 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.768951+0000", 2024-09-18T04:40:08.870 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000874455 2024-09-18T04:40:08.870 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.870 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.870 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:08.870 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.768972+0000", 2024-09-18T04:40:08.870 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.154e-05 2024-09-18T04:40:08.870 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.870 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.871 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:08.871 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.768992+0000", 2024-09-18T04:40:08.871 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9822000000000001e-05 2024-09-18T04:40:08.871 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.871 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.871 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:08.871 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.769110+0000", 2024-09-18T04:40:08.871 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000117762 2024-09-18T04:40:08.871 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.871 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.871 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:08.871 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.769123+0000", 2024-09-18T04:40:08.871 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2473999999999999e-05 2024-09-18T04:40:08.871 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.871 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.871 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:08.872 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.770972+0000", 2024-09-18T04:40:08.872 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001849613 2024-09-18T04:40:08.872 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:08.872 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:08.872 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:08.872 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.872 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.872 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:7, version: 79'64, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:7@79'64, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:40:08.872 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.769830+0000", 2024-09-18T04:40:08.872 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.044825195000001, 2024-09-18T04:40:08.872 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0040223419999999999, 2024-09-18T04:40:08.872 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:08.872 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:08.872 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:08.872 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.873 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:08.873 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.769830+0000", 2024-09-18T04:40:08.873 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:08.873 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.873 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.873 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:08.873 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.769830+0000", 2024-09-18T04:40:08.873 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:08.873 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.873 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.873 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:08.873 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.769829+0000", 2024-09-18T04:40:08.873 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:08.873 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.873 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.874 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:08.874 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.770812+0000", 2024-09-18T04:40:08.874 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00098347800000000009 2024-09-18T04:40:08.874 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.874 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.874 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:08.874 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.770834+0000", 2024-09-18T04:40:08.874 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1648999999999999e-05 2024-09-18T04:40:08.874 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.874 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.874 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:08.874 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.770847+0000", 2024-09-18T04:40:08.874 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3169999999999999e-05 2024-09-18T04:40:08.874 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.875 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.875 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:08.875 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.770969+0000", 2024-09-18T04:40:08.875 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000121721 2024-09-18T04:40:08.875 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.875 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.875 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:08.875 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.770989+0000", 2024-09-18T04:40:08.875 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9944000000000001e-05 2024-09-18T04:40:08.875 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.875 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.875 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:08.875 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.773852+0000", 2024-09-18T04:40:08.875 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028632319999999998 2024-09-18T04:40:08.875 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:08.876 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:08.876 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:08.876 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.876 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.876 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.5 87/86 [PushOp(2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head, version: 79'62, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head@79'62, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-18T04:40:08.876 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.772378+0000", 2024-09-18T04:40:08.876 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.042277072000001, 2024-09-18T04:40:08.876 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002661535, 2024-09-18T04:40:08.876 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:08.876 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:08.876 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:08.876 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.876 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:08.877 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.772378+0000", 2024-09-18T04:40:08.877 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:08.877 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.877 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.877 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:08.877 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.772378+0000", 2024-09-18T04:40:08.877 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:08.877 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.877 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.877 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:08.877 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.772377+0000", 2024-09-18T04:40:08.877 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:08.877 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.877 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.877 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:08.878 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.772393+0000", 2024-09-18T04:40:08.878 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6495e-05 2024-09-18T04:40:08.878 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.878 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.878 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:08.878 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.772417+0000", 2024-09-18T04:40:08.878 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4304e-05 2024-09-18T04:40:08.878 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.878 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.878 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:08.878 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.772431+0000", 2024-09-18T04:40:08.878 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3165e-05 2024-09-18T04:40:08.878 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.878 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.878 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:08.879 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.772499+0000", 2024-09-18T04:40:08.879 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8627000000000005e-05 2024-09-18T04:40:08.879 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.879 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.879 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:08.879 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.772508+0000", 2024-09-18T04:40:08.879 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5609999999999999e-06 2024-09-18T04:40:08.879 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.879 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.879 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:08.879 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.775039+0000", 2024-09-18T04:40:08.879 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002531387 2024-09-18T04:40:08.879 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:08.879 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:08.879 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:08.880 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.880 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.880 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head, version: 79'65, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head@79'65, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-18T04:40:08.880 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.775934+0000", 2024-09-18T04:40:08.880 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.038720318999999, 2024-09-18T04:40:08.880 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023666939999999999, 2024-09-18T04:40:08.880 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:08.880 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:08.880 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:08.880 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.880 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:08.880 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.775934+0000", 2024-09-18T04:40:08.880 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:08.881 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.881 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.881 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:08.881 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.775934+0000", 2024-09-18T04:40:08.881 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:08.881 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.881 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.881 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:08.881 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.775933+0000", 2024-09-18T04:40:08.881 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:08.881 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.881 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.881 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:08.882 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.775955+0000", 2024-09-18T04:40:08.882 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1512e-05 2024-09-18T04:40:08.882 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.882 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.882 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:08.882 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.775975+0000", 2024-09-18T04:40:08.882 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9653999999999999e-05 2024-09-18T04:40:08.882 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.882 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.882 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:08.882 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.775992+0000", 2024-09-18T04:40:08.882 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7326e-05 2024-09-18T04:40:08.882 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.882 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.883 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:08.883 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.776078+0000", 2024-09-18T04:40:08.883 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.6599000000000002e-05 2024-09-18T04:40:08.883 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.883 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.883 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:08.883 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.776088+0000", 2024-09-18T04:40:08.883 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.8120000000000002e-06 2024-09-18T04:40:08.883 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.883 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.883 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:08.883 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.778301+0000", 2024-09-18T04:40:08.883 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002212805 2024-09-18T04:40:08.883 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:08.884 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:08.884 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:08.884 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.884 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.884 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.5 87/86 [PushOp(2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:7, version: 79'61, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:7@79'61, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:40:08.884 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.781808+0000", 2024-09-18T04:40:08.884 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.032847073999999, 2024-09-18T04:40:08.884 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027222169999999999, 2024-09-18T04:40:08.884 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:08.884 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:08.884 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:08.884 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.884 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:08.884 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.781808+0000", 2024-09-18T04:40:08.885 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:08.885 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.885 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.885 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:08.885 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.781808+0000", 2024-09-18T04:40:08.885 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:08.885 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.885 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.885 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:08.885 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.781807+0000", 2024-09-18T04:40:08.885 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:08.885 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.885 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.885 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:08.885 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.782242+0000", 2024-09-18T04:40:08.886 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00043528300000000001 2024-09-18T04:40:08.886 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.886 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.886 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:08.886 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.782265+0000", 2024-09-18T04:40:08.886 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3227e-05 2024-09-18T04:40:08.886 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.886 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.886 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:08.886 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.782283+0000", 2024-09-18T04:40:08.886 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7535999999999999e-05 2024-09-18T04:40:08.886 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.886 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.886 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:08.886 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.782418+0000", 2024-09-18T04:40:08.887 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000135298 2024-09-18T04:40:08.887 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.887 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.887 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:08.887 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.782444+0000", 2024-09-18T04:40:08.887 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6020000000000002e-05 2024-09-18T04:40:08.887 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.887 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.887 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:08.887 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.784530+0000", 2024-09-18T04:40:08.887 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020858890000000001 2024-09-18T04:40:08.887 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:08.887 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:08.887 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:08.887 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.887 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.887 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head, version: 81'75, data_included: [0~49152,1097728~999424], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head@81'75, size: 2146304, copy_subset: [0~2146304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-18T04:40:08.888 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.782052+0000", 2024-09-18T04:40:08.888 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.032602709999999, 2024-09-18T04:40:08.888 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0037389630000000001, 2024-09-18T04:40:08.888 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:08.888 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:08.888 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:08.888 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.888 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:08.888 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.782052+0000", 2024-09-18T04:40:08.888 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:08.888 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.888 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.888 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:08.888 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.782052+0000", 2024-09-18T04:40:08.889 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:08.889 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.889 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.889 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:08.889 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.782051+0000", 2024-09-18T04:40:08.889 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:08.889 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.889 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.889 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:08.889 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.783658+0000", 2024-09-18T04:40:08.889 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0016072149999999999 2024-09-18T04:40:08.889 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.889 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.889 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:08.889 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.783698+0000", 2024-09-18T04:40:08.889 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9394000000000003e-05 2024-09-18T04:40:08.890 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.890 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.890 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:08.890 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.783733+0000", 2024-09-18T04:40:08.890 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5428e-05 2024-09-18T04:40:08.890 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.890 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.890 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:08.890 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.783854+0000", 2024-09-18T04:40:08.890 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012067599999999999 2024-09-18T04:40:08.890 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.890 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.890 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:08.891 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.783869+0000", 2024-09-18T04:40:08.891 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5554999999999999e-05 2024-09-18T04:40:08.891 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.891 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.891 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:08.891 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.785791+0000", 2024-09-18T04:40:08.891 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001921629 2024-09-18T04:40:08.891 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:08.891 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:08.891 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:08.891 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.891 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.891 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head, version: 81'75, data_included: [2097152~49152], data_size: 49152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head@81'75, size: 2146304, copy_subset: [0~2146304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2146304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:40:08.891 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.787740+0000", 2024-09-18T04:40:08.892 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.026915196000001, 2024-09-18T04:40:08.892 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030127370000000001, 2024-09-18T04:40:08.892 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:08.892 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:08.892 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:08.892 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.892 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:08.892 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.787740+0000", 2024-09-18T04:40:08.892 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:08.892 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.892 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.892 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:08.892 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.787740+0000", 2024-09-18T04:40:08.892 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:08.892 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.892 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.893 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:08.893 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.787738+0000", 2024-09-18T04:40:08.893 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:08.893 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.893 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.893 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:08.893 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.787795+0000", 2024-09-18T04:40:08.893 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6753999999999999e-05 2024-09-18T04:40:08.893 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.893 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.893 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:08.893 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.787826+0000", 2024-09-18T04:40:08.893 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0617000000000002e-05 2024-09-18T04:40:08.893 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.894 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.894 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:08.894 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.787853+0000", 2024-09-18T04:40:08.894 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7538999999999998e-05 2024-09-18T04:40:08.894 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.894 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.894 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:08.894 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.787963+0000", 2024-09-18T04:40:08.894 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000109387 2024-09-18T04:40:08.894 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.894 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.894 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:08.894 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.787973+0000", 2024-09-18T04:40:08.894 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0781999999999999e-05 2024-09-18T04:40:08.894 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.895 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.895 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:08.895 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.790752+0000", 2024-09-18T04:40:08.895 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027788520000000001 2024-09-18T04:40:08.895 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:08.895 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:08.895 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:08.895 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.895 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.895 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.5 87/86 [PushOp(2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:7, version: 79'61, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:7@79'61, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:40:08.895 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.791668+0000", 2024-09-18T04:40:08.895 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.022986502999998, 2024-09-18T04:40:08.895 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025634299999999998, 2024-09-18T04:40:08.896 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:08.896 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:08.896 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:08.896 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.896 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:08.896 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.791668+0000", 2024-09-18T04:40:08.896 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:08.896 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.896 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.896 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:08.896 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.791668+0000", 2024-09-18T04:40:08.896 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:08.896 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.896 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.896 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:08.897 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.791667+0000", 2024-09-18T04:40:08.897 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:08.897 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.897 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.897 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:08.897 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.792071+0000", 2024-09-18T04:40:08.897 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00040348599999999999 2024-09-18T04:40:08.897 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.897 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.897 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:08.897 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.792094+0000", 2024-09-18T04:40:08.897 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3105e-05 2024-09-18T04:40:08.897 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.897 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.898 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:08.898 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.792112+0000", 2024-09-18T04:40:08.898 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8502999999999999e-05 2024-09-18T04:40:08.898 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.898 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.898 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:08.898 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.792187+0000", 2024-09-18T04:40:08.898 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5074000000000007e-05 2024-09-18T04:40:08.898 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.898 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.898 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:08.898 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.792197+0000", 2024-09-18T04:40:08.898 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.2979999999999997e-06 2024-09-18T04:40:08.898 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.898 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.898 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:08.899 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.794232+0000", 2024-09-18T04:40:08.899 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002035149 2024-09-18T04:40:08.899 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:08.899 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:08.899 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:08.899 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.899 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.899 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.5 87/86 [PushOp(2:b227e6b1:::rbd_data.125fc09df6d1.000000000000000e:7, version: 79'63, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:b227e6b1:::rbd_data.125fc09df6d1.000000000000000e:7@79'63, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:40:08.899 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.813128+0000", 2024-09-18T04:40:08.899 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.001526642000002, 2024-09-18T04:40:08.899 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025359520000000002, 2024-09-18T04:40:08.899 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:08.899 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:08.899 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:08.899 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.900 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:08.900 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.813128+0000", 2024-09-18T04:40:08.900 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:08.900 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.900 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.900 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:08.900 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.813128+0000", 2024-09-18T04:40:08.900 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:08.900 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.900 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.900 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:08.900 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.813127+0000", 2024-09-18T04:40:08.900 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:08.900 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.901 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.901 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:08.901 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.813544+0000", 2024-09-18T04:40:08.901 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000417248 2024-09-18T04:40:08.901 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.901 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.901 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:08.901 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.813571+0000", 2024-09-18T04:40:08.901 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6789000000000001e-05 2024-09-18T04:40:08.901 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.901 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.901 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:08.901 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.813589+0000", 2024-09-18T04:40:08.902 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8247e-05 2024-09-18T04:40:08.902 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.902 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.902 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:08.902 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.813698+0000", 2024-09-18T04:40:08.902 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000108937 2024-09-18T04:40:08.902 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.902 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.902 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:08.902 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.813715+0000", 2024-09-18T04:40:08.902 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7110000000000001e-05 2024-09-18T04:40:08.902 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:08.902 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:08.902 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:08.902 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.815664+0000", 2024-09-18T04:40:08.903 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001948946 2024-09-18T04:40:08.903 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:08.903 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:08.903 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:08.903 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:08.903 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:08.903 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:40:08.903 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-18T04:40:08.905 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:40:08.899+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:40:08.905 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:40:08.995 INFO:tasks.workunit.client.0.smithi115.stderr:++ rbd journal inspect --pool rbd --image testrbdnbd17342 2024-09-18T04:40:08.995 INFO:tasks.workunit.client.0.smithi115.stderr:++ awk '/entries inspected/ {print $1}' 2024-09-18T04:40:09.005 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:40:09.000+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:40:09.017 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:40:09.018 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:40:09.018 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:40:09.018 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:40:09.028 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:40:09.106 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:40:09.103+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:40:09.129 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:40:09.129 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:40:09.129 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:40:09.129 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:40:09.129 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:40:09.139 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:40:09.208 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:40:09.203+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:40:09.305 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:40:09.305 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:40:09.305 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:40:09.305 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:40:09.305 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.305 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:37 2.5 e51/28 2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:head v 51'1, mlcod=0'0)", 2024-09-18T04:40:09.306 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:40:09.306 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 87.007879861000006, 2024-09-18T04:40:09.306 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012279973, 2024-09-18T04:40:09.306 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:09.306 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:09.306 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:09.306 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.306 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:09.306 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:40:09.306 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.306 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.306 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.307 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:09.307 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:40:09.307 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.307 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.307 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.307 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:09.307 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292468+0000", 2024-09-18T04:40:09.307 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:40:09.307 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.307 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.307 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:09.308 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295837+0000", 2024-09-18T04:40:09.308 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0033691699999999999 2024-09-18T04:40:09.308 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.308 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.308 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:09.308 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295860+0000", 2024-09-18T04:40:09.308 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2756999999999999e-05 2024-09-18T04:40:09.308 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.308 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.308 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:09.308 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295872+0000", 2024-09-18T04:40:09.309 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2442e-05 2024-09-18T04:40:09.309 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.309 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.309 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:09.309 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295934+0000", 2024-09-18T04:40:09.309 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1187999999999998e-05 2024-09-18T04:40:09.309 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.309 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.309 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:09.309 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295967+0000", 2024-09-18T04:40:09.309 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3185999999999997e-05 2024-09-18T04:40:09.310 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.310 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.310 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:09.310 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304697+0000", 2024-09-18T04:40:09.310 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0087298059999999997 2024-09-18T04:40:09.310 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.310 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.310 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:09.310 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304749+0000", 2024-09-18T04:40:09.310 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1922000000000002e-05 2024-09-18T04:40:09.310 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.311 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:09.311 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.311 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.311 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.311 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:49 2.5 e51/28 2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head v 51'2, mlcod=0'0)", 2024-09-18T04:40:09.311 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:40:09.311 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 86.974553022999999, 2024-09-18T04:40:09.311 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018033041999999999, 2024-09-18T04:40:09.311 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:09.311 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:09.312 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:09.312 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.312 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:09.312 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:40:09.312 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.312 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.312 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.312 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:09.312 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:40:09.312 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.313 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.313 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.313 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:09.313 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325793+0000", 2024-09-18T04:40:09.313 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:40:09.313 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.313 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.313 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:09.313 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329038+0000", 2024-09-18T04:40:09.313 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0032444010000000001 2024-09-18T04:40:09.314 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.314 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.314 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:09.314 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329065+0000", 2024-09-18T04:40:09.314 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7500000000000001e-05 2024-09-18T04:40:09.314 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.314 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.314 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:09.314 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329077+0000", 2024-09-18T04:40:09.314 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1919999999999999e-05 2024-09-18T04:40:09.314 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.315 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.315 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:09.315 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329149+0000", 2024-09-18T04:40:09.315 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2349999999999997e-05 2024-09-18T04:40:09.315 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.315 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.315 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:09.315 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329192+0000", 2024-09-18T04:40:09.315 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2744000000000003e-05 2024-09-18T04:40:09.315 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.315 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.316 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:09.316 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343773+0000", 2024-09-18T04:40:09.316 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014581246000000001 2024-09-18T04:40:09.316 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.316 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.316 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:09.316 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343829+0000", 2024-09-18T04:40:09.316 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5143999999999999e-05 2024-09-18T04:40:09.316 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.316 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:09.316 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.317 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.317 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.317 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:47 2.4 e51/17 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 51'3, mlcod=51'1)", 2024-09-18T04:40:09.317 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:40:09.317 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 86.973972758000002, 2024-09-18T04:40:09.317 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018960186, 2024-09-18T04:40:09.317 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:09.317 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:09.317 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:09.317 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.317 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:09.317 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:40:09.318 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.318 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.318 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.318 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:09.318 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:40:09.318 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.318 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.318 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.318 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:09.318 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326374+0000", 2024-09-18T04:40:09.318 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:09.319 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.319 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.319 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:09.319 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330883+0000", 2024-09-18T04:40:09.319 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0045084740000000002 2024-09-18T04:40:09.319 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.319 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.319 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:09.319 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330901+0000", 2024-09-18T04:40:09.319 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7890999999999999e-05 2024-09-18T04:40:09.319 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.319 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.320 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:09.320 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330912+0000", 2024-09-18T04:40:09.320 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1041e-05 2024-09-18T04:40:09.320 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.320 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.320 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:09.320 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.331003+0000", 2024-09-18T04:40:09.320 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1323999999999995e-05 2024-09-18T04:40:09.320 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.320 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.320 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:09.320 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.331030+0000", 2024-09-18T04:40:09.321 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7351999999999998e-05 2024-09-18T04:40:09.321 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.321 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.321 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:09.321 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345287+0000", 2024-09-18T04:40:09.321 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014256332 2024-09-18T04:40:09.321 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.321 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.321 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:09.321 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345336+0000", 2024-09-18T04:40:09.321 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9356e-05 2024-09-18T04:40:09.322 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.322 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:09.322 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.322 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.322 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.322 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:58 2.5 e51/28 2:b227e6b1:::rbd_data.125fc09df6d1.000000000000000e:head v 51'3, mlcod=51'1)", 2024-09-18T04:40:09.322 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:40:09.322 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 86.933546131, 2024-09-18T04:40:09.322 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016513678, 2024-09-18T04:40:09.322 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:09.322 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:09.322 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:09.323 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.323 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:09.323 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:40:09.323 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.323 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.323 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.323 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:09.323 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:40:09.323 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.323 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.323 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.324 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:09.324 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366800+0000", 2024-09-18T04:40:09.324 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:40:09.324 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.324 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.324 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:09.324 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368644+0000", 2024-09-18T04:40:09.324 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001844399 2024-09-18T04:40:09.324 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.324 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.325 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:09.325 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368681+0000", 2024-09-18T04:40:09.325 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6526999999999998e-05 2024-09-18T04:40:09.325 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.325 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.325 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:09.325 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368698+0000", 2024-09-18T04:40:09.325 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7272999999999999e-05 2024-09-18T04:40:09.325 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.325 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.325 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:09.326 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368773+0000", 2024-09-18T04:40:09.326 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4862000000000003e-05 2024-09-18T04:40:09.326 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.326 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.326 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:09.326 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368801+0000", 2024-09-18T04:40:09.326 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7925000000000001e-05 2024-09-18T04:40:09.326 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.326 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.326 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:09.326 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383261+0000", 2024-09-18T04:40:09.326 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014460377999999999 2024-09-18T04:40:09.326 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.326 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.327 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:09.327 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383316+0000", 2024-09-18T04:40:09.327 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4623999999999997e-05 2024-09-18T04:40:09.327 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.327 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:09.327 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.327 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.328 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.328 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:56 2.4 e51/17 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 51'4, mlcod=51'2)", 2024-09-18T04:40:09.328 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:40:09.328 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 86.931542105000005, 2024-09-18T04:40:09.328 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015105221, 2024-09-18T04:40:09.328 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:09.328 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:09.328 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:09.328 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.329 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:09.329 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:40:09.329 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.329 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.329 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.329 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:09.329 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:40:09.329 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.329 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.329 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.329 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:09.329 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368805+0000", 2024-09-18T04:40:09.329 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:09.329 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.329 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.330 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:09.330 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371603+0000", 2024-09-18T04:40:09.330 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027982660000000002 2024-09-18T04:40:09.330 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.330 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.330 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:09.330 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371633+0000", 2024-09-18T04:40:09.330 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0073000000000002e-05 2024-09-18T04:40:09.330 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.330 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.330 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:09.330 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371652+0000", 2024-09-18T04:40:09.331 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8597000000000002e-05 2024-09-18T04:40:09.331 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.331 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.331 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:09.331 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371752+0000", 2024-09-18T04:40:09.331 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9772999999999995e-05 2024-09-18T04:40:09.331 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.331 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.331 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:09.331 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371793+0000", 2024-09-18T04:40:09.331 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1173e-05 2024-09-18T04:40:09.332 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.332 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.332 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:09.332 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383875+0000", 2024-09-18T04:40:09.332 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012081852000000001 2024-09-18T04:40:09.332 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.332 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.332 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:09.332 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383912+0000", 2024-09-18T04:40:09.332 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7045000000000002e-05 2024-09-18T04:40:09.332 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.332 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:09.332 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.332 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.332 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.333 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:36 2.5 e51/28 2:bd5b8dd8:ns::rbd_data.133e813ab5bc.0000000000000002:head v 51'4, mlcod=51'2)", 2024-09-18T04:40:09.333 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:40:09.333 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 86.762823114, 2024-09-18T04:40:09.333 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020758161000000001, 2024-09-18T04:40:09.333 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:09.333 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:09.333 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:09.333 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.333 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:09.333 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:40:09.333 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.333 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.333 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.333 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:09.334 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:40:09.334 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.334 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.334 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.334 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:09.334 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537523+0000", 2024-09-18T04:40:09.334 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:09.334 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.334 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.334 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:09.334 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541578+0000", 2024-09-18T04:40:09.335 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0040549710000000001 2024-09-18T04:40:09.335 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.335 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.335 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:09.335 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541608+0000", 2024-09-18T04:40:09.336 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9349e-05 2024-09-18T04:40:09.336 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.336 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.336 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:09.336 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541621+0000", 2024-09-18T04:40:09.336 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2736e-05 2024-09-18T04:40:09.337 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.337 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.337 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:09.337 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541715+0000", 2024-09-18T04:40:09.337 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4197999999999995e-05 2024-09-18T04:40:09.338 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.338 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.338 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:09.338 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541755+0000", 2024-09-18T04:40:09.338 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9818999999999999e-05 2024-09-18T04:40:09.338 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.338 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.338 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:09.338 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558217+0000", 2024-09-18T04:40:09.338 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016462234999999999 2024-09-18T04:40:09.338 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.339 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.339 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:09.339 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558284+0000", 2024-09-18T04:40:09.339 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6774999999999997e-05 2024-09-18T04:40:09.339 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.339 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:09.339 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.339 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.339 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.339 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:39 2.4 e51/17 2:304364d1:ns::rbd_data.133e813ab5bc.0000000000000004:head v 51'5, mlcod=51'3)", 2024-09-18T04:40:09.339 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:40:09.339 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 86.754683986999993, 2024-09-18T04:40:09.339 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016777558000000001, 2024-09-18T04:40:09.340 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:09.340 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:09.340 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:09.340 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.340 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:09.340 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:40:09.340 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.340 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.340 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.340 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:09.340 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:40:09.340 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.340 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.340 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.341 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:09.341 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545663+0000", 2024-09-18T04:40:09.341 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:09.341 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.341 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.342 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:09.342 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548286+0000", 2024-09-18T04:40:09.342 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026233390000000001 2024-09-18T04:40:09.342 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.342 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.342 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:09.342 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548305+0000", 2024-09-18T04:40:09.343 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8425000000000001e-05 2024-09-18T04:40:09.343 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.343 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.343 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:09.343 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548316+0000", 2024-09-18T04:40:09.343 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1163e-05 2024-09-18T04:40:09.343 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.343 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.343 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:09.343 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548379+0000", 2024-09-18T04:40:09.343 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2780000000000005e-05 2024-09-18T04:40:09.343 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.343 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.343 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:09.344 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548399+0000", 2024-09-18T04:40:09.344 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0080000000000001e-05 2024-09-18T04:40:09.344 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.344 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.344 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:09.344 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562383+0000", 2024-09-18T04:40:09.344 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013984671000000001 2024-09-18T04:40:09.344 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.344 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.344 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:09.344 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562442+0000", 2024-09-18T04:40:09.344 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8700999999999999e-05 2024-09-18T04:40:09.344 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.344 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:09.345 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.345 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.345 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.345 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:37 2.5 e51/28 2:bf39b8e4:ns::rbd_data.133e813ab5bc.0000000000000003:head v 51'5, mlcod=51'2)", 2024-09-18T04:40:09.345 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:40:09.345 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 86.751815718000003, 2024-09-18T04:40:09.345 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014264492, 2024-09-18T04:40:09.345 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:09.345 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:09.345 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:09.345 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.345 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:09.345 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:40:09.346 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.346 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.346 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.346 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:09.346 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:40:09.346 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.346 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.346 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.346 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:09.346 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548531+0000", 2024-09-18T04:40:09.346 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:09.346 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.346 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.346 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:09.347 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551500+0000", 2024-09-18T04:40:09.347 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0029688129999999998 2024-09-18T04:40:09.347 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.347 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.347 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:09.347 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551516+0000", 2024-09-18T04:40:09.347 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5994e-05 2024-09-18T04:40:09.347 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.348 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.348 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:09.348 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551526+0000", 2024-09-18T04:40:09.348 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9299999999999998e-06 2024-09-18T04:40:09.348 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.348 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.348 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:09.348 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551577+0000", 2024-09-18T04:40:09.348 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0871000000000001e-05 2024-09-18T04:40:09.348 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.348 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.348 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:09.348 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551602+0000", 2024-09-18T04:40:09.349 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4813000000000001e-05 2024-09-18T04:40:09.349 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.349 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.349 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:09.349 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562741+0000", 2024-09-18T04:40:09.349 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011139428999999999 2024-09-18T04:40:09.349 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.349 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.349 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:09.349 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562797+0000", 2024-09-18T04:40:09.349 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6329000000000002e-05 2024-09-18T04:40:09.349 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.349 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:09.349 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.349 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.350 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.350 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:49 2.4 e51/17 2:20f0b0c9:ns::rbd_data.133e813ab5bc.0000000000000009:head v 51'6, mlcod=51'4)", 2024-09-18T04:40:09.350 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:40:09.350 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 86.720381160000002, 2024-09-18T04:40:09.350 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013849046, 2024-09-18T04:40:09.350 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:09.350 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:09.350 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:09.350 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.350 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:09.350 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:40:09.350 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.350 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.350 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.351 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:09.351 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:40:09.351 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.351 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.351 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.351 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:09.351 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579966+0000", 2024-09-18T04:40:09.351 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:09.351 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.351 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.351 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:09.351 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582324+0000", 2024-09-18T04:40:09.351 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023578890000000002 2024-09-18T04:40:09.351 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.352 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.352 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:09.352 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582355+0000", 2024-09-18T04:40:09.352 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1485000000000001e-05 2024-09-18T04:40:09.352 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.352 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.352 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:09.352 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582371+0000", 2024-09-18T04:40:09.352 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6147999999999999e-05 2024-09-18T04:40:09.352 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.352 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.352 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:09.352 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582467+0000", 2024-09-18T04:40:09.352 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6059999999999998e-05 2024-09-18T04:40:09.353 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.353 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.353 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:09.353 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582505+0000", 2024-09-18T04:40:09.353 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7679000000000003e-05 2024-09-18T04:40:09.353 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.354 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.354 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:09.354 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593743+0000", 2024-09-18T04:40:09.354 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011238356 2024-09-18T04:40:09.354 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.354 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.354 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:09.355 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593816+0000", 2024-09-18T04:40:09.355 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2935000000000006e-05 2024-09-18T04:40:09.355 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.355 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:09.355 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.355 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.355 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.355 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:53 2.4 e51/17 2:39f7f174:ns::rbd_data.133e813ab5bc.000000000000000b:head v 51'7, mlcod=51'4)", 2024-09-18T04:40:09.355 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:40:09.355 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 86.705680005000005, 2024-09-18T04:40:09.355 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0079914159999999995, 2024-09-18T04:40:09.356 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:09.356 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:09.356 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:09.356 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.356 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:09.356 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:40:09.356 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.356 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.356 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.356 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:09.356 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:40:09.356 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.356 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.356 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.357 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:09.357 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594667+0000", 2024-09-18T04:40:09.357 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:09.357 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.357 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.357 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:09.358 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596446+0000", 2024-09-18T04:40:09.358 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001779033 2024-09-18T04:40:09.358 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.358 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.358 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:09.358 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596472+0000", 2024-09-18T04:40:09.358 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6120000000000001e-05 2024-09-18T04:40:09.358 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.358 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.358 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:09.358 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596500+0000", 2024-09-18T04:40:09.358 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7518e-05 2024-09-18T04:40:09.358 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.359 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.359 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:09.359 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596564+0000", 2024-09-18T04:40:09.359 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3880999999999994e-05 2024-09-18T04:40:09.359 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.359 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.359 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:09.359 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596596+0000", 2024-09-18T04:40:09.359 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2638999999999997e-05 2024-09-18T04:40:09.359 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.359 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.359 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:09.359 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602605+0000", 2024-09-18T04:40:09.360 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0060080699999999999 2024-09-18T04:40:09.360 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.360 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.360 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:09.360 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602660+0000", 2024-09-18T04:40:09.360 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5396000000000003e-05 2024-09-18T04:40:09.360 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.360 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:09.360 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.360 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.360 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.360 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:60 2.5 e51/28 2:a06f51e4:ns::rbd_data.133e813ab5bc.000000000000000f:head v 51'6, mlcod=51'4)", 2024-09-18T04:40:09.360 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:40:09.360 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 86.690863019999995, 2024-09-18T04:40:09.360 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0093954989999999999, 2024-09-18T04:40:09.361 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:09.361 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:09.361 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:09.361 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.361 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:09.361 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:40:09.361 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.361 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.361 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.361 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:09.362 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:40:09.362 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.362 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.362 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.362 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:09.362 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609483+0000", 2024-09-18T04:40:09.362 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:09.362 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.362 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.362 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:09.362 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611767+0000", 2024-09-18T04:40:09.363 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002283724 2024-09-18T04:40:09.363 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.363 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.363 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:09.363 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611791+0000", 2024-09-18T04:40:09.363 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3547e-05 2024-09-18T04:40:09.363 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.363 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.363 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:09.363 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611800+0000", 2024-09-18T04:40:09.363 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4979999999999995e-06 2024-09-18T04:40:09.363 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.363 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.363 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:09.364 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611864+0000", 2024-09-18T04:40:09.364 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3974999999999996e-05 2024-09-18T04:40:09.364 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.364 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.364 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:09.364 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611887+0000", 2024-09-18T04:40:09.364 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2633000000000002e-05 2024-09-18T04:40:09.364 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.364 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.364 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:09.364 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.618777+0000", 2024-09-18T04:40:09.364 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0068897979999999999 2024-09-18T04:40:09.364 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.365 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.365 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:09.365 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.618881+0000", 2024-09-18T04:40:09.365 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010437699999999999 2024-09-18T04:40:09.365 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.365 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:09.365 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.365 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.365 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:663 2.4 e57/17 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 57'13, mlcod=57'11)", 2024-09-18T04:40:09.365 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.355300+0000", 2024-09-18T04:40:09.365 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 77.945048084999996, 2024-09-18T04:40:09.366 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0049192760000000002, 2024-09-18T04:40:09.366 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:09.366 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:09.366 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:09.366 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.366 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:09.366 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355300+0000", 2024-09-18T04:40:09.366 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.366 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.366 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.366 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:09.366 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355300+0000", 2024-09-18T04:40:09.366 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.367 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.367 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:09.367 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355299+0000", 2024-09-18T04:40:09.367 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:09.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.367 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.367 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:09.367 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355876+0000", 2024-09-18T04:40:09.367 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00057730900000000003 2024-09-18T04:40:09.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.367 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.367 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:09.367 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355897+0000", 2024-09-18T04:40:09.367 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0477000000000001e-05 2024-09-18T04:40:09.368 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.368 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.368 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:09.368 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355905+0000", 2024-09-18T04:40:09.368 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5739999999999996e-06 2024-09-18T04:40:09.368 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.369 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.369 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:09.369 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355989+0000", 2024-09-18T04:40:09.369 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4002e-05 2024-09-18T04:40:09.369 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.369 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.369 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:09.369 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.356041+0000", 2024-09-18T04:40:09.369 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1347000000000002e-05 2024-09-18T04:40:09.369 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.369 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.369 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:09.369 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.360172+0000", 2024-09-18T04:40:09.369 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0041315090000000002 2024-09-18T04:40:09.370 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.370 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.370 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:09.370 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.360220+0000", 2024-09-18T04:40:09.370 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7584999999999999e-05 2024-09-18T04:40:09.370 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.370 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:09.370 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.370 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.370 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.370 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:679 2.5 e57/28 2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head v 57'20, mlcod=57'18)", 2024-09-18T04:40:09.370 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.445316+0000", 2024-09-18T04:40:09.370 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 77.855032608000002, 2024-09-18T04:40:09.370 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0047016760000000001, 2024-09-18T04:40:09.371 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:09.371 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:09.371 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:09.371 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.371 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:09.371 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445316+0000", 2024-09-18T04:40:09.371 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.371 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.371 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:09.371 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445316+0000", 2024-09-18T04:40:09.371 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.371 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.372 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:09.372 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445314+0000", 2024-09-18T04:40:09.372 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:09.372 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.372 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.373 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:09.373 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445947+0000", 2024-09-18T04:40:09.373 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00063298900000000003 2024-09-18T04:40:09.373 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.373 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.374 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:09.374 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445968+0000", 2024-09-18T04:40:09.374 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.054e-05 2024-09-18T04:40:09.374 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.374 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.374 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:09.374 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445981+0000", 2024-09-18T04:40:09.375 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2697000000000001e-05 2024-09-18T04:40:09.375 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.375 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.375 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:09.375 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.446102+0000", 2024-09-18T04:40:09.375 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012189300000000001 2024-09-18T04:40:09.375 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.375 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.375 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:09.375 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.446151+0000", 2024-09-18T04:40:09.375 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8279000000000003e-05 2024-09-18T04:40:09.375 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.376 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.376 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:09.376 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.449953+0000", 2024-09-18T04:40:09.376 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003801979 2024-09-18T04:40:09.376 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.376 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.376 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:09.376 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.450018+0000", 2024-09-18T04:40:09.376 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4873e-05 2024-09-18T04:40:09.376 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.376 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:09.376 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.376 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.377 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.377 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:699 2.4 e57/17 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 57'17, mlcod=57'15)", 2024-09-18T04:40:09.377 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.566073+0000", 2024-09-18T04:40:09.377 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 77.734275095000001, 2024-09-18T04:40:09.377 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055398380000000001, 2024-09-18T04:40:09.377 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:09.377 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:09.377 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:09.377 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.377 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:09.377 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566073+0000", 2024-09-18T04:40:09.377 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.378 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.378 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:09.378 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566073+0000", 2024-09-18T04:40:09.378 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.378 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.378 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.378 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:09.378 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566071+0000", 2024-09-18T04:40:09.378 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:09.378 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.378 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.378 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:09.378 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566637+0000", 2024-09-18T04:40:09.378 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00056516699999999999 2024-09-18T04:40:09.379 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.379 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.379 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:09.379 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566668+0000", 2024-09-18T04:40:09.379 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1544999999999997e-05 2024-09-18T04:40:09.379 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.379 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.379 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:09.380 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566685+0000", 2024-09-18T04:40:09.380 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6857000000000001e-05 2024-09-18T04:40:09.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.380 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.380 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:09.380 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566816+0000", 2024-09-18T04:40:09.380 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013092799999999999 2024-09-18T04:40:09.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.380 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.380 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:09.380 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566875+0000", 2024-09-18T04:40:09.380 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9583999999999997e-05 2024-09-18T04:40:09.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.381 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.381 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:09.381 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.571567+0000", 2024-09-18T04:40:09.381 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.004691682 2024-09-18T04:40:09.381 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.381 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.381 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:09.381 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.571613+0000", 2024-09-18T04:40:09.381 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6125000000000003e-05 2024-09-18T04:40:09.381 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.381 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:09.381 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.381 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.382 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.382 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:98 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'27, mlcod=60'25)", 2024-09-18T04:40:09.382 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.847978+0000", 2024-09-18T04:40:09.382 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 68.452370536000004, 2024-09-18T04:40:09.382 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0044617709999999998, 2024-09-18T04:40:09.382 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:09.382 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:09.382 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:09.382 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.382 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:09.382 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.847978+0000", 2024-09-18T04:40:09.382 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.382 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.382 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:09.383 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.847978+0000", 2024-09-18T04:40:09.383 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.383 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.383 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:09.383 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.847977+0000", 2024-09-18T04:40:09.383 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:09.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.383 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.383 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:09.383 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849038+0000", 2024-09-18T04:40:09.383 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0010614820000000001 2024-09-18T04:40:09.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.384 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.384 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:09.384 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849065+0000", 2024-09-18T04:40:09.384 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7127999999999998e-05 2024-09-18T04:40:09.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.384 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.384 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:09.385 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849082+0000", 2024-09-18T04:40:09.385 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7170999999999999e-05 2024-09-18T04:40:09.385 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.385 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.385 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:09.385 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849188+0000", 2024-09-18T04:40:09.385 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010605800000000001 2024-09-18T04:40:09.385 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.385 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.385 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:09.385 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849233+0000", 2024-09-18T04:40:09.385 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4579000000000001e-05 2024-09-18T04:40:09.385 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.385 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.386 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:09.386 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.852370+0000", 2024-09-18T04:40:09.386 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00313751 2024-09-18T04:40:09.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.386 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.386 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:09.386 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.852440+0000", 2024-09-18T04:40:09.386 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9238000000000004e-05 2024-09-18T04:40:09.386 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.386 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:09.386 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.387 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:100 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'29, mlcod=64'27)", 2024-09-18T04:40:09.387 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.858766+0000", 2024-09-18T04:40:09.387 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 68.441582206999996, 2024-09-18T04:40:09.387 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0061964840000000004, 2024-09-18T04:40:09.387 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:09.387 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:09.387 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:09.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.387 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:09.387 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.858766+0000", 2024-09-18T04:40:09.387 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.387 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:09.388 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.858766+0000", 2024-09-18T04:40:09.388 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.388 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.388 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:09.388 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.858765+0000", 2024-09-18T04:40:09.388 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:09.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.388 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.388 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:09.388 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861612+0000", 2024-09-18T04:40:09.389 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028473169999999998 2024-09-18T04:40:09.389 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.389 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.389 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:09.389 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861643+0000", 2024-09-18T04:40:09.389 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1356000000000003e-05 2024-09-18T04:40:09.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.390 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.390 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:09.390 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861656+0000", 2024-09-18T04:40:09.390 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2899e-05 2024-09-18T04:40:09.391 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.391 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.391 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:09.391 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861773+0000", 2024-09-18T04:40:09.391 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011637999999999999 2024-09-18T04:40:09.391 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.391 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.391 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:09.391 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861808+0000", 2024-09-18T04:40:09.391 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5002999999999997e-05 2024-09-18T04:40:09.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.392 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.392 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:09.392 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.864876+0000", 2024-09-18T04:40:09.392 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030680719999999998 2024-09-18T04:40:09.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.392 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.392 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:09.392 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.864963+0000", 2024-09-18T04:40:09.392 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7114000000000007e-05 2024-09-18T04:40:09.392 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.392 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:09.392 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.393 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.393 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:102 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'31, mlcod=64'29)", 2024-09-18T04:40:09.393 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.872098+0000", 2024-09-18T04:40:09.393 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 68.428250950000006, 2024-09-18T04:40:09.393 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0064676450000000002, 2024-09-18T04:40:09.393 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:09.393 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:09.393 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:09.393 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.393 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:09.393 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.872098+0000", 2024-09-18T04:40:09.393 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.394 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.394 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:09.394 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.872098+0000", 2024-09-18T04:40:09.394 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.394 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.394 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:09.394 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.872096+0000", 2024-09-18T04:40:09.394 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:09.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.394 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.394 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:09.394 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875176+0000", 2024-09-18T04:40:09.394 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030799809999999999 2024-09-18T04:40:09.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.395 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.395 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:09.395 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875204+0000", 2024-09-18T04:40:09.395 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8025999999999999e-05 2024-09-18T04:40:09.395 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.395 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.395 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:09.395 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875216+0000", 2024-09-18T04:40:09.395 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2072999999999999e-05 2024-09-18T04:40:09.395 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.395 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.395 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:09.395 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875307+0000", 2024-09-18T04:40:09.395 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0373999999999999e-05 2024-09-18T04:40:09.395 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.395 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.396 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:09.396 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875349+0000", 2024-09-18T04:40:09.396 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2299000000000003e-05 2024-09-18T04:40:09.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.396 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.396 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:09.396 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.878496+0000", 2024-09-18T04:40:09.396 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0031468960000000002 2024-09-18T04:40:09.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.396 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.396 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:09.397 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.878565+0000", 2024-09-18T04:40:09.397 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9465e-05 2024-09-18T04:40:09.397 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.397 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:09.397 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.397 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.397 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:104 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'33, mlcod=64'31)", 2024-09-18T04:40:09.397 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.885100+0000", 2024-09-18T04:40:09.397 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 68.415248414999994, 2024-09-18T04:40:09.397 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0051888159999999997, 2024-09-18T04:40:09.397 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:09.398 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:09.398 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:09.398 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.398 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:09.398 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.885100+0000", 2024-09-18T04:40:09.398 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.398 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.398 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.398 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:09.398 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.885100+0000", 2024-09-18T04:40:09.398 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.398 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.398 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.398 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:09.398 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.885098+0000", 2024-09-18T04:40:09.399 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:09.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.399 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.399 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:09.399 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886809+0000", 2024-09-18T04:40:09.399 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017110459999999999 2024-09-18T04:40:09.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.399 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.399 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:09.399 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886842+0000", 2024-09-18T04:40:09.399 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2638999999999997e-05 2024-09-18T04:40:09.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.399 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.399 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:09.399 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886855+0000", 2024-09-18T04:40:09.400 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3380000000000001e-05 2024-09-18T04:40:09.400 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.400 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.400 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:09.400 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886939+0000", 2024-09-18T04:40:09.400 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3733999999999995e-05 2024-09-18T04:40:09.400 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.400 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.400 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:09.400 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886981+0000", 2024-09-18T04:40:09.400 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1956000000000003e-05 2024-09-18T04:40:09.400 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.400 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.400 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:09.400 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.890201+0000", 2024-09-18T04:40:09.400 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003219927 2024-09-18T04:40:09.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.401 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.401 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:09.401 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.890289+0000", 2024-09-18T04:40:09.401 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7997999999999993e-05 2024-09-18T04:40:09.401 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.401 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:09.401 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.401 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.401 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:145 2.4 e64/62 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 64'37, mlcod=64'35)", 2024-09-18T04:40:09.401 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:01.099398+0000", 2024-09-18T04:40:09.401 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 68.200950926999994, 2024-09-18T04:40:09.401 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0050078459999999998, 2024-09-18T04:40:09.401 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:09.402 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:09.402 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:09.402 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.402 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:09.402 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.099398+0000", 2024-09-18T04:40:09.402 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.402 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.402 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.402 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:09.402 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.099398+0000", 2024-09-18T04:40:09.402 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.402 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.402 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.402 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:09.402 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.099396+0000", 2024-09-18T04:40:09.402 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:09.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.403 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.403 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:09.403 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100199+0000", 2024-09-18T04:40:09.403 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00080338000000000002 2024-09-18T04:40:09.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.403 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.403 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:09.403 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100230+0000", 2024-09-18T04:40:09.403 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0716000000000002e-05 2024-09-18T04:40:09.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.404 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.404 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:09.404 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100243+0000", 2024-09-18T04:40:09.404 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3359e-05 2024-09-18T04:40:09.404 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.404 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.405 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:09.405 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100337+0000", 2024-09-18T04:40:09.405 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4387000000000003e-05 2024-09-18T04:40:09.405 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.405 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.406 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:09.406 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100396+0000", 2024-09-18T04:40:09.406 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8164000000000001e-05 2024-09-18T04:40:09.406 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.406 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.406 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:09.407 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.104309+0000", 2024-09-18T04:40:09.407 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0039137800000000004 2024-09-18T04:40:09.407 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.407 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.407 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:09.407 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.104405+0000", 2024-09-18T04:40:09.407 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6039999999999995e-05 2024-09-18T04:40:09.407 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.407 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:09.407 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.407 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.408 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.408 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:147 2.4 e64/62 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 64'39, mlcod=64'37)", 2024-09-18T04:40:09.408 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:01.110658+0000", 2024-09-18T04:40:09.408 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 68.189690838000004, 2024-09-18T04:40:09.408 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0046316910000000003, 2024-09-18T04:40:09.408 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:09.408 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:09.408 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:09.408 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.408 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:09.408 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.110658+0000", 2024-09-18T04:40:09.408 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.408 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.409 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.409 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:09.409 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.110658+0000", 2024-09-18T04:40:09.409 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.409 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.409 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.409 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:09.409 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.110656+0000", 2024-09-18T04:40:09.409 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:09.409 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.409 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.409 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:09.409 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111444+0000", 2024-09-18T04:40:09.409 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00078762400000000001 2024-09-18T04:40:09.410 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.410 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.410 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:09.410 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111474+0000", 2024-09-18T04:40:09.410 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0085000000000001e-05 2024-09-18T04:40:09.410 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.410 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.410 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:09.410 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111486+0000", 2024-09-18T04:40:09.410 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2645e-05 2024-09-18T04:40:09.410 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.410 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.411 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:09.411 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111581+0000", 2024-09-18T04:40:09.411 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4432000000000007e-05 2024-09-18T04:40:09.411 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.411 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.411 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:09.411 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111631+0000", 2024-09-18T04:40:09.411 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9901000000000002e-05 2024-09-18T04:40:09.412 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.412 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.412 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:09.412 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.115209+0000", 2024-09-18T04:40:09.412 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003577916 2024-09-18T04:40:09.412 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.412 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.412 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:09.412 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.115289+0000", 2024-09-18T04:40:09.412 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0704999999999994e-05 2024-09-18T04:40:09.412 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.412 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:09.412 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.412 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.413 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:09.413 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:40:09.414 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:40:09.415 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:40:09.412+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:40:09.496 INFO:tasks.workunit.client.0.smithi115.stderr:2024-09-18T04:40:09.491+0000 7fb384bd7700 -1 JournalMetadata: 0x56379d489100 failed to locate client: INSPECT 2024-09-18T04:40:09.506 INFO:tasks.workunit.client.0.smithi115.stderr:+ queue_length=4 2024-09-18T04:40:09.506 INFO:tasks.workunit.client.0.smithi115.stderr:+ tries=1 2024-09-18T04:40:09.506 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1 -lt 10 ']' 2024-09-18T04:40:09.507 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 4 -gt 0 ']' 2024-09-18T04:40:09.507 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd journal inspect --pool rbd --image testrbdnbd17342 --verbose 2024-09-18T04:40:09.515 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:40:09.511+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:40:09.530 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:40:09.531 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:40:09.531 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:40:09.531 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:40:09.541 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:40:09.596 INFO:tasks.workunit.client.0.smithi115.stdout:Entry: tag_id=1, commit_tid=1 2024-09-18T04:40:09.596 INFO:tasks.workunit.client.0.smithi115.stdout:{ 2024-09-18T04:40:09.596 INFO:tasks.workunit.client.0.smithi115.stdout: "event_type": "AioDiscard", 2024-09-18T04:40:09.596 INFO:tasks.workunit.client.0.smithi115.stdout: "offset": 0, 2024-09-18T04:40:09.596 INFO:tasks.workunit.client.0.smithi115.stdout: "length": 4096, 2024-09-18T04:40:09.596 INFO:tasks.workunit.client.0.smithi115.stdout: "discard_granularity_bytes": 4096, 2024-09-18T04:40:09.596 INFO:tasks.workunit.client.0.smithi115.stdout: "timestamp": "2024-09-18T04:40:07.691875+0000" 2024-09-18T04:40:09.596 INFO:tasks.workunit.client.0.smithi115.stdout:} 2024-09-18T04:40:09.597 INFO:tasks.workunit.client.0.smithi115.stdout:Entry: tag_id=1, commit_tid=2 2024-09-18T04:40:09.597 INFO:tasks.workunit.client.0.smithi115.stdout:{ 2024-09-18T04:40:09.597 INFO:tasks.workunit.client.0.smithi115.stdout: "event_type": "AioDiscard", 2024-09-18T04:40:09.597 INFO:tasks.workunit.client.0.smithi115.stdout: "offset": 0, 2024-09-18T04:40:09.597 INFO:tasks.workunit.client.0.smithi115.stdout: "length": 4096, 2024-09-18T04:40:09.597 INFO:tasks.workunit.client.0.smithi115.stdout: "discard_granularity_bytes": 4096, 2024-09-18T04:40:09.597 INFO:tasks.workunit.client.0.smithi115.stdout: "timestamp": "2024-09-18T04:40:07.714307+0000" 2024-09-18T04:40:09.597 INFO:tasks.workunit.client.0.smithi115.stdout:} 2024-09-18T04:40:09.597 INFO:tasks.workunit.client.0.smithi115.stdout:Entry: tag_id=1, commit_tid=3 2024-09-18T04:40:09.597 INFO:tasks.workunit.client.0.smithi115.stdout:{ 2024-09-18T04:40:09.597 INFO:tasks.workunit.client.0.smithi115.stdout: "event_type": "AioDiscard", 2024-09-18T04:40:09.597 INFO:tasks.workunit.client.0.smithi115.stdout: "offset": 4096, 2024-09-18T04:40:09.597 INFO:tasks.workunit.client.0.smithi115.stdout: "length": 4096, 2024-09-18T04:40:09.597 INFO:tasks.workunit.client.0.smithi115.stdout: "discard_granularity_bytes": 4096, 2024-09-18T04:40:09.598 INFO:tasks.workunit.client.0.smithi115.stdout: "timestamp": "2024-09-18T04:40:07.745921+0000" 2024-09-18T04:40:09.598 INFO:tasks.workunit.client.0.smithi115.stdout:} 2024-09-18T04:40:09.598 INFO:tasks.workunit.client.0.smithi115.stdout:Entry: tag_id=1, commit_tid=4 2024-09-18T04:40:09.598 INFO:tasks.workunit.client.0.smithi115.stdout:{ 2024-09-18T04:40:09.598 INFO:tasks.workunit.client.0.smithi115.stdout: "event_type": "AioDiscard", 2024-09-18T04:40:09.598 INFO:tasks.workunit.client.0.smithi115.stdout: "offset": 4096, 2024-09-18T04:40:09.598 INFO:tasks.workunit.client.0.smithi115.stdout: "length": 4096, 2024-09-18T04:40:09.598 INFO:tasks.workunit.client.0.smithi115.stdout: "discard_granularity_bytes": 4096, 2024-09-18T04:40:09.598 INFO:tasks.workunit.client.0.smithi115.stdout: "timestamp": "2024-09-18T04:40:07.765789+0000" 2024-09-18T04:40:09.598 INFO:tasks.workunit.client.0.smithi115.stdout:} 2024-09-18T04:40:09.598 INFO:tasks.workunit.client.0.smithi115.stdout:Summary: 2024-09-18T04:40:09.598 INFO:tasks.workunit.client.0.smithi115.stdout: 4 entries inspected, 0 errors 2024-09-18T04:40:09.605 INFO:tasks.workunit.client.0.smithi115.stderr:2024-09-18T04:40:09.599+0000 7fd2d7fff700 -1 JournalMetadata: 0x561f07c32610 failed to locate client: INSPECT 2024-09-18T04:40:09.611 INFO:tasks.workunit.client.0.smithi115.stderr:+ sleep 1 2024-09-18T04:40:09.616 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:40:09.612+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:40:09.651 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:40:09.651 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:40:09.651 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:40:09.651 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:40:09.651 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:40:09.661 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:40:09.717 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:40:09.712+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:40:09.776 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:40:09.777 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:40:09.777 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:40:09.777 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:40:09.777 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.777 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:33 2.1 e51/24 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head v 51'4, mlcod=51'2)", 2024-09-18T04:40:09.777 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:40:09.777 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 87.492170294999994, 2024-09-18T04:40:09.777 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018702106, 2024-09-18T04:40:09.777 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:09.777 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:09.777 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:09.778 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.778 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:09.778 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:40:09.778 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.778 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.778 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.778 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:09.778 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:40:09.778 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.778 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.778 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.778 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:09.779 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279197+0000", 2024-09-18T04:40:09.779 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:09.779 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.779 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.779 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:09.779 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282682+0000", 2024-09-18T04:40:09.779 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0034848689999999998 2024-09-18T04:40:09.779 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.779 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.779 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:09.779 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282709+0000", 2024-09-18T04:40:09.779 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7599000000000001e-05 2024-09-18T04:40:09.780 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.780 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.780 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:09.780 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282720+0000", 2024-09-18T04:40:09.780 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.092e-05 2024-09-18T04:40:09.780 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.780 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.780 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:09.780 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282791+0000", 2024-09-18T04:40:09.780 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1268999999999998e-05 2024-09-18T04:40:09.780 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.781 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.781 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:09.781 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282843+0000", 2024-09-18T04:40:09.781 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1771999999999998e-05 2024-09-18T04:40:09.781 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.781 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.781 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:09.781 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.297830+0000", 2024-09-18T04:40:09.781 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014987228999999999 2024-09-18T04:40:09.781 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.781 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.781 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:09.781 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.297901+0000", 2024-09-18T04:40:09.781 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0209999999999994e-05 2024-09-18T04:40:09.781 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.782 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:09.782 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.782 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.782 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.782 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:38 2.0 e51/17 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head v 51'10, mlcod=51'8)", 2024-09-18T04:40:09.782 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:40:09.782 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 87.490751864999993, 2024-09-18T04:40:09.782 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.023984472, 2024-09-18T04:40:09.782 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:09.782 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:09.782 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:09.782 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.782 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:09.782 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:40:09.783 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.783 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.783 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.783 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:09.783 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:40:09.783 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.783 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.783 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.783 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:09.783 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280614+0000", 2024-09-18T04:40:09.783 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:40:09.783 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.783 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.783 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:09.783 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293094+0000", 2024-09-18T04:40:09.784 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012479571 2024-09-18T04:40:09.784 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.784 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.784 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:09.784 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293111+0000", 2024-09-18T04:40:09.784 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7354999999999999e-05 2024-09-18T04:40:09.784 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.784 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.784 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:09.784 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293121+0000", 2024-09-18T04:40:09.784 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6979999999999993e-06 2024-09-18T04:40:09.784 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.784 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.784 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:09.785 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293180+0000", 2024-09-18T04:40:09.785 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9333000000000003e-05 2024-09-18T04:40:09.785 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.785 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.785 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:09.785 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293203+0000", 2024-09-18T04:40:09.785 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2633000000000002e-05 2024-09-18T04:40:09.785 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.785 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.785 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:09.785 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304534+0000", 2024-09-18T04:40:09.785 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011330761 2024-09-18T04:40:09.785 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.785 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.785 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:09.786 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304601+0000", 2024-09-18T04:40:09.786 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7657000000000006e-05 2024-09-18T04:40:09.786 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.786 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:09.786 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.786 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.786 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.786 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:46 2.6 2:7ba25245:::rbd_data.125fc09df6d1.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:09.786 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:40:09.786 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 87.463209065000001, 2024-09-18T04:40:09.786 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.035604595000000003, 2024-09-18T04:40:09.786 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:09.786 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:09.786 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:09.786 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:40:09.787 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:40:09.787 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 46 2024-09-18T04:40:09.787 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.787 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:09.787 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.787 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:09.787 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:40:09.787 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.787 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.787 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.787 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:09.787 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:40:09.787 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.787 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.788 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.788 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:09.788 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308158+0000", 2024-09-18T04:40:09.788 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:09.788 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.788 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.788 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:09.788 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313191+0000", 2024-09-18T04:40:09.788 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0050323929999999996 2024-09-18T04:40:09.788 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.788 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.788 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:09.788 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313210+0000", 2024-09-18T04:40:09.789 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9673e-05 2024-09-18T04:40:09.789 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.789 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.789 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:09.789 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313221+0000", 2024-09-18T04:40:09.789 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0444e-05 2024-09-18T04:40:09.789 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.789 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.789 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:09.789 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313294+0000", 2024-09-18T04:40:09.789 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2650000000000004e-05 2024-09-18T04:40:09.789 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.789 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.790 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:09.790 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313493+0000", 2024-09-18T04:40:09.790 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019968300000000001 2024-09-18T04:40:09.790 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.790 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.790 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:40:09.790 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313666+0000", 2024-09-18T04:40:09.790 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00017248599999999999 2024-09-18T04:40:09.790 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.790 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.790 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:09.790 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.323175+0000", 2024-09-18T04:40:09.790 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0095088819999999998 2024-09-18T04:40:09.790 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.791 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.791 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:09.791 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343651+0000", 2024-09-18T04:40:09.791 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020475892999999998 2024-09-18T04:40:09.791 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.791 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.791 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:09.791 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343715+0000", 2024-09-18T04:40:09.791 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4443999999999995e-05 2024-09-18T04:40:09.791 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.791 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.791 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:09.792 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343764+0000", 2024-09-18T04:40:09.792 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9280000000000003e-05 2024-09-18T04:40:09.792 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.792 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:09.792 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.792 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.792 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.792 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:47 2.4 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:09.792 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:40:09.792 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 87.458127028000007, 2024-09-18T04:40:09.792 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.032570693999999997, 2024-09-18T04:40:09.792 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:09.793 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:09.793 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:09.793 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:40:09.793 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:40:09.793 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 47 2024-09-18T04:40:09.793 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.793 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:09.793 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.793 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:09.793 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:40:09.793 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.793 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.793 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.794 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:09.794 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:40:09.794 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.794 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.794 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.794 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:09.794 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313240+0000", 2024-09-18T04:40:09.794 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:09.794 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.794 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.794 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:09.795 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325825+0000", 2024-09-18T04:40:09.795 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012584425999999999 2024-09-18T04:40:09.795 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.795 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.795 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:09.795 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325848+0000", 2024-09-18T04:40:09.795 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3025999999999999e-05 2024-09-18T04:40:09.795 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.795 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.795 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:09.795 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325863+0000", 2024-09-18T04:40:09.796 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5152e-05 2024-09-18T04:40:09.796 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.796 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.796 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:09.797 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325938+0000", 2024-09-18T04:40:09.797 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4982999999999997e-05 2024-09-18T04:40:09.797 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.797 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.797 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:09.797 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326090+0000", 2024-09-18T04:40:09.798 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015207900000000001 2024-09-18T04:40:09.798 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.798 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.798 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:40:09.798 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326242+0000", 2024-09-18T04:40:09.798 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015163500000000001 2024-09-18T04:40:09.798 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.798 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.798 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:09.799 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343015+0000", 2024-09-18T04:40:09.799 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016773414 2024-09-18T04:40:09.799 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.799 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.799 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:09.800 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345664+0000", 2024-09-18T04:40:09.800 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026487870000000001 2024-09-18T04:40:09.800 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.800 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.800 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:09.800 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345739+0000", 2024-09-18T04:40:09.800 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5018999999999994e-05 2024-09-18T04:40:09.800 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.800 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.800 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:09.800 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345812+0000", 2024-09-18T04:40:09.800 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3547000000000006e-05 2024-09-18T04:40:09.800 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.801 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:09.801 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.801 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.801 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.801 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:48 2.6 2:61db7a3a:::rbd_data.125fc09df6d1.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:09.801 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:40:09.801 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 87.445470356000001, 2024-09-18T04:40:09.801 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.025064388, 2024-09-18T04:40:09.801 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:09.801 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:09.801 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:09.801 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:40:09.802 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:40:09.802 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 48 2024-09-18T04:40:09.802 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.802 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:09.802 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.802 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:09.802 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:40:09.802 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.802 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.802 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.802 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:09.802 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:40:09.802 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.802 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.803 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.803 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:09.803 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325897+0000", 2024-09-18T04:40:09.803 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:09.803 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.803 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.803 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:09.803 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333697+0000", 2024-09-18T04:40:09.803 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0078002649999999998 2024-09-18T04:40:09.803 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.803 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.803 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:09.803 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333716+0000", 2024-09-18T04:40:09.804 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8910999999999999e-05 2024-09-18T04:40:09.804 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.804 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.804 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:09.804 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333728+0000", 2024-09-18T04:40:09.804 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1936000000000001e-05 2024-09-18T04:40:09.804 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.804 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.804 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:09.804 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333792+0000", 2024-09-18T04:40:09.805 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4010999999999994e-05 2024-09-18T04:40:09.805 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.805 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.805 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:09.805 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.334082+0000", 2024-09-18T04:40:09.805 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00028955799999999998 2024-09-18T04:40:09.805 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.805 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.805 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:40:09.805 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.334326+0000", 2024-09-18T04:40:09.805 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00024399099999999999 2024-09-18T04:40:09.805 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.806 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.806 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:09.806 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350745+0000", 2024-09-18T04:40:09.806 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016419474999999999 2024-09-18T04:40:09.806 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.806 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.806 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:09.806 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350808+0000", 2024-09-18T04:40:09.806 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3244999999999995e-05 2024-09-18T04:40:09.806 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.806 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.806 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:09.807 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350892+0000", 2024-09-18T04:40:09.807 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4074999999999997e-05 2024-09-18T04:40:09.807 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.807 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.807 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:09.807 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350963+0000", 2024-09-18T04:40:09.807 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0412000000000003e-05 2024-09-18T04:40:09.807 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.807 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:09.807 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.807 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.808 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.808 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:51 2.6 2:775d7b11:::rbd_data.125fc09df6d1.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:09.808 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:40:09.808 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 87.437618220000004, 2024-09-18T04:40:09.808 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.023824221, 2024-09-18T04:40:09.808 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:09.808 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:09.808 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:09.808 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:40:09.808 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:40:09.809 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 51 2024-09-18T04:40:09.809 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.809 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:09.809 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.809 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:09.809 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:40:09.809 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.809 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.809 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.809 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:09.809 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:40:09.810 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.810 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.810 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.810 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:09.810 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333749+0000", 2024-09-18T04:40:09.810 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:09.810 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.810 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.810 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:09.810 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340899+0000", 2024-09-18T04:40:09.810 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0071500340000000004 2024-09-18T04:40:09.811 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.811 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.811 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:09.811 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340929+0000", 2024-09-18T04:40:09.811 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0093000000000001e-05 2024-09-18T04:40:09.811 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.811 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.811 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:09.811 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340943+0000", 2024-09-18T04:40:09.811 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4019000000000001e-05 2024-09-18T04:40:09.811 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.811 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.811 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:09.811 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343834+0000", 2024-09-18T04:40:09.812 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028918120000000001 2024-09-18T04:40:09.812 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.812 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.812 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:09.812 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344020+0000", 2024-09-18T04:40:09.812 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000185892 2024-09-18T04:40:09.812 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.812 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.812 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:40:09.812 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344190+0000", 2024-09-18T04:40:09.812 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016946800000000001 2024-09-18T04:40:09.812 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.812 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.812 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:09.813 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.351291+0000", 2024-09-18T04:40:09.813 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.007100805 2024-09-18T04:40:09.813 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.813 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.813 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:09.813 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357470+0000", 2024-09-18T04:40:09.813 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0061797570000000001 2024-09-18T04:40:09.813 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.813 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.813 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:09.813 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357533+0000", 2024-09-18T04:40:09.813 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2906999999999999e-05 2024-09-18T04:40:09.813 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.814 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.814 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:09.814 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357575+0000", 2024-09-18T04:40:09.814 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1403999999999999e-05 2024-09-18T04:40:09.814 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.814 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:09.814 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.814 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.814 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.814 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:56 2.4 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:09.814 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:40:09.814 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 87.421573127000002, 2024-09-18T04:40:09.815 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.034295870999999999, 2024-09-18T04:40:09.815 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:09.815 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:09.815 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:09.815 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:40:09.815 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:40:09.815 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 56 2024-09-18T04:40:09.815 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.815 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:09.815 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.815 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:09.815 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:40:09.815 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.815 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.815 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.816 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:09.816 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:40:09.816 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.816 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.816 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.816 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:09.816 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349794+0000", 2024-09-18T04:40:09.816 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:09.816 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.816 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.816 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:09.816 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366632+0000", 2024-09-18T04:40:09.816 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01683755 2024-09-18T04:40:09.816 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.817 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.817 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:09.817 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366669+0000", 2024-09-18T04:40:09.817 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7351000000000002e-05 2024-09-18T04:40:09.817 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.817 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.817 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:09.817 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366686+0000", 2024-09-18T04:40:09.817 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6507999999999998e-05 2024-09-18T04:40:09.817 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.817 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.817 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:09.817 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366962+0000", 2024-09-18T04:40:09.817 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00027655900000000001 2024-09-18T04:40:09.818 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.818 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.818 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:09.818 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.367452+0000", 2024-09-18T04:40:09.818 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00049025299999999996 2024-09-18T04:40:09.818 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.818 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.818 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:40:09.818 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368113+0000", 2024-09-18T04:40:09.818 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00066093399999999996 2024-09-18T04:40:09.818 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.818 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.818 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:09.818 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383719+0000", 2024-09-18T04:40:09.819 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015606106999999999 2024-09-18T04:40:09.819 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.819 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.819 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:09.819 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384035+0000", 2024-09-18T04:40:09.819 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00031590399999999999 2024-09-18T04:40:09.819 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.819 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.819 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:09.819 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384067+0000", 2024-09-18T04:40:09.819 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.129e-05 2024-09-18T04:40:09.819 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.819 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.819 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:09.819 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384091+0000", 2024-09-18T04:40:09.820 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4788000000000001e-05 2024-09-18T04:40:09.820 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.820 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:09.820 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.820 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.820 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.821 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:57 2.0 e51/17 2:0cdd070e:::rbd_data.125fc09df6d1.000000000000000d:head v 51'21, mlcod=51'19)", 2024-09-18T04:40:09.821 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:40:09.821 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 87.413745640000002, 2024-09-18T04:40:09.821 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.025216311000000002, 2024-09-18T04:40:09.821 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:09.821 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:09.821 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:09.821 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.821 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:09.821 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:40:09.821 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.821 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.821 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.821 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:09.822 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:40:09.822 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.822 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.822 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.822 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:09.822 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:40:09.822 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:40:09.822 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.822 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.822 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:09.822 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363844+0000", 2024-09-18T04:40:09.822 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0062218450000000002 2024-09-18T04:40:09.822 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.823 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.823 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:09.823 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363879+0000", 2024-09-18T04:40:09.823 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4382999999999999e-05 2024-09-18T04:40:09.823 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.823 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.823 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:09.823 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363893+0000", 2024-09-18T04:40:09.823 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4348999999999999e-05 2024-09-18T04:40:09.823 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.824 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.824 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:09.824 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363977+0000", 2024-09-18T04:40:09.824 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3499999999999997e-05 2024-09-18T04:40:09.824 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.824 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.824 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:09.825 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364009+0000", 2024-09-18T04:40:09.825 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2292000000000002e-05 2024-09-18T04:40:09.825 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.825 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.825 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:09.825 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.382781+0000", 2024-09-18T04:40:09.826 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018771626 2024-09-18T04:40:09.826 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.826 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.826 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:09.827 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.382839+0000", 2024-09-18T04:40:09.827 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8758000000000002e-05 2024-09-18T04:40:09.827 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.827 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:09.827 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.827 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.827 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.827 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:59 2.0 e51/17 2:19c52cde:::rbd_data.125fc09df6d1.000000000000000f:head v 51'22, mlcod=51'19)", 2024-09-18T04:40:09.827 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:40:09.827 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 87.407085714000004, 2024-09-18T04:40:09.827 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.019497112, 2024-09-18T04:40:09.827 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:09.828 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:09.828 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:09.828 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.828 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:09.828 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:40:09.828 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.828 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.828 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.828 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:09.828 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:40:09.828 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.829 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.829 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.829 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:09.829 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364282+0000", 2024-09-18T04:40:09.829 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:09.829 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.829 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.829 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:09.829 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371823+0000", 2024-09-18T04:40:09.829 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0075411829999999999 2024-09-18T04:40:09.829 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.829 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.830 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:09.830 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371837+0000", 2024-09-18T04:40:09.830 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4336e-05 2024-09-18T04:40:09.830 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.830 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.830 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:09.830 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371846+0000", 2024-09-18T04:40:09.830 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9770000000000006e-06 2024-09-18T04:40:09.830 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.830 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.830 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:09.831 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.373566+0000", 2024-09-18T04:40:09.831 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017192539999999999 2024-09-18T04:40:09.831 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.831 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.831 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:09.831 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.373595+0000", 2024-09-18T04:40:09.831 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8975999999999998e-05 2024-09-18T04:40:09.831 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.831 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.831 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:09.831 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383740+0000", 2024-09-18T04:40:09.832 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01014525 2024-09-18T04:40:09.832 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.832 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.832 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:09.832 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383780+0000", 2024-09-18T04:40:09.832 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0275999999999999e-05 2024-09-18T04:40:09.832 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.832 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:09.832 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.832 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.832 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.833 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:34 2.6 2:66289171:ns::rbd_data.133e813ab5bc.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:09.833 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.524015+0000", 2024-09-18T04:40:09.833 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 87.247354104999999, 2024-09-18T04:40:09.833 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027582111999999999, 2024-09-18T04:40:09.833 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:09.833 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:09.833 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:09.833 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:40:09.833 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:09.833 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 34 2024-09-18T04:40:09.833 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.833 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:09.834 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.834 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:09.834 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524015+0000", 2024-09-18T04:40:09.834 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.834 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.834 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.834 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:09.834 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524015+0000", 2024-09-18T04:40:09.834 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.834 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.834 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.835 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:09.835 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524012+0000", 2024-09-18T04:40:09.835 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:40:09.835 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.835 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.835 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:09.835 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534956+0000", 2024-09-18T04:40:09.835 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010943403000000001 2024-09-18T04:40:09.835 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.835 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.835 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:09.836 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534978+0000", 2024-09-18T04:40:09.836 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1965999999999999e-05 2024-09-18T04:40:09.836 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.836 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.836 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:09.836 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534990+0000", 2024-09-18T04:40:09.836 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2065999999999999e-05 2024-09-18T04:40:09.836 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.836 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.836 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:09.836 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535065+0000", 2024-09-18T04:40:09.837 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5523000000000002e-05 2024-09-18T04:40:09.837 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.837 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.837 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:09.837 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535208+0000", 2024-09-18T04:40:09.837 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014309500000000001 2024-09-18T04:40:09.837 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.837 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.837 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:40:09.837 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535343+0000", 2024-09-18T04:40:09.837 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013437400000000001 2024-09-18T04:40:09.837 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.837 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.838 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:09.838 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.546576+0000", 2024-09-18T04:40:09.838 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011233425999999999 2024-09-18T04:40:09.838 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.838 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.838 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:09.838 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551498+0000", 2024-09-18T04:40:09.838 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0049219349999999997 2024-09-18T04:40:09.838 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.838 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.838 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:09.838 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551554+0000", 2024-09-18T04:40:09.839 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6400999999999998e-05 2024-09-18T04:40:09.839 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.839 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.839 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:09.839 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551597+0000", 2024-09-18T04:40:09.839 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2283000000000002e-05 2024-09-18T04:40:09.839 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.839 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:09.839 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.839 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.839 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.839 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:33 2.1 e51/24 2:8b40f1c5:ns::rbd_data.133e813ab5bc.0000000000000000:head v 51'7, mlcod=51'5)", 2024-09-18T04:40:09.840 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.531767+0000", 2024-09-18T04:40:09.840 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 87.239602188000006, 2024-09-18T04:40:09.840 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015370295000000001, 2024-09-18T04:40:09.840 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:09.840 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:09.840 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:09.840 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.840 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:09.840 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531767+0000", 2024-09-18T04:40:09.840 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.840 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.840 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.840 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:09.840 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531767+0000", 2024-09-18T04:40:09.841 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.841 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.841 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.841 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:09.841 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531765+0000", 2024-09-18T04:40:09.841 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:09.841 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.841 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.841 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:09.841 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534258+0000", 2024-09-18T04:40:09.841 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002493561 2024-09-18T04:40:09.841 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.841 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.841 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:09.842 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534288+0000", 2024-09-18T04:40:09.842 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.976e-05 2024-09-18T04:40:09.842 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.842 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.842 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:09.842 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534300+0000", 2024-09-18T04:40:09.842 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1928e-05 2024-09-18T04:40:09.842 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.842 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.843 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:09.843 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534367+0000", 2024-09-18T04:40:09.843 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7045999999999995e-05 2024-09-18T04:40:09.843 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.843 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.843 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:09.843 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534394+0000", 2024-09-18T04:40:09.843 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6903999999999999e-05 2024-09-18T04:40:09.843 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.843 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.843 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:09.843 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.547073+0000", 2024-09-18T04:40:09.843 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012679365 2024-09-18T04:40:09.844 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.844 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.844 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:09.844 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.547137+0000", 2024-09-18T04:40:09.844 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3691999999999999e-05 2024-09-18T04:40:09.844 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.844 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:09.844 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.844 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.844 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.844 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:39 2.4 2:304364d1:ns::rbd_data.133e813ab5bc.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:09.844 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.535013+0000", 2024-09-18T04:40:09.844 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 87.236355252999999, 2024-09-18T04:40:09.845 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027764487000000001, 2024-09-18T04:40:09.845 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:09.845 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:09.845 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:09.845 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:40:09.845 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:09.845 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 39 2024-09-18T04:40:09.845 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.845 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:09.845 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.845 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:09.845 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535013+0000", 2024-09-18T04:40:09.846 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.846 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.846 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.846 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:09.846 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535013+0000", 2024-09-18T04:40:09.846 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.846 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.846 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.846 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:09.846 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535012+0000", 2024-09-18T04:40:09.847 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:09.847 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.847 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.847 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:09.847 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545016+0000", 2024-09-18T04:40:09.847 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01000322 2024-09-18T04:40:09.847 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.847 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.847 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:09.847 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545035+0000", 2024-09-18T04:40:09.847 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9845999999999999e-05 2024-09-18T04:40:09.847 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.847 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.848 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:09.848 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545045+0000", 2024-09-18T04:40:09.848 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3640000000000005e-06 2024-09-18T04:40:09.848 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.848 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.848 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:09.848 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545152+0000", 2024-09-18T04:40:09.848 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000107025 2024-09-18T04:40:09.848 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.848 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.848 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:09.848 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545338+0000", 2024-09-18T04:40:09.848 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018643500000000001 2024-09-18T04:40:09.849 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.849 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.849 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:40:09.849 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545492+0000", 2024-09-18T04:40:09.849 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015393199999999999 2024-09-18T04:40:09.849 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.849 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.849 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:09.849 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.553549+0000", 2024-09-18T04:40:09.849 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0080569170000000002 2024-09-18T04:40:09.849 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.849 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.850 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:09.850 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562693+0000", 2024-09-18T04:40:09.850 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0091441609999999996 2024-09-18T04:40:09.850 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.850 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.850 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:09.850 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562748+0000", 2024-09-18T04:40:09.850 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5047000000000003e-05 2024-09-18T04:40:09.850 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.850 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.850 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:09.850 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562778+0000", 2024-09-18T04:40:09.850 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9652e-05 2024-09-18T04:40:09.851 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.851 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:09.851 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.851 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.851 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.851 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:40 2.6 2:784d6cd1:ns::rbd_data.133e813ab5bc.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:09.851 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.545062+0000", 2024-09-18T04:40:09.851 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 87.226306202000004, 2024-09-18T04:40:09.851 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.032771623, 2024-09-18T04:40:09.851 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:09.851 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:09.852 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:09.852 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:40:09.852 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:09.852 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 40 2024-09-18T04:40:09.852 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.852 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:09.852 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.852 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:09.852 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545062+0000", 2024-09-18T04:40:09.852 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.852 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.852 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.852 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:09.853 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545062+0000", 2024-09-18T04:40:09.853 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.853 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.853 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.853 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:09.853 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545061+0000", 2024-09-18T04:40:09.853 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:09.853 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.853 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.853 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:09.853 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552477+0000", 2024-09-18T04:40:09.853 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.007415914 2024-09-18T04:40:09.854 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.854 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.854 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:09.854 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552500+0000", 2024-09-18T04:40:09.854 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3546000000000001e-05 2024-09-18T04:40:09.854 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.854 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.854 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:09.854 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552515+0000", 2024-09-18T04:40:09.854 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4606999999999999e-05 2024-09-18T04:40:09.854 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.854 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.855 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:09.855 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552661+0000", 2024-09-18T04:40:09.855 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000146007 2024-09-18T04:40:09.855 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.855 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.855 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:09.855 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552876+0000", 2024-09-18T04:40:09.855 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000214618 2024-09-18T04:40:09.855 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.855 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.855 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:40:09.855 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.553061+0000", 2024-09-18T04:40:09.856 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018564100000000001 2024-09-18T04:40:09.856 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.856 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.856 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:09.856 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.565612+0000", 2024-09-18T04:40:09.856 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012550301999999999 2024-09-18T04:40:09.856 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.856 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.856 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:09.856 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577695+0000", 2024-09-18T04:40:09.856 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012083493000000001 2024-09-18T04:40:09.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.857 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.857 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:09.857 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577772+0000", 2024-09-18T04:40:09.857 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7158999999999998e-05 2024-09-18T04:40:09.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.857 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.857 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:09.857 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577834+0000", 2024-09-18T04:40:09.857 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1851000000000001e-05 2024-09-18T04:40:09.857 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.858 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:09.858 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.858 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.858 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.858 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:43 2.6 2:759f27b6:ns::rbd_data.133e813ab5bc.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:09.858 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.555868+0000", 2024-09-18T04:40:09.858 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 87.215500707999993, 2024-09-18T04:40:09.858 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.022332497999999999, 2024-09-18T04:40:09.858 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:09.858 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:09.858 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:09.859 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:40:09.859 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:09.859 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 43 2024-09-18T04:40:09.859 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.859 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:09.859 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.859 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:09.859 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555868+0000", 2024-09-18T04:40:09.859 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.859 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.859 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.860 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:09.860 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555868+0000", 2024-09-18T04:40:09.860 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.860 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.860 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.860 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:09.860 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555865+0000", 2024-09-18T04:40:09.860 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999971 2024-09-18T04:40:09.860 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.860 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.860 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:09.860 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561031+0000", 2024-09-18T04:40:09.861 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0051658770000000001 2024-09-18T04:40:09.861 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.861 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.861 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:09.861 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561048+0000", 2024-09-18T04:40:09.861 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6569e-05 2024-09-18T04:40:09.861 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.861 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.861 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:09.861 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561055+0000", 2024-09-18T04:40:09.862 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.328e-06 2024-09-18T04:40:09.862 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.862 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.862 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:09.862 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561114+0000", 2024-09-18T04:40:09.862 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9540000000000003e-05 2024-09-18T04:40:09.862 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.862 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.862 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:09.862 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561314+0000", 2024-09-18T04:40:09.862 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019965799999999999 2024-09-18T04:40:09.862 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.863 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.863 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:40:09.863 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561473+0000", 2024-09-18T04:40:09.863 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000158904 2024-09-18T04:40:09.863 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.863 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.863 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:09.863 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.567015+0000", 2024-09-18T04:40:09.863 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055420859999999999 2024-09-18T04:40:09.863 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.863 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.863 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:09.863 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577982+0000", 2024-09-18T04:40:09.863 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010966872000000001 2024-09-18T04:40:09.864 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.864 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.864 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:09.864 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578162+0000", 2024-09-18T04:40:09.864 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018050999999999999 2024-09-18T04:40:09.864 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.864 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.864 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:09.864 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578200+0000", 2024-09-18T04:40:09.864 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8000999999999998e-05 2024-09-18T04:40:09.864 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.864 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:09.864 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.865 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.865 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.865 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:48 2.6 2:7c225365:ns::rbd_data.133e813ab5bc.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:09.865 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:40:09.865 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 87.207701485000001, 2024-09-18T04:40:09.865 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.024576377, 2024-09-18T04:40:09.865 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:09.865 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:09.865 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:09.865 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:40:09.866 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:09.866 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 48 2024-09-18T04:40:09.866 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.866 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:09.866 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.866 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:09.866 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:40:09.866 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.866 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.866 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.867 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:09.867 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:40:09.867 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.867 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.867 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.867 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:09.867 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.563666+0000", 2024-09-18T04:40:09.867 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:09.867 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.867 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.867 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:09.867 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570331+0000", 2024-09-18T04:40:09.868 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0066653329999999998 2024-09-18T04:40:09.868 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.868 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.868 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:09.868 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570348+0000", 2024-09-18T04:40:09.868 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6507e-05 2024-09-18T04:40:09.868 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.868 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.868 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:09.868 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570355+0000", 2024-09-18T04:40:09.869 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2749999999999998e-06 2024-09-18T04:40:09.869 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.869 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.869 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:09.869 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570396+0000", 2024-09-18T04:40:09.869 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0809000000000002e-05 2024-09-18T04:40:09.869 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.869 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.869 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:09.869 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570571+0000", 2024-09-18T04:40:09.870 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000174823 2024-09-18T04:40:09.870 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.870 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.870 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:40:09.870 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570722+0000", 2024-09-18T04:40:09.870 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015154599999999999 2024-09-18T04:40:09.870 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.870 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.870 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:09.870 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579756+0000", 2024-09-18T04:40:09.870 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0090337000000000004 2024-09-18T04:40:09.870 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.870 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.870 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:09.871 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588093+0000", 2024-09-18T04:40:09.871 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.008336553 2024-09-18T04:40:09.871 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.871 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.871 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:09.871 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588173+0000", 2024-09-18T04:40:09.871 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0333999999999994e-05 2024-09-18T04:40:09.871 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.871 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.871 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:09.871 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588244+0000", 2024-09-18T04:40:09.871 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0573000000000003e-05 2024-09-18T04:40:09.871 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.872 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:09.872 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.872 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.872 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.872 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:49 2.4 2:20f0b0c9:ns::rbd_data.133e813ab5bc.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:09.872 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.570370+0000", 2024-09-18T04:40:09.872 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 87.200999158000002, 2024-09-18T04:40:09.872 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.029633981, 2024-09-18T04:40:09.872 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:09.872 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:09.872 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:09.872 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:40:09.872 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:09.873 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 49 2024-09-18T04:40:09.873 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.873 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:09.873 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.873 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:09.873 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570370+0000", 2024-09-18T04:40:09.873 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.873 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.873 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.873 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:09.873 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570370+0000", 2024-09-18T04:40:09.873 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.873 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.874 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.874 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:09.874 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570369+0000", 2024-09-18T04:40:09.874 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:09.874 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.874 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.874 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:09.874 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579090+0000", 2024-09-18T04:40:09.874 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0087212339999999996 2024-09-18T04:40:09.874 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.874 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.874 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:09.874 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579109+0000", 2024-09-18T04:40:09.874 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9120999999999999e-05 2024-09-18T04:40:09.874 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.875 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.875 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:09.875 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579123+0000", 2024-09-18T04:40:09.875 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4382000000000001e-05 2024-09-18T04:40:09.875 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.875 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.875 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:09.875 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579221+0000", 2024-09-18T04:40:09.875 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.7503999999999993e-05 2024-09-18T04:40:09.875 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.875 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.875 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:09.875 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579553+0000", 2024-09-18T04:40:09.875 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00033254200000000002 2024-09-18T04:40:09.875 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.876 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.876 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:40:09.876 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579817+0000", 2024-09-18T04:40:09.876 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00026317099999999998 2024-09-18T04:40:09.876 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.876 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.876 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:09.876 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.599905+0000", 2024-09-18T04:40:09.876 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020088575000000001 2024-09-18T04:40:09.876 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.876 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.876 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:09.876 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.599923+0000", 2024-09-18T04:40:09.876 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7425999999999999e-05 2024-09-18T04:40:09.877 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.877 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.877 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:09.877 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.599973+0000", 2024-09-18T04:40:09.877 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0649999999999998e-05 2024-09-18T04:40:09.877 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.877 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.877 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:09.877 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.600004+0000", 2024-09-18T04:40:09.877 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0361999999999998e-05 2024-09-18T04:40:09.877 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.877 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:09.877 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.877 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.877 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.878 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:50 2.1 e51/24 2:8c706374:ns::rbd_data.133e813ab5bc.000000000000000a:head v 51'8, mlcod=51'6)", 2024-09-18T04:40:09.878 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.577029+0000", 2024-09-18T04:40:09.878 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 87.194339787000004, 2024-09-18T04:40:09.878 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.017362855, 2024-09-18T04:40:09.878 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:09.878 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:09.878 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:09.878 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.878 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:09.878 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577029+0000", 2024-09-18T04:40:09.878 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.878 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.878 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.878 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:09.879 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577029+0000", 2024-09-18T04:40:09.879 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.879 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.879 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.879 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:09.879 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577027+0000", 2024-09-18T04:40:09.879 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:09.879 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.879 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.879 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:09.879 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578880+0000", 2024-09-18T04:40:09.879 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0018532080000000001 2024-09-18T04:40:09.879 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.879 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.879 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:09.880 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578906+0000", 2024-09-18T04:40:09.880 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5573999999999999e-05 2024-09-18T04:40:09.880 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.880 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.880 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:09.880 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578920+0000", 2024-09-18T04:40:09.880 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4483e-05 2024-09-18T04:40:09.880 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.880 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.880 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:09.880 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578983+0000", 2024-09-18T04:40:09.880 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2248000000000004e-05 2024-09-18T04:40:09.880 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.880 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.881 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:09.881 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579016+0000", 2024-09-18T04:40:09.881 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3364000000000001e-05 2024-09-18T04:40:09.881 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.881 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.881 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:09.881 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594342+0000", 2024-09-18T04:40:09.881 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01532564 2024-09-18T04:40:09.881 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.881 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.881 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:09.881 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594392+0000", 2024-09-18T04:40:09.881 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0068999999999998e-05 2024-09-18T04:40:09.881 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.881 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:09.882 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.882 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.882 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.882 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:45 2.0 e51/17 2:0fbfa3dc:ns::rbd_data.133e813ab5bc.0000000000000007:head v 51'24, mlcod=51'22)", 2024-09-18T04:40:09.882 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.581892+0000", 2024-09-18T04:40:09.882 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 87.189477115000003, 2024-09-18T04:40:09.882 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015646006, 2024-09-18T04:40:09.882 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:09.882 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:09.882 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:09.882 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.882 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:09.882 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581892+0000", 2024-09-18T04:40:09.883 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.883 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.883 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.883 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:09.883 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581892+0000", 2024-09-18T04:40:09.883 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.883 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.883 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.883 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:09.883 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581890+0000", 2024-09-18T04:40:09.883 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:09.883 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.884 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.884 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:09.884 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586188+0000", 2024-09-18T04:40:09.884 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0042985769999999996 2024-09-18T04:40:09.884 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.884 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.884 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:09.884 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586210+0000", 2024-09-18T04:40:09.884 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1069000000000001e-05 2024-09-18T04:40:09.884 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.884 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.884 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:09.885 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586221+0000", 2024-09-18T04:40:09.885 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1287000000000001e-05 2024-09-18T04:40:09.885 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.885 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.885 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:09.885 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586313+0000", 2024-09-18T04:40:09.885 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1935999999999995e-05 2024-09-18T04:40:09.885 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.885 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.885 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:09.885 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586368+0000", 2024-09-18T04:40:09.885 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5578000000000002e-05 2024-09-18T04:40:09.886 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.886 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.886 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:09.886 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.597473+0000", 2024-09-18T04:40:09.886 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011105139999999999 2024-09-18T04:40:09.886 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.886 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.886 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:09.886 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.597538+0000", 2024-09-18T04:40:09.886 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4133000000000004e-05 2024-09-18T04:40:09.886 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.886 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:09.886 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.886 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.887 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.887 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:53 2.4 2:39f7f174:ns::rbd_data.133e813ab5bc.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:09.887 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.584263+0000", 2024-09-18T04:40:09.887 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 87.187105240999998, 2024-09-18T04:40:09.887 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018701180000000001, 2024-09-18T04:40:09.887 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:09.887 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:09.887 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:09.887 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:40:09.887 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:09.888 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 53 2024-09-18T04:40:09.888 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.888 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:09.888 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.888 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:09.888 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584263+0000", 2024-09-18T04:40:09.888 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.888 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.888 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.888 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:09.888 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584263+0000", 2024-09-18T04:40:09.888 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.889 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.889 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.889 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:09.889 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584262+0000", 2024-09-18T04:40:09.889 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:09.889 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.889 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.889 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:09.889 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.591617+0000", 2024-09-18T04:40:09.889 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0073550600000000001 2024-09-18T04:40:09.889 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.889 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.890 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:09.890 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.591637+0000", 2024-09-18T04:40:09.890 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0432999999999999e-05 2024-09-18T04:40:09.890 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.890 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.890 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:09.890 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.591646+0000", 2024-09-18T04:40:09.890 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7849999999999997e-06 2024-09-18T04:40:09.890 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.890 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.890 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:09.890 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593478+0000", 2024-09-18T04:40:09.891 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001831495 2024-09-18T04:40:09.891 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.891 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.891 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:09.891 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593661+0000", 2024-09-18T04:40:09.891 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018364999999999999 2024-09-18T04:40:09.891 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.891 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.891 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:40:09.891 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593834+0000", 2024-09-18T04:40:09.891 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000173065 2024-09-18T04:40:09.891 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.891 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.892 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:09.892 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.600670+0000", 2024-09-18T04:40:09.892 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0068354970000000003 2024-09-18T04:40:09.892 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.892 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.892 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:09.892 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602851+0000", 2024-09-18T04:40:09.892 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002181653 2024-09-18T04:40:09.892 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.892 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.893 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:09.893 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602930+0000", 2024-09-18T04:40:09.893 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8045000000000002e-05 2024-09-18T04:40:09.893 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.893 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.893 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:09.893 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602965+0000", 2024-09-18T04:40:09.893 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5102999999999999e-05 2024-09-18T04:40:09.893 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.893 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:09.893 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.894 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.894 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.894 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:56 2.6 2:72453c3e:ns::rbd_data.133e813ab5bc.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:09.894 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.595988+0000", 2024-09-18T04:40:09.894 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 87.175380838999999, 2024-09-18T04:40:09.894 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.030296172, 2024-09-18T04:40:09.894 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:09.894 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:09.894 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:09.894 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:40:09.895 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:09.895 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 56 2024-09-18T04:40:09.895 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.895 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:09.895 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.895 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:09.895 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.595988+0000", 2024-09-18T04:40:09.895 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.895 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.895 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.896 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:09.896 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.595988+0000", 2024-09-18T04:40:09.896 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:09.896 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.896 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.896 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:09.896 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.595986+0000", 2024-09-18T04:40:09.896 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:40:09.896 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.897 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.897 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:09.897 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605262+0000", 2024-09-18T04:40:09.897 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0092767240000000001 2024-09-18T04:40:09.897 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.897 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.897 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:09.897 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605282+0000", 2024-09-18T04:40:09.897 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9307e-05 2024-09-18T04:40:09.897 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.897 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.897 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:09.898 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605292+0000", 2024-09-18T04:40:09.898 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9510000000000001e-06 2024-09-18T04:40:09.898 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.898 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.898 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:09.898 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605369+0000", 2024-09-18T04:40:09.898 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7846e-05 2024-09-18T04:40:09.898 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.898 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.899 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:09.899 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605668+0000", 2024-09-18T04:40:09.899 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00029896099999999999 2024-09-18T04:40:09.899 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.899 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.899 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:40:09.899 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605906+0000", 2024-09-18T04:40:09.899 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000237758 2024-09-18T04:40:09.899 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.899 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.899 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:09.900 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.613532+0000", 2024-09-18T04:40:09.900 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0076260590000000001 2024-09-18T04:40:09.900 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.900 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.900 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:09.900 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.626116+0000", 2024-09-18T04:40:09.900 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012584092999999999 2024-09-18T04:40:09.900 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.900 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.900 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:09.900 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.626216+0000", 2024-09-18T04:40:09.901 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9630999999999995e-05 2024-09-18T04:40:09.901 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:09.901 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:09.901 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:09.901 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.626284+0000", 2024-09-18T04:40:09.901 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8034999999999995e-05 2024-09-18T04:40:09.901 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.901 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:09.901 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.901 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:09.901 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:09.901 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:40:09.902 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 2, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 2] 2024-09-18T04:40:09.902 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-18T04:40:09.903 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:40:09.900+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:40:09.904 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:40:09.905 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-09-18T04:40:10.004 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:40:09.999+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:40:10.020 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:40:10.021 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:40:10.021 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:40:10.021 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:40:10.031 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:40:10.104 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:40:10.099+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:40:10.136 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:40:10.136 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:40:10.136 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:40:10.137 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:40:10.137 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:40:10.146 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:40:10.205 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:40:10.200+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:40:10.257 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:40:10.258 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:40:10.258 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:40:10.258 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:40:10.258 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.258 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:40:10.258 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:40:10.258 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 162.464798479, 2024-09-18T04:40:10.258 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017399218000000001, 2024-09-18T04:40:10.258 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:10.258 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:10.258 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:10.259 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:40:10.259 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:40:10.259 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 114 2024-09-18T04:40:10.259 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.259 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:10.259 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.259 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:10.259 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:40:10.259 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:10.259 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.259 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.259 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:10.259 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:40:10.260 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:10.260 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.260 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.260 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:10.260 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786900+0000", 2024-09-18T04:40:10.260 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:40:10.260 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.260 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.260 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:10.260 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786908+0000", 2024-09-18T04:40:10.260 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8199999999999997e-06 2024-09-18T04:40:10.260 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.260 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.260 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:10.260 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786925+0000", 2024-09-18T04:40:10.261 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6849000000000001e-05 2024-09-18T04:40:10.261 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.261 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.261 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:10.261 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786934+0000", 2024-09-18T04:40:10.261 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6299999999999993e-06 2024-09-18T04:40:10.261 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.261 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.261 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:10.261 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800926+0000", 2024-09-18T04:40:10.261 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013991632 2024-09-18T04:40:10.261 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.261 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.261 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:10.262 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801060+0000", 2024-09-18T04:40:10.262 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013378199999999999 2024-09-18T04:40:10.262 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.262 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.262 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:40:10.262 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801166+0000", 2024-09-18T04:40:10.262 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010607099999999999 2024-09-18T04:40:10.262 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.262 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.262 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:10.262 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803026+0000", 2024-09-18T04:40:10.262 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001860469 2024-09-18T04:40:10.262 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.262 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.262 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:10.263 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804245+0000", 2024-09-18T04:40:10.263 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0012185900000000001 2024-09-18T04:40:10.263 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.263 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.263 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:10.263 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804278+0000", 2024-09-18T04:40:10.263 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3272000000000003e-05 2024-09-18T04:40:10.263 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.263 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.263 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:10.263 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804300+0000", 2024-09-18T04:40:10.263 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2266e-05 2024-09-18T04:40:10.263 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:10.264 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:10.264 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:10.264 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.264 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.264 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:40:10.264 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:40:10.264 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 162.463885581, 2024-09-18T04:40:10.264 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017521662, 2024-09-18T04:40:10.264 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:10.264 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:10.264 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:10.264 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:40:10.264 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:40:10.264 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 116 2024-09-18T04:40:10.265 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.265 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:10.265 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.265 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:10.265 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:40:10.265 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:10.265 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.265 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.265 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:10.265 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:40:10.265 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:10.265 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.265 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.266 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:10.266 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787813+0000", 2024-09-18T04:40:10.266 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:10.266 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.266 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.266 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:10.266 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787824+0000", 2024-09-18T04:40:10.266 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1157e-05 2024-09-18T04:40:10.266 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.266 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.266 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:10.266 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787840+0000", 2024-09-18T04:40:10.266 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6269e-05 2024-09-18T04:40:10.266 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.266 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.266 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:10.267 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787853+0000", 2024-09-18T04:40:10.267 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3156000000000001e-05 2024-09-18T04:40:10.267 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.267 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.267 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:10.267 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803110+0000", 2024-09-18T04:40:10.267 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015256736 2024-09-18T04:40:10.267 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.267 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.267 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:10.267 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803197+0000", 2024-09-18T04:40:10.268 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7195999999999997e-05 2024-09-18T04:40:10.268 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.268 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.268 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:40:10.268 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803308+0000", 2024-09-18T04:40:10.268 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111005 2024-09-18T04:40:10.268 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.268 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.268 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:10.268 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804316+0000", 2024-09-18T04:40:10.268 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010073949999999999 2024-09-18T04:40:10.268 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.268 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.268 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:10.269 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805232+0000", 2024-09-18T04:40:10.269 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091636300000000001 2024-09-18T04:40:10.269 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.269 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.269 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:10.269 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805277+0000", 2024-09-18T04:40:10.269 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4864000000000003e-05 2024-09-18T04:40:10.269 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.269 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.269 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:10.269 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805336+0000", 2024-09-18T04:40:10.270 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8968000000000002e-05 2024-09-18T04:40:10.270 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:10.270 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:10.270 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:10.270 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.270 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.270 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:40:10.270 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:40:10.270 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 162.463833526, 2024-09-18T04:40:10.270 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017983453, 2024-09-18T04:40:10.270 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:10.270 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:10.270 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:10.270 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:40:10.270 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:40:10.271 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 117 2024-09-18T04:40:10.271 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.271 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:10.271 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.271 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:10.271 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:40:10.271 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:10.271 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.271 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.271 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:10.271 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:40:10.272 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:10.272 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.272 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.272 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:10.272 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787865+0000", 2024-09-18T04:40:10.272 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:40:10.272 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.272 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.272 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:10.272 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787889+0000", 2024-09-18T04:40:10.272 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4244999999999999e-05 2024-09-18T04:40:10.272 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.272 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.272 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:10.273 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787905+0000", 2024-09-18T04:40:10.273 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5739999999999998e-05 2024-09-18T04:40:10.273 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.273 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.273 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:10.273 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787918+0000", 2024-09-18T04:40:10.273 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.252e-05 2024-09-18T04:40:10.273 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.273 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.273 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:10.273 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803693+0000", 2024-09-18T04:40:10.273 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015774872999999998 2024-09-18T04:40:10.273 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.273 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.274 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:10.274 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803775+0000", 2024-09-18T04:40:10.274 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2761000000000003e-05 2024-09-18T04:40:10.274 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.274 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.274 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:40:10.274 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803887+0000", 2024-09-18T04:40:10.274 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111708 2024-09-18T04:40:10.274 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.274 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.274 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:10.274 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804823+0000", 2024-09-18T04:40:10.274 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00093625100000000003 2024-09-18T04:40:10.275 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.275 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.275 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:10.275 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805804+0000", 2024-09-18T04:40:10.276 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00098114699999999992 2024-09-18T04:40:10.276 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.277 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.277 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:10.277 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805831+0000", 2024-09-18T04:40:10.278 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6401999999999999e-05 2024-09-18T04:40:10.278 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.278 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.278 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:10.278 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805850+0000", 2024-09-18T04:40:10.278 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8862e-05 2024-09-18T04:40:10.278 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:10.278 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:10.278 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:10.278 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.279 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.279 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:36 2.7 2:fe1b02ad:::rbd_data.125fc09df6d1.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:10.279 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:40:10.279 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 87.985977415999997, 2024-09-18T04:40:10.279 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.040387804999999999, 2024-09-18T04:40:10.279 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:10.279 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:10.279 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:10.279 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:40:10.279 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:40:10.279 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 36 2024-09-18T04:40:10.279 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.280 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:10.280 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.280 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:10.280 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:40:10.280 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:10.280 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.280 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.280 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:10.280 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:40:10.280 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:10.280 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.280 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.280 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:10.280 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265720+0000", 2024-09-18T04:40:10.281 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:40:10.281 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.281 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.281 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:10.281 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272590+0000", 2024-09-18T04:40:10.281 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.006870508 2024-09-18T04:40:10.281 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.281 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.281 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:10.281 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272623+0000", 2024-09-18T04:40:10.281 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2716000000000003e-05 2024-09-18T04:40:10.281 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.281 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.281 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:10.282 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272637+0000", 2024-09-18T04:40:10.282 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4175999999999999e-05 2024-09-18T04:40:10.282 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.282 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.282 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:10.282 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272779+0000", 2024-09-18T04:40:10.282 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000141392 2024-09-18T04:40:10.282 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.282 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.282 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:10.282 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273031+0000", 2024-09-18T04:40:10.282 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00025249099999999998 2024-09-18T04:40:10.282 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.282 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.282 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:40:10.283 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273219+0000", 2024-09-18T04:40:10.283 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000187531 2024-09-18T04:40:10.283 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.283 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.283 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:10.283 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.302806+0000", 2024-09-18T04:40:10.283 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.029587553999999999 2024-09-18T04:40:10.283 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.283 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.283 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:10.283 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306004+0000", 2024-09-18T04:40:10.284 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0031982629999999998 2024-09-18T04:40:10.284 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.284 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.284 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:10.284 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306071+0000", 2024-09-18T04:40:10.284 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6829999999999995e-05 2024-09-18T04:40:10.284 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.284 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.284 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:10.284 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306110+0000", 2024-09-18T04:40:10.284 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8874000000000001e-05 2024-09-18T04:40:10.284 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:10.284 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:10.284 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:10.285 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.285 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.285 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:38 2.0 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:10.285 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:40:10.285 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 87.979023878999996, 2024-09-18T04:40:10.285 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.033754348000000003, 2024-09-18T04:40:10.285 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:10.285 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:10.285 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:10.285 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:40:10.285 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:40:10.285 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 38 2024-09-18T04:40:10.285 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.285 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:10.286 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.286 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:10.286 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:40:10.286 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:10.286 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.286 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.286 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:10.286 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:40:10.286 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:10.286 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.286 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.286 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:10.286 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272673+0000", 2024-09-18T04:40:10.286 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:40:10.286 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.287 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.287 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:10.287 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277166+0000", 2024-09-18T04:40:10.287 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044922249999999999 2024-09-18T04:40:10.287 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.287 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.287 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:10.287 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277189+0000", 2024-09-18T04:40:10.287 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3283000000000001e-05 2024-09-18T04:40:10.287 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.287 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.287 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:10.287 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277205+0000", 2024-09-18T04:40:10.287 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6305000000000001e-05 2024-09-18T04:40:10.287 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.288 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.288 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:10.288 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277290+0000", 2024-09-18T04:40:10.288 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5135000000000004e-05 2024-09-18T04:40:10.288 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.288 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.288 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:10.289 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277531+0000", 2024-09-18T04:40:10.289 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00024014699999999999 2024-09-18T04:40:10.289 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.289 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.289 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:10.289 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277793+0000", 2024-09-18T04:40:10.289 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00026224499999999998 2024-09-18T04:40:10.289 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.289 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.289 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:10.289 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303990+0000", 2024-09-18T04:40:10.289 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.026197295999999998 2024-09-18T04:40:10.289 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.290 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.290 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:10.290 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306340+0000", 2024-09-18T04:40:10.290 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.002349524 2024-09-18T04:40:10.290 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.290 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.290 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:10.290 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306394+0000", 2024-09-18T04:40:10.290 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3934999999999998e-05 2024-09-18T04:40:10.290 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.291 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.291 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:10.291 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306430+0000", 2024-09-18T04:40:10.291 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6690000000000003e-05 2024-09-18T04:40:10.291 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:10.291 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:10.291 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:10.291 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.291 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.291 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:39 2.7 2:e9c909c5:::rbd_data.125fc09df6d1.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:10.291 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:40:10.291 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 87.974464273999999, 2024-09-18T04:40:10.291 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.029110673, 2024-09-18T04:40:10.291 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:10.291 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:10.292 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:10.292 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:40:10.292 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:40:10.292 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 39 2024-09-18T04:40:10.292 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.292 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:10.292 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.292 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:10.292 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:40:10.292 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:10.292 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.292 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.292 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:10.292 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:40:10.293 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:10.293 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.293 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.293 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:10.293 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277234+0000", 2024-09-18T04:40:10.293 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:10.293 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.293 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.293 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:10.293 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281351+0000", 2024-09-18T04:40:10.293 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0041170920000000001 2024-09-18T04:40:10.293 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.293 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.294 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:10.294 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281375+0000", 2024-09-18T04:40:10.294 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.446e-05 2024-09-18T04:40:10.294 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.294 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.294 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:10.294 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281390+0000", 2024-09-18T04:40:10.294 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4309e-05 2024-09-18T04:40:10.294 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.294 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.294 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:10.294 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.282306+0000", 2024-09-18T04:40:10.294 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091662000000000004 2024-09-18T04:40:10.294 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.294 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.295 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:10.295 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.282613+0000", 2024-09-18T04:40:10.295 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00030663799999999997 2024-09-18T04:40:10.295 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.295 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.295 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:40:10.295 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283001+0000", 2024-09-18T04:40:10.295 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00038801599999999998 2024-09-18T04:40:10.295 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.295 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.295 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:10.295 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306239+0000", 2024-09-18T04:40:10.295 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023238466999999999 2024-09-18T04:40:10.295 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.295 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.296 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:10.296 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306259+0000", 2024-09-18T04:40:10.296 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9837999999999999e-05 2024-09-18T04:40:10.296 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.296 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.296 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:10.296 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306299+0000", 2024-09-18T04:40:10.296 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0194000000000002e-05 2024-09-18T04:40:10.296 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.296 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.296 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:10.296 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306346+0000", 2024-09-18T04:40:10.297 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6845000000000003e-05 2024-09-18T04:40:10.297 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:10.297 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:10.297 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:10.297 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.297 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.297 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:46 2.6 e51/17 2:7ba25245:::rbd_data.125fc09df6d1.0000000000000006:head v 51'2, mlcod=0'0)", 2024-09-18T04:40:10.297 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:40:10.297 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 87.938543142, 2024-09-18T04:40:10.297 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019843098, 2024-09-18T04:40:10.297 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:10.297 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:10.297 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:10.297 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.298 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:10.298 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:40:10.298 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:10.298 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.298 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.298 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:10.298 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:40:10.298 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:10.298 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.298 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.298 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:10.298 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313156+0000", 2024-09-18T04:40:10.298 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:40:10.298 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.298 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.299 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:10.299 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320144+0000", 2024-09-18T04:40:10.299 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.006988345 2024-09-18T04:40:10.299 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.299 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.299 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:10.299 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320175+0000", 2024-09-18T04:40:10.299 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.0861999999999997e-05 2024-09-18T04:40:10.299 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.299 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.299 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:10.299 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320188+0000", 2024-09-18T04:40:10.299 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2714999999999999e-05 2024-09-18T04:40:10.300 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.300 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.300 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:10.300 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320267+0000", 2024-09-18T04:40:10.300 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9586999999999995e-05 2024-09-18T04:40:10.300 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.300 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.300 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:10.300 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320311+0000", 2024-09-18T04:40:10.300 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3529000000000003e-05 2024-09-18T04:40:10.300 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.300 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.300 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:10.300 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.332949+0000", 2024-09-18T04:40:10.301 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012638267999999999 2024-09-18T04:40:10.301 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.301 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.301 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:10.301 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.333000+0000", 2024-09-18T04:40:10.301 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0736999999999999e-05 2024-09-18T04:40:10.301 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:10.301 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:10.301 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:10.301 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.301 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.301 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:57 2.0 2:0cdd070e:::rbd_data.125fc09df6d1.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:10.301 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:40:10.302 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 87.902685567000006, 2024-09-18T04:40:10.302 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.033329974999999998, 2024-09-18T04:40:10.302 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:10.302 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:10.302 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:10.302 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:40:10.302 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:40:10.302 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 57 2024-09-18T04:40:10.302 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.302 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:10.302 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.302 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:10.302 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:40:10.302 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:10.302 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.303 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.303 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:10.303 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:40:10.303 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:10.303 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.303 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.303 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:10.303 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349013+0000", 2024-09-18T04:40:10.303 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:10.303 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.303 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.303 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:10.303 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356138+0000", 2024-09-18T04:40:10.303 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071258010000000002 2024-09-18T04:40:10.303 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.304 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.304 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:10.304 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356168+0000", 2024-09-18T04:40:10.304 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9679000000000002e-05 2024-09-18T04:40:10.304 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.304 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.304 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:10.304 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356181+0000", 2024-09-18T04:40:10.304 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3164e-05 2024-09-18T04:40:10.304 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.304 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.304 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:10.304 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356254+0000", 2024-09-18T04:40:10.305 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3182999999999994e-05 2024-09-18T04:40:10.305 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.305 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.305 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:10.305 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356453+0000", 2024-09-18T04:40:10.305 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019876599999999999 2024-09-18T04:40:10.305 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.305 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.305 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:10.305 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356627+0000", 2024-09-18T04:40:10.305 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017340199999999999 2024-09-18T04:40:10.305 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.305 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.305 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:10.306 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.369200+0000", 2024-09-18T04:40:10.306 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012573302 2024-09-18T04:40:10.306 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.306 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.306 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:10.306 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382262+0000", 2024-09-18T04:40:10.306 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013062549 2024-09-18T04:40:10.306 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.306 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.306 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:10.306 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382310+0000", 2024-09-18T04:40:10.306 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7366e-05 2024-09-18T04:40:10.306 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.306 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.307 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:10.307 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382344+0000", 2024-09-18T04:40:10.307 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4319999999999997e-05 2024-09-18T04:40:10.307 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:10.307 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:10.307 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:10.307 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.307 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.307 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:59 2.0 2:19c52cde:::rbd_data.125fc09df6d1.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:10.307 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:40:10.307 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 87.895494401999997, 2024-09-18T04:40:10.307 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.027137247999999999, 2024-09-18T04:40:10.307 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:10.307 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:10.307 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:10.307 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:40:10.308 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:40:10.308 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 59 2024-09-18T04:40:10.308 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.308 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:10.308 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.308 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:10.308 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:40:10.308 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:10.308 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.308 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.308 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:10.308 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:40:10.308 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:10.308 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.308 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.308 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:10.309 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356204+0000", 2024-09-18T04:40:10.309 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:10.309 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.309 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.309 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:10.309 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362661+0000", 2024-09-18T04:40:10.309 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0064565129999999997 2024-09-18T04:40:10.309 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.309 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.309 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:10.309 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362684+0000", 2024-09-18T04:40:10.309 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3688000000000001e-05 2024-09-18T04:40:10.309 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.309 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.309 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:10.310 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362699+0000", 2024-09-18T04:40:10.310 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4798e-05 2024-09-18T04:40:10.310 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.310 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.310 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:10.310 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363016+0000", 2024-09-18T04:40:10.310 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031737300000000003 2024-09-18T04:40:10.310 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.310 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.310 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:10.310 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363201+0000", 2024-09-18T04:40:10.310 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000184131 2024-09-18T04:40:10.311 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.311 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.311 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:10.311 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363343+0000", 2024-09-18T04:40:10.311 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014236699999999999 2024-09-18T04:40:10.311 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.311 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.311 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:10.311 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.372189+0000", 2024-09-18T04:40:10.311 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0088464819999999993 2024-09-18T04:40:10.311 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.311 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.311 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:10.312 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383269+0000", 2024-09-18T04:40:10.312 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011079594 2024-09-18T04:40:10.312 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.312 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.312 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:10.312 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383314+0000", 2024-09-18T04:40:10.312 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4932000000000003e-05 2024-09-18T04:40:10.312 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.312 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.312 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:10.312 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383343+0000", 2024-09-18T04:40:10.312 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8600999999999999e-05 2024-09-18T04:40:10.312 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:10.312 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:10.313 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:10.313 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.313 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.313 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:40 2.6 e51/17 2:784d6cd1:ns::rbd_data.133e813ab5bc.0000000000000005:head v 51'7, mlcod=51'5)", 2024-09-18T04:40:10.313 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:40:10.313 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 87.699244668000006, 2024-09-18T04:40:10.313 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020497496, 2024-09-18T04:40:10.313 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:10.313 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:10.313 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:10.313 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.313 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:10.313 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:40:10.314 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:10.314 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.314 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.314 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:10.314 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:40:10.314 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:10.314 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.314 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.314 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:10.314 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552454+0000", 2024-09-18T04:40:10.314 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:10.314 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.314 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.314 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:10.314 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556448+0000", 2024-09-18T04:40:10.314 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0039940990000000001 2024-09-18T04:40:10.315 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.315 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.315 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:10.315 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556481+0000", 2024-09-18T04:40:10.315 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3778000000000001e-05 2024-09-18T04:40:10.315 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.315 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.315 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:10.315 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556493+0000", 2024-09-18T04:40:10.315 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1622e-05 2024-09-18T04:40:10.315 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.315 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.315 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:10.316 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556556+0000", 2024-09-18T04:40:10.316 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2582999999999994e-05 2024-09-18T04:40:10.316 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.316 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.316 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:10.316 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556592+0000", 2024-09-18T04:40:10.317 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5880999999999997e-05 2024-09-18T04:40:10.317 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.317 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.317 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:10.317 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.572786+0000", 2024-09-18T04:40:10.318 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016194571000000001 2024-09-18T04:40:10.318 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.318 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.318 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:10.318 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.572953+0000", 2024-09-18T04:40:10.318 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000166436 2024-09-18T04:40:10.318 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:10.318 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:10.318 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:10.318 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.318 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.318 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:45 2.0 2:0fbfa3dc:ns::rbd_data.133e813ab5bc.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:10.318 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:40:10.318 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 87.688030369000003, 2024-09-18T04:40:10.318 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.036558168000000002, 2024-09-18T04:40:10.319 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:10.319 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:10.319 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:10.319 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:40:10.319 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:10.319 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 45 2024-09-18T04:40:10.319 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.319 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:10.319 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.319 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:10.319 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:40:10.319 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:10.319 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.319 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.320 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:10.320 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:40:10.320 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:10.320 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.320 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.320 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:10.320 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:40:10.320 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:40:10.320 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.320 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.321 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:10.321 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578594+0000", 2024-09-18T04:40:10.321 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014927147999999999 2024-09-18T04:40:10.321 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.321 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.321 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:10.322 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578612+0000", 2024-09-18T04:40:10.322 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7623999999999999e-05 2024-09-18T04:40:10.322 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.322 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.322 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:10.322 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578620+0000", 2024-09-18T04:40:10.322 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9070000000000007e-06 2024-09-18T04:40:10.322 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.322 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.322 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:10.322 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578702+0000", 2024-09-18T04:40:10.323 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2349999999999996e-05 2024-09-18T04:40:10.323 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.323 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.323 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:10.323 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.579028+0000", 2024-09-18T04:40:10.323 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032622899999999998 2024-09-18T04:40:10.323 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.323 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.323 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:10.323 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.579343+0000", 2024-09-18T04:40:10.323 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031448899999999999 2024-09-18T04:40:10.323 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.323 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.323 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:10.324 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.590078+0000", 2024-09-18T04:40:10.324 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010734946 2024-09-18T04:40:10.324 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.324 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.324 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:10.324 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600089+0000", 2024-09-18T04:40:10.324 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010011195000000001 2024-09-18T04:40:10.324 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.324 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.324 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:10.324 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600177+0000", 2024-09-18T04:40:10.324 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8425999999999997e-05 2024-09-18T04:40:10.324 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.325 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.325 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:10.325 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600228+0000", 2024-09-18T04:40:10.325 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0216999999999997e-05 2024-09-18T04:40:10.325 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:10.325 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:10.325 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:10.325 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.325 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.325 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:54 2.0 2:04f17ef6:ns::rbd_data.133e813ab5bc.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:10.325 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:40:10.325 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 87.666458532000007, 2024-09-18T04:40:10.325 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.024612999, 2024-09-18T04:40:10.326 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:10.326 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:10.326 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:10.326 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:40:10.326 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:10.326 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 54 2024-09-18T04:40:10.326 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.326 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:10.326 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.326 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:10.326 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:40:10.326 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:10.326 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.327 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.327 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:10.327 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:40:10.327 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:10.327 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.327 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.327 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:10.327 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585239+0000", 2024-09-18T04:40:10.327 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:10.327 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.327 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.327 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:10.327 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593702+0000", 2024-09-18T04:40:10.327 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0084632990000000005 2024-09-18T04:40:10.328 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.328 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.328 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:10.328 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593719+0000", 2024-09-18T04:40:10.328 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.664e-05 2024-09-18T04:40:10.328 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.328 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.328 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:10.328 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593727+0000", 2024-09-18T04:40:10.328 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6839999999999996e-06 2024-09-18T04:40:10.328 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.328 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.328 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:10.328 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593767+0000", 2024-09-18T04:40:10.329 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0330999999999997e-05 2024-09-18T04:40:10.329 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.329 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.329 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:10.329 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593905+0000", 2024-09-18T04:40:10.329 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013754800000000001 2024-09-18T04:40:10.329 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.329 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.329 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:10.329 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.594013+0000", 2024-09-18T04:40:10.329 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000108805 2024-09-18T04:40:10.329 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.329 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.329 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:10.330 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604678+0000", 2024-09-18T04:40:10.330 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010664391 2024-09-18T04:40:10.330 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.330 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.330 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:10.330 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609662+0000", 2024-09-18T04:40:10.330 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0049846100000000004 2024-09-18T04:40:10.330 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.330 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.330 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:10.330 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609788+0000", 2024-09-18T04:40:10.330 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012518800000000001 2024-09-18T04:40:10.330 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.330 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.330 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:10.330 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609854+0000", 2024-09-18T04:40:10.331 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6556000000000004e-05 2024-09-18T04:40:10.331 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:10.331 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:10.331 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:10.331 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.331 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.331 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:58 2.7 2:fc7c6cd3:ns::rbd_data.133e813ab5bc.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:10.331 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:40:10.331 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 87.653616103000005, 2024-09-18T04:40:10.331 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.027690566999999999, 2024-09-18T04:40:10.331 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:10.331 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:10.331 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:10.331 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:40:10.331 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:10.332 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 58 2024-09-18T04:40:10.332 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.332 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:10.332 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.332 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:10.332 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:40:10.332 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:10.332 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.332 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.332 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:10.332 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:40:10.333 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:10.333 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.333 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.333 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:10.333 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598082+0000", 2024-09-18T04:40:10.333 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:10.333 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.333 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.333 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:10.333 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604832+0000", 2024-09-18T04:40:10.333 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0067504510000000002 2024-09-18T04:40:10.333 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.334 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.334 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:10.334 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604852+0000", 2024-09-18T04:40:10.334 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0228999999999999e-05 2024-09-18T04:40:10.334 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.334 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.334 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:10.334 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604870+0000", 2024-09-18T04:40:10.334 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7952e-05 2024-09-18T04:40:10.334 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.334 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.334 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:10.334 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604936+0000", 2024-09-18T04:40:10.334 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5431000000000004e-05 2024-09-18T04:40:10.334 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.335 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.335 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:10.335 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605085+0000", 2024-09-18T04:40:10.335 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014910300000000001 2024-09-18T04:40:10.335 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.335 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.335 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:40:10.335 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605205+0000", 2024-09-18T04:40:10.335 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000119887 2024-09-18T04:40:10.335 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.335 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.335 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:10.335 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625586+0000", 2024-09-18T04:40:10.335 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020381399000000001 2024-09-18T04:40:10.335 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.336 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.336 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:10.336 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625628+0000", 2024-09-18T04:40:10.336 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1873999999999999e-05 2024-09-18T04:40:10.336 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.336 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.336 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:10.336 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625722+0000", 2024-09-18T04:40:10.336 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3787000000000002e-05 2024-09-18T04:40:10.336 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.336 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.336 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:10.336 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625774+0000", 2024-09-18T04:40:10.336 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2503999999999997e-05 2024-09-18T04:40:10.336 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:10.337 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:10.337 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:10.337 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.337 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.337 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:56 2.6 e51/17 2:72453c3e:ns::rbd_data.133e813ab5bc.000000000000000d:head v 51'10, mlcod=51'8)", 2024-09-18T04:40:10.337 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:40:10.337 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 87.646418995000005, 2024-09-18T04:40:10.337 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019766552999999999, 2024-09-18T04:40:10.337 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:10.337 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:10.337 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:10.337 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.337 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:10.337 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:40:10.337 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:10.337 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.338 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.338 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:10.338 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:40:10.338 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:10.338 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.338 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.338 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:10.338 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605279+0000", 2024-09-18T04:40:10.338 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:10.338 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.338 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.338 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:10.338 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609417+0000", 2024-09-18T04:40:10.338 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0041378059999999999 2024-09-18T04:40:10.338 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.339 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.339 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:10.339 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609443+0000", 2024-09-18T04:40:10.340 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6098e-05 2024-09-18T04:40:10.340 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.340 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.340 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:10.340 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609458+0000", 2024-09-18T04:40:10.340 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4671000000000001e-05 2024-09-18T04:40:10.340 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.340 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.340 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:10.340 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609553+0000", 2024-09-18T04:40:10.340 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5265000000000004e-05 2024-09-18T04:40:10.340 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.340 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.340 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:10.341 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609583+0000", 2024-09-18T04:40:10.341 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9839e-05 2024-09-18T04:40:10.341 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.341 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.341 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:10.341 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625004+0000", 2024-09-18T04:40:10.341 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015420639999999999 2024-09-18T04:40:10.341 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.341 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.341 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:10.341 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625047+0000", 2024-09-18T04:40:10.341 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3665999999999998e-05 2024-09-18T04:40:10.341 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:10.341 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:10.341 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:10.341 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.342 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.342 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:99 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:40:10.342 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.582718+0000", 2024-09-18T04:40:10.342 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 73.668982086, 2024-09-18T04:40:10.342 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019362575, 2024-09-18T04:40:10.342 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:10.342 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:10.342 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:10.342 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:40:10.342 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:40:10.342 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 99 2024-09-18T04:40:10.342 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.342 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:10.342 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.343 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:10.343 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582718+0000", 2024-09-18T04:40:10.343 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:10.343 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.343 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.343 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:10.343 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582718+0000", 2024-09-18T04:40:10.343 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:10.343 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.343 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.343 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:10.344 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582717+0000", 2024-09-18T04:40:10.344 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:40:10.344 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.344 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.344 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:10.344 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582720+0000", 2024-09-18T04:40:10.345 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2239999999999998e-06 2024-09-18T04:40:10.345 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.345 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.345 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:10.345 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582723+0000", 2024-09-18T04:40:10.345 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.258e-06 2024-09-18T04:40:10.346 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.346 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.346 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:10.346 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582726+0000", 2024-09-18T04:40:10.346 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8870000000000001e-06 2024-09-18T04:40:10.346 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.346 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.346 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:10.346 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.592018+0000", 2024-09-18T04:40:10.347 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0092926420000000003 2024-09-18T04:40:10.347 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.347 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.347 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:10.347 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.592214+0000", 2024-09-18T04:40:10.347 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000195267 2024-09-18T04:40:10.347 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.347 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.347 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:10.347 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.592913+0000", 2024-09-18T04:40:10.347 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00069924600000000005 2024-09-18T04:40:10.347 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.347 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.347 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:10.348 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597493+0000", 2024-09-18T04:40:10.348 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045797629999999997 2024-09-18T04:40:10.348 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.348 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.348 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:10.348 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.602009+0000", 2024-09-18T04:40:10.348 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045159559999999998 2024-09-18T04:40:10.348 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.348 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.348 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:10.348 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.602042+0000", 2024-09-18T04:40:10.348 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3084999999999999e-05 2024-09-18T04:40:10.348 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.349 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.349 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:10.349 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.602080+0000", 2024-09-18T04:40:10.349 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8600000000000003e-05 2024-09-18T04:40:10.349 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:10.349 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:10.349 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:10.349 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.349 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.349 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:104 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:40:10.349 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.583165+0000", 2024-09-18T04:40:10.349 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 73.668534788000002, 2024-09-18T04:40:10.349 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023679603, 2024-09-18T04:40:10.349 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:10.350 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:10.350 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:10.350 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:40:10.350 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:40:10.350 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 104 2024-09-18T04:40:10.350 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.350 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:10.350 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.350 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:10.350 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583165+0000", 2024-09-18T04:40:10.350 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:10.350 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.351 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.351 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:10.351 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583165+0000", 2024-09-18T04:40:10.351 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:10.351 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.351 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.351 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:10.351 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583164+0000", 2024-09-18T04:40:10.351 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:40:10.351 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.351 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.351 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:10.351 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583167+0000", 2024-09-18T04:40:10.352 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9210000000000001e-06 2024-09-18T04:40:10.352 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.352 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.352 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:10.352 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583171+0000", 2024-09-18T04:40:10.352 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9759999999999997e-06 2024-09-18T04:40:10.352 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.352 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.352 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:10.352 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583174+0000", 2024-09-18T04:40:10.352 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9960000000000002e-06 2024-09-18T04:40:10.352 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.352 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.353 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:10.353 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.594373+0000", 2024-09-18T04:40:10.353 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011198665999999999 2024-09-18T04:40:10.353 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.353 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.353 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:10.353 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.594434+0000", 2024-09-18T04:40:10.353 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.1446999999999996e-05 2024-09-18T04:40:10.353 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.353 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.353 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:10.353 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.594692+0000", 2024-09-18T04:40:10.353 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00025756599999999998 2024-09-18T04:40:10.353 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.353 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.354 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:10.354 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599302+0000", 2024-09-18T04:40:10.354 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0046099640000000002 2024-09-18T04:40:10.354 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.354 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.354 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:10.354 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606710+0000", 2024-09-18T04:40:10.354 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0074077739999999998 2024-09-18T04:40:10.354 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.354 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.354 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:10.354 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606798+0000", 2024-09-18T04:40:10.354 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8133999999999993e-05 2024-09-18T04:40:10.355 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.355 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.355 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:10.355 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606845+0000", 2024-09-18T04:40:10.355 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6653000000000002e-05 2024-09-18T04:40:10.355 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:10.355 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:10.355 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:10.356 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.356 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.356 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:105 2.0 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:40:10.356 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.583193+0000", 2024-09-18T04:40:10.356 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 73.668507168999994, 2024-09-18T04:40:10.356 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023713649, 2024-09-18T04:40:10.356 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:10.356 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:10.356 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:10.357 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:40:10.357 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:40:10.357 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 105 2024-09-18T04:40:10.357 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.357 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:10.357 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.357 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:10.357 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583193+0000", 2024-09-18T04:40:10.357 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:10.357 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.357 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.357 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:10.357 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583193+0000", 2024-09-18T04:40:10.358 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:10.358 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.358 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.358 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:10.358 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583192+0000", 2024-09-18T04:40:10.358 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:40:10.358 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.358 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.358 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:10.358 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583194+0000", 2024-09-18T04:40:10.358 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.305e-06 2024-09-18T04:40:10.358 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.359 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.359 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:10.359 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583198+0000", 2024-09-18T04:40:10.359 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9369999999999997e-06 2024-09-18T04:40:10.359 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.359 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.359 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:10.359 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583202+0000", 2024-09-18T04:40:10.359 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6480000000000001e-06 2024-09-18T04:40:10.359 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.359 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.359 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:10.359 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.595304+0000", 2024-09-18T04:40:10.360 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012102063999999999 2024-09-18T04:40:10.360 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.360 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.360 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:10.360 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.595390+0000", 2024-09-18T04:40:10.360 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5493000000000003e-05 2024-09-18T04:40:10.360 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.360 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.360 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:10.360 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.595509+0000", 2024-09-18T04:40:10.360 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011966199999999999 2024-09-18T04:40:10.360 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.360 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.360 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:10.361 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599358+0000", 2024-09-18T04:40:10.361 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0038486689999999999 2024-09-18T04:40:10.361 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.361 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.361 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:10.361 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606859+0000", 2024-09-18T04:40:10.361 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0075010789999999999 2024-09-18T04:40:10.361 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.361 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.361 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:10.361 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606887+0000", 2024-09-18T04:40:10.361 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7773e-05 2024-09-18T04:40:10.361 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.361 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.362 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:10.362 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606906+0000", 2024-09-18T04:40:10.362 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9417000000000001e-05 2024-09-18T04:40:10.362 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:10.362 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:10.362 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:10.362 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.362 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.362 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:106 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:40:10.362 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.584425+0000", 2024-09-18T04:40:10.362 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 73.667274872999997, 2024-09-18T04:40:10.362 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.022529293999999998, 2024-09-18T04:40:10.362 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:10.362 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:10.363 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:10.363 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:40:10.363 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:40:10.363 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 106 2024-09-18T04:40:10.363 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.363 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:10.363 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.363 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:10.363 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584425+0000", 2024-09-18T04:40:10.363 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:10.363 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.363 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.363 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:10.363 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584425+0000", 2024-09-18T04:40:10.363 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:10.364 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.364 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.364 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:10.364 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584424+0000", 2024-09-18T04:40:10.364 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:40:10.364 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.364 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.364 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:10.364 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584430+0000", 2024-09-18T04:40:10.364 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6389999999999997e-06 2024-09-18T04:40:10.364 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.364 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.364 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:10.365 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584456+0000", 2024-09-18T04:40:10.365 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6188999999999999e-05 2024-09-18T04:40:10.365 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.365 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.365 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:10.365 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584463+0000", 2024-09-18T04:40:10.365 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7340000000000002e-06 2024-09-18T04:40:10.365 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.365 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.365 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:10.365 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597537+0000", 2024-09-18T04:40:10.365 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013074161000000001 2024-09-18T04:40:10.366 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.366 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.366 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:10.366 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597598+0000", 2024-09-18T04:40:10.366 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0930000000000001e-05 2024-09-18T04:40:10.366 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.366 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.366 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:10.366 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597922+0000", 2024-09-18T04:40:10.366 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032473000000000002 2024-09-18T04:40:10.366 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.366 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.366 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:10.366 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.601103+0000", 2024-09-18T04:40:10.367 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0031809849999999999 2024-09-18T04:40:10.367 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.367 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.367 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:10.367 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606912+0000", 2024-09-18T04:40:10.367 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058083550000000003 2024-09-18T04:40:10.367 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.367 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.367 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:10.367 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606935+0000", 2024-09-18T04:40:10.367 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2759999999999999e-05 2024-09-18T04:40:10.368 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.368 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.368 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:10.368 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606954+0000", 2024-09-18T04:40:10.368 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9602999999999998e-05 2024-09-18T04:40:10.368 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:10.368 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:10.368 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:10.368 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.368 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.368 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:109 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:40:10.368 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.585231+0000", 2024-09-18T04:40:10.369 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 73.666468253000005, 2024-09-18T04:40:10.369 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.021767496000000001, 2024-09-18T04:40:10.369 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:10.369 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:10.369 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:10.369 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:40:10.369 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:40:10.369 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 109 2024-09-18T04:40:10.369 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.369 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:10.369 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.370 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:10.370 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585231+0000", 2024-09-18T04:40:10.370 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:10.370 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.370 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.370 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:10.370 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585231+0000", 2024-09-18T04:40:10.370 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:10.370 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.370 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.370 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:10.370 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585230+0000", 2024-09-18T04:40:10.370 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:40:10.371 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.371 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.371 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:10.371 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585237+0000", 2024-09-18T04:40:10.371 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.618e-06 2024-09-18T04:40:10.371 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.371 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.371 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:10.371 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585256+0000", 2024-09-18T04:40:10.371 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9048000000000001e-05 2024-09-18T04:40:10.371 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.371 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.371 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:10.372 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585274+0000", 2024-09-18T04:40:10.372 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8153000000000001e-05 2024-09-18T04:40:10.372 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.372 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.372 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:10.372 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.598284+0000", 2024-09-18T04:40:10.372 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013009682999999999 2024-09-18T04:40:10.372 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.372 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.372 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:10.372 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.598383+0000", 2024-09-18T04:40:10.372 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.8722999999999997e-05 2024-09-18T04:40:10.372 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.372 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.373 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:10.373 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.598829+0000", 2024-09-18T04:40:10.373 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00044623999999999998 2024-09-18T04:40:10.373 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.373 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.373 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:10.373 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.601149+0000", 2024-09-18T04:40:10.373 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0023204150000000002 2024-09-18T04:40:10.373 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.373 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.373 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:10.373 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606960+0000", 2024-09-18T04:40:10.373 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058106390000000003 2024-09-18T04:40:10.374 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.374 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.374 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:10.374 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606982+0000", 2024-09-18T04:40:10.374 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2252e-05 2024-09-18T04:40:10.374 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.374 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.374 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:10.374 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606999+0000", 2024-09-18T04:40:10.374 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6871000000000002e-05 2024-09-18T04:40:10.374 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:10.374 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:10.374 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:10.374 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.375 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.375 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:110 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:40:10.375 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.586640+0000", 2024-09-18T04:40:10.375 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 73.665059933999999, 2024-09-18T04:40:10.375 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020400911000000001, 2024-09-18T04:40:10.375 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:10.375 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:10.375 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:10.375 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:40:10.375 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:40:10.375 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 110 2024-09-18T04:40:10.375 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.375 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:10.375 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.376 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:10.376 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586640+0000", 2024-09-18T04:40:10.376 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:10.376 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.376 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.376 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:10.376 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586640+0000", 2024-09-18T04:40:10.376 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:10.376 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.376 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.376 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:10.376 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586639+0000", 2024-09-18T04:40:10.376 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:40:10.377 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.377 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.377 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:10.377 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586652+0000", 2024-09-18T04:40:10.377 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2482e-05 2024-09-18T04:40:10.377 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.377 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.377 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:10.377 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586660+0000", 2024-09-18T04:40:10.377 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1659999999999994e-06 2024-09-18T04:40:10.377 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.377 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.377 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:10.377 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586666+0000", 2024-09-18T04:40:10.378 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6900000000000003e-06 2024-09-18T04:40:10.378 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.378 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.378 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:10.378 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599381+0000", 2024-09-18T04:40:10.378 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012714328 2024-09-18T04:40:10.378 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.378 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.378 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:10.378 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599436+0000", 2024-09-18T04:40:10.378 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4802999999999997e-05 2024-09-18T04:40:10.379 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.379 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.379 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:10.379 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599629+0000", 2024-09-18T04:40:10.379 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000193921 2024-09-18T04:40:10.379 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.379 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.379 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:10.379 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.601174+0000", 2024-09-18T04:40:10.379 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001544857 2024-09-18T04:40:10.380 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.380 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.380 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:10.380 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.607004+0000", 2024-09-18T04:40:10.380 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058300820000000003 2024-09-18T04:40:10.380 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.380 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.380 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:10.380 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.607024+0000", 2024-09-18T04:40:10.380 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9888e-05 2024-09-18T04:40:10.380 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:10.380 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:10.380 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:10.381 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.607041+0000", 2024-09-18T04:40:10.381 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6450999999999999e-05 2024-09-18T04:40:10.381 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:10.381 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:10.381 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:10.381 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:10.381 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:10.381 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:40:10.383 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:40:10.383 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:40:10.380+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:40:10.464 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T04:40:10.483 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:40:10.480+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:40:10.541 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:40:10.541 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:40:10.541 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:40:10.541 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:40:10.552 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:40:10.584 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:40:10.580+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:40:10.615 INFO:tasks.workunit.client.0.smithi115.stderr:++ rbd journal inspect --pool rbd --image testrbdnbd17342 2024-09-18T04:40:10.615 INFO:tasks.workunit.client.0.smithi115.stderr:++ awk '/entries inspected/ {print $1}' 2024-09-18T04:40:10.621 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:40:10.641 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:40:10.661 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:40:10.667 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:40:10.667 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:40:10.667 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:40:10.667 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:40:10.667 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:40:10.677 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:40:10.683 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:40:10.685 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:40:10.679+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:40:10.705 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:40:10.710 INFO:tasks.workunit.client.0.smithi115.stderr:2024-09-18T04:40:10.707+0000 7f904effd700 -1 JournalMetadata: 0x55df6753adf0 failed to locate client: INSPECT 2024-09-18T04:40:10.721 INFO:tasks.workunit.client.0.smithi115.stderr:+ queue_length=4 2024-09-18T04:40:10.721 INFO:tasks.workunit.client.0.smithi115.stderr:+ tries=2 2024-09-18T04:40:10.721 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 2 -lt 10 ']' 2024-09-18T04:40:10.721 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 4 -gt 0 ']' 2024-09-18T04:40:10.721 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd journal inspect --pool rbd --image testrbdnbd17342 --verbose 2024-09-18T04:40:10.728 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:40:10.730 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:40:10.730 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:40:10.731 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:40:10.731 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:40:10.731 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:40:10.731 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:40:10.786 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:40:10.784+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:40:10.817 INFO:tasks.workunit.client.0.smithi115.stdout:Entry: tag_id=1, commit_tid=1 2024-09-18T04:40:10.817 INFO:tasks.workunit.client.0.smithi115.stdout:{ 2024-09-18T04:40:10.817 INFO:tasks.workunit.client.0.smithi115.stdout: "event_type": "AioDiscard", 2024-09-18T04:40:10.818 INFO:tasks.workunit.client.0.smithi115.stdout: "offset": 0, 2024-09-18T04:40:10.818 INFO:tasks.workunit.client.0.smithi115.stdout: "length": 4096, 2024-09-18T04:40:10.818 INFO:tasks.workunit.client.0.smithi115.stdout: "discard_granularity_bytes": 4096, 2024-09-18T04:40:10.818 INFO:tasks.workunit.client.0.smithi115.stdout: "timestamp": "2024-09-18T04:40:07.691875+0000" 2024-09-18T04:40:10.818 INFO:tasks.workunit.client.0.smithi115.stdout:} 2024-09-18T04:40:10.818 INFO:tasks.workunit.client.0.smithi115.stdout:Entry: tag_id=1, commit_tid=2 2024-09-18T04:40:10.818 INFO:tasks.workunit.client.0.smithi115.stdout:{ 2024-09-18T04:40:10.818 INFO:tasks.workunit.client.0.smithi115.stdout: "event_type": "AioDiscard", 2024-09-18T04:40:10.818 INFO:tasks.workunit.client.0.smithi115.stdout: "offset": 0, 2024-09-18T04:40:10.818 INFO:tasks.workunit.client.0.smithi115.stdout: "length": 4096, 2024-09-18T04:40:10.818 INFO:tasks.workunit.client.0.smithi115.stdout: "discard_granularity_bytes": 4096, 2024-09-18T04:40:10.819 INFO:tasks.workunit.client.0.smithi115.stdout: "timestamp": "2024-09-18T04:40:07.714307+0000" 2024-09-18T04:40:10.819 INFO:tasks.workunit.client.0.smithi115.stdout:} 2024-09-18T04:40:10.819 INFO:tasks.workunit.client.0.smithi115.stdout:Entry: tag_id=1, commit_tid=3 2024-09-18T04:40:10.819 INFO:tasks.workunit.client.0.smithi115.stdout:{ 2024-09-18T04:40:10.819 INFO:tasks.workunit.client.0.smithi115.stdout: "event_type": "AioDiscard", 2024-09-18T04:40:10.819 INFO:tasks.workunit.client.0.smithi115.stdout: "offset": 4096, 2024-09-18T04:40:10.819 INFO:tasks.workunit.client.0.smithi115.stdout: "length": 4096, 2024-09-18T04:40:10.819 INFO:tasks.workunit.client.0.smithi115.stdout: "discard_granularity_bytes": 4096, 2024-09-18T04:40:10.819 INFO:tasks.workunit.client.0.smithi115.stdout: "timestamp": "2024-09-18T04:40:07.745921+0000" 2024-09-18T04:40:10.819 INFO:tasks.workunit.client.0.smithi115.stdout:} 2024-09-18T04:40:10.819 INFO:tasks.workunit.client.0.smithi115.stdout:Entry: tag_id=1, commit_tid=4 2024-09-18T04:40:10.820 INFO:tasks.workunit.client.0.smithi115.stdout:{ 2024-09-18T04:40:10.820 INFO:tasks.workunit.client.0.smithi115.stdout: "event_type": "AioDiscard", 2024-09-18T04:40:10.820 INFO:tasks.workunit.client.0.smithi115.stdout: "offset": 4096, 2024-09-18T04:40:10.820 INFO:tasks.workunit.client.0.smithi115.stdout: "length": 4096, 2024-09-18T04:40:10.820 INFO:tasks.workunit.client.0.smithi115.stdout: "discard_granularity_bytes": 4096, 2024-09-18T04:40:10.820 INFO:tasks.workunit.client.0.smithi115.stdout: "timestamp": "2024-09-18T04:40:07.765789+0000" 2024-09-18T04:40:10.820 INFO:tasks.workunit.client.0.smithi115.stdout:} 2024-09-18T04:40:10.820 INFO:tasks.workunit.client.0.smithi115.stdout:Summary: 2024-09-18T04:40:10.820 INFO:tasks.workunit.client.0.smithi115.stdout: 4 entries inspected, 0 errors 2024-09-18T04:40:10.825 INFO:tasks.workunit.client.0.smithi115.stderr:2024-09-18T04:40:10.819+0000 7ff0e8eee700 -1 JournalMetadata: 0x5622900f94c0 failed to locate client: INSPECT 2024-09-18T04:40:10.832 INFO:tasks.workunit.client.0.smithi115.stderr:+ sleep 1 2024-09-18T04:40:10.840 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:10.840 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:10.854 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:40:10.861 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.0 to 1 (10000) 2024-09-18T04:40:10.876 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-18T04:40:10.887 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:40:10.883+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:40:10.958 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:10.958 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:10.968 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:40:10.988 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:40:10.983+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:40:11.075 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:11.075 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:11.085 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:40:11.089 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:40:11.083+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:40:11.190 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:40:11.188+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:40:11.194 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:11.194 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:11.204 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:40:11.290 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:40:11.288+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:40:11.312 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:11.312 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:11.322 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:40:11.391 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:40:11.387+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:40:11.426 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:11.426 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:11.436 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:40:11.494 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:40:11.487+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:40:11.542 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:11.542 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:11.552 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:40:11.595 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:40:11.592+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:40:11.661 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:11.662 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:11.672 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:40:11.696 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:40:11.692+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:40:11.779 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:11.779 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:11.789 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:40:11.834 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:40:11.831+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:40:11.835 INFO:tasks.workunit.client.0.smithi115.stderr:++ rbd journal inspect --pool rbd --image testrbdnbd17342 2024-09-18T04:40:11.835 INFO:tasks.workunit.client.0.smithi115.stderr:++ awk '/entries inspected/ {print $1}' 2024-09-18T04:40:11.865 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.1 to 1 (10000) 2024-09-18T04:40:11.876 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-18T04:40:11.898 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:40:11.891+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:40:11.905 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:11.905 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:11.905 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-18T04:40:11.919 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:40:11.933 INFO:tasks.workunit.client.0.smithi115.stderr:2024-09-18T04:40:11.927+0000 7f77bc996700 -1 JournalMetadata: 0x5562968f0550 failed to locate client: INSPECT 2024-09-18T04:40:11.942 INFO:tasks.workunit.client.0.smithi115.stderr:+ queue_length=4 2024-09-18T04:40:11.942 INFO:tasks.workunit.client.0.smithi115.stderr:+ tries=3 2024-09-18T04:40:11.943 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 3 -lt 10 ']' 2024-09-18T04:40:11.943 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 4 -gt 0 ']' 2024-09-18T04:40:11.943 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd journal inspect --pool rbd --image testrbdnbd17342 --verbose 2024-09-18T04:40:11.999 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:40:11.996+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:40:12.028 INFO:tasks.workunit.client.0.smithi115.stdout:Entry: tag_id=1, commit_tid=1 2024-09-18T04:40:12.029 INFO:tasks.workunit.client.0.smithi115.stdout:{ 2024-09-18T04:40:12.029 INFO:tasks.workunit.client.0.smithi115.stdout: "event_type": "AioDiscard", 2024-09-18T04:40:12.029 INFO:tasks.workunit.client.0.smithi115.stdout: "offset": 0, 2024-09-18T04:40:12.029 INFO:tasks.workunit.client.0.smithi115.stdout: "length": 4096, 2024-09-18T04:40:12.029 INFO:tasks.workunit.client.0.smithi115.stdout: "discard_granularity_bytes": 4096, 2024-09-18T04:40:12.029 INFO:tasks.workunit.client.0.smithi115.stdout: "timestamp": "2024-09-18T04:40:07.691875+0000" 2024-09-18T04:40:12.029 INFO:tasks.workunit.client.0.smithi115.stdout:} 2024-09-18T04:40:12.029 INFO:tasks.workunit.client.0.smithi115.stdout:Entry: tag_id=1, commit_tid=2 2024-09-18T04:40:12.029 INFO:tasks.workunit.client.0.smithi115.stdout:{ 2024-09-18T04:40:12.030 INFO:tasks.workunit.client.0.smithi115.stdout: "event_type": "AioDiscard", 2024-09-18T04:40:12.030 INFO:tasks.workunit.client.0.smithi115.stdout: "offset": 0, 2024-09-18T04:40:12.030 INFO:tasks.workunit.client.0.smithi115.stdout: "length": 4096, 2024-09-18T04:40:12.030 INFO:tasks.workunit.client.0.smithi115.stdout: "discard_granularity_bytes": 4096, 2024-09-18T04:40:12.030 INFO:tasks.workunit.client.0.smithi115.stdout: "timestamp": "2024-09-18T04:40:07.714307+0000" 2024-09-18T04:40:12.030 INFO:tasks.workunit.client.0.smithi115.stdout:} 2024-09-18T04:40:12.030 INFO:tasks.workunit.client.0.smithi115.stdout:Entry: tag_id=1, commit_tid=3 2024-09-18T04:40:12.030 INFO:tasks.workunit.client.0.smithi115.stdout:{ 2024-09-18T04:40:12.030 INFO:tasks.workunit.client.0.smithi115.stdout: "event_type": "AioDiscard", 2024-09-18T04:40:12.030 INFO:tasks.workunit.client.0.smithi115.stdout: "offset": 4096, 2024-09-18T04:40:12.030 INFO:tasks.workunit.client.0.smithi115.stdout: "length": 4096, 2024-09-18T04:40:12.030 INFO:tasks.workunit.client.0.smithi115.stdout: "discard_granularity_bytes": 4096, 2024-09-18T04:40:12.030 INFO:tasks.workunit.client.0.smithi115.stdout: "timestamp": "2024-09-18T04:40:07.745921+0000" 2024-09-18T04:40:12.030 INFO:tasks.workunit.client.0.smithi115.stdout:} 2024-09-18T04:40:12.031 INFO:tasks.workunit.client.0.smithi115.stdout:Entry: tag_id=1, commit_tid=4 2024-09-18T04:40:12.031 INFO:tasks.workunit.client.0.smithi115.stdout:{ 2024-09-18T04:40:12.031 INFO:tasks.workunit.client.0.smithi115.stdout: "event_type": "AioDiscard", 2024-09-18T04:40:12.031 INFO:tasks.workunit.client.0.smithi115.stdout: "offset": 4096, 2024-09-18T04:40:12.031 INFO:tasks.workunit.client.0.smithi115.stdout: "length": 4096, 2024-09-18T04:40:12.031 INFO:tasks.workunit.client.0.smithi115.stdout: "discard_granularity_bytes": 4096, 2024-09-18T04:40:12.031 INFO:tasks.workunit.client.0.smithi115.stdout: "timestamp": "2024-09-18T04:40:07.765789+0000" 2024-09-18T04:40:12.031 INFO:tasks.workunit.client.0.smithi115.stdout:} 2024-09-18T04:40:12.031 INFO:tasks.workunit.client.0.smithi115.stdout:Summary: 2024-09-18T04:40:12.031 INFO:tasks.workunit.client.0.smithi115.stdout: 4 entries inspected, 0 errors 2024-09-18T04:40:12.037 INFO:tasks.workunit.client.0.smithi115.stderr:2024-09-18T04:40:12.031+0000 7f7196ffd700 -1 JournalMetadata: 0x56461d8df1d0 failed to locate client: INSPECT 2024-09-18T04:40:12.044 INFO:tasks.workunit.client.0.smithi115.stderr:+ sleep 1 2024-09-18T04:40:12.072 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:12.073 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:12.082 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:40:12.100 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:40:12.096+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:40:12.201 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:40:12.196+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:40:12.201 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:12.201 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:12.211 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:40:12.301 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:40:12.295+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:40:12.325 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:12.325 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:12.335 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:40:12.402 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:40:12.396+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:40:12.449 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:12.449 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:12.459 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:40:12.503 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:40:12.499+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:40:12.569 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:12.569 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:12.580 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:40:12.604 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:40:12.599+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:40:12.687 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:12.687 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:12.697 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:40:12.705 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:40:12.699+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:40:12.743 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T04:40:12.806 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:40:12.804+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:40:12.806 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:12.806 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:12.816 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:40:12.878 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.3 to 1 (10000) 2024-09-18T04:40:12.883 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-09-18T04:40:12.893 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-18T04:40:12.894 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:40:12.906 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:40:12.903+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:40:12.918 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:40:12.937 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:40:12.960 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:40:12.963 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:12.963 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:12.973 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:40:12.983 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:40:13.007 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:40:13.008 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:40:13.004+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:40:13.009 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:40:13.010 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:40:13.010 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:40:13.010 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:40:13.010 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:40:13.010 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:40:13.046 INFO:tasks.workunit.client.0.smithi115.stderr:++ rbd journal inspect --pool rbd --image testrbdnbd17342 2024-09-18T04:40:13.047 INFO:tasks.workunit.client.0.smithi115.stderr:++ awk '/entries inspected/ {print $1}' 2024-09-18T04:40:13.081 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:40:13.081 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:40:13.081 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:40:13.081 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:40:13.081 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.081 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.5431.0:1 3.3 3:d80af3cb:::rbd_id.testrbdnbd17342:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-18T04:40:13.081 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:13.542818+0000", 2024-09-18T04:40:13.081 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 59.535838722000001, 2024-09-18T04:40:13.082 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048493400000000002, 2024-09-18T04:40:13.082 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:13.082 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:40:13.082 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:13.082 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.5431", 2024-09-18T04:40:13.082 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3147136083", 2024-09-18T04:40:13.082 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 1 2024-09-18T04:40:13.082 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.082 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:13.082 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.082 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:13.082 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542818+0000", 2024-09-18T04:40:13.082 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:13.082 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.082 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.083 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:13.083 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542818+0000", 2024-09-18T04:40:13.083 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:13.083 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.083 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.083 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:13.083 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542816+0000", 2024-09-18T04:40:13.083 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:13.083 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.083 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.083 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:13.083 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542828+0000", 2024-09-18T04:40:13.083 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2699e-05 2024-09-18T04:40:13.083 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.084 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.084 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:13.084 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542852+0000", 2024-09-18T04:40:13.084 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3176e-05 2024-09-18T04:40:13.084 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.084 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.084 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:13.084 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542874+0000", 2024-09-18T04:40:13.084 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1967000000000002e-05 2024-09-18T04:40:13.084 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.084 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.084 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:13.084 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542984+0000", 2024-09-18T04:40:13.084 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011026099999999999 2024-09-18T04:40:13.085 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.085 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.085 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:13.085 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543161+0000", 2024-09-18T04:40:13.085 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017720299999999999 2024-09-18T04:40:13.085 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.085 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.085 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:13.085 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543303+0000", 2024-09-18T04:40:13.085 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014174700000000001 2024-09-18T04:40:13.085 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:13.085 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:13.085 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:13.086 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.086 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.086 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.5431.0:2 3.3 3:d80af3cb:::rbd_id.testrbdnbd17342:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-18T04:40:13.086 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:13.543564+0000", 2024-09-18T04:40:13.086 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 59.535092235999997, 2024-09-18T04:40:13.086 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010399929999999999, 2024-09-18T04:40:13.086 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:13.086 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:40:13.086 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:13.086 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.5431", 2024-09-18T04:40:13.086 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3147136083", 2024-09-18T04:40:13.086 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 2 2024-09-18T04:40:13.086 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.087 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:13.087 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.087 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:13.087 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543564+0000", 2024-09-18T04:40:13.087 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:13.087 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.087 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.087 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:13.087 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543564+0000", 2024-09-18T04:40:13.087 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:13.087 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.087 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.087 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:13.087 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543563+0000", 2024-09-18T04:40:13.088 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:13.088 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.088 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.088 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:13.088 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543571+0000", 2024-09-18T04:40:13.088 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4270000000000008e-06 2024-09-18T04:40:13.088 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.088 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.088 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:13.088 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543591+0000", 2024-09-18T04:40:13.088 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.929e-05 2024-09-18T04:40:13.088 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.088 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.089 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:13.089 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543600+0000", 2024-09-18T04:40:13.089 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.1390000000000001e-06 2024-09-18T04:40:13.089 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.089 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.089 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:13.089 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543724+0000", 2024-09-18T04:40:13.089 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012458400000000001 2024-09-18T04:40:13.089 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.089 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.089 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:13.089 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543968+0000", 2024-09-18T04:40:13.089 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00024323199999999999 2024-09-18T04:40:13.089 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.090 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.090 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:13.090 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.544604+0000", 2024-09-18T04:40:13.090 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00063675799999999996 2024-09-18T04:40:13.090 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:13.090 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:13.090 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:13.090 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.090 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.090 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "MOSDPGPush(3.3 97/96 [PushOp(3:c4c92e5a:::rbd_trash:head, version: 53'1, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:c4c92e5a:::rbd_trash:head@53'1, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-18T04:40:13.090 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:40:05.833692+0000", 2024-09-18T04:40:13.090 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 7.2449645020000002, 2024-09-18T04:40:13.090 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00061637300000000003, 2024-09-18T04:40:13.091 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:13.091 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:40:13.091 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:13.091 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.091 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:13.091 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:05.833692+0000", 2024-09-18T04:40:13.091 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:13.091 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.091 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.091 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:13.091 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:05.833692+0000", 2024-09-18T04:40:13.091 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:13.091 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.091 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.091 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:13.092 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:05.833691+0000", 2024-09-18T04:40:13.092 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:40:13.092 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.092 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.092 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:13.092 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:05.833709+0000", 2024-09-18T04:40:13.092 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7838000000000001e-05 2024-09-18T04:40:13.092 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.092 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.092 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:13.092 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:05.833745+0000", 2024-09-18T04:40:13.092 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5651000000000001e-05 2024-09-18T04:40:13.092 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.093 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.093 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:13.093 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:05.833763+0000", 2024-09-18T04:40:13.093 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8428e-05 2024-09-18T04:40:13.093 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.093 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.093 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:13.093 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:05.833832+0000", 2024-09-18T04:40:13.093 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8919999999999997e-05 2024-09-18T04:40:13.093 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.093 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.093 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:13.093 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:05.833845+0000", 2024-09-18T04:40:13.093 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2680000000000001e-05 2024-09-18T04:40:13.094 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.094 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.094 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:13.094 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:05.834308+0000", 2024-09-18T04:40:13.094 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00046391200000000003 2024-09-18T04:40:13.094 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:13.094 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:13.094 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:13.094 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.094 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.094 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "MOSDPGPush(3.3 97/96 [PushOp(3:d80af3cb:::rbd_id.testrbdnbd17342:head, version: 69'2, data_included: [0~16], data_size: 16, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d80af3cb:::rbd_id.testrbdnbd17342:head@69'2, size: 16, copy_subset: [0~16], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:16, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-18T04:40:13.094 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:40:05.840323+0000", 2024-09-18T04:40:13.094 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 7.2383333759999999, 2024-09-18T04:40:13.094 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00084282300000000003, 2024-09-18T04:40:13.095 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:13.095 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:40:13.095 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:13.095 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.095 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:13.095 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:05.840323+0000", 2024-09-18T04:40:13.095 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:13.095 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.095 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.095 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:13.095 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:05.840323+0000", 2024-09-18T04:40:13.095 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:13.095 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.095 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.095 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:13.096 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:05.840322+0000", 2024-09-18T04:40:13.096 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:40:13.096 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.096 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.096 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:13.096 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:05.840339+0000", 2024-09-18T04:40:13.096 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6767999999999999e-05 2024-09-18T04:40:13.096 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.096 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.096 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:13.096 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:05.840358+0000", 2024-09-18T04:40:13.096 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8946000000000001e-05 2024-09-18T04:40:13.096 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.097 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.097 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:13.097 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:05.840382+0000", 2024-09-18T04:40:13.097 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4408000000000001e-05 2024-09-18T04:40:13.097 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.097 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.097 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:13.097 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:05.840491+0000", 2024-09-18T04:40:13.097 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000108126 2024-09-18T04:40:13.097 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.097 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.097 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:13.097 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:05.840504+0000", 2024-09-18T04:40:13.098 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3519e-05 2024-09-18T04:40:13.098 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.098 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.098 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:13.098 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:05.841166+0000", 2024-09-18T04:40:13.098 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00066203699999999998 2024-09-18T04:40:13.098 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:13.098 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:13.098 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:13.098 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:13.098 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:13.098 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:40:13.099 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:40:13.142 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:40:13.140+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:40:13.152 INFO:tasks.workunit.client.0.smithi115.stderr:2024-09-18T04:40:13.147+0000 7f778a8a7700 -1 JournalMetadata: 0x5634fa6be290 failed to locate client: INSPECT 2024-09-18T04:40:13.164 INFO:tasks.workunit.client.0.smithi115.stderr:+ queue_length=0 2024-09-18T04:40:13.164 INFO:tasks.workunit.client.0.smithi115.stderr:+ tries=4 2024-09-18T04:40:13.164 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 4 -lt 10 ']' 2024-09-18T04:40:13.164 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 0 -gt 0 ']' 2024-09-18T04:40:13.164 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 0 -eq 0 ']' 2024-09-18T04:40:13.164 INFO:tasks.workunit.client.0.smithi115.stderr:+ unmap_device /dev/nbd1 20186 2024-09-18T04:40:13.164 INFO:tasks.workunit.client.0.smithi115.stderr:+ local args=/dev/nbd1 2024-09-18T04:40:13.164 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pid=20186 2024-09-18T04:40:13.164 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd1 2024-09-18T04:40:13.164 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:40:13.164 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:40:13.165 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:40:13.166 INFO:tasks.workunit.client.0.smithi115.stderr:++ which rbd 2024-09-18T04:40:13.169 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/bin/rbd 2024-09-18T04:40:13.169 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:40:13.169 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd1 2024-09-18T04:40:13.210 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:40:13.208+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:40:13.249 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:40:13.249 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:40:13.249 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:40:13.249 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:40:13.259 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:40:13.304 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd device --device-type nbd list 2024-09-18T04:40:13.305 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false grep '^20186\b' 2024-09-18T04:40:13.305 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep '^20186\b' 2024-09-18T04:40:13.311 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:40:13.307+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:40:13.365 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:40:13.366 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:40:13.366 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:40:13.366 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:40:13.366 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:40:13.375 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:40:13.379 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:40:13.380 INFO:tasks.workunit.client.0.smithi115.stderr:+ ps -C rbd-nbd 2024-09-18T04:40:13.380 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false grep '^ *20186\b' 2024-09-18T04:40:13.380 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep '^ *20186\b' 2024-09-18T04:40:13.387 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:40:13.387 INFO:tasks.workunit.client.0.smithi115.stderr:+ sleep 0.5 2024-09-18T04:40:13.412 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:40:13.407+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:40:13.483 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:40:13.483 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:40:13.483 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:40:13.483 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:40:13.483 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.483 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-18T04:40:13.484 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:40:13.484 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 159.007647504, 2024-09-18T04:40:13.484 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0095781189999999995, 2024-09-18T04:40:13.484 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:13.484 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:13.484 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:13.484 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:40:13.484 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:40:13.484 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 1 2024-09-18T04:40:13.484 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.484 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:13.484 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.484 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:13.485 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:40:13.485 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:13.485 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.485 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.485 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:13.485 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:40:13.485 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:13.485 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.485 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.485 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:13.485 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469862+0000", 2024-09-18T04:40:13.485 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:40:13.485 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.486 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.486 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:13.486 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469879+0000", 2024-09-18T04:40:13.486 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7431999999999998e-05 2024-09-18T04:40:13.486 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.486 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.486 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:13.486 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469896+0000", 2024-09-18T04:40:13.486 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6742999999999999e-05 2024-09-18T04:40:13.486 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.486 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.486 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:13.486 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469915+0000", 2024-09-18T04:40:13.487 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8991000000000002e-05 2024-09-18T04:40:13.487 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.487 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.487 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:13.487 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470397+0000", 2024-09-18T04:40:13.487 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048167799999999998 2024-09-18T04:40:13.487 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.487 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.487 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-09-18T04:40:13.487 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470412+0000", 2024-09-18T04:40:13.487 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5526e-05 2024-09-18T04:40:13.487 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.487 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.487 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:13.488 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473279+0000", 2024-09-18T04:40:13.488 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0028667469999999998 2024-09-18T04:40:13.488 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.488 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.488 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:13.488 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473441+0000", 2024-09-18T04:40:13.488 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000162082 2024-09-18T04:40:13.488 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.488 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.488 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:40:13.488 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473573+0000", 2024-09-18T04:40:13.488 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000132032 2024-09-18T04:40:13.488 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.489 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.489 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:13.489 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474826+0000", 2024-09-18T04:40:13.489 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001252717 2024-09-18T04:40:13.489 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.489 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.489 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:13.489 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479307+0000", 2024-09-18T04:40:13.489 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044813220000000003 2024-09-18T04:40:13.489 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.489 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.489 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:13.489 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479366+0000", 2024-09-18T04:40:13.489 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9024999999999998e-05 2024-09-18T04:40:13.490 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.490 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.490 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:13.490 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479443+0000", 2024-09-18T04:40:13.490 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6339000000000002e-05 2024-09-18T04:40:13.490 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:13.490 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:13.490 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:13.490 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.490 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.490 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-18T04:40:13.490 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:40:13.490 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 156.994639084, 2024-09-18T04:40:13.490 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082699560000000002, 2024-09-18T04:40:13.490 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:13.490 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:13.491 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:13.491 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:40:13.491 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:40:13.491 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 7 2024-09-18T04:40:13.491 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.491 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:13.491 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.491 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:13.491 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:40:13.491 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:13.491 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.491 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.491 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:13.491 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:40:13.492 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:13.492 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.492 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.492 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:13.492 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482870+0000", 2024-09-18T04:40:13.492 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999967 2024-09-18T04:40:13.492 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.492 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.492 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:13.492 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482889+0000", 2024-09-18T04:40:13.492 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.965e-05 2024-09-18T04:40:13.492 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.492 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.492 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:13.492 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482938+0000", 2024-09-18T04:40:13.492 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8548999999999999e-05 2024-09-18T04:40:13.493 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.493 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.493 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:13.493 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482963+0000", 2024-09-18T04:40:13.493 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5075999999999998e-05 2024-09-18T04:40:13.493 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.493 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.493 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:13.493 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483121+0000", 2024-09-18T04:40:13.493 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015819400000000001 2024-09-18T04:40:13.493 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.493 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.493 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op must wait for map", 2024-09-18T04:40:13.493 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483138+0000", 2024-09-18T04:40:13.494 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7068999999999999e-05 2024-09-18T04:40:13.494 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.494 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.494 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:13.494 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488666+0000", 2024-09-18T04:40:13.494 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0055282049999999996 2024-09-18T04:40:13.494 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.494 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.494 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:13.494 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488783+0000", 2024-09-18T04:40:13.494 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000116737 2024-09-18T04:40:13.494 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.494 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.494 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:13.495 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488991+0000", 2024-09-18T04:40:13.495 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020781400000000001 2024-09-18T04:40:13.495 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.495 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.495 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:13.495 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490554+0000", 2024-09-18T04:40:13.495 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0015628370000000001 2024-09-18T04:40:13.495 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.495 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.495 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:13.495 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491019+0000", 2024-09-18T04:40:13.495 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000465608 2024-09-18T04:40:13.495 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.495 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.495 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:13.495 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491087+0000", 2024-09-18T04:40:13.496 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7988e-05 2024-09-18T04:40:13.496 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.496 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.496 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:13.496 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491143+0000", 2024-09-18T04:40:13.496 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5538999999999998e-05 2024-09-18T04:40:13.496 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:13.496 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:13.496 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:13.496 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.496 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.496 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:34 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:13.496 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.261758+0000", 2024-09-18T04:40:13.497 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 91.215753933000002, 2024-09-18T04:40:13.497 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.042024191000000002, 2024-09-18T04:40:13.497 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:13.497 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:13.497 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:13.497 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:40:13.497 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:40:13.497 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 34 2024-09-18T04:40:13.497 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.497 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:13.497 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.497 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:13.497 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.261758+0000", 2024-09-18T04:40:13.497 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:13.498 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.498 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.498 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:13.498 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.261758+0000", 2024-09-18T04:40:13.498 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:13.498 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.498 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.498 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:13.498 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.261756+0000", 2024-09-18T04:40:13.498 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:13.498 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.498 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.498 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:13.498 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.270785+0000", 2024-09-18T04:40:13.499 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0090292340000000006 2024-09-18T04:40:13.499 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.499 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.499 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:13.499 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.270890+0000", 2024-09-18T04:40:13.499 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000104541 2024-09-18T04:40:13.499 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.499 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.499 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:13.499 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.270991+0000", 2024-09-18T04:40:13.499 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000100778 2024-09-18T04:40:13.499 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.499 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.500 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:13.500 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.271157+0000", 2024-09-18T04:40:13.500 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016653999999999999 2024-09-18T04:40:13.500 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.500 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.500 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:13.500 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.271591+0000", 2024-09-18T04:40:13.500 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00043409500000000001 2024-09-18T04:40:13.500 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.500 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.500 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:40:13.500 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272017+0000", 2024-09-18T04:40:13.500 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000425769 2024-09-18T04:40:13.501 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.501 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.501 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:13.501 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.294246+0000", 2024-09-18T04:40:13.501 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.022229237999999998 2024-09-18T04:40:13.501 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.501 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.501 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:13.501 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303571+0000", 2024-09-18T04:40:13.501 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0093250239999999995 2024-09-18T04:40:13.501 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.501 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.501 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:13.502 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303705+0000", 2024-09-18T04:40:13.502 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000133831 2024-09-18T04:40:13.502 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.502 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.502 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:13.502 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303782+0000", 2024-09-18T04:40:13.502 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7156000000000005e-05 2024-09-18T04:40:13.502 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:13.502 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:13.502 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:13.502 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.502 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.502 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:36 2.7 e51/32 2:fe1b02ad:::rbd_data.125fc09df6d1.0000000000000002:head v 51'1, mlcod=0'0)", 2024-09-18T04:40:13.502 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.273441+0000", 2024-09-18T04:40:13.503 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 91.204070951999995, 2024-09-18T04:40:13.503 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.032400768000000003, 2024-09-18T04:40:13.503 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:13.503 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:13.503 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:13.503 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.503 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:13.503 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273441+0000", 2024-09-18T04:40:13.503 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:13.503 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.503 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.503 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:13.503 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273441+0000", 2024-09-18T04:40:13.503 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:13.503 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.504 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.504 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:13.504 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273440+0000", 2024-09-18T04:40:13.504 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:40:13.504 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.504 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.504 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:13.504 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278723+0000", 2024-09-18T04:40:13.504 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0052829909999999999 2024-09-18T04:40:13.504 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.504 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.504 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:13.504 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278752+0000", 2024-09-18T04:40:13.504 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9178000000000001e-05 2024-09-18T04:40:13.505 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.505 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.505 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:13.505 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278761+0000", 2024-09-18T04:40:13.505 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7900000000000005e-06 2024-09-18T04:40:13.505 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.505 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.505 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:13.505 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278864+0000", 2024-09-18T04:40:13.505 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000102699 2024-09-18T04:40:13.505 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.505 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.505 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:13.506 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278900+0000", 2024-09-18T04:40:13.506 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6001999999999999e-05 2024-09-18T04:40:13.506 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.506 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.506 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:13.506 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.305793+0000", 2024-09-18T04:40:13.506 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.026893344 2024-09-18T04:40:13.506 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.506 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.506 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:13.506 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.305842+0000", 2024-09-18T04:40:13.506 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8720000000000001e-05 2024-09-18T04:40:13.506 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:13.506 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:13.506 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:13.507 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.507 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.507 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:39 2.7 e51/32 2:e9c909c5:::rbd_data.125fc09df6d1.0000000000000005:head v 51'2, mlcod=0'0)", 2024-09-18T04:40:13.507 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.283175+0000", 2024-09-18T04:40:13.507 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 91.194337262999994, 2024-09-18T04:40:13.507 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.022864102000000001, 2024-09-18T04:40:13.507 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:13.507 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:13.507 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:13.507 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.507 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:13.507 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283175+0000", 2024-09-18T04:40:13.507 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:13.507 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.508 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.508 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:13.508 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283175+0000", 2024-09-18T04:40:13.508 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:13.508 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.508 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.508 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:13.508 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283173+0000", 2024-09-18T04:40:13.508 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:13.508 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.508 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.508 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:13.508 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.285951+0000", 2024-09-18T04:40:13.509 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0027784540000000001 2024-09-18T04:40:13.509 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.509 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.509 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:13.509 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.285970+0000", 2024-09-18T04:40:13.509 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8855000000000002e-05 2024-09-18T04:40:13.509 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.509 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.509 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:13.509 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.285984+0000", 2024-09-18T04:40:13.509 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4252e-05 2024-09-18T04:40:13.509 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.509 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.509 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:13.510 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.286041+0000", 2024-09-18T04:40:13.510 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6431999999999998e-05 2024-09-18T04:40:13.510 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.510 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.510 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:13.510 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.286086+0000", 2024-09-18T04:40:13.510 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5423000000000002e-05 2024-09-18T04:40:13.510 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.510 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.510 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:13.510 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.305998+0000", 2024-09-18T04:40:13.510 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019911697999999999 2024-09-18T04:40:13.510 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.510 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.511 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:13.511 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306039+0000", 2024-09-18T04:40:13.511 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0973999999999998e-05 2024-09-18T04:40:13.511 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:13.511 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:13.511 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:13.511 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.511 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.511 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:50 2.3 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:13.511 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.312563+0000", 2024-09-18T04:40:13.511 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 91.164949235999998, 2024-09-18T04:40:13.511 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.032636001999999997, 2024-09-18T04:40:13.511 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:13.512 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:13.512 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:13.512 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:40:13.512 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:40:13.512 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 50 2024-09-18T04:40:13.512 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.512 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:13.512 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.512 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:13.512 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.312563+0000", 2024-09-18T04:40:13.512 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:13.512 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.512 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.512 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:13.512 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.312563+0000", 2024-09-18T04:40:13.513 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:13.513 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.513 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.513 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:13.513 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.312561+0000", 2024-09-18T04:40:13.513 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:13.513 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.513 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.513 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:13.513 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321709+0000", 2024-09-18T04:40:13.513 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0091486650000000003 2024-09-18T04:40:13.513 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.513 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.513 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:13.514 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321739+0000", 2024-09-18T04:40:13.514 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9666e-05 2024-09-18T04:40:13.514 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.514 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.514 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:13.514 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321751+0000", 2024-09-18T04:40:13.514 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2086000000000001e-05 2024-09-18T04:40:13.514 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.514 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.514 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:13.514 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321833+0000", 2024-09-18T04:40:13.514 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1842999999999996e-05 2024-09-18T04:40:13.514 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.514 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.515 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:13.515 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321981+0000", 2024-09-18T04:40:13.515 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000147441 2024-09-18T04:40:13.515 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.515 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.515 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:40:13.515 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.322109+0000", 2024-09-18T04:40:13.515 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012851200000000001 2024-09-18T04:40:13.515 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.515 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.515 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:13.515 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.332829+0000", 2024-09-18T04:40:13.515 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010720225999999999 2024-09-18T04:40:13.515 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.515 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.516 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:13.516 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.345039+0000", 2024-09-18T04:40:13.516 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012209831000000001 2024-09-18T04:40:13.516 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.516 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.516 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:13.516 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.345120+0000", 2024-09-18T04:40:13.516 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1145999999999999e-05 2024-09-18T04:40:13.516 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.516 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.516 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:13.516 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.345199+0000", 2024-09-18T04:40:13.517 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8546000000000003e-05 2024-09-18T04:40:13.517 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:13.517 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:13.517 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:13.517 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.517 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.517 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:30 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:13.517 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.516122+0000", 2024-09-18T04:40:13.517 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 90.961390073999993, 2024-09-18T04:40:13.517 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0091865030000000004, 2024-09-18T04:40:13.517 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:13.517 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:13.517 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:13.517 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:40:13.518 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:13.518 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 30 2024-09-18T04:40:13.518 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.518 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:13.518 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.518 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:13.518 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516122+0000", 2024-09-18T04:40:13.518 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:13.518 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.518 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.518 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:13.518 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516122+0000", 2024-09-18T04:40:13.518 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:13.518 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.518 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.519 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:13.519 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516120+0000", 2024-09-18T04:40:13.519 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:13.519 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.519 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.519 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:13.519 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516129+0000", 2024-09-18T04:40:13.519 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.85e-06 2024-09-18T04:40:13.519 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.519 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.519 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:13.519 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516155+0000", 2024-09-18T04:40:13.519 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5882000000000001e-05 2024-09-18T04:40:13.520 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.520 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.520 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:13.520 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516175+0000", 2024-09-18T04:40:13.520 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.022e-05 2024-09-18T04:40:13.520 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.520 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.520 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:13.520 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519780+0000", 2024-09-18T04:40:13.520 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003604187 2024-09-18T04:40:13.520 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.520 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.520 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:13.521 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519964+0000", 2024-09-18T04:40:13.521 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00018391999999999999 2024-09-18T04:40:13.521 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.521 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.521 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:40:13.521 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.520794+0000", 2024-09-18T04:40:13.521 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00083078100000000005 2024-09-18T04:40:13.521 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.521 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.521 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:13.521 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.524129+0000", 2024-09-18T04:40:13.521 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033342419999999998 2024-09-18T04:40:13.521 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.521 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.522 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:13.522 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525239+0000", 2024-09-18T04:40:13.522 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001110778 2024-09-18T04:40:13.522 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.522 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.522 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:13.522 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525277+0000", 2024-09-18T04:40:13.522 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7596999999999999e-05 2024-09-18T04:40:13.522 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.522 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.522 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:13.522 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525308+0000", 2024-09-18T04:40:13.522 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.1535999999999998e-05 2024-09-18T04:40:13.523 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:13.523 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:13.523 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:13.523 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.523 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.523 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:32 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:13.523 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.519217+0000", 2024-09-18T04:40:13.523 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 90.958294753999994, 2024-09-18T04:40:13.523 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082266370000000002, 2024-09-18T04:40:13.523 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:13.523 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:13.523 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:13.523 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:40:13.523 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:13.524 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 32 2024-09-18T04:40:13.524 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.524 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:13.524 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.524 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:13.524 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519217+0000", 2024-09-18T04:40:13.524 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:13.524 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.524 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.524 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:13.524 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519217+0000", 2024-09-18T04:40:13.524 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:13.524 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.525 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.526 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:13.526 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519215+0000", 2024-09-18T04:40:13.526 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:13.526 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.526 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.526 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:13.526 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519229+0000", 2024-09-18T04:40:13.526 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3329999999999999e-05 2024-09-18T04:40:13.526 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.526 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.526 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:13.526 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519268+0000", 2024-09-18T04:40:13.526 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9533000000000002e-05 2024-09-18T04:40:13.526 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.526 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.527 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:13.527 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519285+0000", 2024-09-18T04:40:13.527 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6446000000000001e-05 2024-09-18T04:40:13.527 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.527 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.527 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:13.527 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.522358+0000", 2024-09-18T04:40:13.527 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030734310000000002 2024-09-18T04:40:13.527 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.527 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.527 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:13.527 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.522551+0000", 2024-09-18T04:40:13.527 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019317600000000001 2024-09-18T04:40:13.528 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.528 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.528 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:40:13.528 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.523262+0000", 2024-09-18T04:40:13.528 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00071069999999999998 2024-09-18T04:40:13.528 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.528 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.528 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:13.528 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525318+0000", 2024-09-18T04:40:13.528 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0020561469999999999 2024-09-18T04:40:13.528 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.528 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.528 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:13.528 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.527376+0000", 2024-09-18T04:40:13.528 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0020584969999999998 2024-09-18T04:40:13.529 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.529 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.529 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:13.529 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.527416+0000", 2024-09-18T04:40:13.529 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9054000000000003e-05 2024-09-18T04:40:13.529 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.529 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.529 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:13.529 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.527444+0000", 2024-09-18T04:40:13.529 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8347000000000002e-05 2024-09-18T04:40:13.529 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:13.529 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:13.529 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:13.529 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.530 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.530 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:42 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:13.530 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.551160+0000", 2024-09-18T04:40:13.530 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 90.926352362000003, 2024-09-18T04:40:13.530 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010692757000000001, 2024-09-18T04:40:13.530 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:13.530 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:13.530 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:13.530 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:40:13.530 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:13.530 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 42 2024-09-18T04:40:13.530 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.530 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:13.531 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.531 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:13.531 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551160+0000", 2024-09-18T04:40:13.531 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:13.531 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.531 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.531 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:13.531 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551160+0000", 2024-09-18T04:40:13.531 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:13.531 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.531 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.531 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:13.531 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551159+0000", 2024-09-18T04:40:13.532 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:40:13.532 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.532 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.532 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:13.532 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551166+0000", 2024-09-18T04:40:13.532 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1029999999999996e-06 2024-09-18T04:40:13.532 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.532 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.532 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:13.532 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551177+0000", 2024-09-18T04:40:13.532 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1384e-05 2024-09-18T04:40:13.532 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.532 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.532 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:13.533 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551185+0000", 2024-09-18T04:40:13.533 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1990000000000006e-06 2024-09-18T04:40:13.533 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.533 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.533 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:13.533 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552021+0000", 2024-09-18T04:40:13.533 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00083574400000000003 2024-09-18T04:40:13.533 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.533 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.533 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:13.533 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552339+0000", 2024-09-18T04:40:13.533 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031771399999999997 2024-09-18T04:40:13.533 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.533 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.534 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:40:13.534 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.553140+0000", 2024-09-18T04:40:13.534 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00080088100000000003 2024-09-18T04:40:13.534 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.534 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.534 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:13.534 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.554585+0000", 2024-09-18T04:40:13.534 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001445801 2024-09-18T04:40:13.534 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.534 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.534 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:13.534 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.561711+0000", 2024-09-18T04:40:13.534 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071260109999999998 2024-09-18T04:40:13.535 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.535 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.535 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:13.535 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.561790+0000", 2024-09-18T04:40:13.535 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8967000000000005e-05 2024-09-18T04:40:13.535 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.535 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.535 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:13.535 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.561852+0000", 2024-09-18T04:40:13.535 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2045999999999995e-05 2024-09-18T04:40:13.535 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:13.535 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:13.535 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:13.535 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.535 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.536 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:58 2.7 e51/32 2:fc7c6cd3:ns::rbd_data.133e813ab5bc.000000000000000e:head v 51'7, mlcod=51'5)", 2024-09-18T04:40:13.536 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.605333+0000", 2024-09-18T04:40:13.536 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 90.872179157000005, 2024-09-18T04:40:13.536 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020085902999999999, 2024-09-18T04:40:13.536 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:13.536 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:13.536 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:13.536 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.536 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:13.536 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605333+0000", 2024-09-18T04:40:13.536 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:13.536 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.536 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.537 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:13.537 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605333+0000", 2024-09-18T04:40:13.537 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:13.537 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.537 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.537 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:13.537 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605332+0000", 2024-09-18T04:40:13.537 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:13.537 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.537 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.537 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:13.537 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608667+0000", 2024-09-18T04:40:13.537 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033350300000000001 2024-09-18T04:40:13.537 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.538 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.538 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:13.538 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608706+0000", 2024-09-18T04:40:13.538 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8951e-05 2024-09-18T04:40:13.538 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.538 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.538 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:13.538 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608725+0000", 2024-09-18T04:40:13.538 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9675999999999999e-05 2024-09-18T04:40:13.538 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.538 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.538 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:13.538 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608798+0000", 2024-09-18T04:40:13.538 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3135999999999999e-05 2024-09-18T04:40:13.538 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.539 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.539 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:13.539 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608824+0000", 2024-09-18T04:40:13.539 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5284000000000001e-05 2024-09-18T04:40:13.539 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.539 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.539 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:13.539 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625370+0000", 2024-09-18T04:40:13.539 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016545975000000001 2024-09-18T04:40:13.539 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.539 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.539 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:13.539 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625419+0000", 2024-09-18T04:40:13.539 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9076999999999998e-05 2024-09-18T04:40:13.540 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:13.540 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:13.540 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:13.540 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.540 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.540 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:611 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:40:13.540 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.069561+0000", 2024-09-18T04:40:13.540 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 82.407950603000003, 2024-09-18T04:40:13.540 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0072330299999999997, 2024-09-18T04:40:13.540 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:13.540 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:13.540 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:13.540 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:40:13.540 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:40:13.541 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 611 2024-09-18T04:40:13.541 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.541 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:13.541 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.541 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:13.541 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.069561+0000", 2024-09-18T04:40:13.541 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:13.541 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.541 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.541 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:13.541 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.069561+0000", 2024-09-18T04:40:13.541 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:13.541 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.541 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.542 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:13.542 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.069559+0000", 2024-09-18T04:40:13.542 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:40:13.542 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.542 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.542 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:13.542 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070801+0000", 2024-09-18T04:40:13.542 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0012423359999999999 2024-09-18T04:40:13.542 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.542 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.542 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:13.542 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070828+0000", 2024-09-18T04:40:13.542 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6387e-05 2024-09-18T04:40:13.542 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.543 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.543 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:13.543 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070840+0000", 2024-09-18T04:40:13.543 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2689e-05 2024-09-18T04:40:13.543 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.543 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.543 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:13.543 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070990+0000", 2024-09-18T04:40:13.543 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014946800000000001 2024-09-18T04:40:13.543 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.543 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.543 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:13.543 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.071128+0000", 2024-09-18T04:40:13.544 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013796699999999999 2024-09-18T04:40:13.544 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.544 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.544 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:40:13.544 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.071318+0000", 2024-09-18T04:40:13.544 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000190106 2024-09-18T04:40:13.544 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.544 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.544 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:13.544 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.075711+0000", 2024-09-18T04:40:13.544 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.004393002 2024-09-18T04:40:13.544 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.544 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.544 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:13.545 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.076665+0000", 2024-09-18T04:40:13.545 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00095390399999999999 2024-09-18T04:40:13.546 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.546 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.546 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:13.546 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.076742+0000", 2024-09-18T04:40:13.546 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7112000000000003e-05 2024-09-18T04:40:13.547 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.547 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.547 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:13.547 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.076794+0000", 2024-09-18T04:40:13.547 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2475999999999997e-05 2024-09-18T04:40:13.547 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:13.547 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:13.547 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:13.547 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.547 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.547 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:687 2.3 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:40:13.547 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.487994+0000", 2024-09-18T04:40:13.547 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 81.989517620000001, 2024-09-18T04:40:13.547 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0074167690000000001, 2024-09-18T04:40:13.548 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:13.548 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:13.548 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:13.548 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:40:13.548 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:40:13.548 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 687 2024-09-18T04:40:13.548 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.548 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:13.548 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.548 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:13.548 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.487994+0000", 2024-09-18T04:40:13.548 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:13.548 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.548 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.549 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:13.549 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.487994+0000", 2024-09-18T04:40:13.549 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:13.549 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.549 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.549 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:13.549 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.487993+0000", 2024-09-18T04:40:13.549 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:40:13.549 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.549 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.549 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:13.549 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.488979+0000", 2024-09-18T04:40:13.549 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00098620700000000001 2024-09-18T04:40:13.549 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.549 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.550 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:13.550 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489002+0000", 2024-09-18T04:40:13.550 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2500000000000001e-05 2024-09-18T04:40:13.550 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.550 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.550 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:13.550 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489020+0000", 2024-09-18T04:40:13.550 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7691000000000001e-05 2024-09-18T04:40:13.550 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.550 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.550 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:13.550 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489091+0000", 2024-09-18T04:40:13.550 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1307e-05 2024-09-18T04:40:13.550 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.550 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.551 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:13.551 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489219+0000", 2024-09-18T04:40:13.551 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012766000000000001 2024-09-18T04:40:13.551 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.551 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.551 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:40:13.551 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489387+0000", 2024-09-18T04:40:13.551 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016855400000000001 2024-09-18T04:40:13.551 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.551 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.551 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:13.551 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495243+0000", 2024-09-18T04:40:13.551 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058563189999999996 2024-09-18T04:40:13.551 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.552 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.552 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:13.552 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495266+0000", 2024-09-18T04:40:13.552 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2810999999999999e-05 2024-09-18T04:40:13.552 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.552 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.552 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:13.552 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495332+0000", 2024-09-18T04:40:13.552 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5469999999999995e-05 2024-09-18T04:40:13.552 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.552 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.552 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:13.552 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495411+0000", 2024-09-18T04:40:13.552 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9395999999999997e-05 2024-09-18T04:40:13.553 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:13.553 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:13.553 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:13.553 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.553 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.553 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:689 2.3 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:40:13.553 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.500070+0000", 2024-09-18T04:40:13.553 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 81.977441803999994, 2024-09-18T04:40:13.553 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071096650000000003, 2024-09-18T04:40:13.553 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:13.553 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:13.553 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:13.553 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:40:13.553 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:40:13.553 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 689 2024-09-18T04:40:13.553 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.554 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:13.554 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.554 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:13.554 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.500070+0000", 2024-09-18T04:40:13.554 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:13.554 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.554 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.554 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:13.554 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.500070+0000", 2024-09-18T04:40:13.554 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:13.554 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.554 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.554 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:13.554 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.500069+0000", 2024-09-18T04:40:13.554 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:13.555 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.555 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.555 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:13.555 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501082+0000", 2024-09-18T04:40:13.555 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010136570000000001 2024-09-18T04:40:13.555 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.555 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.555 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:13.555 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501104+0000", 2024-09-18T04:40:13.555 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1883e-05 2024-09-18T04:40:13.555 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.555 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.555 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:13.555 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501115+0000", 2024-09-18T04:40:13.555 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0414000000000001e-05 2024-09-18T04:40:13.555 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.556 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.556 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:13.556 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501178+0000", 2024-09-18T04:40:13.556 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3128999999999998e-05 2024-09-18T04:40:13.556 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.556 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.556 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:13.556 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501257+0000", 2024-09-18T04:40:13.556 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9592999999999995e-05 2024-09-18T04:40:13.556 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.556 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.556 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:40:13.556 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501366+0000", 2024-09-18T04:40:13.556 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000108643 2024-09-18T04:40:13.556 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.557 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.557 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:13.557 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.504041+0000", 2024-09-18T04:40:13.557 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0026750889999999999 2024-09-18T04:40:13.557 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.557 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.557 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:13.557 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.507097+0000", 2024-09-18T04:40:13.557 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030556070000000001 2024-09-18T04:40:13.557 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.557 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.557 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:13.557 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.507140+0000", 2024-09-18T04:40:13.557 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3745000000000002e-05 2024-09-18T04:40:13.558 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.558 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.558 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:13.558 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.507180+0000", 2024-09-18T04:40:13.558 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9475999999999999e-05 2024-09-18T04:40:13.558 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:13.558 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:13.558 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:13.558 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.558 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.558 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4972.0:13 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:40:13.558 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.948263+0000", 2024-09-18T04:40:13.558 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 81.529249012999998, 2024-09-18T04:40:13.558 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0091358970000000005, 2024-09-18T04:40:13.559 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:13.559 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:40:13.559 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:13.559 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4972", 2024-09-18T04:40:13.559 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2280251472", 2024-09-18T04:40:13.559 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 13 2024-09-18T04:40:13.559 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.559 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:13.559 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.559 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:13.559 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948263+0000", 2024-09-18T04:40:13.559 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:13.560 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.560 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.560 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:13.560 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948263+0000", 2024-09-18T04:40:13.560 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:13.560 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.560 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.560 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:13.560 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948261+0000", 2024-09-18T04:40:13.560 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:13.560 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.560 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.560 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:13.560 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948273+0000", 2024-09-18T04:40:13.560 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1576999999999999e-05 2024-09-18T04:40:13.561 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.561 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.561 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:13.561 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948305+0000", 2024-09-18T04:40:13.561 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2209000000000003e-05 2024-09-18T04:40:13.561 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.561 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.561 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:13.561 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948325+0000", 2024-09-18T04:40:13.561 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.022e-05 2024-09-18T04:40:13.561 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.561 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.561 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:13.561 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948455+0000", 2024-09-18T04:40:13.562 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000129624 2024-09-18T04:40:13.562 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.562 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.562 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:13.562 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948598+0000", 2024-09-18T04:40:13.562 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014376699999999999 2024-09-18T04:40:13.562 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.562 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.562 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:13.562 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.957399+0000", 2024-09-18T04:40:13.562 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0088004299999999997 2024-09-18T04:40:13.562 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:13.562 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:13.562 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:13.563 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.563 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.563 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:101 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:40:13.563 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.576367+0000", 2024-09-18T04:40:13.563 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 76.901145305, 2024-09-18T04:40:13.563 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0078671020000000008, 2024-09-18T04:40:13.563 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:13.563 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:13.563 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:13.563 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:40:13.563 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:40:13.563 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 101 2024-09-18T04:40:13.563 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.563 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:13.564 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.564 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:13.564 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576367+0000", 2024-09-18T04:40:13.564 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:13.564 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.564 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.564 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:13.564 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576367+0000", 2024-09-18T04:40:13.564 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:13.564 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.564 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.564 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:13.564 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576364+0000", 2024-09-18T04:40:13.564 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999971 2024-09-18T04:40:13.564 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.565 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.565 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:13.565 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576378+0000", 2024-09-18T04:40:13.565 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3879999999999999e-05 2024-09-18T04:40:13.565 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.565 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.565 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:13.565 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576410+0000", 2024-09-18T04:40:13.565 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2286000000000002e-05 2024-09-18T04:40:13.565 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.565 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.565 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:13.565 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576425+0000", 2024-09-18T04:40:13.566 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5290000000000001e-05 2024-09-18T04:40:13.566 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.566 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.566 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:13.566 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576551+0000", 2024-09-18T04:40:13.566 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012555199999999999 2024-09-18T04:40:13.566 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.566 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.566 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:13.566 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576719+0000", 2024-09-18T04:40:13.566 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016787400000000001 2024-09-18T04:40:13.566 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.566 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.566 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:40:13.566 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576951+0000", 2024-09-18T04:40:13.566 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00023252400000000001 2024-09-18T04:40:13.567 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.567 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.567 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:13.567 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.581459+0000", 2024-09-18T04:40:13.567 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045074410000000001 2024-09-18T04:40:13.567 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.567 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.567 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:13.567 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584076+0000", 2024-09-18T04:40:13.567 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0026173860000000002 2024-09-18T04:40:13.567 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.567 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.567 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:13.567 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584160+0000", 2024-09-18T04:40:13.568 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4191999999999996e-05 2024-09-18T04:40:13.568 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.568 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.568 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:13.568 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584234+0000", 2024-09-18T04:40:13.568 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.339e-05 2024-09-18T04:40:13.568 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:13.568 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:13.568 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:13.568 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.568 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.568 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:44 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:40:13.568 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.550448+0000", 2024-09-18T04:40:13.568 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 72.927064115999997, 2024-09-18T04:40:13.569 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082117590000000008, 2024-09-18T04:40:13.569 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:13.569 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:13.569 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:13.569 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:40:13.569 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:40:13.569 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 44 2024-09-18T04:40:13.569 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.569 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:13.569 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.569 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:13.569 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.550448+0000", 2024-09-18T04:40:13.569 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:13.570 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.570 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.570 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:13.570 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.550448+0000", 2024-09-18T04:40:13.570 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:13.570 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.570 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.570 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:13.570 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.550445+0000", 2024-09-18T04:40:13.570 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999971 2024-09-18T04:40:13.570 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.570 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.570 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:13.570 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.551904+0000", 2024-09-18T04:40:13.570 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001458322 2024-09-18T04:40:13.571 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.571 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.571 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:13.571 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.551938+0000", 2024-09-18T04:40:13.571 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4359000000000001e-05 2024-09-18T04:40:13.571 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.571 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.571 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:13.571 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.551959+0000", 2024-09-18T04:40:13.571 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1403000000000002e-05 2024-09-18T04:40:13.571 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.571 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.571 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:13.571 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.552049+0000", 2024-09-18T04:40:13.572 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9970999999999996e-05 2024-09-18T04:40:13.572 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.572 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.572 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:13.572 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.552190+0000", 2024-09-18T04:40:13.572 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000140271 2024-09-18T04:40:13.572 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.572 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.572 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:40:13.572 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.552385+0000", 2024-09-18T04:40:13.572 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019498700000000001 2024-09-18T04:40:13.572 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.572 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.572 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:13.573 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.556308+0000", 2024-09-18T04:40:13.573 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00392333 2024-09-18T04:40:13.573 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.573 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.573 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:13.573 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.558479+0000", 2024-09-18T04:40:13.573 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0021715889999999998 2024-09-18T04:40:13.573 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.573 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.573 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:13.573 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.558616+0000", 2024-09-18T04:40:13.573 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013613399999999999 2024-09-18T04:40:13.573 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.573 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.574 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:13.574 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.558660+0000", 2024-09-18T04:40:13.574 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4053e-05 2024-09-18T04:40:13.574 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:13.574 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:13.574 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:13.574 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.574 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.574 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:46 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:40:13.574 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.564089+0000", 2024-09-18T04:40:13.574 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 72.913422987000004, 2024-09-18T04:40:13.574 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0069437090000000002, 2024-09-18T04:40:13.574 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:13.575 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:13.575 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:13.575 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:40:13.575 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:40:13.575 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 46 2024-09-18T04:40:13.575 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.575 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:13.575 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.575 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:13.575 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.564089+0000", 2024-09-18T04:40:13.575 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:13.575 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.575 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.575 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:13.576 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.564089+0000", 2024-09-18T04:40:13.576 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:13.576 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.576 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.576 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:13.576 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.564087+0000", 2024-09-18T04:40:13.576 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:13.576 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.576 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.576 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:13.576 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565126+0000", 2024-09-18T04:40:13.576 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001038902 2024-09-18T04:40:13.576 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.576 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.577 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:13.577 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565148+0000", 2024-09-18T04:40:13.577 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1911000000000001e-05 2024-09-18T04:40:13.577 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.577 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.577 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:13.577 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565157+0000", 2024-09-18T04:40:13.577 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6220000000000004e-06 2024-09-18T04:40:13.577 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.577 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.577 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:13.577 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565261+0000", 2024-09-18T04:40:13.577 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000103063 2024-09-18T04:40:13.577 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.578 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.578 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:13.578 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565394+0000", 2024-09-18T04:40:13.578 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000133032 2024-09-18T04:40:13.578 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.578 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.578 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:40:13.578 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.565564+0000", 2024-09-18T04:40:13.578 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017068000000000001 2024-09-18T04:40:13.578 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.578 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.578 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:13.578 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.570556+0000", 2024-09-18T04:40:13.578 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0049920620000000002 2024-09-18T04:40:13.578 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.579 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.579 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:13.579 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.570885+0000", 2024-09-18T04:40:13.579 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032827900000000001 2024-09-18T04:40:13.579 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.579 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.579 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:13.579 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.570959+0000", 2024-09-18T04:40:13.579 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3955999999999995e-05 2024-09-18T04:40:13.579 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.579 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.579 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:13.579 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.571033+0000", 2024-09-18T04:40:13.579 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4191999999999997e-05 2024-09-18T04:40:13.580 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:13.580 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:13.580 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:13.580 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.580 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.580 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:100 2.4 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:40:13.580 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.856389+0000", 2024-09-18T04:40:13.580 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 72.621122874999998, 2024-09-18T04:40:13.580 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0081480800000000003, 2024-09-18T04:40:13.580 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:13.580 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:13.580 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:13.580 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:40:13.580 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:40:13.581 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 100 2024-09-18T04:40:13.581 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.581 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:13.581 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.581 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:13.581 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.856389+0000", 2024-09-18T04:40:13.581 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:13.581 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.581 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.581 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:13.581 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.856389+0000", 2024-09-18T04:40:13.581 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:13.581 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.582 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.582 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:13.582 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.856388+0000", 2024-09-18T04:40:13.582 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:40:13.582 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.582 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.582 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:13.582 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857295+0000", 2024-09-18T04:40:13.582 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00090725599999999995 2024-09-18T04:40:13.582 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.582 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.582 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:13.582 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857311+0000", 2024-09-18T04:40:13.582 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6092000000000001e-05 2024-09-18T04:40:13.583 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.583 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.583 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:13.583 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857319+0000", 2024-09-18T04:40:13.583 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9200000000000004e-06 2024-09-18T04:40:13.583 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.583 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.583 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:13.583 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857388+0000", 2024-09-18T04:40:13.583 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8591999999999997e-05 2024-09-18T04:40:13.583 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.583 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.583 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:13.583 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857535+0000", 2024-09-18T04:40:13.584 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014684299999999999 2024-09-18T04:40:13.584 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.584 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.584 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:40:13.584 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857747+0000", 2024-09-18T04:40:13.584 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000212361 2024-09-18T04:40:13.584 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.584 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.584 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:13.584 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.860831+0000", 2024-09-18T04:40:13.584 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030841789999999998 2024-09-18T04:40:13.584 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.584 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.584 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:13.585 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.864361+0000", 2024-09-18T04:40:13.585 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0035298780000000002 2024-09-18T04:40:13.585 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.585 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.585 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:13.585 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.864456+0000", 2024-09-18T04:40:13.585 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4557999999999998e-05 2024-09-18T04:40:13.585 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.585 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.585 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:13.585 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.864537+0000", 2024-09-18T04:40:13.585 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1577999999999997e-05 2024-09-18T04:40:13.586 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:13.586 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:13.586 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:13.586 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.586 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.586 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:102 2.4 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:40:13.586 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.869773+0000", 2024-09-18T04:40:13.586 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 72.607739218000006, 2024-09-18T04:40:13.586 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0084430359999999992, 2024-09-18T04:40:13.586 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:13.586 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:13.586 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:13.586 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:40:13.587 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:40:13.587 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 102 2024-09-18T04:40:13.587 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.587 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:13.587 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.587 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:13.587 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.869773+0000", 2024-09-18T04:40:13.587 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:13.587 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.587 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.587 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:13.587 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.869773+0000", 2024-09-18T04:40:13.587 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:13.587 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.588 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.588 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:13.588 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.869771+0000", 2024-09-18T04:40:13.588 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:13.588 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.588 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.588 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:13.588 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870629+0000", 2024-09-18T04:40:13.588 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00085760999999999997 2024-09-18T04:40:13.588 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.588 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.588 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:13.588 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870647+0000", 2024-09-18T04:40:13.588 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8363999999999999e-05 2024-09-18T04:40:13.588 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.589 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.589 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:13.589 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870655+0000", 2024-09-18T04:40:13.589 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0099999999999995e-06 2024-09-18T04:40:13.589 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.589 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.589 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:13.589 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870803+0000", 2024-09-18T04:40:13.589 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014798 2024-09-18T04:40:13.589 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.589 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.589 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:13.589 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870951+0000", 2024-09-18T04:40:13.589 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014715999999999999 2024-09-18T04:40:13.589 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.590 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.590 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:40:13.590 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.871145+0000", 2024-09-18T04:40:13.590 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000194253 2024-09-18T04:40:13.590 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.590 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.590 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:13.590 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.877033+0000", 2024-09-18T04:40:13.590 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058883099999999999 2024-09-18T04:40:13.590 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.590 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.590 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:13.590 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.878078+0000", 2024-09-18T04:40:13.590 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001044747 2024-09-18T04:40:13.591 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.591 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.591 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:13.591 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.878160+0000", 2024-09-18T04:40:13.591 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1904999999999996e-05 2024-09-18T04:40:13.591 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.591 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.591 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:13.591 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.878216+0000", 2024-09-18T04:40:13.591 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6082000000000003e-05 2024-09-18T04:40:13.591 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:13.591 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:13.591 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:13.591 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.591 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.592 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:145 2.4 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:40:13.592 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:01.096935+0000", 2024-09-18T04:40:13.592 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 72.380577438000003, 2024-09-18T04:40:13.592 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0069937869999999996, 2024-09-18T04:40:13.592 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:13.592 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:13.592 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:13.592 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:40:13.592 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:40:13.592 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 145 2024-09-18T04:40:13.592 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.592 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:13.592 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.592 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:13.593 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.096935+0000", 2024-09-18T04:40:13.593 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:13.593 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.593 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.593 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:13.593 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.096935+0000", 2024-09-18T04:40:13.593 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:13.593 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.593 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.593 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:13.593 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.096932+0000", 2024-09-18T04:40:13.593 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:40:13.593 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.593 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.594 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:13.594 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097815+0000", 2024-09-18T04:40:13.594 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00088276000000000001 2024-09-18T04:40:13.594 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.594 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.594 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:13.594 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097842+0000", 2024-09-18T04:40:13.594 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6806999999999999e-05 2024-09-18T04:40:13.594 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.594 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.594 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:13.594 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097852+0000", 2024-09-18T04:40:13.594 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0548e-05 2024-09-18T04:40:13.594 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.595 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.595 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:13.595 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097931+0000", 2024-09-18T04:40:13.595 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8417000000000005e-05 2024-09-18T04:40:13.595 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.595 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.595 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:13.595 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.098047+0000", 2024-09-18T04:40:13.595 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011611 2024-09-18T04:40:13.595 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.595 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.595 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:40:13.595 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.098172+0000", 2024-09-18T04:40:13.595 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000125482 2024-09-18T04:40:13.596 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.596 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.596 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:13.596 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.100888+0000", 2024-09-18T04:40:13.596 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0027154430000000001 2024-09-18T04:40:13.596 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.596 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.596 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:13.596 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.103814+0000", 2024-09-18T04:40:13.596 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0029259619999999998 2024-09-18T04:40:13.596 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.596 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.596 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:13.597 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.103891+0000", 2024-09-18T04:40:13.597 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7619000000000003e-05 2024-09-18T04:40:13.597 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:13.597 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:13.597 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:13.597 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.103928+0000", 2024-09-18T04:40:13.597 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6900999999999998e-05 2024-09-18T04:40:13.597 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:13.597 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:13.597 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:13.597 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:13.597 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:13.597 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:40:13.599 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:40:13.600 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:40:13.595+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:40:13.700 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:40:13.695+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:40:13.717 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:40:13.717 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:40:13.717 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:40:13.717 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:40:13.727 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:40:13.801 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:40:13.795+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:40:13.837 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:40:13.837 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:40:13.837 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:40:13.837 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:40:13.837 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:40:13.847 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:40:13.885 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.5 to 1 (10000) 2024-09-18T04:40:13.889 INFO:tasks.workunit.client.0.smithi115.stderr:+ DEV= 2024-09-18T04:40:13.889 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd feature disable rbd/testrbdnbd17342 journaling 2024-09-18T04:40:13.898 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-18T04:40:13.942 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:40:13.939+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:40:13.961 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:40:13.961 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:40:13.961 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:40:13.961 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:40:13.961 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.961 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:7, version: 79'62, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:7@79'62, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 8=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-18T04:40:13.961 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.724969+0000", 2024-09-18T04:40:13.961 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 27.230927566999998, 2024-09-18T04:40:13.961 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0035525930000000002, 2024-09-18T04:40:13.962 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:13.962 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:13.962 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:13.962 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.962 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:13.962 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.724969+0000", 2024-09-18T04:40:13.962 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:13.962 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.962 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.962 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:13.962 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.724969+0000", 2024-09-18T04:40:13.962 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:13.962 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.962 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.963 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:13.963 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.724968+0000", 2024-09-18T04:40:13.963 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:13.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.963 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.963 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:13.963 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.726245+0000", 2024-09-18T04:40:13.963 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001276395 2024-09-18T04:40:13.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.963 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.963 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:13.963 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.726262+0000", 2024-09-18T04:40:13.963 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6973000000000002e-05 2024-09-18T04:40:13.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.964 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.964 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:13.964 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.726293+0000", 2024-09-18T04:40:13.964 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0886999999999998e-05 2024-09-18T04:40:13.964 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.964 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.964 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:13.964 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.726403+0000", 2024-09-18T04:40:13.964 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000110288 2024-09-18T04:40:13.964 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.964 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.964 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:13.964 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.726418+0000", 2024-09-18T04:40:13.965 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5503999999999999e-05 2024-09-18T04:40:13.965 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.965 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.965 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:13.965 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.728522+0000", 2024-09-18T04:40:13.965 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021032630000000002 2024-09-18T04:40:13.965 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:13.965 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:13.965 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:13.965 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.965 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.965 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.1 87/86 [PushOp(2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head, version: 81'220, data_included: [4182016~12288], data_size: 12288, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head@81'220, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:4182016, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:40:13.965 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.730366+0000", 2024-09-18T04:40:13.965 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 27.22553117, 2024-09-18T04:40:13.966 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0032522580000000001, 2024-09-18T04:40:13.966 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:13.966 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:13.966 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:13.966 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.966 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:13.966 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.730366+0000", 2024-09-18T04:40:13.966 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:13.966 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.966 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.966 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:13.966 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.730366+0000", 2024-09-18T04:40:13.966 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:13.966 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.967 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.967 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:13.967 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.730365+0000", 2024-09-18T04:40:13.967 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:13.967 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.967 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.967 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:13.967 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.730385+0000", 2024-09-18T04:40:13.967 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0766000000000001e-05 2024-09-18T04:40:13.967 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.967 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.967 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:13.967 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.730406+0000", 2024-09-18T04:40:13.967 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0706000000000001e-05 2024-09-18T04:40:13.968 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.968 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.968 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:13.968 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.730423+0000", 2024-09-18T04:40:13.968 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7411e-05 2024-09-18T04:40:13.968 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.968 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.968 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:13.968 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.730507+0000", 2024-09-18T04:40:13.968 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3388999999999997e-05 2024-09-18T04:40:13.968 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.968 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.968 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:13.968 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.730523+0000", 2024-09-18T04:40:13.969 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5843999999999999e-05 2024-09-18T04:40:13.969 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.969 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.969 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:13.969 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.733618+0000", 2024-09-18T04:40:13.969 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030951540000000001 2024-09-18T04:40:13.969 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:13.969 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:13.969 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:13.969 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.969 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.969 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:7, version: 79'62, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:7@79'62, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 8=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:40:13.969 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.732222+0000", 2024-09-18T04:40:13.969 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 27.22367521, 2024-09-18T04:40:13.969 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030084410000000002, 2024-09-18T04:40:13.970 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:13.970 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:13.970 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:13.970 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.970 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:13.970 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.732222+0000", 2024-09-18T04:40:13.970 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:13.970 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.970 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.970 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:13.970 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.732222+0000", 2024-09-18T04:40:13.970 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:13.970 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.970 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.970 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:13.971 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.732221+0000", 2024-09-18T04:40:13.971 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:40:13.971 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.971 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.971 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:13.971 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.733174+0000", 2024-09-18T04:40:13.971 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00095350299999999997 2024-09-18T04:40:13.971 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.971 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.971 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:13.971 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.733197+0000", 2024-09-18T04:40:13.971 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2925000000000002e-05 2024-09-18T04:40:13.971 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.971 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.972 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:13.972 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.733218+0000", 2024-09-18T04:40:13.972 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0477999999999999e-05 2024-09-18T04:40:13.972 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.972 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.972 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:13.972 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.733323+0000", 2024-09-18T04:40:13.972 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000105717 2024-09-18T04:40:13.972 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.972 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.972 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:13.972 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.733358+0000", 2024-09-18T04:40:13.972 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4406000000000002e-05 2024-09-18T04:40:13.972 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.973 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.973 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:13.973 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.735230+0000", 2024-09-18T04:40:13.973 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0018720659999999999 2024-09-18T04:40:13.973 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:13.973 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:13.973 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:13.973 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.973 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.973 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:7, version: 79'62, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:7@79'62, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 8=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:40:13.973 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.743433+0000", 2024-09-18T04:40:13.973 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 27.212463729, 2024-09-18T04:40:13.973 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003400237, 2024-09-18T04:40:13.973 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:13.973 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:13.974 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:13.974 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.974 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:13.974 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.743433+0000", 2024-09-18T04:40:13.974 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:13.974 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.974 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.974 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:13.974 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.743433+0000", 2024-09-18T04:40:13.974 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:13.974 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.974 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.974 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:13.974 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.743432+0000", 2024-09-18T04:40:13.974 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:13.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.975 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.975 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:13.975 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.744392+0000", 2024-09-18T04:40:13.975 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00095993499999999998 2024-09-18T04:40:13.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.975 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.975 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:13.975 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.744417+0000", 2024-09-18T04:40:13.975 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5249000000000002e-05 2024-09-18T04:40:13.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.975 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.975 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:13.975 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.744434+0000", 2024-09-18T04:40:13.976 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6597999999999999e-05 2024-09-18T04:40:13.976 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.976 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.976 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:13.976 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.744499+0000", 2024-09-18T04:40:13.976 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4987000000000006e-05 2024-09-18T04:40:13.976 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.976 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.976 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:13.976 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.744506+0000", 2024-09-18T04:40:13.976 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2679999999999997e-06 2024-09-18T04:40:13.976 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.976 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.977 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:13.977 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.746833+0000", 2024-09-18T04:40:13.977 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023270529999999999 2024-09-18T04:40:13.977 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:13.977 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:13.977 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:13.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.977 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.977 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:7, version: 79'64, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:7@79'64, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-18T04:40:13.977 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.750570+0000", 2024-09-18T04:40:13.977 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 27.205326851999999, 2024-09-18T04:40:13.977 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0029248949999999998, 2024-09-18T04:40:13.977 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:13.977 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:13.978 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:13.978 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.978 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:13.978 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.750570+0000", 2024-09-18T04:40:13.978 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:13.978 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.978 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.978 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:13.978 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.750570+0000", 2024-09-18T04:40:13.978 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:13.978 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.978 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.978 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:13.978 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.750569+0000", 2024-09-18T04:40:13.978 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:13.979 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.979 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.979 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:13.979 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.751724+0000", 2024-09-18T04:40:13.979 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001155037 2024-09-18T04:40:13.979 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.979 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.979 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:13.979 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.751747+0000", 2024-09-18T04:40:13.979 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2679000000000001e-05 2024-09-18T04:40:13.979 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.979 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.979 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:13.979 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.751761+0000", 2024-09-18T04:40:13.980 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.467e-05 2024-09-18T04:40:13.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.980 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.980 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:13.980 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.751839+0000", 2024-09-18T04:40:13.980 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7824000000000006e-05 2024-09-18T04:40:13.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.980 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.980 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:13.980 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.751849+0000", 2024-09-18T04:40:13.980 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4159999999999993e-06 2024-09-18T04:40:13.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.980 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.980 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:13.980 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.753495+0000", 2024-09-18T04:40:13.981 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0016461830000000001 2024-09-18T04:40:13.981 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:13.981 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:13.981 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:13.981 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.981 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.981 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.5 87/86 [PushOp(2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:7, version: 79'59, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:7@79'59, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-18T04:40:13.981 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.752479+0000", 2024-09-18T04:40:13.981 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 27.203417484999999, 2024-09-18T04:40:13.981 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027271040000000002, 2024-09-18T04:40:13.981 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:13.981 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:13.981 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:13.981 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.982 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:13.982 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.752479+0000", 2024-09-18T04:40:13.982 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:13.982 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.982 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.982 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:13.982 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.752479+0000", 2024-09-18T04:40:13.982 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:13.982 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.982 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.982 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:13.982 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.752478+0000", 2024-09-18T04:40:13.982 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:13.982 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.982 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.983 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:13.983 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.753784+0000", 2024-09-18T04:40:13.983 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0013058869999999999 2024-09-18T04:40:13.983 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.983 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.983 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:13.983 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.753821+0000", 2024-09-18T04:40:13.983 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7268999999999998e-05 2024-09-18T04:40:13.983 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.983 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.983 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:13.983 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.753873+0000", 2024-09-18T04:40:13.983 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1564000000000003e-05 2024-09-18T04:40:13.984 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.984 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.984 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:13.984 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.753963+0000", 2024-09-18T04:40:13.984 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0636000000000004e-05 2024-09-18T04:40:13.984 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.984 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.984 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:13.984 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.753972+0000", 2024-09-18T04:40:13.984 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0529999999999996e-06 2024-09-18T04:40:13.984 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.984 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.985 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:13.985 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.755206+0000", 2024-09-18T04:40:13.985 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001234113 2024-09-18T04:40:13.985 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:13.985 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:13.985 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:13.985 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.985 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.985 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:7, version: 79'64, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:7@79'64, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:40:13.985 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.756672+0000", 2024-09-18T04:40:13.985 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 27.199224407999999, 2024-09-18T04:40:13.985 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028688770000000001, 2024-09-18T04:40:13.985 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:13.985 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:13.986 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:13.986 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.986 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:13.986 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.756672+0000", 2024-09-18T04:40:13.986 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:13.986 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.986 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.986 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:13.986 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.756672+0000", 2024-09-18T04:40:13.986 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:13.986 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.986 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.986 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:13.986 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.756671+0000", 2024-09-18T04:40:13.986 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:13.987 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.987 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.987 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:13.987 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.757623+0000", 2024-09-18T04:40:13.987 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00095182199999999995 2024-09-18T04:40:13.987 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.987 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.987 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:13.987 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.757644+0000", 2024-09-18T04:40:13.987 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0996999999999999e-05 2024-09-18T04:40:13.987 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.987 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.987 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:13.987 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.757657+0000", 2024-09-18T04:40:13.988 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3264999999999999e-05 2024-09-18T04:40:13.988 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.988 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.988 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:13.988 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.757779+0000", 2024-09-18T04:40:13.988 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000121655 2024-09-18T04:40:13.988 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.988 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.988 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:13.988 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.757801+0000", 2024-09-18T04:40:13.988 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2340999999999998e-05 2024-09-18T04:40:13.988 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.988 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.988 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:13.989 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.759541+0000", 2024-09-18T04:40:13.989 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017398120000000001 2024-09-18T04:40:13.989 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:13.989 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:13.989 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:13.989 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.989 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.989 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.5 87/86 [PushOp(2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:7, version: 79'59, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:7@79'59, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:40:13.989 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.762725+0000", 2024-09-18T04:40:13.989 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 27.193171402000001, 2024-09-18T04:40:13.989 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028321890000000001, 2024-09-18T04:40:13.989 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:13.989 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:13.989 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:13.989 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.990 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:13.990 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.762725+0000", 2024-09-18T04:40:13.990 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:13.990 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.990 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.990 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:13.990 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.762725+0000", 2024-09-18T04:40:13.990 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:13.990 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.990 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.990 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:13.990 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.762724+0000", 2024-09-18T04:40:13.990 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:13.991 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.991 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.991 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:13.991 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.763395+0000", 2024-09-18T04:40:13.991 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00067064800000000003 2024-09-18T04:40:13.991 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.991 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.991 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:13.991 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.763416+0000", 2024-09-18T04:40:13.991 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0984000000000001e-05 2024-09-18T04:40:13.991 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.991 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.991 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:13.991 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.763442+0000", 2024-09-18T04:40:13.991 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6635000000000002e-05 2024-09-18T04:40:13.992 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.992 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.992 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:13.992 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.763584+0000", 2024-09-18T04:40:13.992 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000142037 2024-09-18T04:40:13.992 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.992 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.992 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:13.992 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.763610+0000", 2024-09-18T04:40:13.992 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6027999999999999e-05 2024-09-18T04:40:13.992 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.992 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.992 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:13.993 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.765557+0000", 2024-09-18T04:40:13.993 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0019470939999999999 2024-09-18T04:40:13.993 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:13.993 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:13.993 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:13.993 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.993 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.993 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:7, version: 79'64, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:7@79'64, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:40:13.993 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.763323+0000", 2024-09-18T04:40:13.993 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 27.192573217, 2024-09-18T04:40:13.993 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003042215, 2024-09-18T04:40:13.993 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:13.993 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:13.993 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:13.994 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.994 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:13.994 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.763323+0000", 2024-09-18T04:40:13.994 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:13.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.994 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.994 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:13.994 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.763323+0000", 2024-09-18T04:40:13.994 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:13.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.994 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.994 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:13.994 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.763322+0000", 2024-09-18T04:40:13.994 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:13.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.995 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.995 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:13.995 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.764567+0000", 2024-09-18T04:40:13.995 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001244312 2024-09-18T04:40:13.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.995 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.995 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:13.995 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.764600+0000", 2024-09-18T04:40:13.995 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2787000000000003e-05 2024-09-18T04:40:13.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.995 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.995 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:13.995 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.764640+0000", 2024-09-18T04:40:13.996 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0825000000000003e-05 2024-09-18T04:40:13.996 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.996 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.996 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:13.996 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.764789+0000", 2024-09-18T04:40:13.996 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000148598 2024-09-18T04:40:13.996 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.996 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.996 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:13.996 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.764812+0000", 2024-09-18T04:40:13.996 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2702e-05 2024-09-18T04:40:13.996 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.996 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.996 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:13.996 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.766366+0000", 2024-09-18T04:40:13.997 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0015540110000000001 2024-09-18T04:40:13.997 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:13.997 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:13.997 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:13.997 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.997 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.997 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.5 87/86 [PushOp(2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:7, version: 79'59, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:7@79'59, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:40:13.997 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.768078+0000", 2024-09-18T04:40:13.997 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 27.187818439000001, 2024-09-18T04:40:13.997 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028938509999999998, 2024-09-18T04:40:13.997 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:13.997 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:13.997 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:13.997 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.998 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:13.998 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.768078+0000", 2024-09-18T04:40:13.998 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:13.998 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.998 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.998 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:13.998 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.768078+0000", 2024-09-18T04:40:13.998 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:13.998 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.998 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.998 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:13.998 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.768076+0000", 2024-09-18T04:40:13.998 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:13.998 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.999 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.999 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:13.999 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.768951+0000", 2024-09-18T04:40:13.999 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000874455 2024-09-18T04:40:13.999 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.999 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.999 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:13.999 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.768972+0000", 2024-09-18T04:40:13.999 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.154e-05 2024-09-18T04:40:13.999 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:13.999 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:13.999 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:13.999 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.768992+0000", 2024-09-18T04:40:13.999 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9822000000000001e-05 2024-09-18T04:40:13.999 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.000 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.000 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:14.000 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.769110+0000", 2024-09-18T04:40:14.000 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000117762 2024-09-18T04:40:14.000 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.000 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.000 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:14.000 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.769123+0000", 2024-09-18T04:40:14.000 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2473999999999999e-05 2024-09-18T04:40:14.000 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.000 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.000 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:14.000 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.770972+0000", 2024-09-18T04:40:14.000 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001849613 2024-09-18T04:40:14.001 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.001 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:14.001 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.001 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.001 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.001 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:7, version: 79'64, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:7@79'64, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:40:14.001 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.769830+0000", 2024-09-18T04:40:14.001 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 27.186067175000002, 2024-09-18T04:40:14.001 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0040223419999999999, 2024-09-18T04:40:14.001 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:14.001 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:14.001 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:14.001 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.001 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:14.001 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.769830+0000", 2024-09-18T04:40:14.002 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.002 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.002 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:14.002 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.769830+0000", 2024-09-18T04:40:14.002 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.002 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.002 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:14.002 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.769829+0000", 2024-09-18T04:40:14.002 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:14.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.002 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.003 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:14.003 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.770812+0000", 2024-09-18T04:40:14.003 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00098347800000000009 2024-09-18T04:40:14.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.003 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.003 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:14.003 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.770834+0000", 2024-09-18T04:40:14.003 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1648999999999999e-05 2024-09-18T04:40:14.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.003 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.003 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:14.003 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.770847+0000", 2024-09-18T04:40:14.003 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3169999999999999e-05 2024-09-18T04:40:14.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.004 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.004 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:14.004 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.770969+0000", 2024-09-18T04:40:14.004 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000121721 2024-09-18T04:40:14.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.004 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.004 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:14.004 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.770989+0000", 2024-09-18T04:40:14.004 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9944000000000001e-05 2024-09-18T04:40:14.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.004 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.004 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:14.004 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.773852+0000", 2024-09-18T04:40:14.004 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028632319999999998 2024-09-18T04:40:14.004 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.005 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:14.005 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.005 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.005 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.005 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.5 87/86 [PushOp(2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head, version: 79'62, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head@79'62, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-18T04:40:14.005 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.772378+0000", 2024-09-18T04:40:14.005 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 27.183519052000001, 2024-09-18T04:40:14.005 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002661535, 2024-09-18T04:40:14.005 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:14.005 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:14.005 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:14.005 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.005 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:14.005 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.772378+0000", 2024-09-18T04:40:14.006 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.006 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.006 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.006 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:14.006 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.772378+0000", 2024-09-18T04:40:14.006 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.006 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.006 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.006 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:14.006 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.772377+0000", 2024-09-18T04:40:14.006 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:14.006 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.006 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.006 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:14.006 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.772393+0000", 2024-09-18T04:40:14.007 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6495e-05 2024-09-18T04:40:14.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.007 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.007 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:14.007 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.772417+0000", 2024-09-18T04:40:14.007 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4304e-05 2024-09-18T04:40:14.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.007 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.007 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:14.007 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.772431+0000", 2024-09-18T04:40:14.007 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3165e-05 2024-09-18T04:40:14.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.007 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.007 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:14.008 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.772499+0000", 2024-09-18T04:40:14.008 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8627000000000005e-05 2024-09-18T04:40:14.008 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.008 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.008 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:14.008 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.772508+0000", 2024-09-18T04:40:14.008 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5609999999999999e-06 2024-09-18T04:40:14.008 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.008 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.008 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:14.008 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.775039+0000", 2024-09-18T04:40:14.008 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002531387 2024-09-18T04:40:14.008 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.008 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:14.009 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.009 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.009 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.009 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.5 87/86 [PushOp(2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:7, version: 79'61, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:7@79'61, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:40:14.009 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.781808+0000", 2024-09-18T04:40:14.009 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 27.174089054, 2024-09-18T04:40:14.009 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027222169999999999, 2024-09-18T04:40:14.009 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:14.009 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:14.009 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:14.009 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.009 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:14.009 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.781808+0000", 2024-09-18T04:40:14.009 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.010 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.010 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:14.010 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.781808+0000", 2024-09-18T04:40:14.010 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.010 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.010 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:14.010 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.781807+0000", 2024-09-18T04:40:14.010 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:14.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.010 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.010 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:14.010 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.782242+0000", 2024-09-18T04:40:14.010 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00043528300000000001 2024-09-18T04:40:14.011 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.011 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.011 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:14.011 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.782265+0000", 2024-09-18T04:40:14.011 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3227e-05 2024-09-18T04:40:14.011 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.011 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.011 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:14.011 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.782283+0000", 2024-09-18T04:40:14.011 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7535999999999999e-05 2024-09-18T04:40:14.011 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.011 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.011 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:14.011 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.782418+0000", 2024-09-18T04:40:14.011 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000135298 2024-09-18T04:40:14.012 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.012 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.012 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:14.012 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.782444+0000", 2024-09-18T04:40:14.012 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6020000000000002e-05 2024-09-18T04:40:14.012 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.012 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.012 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:14.012 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.784530+0000", 2024-09-18T04:40:14.012 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020858890000000001 2024-09-18T04:40:14.012 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.012 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:14.012 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.013 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.013 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.013 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head, version: 81'75, data_included: [0~49152,1097728~999424], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head@81'75, size: 2146304, copy_subset: [0~2146304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-18T04:40:14.013 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.782052+0000", 2024-09-18T04:40:14.013 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 27.173844689999999, 2024-09-18T04:40:14.013 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0037389630000000001, 2024-09-18T04:40:14.013 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:14.013 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:14.013 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:14.013 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.013 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:14.013 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.782052+0000", 2024-09-18T04:40:14.013 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.013 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.014 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.014 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:14.014 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.782052+0000", 2024-09-18T04:40:14.014 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.014 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.014 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.014 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:14.014 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.782051+0000", 2024-09-18T04:40:14.014 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:14.014 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.014 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.014 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:14.014 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.783658+0000", 2024-09-18T04:40:14.014 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0016072149999999999 2024-09-18T04:40:14.015 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.015 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.015 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:14.015 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.783698+0000", 2024-09-18T04:40:14.015 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9394000000000003e-05 2024-09-18T04:40:14.015 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.015 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.015 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:14.015 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.783733+0000", 2024-09-18T04:40:14.015 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5428e-05 2024-09-18T04:40:14.015 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.015 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.015 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:14.015 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.783854+0000", 2024-09-18T04:40:14.016 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012067599999999999 2024-09-18T04:40:14.016 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.016 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.016 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:14.016 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.783869+0000", 2024-09-18T04:40:14.016 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5554999999999999e-05 2024-09-18T04:40:14.016 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.016 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.016 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:14.016 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.785791+0000", 2024-09-18T04:40:14.016 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001921629 2024-09-18T04:40:14.016 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.017 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:14.017 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.017 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.017 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.017 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head, version: 81'75, data_included: [2097152~49152], data_size: 49152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head@81'75, size: 2146304, copy_subset: [0~2146304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2146304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:40:14.017 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.787740+0000", 2024-09-18T04:40:14.017 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 27.168157176000001, 2024-09-18T04:40:14.017 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030127370000000001, 2024-09-18T04:40:14.017 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:14.017 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:14.017 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:14.017 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.017 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:14.017 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.787740+0000", 2024-09-18T04:40:14.018 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.018 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.018 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.018 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:14.018 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.787740+0000", 2024-09-18T04:40:14.018 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.018 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.018 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.018 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:14.018 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.787738+0000", 2024-09-18T04:40:14.018 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:14.018 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.018 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.018 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:14.018 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.787795+0000", 2024-09-18T04:40:14.019 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6753999999999999e-05 2024-09-18T04:40:14.019 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.019 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.019 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:14.019 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.787826+0000", 2024-09-18T04:40:14.019 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0617000000000002e-05 2024-09-18T04:40:14.019 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.019 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.019 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:14.019 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.787853+0000", 2024-09-18T04:40:14.019 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7538999999999998e-05 2024-09-18T04:40:14.019 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.019 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.019 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:14.020 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.787963+0000", 2024-09-18T04:40:14.020 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000109387 2024-09-18T04:40:14.020 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.020 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.020 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:14.020 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.787973+0000", 2024-09-18T04:40:14.020 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0781999999999999e-05 2024-09-18T04:40:14.020 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.020 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.020 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:14.020 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.790752+0000", 2024-09-18T04:40:14.020 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027788520000000001 2024-09-18T04:40:14.020 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.020 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:14.021 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.021 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.021 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.021 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.5 87/86 [PushOp(2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:7, version: 79'61, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:7@79'61, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:40:14.021 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.791668+0000", 2024-09-18T04:40:14.021 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 27.164228482999999, 2024-09-18T04:40:14.021 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025634299999999998, 2024-09-18T04:40:14.021 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:14.021 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:14.021 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:14.021 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.021 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:14.021 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.791668+0000", 2024-09-18T04:40:14.021 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.022 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.022 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.022 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:14.022 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.791668+0000", 2024-09-18T04:40:14.022 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.022 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.022 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.022 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:14.022 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.791667+0000", 2024-09-18T04:40:14.022 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:14.022 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.022 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.022 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:14.022 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.792071+0000", 2024-09-18T04:40:14.022 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00040348599999999999 2024-09-18T04:40:14.023 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.023 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.023 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:14.023 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.792094+0000", 2024-09-18T04:40:14.023 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3105e-05 2024-09-18T04:40:14.023 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.023 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.023 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:14.023 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.792112+0000", 2024-09-18T04:40:14.023 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8502999999999999e-05 2024-09-18T04:40:14.023 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.023 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.023 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:14.023 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.792187+0000", 2024-09-18T04:40:14.023 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5074000000000007e-05 2024-09-18T04:40:14.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.024 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.024 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:14.024 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.792197+0000", 2024-09-18T04:40:14.024 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.2979999999999997e-06 2024-09-18T04:40:14.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.024 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.024 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:14.024 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.794232+0000", 2024-09-18T04:40:14.024 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002035149 2024-09-18T04:40:14.024 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.024 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:14.024 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.025 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.025 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.5732.0:3 2.1 2:9370a598:::rbd_header.125fc09df6d1:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e102)", 2024-09-18T04:40:14.025 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:40:13.100500+0000", 2024-09-18T04:40:14.025 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.85539669500000004, 2024-09-18T04:40:14.025 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0035679359999999999, 2024-09-18T04:40:14.025 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:14.025 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:14.025 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:14.025 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.5732", 2024-09-18T04:40:14.025 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/1418224569", 2024-09-18T04:40:14.025 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 3 2024-09-18T04:40:14.025 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.025 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:14.025 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.026 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:14.026 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.100500+0000", 2024-09-18T04:40:14.026 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.026 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.026 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.026 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:14.026 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.100500+0000", 2024-09-18T04:40:14.026 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.026 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.026 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.026 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:14.026 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.100497+0000", 2024-09-18T04:40:14.026 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999971 2024-09-18T04:40:14.026 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.026 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.027 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:14.027 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.100511+0000", 2024-09-18T04:40:14.027 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3753e-05 2024-09-18T04:40:14.027 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.027 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.027 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:14.027 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.100534+0000", 2024-09-18T04:40:14.027 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3108999999999998e-05 2024-09-18T04:40:14.027 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.027 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.027 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:14.027 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.100555+0000", 2024-09-18T04:40:14.027 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.084e-05 2024-09-18T04:40:14.027 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.027 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.028 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:14.028 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.100645+0000", 2024-09-18T04:40:14.028 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9859999999999997e-05 2024-09-18T04:40:14.028 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.028 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.028 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-09-18T04:40:14.028 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.100652+0000", 2024-09-18T04:40:14.028 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4109999999999999e-06 2024-09-18T04:40:14.028 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.028 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.028 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:14.028 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.103598+0000", 2024-09-18T04:40:14.028 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0029458940000000001 2024-09-18T04:40:14.028 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.029 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.029 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:14.029 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.103731+0000", 2024-09-18T04:40:14.029 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013292999999999999 2024-09-18T04:40:14.029 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.029 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.029 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:14.029 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.104068+0000", 2024-09-18T04:40:14.029 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00033679199999999998 2024-09-18T04:40:14.029 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.029 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:14.029 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.029 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.029 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.030 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.5689.0:329 2.1 2:9370a598:::rbd_header.125fc09df6d1:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-18T04:40:14.030 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:40:13.281353+0000", 2024-09-18T04:40:14.030 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.67454374699999997, 2024-09-18T04:40:14.030 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0033936449999999998, 2024-09-18T04:40:14.030 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:14.030 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:14.030 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:14.030 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.5689", 2024-09-18T04:40:14.030 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/1538094101", 2024-09-18T04:40:14.030 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 329 2024-09-18T04:40:14.030 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.030 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:14.030 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.030 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:14.030 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.281353+0000", 2024-09-18T04:40:14.031 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.031 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.031 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.031 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:14.031 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.281353+0000", 2024-09-18T04:40:14.031 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.031 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.031 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.031 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:14.031 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.281350+0000", 2024-09-18T04:40:14.031 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:40:14.031 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.031 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.031 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:14.031 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.281364+0000", 2024-09-18T04:40:14.032 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3936e-05 2024-09-18T04:40:14.032 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.032 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.032 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:14.032 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.281399+0000", 2024-09-18T04:40:14.032 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4282000000000001e-05 2024-09-18T04:40:14.032 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.032 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.032 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:14.032 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.281416+0000", 2024-09-18T04:40:14.032 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7042999999999999e-05 2024-09-18T04:40:14.032 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.032 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.032 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:14.033 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.281571+0000", 2024-09-18T04:40:14.033 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015490999999999999 2024-09-18T04:40:14.033 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.033 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.033 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:14.033 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.281833+0000", 2024-09-18T04:40:14.033 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00026252399999999998 2024-09-18T04:40:14.033 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.033 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.033 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:14.033 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.282340+0000", 2024-09-18T04:40:14.033 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00050714200000000003 2024-09-18T04:40:14.033 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.033 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.034 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:14.034 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.283675+0000", 2024-09-18T04:40:14.034 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001334949 2024-09-18T04:40:14.034 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.034 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.034 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:14.034 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.284605+0000", 2024-09-18T04:40:14.034 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00093018099999999998 2024-09-18T04:40:14.034 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.034 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.034 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:14.034 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.284687+0000", 2024-09-18T04:40:14.034 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1417999999999999e-05 2024-09-18T04:40:14.034 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.034 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.035 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:14.035 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.284747+0000", 2024-09-18T04:40:14.035 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9865999999999999e-05 2024-09-18T04:40:14.035 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.035 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:14.035 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.035 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.035 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.035 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.5689.0:331 2.1 2:9370a598:::rbd_header.125fc09df6d1:head [watch unwatch cookie 140417983158384] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-18T04:40:14.035 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:40:13.285091+0000", 2024-09-18T04:40:14.035 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.67080536300000004, 2024-09-18T04:40:14.035 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0032654429999999998, 2024-09-18T04:40:14.035 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:14.035 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:14.036 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:14.036 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.5689", 2024-09-18T04:40:14.036 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/1538094101", 2024-09-18T04:40:14.036 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 331 2024-09-18T04:40:14.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.036 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:14.036 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.036 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:14.036 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.285091+0000", 2024-09-18T04:40:14.036 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.036 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.036 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:14.036 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.285091+0000", 2024-09-18T04:40:14.037 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.037 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.037 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.037 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:14.037 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.285090+0000", 2024-09-18T04:40:14.037 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:14.037 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.037 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.037 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:14.037 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.285100+0000", 2024-09-18T04:40:14.037 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0477e-05 2024-09-18T04:40:14.037 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.037 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.037 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:14.037 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.285144+0000", 2024-09-18T04:40:14.038 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4227999999999997e-05 2024-09-18T04:40:14.038 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.038 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.038 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:14.038 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.285160+0000", 2024-09-18T04:40:14.038 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6042999999999999e-05 2024-09-18T04:40:14.038 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.038 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.038 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:14.038 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.285639+0000", 2024-09-18T04:40:14.038 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00047824299999999999 2024-09-18T04:40:14.038 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.038 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.038 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:14.039 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.285797+0000", 2024-09-18T04:40:14.039 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015854899999999999 2024-09-18T04:40:14.039 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.039 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.039 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:14.039 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.286015+0000", 2024-09-18T04:40:14.039 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00021824799999999999 2024-09-18T04:40:14.039 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.039 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.039 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:14.039 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.287132+0000", 2024-09-18T04:40:14.039 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0011161980000000001 2024-09-18T04:40:14.039 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.040 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.040 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:14.040 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.288042+0000", 2024-09-18T04:40:14.040 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00091054500000000002 2024-09-18T04:40:14.040 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.040 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.040 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:14.040 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.288119+0000", 2024-09-18T04:40:14.040 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7294999999999998e-05 2024-09-18T04:40:14.040 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.040 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.040 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:14.040 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.288357+0000", 2024-09-18T04:40:14.040 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00023735100000000001 2024-09-18T04:40:14.041 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.041 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:14.041 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.041 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.041 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.041 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.5689.0:332 2.1 2:9370a598:::rbd_header.125fc09df6d1:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e102)", 2024-09-18T04:40:14.041 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:40:13.285656+0000", 2024-09-18T04:40:14.041 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.67024064999999999, 2024-09-18T04:40:14.041 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003014409, 2024-09-18T04:40:14.041 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:14.041 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:14.041 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:14.041 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.5689", 2024-09-18T04:40:14.041 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/1538094101", 2024-09-18T04:40:14.042 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 332 2024-09-18T04:40:14.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.042 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:14.042 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.042 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:14.042 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.285656+0000", 2024-09-18T04:40:14.042 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.042 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.042 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:14.042 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.285656+0000", 2024-09-18T04:40:14.042 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.042 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.043 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:14.043 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.285654+0000", 2024-09-18T04:40:14.043 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:14.043 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.043 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.043 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:14.043 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.285665+0000", 2024-09-18T04:40:14.043 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0823e-05 2024-09-18T04:40:14.043 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.043 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.043 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:14.043 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.285694+0000", 2024-09-18T04:40:14.043 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9128999999999998e-05 2024-09-18T04:40:14.043 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.043 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.044 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:14.044 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.285718+0000", 2024-09-18T04:40:14.044 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4216e-05 2024-09-18T04:40:14.044 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.044 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.044 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:14.044 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.286667+0000", 2024-09-18T04:40:14.044 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00094823499999999996 2024-09-18T04:40:14.044 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.044 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.044 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for rw locks", 2024-09-18T04:40:14.044 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.286812+0000", 2024-09-18T04:40:14.044 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014571300000000001 2024-09-18T04:40:14.044 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.044 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.045 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:14.045 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.288427+0000", 2024-09-18T04:40:14.045 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0016148759999999999 2024-09-18T04:40:14.045 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.045 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.045 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:14.045 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.288547+0000", 2024-09-18T04:40:14.045 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000119387 2024-09-18T04:40:14.045 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.045 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.045 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:14.045 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.288670+0000", 2024-09-18T04:40:14.045 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012381 2024-09-18T04:40:14.046 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.046 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:14.046 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.046 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.046 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:14.046 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:40:14.047 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd config image rm rbd/testrbdnbd17342 rbd_discard_granularity_bytes 2024-09-18T04:40:14.047 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:40:14.043+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:40:14.048 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:40:14.109 INFO:tasks.workunit.client.0.smithi115.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd17342 2024-09-18T04:40:14.109 INFO:tasks.workunit.client.0.smithi115.stderr:++ local cmd 2024-09-18T04:40:14.110 INFO:tasks.workunit.client.0.smithi115.stderr:+++ id -u 2024-09-18T04:40:14.111 INFO:tasks.workunit.client.0.smithi115.stderr:++ '[' 1000 -eq 0 ']' 2024-09-18T04:40:14.112 INFO:tasks.workunit.client.0.smithi115.stderr:+++ which rbd 2024-09-18T04:40:14.114 INFO:tasks.workunit.client.0.smithi115.stderr:++ cmd=/usr/bin/rbd 2024-09-18T04:40:14.114 INFO:tasks.workunit.client.0.smithi115.stderr:++ shift 2024-09-18T04:40:14.115 INFO:tasks.workunit.client.0.smithi115.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd17342 2024-09-18T04:40:14.148 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:40:14.143+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:40:14.154 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:40:14.154 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:40:14.155 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:40:14.155 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:40:14.164 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:40:14.249 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:40:14.244+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:40:14.315 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:40:14.315 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:40:14.316 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:40:14.316 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:40:14.316 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:40:14.325 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:40:14.350 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:40:14.348+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:40:14.436 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:40:14.436 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:40:14.436 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:40:14.436 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:40:14.436 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.436 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:37 2.5 e51/28 2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:head v 51'1, mlcod=0'0)", 2024-09-18T04:40:14.436 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:40:14.436 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 92.138864601999998, 2024-09-18T04:40:14.436 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012279973, 2024-09-18T04:40:14.436 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:14.436 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:14.437 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:14.437 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.437 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:14.437 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:40:14.437 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.437 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.437 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.437 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:14.437 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:40:14.437 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.437 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.437 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.437 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:14.438 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292468+0000", 2024-09-18T04:40:14.438 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:40:14.438 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.438 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.438 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:14.438 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295837+0000", 2024-09-18T04:40:14.438 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0033691699999999999 2024-09-18T04:40:14.438 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.438 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.438 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:14.438 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295860+0000", 2024-09-18T04:40:14.438 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2756999999999999e-05 2024-09-18T04:40:14.438 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.439 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.439 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:14.439 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295872+0000", 2024-09-18T04:40:14.439 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2442e-05 2024-09-18T04:40:14.439 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.439 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.439 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:14.439 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295934+0000", 2024-09-18T04:40:14.439 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1187999999999998e-05 2024-09-18T04:40:14.439 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.439 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.439 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:14.439 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295967+0000", 2024-09-18T04:40:14.439 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3185999999999997e-05 2024-09-18T04:40:14.440 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.440 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.440 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:14.440 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304697+0000", 2024-09-18T04:40:14.440 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0087298059999999997 2024-09-18T04:40:14.440 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.440 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.440 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:14.440 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304749+0000", 2024-09-18T04:40:14.440 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1922000000000002e-05 2024-09-18T04:40:14.440 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.440 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:14.440 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.440 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.441 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.441 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:49 2.5 e51/28 2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head v 51'2, mlcod=0'0)", 2024-09-18T04:40:14.441 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:40:14.441 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 92.105537764000005, 2024-09-18T04:40:14.441 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018033041999999999, 2024-09-18T04:40:14.441 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:14.441 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:14.441 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:14.441 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.441 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:14.441 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:40:14.441 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.441 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.442 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.442 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:14.442 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:40:14.442 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.442 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.442 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.442 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:14.442 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325793+0000", 2024-09-18T04:40:14.442 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:40:14.442 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.442 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.442 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:14.442 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329038+0000", 2024-09-18T04:40:14.442 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0032444010000000001 2024-09-18T04:40:14.443 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.443 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.443 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:14.443 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329065+0000", 2024-09-18T04:40:14.443 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7500000000000001e-05 2024-09-18T04:40:14.443 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.443 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.443 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:14.443 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329077+0000", 2024-09-18T04:40:14.443 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1919999999999999e-05 2024-09-18T04:40:14.443 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.443 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.443 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:14.443 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329149+0000", 2024-09-18T04:40:14.444 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2349999999999997e-05 2024-09-18T04:40:14.444 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.444 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.444 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:14.444 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329192+0000", 2024-09-18T04:40:14.444 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2744000000000003e-05 2024-09-18T04:40:14.444 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.444 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.444 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:14.444 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343773+0000", 2024-09-18T04:40:14.444 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014581246000000001 2024-09-18T04:40:14.444 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.444 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.445 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:14.445 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343829+0000", 2024-09-18T04:40:14.445 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5143999999999999e-05 2024-09-18T04:40:14.445 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.445 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:14.445 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.445 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.445 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.445 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:47 2.4 e51/17 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 51'3, mlcod=51'1)", 2024-09-18T04:40:14.445 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:40:14.445 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 92.104957498999994, 2024-09-18T04:40:14.445 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018960186, 2024-09-18T04:40:14.445 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:14.445 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:14.445 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:14.446 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.446 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:14.446 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:40:14.446 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.446 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.446 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.446 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:14.446 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:40:14.446 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.446 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.446 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.446 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:14.446 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326374+0000", 2024-09-18T04:40:14.446 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:14.447 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.447 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.447 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:14.447 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330883+0000", 2024-09-18T04:40:14.447 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0045084740000000002 2024-09-18T04:40:14.447 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.447 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.447 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:14.447 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330901+0000", 2024-09-18T04:40:14.447 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7890999999999999e-05 2024-09-18T04:40:14.447 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.447 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.447 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:14.447 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330912+0000", 2024-09-18T04:40:14.447 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1041e-05 2024-09-18T04:40:14.448 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.448 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.448 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:14.448 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.331003+0000", 2024-09-18T04:40:14.448 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1323999999999995e-05 2024-09-18T04:40:14.448 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.448 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.448 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:14.448 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.331030+0000", 2024-09-18T04:40:14.448 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7351999999999998e-05 2024-09-18T04:40:14.448 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.448 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.448 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:14.449 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345287+0000", 2024-09-18T04:40:14.449 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014256332 2024-09-18T04:40:14.449 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.449 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.449 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:14.449 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345336+0000", 2024-09-18T04:40:14.449 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9356e-05 2024-09-18T04:40:14.449 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.449 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:14.449 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.449 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.449 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.449 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:58 2.5 e51/28 2:b227e6b1:::rbd_data.125fc09df6d1.000000000000000e:head v 51'3, mlcod=51'1)", 2024-09-18T04:40:14.449 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:40:14.449 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 92.064530872000006, 2024-09-18T04:40:14.450 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016513678, 2024-09-18T04:40:14.450 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:14.450 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:14.450 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:14.450 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.450 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:14.450 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:40:14.450 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.450 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.450 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.450 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:14.450 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:40:14.450 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.450 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.451 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.451 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:14.451 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366800+0000", 2024-09-18T04:40:14.451 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:40:14.451 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.451 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.451 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:14.451 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368644+0000", 2024-09-18T04:40:14.451 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001844399 2024-09-18T04:40:14.451 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.451 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.451 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:14.451 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368681+0000", 2024-09-18T04:40:14.451 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6526999999999998e-05 2024-09-18T04:40:14.452 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.452 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.452 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:14.452 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368698+0000", 2024-09-18T04:40:14.452 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7272999999999999e-05 2024-09-18T04:40:14.452 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.452 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.452 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:14.452 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368773+0000", 2024-09-18T04:40:14.452 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4862000000000003e-05 2024-09-18T04:40:14.452 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.452 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.452 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:14.452 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368801+0000", 2024-09-18T04:40:14.453 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7925000000000001e-05 2024-09-18T04:40:14.453 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.453 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.453 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:14.453 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383261+0000", 2024-09-18T04:40:14.453 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014460377999999999 2024-09-18T04:40:14.453 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.453 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.453 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:14.453 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383316+0000", 2024-09-18T04:40:14.453 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4623999999999997e-05 2024-09-18T04:40:14.453 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.453 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:14.453 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.454 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.454 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.454 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:56 2.4 e51/17 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 51'4, mlcod=51'2)", 2024-09-18T04:40:14.454 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:40:14.454 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 92.062526845999997, 2024-09-18T04:40:14.454 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015105221, 2024-09-18T04:40:14.454 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:14.454 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:14.454 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:14.454 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.454 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:14.454 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:40:14.454 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.455 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.455 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.455 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:14.455 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:40:14.455 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.455 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.455 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.455 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:14.455 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368805+0000", 2024-09-18T04:40:14.455 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:14.455 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.455 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.455 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:14.455 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371603+0000", 2024-09-18T04:40:14.455 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027982660000000002 2024-09-18T04:40:14.456 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.456 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.456 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:14.456 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371633+0000", 2024-09-18T04:40:14.456 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0073000000000002e-05 2024-09-18T04:40:14.456 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.456 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.456 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:14.456 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371652+0000", 2024-09-18T04:40:14.456 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8597000000000002e-05 2024-09-18T04:40:14.456 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.456 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.456 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:14.456 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371752+0000", 2024-09-18T04:40:14.457 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9772999999999995e-05 2024-09-18T04:40:14.457 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.457 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.457 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:14.457 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371793+0000", 2024-09-18T04:40:14.457 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1173e-05 2024-09-18T04:40:14.457 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.457 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.457 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:14.457 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383875+0000", 2024-09-18T04:40:14.457 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012081852000000001 2024-09-18T04:40:14.457 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.457 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.457 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:14.457 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383912+0000", 2024-09-18T04:40:14.458 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7045000000000002e-05 2024-09-18T04:40:14.458 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.458 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:14.458 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.458 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.458 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.458 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:36 2.5 e51/28 2:bd5b8dd8:ns::rbd_data.133e813ab5bc.0000000000000002:head v 51'4, mlcod=51'2)", 2024-09-18T04:40:14.458 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:40:14.458 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 91.893807855000006, 2024-09-18T04:40:14.458 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020758161000000001, 2024-09-18T04:40:14.458 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:14.458 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:14.458 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:14.458 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.459 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:14.459 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:40:14.459 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.459 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.459 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.459 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:14.459 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:40:14.459 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.459 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.459 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.459 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:14.459 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537523+0000", 2024-09-18T04:40:14.459 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:14.459 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.460 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.460 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:14.460 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541578+0000", 2024-09-18T04:40:14.460 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0040549710000000001 2024-09-18T04:40:14.460 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.460 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.460 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:14.460 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541608+0000", 2024-09-18T04:40:14.460 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9349e-05 2024-09-18T04:40:14.460 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.460 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.460 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:14.460 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541621+0000", 2024-09-18T04:40:14.460 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2736e-05 2024-09-18T04:40:14.461 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.461 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.461 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:14.461 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541715+0000", 2024-09-18T04:40:14.461 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4197999999999995e-05 2024-09-18T04:40:14.461 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.461 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.461 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:14.461 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541755+0000", 2024-09-18T04:40:14.461 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9818999999999999e-05 2024-09-18T04:40:14.461 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.461 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.461 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:14.461 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558217+0000", 2024-09-18T04:40:14.462 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016462234999999999 2024-09-18T04:40:14.462 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.462 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.462 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:14.462 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558284+0000", 2024-09-18T04:40:14.462 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6774999999999997e-05 2024-09-18T04:40:14.462 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.462 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:14.462 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.462 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.462 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.462 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:39 2.4 e51/17 2:304364d1:ns::rbd_data.133e813ab5bc.0000000000000004:head v 51'5, mlcod=51'3)", 2024-09-18T04:40:14.462 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:40:14.462 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 91.885668727999999, 2024-09-18T04:40:14.462 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016777558000000001, 2024-09-18T04:40:14.463 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:14.463 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:14.463 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:14.463 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.463 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:14.463 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:40:14.463 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.463 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.463 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.463 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:14.463 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:40:14.463 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.463 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.463 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.463 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:14.464 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545663+0000", 2024-09-18T04:40:14.464 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:14.464 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.464 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.464 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:14.464 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548286+0000", 2024-09-18T04:40:14.464 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026233390000000001 2024-09-18T04:40:14.464 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.464 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.464 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:14.464 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548305+0000", 2024-09-18T04:40:14.464 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8425000000000001e-05 2024-09-18T04:40:14.464 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.465 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.465 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:14.465 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548316+0000", 2024-09-18T04:40:14.465 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1163e-05 2024-09-18T04:40:14.465 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.465 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.465 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:14.465 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548379+0000", 2024-09-18T04:40:14.465 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2780000000000005e-05 2024-09-18T04:40:14.465 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.465 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.465 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:14.465 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548399+0000", 2024-09-18T04:40:14.465 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0080000000000001e-05 2024-09-18T04:40:14.466 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.466 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.466 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:14.466 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562383+0000", 2024-09-18T04:40:14.466 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013984671000000001 2024-09-18T04:40:14.466 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.466 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.466 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:14.466 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562442+0000", 2024-09-18T04:40:14.466 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8700999999999999e-05 2024-09-18T04:40:14.466 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.466 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:14.466 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.466 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.467 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.467 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:37 2.5 e51/28 2:bf39b8e4:ns::rbd_data.133e813ab5bc.0000000000000003:head v 51'5, mlcod=51'2)", 2024-09-18T04:40:14.467 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:40:14.467 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 91.882800458999995, 2024-09-18T04:40:14.467 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014264492, 2024-09-18T04:40:14.467 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:14.467 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:14.467 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:14.467 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.467 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:14.467 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:40:14.467 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.467 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.468 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.468 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:14.468 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:40:14.468 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.468 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.468 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.468 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:14.468 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548531+0000", 2024-09-18T04:40:14.468 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:14.468 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.468 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.468 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:14.468 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551500+0000", 2024-09-18T04:40:14.468 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0029688129999999998 2024-09-18T04:40:14.469 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.469 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.469 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:14.469 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551516+0000", 2024-09-18T04:40:14.469 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5994e-05 2024-09-18T04:40:14.469 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.469 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.469 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:14.469 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551526+0000", 2024-09-18T04:40:14.469 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9299999999999998e-06 2024-09-18T04:40:14.469 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.469 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.469 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:14.469 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551577+0000", 2024-09-18T04:40:14.470 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0871000000000001e-05 2024-09-18T04:40:14.470 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.470 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.470 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:14.470 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551602+0000", 2024-09-18T04:40:14.470 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4813000000000001e-05 2024-09-18T04:40:14.470 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.470 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.470 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:14.470 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562741+0000", 2024-09-18T04:40:14.470 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011139428999999999 2024-09-18T04:40:14.470 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.470 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.470 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:14.470 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562797+0000", 2024-09-18T04:40:14.471 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6329000000000002e-05 2024-09-18T04:40:14.471 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.471 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:14.471 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.471 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.471 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.471 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:49 2.4 e51/17 2:20f0b0c9:ns::rbd_data.133e813ab5bc.0000000000000009:head v 51'6, mlcod=51'4)", 2024-09-18T04:40:14.471 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:40:14.471 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 91.851365900999994, 2024-09-18T04:40:14.471 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013849046, 2024-09-18T04:40:14.471 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:14.471 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:14.471 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:14.471 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.472 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:14.472 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:40:14.472 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.472 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.472 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.472 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:14.472 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:40:14.472 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.472 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.472 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.472 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:14.472 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579966+0000", 2024-09-18T04:40:14.472 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:14.473 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.473 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.473 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:14.473 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582324+0000", 2024-09-18T04:40:14.473 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023578890000000002 2024-09-18T04:40:14.473 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.473 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.473 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:14.473 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582355+0000", 2024-09-18T04:40:14.473 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1485000000000001e-05 2024-09-18T04:40:14.473 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.473 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.473 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:14.473 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582371+0000", 2024-09-18T04:40:14.474 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6147999999999999e-05 2024-09-18T04:40:14.474 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.474 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.474 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:14.474 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582467+0000", 2024-09-18T04:40:14.474 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6059999999999998e-05 2024-09-18T04:40:14.474 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.474 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.474 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:14.474 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582505+0000", 2024-09-18T04:40:14.474 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7679000000000003e-05 2024-09-18T04:40:14.474 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.474 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.474 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:14.474 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593743+0000", 2024-09-18T04:40:14.475 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011238356 2024-09-18T04:40:14.475 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.475 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.475 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:14.475 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593816+0000", 2024-09-18T04:40:14.475 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2935000000000006e-05 2024-09-18T04:40:14.475 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.475 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:14.475 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.475 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.475 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.475 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:53 2.4 e51/17 2:39f7f174:ns::rbd_data.133e813ab5bc.000000000000000b:head v 51'7, mlcod=51'4)", 2024-09-18T04:40:14.475 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:40:14.475 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 91.836664745999997, 2024-09-18T04:40:14.476 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0079914159999999995, 2024-09-18T04:40:14.476 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:14.476 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:14.476 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:14.476 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.476 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:14.476 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:40:14.476 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.476 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.476 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.476 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:14.476 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:40:14.476 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.476 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.476 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.477 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:14.477 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594667+0000", 2024-09-18T04:40:14.477 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:14.477 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.477 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.477 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:14.477 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596446+0000", 2024-09-18T04:40:14.477 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001779033 2024-09-18T04:40:14.477 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.477 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.477 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:14.477 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596472+0000", 2024-09-18T04:40:14.477 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6120000000000001e-05 2024-09-18T04:40:14.477 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.478 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.478 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:14.478 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596500+0000", 2024-09-18T04:40:14.478 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7518e-05 2024-09-18T04:40:14.478 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.478 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.478 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:14.478 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596564+0000", 2024-09-18T04:40:14.478 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3880999999999994e-05 2024-09-18T04:40:14.478 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.478 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.478 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:14.478 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596596+0000", 2024-09-18T04:40:14.478 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2638999999999997e-05 2024-09-18T04:40:14.479 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.479 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.479 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:14.479 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602605+0000", 2024-09-18T04:40:14.479 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0060080699999999999 2024-09-18T04:40:14.479 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.479 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.479 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:14.479 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602660+0000", 2024-09-18T04:40:14.479 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5396000000000003e-05 2024-09-18T04:40:14.479 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.479 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:14.479 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.479 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.479 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.480 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:60 2.5 e51/28 2:a06f51e4:ns::rbd_data.133e813ab5bc.000000000000000f:head v 51'6, mlcod=51'4)", 2024-09-18T04:40:14.480 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:40:14.480 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 91.821847761000001, 2024-09-18T04:40:14.480 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0093954989999999999, 2024-09-18T04:40:14.480 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:14.480 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:14.480 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:14.480 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.480 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:14.480 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:40:14.480 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.480 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.480 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.481 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:14.481 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:40:14.481 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.481 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.481 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.481 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:14.481 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609483+0000", 2024-09-18T04:40:14.481 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:14.481 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.481 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.481 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:14.481 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611767+0000", 2024-09-18T04:40:14.481 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002283724 2024-09-18T04:40:14.482 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.482 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.482 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:14.482 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611791+0000", 2024-09-18T04:40:14.482 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3547e-05 2024-09-18T04:40:14.482 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.482 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.482 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:14.482 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611800+0000", 2024-09-18T04:40:14.482 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4979999999999995e-06 2024-09-18T04:40:14.482 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.482 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.482 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:14.482 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611864+0000", 2024-09-18T04:40:14.482 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3974999999999996e-05 2024-09-18T04:40:14.483 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.483 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.483 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:14.483 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611887+0000", 2024-09-18T04:40:14.483 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2633000000000002e-05 2024-09-18T04:40:14.483 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.483 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.483 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:14.483 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.618777+0000", 2024-09-18T04:40:14.483 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0068897979999999999 2024-09-18T04:40:14.483 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.483 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.483 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:14.483 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.618881+0000", 2024-09-18T04:40:14.484 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010437699999999999 2024-09-18T04:40:14.484 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.484 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:14.484 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.484 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.484 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.484 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:663 2.4 e57/17 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 57'13, mlcod=57'11)", 2024-09-18T04:40:14.484 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.355300+0000", 2024-09-18T04:40:14.484 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 83.076032826000002, 2024-09-18T04:40:14.484 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0049192760000000002, 2024-09-18T04:40:14.484 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:14.484 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:14.485 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:14.485 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.485 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:14.485 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355300+0000", 2024-09-18T04:40:14.485 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.485 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.485 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.485 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:14.485 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355300+0000", 2024-09-18T04:40:14.485 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.485 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.485 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.485 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:14.485 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355299+0000", 2024-09-18T04:40:14.486 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:14.486 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.486 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.486 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:14.486 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355876+0000", 2024-09-18T04:40:14.486 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00057730900000000003 2024-09-18T04:40:14.486 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.486 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.486 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:14.486 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355897+0000", 2024-09-18T04:40:14.486 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0477000000000001e-05 2024-09-18T04:40:14.486 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.486 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.486 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:14.487 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355905+0000", 2024-09-18T04:40:14.487 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5739999999999996e-06 2024-09-18T04:40:14.487 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.487 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.487 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:14.487 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355989+0000", 2024-09-18T04:40:14.487 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4002e-05 2024-09-18T04:40:14.487 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.487 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.487 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:14.487 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.356041+0000", 2024-09-18T04:40:14.487 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1347000000000002e-05 2024-09-18T04:40:14.487 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.487 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.487 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:14.488 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.360172+0000", 2024-09-18T04:40:14.488 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0041315090000000002 2024-09-18T04:40:14.488 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.488 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.488 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:14.488 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.360220+0000", 2024-09-18T04:40:14.488 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7584999999999999e-05 2024-09-18T04:40:14.488 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.488 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:14.488 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.488 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.488 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.488 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:679 2.5 e57/28 2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head v 57'20, mlcod=57'18)", 2024-09-18T04:40:14.488 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.445316+0000", 2024-09-18T04:40:14.489 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 82.986017348999994, 2024-09-18T04:40:14.489 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0047016760000000001, 2024-09-18T04:40:14.489 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:14.489 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:14.489 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:14.489 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.489 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:14.489 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445316+0000", 2024-09-18T04:40:14.489 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.489 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.489 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.489 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:14.489 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445316+0000", 2024-09-18T04:40:14.489 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.490 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.490 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.490 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:14.490 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445314+0000", 2024-09-18T04:40:14.490 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:14.490 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.490 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.490 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:14.490 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445947+0000", 2024-09-18T04:40:14.490 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00063298900000000003 2024-09-18T04:40:14.490 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.490 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.490 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:14.490 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445968+0000", 2024-09-18T04:40:14.490 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.054e-05 2024-09-18T04:40:14.491 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.491 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.491 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:14.491 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445981+0000", 2024-09-18T04:40:14.491 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2697000000000001e-05 2024-09-18T04:40:14.491 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.491 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.491 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:14.491 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.446102+0000", 2024-09-18T04:40:14.491 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012189300000000001 2024-09-18T04:40:14.491 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.491 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.491 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:14.491 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.446151+0000", 2024-09-18T04:40:14.492 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8279000000000003e-05 2024-09-18T04:40:14.492 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.492 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.492 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:14.492 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.449953+0000", 2024-09-18T04:40:14.492 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003801979 2024-09-18T04:40:14.492 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.492 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.492 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:14.492 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.450018+0000", 2024-09-18T04:40:14.492 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4873e-05 2024-09-18T04:40:14.492 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.492 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:14.492 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.493 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.493 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.493 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:699 2.4 e57/17 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 57'17, mlcod=57'15)", 2024-09-18T04:40:14.493 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.566073+0000", 2024-09-18T04:40:14.493 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 82.865259836000007, 2024-09-18T04:40:14.493 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055398380000000001, 2024-09-18T04:40:14.493 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:14.493 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:14.493 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:14.493 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.493 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:14.493 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566073+0000", 2024-09-18T04:40:14.493 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.493 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.494 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.494 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:14.494 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566073+0000", 2024-09-18T04:40:14.494 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.494 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.494 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.494 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:14.494 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566071+0000", 2024-09-18T04:40:14.494 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:14.494 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.494 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.494 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:14.494 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566637+0000", 2024-09-18T04:40:14.494 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00056516699999999999 2024-09-18T04:40:14.494 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.495 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.495 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:14.495 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566668+0000", 2024-09-18T04:40:14.495 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1544999999999997e-05 2024-09-18T04:40:14.495 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.495 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.495 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:14.495 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566685+0000", 2024-09-18T04:40:14.495 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6857000000000001e-05 2024-09-18T04:40:14.495 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.495 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.495 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:14.495 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566816+0000", 2024-09-18T04:40:14.496 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013092799999999999 2024-09-18T04:40:14.496 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.496 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.496 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:14.496 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566875+0000", 2024-09-18T04:40:14.496 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9583999999999997e-05 2024-09-18T04:40:14.496 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.496 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.496 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:14.496 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.571567+0000", 2024-09-18T04:40:14.496 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.004691682 2024-09-18T04:40:14.496 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.496 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.496 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:14.497 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.571613+0000", 2024-09-18T04:40:14.497 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6125000000000003e-05 2024-09-18T04:40:14.497 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.497 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:14.497 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.497 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.497 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.497 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:98 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'27, mlcod=60'25)", 2024-09-18T04:40:14.497 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.847978+0000", 2024-09-18T04:40:14.497 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 73.583355276999995, 2024-09-18T04:40:14.497 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0044617709999999998, 2024-09-18T04:40:14.497 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:14.497 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:14.497 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:14.498 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.498 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:14.498 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.847978+0000", 2024-09-18T04:40:14.498 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.498 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.498 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.498 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:14.498 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.847978+0000", 2024-09-18T04:40:14.498 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.498 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.498 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.498 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:14.498 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.847977+0000", 2024-09-18T04:40:14.498 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:14.498 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.499 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.499 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:14.499 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849038+0000", 2024-09-18T04:40:14.499 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0010614820000000001 2024-09-18T04:40:14.499 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.499 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.499 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:14.499 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849065+0000", 2024-09-18T04:40:14.499 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7127999999999998e-05 2024-09-18T04:40:14.499 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.499 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.499 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:14.499 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849082+0000", 2024-09-18T04:40:14.499 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7170999999999999e-05 2024-09-18T04:40:14.499 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.500 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.500 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:14.500 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849188+0000", 2024-09-18T04:40:14.500 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010605800000000001 2024-09-18T04:40:14.500 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.500 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.500 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:14.500 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849233+0000", 2024-09-18T04:40:14.500 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4579000000000001e-05 2024-09-18T04:40:14.500 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.500 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.500 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:14.500 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.852370+0000", 2024-09-18T04:40:14.501 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00313751 2024-09-18T04:40:14.501 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.501 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.501 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:14.501 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.852440+0000", 2024-09-18T04:40:14.501 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9238000000000004e-05 2024-09-18T04:40:14.501 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.501 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:14.501 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.501 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.501 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.501 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:100 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'29, mlcod=64'27)", 2024-09-18T04:40:14.501 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.858766+0000", 2024-09-18T04:40:14.501 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 73.572566948000002, 2024-09-18T04:40:14.502 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0061964840000000004, 2024-09-18T04:40:14.502 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:14.502 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:14.502 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:14.502 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.502 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:14.502 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.858766+0000", 2024-09-18T04:40:14.502 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.502 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.502 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.502 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:14.502 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.858766+0000", 2024-09-18T04:40:14.502 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.502 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.502 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.503 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:14.503 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.858765+0000", 2024-09-18T04:40:14.503 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:14.503 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.503 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.503 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:14.503 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861612+0000", 2024-09-18T04:40:14.503 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028473169999999998 2024-09-18T04:40:14.503 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.503 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.503 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:14.503 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861643+0000", 2024-09-18T04:40:14.503 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1356000000000003e-05 2024-09-18T04:40:14.503 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.504 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.504 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:14.504 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861656+0000", 2024-09-18T04:40:14.504 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2899e-05 2024-09-18T04:40:14.504 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.504 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.504 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:14.504 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861773+0000", 2024-09-18T04:40:14.504 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011637999999999999 2024-09-18T04:40:14.504 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.504 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.504 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:14.504 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861808+0000", 2024-09-18T04:40:14.504 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5002999999999997e-05 2024-09-18T04:40:14.505 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.505 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.505 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:14.505 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.864876+0000", 2024-09-18T04:40:14.505 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030680719999999998 2024-09-18T04:40:14.505 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.505 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.505 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:14.505 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.864963+0000", 2024-09-18T04:40:14.505 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7114000000000007e-05 2024-09-18T04:40:14.505 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.505 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:14.505 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.506 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.506 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.506 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:102 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'31, mlcod=64'29)", 2024-09-18T04:40:14.506 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.872098+0000", 2024-09-18T04:40:14.506 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 73.559235690999998, 2024-09-18T04:40:14.506 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0064676450000000002, 2024-09-18T04:40:14.506 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:14.506 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:14.506 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:14.506 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.506 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:14.506 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.872098+0000", 2024-09-18T04:40:14.506 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.506 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.507 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.507 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:14.507 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.872098+0000", 2024-09-18T04:40:14.507 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.507 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.507 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.507 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:14.507 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.872096+0000", 2024-09-18T04:40:14.507 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:14.507 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.507 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.507 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:14.507 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875176+0000", 2024-09-18T04:40:14.507 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030799809999999999 2024-09-18T04:40:14.508 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.508 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.508 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:14.508 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875204+0000", 2024-09-18T04:40:14.508 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8025999999999999e-05 2024-09-18T04:40:14.508 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.508 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.508 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:14.508 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875216+0000", 2024-09-18T04:40:14.508 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2072999999999999e-05 2024-09-18T04:40:14.508 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.508 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.508 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:14.508 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875307+0000", 2024-09-18T04:40:14.509 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0373999999999999e-05 2024-09-18T04:40:14.509 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.509 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.509 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:14.509 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875349+0000", 2024-09-18T04:40:14.509 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2299000000000003e-05 2024-09-18T04:40:14.509 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.509 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.509 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:14.509 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.878496+0000", 2024-09-18T04:40:14.509 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0031468960000000002 2024-09-18T04:40:14.509 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.509 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.510 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:14.510 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.878565+0000", 2024-09-18T04:40:14.510 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9465e-05 2024-09-18T04:40:14.510 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.510 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:14.510 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.510 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.510 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.510 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:104 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'33, mlcod=64'31)", 2024-09-18T04:40:14.510 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.885100+0000", 2024-09-18T04:40:14.510 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 73.546233156, 2024-09-18T04:40:14.510 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0051888159999999997, 2024-09-18T04:40:14.510 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:14.510 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:14.511 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:14.511 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.511 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:14.511 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.885100+0000", 2024-09-18T04:40:14.511 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.511 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.511 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.511 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:14.511 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.885100+0000", 2024-09-18T04:40:14.511 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.511 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.511 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.511 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:14.511 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.885098+0000", 2024-09-18T04:40:14.512 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:14.512 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.512 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.512 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:14.512 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886809+0000", 2024-09-18T04:40:14.512 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017110459999999999 2024-09-18T04:40:14.512 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.512 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.512 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:14.512 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886842+0000", 2024-09-18T04:40:14.512 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2638999999999997e-05 2024-09-18T04:40:14.512 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.512 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.513 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:14.513 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886855+0000", 2024-09-18T04:40:14.513 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3380000000000001e-05 2024-09-18T04:40:14.513 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.513 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.513 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:14.513 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886939+0000", 2024-09-18T04:40:14.513 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3733999999999995e-05 2024-09-18T04:40:14.513 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.513 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.513 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:14.513 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886981+0000", 2024-09-18T04:40:14.513 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1956000000000003e-05 2024-09-18T04:40:14.513 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.513 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.514 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:14.514 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.890201+0000", 2024-09-18T04:40:14.514 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003219927 2024-09-18T04:40:14.514 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.514 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.514 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:14.514 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.890289+0000", 2024-09-18T04:40:14.514 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7997999999999993e-05 2024-09-18T04:40:14.514 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.514 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:14.514 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.514 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.514 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.514 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:145 2.4 e64/62 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 64'37, mlcod=64'35)", 2024-09-18T04:40:14.514 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:01.099398+0000", 2024-09-18T04:40:14.515 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 73.331935668, 2024-09-18T04:40:14.515 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0050078459999999998, 2024-09-18T04:40:14.515 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:14.515 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:14.515 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:14.515 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.515 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:14.515 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.099398+0000", 2024-09-18T04:40:14.515 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.515 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.515 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.515 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:14.515 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.099398+0000", 2024-09-18T04:40:14.515 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.516 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.516 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.516 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:14.516 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.099396+0000", 2024-09-18T04:40:14.516 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:14.516 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.516 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.516 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:14.516 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100199+0000", 2024-09-18T04:40:14.516 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00080338000000000002 2024-09-18T04:40:14.516 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.516 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.516 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:14.516 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100230+0000", 2024-09-18T04:40:14.517 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0716000000000002e-05 2024-09-18T04:40:14.517 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.517 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.517 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:14.517 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100243+0000", 2024-09-18T04:40:14.517 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3359e-05 2024-09-18T04:40:14.517 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.517 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.517 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:14.517 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100337+0000", 2024-09-18T04:40:14.517 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4387000000000003e-05 2024-09-18T04:40:14.517 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.517 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.517 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:14.518 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100396+0000", 2024-09-18T04:40:14.518 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8164000000000001e-05 2024-09-18T04:40:14.518 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.518 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.518 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:14.518 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.104309+0000", 2024-09-18T04:40:14.518 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0039137800000000004 2024-09-18T04:40:14.518 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.518 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.518 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:14.518 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.104405+0000", 2024-09-18T04:40:14.518 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6039999999999995e-05 2024-09-18T04:40:14.518 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.518 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:14.519 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.519 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.519 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.519 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:147 2.4 e64/62 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 64'39, mlcod=64'37)", 2024-09-18T04:40:14.519 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:01.110658+0000", 2024-09-18T04:40:14.519 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 73.320675578999996, 2024-09-18T04:40:14.519 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0046316910000000003, 2024-09-18T04:40:14.519 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:14.519 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:14.519 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:14.519 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.519 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:14.519 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.110658+0000", 2024-09-18T04:40:14.519 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.520 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.520 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.520 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:14.520 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.110658+0000", 2024-09-18T04:40:14.520 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.520 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.520 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.520 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:14.520 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.110656+0000", 2024-09-18T04:40:14.520 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:14.520 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.520 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.520 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:14.520 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111444+0000", 2024-09-18T04:40:14.521 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00078762400000000001 2024-09-18T04:40:14.521 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.521 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.521 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:14.521 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111474+0000", 2024-09-18T04:40:14.521 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0085000000000001e-05 2024-09-18T04:40:14.521 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.521 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.521 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:14.521 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111486+0000", 2024-09-18T04:40:14.521 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2645e-05 2024-09-18T04:40:14.521 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.521 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.521 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:14.521 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111581+0000", 2024-09-18T04:40:14.522 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4432000000000007e-05 2024-09-18T04:40:14.522 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.522 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.522 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:14.522 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111631+0000", 2024-09-18T04:40:14.522 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9901000000000002e-05 2024-09-18T04:40:14.522 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.522 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.522 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:14.522 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.115209+0000", 2024-09-18T04:40:14.522 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003577916 2024-09-18T04:40:14.522 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.522 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.522 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:14.523 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.115289+0000", 2024-09-18T04:40:14.523 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0704999999999994e-05 2024-09-18T04:40:14.523 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.523 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:14.523 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.523 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.523 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:14.523 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:40:14.525 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:40:14.525 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:40:14.520+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:40:14.548 INFO:tasks.workunit.client.0.smithi115.stderr:+ DEV=/dev/nbd1 2024-09-18T04:40:14.549 INFO:tasks.workunit.client.0.smithi115.stderr:+ get_pid rbd 2024-09-18T04:40:14.549 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pool=rbd 2024-09-18T04:40:14.549 INFO:tasks.workunit.client.0.smithi115.stderr:+ local ns= 2024-09-18T04:40:14.549 INFO:tasks.workunit.client.0.smithi115.stderr:++ rbd device --device-type nbd --format xml list 2024-09-18T04:40:14.549 INFO:tasks.workunit.client.0.smithi115.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd17342'\''][device='\''/dev/nbd1'\'']/id' 2024-09-18T04:40:14.625 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:40:14.619+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:40:14.637 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:40:14.637 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:40:14.637 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:40:14.637 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:40:14.646 INFO:tasks.workunit.client.0.smithi115.stderr:+ PID=20625 2024-09-18T04:40:14.646 INFO:tasks.workunit.client.0.smithi115.stderr:+ test -n 20625 2024-09-18T04:40:14.647 INFO:tasks.workunit.client.0.smithi115.stderr:+ ps -p 20625 -C rbd-nbd 2024-09-18T04:40:14.648 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:40:14.651 INFO:tasks.workunit.client.0.smithi115.stdout: PID TTY TIME CMD 2024-09-18T04:40:14.652 INFO:tasks.workunit.client.0.smithi115.stdout: 20625 ? 00:00:00 rbd-nbd 2024-09-18T04:40:14.652 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd feature enable rbd/testrbdnbd17342 journaling 2024-09-18T04:40:14.726 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:40:14.724+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:40:14.759 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:40:14.760 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:40:14.760 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:40:14.760 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:40:14.760 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:40:14.763 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo blkdiscard --offset 0 --length 4096 /dev/nbd1 2024-09-18T04:40:14.763 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:40:14.764 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:40:14.766 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:40:14.766 INFO:tasks.workunit.client.0.smithi115.stderr:++ which blkdiscard 2024-09-18T04:40:14.769 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/sbin/blkdiscard 2024-09-18T04:40:14.769 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:40:14.769 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 0 --length 4096 /dev/nbd1 2024-09-18T04:40:14.770 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:40:14.804 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd feature disable rbd/testrbdnbd17342 journaling 2024-09-18T04:40:14.827 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:40:14.823+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:40:14.890 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.2 to 1 (10000) 2024-09-18T04:40:14.893 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:40:14.894 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:40:14.894 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:40:14.894 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:40:14.894 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.894 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:33 2.1 e51/24 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head v 51'4, mlcod=51'2)", 2024-09-18T04:40:14.894 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:40:14.894 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 92.609009624999999, 2024-09-18T04:40:14.894 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018702106, 2024-09-18T04:40:14.894 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:14.894 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:14.894 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:14.894 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.895 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:14.895 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:40:14.895 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.895 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.895 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.895 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:14.895 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:40:14.895 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.895 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.895 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.895 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:14.895 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279197+0000", 2024-09-18T04:40:14.895 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:14.895 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.896 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.896 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:14.896 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282682+0000", 2024-09-18T04:40:14.896 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0034848689999999998 2024-09-18T04:40:14.896 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.896 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.896 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:14.896 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282709+0000", 2024-09-18T04:40:14.896 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7599000000000001e-05 2024-09-18T04:40:14.896 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.896 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.896 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:14.896 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282720+0000", 2024-09-18T04:40:14.897 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.092e-05 2024-09-18T04:40:14.897 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.897 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.897 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:14.897 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282791+0000", 2024-09-18T04:40:14.897 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1268999999999998e-05 2024-09-18T04:40:14.897 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.897 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.897 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:14.897 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282843+0000", 2024-09-18T04:40:14.897 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1771999999999998e-05 2024-09-18T04:40:14.897 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.897 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.898 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:14.898 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.297830+0000", 2024-09-18T04:40:14.898 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014987228999999999 2024-09-18T04:40:14.898 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.898 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.898 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:14.898 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.297901+0000", 2024-09-18T04:40:14.898 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0209999999999994e-05 2024-09-18T04:40:14.898 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.898 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:14.898 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.898 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.898 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.898 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:38 2.0 e51/17 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head v 51'10, mlcod=51'8)", 2024-09-18T04:40:14.899 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:40:14.899 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 92.607591194999998, 2024-09-18T04:40:14.899 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.023984472, 2024-09-18T04:40:14.899 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:14.899 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:14.899 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:14.899 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.899 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:14.899 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:40:14.900 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.900 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.900 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.900 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:14.900 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:40:14.900 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.900 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.900 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.900 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:14.900 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280614+0000", 2024-09-18T04:40:14.900 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:40:14.900 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.900 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.901 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:14.901 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293094+0000", 2024-09-18T04:40:14.901 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012479571 2024-09-18T04:40:14.901 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.901 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.901 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:14.901 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293111+0000", 2024-09-18T04:40:14.901 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7354999999999999e-05 2024-09-18T04:40:14.901 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.901 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.901 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:14.901 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293121+0000", 2024-09-18T04:40:14.902 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6979999999999993e-06 2024-09-18T04:40:14.902 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.902 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.902 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:14.902 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293180+0000", 2024-09-18T04:40:14.902 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9333000000000003e-05 2024-09-18T04:40:14.902 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.902 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.902 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:14.902 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293203+0000", 2024-09-18T04:40:14.902 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2633000000000002e-05 2024-09-18T04:40:14.902 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.902 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.903 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:14.903 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304534+0000", 2024-09-18T04:40:14.903 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011330761 2024-09-18T04:40:14.903 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.903 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.903 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:14.903 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304601+0000", 2024-09-18T04:40:14.903 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7657000000000006e-05 2024-09-18T04:40:14.903 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.903 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:14.903 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.903 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.904 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.904 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:46 2.6 2:7ba25245:::rbd_data.125fc09df6d1.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:14.904 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:40:14.904 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 92.580048395000006, 2024-09-18T04:40:14.904 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.035604595000000003, 2024-09-18T04:40:14.904 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:14.904 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:14.904 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:14.904 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:40:14.904 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:40:14.904 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 46 2024-09-18T04:40:14.904 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.904 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:14.905 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.905 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:14.905 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:40:14.905 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.905 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.905 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.905 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:14.905 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:40:14.906 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.906 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.906 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.906 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:14.906 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308158+0000", 2024-09-18T04:40:14.906 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:14.906 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.906 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.906 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:14.906 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313191+0000", 2024-09-18T04:40:14.906 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0050323929999999996 2024-09-18T04:40:14.906 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.906 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.907 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:14.907 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313210+0000", 2024-09-18T04:40:14.907 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9673e-05 2024-09-18T04:40:14.907 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.907 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.907 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:14.907 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313221+0000", 2024-09-18T04:40:14.907 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0444e-05 2024-09-18T04:40:14.907 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.907 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.907 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:14.907 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313294+0000", 2024-09-18T04:40:14.907 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2650000000000004e-05 2024-09-18T04:40:14.907 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.908 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.908 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:14.908 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313493+0000", 2024-09-18T04:40:14.908 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019968300000000001 2024-09-18T04:40:14.908 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.908 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.908 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:40:14.908 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313666+0000", 2024-09-18T04:40:14.908 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00017248599999999999 2024-09-18T04:40:14.908 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.908 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.908 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:14.908 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.323175+0000", 2024-09-18T04:40:14.908 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0095088819999999998 2024-09-18T04:40:14.909 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.909 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.909 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:14.909 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343651+0000", 2024-09-18T04:40:14.909 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020475892999999998 2024-09-18T04:40:14.909 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.909 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.909 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:14.909 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343715+0000", 2024-09-18T04:40:14.909 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4443999999999995e-05 2024-09-18T04:40:14.909 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.909 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.910 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:14.910 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343764+0000", 2024-09-18T04:40:14.910 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9280000000000003e-05 2024-09-18T04:40:14.910 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.910 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:14.910 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.910 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.910 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.910 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:47 2.4 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:14.910 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:40:14.910 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 92.574966357999998, 2024-09-18T04:40:14.910 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.032570693999999997, 2024-09-18T04:40:14.911 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:14.911 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:14.911 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:14.911 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:40:14.911 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:40:14.911 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 47 2024-09-18T04:40:14.911 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.911 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:14.911 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.911 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:14.911 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:40:14.911 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.911 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.912 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.912 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:14.912 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:40:14.912 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.912 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.912 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.912 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:14.912 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313240+0000", 2024-09-18T04:40:14.912 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:14.912 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.913 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.913 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:14.913 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325825+0000", 2024-09-18T04:40:14.913 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012584425999999999 2024-09-18T04:40:14.913 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.913 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.913 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:14.913 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325848+0000", 2024-09-18T04:40:14.913 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3025999999999999e-05 2024-09-18T04:40:14.914 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.914 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.914 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:14.914 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325863+0000", 2024-09-18T04:40:14.914 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5152e-05 2024-09-18T04:40:14.914 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.914 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.914 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:14.914 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325938+0000", 2024-09-18T04:40:14.914 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4982999999999997e-05 2024-09-18T04:40:14.914 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.915 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.915 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:14.915 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326090+0000", 2024-09-18T04:40:14.915 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015207900000000001 2024-09-18T04:40:14.915 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.915 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.915 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:40:14.915 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326242+0000", 2024-09-18T04:40:14.916 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015163500000000001 2024-09-18T04:40:14.916 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.916 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.916 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:14.916 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343015+0000", 2024-09-18T04:40:14.916 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016773414 2024-09-18T04:40:14.916 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.916 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.916 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:14.916 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345664+0000", 2024-09-18T04:40:14.917 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026487870000000001 2024-09-18T04:40:14.917 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.917 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.917 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:14.917 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345739+0000", 2024-09-18T04:40:14.917 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5018999999999994e-05 2024-09-18T04:40:14.917 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.917 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.917 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:14.917 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345812+0000", 2024-09-18T04:40:14.917 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3547000000000006e-05 2024-09-18T04:40:14.917 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.917 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:14.918 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.918 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.918 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.918 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:48 2.6 2:61db7a3a:::rbd_data.125fc09df6d1.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:14.918 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:40:14.918 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 92.562309686000006, 2024-09-18T04:40:14.918 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.025064388, 2024-09-18T04:40:14.918 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:14.918 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:14.918 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:14.918 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:40:14.918 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:40:14.918 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 48 2024-09-18T04:40:14.918 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.918 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:14.919 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.919 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:14.919 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:40:14.919 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.919 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.919 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.919 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:14.919 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:40:14.919 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.919 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.919 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.919 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:14.919 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325897+0000", 2024-09-18T04:40:14.919 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:14.919 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.920 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.920 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:14.920 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333697+0000", 2024-09-18T04:40:14.920 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0078002649999999998 2024-09-18T04:40:14.920 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.920 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.920 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:14.920 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333716+0000", 2024-09-18T04:40:14.920 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8910999999999999e-05 2024-09-18T04:40:14.920 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.920 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.920 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:14.920 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333728+0000", 2024-09-18T04:40:14.920 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1936000000000001e-05 2024-09-18T04:40:14.921 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.921 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.921 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:14.921 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333792+0000", 2024-09-18T04:40:14.921 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4010999999999994e-05 2024-09-18T04:40:14.921 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.921 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.921 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:14.921 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.334082+0000", 2024-09-18T04:40:14.921 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00028955799999999998 2024-09-18T04:40:14.921 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.921 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.921 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:40:14.921 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.334326+0000", 2024-09-18T04:40:14.922 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00024399099999999999 2024-09-18T04:40:14.922 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.922 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.922 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:14.922 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350745+0000", 2024-09-18T04:40:14.922 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016419474999999999 2024-09-18T04:40:14.922 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.922 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.922 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:14.922 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350808+0000", 2024-09-18T04:40:14.922 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3244999999999995e-05 2024-09-18T04:40:14.922 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.922 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.922 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:14.922 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350892+0000", 2024-09-18T04:40:14.923 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4074999999999997e-05 2024-09-18T04:40:14.923 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.923 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.923 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:14.923 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350963+0000", 2024-09-18T04:40:14.923 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0412000000000003e-05 2024-09-18T04:40:14.923 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.923 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:14.923 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.923 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.923 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.923 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:51 2.6 2:775d7b11:::rbd_data.125fc09df6d1.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:14.923 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:40:14.923 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 92.554457549999995, 2024-09-18T04:40:14.924 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.023824221, 2024-09-18T04:40:14.924 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:14.924 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:14.924 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:14.924 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:40:14.924 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:40:14.924 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 51 2024-09-18T04:40:14.924 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.924 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:14.924 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.924 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:14.924 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:40:14.924 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.924 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.925 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.925 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:14.925 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:40:14.925 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.925 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.925 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.925 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:14.925 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333749+0000", 2024-09-18T04:40:14.925 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:14.925 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.925 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.925 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:14.925 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340899+0000", 2024-09-18T04:40:14.926 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0071500340000000004 2024-09-18T04:40:14.926 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.926 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.926 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:14.926 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340929+0000", 2024-09-18T04:40:14.926 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0093000000000001e-05 2024-09-18T04:40:14.926 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.926 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.926 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:14.926 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340943+0000", 2024-09-18T04:40:14.926 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4019000000000001e-05 2024-09-18T04:40:14.927 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.927 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.927 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:14.927 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343834+0000", 2024-09-18T04:40:14.927 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028918120000000001 2024-09-18T04:40:14.927 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.927 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.927 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:14.927 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344020+0000", 2024-09-18T04:40:14.927 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000185892 2024-09-18T04:40:14.927 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.927 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.927 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:40:14.927 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344190+0000", 2024-09-18T04:40:14.927 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016946800000000001 2024-09-18T04:40:14.928 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.928 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.928 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:14.928 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.351291+0000", 2024-09-18T04:40:14.928 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.007100805 2024-09-18T04:40:14.928 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.928 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.928 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:14.928 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357470+0000", 2024-09-18T04:40:14.928 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0061797570000000001 2024-09-18T04:40:14.928 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.928 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.928 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:14.929 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357533+0000", 2024-09-18T04:40:14.929 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2906999999999999e-05 2024-09-18T04:40:14.929 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.929 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.929 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:14.929 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357575+0000", 2024-09-18T04:40:14.929 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1403999999999999e-05 2024-09-18T04:40:14.929 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.929 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:14.929 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.929 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.929 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.929 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:56 2.4 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:14.929 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:40:14.930 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 92.538412457000007, 2024-09-18T04:40:14.930 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.034295870999999999, 2024-09-18T04:40:14.930 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:14.930 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:14.930 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:14.930 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:40:14.930 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:40:14.930 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 56 2024-09-18T04:40:14.930 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.930 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:14.930 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.930 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:14.930 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:40:14.931 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.931 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.931 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.931 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:14.931 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:40:14.931 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.931 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.931 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.931 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:14.931 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349794+0000", 2024-09-18T04:40:14.931 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:14.931 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.931 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.931 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:14.931 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366632+0000", 2024-09-18T04:40:14.932 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01683755 2024-09-18T04:40:14.932 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.932 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.932 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:14.932 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366669+0000", 2024-09-18T04:40:14.932 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7351000000000002e-05 2024-09-18T04:40:14.932 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.932 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.932 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:14.932 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366686+0000", 2024-09-18T04:40:14.932 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6507999999999998e-05 2024-09-18T04:40:14.932 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.932 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.933 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:14.933 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366962+0000", 2024-09-18T04:40:14.933 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00027655900000000001 2024-09-18T04:40:14.933 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.933 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.933 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:14.933 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.367452+0000", 2024-09-18T04:40:14.933 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00049025299999999996 2024-09-18T04:40:14.933 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.933 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.933 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:40:14.933 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368113+0000", 2024-09-18T04:40:14.933 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00066093399999999996 2024-09-18T04:40:14.933 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.933 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.934 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:14.934 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383719+0000", 2024-09-18T04:40:14.934 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015606106999999999 2024-09-18T04:40:14.934 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.934 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.934 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:14.934 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384035+0000", 2024-09-18T04:40:14.934 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00031590399999999999 2024-09-18T04:40:14.934 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.934 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.934 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:14.934 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384067+0000", 2024-09-18T04:40:14.934 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.129e-05 2024-09-18T04:40:14.934 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.934 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.935 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:14.935 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384091+0000", 2024-09-18T04:40:14.935 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4788000000000001e-05 2024-09-18T04:40:14.935 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.935 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:14.935 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.935 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.935 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.935 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:57 2.0 e51/17 2:0cdd070e:::rbd_data.125fc09df6d1.000000000000000d:head v 51'21, mlcod=51'19)", 2024-09-18T04:40:14.935 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:40:14.935 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 92.530584970000007, 2024-09-18T04:40:14.935 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.025216311000000002, 2024-09-18T04:40:14.935 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:14.935 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:14.935 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:14.935 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.936 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:14.936 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:40:14.936 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.936 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.936 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.936 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:14.936 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:40:14.936 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.936 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.936 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.936 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:14.936 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:40:14.937 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:40:14.937 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.937 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.937 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:14.937 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363844+0000", 2024-09-18T04:40:14.937 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0062218450000000002 2024-09-18T04:40:14.937 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.937 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.937 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:14.937 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363879+0000", 2024-09-18T04:40:14.937 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4382999999999999e-05 2024-09-18T04:40:14.937 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.938 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.938 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:14.938 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363893+0000", 2024-09-18T04:40:14.938 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4348999999999999e-05 2024-09-18T04:40:14.938 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.938 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.938 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:14.938 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363977+0000", 2024-09-18T04:40:14.938 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3499999999999997e-05 2024-09-18T04:40:14.938 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.938 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.938 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:14.938 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364009+0000", 2024-09-18T04:40:14.938 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2292000000000002e-05 2024-09-18T04:40:14.939 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.939 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.939 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:14.939 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.382781+0000", 2024-09-18T04:40:14.939 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018771626 2024-09-18T04:40:14.939 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.939 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.939 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:14.939 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.382839+0000", 2024-09-18T04:40:14.939 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8758000000000002e-05 2024-09-18T04:40:14.939 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.939 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:14.939 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.939 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.939 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.940 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:59 2.0 e51/17 2:19c52cde:::rbd_data.125fc09df6d1.000000000000000f:head v 51'22, mlcod=51'19)", 2024-09-18T04:40:14.940 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:40:14.940 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 92.523925043999995, 2024-09-18T04:40:14.940 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.019497112, 2024-09-18T04:40:14.940 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:14.940 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:14.940 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:14.940 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.940 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:14.940 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:40:14.940 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.940 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.940 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.940 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:14.941 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:40:14.941 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.941 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.941 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.941 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:14.941 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364282+0000", 2024-09-18T04:40:14.941 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:14.941 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.941 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.941 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:14.941 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371823+0000", 2024-09-18T04:40:14.941 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0075411829999999999 2024-09-18T04:40:14.941 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.942 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.942 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:14.942 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371837+0000", 2024-09-18T04:40:14.942 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4336e-05 2024-09-18T04:40:14.942 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.942 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.942 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:14.942 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371846+0000", 2024-09-18T04:40:14.942 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9770000000000006e-06 2024-09-18T04:40:14.942 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.942 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.942 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:14.942 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.373566+0000", 2024-09-18T04:40:14.943 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017192539999999999 2024-09-18T04:40:14.943 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.943 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.943 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:14.943 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.373595+0000", 2024-09-18T04:40:14.943 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8975999999999998e-05 2024-09-18T04:40:14.943 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.943 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.943 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:14.943 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383740+0000", 2024-09-18T04:40:14.943 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01014525 2024-09-18T04:40:14.943 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.943 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.944 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:14.944 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383780+0000", 2024-09-18T04:40:14.944 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0275999999999999e-05 2024-09-18T04:40:14.944 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.944 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:14.944 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.944 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.944 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.944 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:34 2.6 2:66289171:ns::rbd_data.133e813ab5bc.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:14.944 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.524015+0000", 2024-09-18T04:40:14.944 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 92.364193435000004, 2024-09-18T04:40:14.945 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027582111999999999, 2024-09-18T04:40:14.945 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:14.945 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:14.945 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:14.945 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:40:14.945 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:14.945 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 34 2024-09-18T04:40:14.945 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.945 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:14.945 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.945 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:14.945 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524015+0000", 2024-09-18T04:40:14.945 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.946 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.946 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.946 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:14.946 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524015+0000", 2024-09-18T04:40:14.946 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.946 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.946 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.946 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:14.946 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524012+0000", 2024-09-18T04:40:14.946 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:40:14.946 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.946 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.947 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:14.947 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534956+0000", 2024-09-18T04:40:14.947 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010943403000000001 2024-09-18T04:40:14.947 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.947 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.947 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:14.947 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534978+0000", 2024-09-18T04:40:14.947 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1965999999999999e-05 2024-09-18T04:40:14.947 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.948 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.948 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:14.948 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534990+0000", 2024-09-18T04:40:14.948 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2065999999999999e-05 2024-09-18T04:40:14.948 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.948 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.948 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:14.948 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535065+0000", 2024-09-18T04:40:14.948 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5523000000000002e-05 2024-09-18T04:40:14.948 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.948 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.948 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:14.948 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535208+0000", 2024-09-18T04:40:14.948 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014309500000000001 2024-09-18T04:40:14.949 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.949 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.949 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:40:14.949 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535343+0000", 2024-09-18T04:40:14.949 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013437400000000001 2024-09-18T04:40:14.949 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.949 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.949 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:14.949 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.546576+0000", 2024-09-18T04:40:14.949 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011233425999999999 2024-09-18T04:40:14.950 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.950 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.950 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:14.950 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551498+0000", 2024-09-18T04:40:14.950 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0049219349999999997 2024-09-18T04:40:14.950 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.950 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.950 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:14.950 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551554+0000", 2024-09-18T04:40:14.950 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6400999999999998e-05 2024-09-18T04:40:14.950 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.950 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.950 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:14.950 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551597+0000", 2024-09-18T04:40:14.950 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2283000000000002e-05 2024-09-18T04:40:14.951 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.951 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:14.951 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.951 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.951 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.951 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:33 2.1 e51/24 2:8b40f1c5:ns::rbd_data.133e813ab5bc.0000000000000000:head v 51'7, mlcod=51'5)", 2024-09-18T04:40:14.951 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.531767+0000", 2024-09-18T04:40:14.951 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 92.356441517999997, 2024-09-18T04:40:14.951 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015370295000000001, 2024-09-18T04:40:14.951 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:14.951 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:14.951 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:14.951 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.951 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:14.951 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531767+0000", 2024-09-18T04:40:14.952 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.952 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.952 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.952 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:14.952 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531767+0000", 2024-09-18T04:40:14.952 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.952 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.952 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.952 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:14.952 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531765+0000", 2024-09-18T04:40:14.952 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:14.952 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.952 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.952 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:14.953 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534258+0000", 2024-09-18T04:40:14.953 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002493561 2024-09-18T04:40:14.953 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.953 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.953 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:14.953 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534288+0000", 2024-09-18T04:40:14.953 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.976e-05 2024-09-18T04:40:14.953 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.953 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.953 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:14.953 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534300+0000", 2024-09-18T04:40:14.953 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1928e-05 2024-09-18T04:40:14.953 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.953 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.953 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:14.953 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534367+0000", 2024-09-18T04:40:14.954 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7045999999999995e-05 2024-09-18T04:40:14.954 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.954 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.954 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:14.954 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534394+0000", 2024-09-18T04:40:14.954 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6903999999999999e-05 2024-09-18T04:40:14.954 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.954 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.954 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:14.954 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.547073+0000", 2024-09-18T04:40:14.954 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012679365 2024-09-18T04:40:14.954 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.954 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.954 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:14.954 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.547137+0000", 2024-09-18T04:40:14.955 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3691999999999999e-05 2024-09-18T04:40:14.955 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.955 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:14.955 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.955 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.955 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.955 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:39 2.4 2:304364d1:ns::rbd_data.133e813ab5bc.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:14.955 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.535013+0000", 2024-09-18T04:40:14.955 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 92.353194583000004, 2024-09-18T04:40:14.955 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027764487000000001, 2024-09-18T04:40:14.955 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:14.955 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:14.955 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:14.955 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:40:14.956 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:14.956 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 39 2024-09-18T04:40:14.956 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.956 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:14.956 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.956 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:14.956 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535013+0000", 2024-09-18T04:40:14.956 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.956 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.956 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.956 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:14.956 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535013+0000", 2024-09-18T04:40:14.956 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.956 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.957 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.957 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:14.957 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535012+0000", 2024-09-18T04:40:14.957 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:14.957 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.957 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.957 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:14.957 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545016+0000", 2024-09-18T04:40:14.957 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01000322 2024-09-18T04:40:14.957 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.957 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.957 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:14.958 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545035+0000", 2024-09-18T04:40:14.958 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9845999999999999e-05 2024-09-18T04:40:14.958 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.958 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.958 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:14.958 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545045+0000", 2024-09-18T04:40:14.958 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3640000000000005e-06 2024-09-18T04:40:14.958 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.958 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.958 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:14.958 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545152+0000", 2024-09-18T04:40:14.958 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000107025 2024-09-18T04:40:14.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.959 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.959 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:14.959 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545338+0000", 2024-09-18T04:40:14.959 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018643500000000001 2024-09-18T04:40:14.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.959 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.959 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:40:14.959 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545492+0000", 2024-09-18T04:40:14.959 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015393199999999999 2024-09-18T04:40:14.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.959 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.959 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:14.959 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.553549+0000", 2024-09-18T04:40:14.959 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0080569170000000002 2024-09-18T04:40:14.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.960 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.960 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:14.960 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562693+0000", 2024-09-18T04:40:14.960 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0091441609999999996 2024-09-18T04:40:14.960 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.960 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.960 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:14.960 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562748+0000", 2024-09-18T04:40:14.960 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5047000000000003e-05 2024-09-18T04:40:14.960 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.960 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.960 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:14.960 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562778+0000", 2024-09-18T04:40:14.961 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9652e-05 2024-09-18T04:40:14.961 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.961 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:14.961 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.961 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.961 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.961 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:40 2.6 2:784d6cd1:ns::rbd_data.133e813ab5bc.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:14.961 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.545062+0000", 2024-09-18T04:40:14.961 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 92.343145531999994, 2024-09-18T04:40:14.961 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.032771623, 2024-09-18T04:40:14.961 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:14.961 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:14.961 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:14.961 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:40:14.962 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:14.962 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 40 2024-09-18T04:40:14.962 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.962 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:14.962 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.962 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:14.962 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545062+0000", 2024-09-18T04:40:14.962 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.962 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.963 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.963 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:14.963 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545062+0000", 2024-09-18T04:40:14.963 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.963 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.963 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:14.963 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545061+0000", 2024-09-18T04:40:14.963 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:14.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.963 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.963 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:14.963 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552477+0000", 2024-09-18T04:40:14.963 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.007415914 2024-09-18T04:40:14.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.964 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.964 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:14.964 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552500+0000", 2024-09-18T04:40:14.964 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3546000000000001e-05 2024-09-18T04:40:14.964 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.964 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.964 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:14.964 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552515+0000", 2024-09-18T04:40:14.964 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4606999999999999e-05 2024-09-18T04:40:14.964 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.964 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.964 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:14.964 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552661+0000", 2024-09-18T04:40:14.965 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000146007 2024-09-18T04:40:14.965 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.965 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.965 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:14.965 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552876+0000", 2024-09-18T04:40:14.965 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000214618 2024-09-18T04:40:14.966 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.966 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.966 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:40:14.966 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.553061+0000", 2024-09-18T04:40:14.966 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018564100000000001 2024-09-18T04:40:14.966 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.966 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.967 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:14.967 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.565612+0000", 2024-09-18T04:40:14.967 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012550301999999999 2024-09-18T04:40:14.967 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.967 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.967 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:14.968 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577695+0000", 2024-09-18T04:40:14.968 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012083493000000001 2024-09-18T04:40:14.968 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.968 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.968 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:14.968 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577772+0000", 2024-09-18T04:40:14.968 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7158999999999998e-05 2024-09-18T04:40:14.969 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.969 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.969 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:14.969 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577834+0000", 2024-09-18T04:40:14.969 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1851000000000001e-05 2024-09-18T04:40:14.969 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.969 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:14.970 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.970 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.970 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.970 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:43 2.6 2:759f27b6:ns::rbd_data.133e813ab5bc.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:14.970 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.555868+0000", 2024-09-18T04:40:14.970 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 92.332340037999998, 2024-09-18T04:40:14.970 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.022332497999999999, 2024-09-18T04:40:14.971 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:14.971 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:14.971 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:14.971 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:40:14.971 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:14.971 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 43 2024-09-18T04:40:14.971 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.972 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:14.972 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.972 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:14.972 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555868+0000", 2024-09-18T04:40:14.972 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.972 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.973 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.973 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:14.973 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555868+0000", 2024-09-18T04:40:14.973 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.973 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.973 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.974 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:14.974 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555865+0000", 2024-09-18T04:40:14.974 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999971 2024-09-18T04:40:14.974 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.974 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.974 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:14.974 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561031+0000", 2024-09-18T04:40:14.975 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0051658770000000001 2024-09-18T04:40:14.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.975 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.975 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:14.975 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561048+0000", 2024-09-18T04:40:14.975 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6569e-05 2024-09-18T04:40:14.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.975 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.975 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:14.975 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561055+0000", 2024-09-18T04:40:14.975 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.328e-06 2024-09-18T04:40:14.976 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.976 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.976 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:14.976 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561114+0000", 2024-09-18T04:40:14.976 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9540000000000003e-05 2024-09-18T04:40:14.976 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.976 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.976 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:14.976 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561314+0000", 2024-09-18T04:40:14.976 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019965799999999999 2024-09-18T04:40:14.976 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.976 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.977 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:40:14.977 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561473+0000", 2024-09-18T04:40:14.977 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000158904 2024-09-18T04:40:14.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.977 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.977 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:14.977 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.567015+0000", 2024-09-18T04:40:14.977 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055420859999999999 2024-09-18T04:40:14.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.977 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.977 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:14.977 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577982+0000", 2024-09-18T04:40:14.977 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010966872000000001 2024-09-18T04:40:14.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.978 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.978 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:14.978 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578162+0000", 2024-09-18T04:40:14.978 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018050999999999999 2024-09-18T04:40:14.978 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.978 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.978 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:14.978 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578200+0000", 2024-09-18T04:40:14.978 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8000999999999998e-05 2024-09-18T04:40:14.978 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.978 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:14.978 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.978 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.978 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.979 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:48 2.6 2:7c225365:ns::rbd_data.133e813ab5bc.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:14.979 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:40:14.979 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 92.324540815000006, 2024-09-18T04:40:14.979 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.024576377, 2024-09-18T04:40:14.979 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:14.979 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:14.979 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:14.979 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:40:14.979 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:14.979 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 48 2024-09-18T04:40:14.979 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.979 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:14.979 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.979 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:14.980 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:40:14.980 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.980 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.980 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:14.980 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:40:14.980 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.980 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.980 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:14.980 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.563666+0000", 2024-09-18T04:40:14.980 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:14.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.980 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.980 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:14.981 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570331+0000", 2024-09-18T04:40:14.981 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0066653329999999998 2024-09-18T04:40:14.981 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.981 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.981 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:14.981 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570348+0000", 2024-09-18T04:40:14.981 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6507e-05 2024-09-18T04:40:14.981 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.981 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.981 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:14.981 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570355+0000", 2024-09-18T04:40:14.981 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2749999999999998e-06 2024-09-18T04:40:14.981 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.982 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.982 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:14.982 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570396+0000", 2024-09-18T04:40:14.982 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0809000000000002e-05 2024-09-18T04:40:14.982 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.982 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.982 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:14.982 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570571+0000", 2024-09-18T04:40:14.982 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000174823 2024-09-18T04:40:14.982 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.982 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.982 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:40:14.983 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570722+0000", 2024-09-18T04:40:14.983 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015154599999999999 2024-09-18T04:40:14.983 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.983 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.983 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:14.983 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579756+0000", 2024-09-18T04:40:14.983 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0090337000000000004 2024-09-18T04:40:14.983 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.983 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.983 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:14.983 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588093+0000", 2024-09-18T04:40:14.983 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.008336553 2024-09-18T04:40:14.984 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.984 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.984 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:14.984 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588173+0000", 2024-09-18T04:40:14.984 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0333999999999994e-05 2024-09-18T04:40:14.984 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.984 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.984 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:14.984 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588244+0000", 2024-09-18T04:40:14.984 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0573000000000003e-05 2024-09-18T04:40:14.984 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.984 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:14.984 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.985 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.985 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.985 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:49 2.4 2:20f0b0c9:ns::rbd_data.133e813ab5bc.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:14.985 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.570370+0000", 2024-09-18T04:40:14.985 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 92.317838488000007, 2024-09-18T04:40:14.985 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.029633981, 2024-09-18T04:40:14.985 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:14.985 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:14.985 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:14.985 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:40:14.985 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:14.986 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 49 2024-09-18T04:40:14.986 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.986 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:14.986 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.986 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:14.986 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570370+0000", 2024-09-18T04:40:14.986 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.986 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.986 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.986 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:14.986 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570370+0000", 2024-09-18T04:40:14.987 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.987 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.987 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.987 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:14.987 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570369+0000", 2024-09-18T04:40:14.987 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:14.987 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.987 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.987 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:14.987 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579090+0000", 2024-09-18T04:40:14.987 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0087212339999999996 2024-09-18T04:40:14.988 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.988 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.988 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:14.988 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579109+0000", 2024-09-18T04:40:14.988 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9120999999999999e-05 2024-09-18T04:40:14.988 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.988 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.988 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:14.988 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579123+0000", 2024-09-18T04:40:14.988 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4382000000000001e-05 2024-09-18T04:40:14.988 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.988 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.988 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:14.989 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579221+0000", 2024-09-18T04:40:14.989 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.7503999999999993e-05 2024-09-18T04:40:14.989 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.989 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.989 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:14.989 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579553+0000", 2024-09-18T04:40:14.989 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00033254200000000002 2024-09-18T04:40:14.989 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.989 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.989 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:40:14.989 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579817+0000", 2024-09-18T04:40:14.989 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00026317099999999998 2024-09-18T04:40:14.989 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.989 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.990 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:14.990 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.599905+0000", 2024-09-18T04:40:14.990 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020088575000000001 2024-09-18T04:40:14.990 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.990 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.990 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:14.990 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.599923+0000", 2024-09-18T04:40:14.990 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7425999999999999e-05 2024-09-18T04:40:14.990 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.990 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.990 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:14.990 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.599973+0000", 2024-09-18T04:40:14.990 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0649999999999998e-05 2024-09-18T04:40:14.991 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.991 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.991 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:14.991 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.600004+0000", 2024-09-18T04:40:14.991 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0361999999999998e-05 2024-09-18T04:40:14.991 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.991 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:14.991 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.991 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.991 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.991 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:50 2.1 e51/24 2:8c706374:ns::rbd_data.133e813ab5bc.000000000000000a:head v 51'8, mlcod=51'6)", 2024-09-18T04:40:14.992 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.577029+0000", 2024-09-18T04:40:14.992 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 92.311179116999995, 2024-09-18T04:40:14.992 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.017362855, 2024-09-18T04:40:14.992 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:14.992 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:14.992 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:14.992 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.992 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:14.992 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577029+0000", 2024-09-18T04:40:14.992 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.992 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.993 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.993 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:14.993 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577029+0000", 2024-09-18T04:40:14.993 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.993 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.993 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.993 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:14.993 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577027+0000", 2024-09-18T04:40:14.993 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:14.993 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.993 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.993 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:14.994 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578880+0000", 2024-09-18T04:40:14.994 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0018532080000000001 2024-09-18T04:40:14.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.994 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.994 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:14.994 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578906+0000", 2024-09-18T04:40:14.994 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5573999999999999e-05 2024-09-18T04:40:14.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.994 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.994 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:14.994 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578920+0000", 2024-09-18T04:40:14.994 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4483e-05 2024-09-18T04:40:14.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.994 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.995 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:14.995 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578983+0000", 2024-09-18T04:40:14.995 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2248000000000004e-05 2024-09-18T04:40:14.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.995 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.995 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:14.995 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579016+0000", 2024-09-18T04:40:14.995 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3364000000000001e-05 2024-09-18T04:40:14.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.995 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.995 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:14.995 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594342+0000", 2024-09-18T04:40:14.995 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01532564 2024-09-18T04:40:14.996 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.996 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.996 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:14.996 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594392+0000", 2024-09-18T04:40:14.996 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0068999999999998e-05 2024-09-18T04:40:14.996 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.996 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:14.996 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:14.996 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.996 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.996 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:45 2.0 e51/17 2:0fbfa3dc:ns::rbd_data.133e813ab5bc.0000000000000007:head v 51'24, mlcod=51'22)", 2024-09-18T04:40:14.996 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.581892+0000", 2024-09-18T04:40:14.996 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 92.306316444999993, 2024-09-18T04:40:14.996 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015646006, 2024-09-18T04:40:14.997 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:14.997 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:14.997 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:14.997 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.997 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:14.997 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581892+0000", 2024-09-18T04:40:14.997 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.997 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.997 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.997 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:14.997 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581892+0000", 2024-09-18T04:40:14.997 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:14.997 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.997 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.998 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:14.998 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581890+0000", 2024-09-18T04:40:14.998 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:14.998 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.998 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.998 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:14.998 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586188+0000", 2024-09-18T04:40:14.998 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0042985769999999996 2024-09-18T04:40:14.998 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.998 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.998 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:14.998 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586210+0000", 2024-09-18T04:40:14.998 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1069000000000001e-05 2024-09-18T04:40:14.998 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.999 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.999 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:14.999 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586221+0000", 2024-09-18T04:40:14.999 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1287000000000001e-05 2024-09-18T04:40:14.999 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.999 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.999 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:14.999 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586313+0000", 2024-09-18T04:40:14.999 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1935999999999995e-05 2024-09-18T04:40:14.999 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:14.999 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:14.999 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:14.999 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586368+0000", 2024-09-18T04:40:14.999 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5578000000000002e-05 2024-09-18T04:40:14.999 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:15.000 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:15.000 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:15.000 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.597473+0000", 2024-09-18T04:40:15.000 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011105139999999999 2024-09-18T04:40:15.000 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:15.000 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:15.000 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:15.000 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.597538+0000", 2024-09-18T04:40:15.000 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4133000000000004e-05 2024-09-18T04:40:15.000 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:15.001 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:15.001 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:15.001 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:15.001 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:15.001 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:53 2.4 2:39f7f174:ns::rbd_data.133e813ab5bc.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:15.001 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.584263+0000", 2024-09-18T04:40:15.001 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 92.303944571000002, 2024-09-18T04:40:15.001 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018701180000000001, 2024-09-18T04:40:15.001 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:15.001 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:15.001 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:15.001 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:40:15.001 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:15.002 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 53 2024-09-18T04:40:15.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:15.002 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:15.002 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:15.002 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:15.002 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584263+0000", 2024-09-18T04:40:15.002 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:15.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:15.002 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:15.002 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:15.002 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584263+0000", 2024-09-18T04:40:15.002 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:15.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:15.003 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:15.003 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:15.003 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584262+0000", 2024-09-18T04:40:15.003 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:15.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:15.003 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:15.003 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:15.003 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.591617+0000", 2024-09-18T04:40:15.003 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0073550600000000001 2024-09-18T04:40:15.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:15.003 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:15.004 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:15.004 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.591637+0000", 2024-09-18T04:40:15.004 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0432999999999999e-05 2024-09-18T04:40:15.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:15.004 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:15.004 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:15.004 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.591646+0000", 2024-09-18T04:40:15.004 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7849999999999997e-06 2024-09-18T04:40:15.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:15.004 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:15.004 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:15.005 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593478+0000", 2024-09-18T04:40:15.005 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001831495 2024-09-18T04:40:15.005 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:15.005 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:15.005 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:15.005 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593661+0000", 2024-09-18T04:40:15.005 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018364999999999999 2024-09-18T04:40:15.005 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:15.005 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:15.005 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:40:15.005 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593834+0000", 2024-09-18T04:40:15.006 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000173065 2024-09-18T04:40:15.006 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:15.006 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:15.006 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:15.006 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.600670+0000", 2024-09-18T04:40:15.006 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0068354970000000003 2024-09-18T04:40:15.006 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:15.006 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:15.007 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:15.007 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602851+0000", 2024-09-18T04:40:15.007 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002181653 2024-09-18T04:40:15.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:15.007 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:15.007 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:15.007 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602930+0000", 2024-09-18T04:40:15.007 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8045000000000002e-05 2024-09-18T04:40:15.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:15.007 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:15.007 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:15.007 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602965+0000", 2024-09-18T04:40:15.007 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5102999999999999e-05 2024-09-18T04:40:15.008 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:15.008 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:15.008 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:15.008 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:15.008 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:15.008 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:56 2.6 2:72453c3e:ns::rbd_data.133e813ab5bc.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:15.008 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.595988+0000", 2024-09-18T04:40:15.008 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 92.292220169000004, 2024-09-18T04:40:15.008 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.030296172, 2024-09-18T04:40:15.008 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:15.008 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:15.008 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:15.008 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:40:15.009 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:15.009 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 56 2024-09-18T04:40:15.009 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:15.009 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:15.009 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:15.009 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:15.009 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.595988+0000", 2024-09-18T04:40:15.009 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:15.009 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:15.009 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:15.009 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:15.009 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.595988+0000", 2024-09-18T04:40:15.009 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:15.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:15.010 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:15.010 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:15.010 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.595986+0000", 2024-09-18T04:40:15.010 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:40:15.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:15.010 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:15.010 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:15.010 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605262+0000", 2024-09-18T04:40:15.010 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0092767240000000001 2024-09-18T04:40:15.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:15.010 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:15.010 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:15.010 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605282+0000", 2024-09-18T04:40:15.010 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9307e-05 2024-09-18T04:40:15.011 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:15.011 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:15.011 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:15.011 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605292+0000", 2024-09-18T04:40:15.011 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9510000000000001e-06 2024-09-18T04:40:15.011 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:15.011 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:15.011 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:15.011 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605369+0000", 2024-09-18T04:40:15.011 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7846e-05 2024-09-18T04:40:15.011 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:15.011 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:15.011 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:15.011 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605668+0000", 2024-09-18T04:40:15.011 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00029896099999999999 2024-09-18T04:40:15.012 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:15.012 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:15.012 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:40:15.012 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605906+0000", 2024-09-18T04:40:15.012 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000237758 2024-09-18T04:40:15.012 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:15.012 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:15.012 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:15.012 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.613532+0000", 2024-09-18T04:40:15.012 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0076260590000000001 2024-09-18T04:40:15.012 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:15.012 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:15.012 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:15.012 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.626116+0000", 2024-09-18T04:40:15.013 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012584092999999999 2024-09-18T04:40:15.013 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:15.013 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:15.013 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:15.013 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.626216+0000", 2024-09-18T04:40:15.013 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9630999999999995e-05 2024-09-18T04:40:15.013 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:15.013 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:15.013 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:15.013 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.626284+0000", 2024-09-18T04:40:15.013 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8034999999999995e-05 2024-09-18T04:40:15.014 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:15.014 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:15.014 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:15.014 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:15.014 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:15.014 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:40:15.014 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-18T04:40:15.017 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-18T04:40:15.018 INFO:tasks.workunit.client.0.smithi115.stderr:+ unmap_device /dev/nbd1 20625 2024-09-18T04:40:15.018 INFO:tasks.workunit.client.0.smithi115.stderr:+ local args=/dev/nbd1 2024-09-18T04:40:15.018 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pid=20625 2024-09-18T04:40:15.018 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd1 2024-09-18T04:40:15.019 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:40:15.019 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:40:15.019 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:40:15.019 INFO:tasks.workunit.client.0.smithi115.stderr:++ which rbd 2024-09-18T04:40:15.019 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/bin/rbd 2024-09-18T04:40:15.019 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:40:15.019 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd1 2024-09-18T04:40:15.019 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:40:15.020 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:40:15.012+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:40:15.022 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T04:40:15.051 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd device --device-type nbd list 2024-09-18T04:40:15.052 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false grep '^20625\b' 2024-09-18T04:40:15.052 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep '^20625\b' 2024-09-18T04:40:15.117 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:40:15.112+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:40:15.126 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:40:15.126 INFO:tasks.workunit.client.0.smithi115.stderr:+ ps -C rbd-nbd 2024-09-18T04:40:15.127 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false grep '^ *20625\b' 2024-09-18T04:40:15.127 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep '^ *20625\b' 2024-09-18T04:40:15.132 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:40:15.133 INFO:tasks.workunit.client.0.smithi115.stderr:+ sleep 0.5 2024-09-18T04:40:15.134 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:40:15.134 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:40:15.134 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:40:15.134 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:40:15.146 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:40:15.203 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:40:15.217 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:40:15.211+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:40:15.242 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:40:15.254 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:40:15.254 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:40:15.254 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:40:15.254 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:40:15.254 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:40:15.264 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:40:15.276 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:40:15.310 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:40:15.318 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:40:15.316+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:40:15.345 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:40:15.371 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:15.371 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:15.380 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:40:15.381 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:40:15.382 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:40:15.382 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:40:15.382 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:40:15.383 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:40:15.383 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:40:15.383 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:40:15.419 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:40:15.415+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:40:15.521 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:40:15.519+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:40:15.535 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:15.535 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:15.544 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:40:15.623 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:40:15.619+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:40:15.634 INFO:tasks.workunit.client.0.smithi115.stderr:+ DEV= 2024-09-18T04:40:15.635 INFO:tasks.workunit.client.0.smithi115.stderr:++ ceph-conf --show-config-value librados_thread_count 2024-09-18T04:40:15.649 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:15.649 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:15.658 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:40:15.667 INFO:tasks.workunit.client.0.smithi115.stderr:+ EXPECTED=2 2024-09-18T04:40:15.667 INFO:tasks.workunit.client.0.smithi115.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd17342 2024-09-18T04:40:15.667 INFO:tasks.workunit.client.0.smithi115.stderr:++ local cmd 2024-09-18T04:40:15.668 INFO:tasks.workunit.client.0.smithi115.stderr:+++ id -u 2024-09-18T04:40:15.670 INFO:tasks.workunit.client.0.smithi115.stderr:++ '[' 1000 -eq 0 ']' 2024-09-18T04:40:15.670 INFO:tasks.workunit.client.0.smithi115.stderr:+++ which rbd 2024-09-18T04:40:15.673 INFO:tasks.workunit.client.0.smithi115.stderr:++ cmd=/usr/bin/rbd 2024-09-18T04:40:15.673 INFO:tasks.workunit.client.0.smithi115.stderr:++ shift 2024-09-18T04:40:15.673 INFO:tasks.workunit.client.0.smithi115.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd17342 2024-09-18T04:40:15.724 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:40:15.720+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:40:15.762 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:15.762 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:15.773 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:40:15.825 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:40:15.820+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:40:15.874 INFO:tasks.workunit.client.0.smithi115.stderr:+ DEV=/dev/nbd1 2024-09-18T04:40:15.874 INFO:tasks.workunit.client.0.smithi115.stderr:+ get_pid rbd 2024-09-18T04:40:15.874 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pool=rbd 2024-09-18T04:40:15.874 INFO:tasks.workunit.client.0.smithi115.stderr:+ local ns= 2024-09-18T04:40:15.875 INFO:tasks.workunit.client.0.smithi115.stderr:++ rbd device --device-type nbd --format xml list 2024-09-18T04:40:15.875 INFO:tasks.workunit.client.0.smithi115.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd17342'\''][device='\''/dev/nbd1'\'']/id' 2024-09-18T04:40:15.887 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:15.887 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:15.897 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:40:15.903 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-09-18T04:40:15.904 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.4 to 1 (10000) 2024-09-18T04:40:15.917 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-18T04:40:15.918 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T04:40:15.926 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:40:15.920+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:40:15.967 INFO:tasks.workunit.client.0.smithi115.stderr:+ PID=20738 2024-09-18T04:40:15.967 INFO:tasks.workunit.client.0.smithi115.stderr:+ test -n 20738 2024-09-18T04:40:15.967 INFO:tasks.workunit.client.0.smithi115.stderr:+ ps -p 20738 -C rbd-nbd 2024-09-18T04:40:15.971 INFO:tasks.workunit.client.0.smithi115.stdout: PID TTY TIME CMD 2024-09-18T04:40:15.971 INFO:tasks.workunit.client.0.smithi115.stdout: 20738 ? 00:00:00 rbd-nbd 2024-09-18T04:40:15.972 INFO:tasks.workunit.client.0.smithi115.stderr:++ ps -p 20738 -T 2024-09-18T04:40:15.972 INFO:tasks.workunit.client.0.smithi115.stderr:++ grep -c io_context_pool 2024-09-18T04:40:15.986 INFO:tasks.workunit.client.0.smithi115.stderr:+ ACTUAL=2 2024-09-18T04:40:15.986 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 2 -eq 2 ']' 2024-09-18T04:40:15.986 INFO:tasks.workunit.client.0.smithi115.stderr:+ unmap_device /dev/nbd1 20738 2024-09-18T04:40:15.986 INFO:tasks.workunit.client.0.smithi115.stderr:+ local args=/dev/nbd1 2024-09-18T04:40:15.987 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pid=20738 2024-09-18T04:40:15.987 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd1 2024-09-18T04:40:15.987 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:40:15.987 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:40:15.987 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:40:15.988 INFO:tasks.workunit.client.0.smithi115.stderr:++ which rbd 2024-09-18T04:40:15.989 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/bin/rbd 2024-09-18T04:40:15.989 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:40:15.989 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd1 2024-09-18T04:40:15.997 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:15.997 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:16.007 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:40:16.053 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:40:16.048+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:40:16.105 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd device --device-type nbd list 2024-09-18T04:40:16.105 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false grep '^20738\b' 2024-09-18T04:40:16.105 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep '^20738\b' 2024-09-18T04:40:16.128 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:40:16.124+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:40:16.165 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:16.166 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:16.174 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:40:16.181 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:40:16.182 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false grep '^ *20738\b' 2024-09-18T04:40:16.182 INFO:tasks.workunit.client.0.smithi115.stderr:+ ps -C rbd-nbd 2024-09-18T04:40:16.182 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep '^ *20738\b' 2024-09-18T04:40:16.192 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:40:16.193 INFO:tasks.workunit.client.0.smithi115.stderr:+ sleep 0.5 2024-09-18T04:40:16.229 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:40:16.223+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:40:16.287 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:16.287 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:16.291 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:40:16.295 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-18T04:40:16.298 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:40:16.306 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":185,"stamp":"2024-09-18T04:40:15.892708+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":147465357,"num_objects":80,"num_object_clones":19,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":13,"num_read":7277,"num_read_kb":179969,"num_write":977,"num_write_kb":273266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":39960608,"num_keys_recovered":25,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":952,"ondisk_log_size":952,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":470356,"kb_used_data":277952,"kb_used_omap":17,"kb_used_meta":192302,"kb_avail":565760684,"statfs":{"total":579820584960,"available":579338940416,"internally_reserved":0,"allocated":284622848,"data_stored":302102808,"data_compressed":482441,"data_compressed_allocated":516096,"data_compressed_original":9863168,"omap_allocated":18101,"internal_metadata":196917579},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-78,"num_objects":-1,"num_object_clones":0,"num_object_copies":-2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":641,"num_read_kb":1709,"num_write":17,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.039762"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":247,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-18T04:40:13.880739+0000","last_change":"2024-09-18T04:37:27.423069+0000","last_active":"2024-09-18T04:40:13.880739+0000","last_peered":"2024-09-18T04:40:13.880739+0000","last_clean":"2024-09-18T04:40:13.880739+0000","last_became_active":"2024-09-18T04:37:27.422350+0000","last_became_peered":"2024-09-18T04:37:27.422350+0000","last_unstale":"2024-09-18T04:40:13.880739+0000","last_undegraded":"2024-09-18T04:40:13.880739+0000","last_fullsized":"2024-09-18T04:40:13.880739+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:26.406292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:26.406292+0000","last_clean_scrub_stamp":"2024-09-18T04:37:26.406292+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:31:45.852149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"99'107","reported_seq":580,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-18T04:40:10.823393+0000","last_change":"2024-09-18T04:39:46.795376+0000","last_active":"2024-09-18T04:40:10.823393+0000","last_peered":"2024-09-18T04:40:10.823393+0000","last_clean":"2024-09-18T04:40:10.823393+0000","last_became_active":"2024-09-18T04:39:46.718066+0000","last_became_peered":"2024-09-18T04:39:46.718066+0000","last_unstale":"2024-09-18T04:40:10.823393+0000","last_undegraded":"2024-09-18T04:40:10.823393+0000","last_fullsized":"2024-09-18T04:40:10.823393+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T07:09:22.599338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018293420000000001,"stat_sum":{"num_bytes":10535160,"num_objects":11,"num_object_clones":2,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":550,"num_read_kb":17110,"num_write":115,"num_write_kb":25705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":9486336,"num_keys_recovered":9,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"69'3","reported_seq":71,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-18T04:40:13.143324+0000","last_change":"2024-09-18T04:38:46.966799+0000","last_active":"2024-09-18T04:40:13.143324+0000","last_peered":"2024-09-18T04:40:13.143324+0000","last_clean":"2024-09-18T04:40:13.143324+0000","last_became_active":"2024-09-18T04:38:44.953092+0000","last_became_peered":"2024-09-18T04:38:44.953092+0000","last_unstale":"2024-09-18T04:40:13.143324+0000","last_undegraded":"2024-09-18T04:40:13.143324+0000","last_fullsized":"2024-09-18T04:40:13.143324+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_clean_scrub_stamp":"2024-09-18T04:38:43.928423+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:04:46.888906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010424099999999999,"stat_sum":{"num_bytes":54,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"104'215","reported_seq":1321,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-18T04:40:14.883246+0000","last_change":"2024-09-18T04:37:36.484950+0000","last_active":"2024-09-18T04:40:14.883246+0000","last_peered":"2024-09-18T04:40:14.883246+0000","last_clean":"2024-09-18T04:40:14.883246+0000","last_became_active":"2024-09-18T04:37:34.470841+0000","last_became_peered":"2024-09-18T04:37:34.470841+0000","last_unstale":"2024-09-18T04:40:14.883246+0000","last_undegraded":"2024-09-18T04:40:14.883246+0000","last_fullsized":"2024-09-18T04:40:14.883246+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":215,"log_dups_size":0,"ondisk_log_size":215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:16:34.528783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090513900000000001,"stat_sum":{"num_bytes":25165900,"num_objects":11,"num_object_clones":3,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":2,"num_read":1718,"num_read_kb":39765,"num_write":314,"num_write_kb":45334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":88,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-18T04:40:09.850372+0000","last_change":"2024-09-18T04:40:05.832504+0000","last_active":"2024-09-18T04:40:09.850372+0000","last_peered":"2024-09-18T04:40:09.850372+0000","last_clean":"2024-09-18T04:40:09.850372+0000","last_became_active":"2024-09-18T04:40:05.832216+0000","last_became_peered":"2024-09-18T04:40:05.832216+0000","last_unstale":"2024-09-18T04:40:09.850372+0000","last_undegraded":"2024-09-18T04:40:09.850372+0000","last_fullsized":"2024-09-18T04:40:09.850372+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_clean_scrub_stamp":"2024-09-18T04:38:43.928423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:03:41.967712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075989799999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"103'260","reported_seq":1983,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-18T04:40:14.714035+0000","last_change":"2024-09-18T04:39:46.746657+0000","last_active":"2024-09-18T04:40:14.714035+0000","last_peered":"2024-09-18T04:40:14.714035+0000","last_clean":"2024-09-18T04:40:14.714035+0000","last_became_active":"2024-09-18T04:39:46.716998+0000","last_became_peered":"2024-09-18T04:39:46.716998+0000","last_unstale":"2024-09-18T04:40:14.714035+0000","last_undegraded":"2024-09-18T04:40:14.714035+0000","last_fullsized":"2024-09-18T04:40:14.714035+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":260,"log_dups_size":0,"ondisk_log_size":260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:37:53.810382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18898982,"num_objects":9,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3943,"num_read_kb":29480,"num_write":162,"num_write_kb":27759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":5308416,"num_keys_recovered":13,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":73,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-18T04:40:13.103431+0000","last_change":"2024-09-18T04:39:59.780545+0000","last_active":"2024-09-18T04:40:13.103431+0000","last_peered":"2024-09-18T04:40:13.103431+0000","last_clean":"2024-09-18T04:40:13.103431+0000","last_became_active":"2024-09-18T04:39:59.780106+0000","last_became_peered":"2024-09-18T04:39:59.780106+0000","last_unstale":"2024-09-18T04:40:13.103431+0000","last_undegraded":"2024-09-18T04:40:13.103431+0000","last_fullsized":"2024-09-18T04:40:13.103431+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_clean_scrub_stamp":"2024-09-18T04:38:43.928423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T08:02:28.727480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011574549999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"20'2","reported_seq":201,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-18T04:40:10.675453+0000","last_change":"2024-09-18T04:37:36.490461+0000","last_active":"2024-09-18T04:40:10.675453+0000","last_peered":"2024-09-18T04:40:10.675453+0000","last_clean":"2024-09-18T04:40:10.675453+0000","last_became_active":"2024-09-18T04:37:34.473088+0000","last_became_peered":"2024-09-18T04:37:34.473088+0000","last_unstale":"2024-09-18T04:40:10.675453+0000","last_undegraded":"2024-09-18T04:40:10.675453+0000","last_fullsized":"2024-09-18T04:40:10.675453+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:17:57.352397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001879932,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":105,"num_read_kb":105,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"69'2","reported_seq":18,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-18T04:40:07.449849+0000","last_change":"2024-09-18T04:40:05.842027+0000","last_active":"2024-09-18T04:40:07.449849+0000","last_peered":"2024-09-18T04:40:07.449849+0000","last_clean":"2024-09-18T04:40:07.449849+0000","last_became_active":"2024-09-18T04:40:05.831890+0000","last_became_peered":"2024-09-18T04:40:05.831890+0000","last_unstale":"2024-09-18T04:40:07.449849+0000","last_undegraded":"2024-09-18T04:40:07.449849+0000","last_fullsized":"2024-09-18T04:40:07.449849+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_clean_scrub_stamp":"2024-09-18T04:38:43.928423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:27:26.157495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010091900000000001,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":32,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"69'3","reported_seq":85,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-18T04:40:06.840655+0000","last_change":"2024-09-18T04:39:46.720085+0000","last_active":"2024-09-18T04:40:06.840655+0000","last_peered":"2024-09-18T04:40:06.840655+0000","last_clean":"2024-09-18T04:40:06.840655+0000","last_became_active":"2024-09-18T04:39:46.719365+0000","last_became_peered":"2024-09-18T04:39:46.719365+0000","last_unstale":"2024-09-18T04:40:06.840655+0000","last_undegraded":"2024-09-18T04:40:06.840655+0000","last_fullsized":"2024-09-18T04:40:06.840655+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_clean_scrub_stamp":"2024-09-18T04:38:43.928423+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:38:16.629282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065509499999999996,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"98'46","reported_seq":88,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-18T04:40:10.813068+0000","last_change":"2024-09-18T04:38:59.528402+0000","last_active":"2024-09-18T04:40:10.813068+0000","last_peered":"2024-09-18T04:40:10.813068+0000","last_clean":"2024-09-18T04:40:10.813068+0000","last_became_active":"2024-09-18T04:38:59.462279+0000","last_became_peered":"2024-09-18T04:38:59.462279+0000","last_unstale":"2024-09-18T04:40:10.813068+0000","last_undegraded":"2024-09-18T04:40:10.813068+0000","last_fullsized":"2024-09-18T04:40:10.813068+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:32:58.894411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081514499999999998,"stat_sum":{"num_bytes":20971598,"num_objects":10,"num_object_clones":2,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":2,"num_read":91,"num_read_kb":16656,"num_write":44,"num_write_kb":36867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":12582912,"num_keys_recovered":3,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"69'1","reported_seq":79,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-18T04:40:13.143625+0000","last_change":"2024-09-18T04:39:46.718266+0000","last_active":"2024-09-18T04:40:13.143625+0000","last_peered":"2024-09-18T04:40:13.143625+0000","last_clean":"2024-09-18T04:40:13.143625+0000","last_became_active":"2024-09-18T04:39:46.717950+0000","last_became_peered":"2024-09-18T04:39:46.717950+0000","last_unstale":"2024-09-18T04:40:13.143625+0000","last_undegraded":"2024-09-18T04:40:13.143625+0000","last_fullsized":"2024-09-18T04:40:13.143625+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_clean_scrub_stamp":"2024-09-18T04:38:43.928423+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:17:31.749220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016757619999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"79'64","reported_seq":393,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-18T04:40:13.103529+0000","last_change":"2024-09-18T04:39:46.817083+0000","last_active":"2024-09-18T04:40:13.103529+0000","last_peered":"2024-09-18T04:40:13.103529+0000","last_clean":"2024-09-18T04:40:13.103529+0000","last_became_active":"2024-09-18T04:39:46.717423+0000","last_became_peered":"2024-09-18T04:39:46.717423+0000","last_unstale":"2024-09-18T04:40:13.103529+0000","last_undegraded":"2024-09-18T04:40:13.103529+0000","last_fullsized":"2024-09-18T04:40:13.103529+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T09:52:29.555572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165862,"num_objects":10,"num_object_clones":3,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":3,"num_read":244,"num_read_kb":31404,"num_write":61,"num_write_kb":49153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":48,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-18T04:40:13.103561+0000","last_change":"2024-09-18T04:39:46.717470+0000","last_active":"2024-09-18T04:40:13.103561+0000","last_peered":"2024-09-18T04:40:13.103561+0000","last_clean":"2024-09-18T04:40:13.103561+0000","last_became_active":"2024-09-18T04:39:46.717099+0000","last_became_peered":"2024-09-18T04:39:46.717099+0000","last_unstale":"2024-09-18T04:40:13.103561+0000","last_undegraded":"2024-09-18T04:40:13.103561+0000","last_fullsized":"2024-09-18T04:40:13.103561+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_clean_scrub_stamp":"2024-09-18T04:38:43.928423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:08:02.208525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"103'70","reported_seq":293,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-18T04:40:13.880048+0000","last_change":"2024-09-18T04:37:36.494193+0000","last_active":"2024-09-18T04:40:13.880048+0000","last_peered":"2024-09-18T04:40:13.880048+0000","last_clean":"2024-09-18T04:40:13.880048+0000","last_became_active":"2024-09-18T04:37:34.472940+0000","last_became_peered":"2024-09-18T04:37:34.472940+0000","last_unstale":"2024-09-18T04:40:13.880048+0000","last_undegraded":"2024-09-18T04:40:13.880048+0000","last_fullsized":"2024-09-18T04:40:13.880048+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:18:29.266815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070553900000000004,"stat_sum":{"num_bytes":33554470,"num_objects":13,"num_object_clones":3,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":3,"num_read":131,"num_read_kb":28707,"num_write":67,"num_write_kb":57348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":75,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-18T04:40:09.850785+0000","last_change":"2024-09-18T04:39:18.577075+0000","last_active":"2024-09-18T04:40:09.850785+0000","last_peered":"2024-09-18T04:40:09.850785+0000","last_clean":"2024-09-18T04:40:09.850785+0000","last_became_active":"2024-09-18T04:39:18.576593+0000","last_became_peered":"2024-09-18T04:39:18.576593+0000","last_unstale":"2024-09-18T04:40:09.850785+0000","last_undegraded":"2024-09-18T04:40:09.850785+0000","last_fullsized":"2024-09-18T04:40:09.850785+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_clean_scrub_stamp":"2024-09-18T04:38:43.928423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:27:02.258820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058611500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"79'62","reported_seq":340,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-18T04:40:13.880675+0000","last_change":"2024-09-18T04:38:05.754820+0000","last_active":"2024-09-18T04:40:13.880675+0000","last_peered":"2024-09-18T04:40:13.880675+0000","last_clean":"2024-09-18T04:40:13.880675+0000","last_became_active":"2024-09-18T04:38:05.753962+0000","last_became_peered":"2024-09-18T04:38:05.753962+0000","last_unstale":"2024-09-18T04:40:13.880675+0000","last_undegraded":"2024-09-18T04:40:13.880675+0000","last_fullsized":"2024-09-18T04:40:13.880675+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:53:52.224098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047343899999999998,"stat_sum":{"num_bytes":12582928,"num_objects":7,"num_object_clones":2,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":2,"num_read":344,"num_read_kb":16613,"num_write":47,"num_write_kb":28676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":70,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":32,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1835,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":146874919,"num_objects":72,"num_object_clones":19,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":13,"num_read":7126,"num_read_kb":179840,"num_write":812,"num_write_kb":270844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":39960576,"num_keys_recovered":25,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":277364736,"data_stored":298140594,"data_compressed":416401,"data_compressed_allocated":425984,"data_compressed_original":8716288,"omap_allocated":3627,"internal_metadata":0},"log_size":826,"ondisk_log_size":826,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":66040,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":13,"seq":55834574885,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74304,"kb_used_data":43192,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94297536,"statfs":{"total":96636764160,"available":96560676864,"internally_reserved":0,"allocated":44228608,"data_stored":43489464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:39:28 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.85099999999999998,"15min":0.86499999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.0469999999999999,"5min":1.5249999999999999,"15min":1.5249999999999999},"last":0.879},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.79700000000000004,"15min":0.80800000000000005},"min":{"1min":0.53500000000000003,"5min":0.502,"15min":0.502},"max":{"1min":0.996,"5min":1.369,"15min":1.369},"last":0.76400000000000001}]},{"osd":1,"last update":"Wed Sep 18 04:39:28 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.83999999999999997,"15min":0.85199999999999998},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.091,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.79500000000000004,"15min":0.79900000000000004},"min":{"1min":0.59999999999999998,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.52700000000000002}]},{"osd":2,"last update":"Wed Sep 18 04:38:26 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.4930000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.85199999999999998}]},{"osd":3,"last update":"Wed Sep 18 04:39:28 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.75700000000000001,"15min":0.77000000000000002},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.0169999999999999,"5min":1.22,"15min":1.22},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.78900000000000003,"15min":0.79700000000000004},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.78900000000000003}]},{"osd":4,"last update":"Wed Sep 18 04:39:28 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.77800000000000002,"15min":0.78500000000000003},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.998,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.751,"5min":0.82499999999999996,"15min":0.83699999999999997},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.075,"5min":1.121,"15min":1.121},"last":0.81999999999999995}]}]},{"osd":4,"up_from":13,"seq":55834574885,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32252,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339588,"statfs":{"total":96636764160,"available":96603738112,"internally_reserved":0,"allocated":1163264,"data_stored":453383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:39:31 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.76600000000000001,"15min":0.77200000000000002},"min":{"1min":0.52200000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.044,"5min":1.175,"15min":1.175},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.78500000000000003,"15min":0.79500000000000004},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.1060000000000001,"5min":1.258,"15min":1.258},"last":0.66300000000000003}]},{"osd":1,"last update":"Wed Sep 18 04:39:31 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.72199999999999998,"15min":0.72799999999999998},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.95899999999999996,"5min":1.002,"15min":1.002},"last":0.63},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.875,"15min":0.88400000000000001},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":0.93500000000000005}]},{"osd":2,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.751},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.619,"5min":1.619,"15min":1.619},"last":0.81299999999999994}]},{"osd":3,"last update":"Wed Sep 18 04:39:31 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.81299999999999994,"15min":0.82299999999999995},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0129999999999999,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":2.302,"5min":2.302,"15min":2.302},"last":0.70699999999999996}]},{"osd":5,"last update":"Wed Sep 18 04:39:31 2024","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.88200000000000001,"15min":0.88700000000000001},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.365,"5min":1.365,"15min":1.365},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.89000000000000001,"15min":0.88800000000000001},"min":{"1min":0.42599999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":2.0979999999999999,"5min":2.0979999999999999,"15min":2.0979999999999999},"last":0.73799999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574886,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81476,"kb_used_data":46264,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94290364,"statfs":{"total":96636764160,"available":96553332736,"internally_reserved":0,"allocated":47374336,"data_stored":47181211,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 04:39:28 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.59999999999999998,"15min":0.60899999999999999},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.78000000000000003,"5min":1.113,"15min":1.113},"last":0.621},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.63,"15min":0.64200000000000002},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.89900000000000002,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.56899999999999995}]},{"osd":2,"last update":"Wed Sep 18 04:38:28 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.8180000000000001,"5min":1.8180000000000001,"15min":1.8180000000000001},"last":0.41999999999999998}]},{"osd":3,"last update":"Wed Sep 18 04:39:28 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.81499999999999995,"15min":0.81799999999999995},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":0.99399999999999999,"5min":1.169,"15min":1.169},"last":0.97599999999999998},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.81899999999999995,"15min":0.82599999999999996},"min":{"1min":0.49199999999999999,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.97699999999999998,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.66700000000000004}]},{"osd":4,"last update":"Wed Sep 18 04:39:28 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.84599999999999997,"15min":0.85399999999999998},"min":{"1min":0.61699999999999999,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.0600000000000001,"5min":1.073,"15min":1.073},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.89600000000000002,"15min":0.90800000000000003},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.879}]},{"osd":5,"last update":"Wed Sep 18 04:39:28 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.91600000000000004,"15min":0.93100000000000005},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.1479999999999999,"5min":1.8919999999999999,"15min":1.8919999999999999},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.95399999999999996,"15min":0.97199999999999998},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.3140000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":0.94499999999999995}]}]},{"osd":3,"up_from":12,"seq":51539607590,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":97928,"kb_used_data":66812,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94273912,"statfs":{"total":96636764160,"available":96536485888,"internally_reserved":0,"allocated":68415488,"data_stored":68243735,"data_compressed":36109,"data_compressed_allocated":49152,"data_compressed_original":638976,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.93200000000000005,"15min":0.95299999999999996},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.264,"5min":2.665,"15min":2.665},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.86799999999999999,"15min":0.878},"min":{"1min":0.55700000000000005,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.0649999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.57599999999999996}]},{"osd":1,"last update":"Wed Sep 18 04:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.86299999999999999,"15min":0.88300000000000001},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.121,"5min":2.4510000000000001,"15min":2.4510000000000001},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.77200000000000002,"15min":0.78600000000000003},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.046,"5min":1.077,"15min":1.077},"last":0.78000000000000003}]},{"osd":2,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":2.577,"5min":2.577,"15min":2.577},"last":0.95299999999999996},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":1.0629999999999999}]},{"osd":4,"last update":"Wed Sep 18 04:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.72499999999999998,"15min":0.72699999999999998},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.97399999999999998,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.751,"15min":0.749},"min":{"1min":0.49199999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.0209999999999999,"5min":1.044,"15min":1.044},"last":0.68400000000000005}]},{"osd":5,"last update":"Wed Sep 18 04:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.73599999999999999,"15min":0.73899999999999999},"min":{"1min":0.56399999999999995,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.069,"5min":1.173,"15min":1.173},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.877,"5min":0.86899999999999999,"15min":0.86799999999999999},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.087,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":1.0349999999999999}]}]},{"osd":2,"up_from":86,"seq":369367187463,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79248,"kb_used_data":50620,"kb_used_omap":9,"kb_used_meta":28598,"kb_avail":94292592,"statfs":{"total":96636764160,"available":96555614208,"internally_reserved":0,"allocated":51834880,"data_stored":61397815,"data_compressed":205860,"data_compressed_allocated":274432,"data_compressed_original":4300800,"omap_allocated":10134,"internal_metadata":29284458},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80800000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86799999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90500000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.359}]}]},{"osd":1,"up_from":12,"seq":51539607590,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":105148,"kb_used_data":69928,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94266692,"statfs":{"total":96636764160,"available":96529092608,"internally_reserved":0,"allocated":71606272,"data_stored":81337200,"data_compressed":207452,"data_compressed_allocated":147456,"data_compressed_original":4349952,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:39:27 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.60299999999999998,"15min":0.59499999999999997},"min":{"1min":0.433,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.746},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.625,"15min":0.627},"min":{"1min":0.42599999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.871,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":1.002}]},{"osd":2,"last update":"Wed Sep 18 04:39:27 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.66200000000000003,"15min":0.66300000000000003},"min":{"1min":0.46700000000000003,"5min":0.436,"15min":0.436},"max":{"1min":1.05,"5min":1.327,"15min":1.327},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.66400000000000003,"15min":0.66700000000000004},"min":{"1min":0.43099999999999999,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.872,"5min":1.002,"15min":1.002},"last":0.59699999999999998}]},{"osd":3,"last update":"Wed Sep 18 04:39:27 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.79900000000000004,"15min":0.80100000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.22,"5min":1.22,"15min":1.22},"last":1.0329999999999999},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.86799999999999999,"15min":0.871},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.97199999999999998}]},{"osd":4,"last update":"Wed Sep 18 04:39:27 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"min":{"1min":0.63600000000000001,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.018,"5min":1.236,"15min":1.236},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.93000000000000005,"15min":0.93100000000000005},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.268,"5min":1.268,"15min":1.268},"last":1.0720000000000001}]},{"osd":5,"last update":"Wed Sep 18 04:39:27 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.90700000000000003,"15min":0.91100000000000003},"min":{"1min":0.71899999999999997,"5min":0.628,"15min":0.628},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":1.105},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.159,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":1.1879999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":46145536,"data_stored":46137460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":70389760,"data_stored":80863403,"data_compressed":207452,"data_compressed_allocated":147456,"data_compressed_original":4349952,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":50626560,"data_stored":60924072,"data_compressed":205860,"data_compressed_allocated":274432,"data_compressed_original":4300800,"omap_allocated":3627,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":67129344,"data_stored":67174530,"data_compressed":3089,"data_compressed_allocated":4096,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":43073536,"data_stored":43041129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":16,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":54,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1835,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":16,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":16,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":54,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T04:40:16.307 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T04:40:16.355 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:40:16.351+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:40:16.401 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:16.401 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:16.411 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:40:16.431 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:40:16.427+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:40:16.528 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:16.528 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:16.531 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:40:16.528+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:40:16.538 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:40:16.618 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:40:16.620 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-18T04:40:16.633 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":185,"stamp":"2024-09-18T04:40:15.892708+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":147465357,"num_objects":80,"num_object_clones":19,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":13,"num_read":7277,"num_read_kb":179969,"num_write":977,"num_write_kb":273266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":39960608,"num_keys_recovered":25,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":952,"ondisk_log_size":952,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":470356,"kb_used_data":277952,"kb_used_omap":17,"kb_used_meta":192302,"kb_avail":565760684,"statfs":{"total":579820584960,"available":579338940416,"internally_reserved":0,"allocated":284622848,"data_stored":302102808,"data_compressed":482441,"data_compressed_allocated":516096,"data_compressed_original":9863168,"omap_allocated":18101,"internal_metadata":196917579},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-78,"num_objects":-1,"num_object_clones":0,"num_object_copies":-2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":641,"num_read_kb":1709,"num_write":17,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.039762"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":247,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-18T04:40:13.880739+0000","last_change":"2024-09-18T04:37:27.423069+0000","last_active":"2024-09-18T04:40:13.880739+0000","last_peered":"2024-09-18T04:40:13.880739+0000","last_clean":"2024-09-18T04:40:13.880739+0000","last_became_active":"2024-09-18T04:37:27.422350+0000","last_became_peered":"2024-09-18T04:37:27.422350+0000","last_unstale":"2024-09-18T04:40:13.880739+0000","last_undegraded":"2024-09-18T04:40:13.880739+0000","last_fullsized":"2024-09-18T04:40:13.880739+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:26.406292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:26.406292+0000","last_clean_scrub_stamp":"2024-09-18T04:37:26.406292+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:31:45.852149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"99'107","reported_seq":580,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-18T04:40:10.823393+0000","last_change":"2024-09-18T04:39:46.795376+0000","last_active":"2024-09-18T04:40:10.823393+0000","last_peered":"2024-09-18T04:40:10.823393+0000","last_clean":"2024-09-18T04:40:10.823393+0000","last_became_active":"2024-09-18T04:39:46.718066+0000","last_became_peered":"2024-09-18T04:39:46.718066+0000","last_unstale":"2024-09-18T04:40:10.823393+0000","last_undegraded":"2024-09-18T04:40:10.823393+0000","last_fullsized":"2024-09-18T04:40:10.823393+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T07:09:22.599338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018293420000000001,"stat_sum":{"num_bytes":10535160,"num_objects":11,"num_object_clones":2,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":550,"num_read_kb":17110,"num_write":115,"num_write_kb":25705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":9486336,"num_keys_recovered":9,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"69'3","reported_seq":71,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-18T04:40:13.143324+0000","last_change":"2024-09-18T04:38:46.966799+0000","last_active":"2024-09-18T04:40:13.143324+0000","last_peered":"2024-09-18T04:40:13.143324+0000","last_clean":"2024-09-18T04:40:13.143324+0000","last_became_active":"2024-09-18T04:38:44.953092+0000","last_became_peered":"2024-09-18T04:38:44.953092+0000","last_unstale":"2024-09-18T04:40:13.143324+0000","last_undegraded":"2024-09-18T04:40:13.143324+0000","last_fullsized":"2024-09-18T04:40:13.143324+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_clean_scrub_stamp":"2024-09-18T04:38:43.928423+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:04:46.888906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010424099999999999,"stat_sum":{"num_bytes":54,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"104'215","reported_seq":1321,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-18T04:40:14.883246+0000","last_change":"2024-09-18T04:37:36.484950+0000","last_active":"2024-09-18T04:40:14.883246+0000","last_peered":"2024-09-18T04:40:14.883246+0000","last_clean":"2024-09-18T04:40:14.883246+0000","last_became_active":"2024-09-18T04:37:34.470841+0000","last_became_peered":"2024-09-18T04:37:34.470841+0000","last_unstale":"2024-09-18T04:40:14.883246+0000","last_undegraded":"2024-09-18T04:40:14.883246+0000","last_fullsized":"2024-09-18T04:40:14.883246+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":215,"log_dups_size":0,"ondisk_log_size":215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:16:34.528783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090513900000000001,"stat_sum":{"num_bytes":25165900,"num_objects":11,"num_object_clones":3,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":2,"num_read":1718,"num_read_kb":39765,"num_write":314,"num_write_kb":45334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":88,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-18T04:40:09.850372+0000","last_change":"2024-09-18T04:40:05.832504+0000","last_active":"2024-09-18T04:40:09.850372+0000","last_peered":"2024-09-18T04:40:09.850372+0000","last_clean":"2024-09-18T04:40:09.850372+0000","last_became_active":"2024-09-18T04:40:05.832216+0000","last_became_peered":"2024-09-18T04:40:05.832216+0000","last_unstale":"2024-09-18T04:40:09.850372+0000","last_undegraded":"2024-09-18T04:40:09.850372+0000","last_fullsized":"2024-09-18T04:40:09.850372+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_clean_scrub_stamp":"2024-09-18T04:38:43.928423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:03:41.967712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075989799999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"103'260","reported_seq":1983,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-18T04:40:14.714035+0000","last_change":"2024-09-18T04:39:46.746657+0000","last_active":"2024-09-18T04:40:14.714035+0000","last_peered":"2024-09-18T04:40:14.714035+0000","last_clean":"2024-09-18T04:40:14.714035+0000","last_became_active":"2024-09-18T04:39:46.716998+0000","last_became_peered":"2024-09-18T04:39:46.716998+0000","last_unstale":"2024-09-18T04:40:14.714035+0000","last_undegraded":"2024-09-18T04:40:14.714035+0000","last_fullsized":"2024-09-18T04:40:14.714035+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":260,"log_dups_size":0,"ondisk_log_size":260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:37:53.810382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18898982,"num_objects":9,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3943,"num_read_kb":29480,"num_write":162,"num_write_kb":27759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":5308416,"num_keys_recovered":13,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":73,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-18T04:40:13.103431+0000","last_change":"2024-09-18T04:39:59.780545+0000","last_active":"2024-09-18T04:40:13.103431+0000","last_peered":"2024-09-18T04:40:13.103431+0000","last_clean":"2024-09-18T04:40:13.103431+0000","last_became_active":"2024-09-18T04:39:59.780106+0000","last_became_peered":"2024-09-18T04:39:59.780106+0000","last_unstale":"2024-09-18T04:40:13.103431+0000","last_undegraded":"2024-09-18T04:40:13.103431+0000","last_fullsized":"2024-09-18T04:40:13.103431+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_clean_scrub_stamp":"2024-09-18T04:38:43.928423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T08:02:28.727480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011574549999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"20'2","reported_seq":201,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-18T04:40:10.675453+0000","last_change":"2024-09-18T04:37:36.490461+0000","last_active":"2024-09-18T04:40:10.675453+0000","last_peered":"2024-09-18T04:40:10.675453+0000","last_clean":"2024-09-18T04:40:10.675453+0000","last_became_active":"2024-09-18T04:37:34.473088+0000","last_became_peered":"2024-09-18T04:37:34.473088+0000","last_unstale":"2024-09-18T04:40:10.675453+0000","last_undegraded":"2024-09-18T04:40:10.675453+0000","last_fullsized":"2024-09-18T04:40:10.675453+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:17:57.352397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001879932,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":105,"num_read_kb":105,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"69'2","reported_seq":18,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-18T04:40:07.449849+0000","last_change":"2024-09-18T04:40:05.842027+0000","last_active":"2024-09-18T04:40:07.449849+0000","last_peered":"2024-09-18T04:40:07.449849+0000","last_clean":"2024-09-18T04:40:07.449849+0000","last_became_active":"2024-09-18T04:40:05.831890+0000","last_became_peered":"2024-09-18T04:40:05.831890+0000","last_unstale":"2024-09-18T04:40:07.449849+0000","last_undegraded":"2024-09-18T04:40:07.449849+0000","last_fullsized":"2024-09-18T04:40:07.449849+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_clean_scrub_stamp":"2024-09-18T04:38:43.928423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:27:26.157495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010091900000000001,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":32,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"69'3","reported_seq":85,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-18T04:40:06.840655+0000","last_change":"2024-09-18T04:39:46.720085+0000","last_active":"2024-09-18T04:40:06.840655+0000","last_peered":"2024-09-18T04:40:06.840655+0000","last_clean":"2024-09-18T04:40:06.840655+0000","last_became_active":"2024-09-18T04:39:46.719365+0000","last_became_peered":"2024-09-18T04:39:46.719365+0000","last_unstale":"2024-09-18T04:40:06.840655+0000","last_undegraded":"2024-09-18T04:40:06.840655+0000","last_fullsized":"2024-09-18T04:40:06.840655+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_clean_scrub_stamp":"2024-09-18T04:38:43.928423+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:38:16.629282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065509499999999996,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"98'46","reported_seq":88,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-18T04:40:10.813068+0000","last_change":"2024-09-18T04:38:59.528402+0000","last_active":"2024-09-18T04:40:10.813068+0000","last_peered":"2024-09-18T04:40:10.813068+0000","last_clean":"2024-09-18T04:40:10.813068+0000","last_became_active":"2024-09-18T04:38:59.462279+0000","last_became_peered":"2024-09-18T04:38:59.462279+0000","last_unstale":"2024-09-18T04:40:10.813068+0000","last_undegraded":"2024-09-18T04:40:10.813068+0000","last_fullsized":"2024-09-18T04:40:10.813068+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:32:58.894411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081514499999999998,"stat_sum":{"num_bytes":20971598,"num_objects":10,"num_object_clones":2,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":2,"num_read":91,"num_read_kb":16656,"num_write":44,"num_write_kb":36867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":12582912,"num_keys_recovered":3,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"69'1","reported_seq":79,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-18T04:40:13.143625+0000","last_change":"2024-09-18T04:39:46.718266+0000","last_active":"2024-09-18T04:40:13.143625+0000","last_peered":"2024-09-18T04:40:13.143625+0000","last_clean":"2024-09-18T04:40:13.143625+0000","last_became_active":"2024-09-18T04:39:46.717950+0000","last_became_peered":"2024-09-18T04:39:46.717950+0000","last_unstale":"2024-09-18T04:40:13.143625+0000","last_undegraded":"2024-09-18T04:40:13.143625+0000","last_fullsized":"2024-09-18T04:40:13.143625+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_clean_scrub_stamp":"2024-09-18T04:38:43.928423+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:17:31.749220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016757619999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"79'64","reported_seq":393,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-18T04:40:13.103529+0000","last_change":"2024-09-18T04:39:46.817083+0000","last_active":"2024-09-18T04:40:13.103529+0000","last_peered":"2024-09-18T04:40:13.103529+0000","last_clean":"2024-09-18T04:40:13.103529+0000","last_became_active":"2024-09-18T04:39:46.717423+0000","last_became_peered":"2024-09-18T04:39:46.717423+0000","last_unstale":"2024-09-18T04:40:13.103529+0000","last_undegraded":"2024-09-18T04:40:13.103529+0000","last_fullsized":"2024-09-18T04:40:13.103529+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T09:52:29.555572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165862,"num_objects":10,"num_object_clones":3,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":3,"num_read":244,"num_read_kb":31404,"num_write":61,"num_write_kb":49153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":48,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-18T04:40:13.103561+0000","last_change":"2024-09-18T04:39:46.717470+0000","last_active":"2024-09-18T04:40:13.103561+0000","last_peered":"2024-09-18T04:40:13.103561+0000","last_clean":"2024-09-18T04:40:13.103561+0000","last_became_active":"2024-09-18T04:39:46.717099+0000","last_became_peered":"2024-09-18T04:39:46.717099+0000","last_unstale":"2024-09-18T04:40:13.103561+0000","last_undegraded":"2024-09-18T04:40:13.103561+0000","last_fullsized":"2024-09-18T04:40:13.103561+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_clean_scrub_stamp":"2024-09-18T04:38:43.928423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:08:02.208525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"103'70","reported_seq":293,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-18T04:40:13.880048+0000","last_change":"2024-09-18T04:37:36.494193+0000","last_active":"2024-09-18T04:40:13.880048+0000","last_peered":"2024-09-18T04:40:13.880048+0000","last_clean":"2024-09-18T04:40:13.880048+0000","last_became_active":"2024-09-18T04:37:34.472940+0000","last_became_peered":"2024-09-18T04:37:34.472940+0000","last_unstale":"2024-09-18T04:40:13.880048+0000","last_undegraded":"2024-09-18T04:40:13.880048+0000","last_fullsized":"2024-09-18T04:40:13.880048+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:18:29.266815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070553900000000004,"stat_sum":{"num_bytes":33554470,"num_objects":13,"num_object_clones":3,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":3,"num_read":131,"num_read_kb":28707,"num_write":67,"num_write_kb":57348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":75,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-18T04:40:09.850785+0000","last_change":"2024-09-18T04:39:18.577075+0000","last_active":"2024-09-18T04:40:09.850785+0000","last_peered":"2024-09-18T04:40:09.850785+0000","last_clean":"2024-09-18T04:40:09.850785+0000","last_became_active":"2024-09-18T04:39:18.576593+0000","last_became_peered":"2024-09-18T04:39:18.576593+0000","last_unstale":"2024-09-18T04:40:09.850785+0000","last_undegraded":"2024-09-18T04:40:09.850785+0000","last_fullsized":"2024-09-18T04:40:09.850785+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:38:43.928423+0000","last_clean_scrub_stamp":"2024-09-18T04:38:43.928423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:27:02.258820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058611500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"79'62","reported_seq":340,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-18T04:40:13.880675+0000","last_change":"2024-09-18T04:38:05.754820+0000","last_active":"2024-09-18T04:40:13.880675+0000","last_peered":"2024-09-18T04:40:13.880675+0000","last_clean":"2024-09-18T04:40:13.880675+0000","last_became_active":"2024-09-18T04:38:05.753962+0000","last_became_peered":"2024-09-18T04:38:05.753962+0000","last_unstale":"2024-09-18T04:40:13.880675+0000","last_undegraded":"2024-09-18T04:40:13.880675+0000","last_fullsized":"2024-09-18T04:40:13.880675+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:53:52.224098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047343899999999998,"stat_sum":{"num_bytes":12582928,"num_objects":7,"num_object_clones":2,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":2,"num_read":344,"num_read_kb":16613,"num_write":47,"num_write_kb":28676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":70,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":32,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1835,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":146874919,"num_objects":72,"num_object_clones":19,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":13,"num_read":7126,"num_read_kb":179840,"num_write":812,"num_write_kb":270844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":39960576,"num_keys_recovered":25,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":277364736,"data_stored":298140594,"data_compressed":416401,"data_compressed_allocated":425984,"data_compressed_original":8716288,"omap_allocated":3627,"internal_metadata":0},"log_size":826,"ondisk_log_size":826,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":66040,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":13,"seq":55834574885,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74304,"kb_used_data":43192,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94297536,"statfs":{"total":96636764160,"available":96560676864,"internally_reserved":0,"allocated":44228608,"data_stored":43489464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:39:28 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.85099999999999998,"15min":0.86499999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.0469999999999999,"5min":1.5249999999999999,"15min":1.5249999999999999},"last":0.879},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.79700000000000004,"15min":0.80800000000000005},"min":{"1min":0.53500000000000003,"5min":0.502,"15min":0.502},"max":{"1min":0.996,"5min":1.369,"15min":1.369},"last":0.76400000000000001}]},{"osd":1,"last update":"Wed Sep 18 04:39:28 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.83999999999999997,"15min":0.85199999999999998},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.091,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.79500000000000004,"15min":0.79900000000000004},"min":{"1min":0.59999999999999998,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.52700000000000002}]},{"osd":2,"last update":"Wed Sep 18 04:38:26 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.4930000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.85199999999999998}]},{"osd":3,"last update":"Wed Sep 18 04:39:28 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.75700000000000001,"15min":0.77000000000000002},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.0169999999999999,"5min":1.22,"15min":1.22},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.78900000000000003,"15min":0.79700000000000004},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.78900000000000003}]},{"osd":4,"last update":"Wed Sep 18 04:39:28 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.77800000000000002,"15min":0.78500000000000003},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.998,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.751,"5min":0.82499999999999996,"15min":0.83699999999999997},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.075,"5min":1.121,"15min":1.121},"last":0.81999999999999995}]}]},{"osd":4,"up_from":13,"seq":55834574885,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32252,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339588,"statfs":{"total":96636764160,"available":96603738112,"internally_reserved":0,"allocated":1163264,"data_stored":453383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:39:31 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.76600000000000001,"15min":0.77200000000000002},"min":{"1min":0.52200000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.044,"5min":1.175,"15min":1.175},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.78500000000000003,"15min":0.79500000000000004},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.1060000000000001,"5min":1.258,"15min":1.258},"last":0.66300000000000003}]},{"osd":1,"last update":"Wed Sep 18 04:39:31 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.72199999999999998,"15min":0.72799999999999998},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.95899999999999996,"5min":1.002,"15min":1.002},"last":0.63},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.875,"15min":0.88400000000000001},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":0.93500000000000005}]},{"osd":2,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.751},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.619,"5min":1.619,"15min":1.619},"last":0.81299999999999994}]},{"osd":3,"last update":"Wed Sep 18 04:39:31 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.81299999999999994,"15min":0.82299999999999995},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0129999999999999,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":2.302,"5min":2.302,"15min":2.302},"last":0.70699999999999996}]},{"osd":5,"last update":"Wed Sep 18 04:39:31 2024","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.88200000000000001,"15min":0.88700000000000001},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.365,"5min":1.365,"15min":1.365},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.89000000000000001,"15min":0.88800000000000001},"min":{"1min":0.42599999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":2.0979999999999999,"5min":2.0979999999999999,"15min":2.0979999999999999},"last":0.73799999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574886,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81476,"kb_used_data":46264,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94290364,"statfs":{"total":96636764160,"available":96553332736,"internally_reserved":0,"allocated":47374336,"data_stored":47181211,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 04:39:28 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.59999999999999998,"15min":0.60899999999999999},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.78000000000000003,"5min":1.113,"15min":1.113},"last":0.621},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.63,"15min":0.64200000000000002},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.89900000000000002,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.56899999999999995}]},{"osd":2,"last update":"Wed Sep 18 04:38:28 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.8180000000000001,"5min":1.8180000000000001,"15min":1.8180000000000001},"last":0.41999999999999998}]},{"osd":3,"last update":"Wed Sep 18 04:39:28 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.81499999999999995,"15min":0.81799999999999995},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":0.99399999999999999,"5min":1.169,"15min":1.169},"last":0.97599999999999998},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.81899999999999995,"15min":0.82599999999999996},"min":{"1min":0.49199999999999999,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.97699999999999998,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.66700000000000004}]},{"osd":4,"last update":"Wed Sep 18 04:39:28 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.84599999999999997,"15min":0.85399999999999998},"min":{"1min":0.61699999999999999,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.0600000000000001,"5min":1.073,"15min":1.073},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.89600000000000002,"15min":0.90800000000000003},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.879}]},{"osd":5,"last update":"Wed Sep 18 04:39:28 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.91600000000000004,"15min":0.93100000000000005},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.1479999999999999,"5min":1.8919999999999999,"15min":1.8919999999999999},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.95399999999999996,"15min":0.97199999999999998},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.3140000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":0.94499999999999995}]}]},{"osd":3,"up_from":12,"seq":51539607590,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":97928,"kb_used_data":66812,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94273912,"statfs":{"total":96636764160,"available":96536485888,"internally_reserved":0,"allocated":68415488,"data_stored":68243735,"data_compressed":36109,"data_compressed_allocated":49152,"data_compressed_original":638976,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.93200000000000005,"15min":0.95299999999999996},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.264,"5min":2.665,"15min":2.665},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.86799999999999999,"15min":0.878},"min":{"1min":0.55700000000000005,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.0649999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.57599999999999996}]},{"osd":1,"last update":"Wed Sep 18 04:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.86299999999999999,"15min":0.88300000000000001},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.121,"5min":2.4510000000000001,"15min":2.4510000000000001},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.77200000000000002,"15min":0.78600000000000003},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.046,"5min":1.077,"15min":1.077},"last":0.78000000000000003}]},{"osd":2,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":2.577,"5min":2.577,"15min":2.577},"last":0.95299999999999996},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":1.0629999999999999}]},{"osd":4,"last update":"Wed Sep 18 04:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.72499999999999998,"15min":0.72699999999999998},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.97399999999999998,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.751,"15min":0.749},"min":{"1min":0.49199999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.0209999999999999,"5min":1.044,"15min":1.044},"last":0.68400000000000005}]},{"osd":5,"last update":"Wed Sep 18 04:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.73599999999999999,"15min":0.73899999999999999},"min":{"1min":0.56399999999999995,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.069,"5min":1.173,"15min":1.173},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.877,"5min":0.86899999999999999,"15min":0.86799999999999999},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.087,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":1.0349999999999999}]}]},{"osd":2,"up_from":86,"seq":369367187463,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79248,"kb_used_data":50620,"kb_used_omap":9,"kb_used_meta":28598,"kb_avail":94292592,"statfs":{"total":96636764160,"available":96555614208,"internally_reserved":0,"allocated":51834880,"data_stored":61397815,"data_compressed":205860,"data_compressed_allocated":274432,"data_compressed_original":4300800,"omap_allocated":10134,"internal_metadata":29284458},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80800000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86799999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90500000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.359}]}]},{"osd":1,"up_from":12,"seq":51539607590,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":105148,"kb_used_data":69928,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94266692,"statfs":{"total":96636764160,"available":96529092608,"internally_reserved":0,"allocated":71606272,"data_stored":81337200,"data_compressed":207452,"data_compressed_allocated":147456,"data_compressed_original":4349952,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:39:27 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.60299999999999998,"15min":0.59499999999999997},"min":{"1min":0.433,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.746},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.625,"15min":0.627},"min":{"1min":0.42599999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.871,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":1.002}]},{"osd":2,"last update":"Wed Sep 18 04:39:27 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.66200000000000003,"15min":0.66300000000000003},"min":{"1min":0.46700000000000003,"5min":0.436,"15min":0.436},"max":{"1min":1.05,"5min":1.327,"15min":1.327},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.66400000000000003,"15min":0.66700000000000004},"min":{"1min":0.43099999999999999,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.872,"5min":1.002,"15min":1.002},"last":0.59699999999999998}]},{"osd":3,"last update":"Wed Sep 18 04:39:27 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.79900000000000004,"15min":0.80100000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.22,"5min":1.22,"15min":1.22},"last":1.0329999999999999},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.86799999999999999,"15min":0.871},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.97199999999999998}]},{"osd":4,"last update":"Wed Sep 18 04:39:27 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"min":{"1min":0.63600000000000001,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.018,"5min":1.236,"15min":1.236},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.93000000000000005,"15min":0.93100000000000005},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.268,"5min":1.268,"15min":1.268},"last":1.0720000000000001}]},{"osd":5,"last update":"Wed Sep 18 04:39:27 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.90700000000000003,"15min":0.91100000000000003},"min":{"1min":0.71899999999999997,"5min":0.628,"15min":0.628},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":1.105},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.159,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":1.1879999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":46145536,"data_stored":46137460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":70389760,"data_stored":80863403,"data_compressed":207452,"data_compressed_allocated":147456,"data_compressed_original":4349952,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":50626560,"data_stored":60924072,"data_compressed":205860,"data_compressed_allocated":274432,"data_compressed_original":4300800,"omap_allocated":3627,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":67129344,"data_stored":67174530,"data_compressed":3089,"data_compressed_allocated":4096,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":43073536,"data_stored":43041129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":16,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":54,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1835,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":16,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":16,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":54,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T04:40:16.635 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-18T04:40:16.635 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T04:40:16.635 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-09-18T04:40:16.635 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-18T04:40:16.635 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-18T04:40:16.636 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config set filestore_inject_stall 3 2024-09-18T04:40:16.636 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:40:16.628+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:40:16.654 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:16.654 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:16.663 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:40:16.694 INFO:tasks.workunit.client.0.smithi115.stderr:+ EXPECTED=7 2024-09-18T04:40:16.695 INFO:tasks.workunit.client.0.smithi115.stderr:++ _sudo rbd device --device-type nbd --rbd-op-threads 7 map rbd/testrbdnbd17342 2024-09-18T04:40:16.695 INFO:tasks.workunit.client.0.smithi115.stderr:++ local cmd 2024-09-18T04:40:16.696 INFO:tasks.workunit.client.0.smithi115.stderr:+++ id -u 2024-09-18T04:40:16.698 INFO:tasks.workunit.client.0.smithi115.stderr:++ '[' 1000 -eq 0 ']' 2024-09-18T04:40:16.698 INFO:tasks.workunit.client.0.smithi115.stderr:+++ which rbd 2024-09-18T04:40:16.701 INFO:tasks.workunit.client.0.smithi115.stderr:++ cmd=/usr/bin/rbd 2024-09-18T04:40:16.701 INFO:tasks.workunit.client.0.smithi115.stderr:++ shift 2024-09-18T04:40:16.701 INFO:tasks.workunit.client.0.smithi115.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd --rbd-op-threads 7 map rbd/testrbdnbd17342 2024-09-18T04:40:16.733 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:40:16.727+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:40:16.741 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:40:16.741 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "" 2024-09-18T04:40:16.741 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:40:16.770 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:16.770 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:16.780 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:40:16.834 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:40:16.832+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:40:16.933 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:16.933 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:16.935 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:40:16.932+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:40:16.943 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:40:16.963 INFO:tasks.workunit.client.0.smithi115.stderr:+ DEV=/dev/nbd1 2024-09-18T04:40:16.963 INFO:tasks.workunit.client.0.smithi115.stderr:+ get_pid rbd 2024-09-18T04:40:16.963 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pool=rbd 2024-09-18T04:40:16.963 INFO:tasks.workunit.client.0.smithi115.stderr:+ local ns= 2024-09-18T04:40:16.964 INFO:tasks.workunit.client.0.smithi115.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd17342'\''][device='\''/dev/nbd1'\'']/id' 2024-09-18T04:40:16.964 INFO:tasks.workunit.client.0.smithi115.stderr:++ rbd device --device-type nbd --format xml list 2024-09-18T04:40:17.036 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:40:17.032+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:40:17.045 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:17.045 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:17.055 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:40:17.069 INFO:tasks.workunit.client.0.smithi115.stderr:+ PID=20805 2024-09-18T04:40:17.069 INFO:tasks.workunit.client.0.smithi115.stderr:+ test -n 20805 2024-09-18T04:40:17.069 INFO:tasks.workunit.client.0.smithi115.stderr:+ ps -p 20805 -C rbd-nbd 2024-09-18T04:40:17.078 INFO:tasks.workunit.client.0.smithi115.stdout: PID TTY TIME CMD 2024-09-18T04:40:17.078 INFO:tasks.workunit.client.0.smithi115.stdout: 20805 ? 00:00:00 rbd-nbd 2024-09-18T04:40:17.079 INFO:tasks.workunit.client.0.smithi115.stderr:++ ps -p 20805 -T 2024-09-18T04:40:17.079 INFO:tasks.workunit.client.0.smithi115.stderr:++ grep -c io_context_pool 2024-09-18T04:40:17.089 INFO:tasks.workunit.client.0.smithi115.stderr:+ ACTUAL=7 2024-09-18T04:40:17.089 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 7 -eq 7 ']' 2024-09-18T04:40:17.089 INFO:tasks.workunit.client.0.smithi115.stderr:+ unmap_device /dev/nbd1 20805 2024-09-18T04:40:17.090 INFO:tasks.workunit.client.0.smithi115.stderr:+ local args=/dev/nbd1 2024-09-18T04:40:17.090 INFO:tasks.workunit.client.0.smithi115.stderr:+ local pid=20805 2024-09-18T04:40:17.090 INFO:tasks.workunit.client.0.smithi115.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd1 2024-09-18T04:40:17.090 INFO:tasks.workunit.client.0.smithi115.stderr:+ local cmd 2024-09-18T04:40:17.090 INFO:tasks.workunit.client.0.smithi115.stderr:++ id -u 2024-09-18T04:40:17.090 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' 1000 -eq 0 ']' 2024-09-18T04:40:17.091 INFO:tasks.workunit.client.0.smithi115.stderr:++ which rbd 2024-09-18T04:40:17.093 INFO:tasks.workunit.client.0.smithi115.stderr:+ cmd=/usr/bin/rbd 2024-09-18T04:40:17.093 INFO:tasks.workunit.client.0.smithi115.stderr:+ shift 2024-09-18T04:40:17.094 INFO:tasks.workunit.client.0.smithi115.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd1 2024-09-18T04:40:17.136 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:40:17.132+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:40:17.166 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:17.166 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:17.176 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:40:17.203 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false grep '^20805\b' 2024-09-18T04:40:17.203 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep '^20805\b' 2024-09-18T04:40:17.203 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd device --device-type nbd list 2024-09-18T04:40:17.237 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:40:17.232+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:40:17.278 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:40:17.278 INFO:tasks.workunit.client.0.smithi115.stderr:+ ps -C rbd-nbd 2024-09-18T04:40:17.279 INFO:tasks.workunit.client.0.smithi115.stderr:+ expect_false grep '^ *20805\b' 2024-09-18T04:40:17.279 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep '^ *20805\b' 2024-09-18T04:40:17.279 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:17.279 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:17.285 INFO:tasks.workunit.client.0.smithi115.stderr:+ return 0 2024-09-18T04:40:17.286 INFO:tasks.workunit.client.0.smithi115.stderr:+ sleep 0.5 2024-09-18T04:40:17.290 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:40:17.338 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:40:17.332+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:40:17.397 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T04:40:17.408 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:17.408 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:17.418 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:40:17.438 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:40:17.436+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:40:17.528 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:17.529 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:17.539 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:40:17.539 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:40:17.536+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:40:17.546 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:40:17.565 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:40:17.583 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:40:17.605 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:40:17.627 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:40:17.640 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:40:17.636+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:40:17.650 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:40:17.652 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:40:17.652 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:40:17.652 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:40:17.653 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:40:17.653 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:40:17.653 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:40:17.696 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:40:17.696 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:40:17.696 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:40:17.696 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:40:17.696 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.696 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:40:17.696 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:40:17.696 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 169.90347003299999, 2024-09-18T04:40:17.696 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017399218000000001, 2024-09-18T04:40:17.696 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:17.697 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:17.697 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:17.697 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:40:17.697 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:40:17.697 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 114 2024-09-18T04:40:17.697 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.697 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:17.697 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.697 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:17.697 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:40:17.697 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:17.697 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.697 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.697 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:17.698 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:40:17.698 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:17.698 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.698 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.698 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:17.698 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786900+0000", 2024-09-18T04:40:17.698 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:40:17.698 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.698 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.698 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:17.698 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786908+0000", 2024-09-18T04:40:17.698 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8199999999999997e-06 2024-09-18T04:40:17.698 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.698 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.698 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:17.698 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786925+0000", 2024-09-18T04:40:17.699 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6849000000000001e-05 2024-09-18T04:40:17.699 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.699 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.699 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:17.699 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786934+0000", 2024-09-18T04:40:17.699 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6299999999999993e-06 2024-09-18T04:40:17.699 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.699 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.699 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:17.699 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800926+0000", 2024-09-18T04:40:17.699 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013991632 2024-09-18T04:40:17.699 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.699 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.699 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:17.700 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801060+0000", 2024-09-18T04:40:17.700 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013378199999999999 2024-09-18T04:40:17.700 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.700 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.700 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:40:17.700 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801166+0000", 2024-09-18T04:40:17.700 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010607099999999999 2024-09-18T04:40:17.700 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.700 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.700 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:17.700 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803026+0000", 2024-09-18T04:40:17.700 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001860469 2024-09-18T04:40:17.700 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.700 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.701 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:17.701 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804245+0000", 2024-09-18T04:40:17.701 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0012185900000000001 2024-09-18T04:40:17.701 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.701 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.701 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:17.701 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804278+0000", 2024-09-18T04:40:17.701 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3272000000000003e-05 2024-09-18T04:40:17.701 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.701 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.701 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:17.701 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804300+0000", 2024-09-18T04:40:17.701 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2266e-05 2024-09-18T04:40:17.701 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:17.702 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:17.702 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:17.702 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.702 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.702 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:40:17.702 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:40:17.702 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 169.902557135, 2024-09-18T04:40:17.702 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017521662, 2024-09-18T04:40:17.702 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:17.702 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:17.702 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:17.702 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:40:17.702 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:40:17.702 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 116 2024-09-18T04:40:17.702 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.703 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:17.703 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.703 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:17.703 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:40:17.703 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:17.703 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.703 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.703 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:17.703 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:40:17.703 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:17.703 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.703 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.703 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:17.703 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787813+0000", 2024-09-18T04:40:17.704 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:17.704 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.704 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.704 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:17.704 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787824+0000", 2024-09-18T04:40:17.704 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1157e-05 2024-09-18T04:40:17.704 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.704 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.704 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:17.704 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787840+0000", 2024-09-18T04:40:17.704 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6269e-05 2024-09-18T04:40:17.704 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.704 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.704 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:17.705 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787853+0000", 2024-09-18T04:40:17.705 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3156000000000001e-05 2024-09-18T04:40:17.705 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.705 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.705 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:17.705 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803110+0000", 2024-09-18T04:40:17.705 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015256736 2024-09-18T04:40:17.705 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.705 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.705 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:17.705 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803197+0000", 2024-09-18T04:40:17.705 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7195999999999997e-05 2024-09-18T04:40:17.705 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.705 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.706 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:40:17.706 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803308+0000", 2024-09-18T04:40:17.706 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111005 2024-09-18T04:40:17.706 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.706 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.706 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:17.706 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804316+0000", 2024-09-18T04:40:17.706 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010073949999999999 2024-09-18T04:40:17.706 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.706 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.706 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:17.706 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805232+0000", 2024-09-18T04:40:17.706 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091636300000000001 2024-09-18T04:40:17.706 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.707 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.707 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:17.707 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805277+0000", 2024-09-18T04:40:17.707 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4864000000000003e-05 2024-09-18T04:40:17.707 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.707 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.707 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:17.707 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805336+0000", 2024-09-18T04:40:17.707 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8968000000000002e-05 2024-09-18T04:40:17.707 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:17.707 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:17.707 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:17.707 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.707 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.707 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:40:17.708 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:40:17.708 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 169.90250508, 2024-09-18T04:40:17.708 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017983453, 2024-09-18T04:40:17.708 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:17.708 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:17.708 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:17.708 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:40:17.708 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:40:17.708 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 117 2024-09-18T04:40:17.708 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.708 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:17.708 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.708 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:17.708 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:40:17.708 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:17.708 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.709 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.709 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:17.709 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:40:17.709 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:17.709 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.709 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.709 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:17.709 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787865+0000", 2024-09-18T04:40:17.709 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:40:17.709 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.709 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.709 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:17.709 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787889+0000", 2024-09-18T04:40:17.709 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4244999999999999e-05 2024-09-18T04:40:17.710 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.710 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.710 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:17.710 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787905+0000", 2024-09-18T04:40:17.710 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5739999999999998e-05 2024-09-18T04:40:17.710 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.710 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.710 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:17.710 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787918+0000", 2024-09-18T04:40:17.710 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.252e-05 2024-09-18T04:40:17.710 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.710 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.710 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:17.710 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803693+0000", 2024-09-18T04:40:17.711 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015774872999999998 2024-09-18T04:40:17.711 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.711 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.711 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:17.711 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803775+0000", 2024-09-18T04:40:17.711 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2761000000000003e-05 2024-09-18T04:40:17.711 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.711 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.711 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:40:17.711 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803887+0000", 2024-09-18T04:40:17.711 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111708 2024-09-18T04:40:17.711 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.711 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.711 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:17.711 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804823+0000", 2024-09-18T04:40:17.711 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00093625100000000003 2024-09-18T04:40:17.712 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.712 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.712 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:17.712 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805804+0000", 2024-09-18T04:40:17.712 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00098114699999999992 2024-09-18T04:40:17.712 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.712 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.712 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:17.712 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805831+0000", 2024-09-18T04:40:17.712 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6401999999999999e-05 2024-09-18T04:40:17.712 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.712 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.712 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:17.712 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805850+0000", 2024-09-18T04:40:17.713 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8862e-05 2024-09-18T04:40:17.713 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:17.713 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:17.713 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:17.713 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.713 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.713 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:36 2.7 2:fe1b02ad:::rbd_data.125fc09df6d1.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:17.713 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:40:17.713 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 95.424648970000007, 2024-09-18T04:40:17.713 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.040387804999999999, 2024-09-18T04:40:17.713 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:17.713 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:17.713 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:17.713 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:40:17.714 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:40:17.714 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 36 2024-09-18T04:40:17.714 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.714 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:17.714 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.714 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:17.714 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:40:17.714 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:17.714 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.714 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.714 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:17.714 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:40:17.714 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:17.714 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.715 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.715 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:17.715 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265720+0000", 2024-09-18T04:40:17.715 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:40:17.715 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.715 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.715 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:17.715 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272590+0000", 2024-09-18T04:40:17.715 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.006870508 2024-09-18T04:40:17.715 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.715 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.715 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:17.715 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272623+0000", 2024-09-18T04:40:17.715 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2716000000000003e-05 2024-09-18T04:40:17.715 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.716 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.716 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:17.716 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272637+0000", 2024-09-18T04:40:17.716 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4175999999999999e-05 2024-09-18T04:40:17.716 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.716 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.716 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:17.716 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272779+0000", 2024-09-18T04:40:17.716 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000141392 2024-09-18T04:40:17.716 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.716 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.716 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:17.716 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273031+0000", 2024-09-18T04:40:17.716 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00025249099999999998 2024-09-18T04:40:17.717 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.717 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.717 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:40:17.717 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273219+0000", 2024-09-18T04:40:17.717 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000187531 2024-09-18T04:40:17.717 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.717 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.717 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:17.717 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.302806+0000", 2024-09-18T04:40:17.717 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.029587553999999999 2024-09-18T04:40:17.717 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.717 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.717 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:17.718 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306004+0000", 2024-09-18T04:40:17.718 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0031982629999999998 2024-09-18T04:40:17.718 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.718 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.718 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:17.718 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306071+0000", 2024-09-18T04:40:17.718 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6829999999999995e-05 2024-09-18T04:40:17.718 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.718 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.718 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:17.718 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306110+0000", 2024-09-18T04:40:17.718 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8874000000000001e-05 2024-09-18T04:40:17.718 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:17.718 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:17.719 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:17.719 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.719 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.719 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:38 2.0 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:17.719 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:40:17.719 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 95.417695433000006, 2024-09-18T04:40:17.719 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.033754348000000003, 2024-09-18T04:40:17.719 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:17.719 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:17.719 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:17.719 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:40:17.719 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:40:17.719 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 38 2024-09-18T04:40:17.719 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.720 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:17.720 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.720 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:17.720 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:40:17.720 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:17.720 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.720 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.720 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:17.720 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:40:17.720 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:17.720 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.720 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.720 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:17.720 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272673+0000", 2024-09-18T04:40:17.721 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:40:17.721 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.721 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.721 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:17.721 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277166+0000", 2024-09-18T04:40:17.721 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044922249999999999 2024-09-18T04:40:17.721 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.721 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.721 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:17.721 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277189+0000", 2024-09-18T04:40:17.721 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3283000000000001e-05 2024-09-18T04:40:17.721 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.721 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.721 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:17.722 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277205+0000", 2024-09-18T04:40:17.722 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6305000000000001e-05 2024-09-18T04:40:17.722 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.722 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.722 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:17.722 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277290+0000", 2024-09-18T04:40:17.722 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5135000000000004e-05 2024-09-18T04:40:17.722 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.722 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.722 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:17.722 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277531+0000", 2024-09-18T04:40:17.722 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00024014699999999999 2024-09-18T04:40:17.722 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.722 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.723 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:17.723 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277793+0000", 2024-09-18T04:40:17.723 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00026224499999999998 2024-09-18T04:40:17.723 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.723 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.723 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:17.723 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303990+0000", 2024-09-18T04:40:17.723 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.026197295999999998 2024-09-18T04:40:17.723 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.723 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.723 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:17.723 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306340+0000", 2024-09-18T04:40:17.723 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.002349524 2024-09-18T04:40:17.723 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.724 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.724 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:17.724 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306394+0000", 2024-09-18T04:40:17.724 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3934999999999998e-05 2024-09-18T04:40:17.724 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.724 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.724 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:17.724 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306430+0000", 2024-09-18T04:40:17.724 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6690000000000003e-05 2024-09-18T04:40:17.724 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:17.724 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:17.724 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:17.724 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.724 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.725 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:39 2.7 2:e9c909c5:::rbd_data.125fc09df6d1.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:17.725 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:40:17.725 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 95.413135827999994, 2024-09-18T04:40:17.725 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.029110673, 2024-09-18T04:40:17.725 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:17.725 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:17.725 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:17.725 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:40:17.725 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:40:17.725 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 39 2024-09-18T04:40:17.725 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.725 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:17.725 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.725 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:17.726 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:40:17.726 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:17.726 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.726 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.726 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:17.726 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:40:17.726 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:17.726 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.726 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.726 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:17.726 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277234+0000", 2024-09-18T04:40:17.726 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:17.726 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.726 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.726 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:17.727 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281351+0000", 2024-09-18T04:40:17.727 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0041170920000000001 2024-09-18T04:40:17.727 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.727 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.727 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:17.727 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281375+0000", 2024-09-18T04:40:17.727 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.446e-05 2024-09-18T04:40:17.727 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.727 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.727 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:17.727 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281390+0000", 2024-09-18T04:40:17.727 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4309e-05 2024-09-18T04:40:17.727 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.727 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.728 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:17.728 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.282306+0000", 2024-09-18T04:40:17.728 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091662000000000004 2024-09-18T04:40:17.728 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.728 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.728 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:17.728 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.282613+0000", 2024-09-18T04:40:17.728 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00030663799999999997 2024-09-18T04:40:17.728 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.728 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.728 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:40:17.728 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283001+0000", 2024-09-18T04:40:17.728 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00038801599999999998 2024-09-18T04:40:17.728 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.728 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.728 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:17.729 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306239+0000", 2024-09-18T04:40:17.729 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023238466999999999 2024-09-18T04:40:17.729 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.729 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.729 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:17.729 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306259+0000", 2024-09-18T04:40:17.729 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9837999999999999e-05 2024-09-18T04:40:17.729 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.729 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.729 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:17.729 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306299+0000", 2024-09-18T04:40:17.729 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0194000000000002e-05 2024-09-18T04:40:17.729 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.729 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.729 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:17.729 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306346+0000", 2024-09-18T04:40:17.730 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6845000000000003e-05 2024-09-18T04:40:17.730 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:17.730 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:17.730 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:17.730 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.730 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.730 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:46 2.6 e51/17 2:7ba25245:::rbd_data.125fc09df6d1.0000000000000006:head v 51'2, mlcod=0'0)", 2024-09-18T04:40:17.730 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:40:17.730 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 95.377214695999996, 2024-09-18T04:40:17.730 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019843098, 2024-09-18T04:40:17.730 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:17.730 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:17.730 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:17.730 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.730 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:17.731 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:40:17.731 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:17.731 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.731 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.731 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:17.731 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:40:17.731 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:17.731 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.731 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.731 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:17.731 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313156+0000", 2024-09-18T04:40:17.731 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:40:17.731 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.731 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.731 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:17.731 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320144+0000", 2024-09-18T04:40:17.732 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.006988345 2024-09-18T04:40:17.732 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.732 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.732 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:17.732 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320175+0000", 2024-09-18T04:40:17.732 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.0861999999999997e-05 2024-09-18T04:40:17.732 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.732 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.732 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:17.732 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320188+0000", 2024-09-18T04:40:17.732 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2714999999999999e-05 2024-09-18T04:40:17.732 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.732 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.732 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:17.732 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320267+0000", 2024-09-18T04:40:17.732 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9586999999999995e-05 2024-09-18T04:40:17.733 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.733 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.733 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:17.733 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320311+0000", 2024-09-18T04:40:17.733 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3529000000000003e-05 2024-09-18T04:40:17.733 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.733 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.733 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:17.733 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.332949+0000", 2024-09-18T04:40:17.733 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012638267999999999 2024-09-18T04:40:17.734 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.734 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.734 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:17.734 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.333000+0000", 2024-09-18T04:40:17.734 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0736999999999999e-05 2024-09-18T04:40:17.734 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:17.734 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:17.734 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:17.734 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.734 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.734 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:57 2.0 2:0cdd070e:::rbd_data.125fc09df6d1.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:17.734 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:40:17.734 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 95.341357121000001, 2024-09-18T04:40:17.734 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.033329974999999998, 2024-09-18T04:40:17.734 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:17.735 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:17.735 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:17.735 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:40:17.735 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:40:17.735 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 57 2024-09-18T04:40:17.735 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.735 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:17.735 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.735 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:17.735 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:40:17.735 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:17.735 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.735 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.735 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:17.735 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:40:17.736 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:17.736 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.736 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.736 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:17.736 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349013+0000", 2024-09-18T04:40:17.736 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:17.736 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.736 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.736 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:17.736 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356138+0000", 2024-09-18T04:40:17.736 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071258010000000002 2024-09-18T04:40:17.736 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.736 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.736 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:17.737 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356168+0000", 2024-09-18T04:40:17.737 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9679000000000002e-05 2024-09-18T04:40:17.737 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.737 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.737 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:17.737 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356181+0000", 2024-09-18T04:40:17.737 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3164e-05 2024-09-18T04:40:17.737 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.737 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.737 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:17.737 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356254+0000", 2024-09-18T04:40:17.737 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3182999999999994e-05 2024-09-18T04:40:17.737 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.738 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.738 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:17.738 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356453+0000", 2024-09-18T04:40:17.738 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019876599999999999 2024-09-18T04:40:17.738 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.738 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.738 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:17.738 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356627+0000", 2024-09-18T04:40:17.738 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017340199999999999 2024-09-18T04:40:17.738 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.738 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.738 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:17.738 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.369200+0000", 2024-09-18T04:40:17.738 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012573302 2024-09-18T04:40:17.738 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.739 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.739 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:17.739 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382262+0000", 2024-09-18T04:40:17.739 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013062549 2024-09-18T04:40:17.739 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.739 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.739 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:17.739 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382310+0000", 2024-09-18T04:40:17.739 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7366e-05 2024-09-18T04:40:17.739 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.739 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.739 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:17.739 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382344+0000", 2024-09-18T04:40:17.740 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4319999999999997e-05 2024-09-18T04:40:17.740 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:17.740 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:17.740 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:17.740 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.740 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.740 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:59 2.0 2:19c52cde:::rbd_data.125fc09df6d1.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:17.740 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:40:17.740 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 95.334165956000007, 2024-09-18T04:40:17.740 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.027137247999999999, 2024-09-18T04:40:17.740 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:17.740 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:17.740 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:17.740 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:40:17.741 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:40:17.741 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 59 2024-09-18T04:40:17.741 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.741 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:17.741 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.741 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:17.741 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:40:17.741 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:17.741 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.741 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.741 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:17.741 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:40:17.741 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:17.741 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.742 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.742 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:17.742 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356204+0000", 2024-09-18T04:40:17.742 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:17.742 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.742 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.742 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:17.742 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362661+0000", 2024-09-18T04:40:17.742 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0064565129999999997 2024-09-18T04:40:17.742 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.742 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.742 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:17.742 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362684+0000", 2024-09-18T04:40:17.742 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3688000000000001e-05 2024-09-18T04:40:17.742 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.743 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.743 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:17.743 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362699+0000", 2024-09-18T04:40:17.743 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4798e-05 2024-09-18T04:40:17.743 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.743 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.743 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:17.743 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363016+0000", 2024-09-18T04:40:17.743 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031737300000000003 2024-09-18T04:40:17.743 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.743 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.743 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:17.743 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363201+0000", 2024-09-18T04:40:17.743 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000184131 2024-09-18T04:40:17.743 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.744 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.744 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:17.744 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363343+0000", 2024-09-18T04:40:17.744 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014236699999999999 2024-09-18T04:40:17.744 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.744 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.744 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:17.744 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.372189+0000", 2024-09-18T04:40:17.744 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0088464819999999993 2024-09-18T04:40:17.744 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.744 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.744 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:17.744 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383269+0000", 2024-09-18T04:40:17.744 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011079594 2024-09-18T04:40:17.744 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.744 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.745 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:17.745 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383314+0000", 2024-09-18T04:40:17.745 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4932000000000003e-05 2024-09-18T04:40:17.745 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.745 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.745 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:17.745 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383343+0000", 2024-09-18T04:40:17.745 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8600999999999999e-05 2024-09-18T04:40:17.745 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:17.745 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:17.745 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:17.745 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.745 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.745 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:40 2.6 e51/17 2:784d6cd1:ns::rbd_data.133e813ab5bc.0000000000000005:head v 51'7, mlcod=51'5)", 2024-09-18T04:40:17.745 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:40:17.746 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 95.137916222000001, 2024-09-18T04:40:17.746 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020497496, 2024-09-18T04:40:17.746 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:17.746 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:17.746 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:17.746 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.746 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:17.746 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:40:17.746 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:17.746 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.746 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.746 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:17.746 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:40:17.746 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:17.746 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.746 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.747 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:17.747 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552454+0000", 2024-09-18T04:40:17.747 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:17.747 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.747 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.747 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:17.747 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556448+0000", 2024-09-18T04:40:17.747 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0039940990000000001 2024-09-18T04:40:17.747 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.747 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.747 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:17.747 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556481+0000", 2024-09-18T04:40:17.747 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3778000000000001e-05 2024-09-18T04:40:17.747 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.747 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.747 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:17.748 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556493+0000", 2024-09-18T04:40:17.748 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1622e-05 2024-09-18T04:40:17.748 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.748 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.748 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:17.748 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556556+0000", 2024-09-18T04:40:17.748 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2582999999999994e-05 2024-09-18T04:40:17.748 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.748 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.748 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:17.748 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556592+0000", 2024-09-18T04:40:17.748 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5880999999999997e-05 2024-09-18T04:40:17.748 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.748 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.749 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:17.749 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.572786+0000", 2024-09-18T04:40:17.749 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016194571000000001 2024-09-18T04:40:17.749 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.749 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.749 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:17.749 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.572953+0000", 2024-09-18T04:40:17.749 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000166436 2024-09-18T04:40:17.749 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:17.749 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:17.749 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:17.749 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.749 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.750 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:45 2.0 2:0fbfa3dc:ns::rbd_data.133e813ab5bc.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:17.750 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:40:17.750 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 95.126701922999999, 2024-09-18T04:40:17.750 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.036558168000000002, 2024-09-18T04:40:17.750 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:17.750 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:17.750 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:17.750 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:40:17.750 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:17.750 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 45 2024-09-18T04:40:17.750 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.750 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:17.750 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.750 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:17.750 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:40:17.750 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:17.751 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.751 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.751 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:17.751 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:40:17.751 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:17.751 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.751 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.751 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:17.751 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:40:17.751 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:40:17.751 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.751 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.751 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:17.751 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578594+0000", 2024-09-18T04:40:17.751 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014927147999999999 2024-09-18T04:40:17.752 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.752 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.752 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:17.752 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578612+0000", 2024-09-18T04:40:17.752 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7623999999999999e-05 2024-09-18T04:40:17.752 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.752 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.752 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:17.752 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578620+0000", 2024-09-18T04:40:17.752 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9070000000000007e-06 2024-09-18T04:40:17.752 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.752 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.752 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:17.752 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578702+0000", 2024-09-18T04:40:17.753 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2349999999999996e-05 2024-09-18T04:40:17.753 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.753 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.753 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:17.753 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.579028+0000", 2024-09-18T04:40:17.753 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032622899999999998 2024-09-18T04:40:17.753 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.753 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.753 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:17.753 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.579343+0000", 2024-09-18T04:40:17.753 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031448899999999999 2024-09-18T04:40:17.753 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.753 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.753 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:17.753 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.590078+0000", 2024-09-18T04:40:17.754 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010734946 2024-09-18T04:40:17.754 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.754 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.754 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:17.754 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600089+0000", 2024-09-18T04:40:17.754 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010011195000000001 2024-09-18T04:40:17.754 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.754 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.754 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:17.754 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600177+0000", 2024-09-18T04:40:17.755 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8425999999999997e-05 2024-09-18T04:40:17.755 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.755 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.755 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:17.755 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600228+0000", 2024-09-18T04:40:17.755 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0216999999999997e-05 2024-09-18T04:40:17.755 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:17.755 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:17.755 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:17.755 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.755 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.755 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:54 2.0 2:04f17ef6:ns::rbd_data.133e813ab5bc.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:17.755 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:40:17.756 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 95.105130086000003, 2024-09-18T04:40:17.756 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.024612999, 2024-09-18T04:40:17.756 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:17.756 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:17.756 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:17.756 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:40:17.756 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:17.756 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 54 2024-09-18T04:40:17.756 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.756 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:17.756 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.756 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:17.756 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:40:17.756 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:17.757 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.757 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.757 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:17.757 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:40:17.757 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:17.757 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.757 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.757 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:17.757 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585239+0000", 2024-09-18T04:40:17.757 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:17.757 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.757 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.757 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:17.758 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593702+0000", 2024-09-18T04:40:17.758 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0084632990000000005 2024-09-18T04:40:17.758 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.758 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.758 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:17.758 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593719+0000", 2024-09-18T04:40:17.758 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.664e-05 2024-09-18T04:40:17.758 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.758 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.758 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:17.758 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593727+0000", 2024-09-18T04:40:17.758 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6839999999999996e-06 2024-09-18T04:40:17.758 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.758 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.758 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:17.759 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593767+0000", 2024-09-18T04:40:17.759 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0330999999999997e-05 2024-09-18T04:40:17.759 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.759 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.759 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:17.759 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593905+0000", 2024-09-18T04:40:17.759 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013754800000000001 2024-09-18T04:40:17.759 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.759 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.759 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:17.759 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.594013+0000", 2024-09-18T04:40:17.759 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000108805 2024-09-18T04:40:17.759 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.759 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.759 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:17.760 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604678+0000", 2024-09-18T04:40:17.760 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010664391 2024-09-18T04:40:17.760 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.760 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.760 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:17.760 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609662+0000", 2024-09-18T04:40:17.760 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0049846100000000004 2024-09-18T04:40:17.760 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.760 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.760 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:17.760 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609788+0000", 2024-09-18T04:40:17.760 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012518800000000001 2024-09-18T04:40:17.760 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.760 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.760 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:17.761 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609854+0000", 2024-09-18T04:40:17.761 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6556000000000004e-05 2024-09-18T04:40:17.761 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:17.761 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:17.761 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:17.761 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.761 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.761 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:58 2.7 2:fc7c6cd3:ns::rbd_data.133e813ab5bc.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:17.761 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:40:17.761 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 95.092287657, 2024-09-18T04:40:17.761 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.027690566999999999, 2024-09-18T04:40:17.761 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:17.761 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:17.761 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:17.761 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:40:17.762 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:17.762 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 58 2024-09-18T04:40:17.762 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.762 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:17.762 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.762 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:17.762 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:40:17.762 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:17.762 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.762 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.762 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:17.762 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:40:17.762 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:17.762 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.762 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.763 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:17.763 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598082+0000", 2024-09-18T04:40:17.763 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:17.763 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.763 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.763 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:17.763 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604832+0000", 2024-09-18T04:40:17.763 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0067504510000000002 2024-09-18T04:40:17.763 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.763 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.763 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:17.763 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604852+0000", 2024-09-18T04:40:17.763 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0228999999999999e-05 2024-09-18T04:40:17.763 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.763 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.763 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:17.764 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604870+0000", 2024-09-18T04:40:17.764 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7952e-05 2024-09-18T04:40:17.764 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.764 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.764 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:17.764 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604936+0000", 2024-09-18T04:40:17.764 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5431000000000004e-05 2024-09-18T04:40:17.764 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.764 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.764 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:17.764 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605085+0000", 2024-09-18T04:40:17.764 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014910300000000001 2024-09-18T04:40:17.764 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.764 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.765 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:40:17.765 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605205+0000", 2024-09-18T04:40:17.765 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000119887 2024-09-18T04:40:17.765 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.765 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.765 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:17.765 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625586+0000", 2024-09-18T04:40:17.765 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020381399000000001 2024-09-18T04:40:17.765 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.765 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.765 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:17.765 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625628+0000", 2024-09-18T04:40:17.765 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1873999999999999e-05 2024-09-18T04:40:17.765 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.766 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.766 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:17.766 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625722+0000", 2024-09-18T04:40:17.766 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3787000000000002e-05 2024-09-18T04:40:17.766 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.766 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.766 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:17.766 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625774+0000", 2024-09-18T04:40:17.766 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2503999999999997e-05 2024-09-18T04:40:17.766 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:17.766 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:17.766 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:17.766 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.766 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.767 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:56 2.6 e51/17 2:72453c3e:ns::rbd_data.133e813ab5bc.000000000000000d:head v 51'10, mlcod=51'8)", 2024-09-18T04:40:17.767 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:40:17.767 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 95.085090549, 2024-09-18T04:40:17.767 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019766552999999999, 2024-09-18T04:40:17.767 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:17.767 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:17.767 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:17.767 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.767 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:17.767 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:40:17.767 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:17.767 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.767 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.767 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:17.767 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:40:17.768 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:17.768 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.768 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.768 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:17.768 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605279+0000", 2024-09-18T04:40:17.768 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:17.768 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.768 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.768 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:17.768 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609417+0000", 2024-09-18T04:40:17.768 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0041378059999999999 2024-09-18T04:40:17.768 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.768 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.768 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:17.769 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609443+0000", 2024-09-18T04:40:17.769 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6098e-05 2024-09-18T04:40:17.769 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.769 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.769 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:17.769 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609458+0000", 2024-09-18T04:40:17.769 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4671000000000001e-05 2024-09-18T04:40:17.769 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.769 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.769 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:17.769 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609553+0000", 2024-09-18T04:40:17.769 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5265000000000004e-05 2024-09-18T04:40:17.769 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.769 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.770 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:17.770 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609583+0000", 2024-09-18T04:40:17.770 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9839e-05 2024-09-18T04:40:17.770 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.770 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.770 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:17.770 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625004+0000", 2024-09-18T04:40:17.770 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015420639999999999 2024-09-18T04:40:17.770 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.770 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.770 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:17.770 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625047+0000", 2024-09-18T04:40:17.770 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3665999999999998e-05 2024-09-18T04:40:17.770 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:17.770 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:17.771 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:17.771 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.771 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.771 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:99 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:40:17.771 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.582718+0000", 2024-09-18T04:40:17.771 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 81.107653639999995, 2024-09-18T04:40:17.771 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019362575, 2024-09-18T04:40:17.771 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:17.771 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:17.771 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:17.771 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:40:17.771 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:40:17.771 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 99 2024-09-18T04:40:17.771 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.771 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:17.772 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.772 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:17.772 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582718+0000", 2024-09-18T04:40:17.772 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:17.772 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.772 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.772 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:17.772 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582718+0000", 2024-09-18T04:40:17.772 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:17.772 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.772 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.772 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:17.772 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582717+0000", 2024-09-18T04:40:17.772 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:40:17.773 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.773 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.773 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:17.773 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582720+0000", 2024-09-18T04:40:17.773 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2239999999999998e-06 2024-09-18T04:40:17.773 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.773 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.773 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:17.773 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582723+0000", 2024-09-18T04:40:17.773 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.258e-06 2024-09-18T04:40:17.773 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.773 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.773 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:17.774 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582726+0000", 2024-09-18T04:40:17.774 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8870000000000001e-06 2024-09-18T04:40:17.774 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.774 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.774 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:17.774 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.592018+0000", 2024-09-18T04:40:17.774 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0092926420000000003 2024-09-18T04:40:17.774 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.774 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.774 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:17.774 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.592214+0000", 2024-09-18T04:40:17.774 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000195267 2024-09-18T04:40:17.774 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.774 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.774 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:17.775 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.592913+0000", 2024-09-18T04:40:17.775 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00069924600000000005 2024-09-18T04:40:17.775 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.775 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.775 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:17.775 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597493+0000", 2024-09-18T04:40:17.775 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045797629999999997 2024-09-18T04:40:17.775 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.775 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.775 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:17.775 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.602009+0000", 2024-09-18T04:40:17.775 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045159559999999998 2024-09-18T04:40:17.775 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.775 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.775 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:17.776 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.602042+0000", 2024-09-18T04:40:17.776 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3084999999999999e-05 2024-09-18T04:40:17.776 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.776 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.776 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:17.776 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.602080+0000", 2024-09-18T04:40:17.776 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8600000000000003e-05 2024-09-18T04:40:17.776 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:17.776 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:17.776 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:17.776 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.776 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.776 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:104 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:40:17.776 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.583165+0000", 2024-09-18T04:40:17.777 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 81.107206341999998, 2024-09-18T04:40:17.777 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023679603, 2024-09-18T04:40:17.777 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:17.777 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:17.777 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:17.777 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:40:17.777 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:40:17.777 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 104 2024-09-18T04:40:17.777 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.777 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:17.777 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.777 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:17.777 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583165+0000", 2024-09-18T04:40:17.778 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:17.778 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.778 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.778 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:17.778 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583165+0000", 2024-09-18T04:40:17.778 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:17.778 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.778 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.778 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:17.778 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583164+0000", 2024-09-18T04:40:17.778 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:40:17.778 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.778 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.778 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:17.779 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583167+0000", 2024-09-18T04:40:17.779 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9210000000000001e-06 2024-09-18T04:40:17.779 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.779 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.779 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:17.779 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583171+0000", 2024-09-18T04:40:17.779 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9759999999999997e-06 2024-09-18T04:40:17.779 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.779 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.779 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:17.779 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583174+0000", 2024-09-18T04:40:17.779 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9960000000000002e-06 2024-09-18T04:40:17.779 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.779 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.780 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:17.780 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.594373+0000", 2024-09-18T04:40:17.780 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011198665999999999 2024-09-18T04:40:17.780 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.780 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.780 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:17.780 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.594434+0000", 2024-09-18T04:40:17.780 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.1446999999999996e-05 2024-09-18T04:40:17.780 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.780 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.780 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:17.780 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.594692+0000", 2024-09-18T04:40:17.780 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00025756599999999998 2024-09-18T04:40:17.780 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.781 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.781 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:17.781 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599302+0000", 2024-09-18T04:40:17.781 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0046099640000000002 2024-09-18T04:40:17.781 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.781 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.781 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:17.781 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606710+0000", 2024-09-18T04:40:17.781 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0074077739999999998 2024-09-18T04:40:17.781 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.781 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.781 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:17.781 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606798+0000", 2024-09-18T04:40:17.782 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8133999999999993e-05 2024-09-18T04:40:17.782 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.782 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.782 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:17.782 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606845+0000", 2024-09-18T04:40:17.782 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6653000000000002e-05 2024-09-18T04:40:17.782 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:17.782 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:17.782 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:17.782 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.782 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.782 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:105 2.0 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:40:17.782 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.583193+0000", 2024-09-18T04:40:17.782 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 81.107178723000004, 2024-09-18T04:40:17.782 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023713649, 2024-09-18T04:40:17.783 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:17.783 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:17.783 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:17.783 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:40:17.783 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:40:17.783 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 105 2024-09-18T04:40:17.783 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.783 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:17.783 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.783 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:17.783 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583193+0000", 2024-09-18T04:40:17.783 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:17.783 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.783 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.783 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:17.784 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583193+0000", 2024-09-18T04:40:17.784 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:17.784 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.784 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.784 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:17.784 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583192+0000", 2024-09-18T04:40:17.784 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:40:17.784 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.784 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.784 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:17.784 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583194+0000", 2024-09-18T04:40:17.784 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.305e-06 2024-09-18T04:40:17.784 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.784 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.785 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:17.785 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583198+0000", 2024-09-18T04:40:17.785 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9369999999999997e-06 2024-09-18T04:40:17.785 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.785 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.785 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:17.785 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583202+0000", 2024-09-18T04:40:17.785 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6480000000000001e-06 2024-09-18T04:40:17.785 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.785 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.785 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:17.785 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.595304+0000", 2024-09-18T04:40:17.785 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012102063999999999 2024-09-18T04:40:17.785 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.786 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.786 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:17.786 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.595390+0000", 2024-09-18T04:40:17.786 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5493000000000003e-05 2024-09-18T04:40:17.786 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.786 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.786 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:17.786 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.595509+0000", 2024-09-18T04:40:17.786 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011966199999999999 2024-09-18T04:40:17.786 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.786 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.786 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:17.786 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599358+0000", 2024-09-18T04:40:17.786 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0038486689999999999 2024-09-18T04:40:17.786 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.787 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.787 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:17.787 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606859+0000", 2024-09-18T04:40:17.787 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0075010789999999999 2024-09-18T04:40:17.787 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.787 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.787 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:17.787 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606887+0000", 2024-09-18T04:40:17.787 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7773e-05 2024-09-18T04:40:17.787 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.787 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.787 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:17.787 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606906+0000", 2024-09-18T04:40:17.787 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9417000000000001e-05 2024-09-18T04:40:17.787 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:17.788 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:17.788 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:17.788 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.788 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.788 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:106 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:40:17.788 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.584425+0000", 2024-09-18T04:40:17.788 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 81.105946427000006, 2024-09-18T04:40:17.788 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.022529293999999998, 2024-09-18T04:40:17.788 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:17.788 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:17.788 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:17.788 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:40:17.788 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:40:17.789 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 106 2024-09-18T04:40:17.789 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.789 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:17.789 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.789 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:17.789 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584425+0000", 2024-09-18T04:40:17.789 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:17.789 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.789 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.789 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:17.789 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584425+0000", 2024-09-18T04:40:17.789 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:17.789 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.790 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.790 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:17.790 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584424+0000", 2024-09-18T04:40:17.790 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:40:17.790 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.790 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.790 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:17.790 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584430+0000", 2024-09-18T04:40:17.790 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6389999999999997e-06 2024-09-18T04:40:17.790 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.790 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.790 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:17.790 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584456+0000", 2024-09-18T04:40:17.790 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6188999999999999e-05 2024-09-18T04:40:17.790 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.791 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.791 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:17.791 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584463+0000", 2024-09-18T04:40:17.791 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7340000000000002e-06 2024-09-18T04:40:17.791 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.791 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.791 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:17.791 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597537+0000", 2024-09-18T04:40:17.791 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013074161000000001 2024-09-18T04:40:17.791 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.791 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.791 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:17.791 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597598+0000", 2024-09-18T04:40:17.791 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0930000000000001e-05 2024-09-18T04:40:17.791 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.792 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.792 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:17.792 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597922+0000", 2024-09-18T04:40:17.792 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032473000000000002 2024-09-18T04:40:17.792 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.792 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.792 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:17.792 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.601103+0000", 2024-09-18T04:40:17.792 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0031809849999999999 2024-09-18T04:40:17.792 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.792 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.792 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:17.792 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606912+0000", 2024-09-18T04:40:17.792 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058083550000000003 2024-09-18T04:40:17.792 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.793 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.793 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:17.793 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606935+0000", 2024-09-18T04:40:17.793 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2759999999999999e-05 2024-09-18T04:40:17.793 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.793 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.793 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:17.793 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606954+0000", 2024-09-18T04:40:17.793 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9602999999999998e-05 2024-09-18T04:40:17.793 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:17.793 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:17.793 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:17.793 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.794 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.794 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:109 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:40:17.794 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.585231+0000", 2024-09-18T04:40:17.794 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 81.105139807, 2024-09-18T04:40:17.794 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.021767496000000001, 2024-09-18T04:40:17.794 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:17.794 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:17.794 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:17.794 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:40:17.794 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:40:17.794 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 109 2024-09-18T04:40:17.794 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.795 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:17.795 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.795 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:17.795 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585231+0000", 2024-09-18T04:40:17.795 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:17.795 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.795 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.795 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:17.795 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585231+0000", 2024-09-18T04:40:17.795 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:17.795 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.795 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.795 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:17.795 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585230+0000", 2024-09-18T04:40:17.795 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:40:17.796 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.796 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.796 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:17.796 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585237+0000", 2024-09-18T04:40:17.796 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.618e-06 2024-09-18T04:40:17.796 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.796 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.796 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:17.796 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585256+0000", 2024-09-18T04:40:17.796 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9048000000000001e-05 2024-09-18T04:40:17.796 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.796 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.796 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:17.797 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585274+0000", 2024-09-18T04:40:17.797 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8153000000000001e-05 2024-09-18T04:40:17.797 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.797 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.797 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:17.797 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.598284+0000", 2024-09-18T04:40:17.797 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013009682999999999 2024-09-18T04:40:17.797 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.797 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.797 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:17.797 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.598383+0000", 2024-09-18T04:40:17.797 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.8722999999999997e-05 2024-09-18T04:40:17.797 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.798 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.798 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:17.798 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.598829+0000", 2024-09-18T04:40:17.798 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00044623999999999998 2024-09-18T04:40:17.798 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.798 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.798 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:17.798 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.601149+0000", 2024-09-18T04:40:17.798 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0023204150000000002 2024-09-18T04:40:17.798 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.798 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.798 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:17.798 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606960+0000", 2024-09-18T04:40:17.798 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058106390000000003 2024-09-18T04:40:17.798 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.800 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.800 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:17.800 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606982+0000", 2024-09-18T04:40:17.800 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2252e-05 2024-09-18T04:40:17.800 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.800 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.800 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:17.800 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606999+0000", 2024-09-18T04:40:17.800 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6871000000000002e-05 2024-09-18T04:40:17.800 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:17.800 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:17.800 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:17.800 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.800 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.801 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:110 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:40:17.801 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.586640+0000", 2024-09-18T04:40:17.801 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 81.103731487999994, 2024-09-18T04:40:17.801 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020400911000000001, 2024-09-18T04:40:17.801 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:17.801 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:17.801 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:17.801 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:40:17.801 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:40:17.801 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 110 2024-09-18T04:40:17.801 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.801 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:17.801 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.801 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:17.802 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586640+0000", 2024-09-18T04:40:17.802 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:17.802 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.802 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.802 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:17.802 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586640+0000", 2024-09-18T04:40:17.802 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:17.802 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.802 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.802 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:17.802 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586639+0000", 2024-09-18T04:40:17.802 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:40:17.802 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.802 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.803 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:17.803 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586652+0000", 2024-09-18T04:40:17.803 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2482e-05 2024-09-18T04:40:17.803 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.803 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.803 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:17.803 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586660+0000", 2024-09-18T04:40:17.803 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1659999999999994e-06 2024-09-18T04:40:17.803 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.803 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.803 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:17.803 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586666+0000", 2024-09-18T04:40:17.803 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6900000000000003e-06 2024-09-18T04:40:17.803 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.803 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.804 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:17.804 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599381+0000", 2024-09-18T04:40:17.804 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012714328 2024-09-18T04:40:17.804 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.804 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.804 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:17.804 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599436+0000", 2024-09-18T04:40:17.804 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4802999999999997e-05 2024-09-18T04:40:17.804 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.804 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.804 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:17.804 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599629+0000", 2024-09-18T04:40:17.804 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000193921 2024-09-18T04:40:17.804 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.805 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.805 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:17.805 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.601174+0000", 2024-09-18T04:40:17.805 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001544857 2024-09-18T04:40:17.805 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.805 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.805 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:17.805 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.607004+0000", 2024-09-18T04:40:17.805 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058300820000000003 2024-09-18T04:40:17.805 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.805 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.805 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:17.805 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.607024+0000", 2024-09-18T04:40:17.805 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9888e-05 2024-09-18T04:40:17.806 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:17.806 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:17.806 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:17.806 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.607041+0000", 2024-09-18T04:40:17.806 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6450999999999999e-05 2024-09-18T04:40:17.806 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:17.806 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:17.806 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:17.806 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:17.806 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:17.806 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:40:17.807 INFO:tasks.workunit.client.0.smithi115.stdout:OK 2024-09-18T04:40:17.808 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:40:17.809 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:40:17.804+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:40:17.809 INFO:tasks.workunit.client.0.smithi115.stderr:+ DEV= 2024-09-18T04:40:17.809 INFO:tasks.workunit.client.0.smithi115.stderr:+ echo OK 2024-09-18T04:40:17.809 INFO:tasks.workunit.client.0.smithi115.stderr:+ cleanup 2024-09-18T04:40:17.809 INFO:tasks.workunit.client.0.smithi115.stderr:+ local ns s 2024-09-18T04:40:17.809 INFO:tasks.workunit.client.0.smithi115.stderr:+ set +e 2024-09-18T04:40:17.809 INFO:tasks.workunit.client.0.smithi115.stderr:+ mount 2024-09-18T04:40:17.809 INFO:tasks.workunit.client.0.smithi115.stderr:+ fgrep /tmp/tmp.WJMNJxIdAN/mnt 2024-09-18T04:40:17.809 INFO:tasks.workunit.client.0.smithi115.stderr:+ rm -Rf /tmp/tmp.WJMNJxIdAN 2024-09-18T04:40:17.818 INFO:tasks.workunit.client.0.smithi115.stderr:+ '[' -n '' ']' 2024-09-18T04:40:17.818 INFO:tasks.workunit.client.0.smithi115.stderr:+ for ns in '' ${NS} 2024-09-18T04:40:17.818 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd -p rbd --namespace '' status testrbdnbd17342 2024-09-18T04:40:17.884 INFO:tasks.workunit.client.0.smithi115.stdout:Watchers: none 2024-09-18T04:40:17.892 INFO:tasks.workunit.client.0.smithi115.stderr:+ for s in 0.5 1 2 4 8 16 32 2024-09-18T04:40:17.892 INFO:tasks.workunit.client.0.smithi115.stderr:+ sleep 0.5 2024-09-18T04:40:17.908 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:40:17.904+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:40:17.919 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-18T04:40:17.963 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:40:17.963 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:40:17.963 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:40:17.963 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:40:17.973 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:40:18.018 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:40:18.016+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:40:18.079 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:40:18.079 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:40:18.079 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:40:18.079 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:40:18.079 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:40:18.088 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:40:18.109 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:40:18.104+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:40:18.200 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:40:18.200 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:40:18.200 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:40:18.200 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:40:18.200 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.201 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.5431.0:1 3.3 3:d80af3cb:::rbd_id.testrbdnbd17342:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-18T04:40:18.201 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:13.542818+0000", 2024-09-18T04:40:18.201 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 64.654483839999997, 2024-09-18T04:40:18.201 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048493400000000002, 2024-09-18T04:40:18.201 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:18.201 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:40:18.201 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:18.201 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.5431", 2024-09-18T04:40:18.201 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3147136083", 2024-09-18T04:40:18.201 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 1 2024-09-18T04:40:18.201 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.201 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:18.202 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.202 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:18.202 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542818+0000", 2024-09-18T04:40:18.202 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:18.202 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.202 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.202 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:18.202 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542818+0000", 2024-09-18T04:40:18.202 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:18.202 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.202 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.202 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:18.202 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542816+0000", 2024-09-18T04:40:18.202 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:18.203 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.203 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.203 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:18.203 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542828+0000", 2024-09-18T04:40:18.203 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2699e-05 2024-09-18T04:40:18.203 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.203 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.203 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:18.203 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542852+0000", 2024-09-18T04:40:18.203 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3176e-05 2024-09-18T04:40:18.203 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.203 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.203 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:18.203 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542874+0000", 2024-09-18T04:40:18.204 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1967000000000002e-05 2024-09-18T04:40:18.204 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.204 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.204 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:18.204 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542984+0000", 2024-09-18T04:40:18.204 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011026099999999999 2024-09-18T04:40:18.204 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.204 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.204 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:18.204 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543161+0000", 2024-09-18T04:40:18.204 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017720299999999999 2024-09-18T04:40:18.204 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.204 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.205 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:18.205 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543303+0000", 2024-09-18T04:40:18.205 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014174700000000001 2024-09-18T04:40:18.205 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:18.205 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:18.205 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:18.205 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.205 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.206 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.5431.0:2 3.3 3:d80af3cb:::rbd_id.testrbdnbd17342:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-18T04:40:18.206 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:13.543564+0000", 2024-09-18T04:40:18.206 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 64.653737354, 2024-09-18T04:40:18.206 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010399929999999999, 2024-09-18T04:40:18.206 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:18.206 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:40:18.206 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:18.207 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.5431", 2024-09-18T04:40:18.207 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3147136083", 2024-09-18T04:40:18.207 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 2 2024-09-18T04:40:18.207 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.207 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:18.207 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.208 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:18.208 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543564+0000", 2024-09-18T04:40:18.209 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:18.209 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.209 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.209 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:18.209 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543564+0000", 2024-09-18T04:40:18.209 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:18.209 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.209 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.209 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:18.209 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543563+0000", 2024-09-18T04:40:18.209 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:18.209 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.210 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.210 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:18.210 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543571+0000", 2024-09-18T04:40:18.210 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4270000000000008e-06 2024-09-18T04:40:18.210 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.210 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.210 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:18.210 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543591+0000", 2024-09-18T04:40:18.210 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.929e-05 2024-09-18T04:40:18.210 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.210 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.210 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:18.211 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543600+0000", 2024-09-18T04:40:18.211 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.1390000000000001e-06 2024-09-18T04:40:18.211 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.211 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.211 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:18.211 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543724+0000", 2024-09-18T04:40:18.211 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012458400000000001 2024-09-18T04:40:18.211 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.211 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.211 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:18.211 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543968+0000", 2024-09-18T04:40:18.212 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00024323199999999999 2024-09-18T04:40:18.212 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.212 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.212 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:18.212 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.544604+0000", 2024-09-18T04:40:18.212 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00063675799999999996 2024-09-18T04:40:18.212 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:18.212 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:18.212 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:18.212 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.212 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.214 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "MOSDPGPush(3.3 97/96 [PushOp(3:c4c92e5a:::rbd_trash:head, version: 53'1, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:c4c92e5a:::rbd_trash:head@53'1, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-18T04:40:18.214 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:40:05.833692+0000", 2024-09-18T04:40:18.214 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 12.36360962, 2024-09-18T04:40:18.214 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00061637300000000003, 2024-09-18T04:40:18.214 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:18.214 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:40:18.214 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:18.214 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.214 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:18.214 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:05.833692+0000", 2024-09-18T04:40:18.214 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:18.215 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.215 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.215 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:18.215 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:05.833692+0000", 2024-09-18T04:40:18.215 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:18.215 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.215 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.215 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:18.215 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:05.833691+0000", 2024-09-18T04:40:18.215 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:40:18.215 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.216 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.216 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:18.216 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:05.833709+0000", 2024-09-18T04:40:18.216 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7838000000000001e-05 2024-09-18T04:40:18.216 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.216 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.216 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:18.216 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:05.833745+0000", 2024-09-18T04:40:18.216 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5651000000000001e-05 2024-09-18T04:40:18.216 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.216 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.216 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:18.217 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:05.833763+0000", 2024-09-18T04:40:18.217 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8428e-05 2024-09-18T04:40:18.217 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.217 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.217 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:18.217 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:05.833832+0000", 2024-09-18T04:40:18.217 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8919999999999997e-05 2024-09-18T04:40:18.217 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.217 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.217 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:18.217 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:05.833845+0000", 2024-09-18T04:40:18.218 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2680000000000001e-05 2024-09-18T04:40:18.218 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.218 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.218 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:18.218 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:05.834308+0000", 2024-09-18T04:40:18.218 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00046391200000000003 2024-09-18T04:40:18.218 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:18.218 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:18.218 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:18.218 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.218 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.219 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "MOSDPGPush(3.3 97/96 [PushOp(3:d80af3cb:::rbd_id.testrbdnbd17342:head, version: 69'2, data_included: [0~16], data_size: 16, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d80af3cb:::rbd_id.testrbdnbd17342:head@69'2, size: 16, copy_subset: [0~16], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:16, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-18T04:40:18.219 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:40:05.840323+0000", 2024-09-18T04:40:18.219 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 12.356978494, 2024-09-18T04:40:18.219 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00084282300000000003, 2024-09-18T04:40:18.219 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:18.219 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:40:18.219 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:18.219 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.219 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:18.219 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:05.840323+0000", 2024-09-18T04:40:18.219 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:18.219 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.220 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.220 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:18.220 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:05.840323+0000", 2024-09-18T04:40:18.220 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:18.220 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.220 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.220 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:18.220 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:05.840322+0000", 2024-09-18T04:40:18.220 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:40:18.220 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.220 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.221 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:18.221 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:05.840339+0000", 2024-09-18T04:40:18.221 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6767999999999999e-05 2024-09-18T04:40:18.221 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.221 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.221 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:18.221 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:05.840358+0000", 2024-09-18T04:40:18.221 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8946000000000001e-05 2024-09-18T04:40:18.221 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.221 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.221 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:18.221 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:05.840382+0000", 2024-09-18T04:40:18.222 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4408000000000001e-05 2024-09-18T04:40:18.222 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.222 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.222 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:18.222 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:05.840491+0000", 2024-09-18T04:40:18.222 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000108126 2024-09-18T04:40:18.222 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.222 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.222 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:18.222 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:05.840504+0000", 2024-09-18T04:40:18.222 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3519e-05 2024-09-18T04:40:18.223 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.223 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.223 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:18.223 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:05.841166+0000", 2024-09-18T04:40:18.223 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00066203699999999998 2024-09-18T04:40:18.223 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:18.223 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:18.223 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:18.223 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:18.223 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:18.223 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:40:18.225 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:40:18.226 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:40:18.220+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:40:18.325 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:40:18.320+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:40:18.377 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:40:18.377 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:40:18.377 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:40:18.377 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:40:18.387 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:40:18.394 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd -p rbd --namespace '' status testrbdnbd17342 2024-09-18T04:40:18.394 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep 'Watchers: none' 2024-09-18T04:40:18.426 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:40:18.424+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:40:18.466 INFO:tasks.workunit.client.0.smithi115.stdout:Watchers: none 2024-09-18T04:40:18.466 INFO:tasks.workunit.client.0.smithi115.stderr:+ break 2024-09-18T04:40:18.466 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd -p rbd --namespace '' snap purge testrbdnbd17342 2024-09-18T04:40:18.492 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:40:18.492 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:40:18.492 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:40:18.492 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:40:18.492 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:40:18.501 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:40:18.527 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:40:18.523+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:40:18.628 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:40:18.624+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:40:18.657 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:40:18.657 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:40:18.657 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:40:18.657 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:40:18.657 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.657 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-18T04:40:18.657 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:40:18.657 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 164.18299195500001, 2024-09-18T04:40:18.658 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0095781189999999995, 2024-09-18T04:40:18.658 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:18.658 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:18.658 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:18.658 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:40:18.658 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:40:18.658 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 1 2024-09-18T04:40:18.658 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.658 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:18.658 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.659 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:18.659 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:40:18.659 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:18.659 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.659 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.659 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:18.659 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:40:18.659 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:18.659 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.659 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.659 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:18.660 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469862+0000", 2024-09-18T04:40:18.660 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:40:18.660 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.660 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.660 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:18.660 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469879+0000", 2024-09-18T04:40:18.660 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7431999999999998e-05 2024-09-18T04:40:18.660 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.660 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.660 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:18.660 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469896+0000", 2024-09-18T04:40:18.660 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6742999999999999e-05 2024-09-18T04:40:18.661 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.661 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.661 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:18.661 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469915+0000", 2024-09-18T04:40:18.661 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8991000000000002e-05 2024-09-18T04:40:18.661 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.661 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.661 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:18.661 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470397+0000", 2024-09-18T04:40:18.661 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048167799999999998 2024-09-18T04:40:18.661 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.661 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.661 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-09-18T04:40:18.662 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470412+0000", 2024-09-18T04:40:18.662 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5526e-05 2024-09-18T04:40:18.662 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.662 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.662 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:18.662 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473279+0000", 2024-09-18T04:40:18.662 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0028667469999999998 2024-09-18T04:40:18.662 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.662 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.662 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:18.662 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473441+0000", 2024-09-18T04:40:18.662 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000162082 2024-09-18T04:40:18.662 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.662 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.663 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:40:18.663 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473573+0000", 2024-09-18T04:40:18.663 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000132032 2024-09-18T04:40:18.663 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.663 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.663 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:18.663 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474826+0000", 2024-09-18T04:40:18.663 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001252717 2024-09-18T04:40:18.663 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.663 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.663 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:18.663 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479307+0000", 2024-09-18T04:40:18.663 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044813220000000003 2024-09-18T04:40:18.664 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.664 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.664 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:18.664 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479366+0000", 2024-09-18T04:40:18.664 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9024999999999998e-05 2024-09-18T04:40:18.664 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.664 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.664 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:18.664 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479443+0000", 2024-09-18T04:40:18.664 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6339000000000002e-05 2024-09-18T04:40:18.664 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:18.664 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:18.664 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:18.665 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.665 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.665 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-18T04:40:18.665 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:40:18.665 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 162.169983535, 2024-09-18T04:40:18.665 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082699560000000002, 2024-09-18T04:40:18.665 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:18.665 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:18.665 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:18.665 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:40:18.665 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:40:18.665 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 7 2024-09-18T04:40:18.666 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.666 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:18.666 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.666 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:18.666 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:40:18.666 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:18.666 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.666 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.666 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:18.666 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:40:18.666 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:18.666 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.667 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.667 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:18.667 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482870+0000", 2024-09-18T04:40:18.667 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999967 2024-09-18T04:40:18.667 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.667 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.667 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:18.667 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482889+0000", 2024-09-18T04:40:18.667 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.965e-05 2024-09-18T04:40:18.667 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.667 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.667 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:18.667 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482938+0000", 2024-09-18T04:40:18.668 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8548999999999999e-05 2024-09-18T04:40:18.668 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.668 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.668 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:18.668 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482963+0000", 2024-09-18T04:40:18.668 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5075999999999998e-05 2024-09-18T04:40:18.668 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.668 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.668 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:18.668 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483121+0000", 2024-09-18T04:40:18.668 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015819400000000001 2024-09-18T04:40:18.668 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.668 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.668 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op must wait for map", 2024-09-18T04:40:18.669 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483138+0000", 2024-09-18T04:40:18.669 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7068999999999999e-05 2024-09-18T04:40:18.669 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.669 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.669 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:18.669 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488666+0000", 2024-09-18T04:40:18.669 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0055282049999999996 2024-09-18T04:40:18.669 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.669 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.669 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:18.669 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488783+0000", 2024-09-18T04:40:18.669 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000116737 2024-09-18T04:40:18.669 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.669 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.670 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:18.670 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488991+0000", 2024-09-18T04:40:18.670 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020781400000000001 2024-09-18T04:40:18.670 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.670 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.670 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:18.670 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490554+0000", 2024-09-18T04:40:18.670 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0015628370000000001 2024-09-18T04:40:18.670 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.670 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.670 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:18.670 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491019+0000", 2024-09-18T04:40:18.670 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000465608 2024-09-18T04:40:18.671 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.671 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.671 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:18.671 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491087+0000", 2024-09-18T04:40:18.671 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7988e-05 2024-09-18T04:40:18.671 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.671 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.671 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:18.671 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491143+0000", 2024-09-18T04:40:18.671 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5538999999999998e-05 2024-09-18T04:40:18.671 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:18.671 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:18.672 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:18.672 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.672 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.672 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:34 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:18.672 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.261758+0000", 2024-09-18T04:40:18.672 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 96.391098384000003, 2024-09-18T04:40:18.672 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.042024191000000002, 2024-09-18T04:40:18.672 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:18.672 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:18.672 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:18.673 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:40:18.673 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:40:18.673 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 34 2024-09-18T04:40:18.673 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.673 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:18.673 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.673 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:18.673 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.261758+0000", 2024-09-18T04:40:18.673 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:18.673 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.673 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.673 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:18.673 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.261758+0000", 2024-09-18T04:40:18.673 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:18.674 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.674 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.674 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:18.674 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.261756+0000", 2024-09-18T04:40:18.674 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:18.674 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.674 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.674 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:18.674 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.270785+0000", 2024-09-18T04:40:18.674 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0090292340000000006 2024-09-18T04:40:18.674 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.675 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.675 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:18.675 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.270890+0000", 2024-09-18T04:40:18.675 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000104541 2024-09-18T04:40:18.675 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.675 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.675 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:18.675 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.270991+0000", 2024-09-18T04:40:18.675 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000100778 2024-09-18T04:40:18.675 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.676 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.676 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:18.676 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.271157+0000", 2024-09-18T04:40:18.676 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016653999999999999 2024-09-18T04:40:18.676 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.676 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.676 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:18.676 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.271591+0000", 2024-09-18T04:40:18.676 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00043409500000000001 2024-09-18T04:40:18.676 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.677 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.677 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:40:18.677 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272017+0000", 2024-09-18T04:40:18.677 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000425769 2024-09-18T04:40:18.677 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.677 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.677 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:18.677 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.294246+0000", 2024-09-18T04:40:18.677 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.022229237999999998 2024-09-18T04:40:18.677 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.678 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.678 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:18.678 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303571+0000", 2024-09-18T04:40:18.678 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0093250239999999995 2024-09-18T04:40:18.678 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.678 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.678 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:18.678 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303705+0000", 2024-09-18T04:40:18.678 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000133831 2024-09-18T04:40:18.678 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.679 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.679 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:18.679 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303782+0000", 2024-09-18T04:40:18.679 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7156000000000005e-05 2024-09-18T04:40:18.679 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:18.679 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:18.679 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:18.679 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.679 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.679 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:36 2.7 e51/32 2:fe1b02ad:::rbd_data.125fc09df6d1.0000000000000002:head v 51'1, mlcod=0'0)", 2024-09-18T04:40:18.679 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.273441+0000", 2024-09-18T04:40:18.680 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 96.379415402999996, 2024-09-18T04:40:18.680 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.032400768000000003, 2024-09-18T04:40:18.680 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:18.680 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:18.680 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:18.680 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.680 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:18.680 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273441+0000", 2024-09-18T04:40:18.680 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:18.680 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.681 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.681 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:18.681 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273441+0000", 2024-09-18T04:40:18.681 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:18.681 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.681 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.681 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:18.681 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273440+0000", 2024-09-18T04:40:18.681 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:40:18.681 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.681 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.682 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:18.682 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278723+0000", 2024-09-18T04:40:18.682 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0052829909999999999 2024-09-18T04:40:18.682 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.682 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.682 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:18.682 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278752+0000", 2024-09-18T04:40:18.682 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9178000000000001e-05 2024-09-18T04:40:18.682 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.682 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.683 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:18.683 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278761+0000", 2024-09-18T04:40:18.683 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7900000000000005e-06 2024-09-18T04:40:18.683 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.683 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.683 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:18.683 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278864+0000", 2024-09-18T04:40:18.683 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000102699 2024-09-18T04:40:18.683 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.683 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.683 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:18.683 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278900+0000", 2024-09-18T04:40:18.684 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6001999999999999e-05 2024-09-18T04:40:18.684 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.684 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.684 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:18.684 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.305793+0000", 2024-09-18T04:40:18.684 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.026893344 2024-09-18T04:40:18.684 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.684 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.684 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:18.684 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.305842+0000", 2024-09-18T04:40:18.684 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8720000000000001e-05 2024-09-18T04:40:18.685 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:18.685 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:18.685 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:18.685 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.685 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.685 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:39 2.7 e51/32 2:e9c909c5:::rbd_data.125fc09df6d1.0000000000000005:head v 51'2, mlcod=0'0)", 2024-09-18T04:40:18.685 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.283175+0000", 2024-09-18T04:40:18.685 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 96.369681713999995, 2024-09-18T04:40:18.685 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.022864102000000001, 2024-09-18T04:40:18.685 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:18.685 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:18.686 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:18.686 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.686 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:18.686 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283175+0000", 2024-09-18T04:40:18.686 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:18.686 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.686 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.686 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:18.686 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283175+0000", 2024-09-18T04:40:18.686 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:18.686 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.687 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.687 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:18.687 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283173+0000", 2024-09-18T04:40:18.687 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:18.687 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.687 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.687 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:18.687 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.285951+0000", 2024-09-18T04:40:18.687 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0027784540000000001 2024-09-18T04:40:18.687 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.688 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.688 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:18.688 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.285970+0000", 2024-09-18T04:40:18.688 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8855000000000002e-05 2024-09-18T04:40:18.688 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.688 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.688 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:18.688 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.285984+0000", 2024-09-18T04:40:18.688 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4252e-05 2024-09-18T04:40:18.688 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.689 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.689 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:18.689 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.286041+0000", 2024-09-18T04:40:18.689 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6431999999999998e-05 2024-09-18T04:40:18.689 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.689 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.689 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:18.689 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.286086+0000", 2024-09-18T04:40:18.689 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5423000000000002e-05 2024-09-18T04:40:18.689 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.689 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.689 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:18.689 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.305998+0000", 2024-09-18T04:40:18.690 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019911697999999999 2024-09-18T04:40:18.690 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.690 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.690 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:18.690 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306039+0000", 2024-09-18T04:40:18.690 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0973999999999998e-05 2024-09-18T04:40:18.690 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:18.690 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:18.690 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:18.690 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.690 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.690 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:50 2.3 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:18.690 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.312563+0000", 2024-09-18T04:40:18.690 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 96.340293686999999, 2024-09-18T04:40:18.690 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.032636001999999997, 2024-09-18T04:40:18.690 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:18.691 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:18.691 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:18.691 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:40:18.691 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:40:18.691 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 50 2024-09-18T04:40:18.691 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.691 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:18.691 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.691 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:18.691 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.312563+0000", 2024-09-18T04:40:18.691 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:18.691 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.691 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.691 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:18.691 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.312563+0000", 2024-09-18T04:40:18.692 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:18.692 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.692 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.692 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:18.692 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.312561+0000", 2024-09-18T04:40:18.692 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:18.692 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.692 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.692 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:18.692 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321709+0000", 2024-09-18T04:40:18.692 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0091486650000000003 2024-09-18T04:40:18.692 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.692 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.692 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:18.692 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321739+0000", 2024-09-18T04:40:18.693 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9666e-05 2024-09-18T04:40:18.693 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.693 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.693 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:18.693 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321751+0000", 2024-09-18T04:40:18.693 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2086000000000001e-05 2024-09-18T04:40:18.693 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.693 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.693 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:18.693 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321833+0000", 2024-09-18T04:40:18.693 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1842999999999996e-05 2024-09-18T04:40:18.693 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.693 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.693 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:18.694 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321981+0000", 2024-09-18T04:40:18.694 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000147441 2024-09-18T04:40:18.694 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.694 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.694 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:40:18.694 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.322109+0000", 2024-09-18T04:40:18.694 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012851200000000001 2024-09-18T04:40:18.694 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.694 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.694 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:18.694 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.332829+0000", 2024-09-18T04:40:18.694 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010720225999999999 2024-09-18T04:40:18.695 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.695 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.695 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:18.695 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.345039+0000", 2024-09-18T04:40:18.695 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012209831000000001 2024-09-18T04:40:18.695 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.695 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.695 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:18.695 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.345120+0000", 2024-09-18T04:40:18.695 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1145999999999999e-05 2024-09-18T04:40:18.695 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.695 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.696 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:18.696 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.345199+0000", 2024-09-18T04:40:18.696 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8546000000000003e-05 2024-09-18T04:40:18.696 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:18.696 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:18.696 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:18.696 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.696 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.696 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:30 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:18.696 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.516122+0000", 2024-09-18T04:40:18.696 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 96.136734524999994, 2024-09-18T04:40:18.696 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0091865030000000004, 2024-09-18T04:40:18.696 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:18.697 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:18.697 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:18.697 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:40:18.697 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:18.697 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 30 2024-09-18T04:40:18.697 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.697 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:18.697 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.697 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:18.697 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516122+0000", 2024-09-18T04:40:18.697 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:18.697 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.697 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.698 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:18.698 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516122+0000", 2024-09-18T04:40:18.698 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:18.698 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.698 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.698 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:18.698 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516120+0000", 2024-09-18T04:40:18.698 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:18.698 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.698 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.698 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:18.698 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516129+0000", 2024-09-18T04:40:18.699 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.85e-06 2024-09-18T04:40:18.699 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.699 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.699 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:18.699 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516155+0000", 2024-09-18T04:40:18.699 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5882000000000001e-05 2024-09-18T04:40:18.699 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.699 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.699 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:18.699 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516175+0000", 2024-09-18T04:40:18.699 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.022e-05 2024-09-18T04:40:18.699 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.700 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.700 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:18.700 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519780+0000", 2024-09-18T04:40:18.700 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003604187 2024-09-18T04:40:18.700 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.700 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.700 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:18.700 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519964+0000", 2024-09-18T04:40:18.700 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00018391999999999999 2024-09-18T04:40:18.700 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.700 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.701 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:40:18.701 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.520794+0000", 2024-09-18T04:40:18.701 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00083078100000000005 2024-09-18T04:40:18.701 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.701 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.701 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:18.701 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.524129+0000", 2024-09-18T04:40:18.701 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033342419999999998 2024-09-18T04:40:18.701 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.701 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.701 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:18.701 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525239+0000", 2024-09-18T04:40:18.702 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001110778 2024-09-18T04:40:18.702 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.702 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.702 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:18.702 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525277+0000", 2024-09-18T04:40:18.702 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7596999999999999e-05 2024-09-18T04:40:18.702 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.702 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.702 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:18.702 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525308+0000", 2024-09-18T04:40:18.702 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.1535999999999998e-05 2024-09-18T04:40:18.703 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:18.703 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:18.703 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:18.703 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.703 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.703 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:32 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:18.703 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.519217+0000", 2024-09-18T04:40:18.703 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 96.133639204999994, 2024-09-18T04:40:18.703 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082266370000000002, 2024-09-18T04:40:18.703 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:18.703 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:18.704 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:18.704 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:40:18.704 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:18.704 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 32 2024-09-18T04:40:18.704 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.704 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:18.704 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.704 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:18.704 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519217+0000", 2024-09-18T04:40:18.704 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:18.704 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.705 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.705 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:18.705 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519217+0000", 2024-09-18T04:40:18.705 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:18.705 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.705 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.705 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:18.705 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519215+0000", 2024-09-18T04:40:18.705 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:18.705 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.706 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.706 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:18.706 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519229+0000", 2024-09-18T04:40:18.706 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3329999999999999e-05 2024-09-18T04:40:18.706 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.706 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.706 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:18.706 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519268+0000", 2024-09-18T04:40:18.706 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9533000000000002e-05 2024-09-18T04:40:18.706 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.706 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.707 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:18.707 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519285+0000", 2024-09-18T04:40:18.707 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6446000000000001e-05 2024-09-18T04:40:18.707 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.707 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.707 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:18.707 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.522358+0000", 2024-09-18T04:40:18.707 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030734310000000002 2024-09-18T04:40:18.707 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.707 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.707 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:18.708 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.522551+0000", 2024-09-18T04:40:18.708 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019317600000000001 2024-09-18T04:40:18.708 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.708 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.708 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:40:18.708 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.523262+0000", 2024-09-18T04:40:18.708 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00071069999999999998 2024-09-18T04:40:18.708 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.708 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.708 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:18.708 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525318+0000", 2024-09-18T04:40:18.708 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0020561469999999999 2024-09-18T04:40:18.708 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.709 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.709 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:18.709 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.527376+0000", 2024-09-18T04:40:18.709 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0020584969999999998 2024-09-18T04:40:18.709 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.709 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.709 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:18.709 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.527416+0000", 2024-09-18T04:40:18.709 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9054000000000003e-05 2024-09-18T04:40:18.709 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.709 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.709 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:18.709 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.527444+0000", 2024-09-18T04:40:18.709 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8347000000000002e-05 2024-09-18T04:40:18.710 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:18.710 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:18.710 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:18.710 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.710 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.710 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:42 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:18.710 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.551160+0000", 2024-09-18T04:40:18.710 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 96.101696813000004, 2024-09-18T04:40:18.710 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010692757000000001, 2024-09-18T04:40:18.711 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:18.711 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:18.711 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:18.711 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:40:18.711 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:18.711 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 42 2024-09-18T04:40:18.711 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.711 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:18.711 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.711 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:18.711 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551160+0000", 2024-09-18T04:40:18.711 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:18.712 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.712 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.712 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:18.712 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551160+0000", 2024-09-18T04:40:18.712 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:18.712 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.712 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.712 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:18.712 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551159+0000", 2024-09-18T04:40:18.712 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:40:18.712 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.712 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.713 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:18.713 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551166+0000", 2024-09-18T04:40:18.713 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1029999999999996e-06 2024-09-18T04:40:18.713 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.713 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.713 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:18.713 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551177+0000", 2024-09-18T04:40:18.713 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1384e-05 2024-09-18T04:40:18.713 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.713 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.713 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:18.713 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551185+0000", 2024-09-18T04:40:18.714 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1990000000000006e-06 2024-09-18T04:40:18.714 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.714 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.714 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:18.714 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552021+0000", 2024-09-18T04:40:18.714 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00083574400000000003 2024-09-18T04:40:18.714 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.714 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.714 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:18.714 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552339+0000", 2024-09-18T04:40:18.714 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031771399999999997 2024-09-18T04:40:18.715 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.715 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.715 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:40:18.715 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.553140+0000", 2024-09-18T04:40:18.715 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00080088100000000003 2024-09-18T04:40:18.715 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.715 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.715 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:18.715 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.554585+0000", 2024-09-18T04:40:18.715 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001445801 2024-09-18T04:40:18.715 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.716 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.716 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:18.716 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.561711+0000", 2024-09-18T04:40:18.716 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071260109999999998 2024-09-18T04:40:18.716 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.716 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.716 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:18.716 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.561790+0000", 2024-09-18T04:40:18.716 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8967000000000005e-05 2024-09-18T04:40:18.716 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.717 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.717 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:18.717 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.561852+0000", 2024-09-18T04:40:18.717 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2045999999999995e-05 2024-09-18T04:40:18.717 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:18.717 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:18.717 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:18.717 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.717 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.717 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:58 2.7 e51/32 2:fc7c6cd3:ns::rbd_data.133e813ab5bc.000000000000000e:head v 51'7, mlcod=51'5)", 2024-09-18T04:40:18.717 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.605333+0000", 2024-09-18T04:40:18.718 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 96.047523608000006, 2024-09-18T04:40:18.718 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020085902999999999, 2024-09-18T04:40:18.718 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:18.718 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:18.718 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:18.718 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.718 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:18.718 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605333+0000", 2024-09-18T04:40:18.718 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:18.718 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.719 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.719 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:18.719 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605333+0000", 2024-09-18T04:40:18.719 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:18.719 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.719 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.719 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:18.719 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605332+0000", 2024-09-18T04:40:18.719 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:18.719 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.719 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.719 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:18.719 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608667+0000", 2024-09-18T04:40:18.720 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033350300000000001 2024-09-18T04:40:18.720 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.720 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.720 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:18.720 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608706+0000", 2024-09-18T04:40:18.720 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8951e-05 2024-09-18T04:40:18.720 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.720 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.720 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:18.720 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608725+0000", 2024-09-18T04:40:18.720 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9675999999999999e-05 2024-09-18T04:40:18.720 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.720 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.721 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:18.721 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608798+0000", 2024-09-18T04:40:18.721 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3135999999999999e-05 2024-09-18T04:40:18.721 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.721 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.721 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:18.721 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608824+0000", 2024-09-18T04:40:18.721 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5284000000000001e-05 2024-09-18T04:40:18.721 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.721 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.721 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:18.721 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625370+0000", 2024-09-18T04:40:18.721 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016545975000000001 2024-09-18T04:40:18.721 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.722 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.722 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:18.722 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625419+0000", 2024-09-18T04:40:18.722 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9076999999999998e-05 2024-09-18T04:40:18.722 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:18.722 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:18.722 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:18.722 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.722 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.722 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:611 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:40:18.722 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.069561+0000", 2024-09-18T04:40:18.722 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 87.583295054000004, 2024-09-18T04:40:18.722 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0072330299999999997, 2024-09-18T04:40:18.722 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:18.722 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:18.723 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:18.723 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:40:18.723 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:40:18.723 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 611 2024-09-18T04:40:18.723 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.723 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:18.723 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.723 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:18.723 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.069561+0000", 2024-09-18T04:40:18.723 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:18.723 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.723 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.723 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:18.724 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.069561+0000", 2024-09-18T04:40:18.724 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:18.724 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.724 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.724 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:18.724 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.069559+0000", 2024-09-18T04:40:18.724 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:40:18.724 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.724 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.724 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:18.724 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070801+0000", 2024-09-18T04:40:18.724 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0012423359999999999 2024-09-18T04:40:18.724 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.724 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.725 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:18.725 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070828+0000", 2024-09-18T04:40:18.725 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6387e-05 2024-09-18T04:40:18.725 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.725 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.725 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:18.725 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070840+0000", 2024-09-18T04:40:18.725 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2689e-05 2024-09-18T04:40:18.725 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.725 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.725 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:18.725 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070990+0000", 2024-09-18T04:40:18.725 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014946800000000001 2024-09-18T04:40:18.725 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.726 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.726 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:18.726 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.071128+0000", 2024-09-18T04:40:18.726 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013796699999999999 2024-09-18T04:40:18.726 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.726 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.726 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:40:18.726 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.071318+0000", 2024-09-18T04:40:18.726 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000190106 2024-09-18T04:40:18.726 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.726 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.726 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:18.727 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.075711+0000", 2024-09-18T04:40:18.727 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.004393002 2024-09-18T04:40:18.727 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.727 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.727 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:18.727 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.076665+0000", 2024-09-18T04:40:18.727 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00095390399999999999 2024-09-18T04:40:18.727 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.727 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.727 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:18.727 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.076742+0000", 2024-09-18T04:40:18.727 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7112000000000003e-05 2024-09-18T04:40:18.727 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.727 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.728 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:18.728 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.076794+0000", 2024-09-18T04:40:18.728 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2475999999999997e-05 2024-09-18T04:40:18.728 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:18.728 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:18.728 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:18.728 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.728 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.728 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:687 2.3 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:40:18.728 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.487994+0000", 2024-09-18T04:40:18.728 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 87.164862071000002, 2024-09-18T04:40:18.728 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0074167690000000001, 2024-09-18T04:40:18.728 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:18.729 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:18.729 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:18.729 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:40:18.729 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:40:18.729 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 687 2024-09-18T04:40:18.729 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.729 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:18.729 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.729 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:18.729 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.487994+0000", 2024-09-18T04:40:18.729 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:18.729 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.729 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.729 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:18.729 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.487994+0000", 2024-09-18T04:40:18.730 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:18.730 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.730 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.730 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:18.730 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.487993+0000", 2024-09-18T04:40:18.730 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:40:18.730 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.730 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.730 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:18.730 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.488979+0000", 2024-09-18T04:40:18.731 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00098620700000000001 2024-09-18T04:40:18.731 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.731 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.731 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:18.731 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489002+0000", 2024-09-18T04:40:18.731 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2500000000000001e-05 2024-09-18T04:40:18.731 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.731 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.731 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:18.731 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489020+0000", 2024-09-18T04:40:18.731 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7691000000000001e-05 2024-09-18T04:40:18.732 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.732 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.732 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:18.732 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489091+0000", 2024-09-18T04:40:18.732 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1307e-05 2024-09-18T04:40:18.732 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.732 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.732 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:18.732 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489219+0000", 2024-09-18T04:40:18.732 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012766000000000001 2024-09-18T04:40:18.732 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.732 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.733 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:40:18.733 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489387+0000", 2024-09-18T04:40:18.733 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016855400000000001 2024-09-18T04:40:18.733 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.733 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.733 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:18.733 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495243+0000", 2024-09-18T04:40:18.733 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058563189999999996 2024-09-18T04:40:18.733 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.733 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.733 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:18.734 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495266+0000", 2024-09-18T04:40:18.734 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2810999999999999e-05 2024-09-18T04:40:18.734 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.734 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.734 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:18.734 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495332+0000", 2024-09-18T04:40:18.734 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5469999999999995e-05 2024-09-18T04:40:18.734 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.734 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.734 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:18.734 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495411+0000", 2024-09-18T04:40:18.735 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9395999999999997e-05 2024-09-18T04:40:18.735 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:18.735 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:18.735 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:18.735 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.735 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.735 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:689 2.3 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:40:18.736 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.500070+0000", 2024-09-18T04:40:18.736 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 87.152786254999995, 2024-09-18T04:40:18.736 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071096650000000003, 2024-09-18T04:40:18.736 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:18.736 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:18.736 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:18.736 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:40:18.736 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:40:18.736 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 689 2024-09-18T04:40:18.736 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.737 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:18.737 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.737 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:18.737 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.500070+0000", 2024-09-18T04:40:18.737 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:18.737 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.737 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.737 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:18.737 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.500070+0000", 2024-09-18T04:40:18.737 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:18.738 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.738 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.738 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:18.738 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.500069+0000", 2024-09-18T04:40:18.738 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:18.738 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.738 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.738 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:18.738 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501082+0000", 2024-09-18T04:40:18.739 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010136570000000001 2024-09-18T04:40:18.739 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.739 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.739 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:18.739 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501104+0000", 2024-09-18T04:40:18.739 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1883e-05 2024-09-18T04:40:18.739 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.739 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.739 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:18.739 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501115+0000", 2024-09-18T04:40:18.740 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0414000000000001e-05 2024-09-18T04:40:18.740 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.740 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.740 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:18.740 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501178+0000", 2024-09-18T04:40:18.740 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3128999999999998e-05 2024-09-18T04:40:18.740 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.740 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.740 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:18.740 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501257+0000", 2024-09-18T04:40:18.740 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9592999999999995e-05 2024-09-18T04:40:18.741 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.741 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.741 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:40:18.741 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.501366+0000", 2024-09-18T04:40:18.741 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000108643 2024-09-18T04:40:18.741 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.741 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.741 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:18.741 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.504041+0000", 2024-09-18T04:40:18.741 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0026750889999999999 2024-09-18T04:40:18.741 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.742 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.742 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:18.742 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.507097+0000", 2024-09-18T04:40:18.742 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030556070000000001 2024-09-18T04:40:18.742 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.742 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.742 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:18.742 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.507140+0000", 2024-09-18T04:40:18.742 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3745000000000002e-05 2024-09-18T04:40:18.742 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.742 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.742 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:18.743 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.507180+0000", 2024-09-18T04:40:18.743 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9475999999999999e-05 2024-09-18T04:40:18.743 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:18.743 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:18.743 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:18.743 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.743 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.743 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4972.0:13 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:40:18.743 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.948263+0000", 2024-09-18T04:40:18.743 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 86.704593463999998, 2024-09-18T04:40:18.743 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0091358970000000005, 2024-09-18T04:40:18.743 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:18.744 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:40:18.744 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:18.744 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4972", 2024-09-18T04:40:18.744 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2280251472", 2024-09-18T04:40:18.744 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 13 2024-09-18T04:40:18.744 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.744 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:18.744 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.744 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:18.744 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948263+0000", 2024-09-18T04:40:18.744 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:18.745 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.745 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.745 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:18.745 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948263+0000", 2024-09-18T04:40:18.745 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:18.745 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.745 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.745 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:18.745 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948261+0000", 2024-09-18T04:40:18.745 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:18.746 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.746 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.746 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:18.746 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948273+0000", 2024-09-18T04:40:18.746 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1576999999999999e-05 2024-09-18T04:40:18.746 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.746 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.746 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:18.746 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948305+0000", 2024-09-18T04:40:18.746 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2209000000000003e-05 2024-09-18T04:40:18.746 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.746 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.747 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:18.747 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948325+0000", 2024-09-18T04:40:18.747 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.022e-05 2024-09-18T04:40:18.747 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.747 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.747 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:18.747 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948455+0000", 2024-09-18T04:40:18.747 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000129624 2024-09-18T04:40:18.747 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.747 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.747 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:18.747 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948598+0000", 2024-09-18T04:40:18.747 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014376699999999999 2024-09-18T04:40:18.748 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.748 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.748 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:18.748 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.957399+0000", 2024-09-18T04:40:18.748 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0088004299999999997 2024-09-18T04:40:18.748 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:18.748 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:18.748 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:18.748 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.748 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.748 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:101 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:40:18.749 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.576367+0000", 2024-09-18T04:40:18.749 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 82.076489756000001, 2024-09-18T04:40:18.749 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0078671020000000008, 2024-09-18T04:40:18.749 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:18.749 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:18.749 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:18.749 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:40:18.749 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:40:18.749 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 101 2024-09-18T04:40:18.749 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.749 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:18.749 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.749 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:18.749 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576367+0000", 2024-09-18T04:40:18.750 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:18.750 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.750 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.750 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:18.750 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576367+0000", 2024-09-18T04:40:18.750 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:18.750 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.750 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.750 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:18.750 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576364+0000", 2024-09-18T04:40:18.750 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999971 2024-09-18T04:40:18.750 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.750 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.750 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:18.750 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576378+0000", 2024-09-18T04:40:18.751 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3879999999999999e-05 2024-09-18T04:40:18.751 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.751 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.751 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:18.751 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576410+0000", 2024-09-18T04:40:18.751 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2286000000000002e-05 2024-09-18T04:40:18.751 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.751 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.751 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:18.751 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576425+0000", 2024-09-18T04:40:18.751 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5290000000000001e-05 2024-09-18T04:40:18.751 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.751 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.752 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:18.752 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576551+0000", 2024-09-18T04:40:18.752 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012555199999999999 2024-09-18T04:40:18.752 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.752 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.752 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:18.752 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576719+0000", 2024-09-18T04:40:18.752 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016787400000000001 2024-09-18T04:40:18.752 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.752 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.752 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:40:18.752 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576951+0000", 2024-09-18T04:40:18.752 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00023252400000000001 2024-09-18T04:40:18.753 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.753 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.753 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:18.753 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.581459+0000", 2024-09-18T04:40:18.753 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045074410000000001 2024-09-18T04:40:18.753 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.753 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.753 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:18.753 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584076+0000", 2024-09-18T04:40:18.753 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0026173860000000002 2024-09-18T04:40:18.753 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.753 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.753 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:18.753 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584160+0000", 2024-09-18T04:40:18.754 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4191999999999996e-05 2024-09-18T04:40:18.754 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.754 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.754 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:18.754 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584234+0000", 2024-09-18T04:40:18.754 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.339e-05 2024-09-18T04:40:18.754 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:18.754 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:18.754 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:18.754 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.754 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.754 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:44 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:40:18.754 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.550448+0000", 2024-09-18T04:40:18.755 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 78.102408566999998, 2024-09-18T04:40:18.755 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082117590000000008, 2024-09-18T04:40:18.755 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:18.755 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:18.755 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:18.755 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:40:18.755 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:40:18.755 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 44 2024-09-18T04:40:18.755 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.755 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:18.755 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.755 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:18.755 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.550448+0000", 2024-09-18T04:40:18.755 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:18.756 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.756 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.756 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:18.756 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.550448+0000", 2024-09-18T04:40:18.756 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:18.756 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.756 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.756 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:18.756 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.550445+0000", 2024-09-18T04:40:18.756 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999971 2024-09-18T04:40:18.756 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.756 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.756 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:18.757 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.551904+0000", 2024-09-18T04:40:18.757 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001458322 2024-09-18T04:40:18.757 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.757 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.757 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:18.757 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.551938+0000", 2024-09-18T04:40:18.757 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4359000000000001e-05 2024-09-18T04:40:18.757 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.757 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.757 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:18.757 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.551959+0000", 2024-09-18T04:40:18.757 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1403000000000002e-05 2024-09-18T04:40:18.757 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.758 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.758 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:18.758 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.552049+0000", 2024-09-18T04:40:18.758 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9970999999999996e-05 2024-09-18T04:40:18.758 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.758 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.758 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:18.758 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.552190+0000", 2024-09-18T04:40:18.758 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000140271 2024-09-18T04:40:18.758 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.758 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.759 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:40:18.759 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.552385+0000", 2024-09-18T04:40:18.759 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019498700000000001 2024-09-18T04:40:18.759 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.759 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.759 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:18.759 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.556308+0000", 2024-09-18T04:40:18.759 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00392333 2024-09-18T04:40:18.759 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.759 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.759 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:18.760 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.558479+0000", 2024-09-18T04:40:18.760 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0021715889999999998 2024-09-18T04:40:18.760 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.760 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.760 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:18.760 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.558616+0000", 2024-09-18T04:40:18.760 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013613399999999999 2024-09-18T04:40:18.760 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.760 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.760 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:18.761 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.558660+0000", 2024-09-18T04:40:18.761 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4053e-05 2024-09-18T04:40:18.761 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:18.761 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:18.761 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:18.761 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.761 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.761 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:100 2.4 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:40:18.761 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.856389+0000", 2024-09-18T04:40:18.761 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 77.796467325999998, 2024-09-18T04:40:18.761 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0081480800000000003, 2024-09-18T04:40:18.762 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:18.762 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:18.762 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:18.762 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:40:18.762 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:40:18.762 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 100 2024-09-18T04:40:18.762 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.762 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:18.762 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.763 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:18.763 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.856389+0000", 2024-09-18T04:40:18.763 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:18.763 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.763 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.763 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:18.763 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.856389+0000", 2024-09-18T04:40:18.763 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:18.763 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.763 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.764 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:18.764 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.856388+0000", 2024-09-18T04:40:18.764 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:40:18.764 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.764 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.764 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:18.764 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857295+0000", 2024-09-18T04:40:18.764 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00090725599999999995 2024-09-18T04:40:18.764 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.764 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.764 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:18.765 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857311+0000", 2024-09-18T04:40:18.765 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6092000000000001e-05 2024-09-18T04:40:18.765 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.765 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.765 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:18.765 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857319+0000", 2024-09-18T04:40:18.765 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9200000000000004e-06 2024-09-18T04:40:18.765 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.765 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.765 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:18.765 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857388+0000", 2024-09-18T04:40:18.765 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8591999999999997e-05 2024-09-18T04:40:18.766 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.766 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.766 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:18.766 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857535+0000", 2024-09-18T04:40:18.766 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014684299999999999 2024-09-18T04:40:18.766 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.766 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.766 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:40:18.766 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857747+0000", 2024-09-18T04:40:18.766 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000212361 2024-09-18T04:40:18.766 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.767 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.767 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:18.767 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.860831+0000", 2024-09-18T04:40:18.767 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030841789999999998 2024-09-18T04:40:18.767 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.767 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.767 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:18.767 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.864361+0000", 2024-09-18T04:40:18.767 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0035298780000000002 2024-09-18T04:40:18.767 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.768 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.768 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:18.768 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.864456+0000", 2024-09-18T04:40:18.768 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4557999999999998e-05 2024-09-18T04:40:18.768 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.768 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.768 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:18.768 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.864537+0000", 2024-09-18T04:40:18.768 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1577999999999997e-05 2024-09-18T04:40:18.768 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:18.769 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:18.769 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:18.769 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.769 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.769 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:102 2.4 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:40:18.769 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.869773+0000", 2024-09-18T04:40:18.769 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 77.783083669000007, 2024-09-18T04:40:18.769 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0084430359999999992, 2024-09-18T04:40:18.769 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:18.770 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:18.770 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:18.770 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:40:18.770 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:40:18.770 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 102 2024-09-18T04:40:18.770 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.770 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:18.770 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.770 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:18.770 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.869773+0000", 2024-09-18T04:40:18.770 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:18.771 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.771 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.771 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:18.771 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.869773+0000", 2024-09-18T04:40:18.771 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:18.771 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.771 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.771 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:18.771 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.869771+0000", 2024-09-18T04:40:18.771 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:18.771 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.771 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.771 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:18.772 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870629+0000", 2024-09-18T04:40:18.772 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00085760999999999997 2024-09-18T04:40:18.772 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.772 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.772 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:18.772 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870647+0000", 2024-09-18T04:40:18.772 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8363999999999999e-05 2024-09-18T04:40:18.772 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.772 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.772 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:18.772 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870655+0000", 2024-09-18T04:40:18.772 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0099999999999995e-06 2024-09-18T04:40:18.772 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.773 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.773 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:18.773 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870803+0000", 2024-09-18T04:40:18.773 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014798 2024-09-18T04:40:18.773 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.773 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.773 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:18.773 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870951+0000", 2024-09-18T04:40:18.773 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014715999999999999 2024-09-18T04:40:18.773 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.773 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.773 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:40:18.773 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.871145+0000", 2024-09-18T04:40:18.774 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000194253 2024-09-18T04:40:18.774 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.774 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.774 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:18.774 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.877033+0000", 2024-09-18T04:40:18.774 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058883099999999999 2024-09-18T04:40:18.774 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.774 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.774 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:18.774 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.878078+0000", 2024-09-18T04:40:18.774 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001044747 2024-09-18T04:40:18.775 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.775 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.775 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:18.775 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.878160+0000", 2024-09-18T04:40:18.775 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1904999999999996e-05 2024-09-18T04:40:18.775 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.775 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.775 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:18.775 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.878216+0000", 2024-09-18T04:40:18.775 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6082000000000003e-05 2024-09-18T04:40:18.775 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:18.776 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:18.776 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:18.776 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.776 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.776 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:145 2.4 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:40:18.776 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:01.096935+0000", 2024-09-18T04:40:18.776 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 77.555921889000004, 2024-09-18T04:40:18.776 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0069937869999999996, 2024-09-18T04:40:18.776 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:18.776 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:18.776 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:18.777 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:40:18.777 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:40:18.777 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 145 2024-09-18T04:40:18.777 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.777 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:18.777 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.777 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:18.777 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.096935+0000", 2024-09-18T04:40:18.777 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:18.777 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.777 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.777 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:18.777 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.096935+0000", 2024-09-18T04:40:18.778 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:18.778 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.778 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.778 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:18.778 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.096932+0000", 2024-09-18T04:40:18.778 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:40:18.778 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.778 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.778 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:18.778 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097815+0000", 2024-09-18T04:40:18.779 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00088276000000000001 2024-09-18T04:40:18.779 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.779 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.779 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:18.779 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097842+0000", 2024-09-18T04:40:18.779 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6806999999999999e-05 2024-09-18T04:40:18.779 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.779 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.779 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:18.779 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097852+0000", 2024-09-18T04:40:18.779 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0548e-05 2024-09-18T04:40:18.779 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.780 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.780 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:18.780 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.097931+0000", 2024-09-18T04:40:18.780 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8417000000000005e-05 2024-09-18T04:40:18.780 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.780 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.780 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:18.780 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.098047+0000", 2024-09-18T04:40:18.780 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011611 2024-09-18T04:40:18.780 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.780 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.780 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:40:18.780 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.098172+0000", 2024-09-18T04:40:18.780 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000125482 2024-09-18T04:40:18.781 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.781 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.781 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:18.781 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.100888+0000", 2024-09-18T04:40:18.781 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0027154430000000001 2024-09-18T04:40:18.781 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.781 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.781 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:18.781 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.103814+0000", 2024-09-18T04:40:18.781 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0029259619999999998 2024-09-18T04:40:18.781 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.782 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.782 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:18.782 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.103891+0000", 2024-09-18T04:40:18.782 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7619000000000003e-05 2024-09-18T04:40:18.782 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.782 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.782 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:18.782 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:01.103928+0000", 2024-09-18T04:40:18.782 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6900999999999998e-05 2024-09-18T04:40:18.782 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:18.782 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:18.782 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:18.782 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.783 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.783 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.6061.0:189 2.3 2:d670d898:::journal.125fc09df6d1:head [watch watch cookie 140218200024176] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-18T04:40:18.783 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:40:14.880927+0000", 2024-09-18T04:40:18.783 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 3.7719291039999998, 2024-09-18T04:40:18.783 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0085105679999999996, 2024-09-18T04:40:18.783 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:18.783 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:18.783 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:18.783 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.6061", 2024-09-18T04:40:18.783 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2011291291", 2024-09-18T04:40:18.783 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 189 2024-09-18T04:40:18.783 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.783 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:18.783 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.784 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:18.784 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:14.880927+0000", 2024-09-18T04:40:18.784 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:18.784 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.784 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.784 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:18.784 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:14.880927+0000", 2024-09-18T04:40:18.784 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:18.784 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.784 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.784 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:18.784 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:14.880925+0000", 2024-09-18T04:40:18.784 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:18.785 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.785 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.785 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:18.785 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:14.880937+0000", 2024-09-18T04:40:18.785 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1659e-05 2024-09-18T04:40:18.785 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.785 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.785 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:18.785 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:14.880960+0000", 2024-09-18T04:40:18.785 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3113e-05 2024-09-18T04:40:18.785 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.785 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.785 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:18.786 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:14.880972+0000", 2024-09-18T04:40:18.786 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.222e-05 2024-09-18T04:40:18.786 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.786 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.786 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:18.786 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:14.881121+0000", 2024-09-18T04:40:18.786 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014843699999999999 2024-09-18T04:40:18.786 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.786 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.786 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op must wait for map", 2024-09-18T04:40:18.786 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:14.881136+0000", 2024-09-18T04:40:18.786 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5373e-05 2024-09-18T04:40:18.786 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.786 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.787 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:18.787 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:14.887284+0000", 2024-09-18T04:40:18.787 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0061473919999999998 2024-09-18T04:40:18.787 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.787 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.787 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:18.787 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:14.887438+0000", 2024-09-18T04:40:18.787 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000154117 2024-09-18T04:40:18.787 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.787 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.787 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:40:18.787 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:14.887918+0000", 2024-09-18T04:40:18.787 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048078599999999997 2024-09-18T04:40:18.787 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.788 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.788 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:18.788 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:14.889178+0000", 2024-09-18T04:40:18.788 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001259453 2024-09-18T04:40:18.788 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.788 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.788 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:18.788 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:14.889246+0000", 2024-09-18T04:40:18.788 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8125000000000003e-05 2024-09-18T04:40:18.788 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.788 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.788 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:18.789 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:14.889330+0000", 2024-09-18T04:40:18.789 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4011999999999995e-05 2024-09-18T04:40:18.789 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:18.789 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:18.789 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:18.789 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:14.889438+0000", 2024-09-18T04:40:18.789 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010788100000000001 2024-09-18T04:40:18.789 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:18.789 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:18.789 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:18.789 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:18.789 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:18.789 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:40:18.791 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:40:18.792 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:40:18.788+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:40:18.891 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:40:18.888+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:40:18.909 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:40:18.909 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:40:18.909 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:40:18.909 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:40:18.923 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:40:18.926 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-09-18T04:40:18.992 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:40:18.988+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:40:19.021 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:40:19.021 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:40:19.021 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:40:19.021 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:40:19.021 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:40:19.031 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:40:19.093 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:40:19.087+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:40:19.138 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:40:19.139 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:40:19.139 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:40:19.139 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:40:19.139 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.139 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:7, version: 79'62, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:7@79'62, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 8=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-18T04:40:19.139 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.724969+0000", 2024-09-18T04:40:19.139 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 32.408777203, 2024-09-18T04:40:19.139 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0035525930000000002, 2024-09-18T04:40:19.139 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:19.139 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:19.139 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:19.139 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.140 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:19.140 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.724969+0000", 2024-09-18T04:40:19.140 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.140 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.140 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.140 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:19.140 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.724969+0000", 2024-09-18T04:40:19.140 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.140 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.140 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.140 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:19.140 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.724968+0000", 2024-09-18T04:40:19.141 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:19.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.141 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.141 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:19.141 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.726245+0000", 2024-09-18T04:40:19.141 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001276395 2024-09-18T04:40:19.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.141 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.141 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:19.141 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.726262+0000", 2024-09-18T04:40:19.141 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6973000000000002e-05 2024-09-18T04:40:19.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.142 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.142 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:19.142 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.726293+0000", 2024-09-18T04:40:19.142 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0886999999999998e-05 2024-09-18T04:40:19.142 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.142 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.142 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:19.142 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.726403+0000", 2024-09-18T04:40:19.142 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000110288 2024-09-18T04:40:19.142 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.142 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.142 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:19.142 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.726418+0000", 2024-09-18T04:40:19.143 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5503999999999999e-05 2024-09-18T04:40:19.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.143 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.143 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:19.143 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.728522+0000", 2024-09-18T04:40:19.143 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021032630000000002 2024-09-18T04:40:19.143 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.143 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:19.143 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.143 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.144 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.1 87/86 [PushOp(2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head, version: 81'220, data_included: [4182016~12288], data_size: 12288, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head@81'220, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:4182016, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:40:19.144 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.730366+0000", 2024-09-18T04:40:19.144 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 32.403380806000001, 2024-09-18T04:40:19.144 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0032522580000000001, 2024-09-18T04:40:19.144 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:19.144 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:19.144 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:19.144 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.144 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:19.144 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.730366+0000", 2024-09-18T04:40:19.144 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.145 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.145 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.145 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:19.145 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.730366+0000", 2024-09-18T04:40:19.145 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.145 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.145 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.145 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:19.145 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.730365+0000", 2024-09-18T04:40:19.145 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:19.145 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.145 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.146 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:19.146 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.730385+0000", 2024-09-18T04:40:19.146 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0766000000000001e-05 2024-09-18T04:40:19.146 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.146 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.146 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:19.146 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.730406+0000", 2024-09-18T04:40:19.146 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0706000000000001e-05 2024-09-18T04:40:19.146 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.146 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.146 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:19.147 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.730423+0000", 2024-09-18T04:40:19.147 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7411e-05 2024-09-18T04:40:19.147 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.147 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.147 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:19.147 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.730507+0000", 2024-09-18T04:40:19.147 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3388999999999997e-05 2024-09-18T04:40:19.147 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.147 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.147 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:19.147 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.730523+0000", 2024-09-18T04:40:19.147 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5843999999999999e-05 2024-09-18T04:40:19.147 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.148 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.148 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:19.148 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.733618+0000", 2024-09-18T04:40:19.148 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030951540000000001 2024-09-18T04:40:19.148 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.148 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:19.148 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.148 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.148 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.148 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:7, version: 79'62, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:7@79'62, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 8=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:40:19.149 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.743433+0000", 2024-09-18T04:40:19.149 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 32.390313364999997, 2024-09-18T04:40:19.149 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003400237, 2024-09-18T04:40:19.149 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:19.149 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:19.149 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:19.149 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.149 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:19.149 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.743433+0000", 2024-09-18T04:40:19.149 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.149 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.149 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:19.150 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.743433+0000", 2024-09-18T04:40:19.150 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.150 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.150 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.150 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:19.150 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.743432+0000", 2024-09-18T04:40:19.150 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:19.150 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.150 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.150 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:19.150 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.744392+0000", 2024-09-18T04:40:19.150 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00095993499999999998 2024-09-18T04:40:19.150 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.151 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.151 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:19.151 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.744417+0000", 2024-09-18T04:40:19.151 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5249000000000002e-05 2024-09-18T04:40:19.151 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.151 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.151 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:19.151 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.744434+0000", 2024-09-18T04:40:19.151 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6597999999999999e-05 2024-09-18T04:40:19.151 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.151 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.151 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:19.151 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.744499+0000", 2024-09-18T04:40:19.151 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4987000000000006e-05 2024-09-18T04:40:19.152 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.152 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.152 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:19.152 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.744506+0000", 2024-09-18T04:40:19.152 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2679999999999997e-06 2024-09-18T04:40:19.152 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.152 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.152 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:19.152 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.746833+0000", 2024-09-18T04:40:19.152 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023270529999999999 2024-09-18T04:40:19.152 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.152 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:19.152 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.153 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.153 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.153 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:7, version: 79'64, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:7@79'64, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:40:19.153 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.769830+0000", 2024-09-18T04:40:19.153 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 32.363916811000003, 2024-09-18T04:40:19.153 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0040223419999999999, 2024-09-18T04:40:19.153 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:19.153 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:19.153 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:19.153 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.153 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:19.153 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.769830+0000", 2024-09-18T04:40:19.153 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.153 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.154 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.154 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:19.154 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.769830+0000", 2024-09-18T04:40:19.154 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.154 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.154 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.154 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:19.154 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.769829+0000", 2024-09-18T04:40:19.154 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:19.154 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.154 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.154 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:19.154 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.770812+0000", 2024-09-18T04:40:19.155 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00098347800000000009 2024-09-18T04:40:19.155 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.155 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.155 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:19.155 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.770834+0000", 2024-09-18T04:40:19.155 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1648999999999999e-05 2024-09-18T04:40:19.155 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.155 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.155 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:19.155 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.770847+0000", 2024-09-18T04:40:19.155 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3169999999999999e-05 2024-09-18T04:40:19.155 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.155 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.155 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:19.155 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.770969+0000", 2024-09-18T04:40:19.155 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000121721 2024-09-18T04:40:19.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.156 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.156 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:19.156 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.770989+0000", 2024-09-18T04:40:19.156 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9944000000000001e-05 2024-09-18T04:40:19.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.156 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.156 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:19.156 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.773852+0000", 2024-09-18T04:40:19.156 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028632319999999998 2024-09-18T04:40:19.156 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.156 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:19.156 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.157 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.157 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head, version: 81'75, data_included: [0~49152,1097728~999424], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head@81'75, size: 2146304, copy_subset: [0~2146304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-18T04:40:19.157 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.782052+0000", 2024-09-18T04:40:19.157 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 32.351694326000001, 2024-09-18T04:40:19.157 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0037389630000000001, 2024-09-18T04:40:19.157 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:19.157 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:19.157 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:19.157 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.157 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:19.158 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.782052+0000", 2024-09-18T04:40:19.158 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.158 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.158 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.158 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:19.158 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.782052+0000", 2024-09-18T04:40:19.158 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.158 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.158 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.158 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:19.158 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.782051+0000", 2024-09-18T04:40:19.158 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:19.158 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.158 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.158 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:19.159 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.783658+0000", 2024-09-18T04:40:19.159 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0016072149999999999 2024-09-18T04:40:19.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.159 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.159 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:19.159 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.783698+0000", 2024-09-18T04:40:19.159 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9394000000000003e-05 2024-09-18T04:40:19.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.159 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.159 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:19.159 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.783733+0000", 2024-09-18T04:40:19.159 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5428e-05 2024-09-18T04:40:19.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.160 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.160 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:19.160 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.783854+0000", 2024-09-18T04:40:19.160 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012067599999999999 2024-09-18T04:40:19.160 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.160 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.160 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:19.160 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.783869+0000", 2024-09-18T04:40:19.160 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5554999999999999e-05 2024-09-18T04:40:19.160 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.160 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.160 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:19.161 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.785791+0000", 2024-09-18T04:40:19.161 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001921629 2024-09-18T04:40:19.161 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.161 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:19.161 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.161 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.161 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.5732.0:3 2.1 2:9370a598:::rbd_header.125fc09df6d1:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e102)", 2024-09-18T04:40:19.161 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:40:13.100500+0000", 2024-09-18T04:40:19.161 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 6.033246331, 2024-09-18T04:40:19.161 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0035679359999999999, 2024-09-18T04:40:19.161 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:19.161 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:19.162 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:19.162 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.5732", 2024-09-18T04:40:19.162 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/1418224569", 2024-09-18T04:40:19.162 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 3 2024-09-18T04:40:19.162 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.162 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:19.162 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.162 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:19.162 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.100500+0000", 2024-09-18T04:40:19.162 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.163 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.163 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.163 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:19.163 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.100500+0000", 2024-09-18T04:40:19.163 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.163 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.163 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.163 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:19.163 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.100497+0000", 2024-09-18T04:40:19.163 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999971 2024-09-18T04:40:19.163 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.163 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.163 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:19.163 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.100511+0000", 2024-09-18T04:40:19.164 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3753e-05 2024-09-18T04:40:19.164 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.164 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.164 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:19.164 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.100534+0000", 2024-09-18T04:40:19.164 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3108999999999998e-05 2024-09-18T04:40:19.164 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.164 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.164 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:19.164 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.100555+0000", 2024-09-18T04:40:19.164 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.084e-05 2024-09-18T04:40:19.164 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.164 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.164 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:19.165 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.100645+0000", 2024-09-18T04:40:19.165 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9859999999999997e-05 2024-09-18T04:40:19.165 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.165 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.165 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-09-18T04:40:19.165 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.100652+0000", 2024-09-18T04:40:19.165 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4109999999999999e-06 2024-09-18T04:40:19.165 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.165 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.165 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:19.166 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.103598+0000", 2024-09-18T04:40:19.166 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0029458940000000001 2024-09-18T04:40:19.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.166 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.166 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:19.166 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.103731+0000", 2024-09-18T04:40:19.166 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013292999999999999 2024-09-18T04:40:19.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.166 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.166 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:19.166 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.104068+0000", 2024-09-18T04:40:19.166 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00033679199999999998 2024-09-18T04:40:19.166 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.166 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:19.167 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.167 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.167 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.167 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.5689.0:329 2.1 2:9370a598:::rbd_header.125fc09df6d1:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-18T04:40:19.167 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:40:13.281353+0000", 2024-09-18T04:40:19.167 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 5.8523933829999999, 2024-09-18T04:40:19.167 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0033936449999999998, 2024-09-18T04:40:19.167 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:19.167 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:19.167 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:19.167 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.5689", 2024-09-18T04:40:19.167 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/1538094101", 2024-09-18T04:40:19.168 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 329 2024-09-18T04:40:19.168 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.168 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:19.168 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.168 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:19.168 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.281353+0000", 2024-09-18T04:40:19.168 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.168 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.168 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.168 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:19.168 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.281353+0000", 2024-09-18T04:40:19.168 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.168 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.169 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.169 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:19.169 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.281350+0000", 2024-09-18T04:40:19.169 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:40:19.169 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.169 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.169 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:19.169 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.281364+0000", 2024-09-18T04:40:19.169 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3936e-05 2024-09-18T04:40:19.169 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.169 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.169 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:19.169 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.281399+0000", 2024-09-18T04:40:19.170 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4282000000000001e-05 2024-09-18T04:40:19.170 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.170 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.170 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:19.170 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.281416+0000", 2024-09-18T04:40:19.170 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7042999999999999e-05 2024-09-18T04:40:19.170 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.170 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.171 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:19.171 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.281571+0000", 2024-09-18T04:40:19.171 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015490999999999999 2024-09-18T04:40:19.171 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.171 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.172 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:19.172 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.281833+0000", 2024-09-18T04:40:19.172 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00026252399999999998 2024-09-18T04:40:19.172 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.172 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.173 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:19.173 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.282340+0000", 2024-09-18T04:40:19.173 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00050714200000000003 2024-09-18T04:40:19.173 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.173 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.173 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:19.173 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.283675+0000", 2024-09-18T04:40:19.173 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001334949 2024-09-18T04:40:19.173 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.173 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.173 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:19.173 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.284605+0000", 2024-09-18T04:40:19.174 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00093018099999999998 2024-09-18T04:40:19.174 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.174 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.174 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:19.174 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.284687+0000", 2024-09-18T04:40:19.174 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1417999999999999e-05 2024-09-18T04:40:19.174 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.174 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.174 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:19.174 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.284747+0000", 2024-09-18T04:40:19.174 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9865999999999999e-05 2024-09-18T04:40:19.174 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.174 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:19.175 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.175 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.175 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.175 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.5689.0:331 2.1 2:9370a598:::rbd_header.125fc09df6d1:head [watch unwatch cookie 140417983158384] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-18T04:40:19.175 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:40:13.285091+0000", 2024-09-18T04:40:19.175 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 5.8486549989999999, 2024-09-18T04:40:19.175 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0032654429999999998, 2024-09-18T04:40:19.175 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:19.175 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:19.175 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:19.175 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.5689", 2024-09-18T04:40:19.176 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/1538094101", 2024-09-18T04:40:19.176 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 331 2024-09-18T04:40:19.176 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.176 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:19.176 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.176 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:19.176 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.285091+0000", 2024-09-18T04:40:19.176 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.176 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.176 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.176 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:19.177 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.285091+0000", 2024-09-18T04:40:19.177 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.177 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.177 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.177 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:19.177 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.285090+0000", 2024-09-18T04:40:19.177 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:19.177 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.177 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.177 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:19.177 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.285100+0000", 2024-09-18T04:40:19.177 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0477e-05 2024-09-18T04:40:19.177 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.177 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.178 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:19.178 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.285144+0000", 2024-09-18T04:40:19.178 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4227999999999997e-05 2024-09-18T04:40:19.178 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.178 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.178 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:19.178 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.285160+0000", 2024-09-18T04:40:19.178 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6042999999999999e-05 2024-09-18T04:40:19.178 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.178 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.178 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:19.179 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.285639+0000", 2024-09-18T04:40:19.179 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00047824299999999999 2024-09-18T04:40:19.179 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.179 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.179 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:19.179 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.285797+0000", 2024-09-18T04:40:19.179 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015854899999999999 2024-09-18T04:40:19.179 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.179 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.179 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:19.179 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.286015+0000", 2024-09-18T04:40:19.179 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00021824799999999999 2024-09-18T04:40:19.180 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.180 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.180 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:19.180 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.287132+0000", 2024-09-18T04:40:19.180 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0011161980000000001 2024-09-18T04:40:19.180 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.180 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.180 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:19.180 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.288042+0000", 2024-09-18T04:40:19.180 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00091054500000000002 2024-09-18T04:40:19.180 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.180 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.180 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:19.181 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.288119+0000", 2024-09-18T04:40:19.181 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7294999999999998e-05 2024-09-18T04:40:19.181 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.181 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.181 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:19.181 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.288357+0000", 2024-09-18T04:40:19.181 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00023735100000000001 2024-09-18T04:40:19.181 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.181 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:19.181 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.181 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.181 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.181 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.6267.0:3 2.1 2:9370a598:::rbd_header.125fc09df6d1:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e103)", 2024-09-18T04:40:19.182 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:40:13.945412+0000", 2024-09-18T04:40:19.182 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 5.1883345729999997, 2024-09-18T04:40:19.182 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0035634999999999998, 2024-09-18T04:40:19.182 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:19.182 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:19.182 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:19.182 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.6267", 2024-09-18T04:40:19.182 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3449046013", 2024-09-18T04:40:19.182 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 3 2024-09-18T04:40:19.182 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.182 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:19.182 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.182 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:19.182 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.945412+0000", 2024-09-18T04:40:19.183 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.183 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.183 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.183 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:19.183 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.945412+0000", 2024-09-18T04:40:19.183 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.183 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.183 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.183 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:19.183 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.945409+0000", 2024-09-18T04:40:19.183 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:40:19.183 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.183 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.183 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:19.183 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.945423+0000", 2024-09-18T04:40:19.184 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4014e-05 2024-09-18T04:40:19.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.184 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.184 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:19.184 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.945448+0000", 2024-09-18T04:40:19.184 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5065000000000001e-05 2024-09-18T04:40:19.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.184 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.184 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:19.184 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.945470+0000", 2024-09-18T04:40:19.184 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2127e-05 2024-09-18T04:40:19.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.184 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.184 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:19.185 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.945636+0000", 2024-09-18T04:40:19.185 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016542299999999999 2024-09-18T04:40:19.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.185 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.185 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-09-18T04:40:19.185 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.945654+0000", 2024-09-18T04:40:19.185 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8165999999999999e-05 2024-09-18T04:40:19.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.185 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.186 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:19.186 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.948489+0000", 2024-09-18T04:40:19.186 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002835317 2024-09-18T04:40:19.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.186 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.186 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:19.186 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.948622+0000", 2024-09-18T04:40:19.186 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013259400000000001 2024-09-18T04:40:19.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.186 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.186 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:19.186 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.948975+0000", 2024-09-18T04:40:19.186 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00035329099999999999 2024-09-18T04:40:19.186 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.186 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:19.187 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.187 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.187 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.6061.0:21 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [read 2560~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e103)", 2024-09-18T04:40:19.187 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:40:14.538767+0000", 2024-09-18T04:40:19.187 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.5949789189999999, 2024-09-18T04:40:19.187 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003584575, 2024-09-18T04:40:19.187 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:19.187 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:19.187 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:19.187 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.6061", 2024-09-18T04:40:19.187 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2011291291", 2024-09-18T04:40:19.187 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 21 2024-09-18T04:40:19.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.188 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:19.188 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.188 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:19.188 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.538767+0000", 2024-09-18T04:40:19.188 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.188 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.188 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:19.188 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.538767+0000", 2024-09-18T04:40:19.188 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.188 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.189 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:19.189 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.538767+0000", 2024-09-18T04:40:19.189 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:40:19.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.189 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.189 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:19.189 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.538769+0000", 2024-09-18T04:40:19.189 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8870000000000001e-06 2024-09-18T04:40:19.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.189 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.189 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:19.189 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.538773+0000", 2024-09-18T04:40:19.189 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8e-06 2024-09-18T04:40:19.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.190 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.190 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:19.190 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.538776+0000", 2024-09-18T04:40:19.190 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3629999999999998e-06 2024-09-18T04:40:19.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.190 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.190 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:19.190 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.541842+0000", 2024-09-18T04:40:19.190 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030658730000000002 2024-09-18T04:40:19.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.190 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.190 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:19.191 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.542072+0000", 2024-09-18T04:40:19.191 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00023009500000000001 2024-09-18T04:40:19.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.191 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.191 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:19.191 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.542352+0000", 2024-09-18T04:40:19.191 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00027983700000000001 2024-09-18T04:40:19.191 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.191 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:19.191 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.191 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.191 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.6061.0:22 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [read 3072~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e103)", 2024-09-18T04:40:19.191 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:40:14.538779+0000", 2024-09-18T04:40:19.192 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.5949672850000001, 2024-09-18T04:40:19.192 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0042496519999999996, 2024-09-18T04:40:19.192 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:19.192 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:19.192 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:19.192 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.6061", 2024-09-18T04:40:19.192 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2011291291", 2024-09-18T04:40:19.192 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 22 2024-09-18T04:40:19.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.192 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:19.192 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.192 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:19.192 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.538779+0000", 2024-09-18T04:40:19.193 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.193 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.193 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:19.193 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.538779+0000", 2024-09-18T04:40:19.193 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.193 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.193 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:19.193 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.538779+0000", 2024-09-18T04:40:19.193 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:40:19.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.194 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.194 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:19.194 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.538781+0000", 2024-09-18T04:40:19.194 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.835e-06 2024-09-18T04:40:19.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.194 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.194 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:19.194 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.538784+0000", 2024-09-18T04:40:19.194 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7340000000000002e-06 2024-09-18T04:40:19.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.194 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.194 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:19.194 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.538788+0000", 2024-09-18T04:40:19.194 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5559999999999999e-06 2024-09-18T04:40:19.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.195 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.195 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:19.195 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.542460+0000", 2024-09-18T04:40:19.195 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0036717410000000001 2024-09-18T04:40:19.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.195 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.195 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:19.195 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.542705+0000", 2024-09-18T04:40:19.195 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00024554699999999999 2024-09-18T04:40:19.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.195 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.195 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:19.195 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.543029+0000", 2024-09-18T04:40:19.195 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00032351999999999999 2024-09-18T04:40:19.196 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.196 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:19.196 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.196 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.196 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.6061.0:23 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [read 3584~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e103)", 2024-09-18T04:40:19.196 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:40:14.538791+0000", 2024-09-18T04:40:19.196 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.5949552469999997, 2024-09-18T04:40:19.196 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0047975300000000004, 2024-09-18T04:40:19.196 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:19.196 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:19.196 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:19.197 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.6061", 2024-09-18T04:40:19.197 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2011291291", 2024-09-18T04:40:19.197 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 23 2024-09-18T04:40:19.197 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.197 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:19.197 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.197 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:19.197 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.538791+0000", 2024-09-18T04:40:19.197 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.197 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.197 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.197 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:19.197 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.538791+0000", 2024-09-18T04:40:19.198 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.198 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.198 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:19.198 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.538791+0000", 2024-09-18T04:40:19.198 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:40:19.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.198 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.198 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:19.198 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.538793+0000", 2024-09-18T04:40:19.198 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6950000000000001e-06 2024-09-18T04:40:19.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.198 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.199 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:19.199 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.538796+0000", 2024-09-18T04:40:19.199 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7079999999999999e-06 2024-09-18T04:40:19.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.199 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.199 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:19.199 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.538799+0000", 2024-09-18T04:40:19.199 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1659999999999998e-06 2024-09-18T04:40:19.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.199 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.199 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:19.199 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.543113+0000", 2024-09-18T04:40:19.199 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0043137690000000003 2024-09-18T04:40:19.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.200 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.200 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:19.200 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.543364+0000", 2024-09-18T04:40:19.200 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00025053299999999999 2024-09-18T04:40:19.200 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.200 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.200 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:19.200 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.543589+0000", 2024-09-18T04:40:19.200 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000224942 2024-09-18T04:40:19.200 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.200 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:19.200 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.201 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.201 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.201 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.6061.0:29 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [read 6656~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e103)", 2024-09-18T04:40:19.201 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:40:14.544304+0000", 2024-09-18T04:40:19.201 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.5894421789999997, 2024-09-18T04:40:19.201 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0034195860000000001, 2024-09-18T04:40:19.201 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:19.201 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:19.201 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:19.201 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.6061", 2024-09-18T04:40:19.201 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2011291291", 2024-09-18T04:40:19.201 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 29 2024-09-18T04:40:19.201 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.202 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:19.202 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.202 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:19.202 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.544304+0000", 2024-09-18T04:40:19.202 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.202 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.202 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.202 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:19.202 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.544304+0000", 2024-09-18T04:40:19.203 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.203 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.203 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:19.203 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.544304+0000", 2024-09-18T04:40:19.203 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:40:19.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.203 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.204 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:19.204 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.544306+0000", 2024-09-18T04:40:19.204 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7630000000000001e-06 2024-09-18T04:40:19.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.204 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.204 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:19.204 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.544309+0000", 2024-09-18T04:40:19.204 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.754e-06 2024-09-18T04:40:19.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.204 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.204 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:19.204 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.544312+0000", 2024-09-18T04:40:19.204 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0639999999999998e-06 2024-09-18T04:40:19.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.205 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.205 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:19.205 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.547118+0000", 2024-09-18T04:40:19.205 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002805358 2024-09-18T04:40:19.205 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.205 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.205 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:19.205 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.547366+0000", 2024-09-18T04:40:19.205 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000247748 2024-09-18T04:40:19.205 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.205 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.205 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:19.205 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.547724+0000", 2024-09-18T04:40:19.205 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00035818100000000001 2024-09-18T04:40:19.206 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.206 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:19.206 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.206 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.206 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.206 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.6061.0:30 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [read 7168~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e103)", 2024-09-18T04:40:19.206 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:40:14.544315+0000", 2024-09-18T04:40:19.206 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.5894310650000003, 2024-09-18T04:40:19.206 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0038419959999999999, 2024-09-18T04:40:19.206 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:19.206 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:19.206 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:19.207 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.6061", 2024-09-18T04:40:19.207 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2011291291", 2024-09-18T04:40:19.207 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 30 2024-09-18T04:40:19.207 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.207 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:19.207 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.207 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:19.207 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.544315+0000", 2024-09-18T04:40:19.207 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.207 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.207 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.207 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:19.207 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.544315+0000", 2024-09-18T04:40:19.207 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.208 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.208 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.208 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:19.208 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.544315+0000", 2024-09-18T04:40:19.208 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:40:19.208 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.208 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.208 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:19.208 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.544317+0000", 2024-09-18T04:40:19.208 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.782e-06 2024-09-18T04:40:19.208 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.209 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.209 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:19.209 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.544320+0000", 2024-09-18T04:40:19.209 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6669999999999998e-06 2024-09-18T04:40:19.209 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.209 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.209 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:19.209 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.544324+0000", 2024-09-18T04:40:19.209 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3909999999999998e-06 2024-09-18T04:40:19.209 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.209 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.209 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:19.209 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.547774+0000", 2024-09-18T04:40:19.209 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0034506329999999998 2024-09-18T04:40:19.210 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.210 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.210 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:19.210 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.547934+0000", 2024-09-18T04:40:19.210 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000159576 2024-09-18T04:40:19.210 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.210 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.210 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:19.210 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.548157+0000", 2024-09-18T04:40:19.210 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000223255 2024-09-18T04:40:19.210 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.210 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:19.211 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.211 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.211 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.211 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.6061.0:31 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [read 7680~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e103)", 2024-09-18T04:40:19.211 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:40:14.544327+0000", 2024-09-18T04:40:19.211 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.5894198289999997, 2024-09-18T04:40:19.211 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0044879840000000004, 2024-09-18T04:40:19.211 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:19.211 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:19.211 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:19.211 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.6061", 2024-09-18T04:40:19.211 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2011291291", 2024-09-18T04:40:19.211 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 31 2024-09-18T04:40:19.212 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.212 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:19.212 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.212 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:19.212 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.544327+0000", 2024-09-18T04:40:19.212 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.212 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.212 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.212 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:19.212 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.544327+0000", 2024-09-18T04:40:19.213 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.213 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.213 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.213 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:19.213 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.544326+0000", 2024-09-18T04:40:19.213 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:40:19.213 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.213 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.213 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:19.213 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.544338+0000", 2024-09-18T04:40:19.213 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2163e-05 2024-09-18T04:40:19.213 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.213 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.213 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:19.214 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.544343+0000", 2024-09-18T04:40:19.214 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.605e-06 2024-09-18T04:40:19.214 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.214 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.214 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:19.214 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.544346+0000", 2024-09-18T04:40:19.214 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4369999999999999e-06 2024-09-18T04:40:19.214 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.214 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.214 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:19.214 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.548234+0000", 2024-09-18T04:40:19.214 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0038871259999999999 2024-09-18T04:40:19.214 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.215 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.215 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:19.215 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.548480+0000", 2024-09-18T04:40:19.215 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00024613800000000002 2024-09-18T04:40:19.215 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.215 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.215 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:19.215 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.548815+0000", 2024-09-18T04:40:19.215 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000334796 2024-09-18T04:40:19.215 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.215 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:19.215 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.215 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.215 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.216 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.6061.0:86 2.1 2:9370a598:::rbd_header.125fc09df6d1:head [call rbd.snapshot_get in=15b,call rbd.parent_overlap_get in=21b,call rbd.get_flags in=12b,call rbd.get_protection_status in=24b,call rbd.snapshot_get in=15b,call rbd.parent_overlap_get in=21b,call rbd.get_flags in=12b,call rbd.get_protection_status in=24b,call rbd.snapshot_get in=15b,call rbd.parent_overlap_get in=21b,call rbd.get_flags in=12b,call rbd.get_protection_status in=24b,call rbd.snapshot_get in=15b,call rbd.parent_overlap_get in=21b,call rbd.get_flags in=12b,call rbd.get_protection_status in=24b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e103)", 2024-09-18T04:40:19.216 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:40:14.750931+0000", 2024-09-18T04:40:19.216 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.3828153680000002, 2024-09-18T04:40:19.216 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0035763380000000001, 2024-09-18T04:40:19.216 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:19.216 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:19.216 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:19.216 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.6061", 2024-09-18T04:40:19.216 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2011291291", 2024-09-18T04:40:19.216 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 86 2024-09-18T04:40:19.216 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.216 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:19.216 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.216 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:19.217 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.750931+0000", 2024-09-18T04:40:19.217 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.217 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.217 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.217 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:19.217 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.750931+0000", 2024-09-18T04:40:19.217 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.217 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.217 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.217 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:19.217 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.750930+0000", 2024-09-18T04:40:19.217 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:19.217 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.217 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.218 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:19.218 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.750936+0000", 2024-09-18T04:40:19.218 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2659999999999998e-06 2024-09-18T04:40:19.218 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.218 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.218 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:19.218 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.750947+0000", 2024-09-18T04:40:19.218 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0305999999999999e-05 2024-09-18T04:40:19.218 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.218 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.218 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:19.218 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.750954+0000", 2024-09-18T04:40:19.218 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1740000000000003e-06 2024-09-18T04:40:19.218 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.219 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.219 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:19.219 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.752128+0000", 2024-09-18T04:40:19.219 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0011746809999999999 2024-09-18T04:40:19.219 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.219 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.219 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for rw locks", 2024-09-18T04:40:19.219 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.752369+0000", 2024-09-18T04:40:19.219 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00024055499999999999 2024-09-18T04:40:19.219 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.219 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.219 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:19.220 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.753313+0000", 2024-09-18T04:40:19.220 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00094424699999999995 2024-09-18T04:40:19.220 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.220 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.220 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:19.220 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.753424+0000", 2024-09-18T04:40:19.220 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000110958 2024-09-18T04:40:19.221 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.221 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.221 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:19.221 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.754507+0000", 2024-09-18T04:40:19.221 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0010830989999999999 2024-09-18T04:40:19.221 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.221 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:19.221 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.221 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.221 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.221 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.6061.0:87 2.1 2:9370a598:::rbd_header.125fc09df6d1:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e103)", 2024-09-18T04:40:19.221 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:40:14.751277+0000", 2024-09-18T04:40:19.221 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.3824693520000002, 2024-09-18T04:40:19.221 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0034438260000000001, 2024-09-18T04:40:19.222 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:19.222 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:19.222 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:19.222 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.6061", 2024-09-18T04:40:19.222 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2011291291", 2024-09-18T04:40:19.222 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 87 2024-09-18T04:40:19.222 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.222 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:19.222 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.222 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:19.222 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.751277+0000", 2024-09-18T04:40:19.222 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.222 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.222 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.223 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:19.223 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.751277+0000", 2024-09-18T04:40:19.223 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.223 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.223 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.223 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:19.223 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.751276+0000", 2024-09-18T04:40:19.223 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:19.223 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.223 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.223 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:19.223 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.751282+0000", 2024-09-18T04:40:19.223 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3519999999999997e-06 2024-09-18T04:40:19.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.224 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.224 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:19.224 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.751295+0000", 2024-09-18T04:40:19.224 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3913999999999999e-05 2024-09-18T04:40:19.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.224 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.224 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:19.224 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.751303+0000", 2024-09-18T04:40:19.224 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.588e-06 2024-09-18T04:40:19.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.224 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.224 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:19.224 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.752477+0000", 2024-09-18T04:40:19.224 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0011738810000000001 2024-09-18T04:40:19.225 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.225 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.225 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for rw locks", 2024-09-18T04:40:19.225 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.752684+0000", 2024-09-18T04:40:19.225 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00020664300000000001 2024-09-18T04:40:19.225 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.225 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.225 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:19.225 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.754573+0000", 2024-09-18T04:40:19.225 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0018898859999999999 2024-09-18T04:40:19.225 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.225 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.225 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:19.225 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.754637+0000", 2024-09-18T04:40:19.226 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3100000000000002e-05 2024-09-18T04:40:19.226 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.226 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.226 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:19.226 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.754721+0000", 2024-09-18T04:40:19.226 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4240999999999995e-05 2024-09-18T04:40:19.226 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.226 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:19.226 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.226 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.226 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.226 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.6291.0:35 2.1 2:9370a598:::rbd_header.125fc09df6d1:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e103)", 2024-09-18T04:40:19.226 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:40:14.751317+0000", 2024-09-18T04:40:19.226 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.3824295639999997, 2024-09-18T04:40:19.227 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003608694, 2024-09-18T04:40:19.227 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:19.227 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:19.227 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:19.227 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.6291", 2024-09-18T04:40:19.229 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/1667696675", 2024-09-18T04:40:19.229 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 35 2024-09-18T04:40:19.229 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.229 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:19.229 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.229 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:19.230 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.751317+0000", 2024-09-18T04:40:19.230 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.230 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.230 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:19.230 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.751317+0000", 2024-09-18T04:40:19.231 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.231 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.231 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.231 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:19.231 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.751316+0000", 2024-09-18T04:40:19.232 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:40:19.232 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.232 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.232 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:19.232 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.751320+0000", 2024-09-18T04:40:19.232 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7529999999999999e-06 2024-09-18T04:40:19.232 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.232 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.232 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:19.232 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.751327+0000", 2024-09-18T04:40:19.232 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1790000000000002e-06 2024-09-18T04:40:19.232 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.232 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.233 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:19.233 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.751333+0000", 2024-09-18T04:40:19.233 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4870000000000002e-06 2024-09-18T04:40:19.233 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.233 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.233 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:19.233 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.752785+0000", 2024-09-18T04:40:19.233 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001452577 2024-09-18T04:40:19.233 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.233 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.233 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for rw locks", 2024-09-18T04:40:19.233 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.752865+0000", 2024-09-18T04:40:19.234 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9290999999999997e-05 2024-09-18T04:40:19.234 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.234 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.234 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:19.234 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.754760+0000", 2024-09-18T04:40:19.234 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001895189 2024-09-18T04:40:19.234 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.234 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.234 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:19.234 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.754845+0000", 2024-09-18T04:40:19.234 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5567000000000003e-05 2024-09-18T04:40:19.234 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.234 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.234 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:19.234 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.754925+0000", 2024-09-18T04:40:19.235 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0186999999999996e-05 2024-09-18T04:40:19.235 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.235 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:19.235 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.235 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.235 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.235 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.6061.0:220 2.1 2:9370a598:::rbd_header.125fc09df6d1:head [watch unwatch cookie 140218401383712] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-18T04:40:19.235 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:40:15.030311+0000", 2024-09-18T04:40:19.235 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.1034356360000004, 2024-09-18T04:40:19.235 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0034368469999999998, 2024-09-18T04:40:19.235 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:19.235 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:19.235 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:19.235 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.6061", 2024-09-18T04:40:19.236 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2011291291", 2024-09-18T04:40:19.236 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 220 2024-09-18T04:40:19.236 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.236 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:19.236 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.236 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:19.236 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:15.030311+0000", 2024-09-18T04:40:19.236 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.236 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.236 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.236 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:19.236 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:15.030311+0000", 2024-09-18T04:40:19.236 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.236 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.237 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.237 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:19.237 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:15.030309+0000", 2024-09-18T04:40:19.237 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:19.237 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.237 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.237 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:19.237 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:15.030317+0000", 2024-09-18T04:40:19.237 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9459999999999998e-06 2024-09-18T04:40:19.237 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.238 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.238 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:19.238 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:15.030336+0000", 2024-09-18T04:40:19.238 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8430000000000001e-05 2024-09-18T04:40:19.238 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.238 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.238 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:19.238 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:15.030351+0000", 2024-09-18T04:40:19.238 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5213e-05 2024-09-18T04:40:19.238 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.238 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.238 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:19.239 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:15.030826+0000", 2024-09-18T04:40:19.239 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00047490500000000002 2024-09-18T04:40:19.239 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.239 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.239 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:19.239 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:15.031080+0000", 2024-09-18T04:40:19.239 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00025388299999999999 2024-09-18T04:40:19.239 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.239 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.239 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:19.239 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:15.031460+0000", 2024-09-18T04:40:19.239 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00037980000000000002 2024-09-18T04:40:19.239 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.239 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.240 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:19.240 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:15.032713+0000", 2024-09-18T04:40:19.240 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001253737 2024-09-18T04:40:19.240 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.240 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.240 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:19.240 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:15.033521+0000", 2024-09-18T04:40:19.240 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00080719699999999995 2024-09-18T04:40:19.240 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.240 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.240 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:19.240 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:15.033605+0000", 2024-09-18T04:40:19.240 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4024999999999996e-05 2024-09-18T04:40:19.240 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.241 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.241 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:19.241 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:15.033748+0000", 2024-09-18T04:40:19.241 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014293699999999999 2024-09-18T04:40:19.241 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.241 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:19.241 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.241 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.241 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.241 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.5837.0:23 2.5 2:bb27d472:::rbd_object_map.125fc09df6d1.0000000000000006:head [call rbd.object_map_snap_remove in=25b] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e105)", 2024-09-18T04:40:19.241 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:40:18.547792+0000", 2024-09-18T04:40:19.241 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.58595469200000005, 2024-09-18T04:40:19.241 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003224242, 2024-09-18T04:40:19.242 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:19.242 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:19.242 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:19.242 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.5837", 2024-09-18T04:40:19.242 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2915691943", 2024-09-18T04:40:19.242 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 23 2024-09-18T04:40:19.242 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.242 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:19.242 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.242 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:19.242 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:18.547792+0000", 2024-09-18T04:40:19.242 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.242 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.242 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.242 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:19.243 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:18.547792+0000", 2024-09-18T04:40:19.243 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.243 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.243 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.243 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:19.243 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:18.547789+0000", 2024-09-18T04:40:19.243 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:40:19.243 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.243 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.243 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:19.243 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:18.547810+0000", 2024-09-18T04:40:19.244 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.069e-05 2024-09-18T04:40:19.244 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.244 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.244 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:19.244 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:18.547839+0000", 2024-09-18T04:40:19.244 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8972999999999999e-05 2024-09-18T04:40:19.244 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.244 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.244 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:19.244 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:18.547852+0000", 2024-09-18T04:40:19.244 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3336000000000001e-05 2024-09-18T04:40:19.245 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.245 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.245 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:19.245 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:18.547964+0000", 2024-09-18T04:40:19.245 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000111694 2024-09-18T04:40:19.245 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.245 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.245 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:19.245 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:18.548322+0000", 2024-09-18T04:40:19.245 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00035789200000000001 2024-09-18T04:40:19.245 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.245 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.245 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:40:19.245 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:18.548853+0000", 2024-09-18T04:40:19.246 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00053145099999999995 2024-09-18T04:40:19.246 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.246 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.246 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:19.246 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:18.550126+0000", 2024-09-18T04:40:19.246 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001273293 2024-09-18T04:40:19.246 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.246 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.246 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:19.246 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:18.550873+0000", 2024-09-18T04:40:19.246 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00074619699999999998 2024-09-18T04:40:19.246 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.246 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.246 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:19.246 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:18.550961+0000", 2024-09-18T04:40:19.247 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8617999999999998e-05 2024-09-18T04:40:19.247 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.247 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.247 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:19.247 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:18.551016+0000", 2024-09-18T04:40:19.247 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4718e-05 2024-09-18T04:40:19.247 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.247 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:19.247 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.247 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.247 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:19.247 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:40:19.249 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:40:19.243+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:40:19.249 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:40:19.350 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:40:19.347+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:40:19.366 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:40:19.366 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:40:19.366 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:40:19.366 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:40:19.377 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:40:19.451 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:40:19.447+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:40:19.487 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:40:19.487 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:40:19.487 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:40:19.487 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:40:19.487 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:40:19.498 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:40:19.553 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:40:19.547+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:40:19.611 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:40:19.611 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:40:19.611 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:40:19.611 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:40:19.611 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.611 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:37 2.5 e51/28 2:ba37578c:::rbd_data.125fc09df6d1.0000000000000003:head v 51'1, mlcod=0'0)", 2024-09-18T04:40:19.611 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:40:19.611 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 97.314008584000007, 2024-09-18T04:40:19.612 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012279973, 2024-09-18T04:40:19.612 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:19.612 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:19.612 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:19.612 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.612 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:19.612 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:40:19.612 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.612 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.612 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.612 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:19.612 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292469+0000", 2024-09-18T04:40:19.612 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.612 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.613 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.613 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:19.613 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.292468+0000", 2024-09-18T04:40:19.613 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:40:19.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.613 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.613 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:19.613 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295837+0000", 2024-09-18T04:40:19.613 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0033691699999999999 2024-09-18T04:40:19.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.613 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.613 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:19.614 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295860+0000", 2024-09-18T04:40:19.614 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2756999999999999e-05 2024-09-18T04:40:19.614 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.614 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.614 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:19.614 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295872+0000", 2024-09-18T04:40:19.614 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2442e-05 2024-09-18T04:40:19.614 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.614 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.614 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:19.614 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295934+0000", 2024-09-18T04:40:19.614 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1187999999999998e-05 2024-09-18T04:40:19.614 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.614 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.615 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:19.615 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.295967+0000", 2024-09-18T04:40:19.615 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3185999999999997e-05 2024-09-18T04:40:19.615 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.615 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.616 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:19.616 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304697+0000", 2024-09-18T04:40:19.616 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0087298059999999997 2024-09-18T04:40:19.616 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.616 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.616 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:19.616 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304749+0000", 2024-09-18T04:40:19.616 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1922000000000002e-05 2024-09-18T04:40:19.616 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.616 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:19.616 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.617 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.617 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.617 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:49 2.5 e51/28 2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head v 51'2, mlcod=0'0)", 2024-09-18T04:40:19.617 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:40:19.617 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 97.280681745999999, 2024-09-18T04:40:19.617 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018033041999999999, 2024-09-18T04:40:19.617 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:19.617 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:19.617 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:19.617 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.617 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:19.617 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:40:19.617 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.618 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.618 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.618 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:19.618 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325795+0000", 2024-09-18T04:40:19.618 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.618 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.618 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.618 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:19.618 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325793+0000", 2024-09-18T04:40:19.618 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:40:19.618 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.618 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.618 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:19.618 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329038+0000", 2024-09-18T04:40:19.619 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0032444010000000001 2024-09-18T04:40:19.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.619 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.619 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:19.619 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329065+0000", 2024-09-18T04:40:19.619 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7500000000000001e-05 2024-09-18T04:40:19.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.619 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.619 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:19.619 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329077+0000", 2024-09-18T04:40:19.619 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1919999999999999e-05 2024-09-18T04:40:19.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.620 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.620 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:19.620 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329149+0000", 2024-09-18T04:40:19.620 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2349999999999997e-05 2024-09-18T04:40:19.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.620 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.620 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:19.620 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.329192+0000", 2024-09-18T04:40:19.620 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2744000000000003e-05 2024-09-18T04:40:19.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.620 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.621 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:19.621 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343773+0000", 2024-09-18T04:40:19.621 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014581246000000001 2024-09-18T04:40:19.621 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.621 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.621 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:19.621 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343829+0000", 2024-09-18T04:40:19.621 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5143999999999999e-05 2024-09-18T04:40:19.622 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.622 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:19.622 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.622 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.622 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:47 2.4 e51/17 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 51'3, mlcod=51'1)", 2024-09-18T04:40:19.622 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:40:19.622 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 97.280101481000003, 2024-09-18T04:40:19.622 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018960186, 2024-09-18T04:40:19.622 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:19.622 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:19.622 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:19.622 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.623 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:19.623 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:40:19.623 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.623 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.623 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.623 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:19.623 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326376+0000", 2024-09-18T04:40:19.623 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.623 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.623 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.623 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:19.623 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326374+0000", 2024-09-18T04:40:19.623 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:19.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.624 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.624 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:19.624 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330883+0000", 2024-09-18T04:40:19.624 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0045084740000000002 2024-09-18T04:40:19.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.624 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.624 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:19.624 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330901+0000", 2024-09-18T04:40:19.624 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7890999999999999e-05 2024-09-18T04:40:19.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.624 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.624 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:19.625 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.330912+0000", 2024-09-18T04:40:19.625 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1041e-05 2024-09-18T04:40:19.625 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.625 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.625 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:19.625 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.331003+0000", 2024-09-18T04:40:19.625 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1323999999999995e-05 2024-09-18T04:40:19.625 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.625 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.625 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:19.626 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.331030+0000", 2024-09-18T04:40:19.626 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7351999999999998e-05 2024-09-18T04:40:19.626 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.626 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.626 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:19.626 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345287+0000", 2024-09-18T04:40:19.626 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014256332 2024-09-18T04:40:19.626 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.626 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.626 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:19.626 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345336+0000", 2024-09-18T04:40:19.626 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9356e-05 2024-09-18T04:40:19.626 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.626 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:19.627 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.627 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.627 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.627 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:58 2.5 e51/28 2:b227e6b1:::rbd_data.125fc09df6d1.000000000000000e:head v 51'3, mlcod=51'1)", 2024-09-18T04:40:19.627 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:40:19.627 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 97.239674854, 2024-09-18T04:40:19.627 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016513678, 2024-09-18T04:40:19.627 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:19.627 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:19.627 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:19.627 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.627 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:19.627 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:40:19.628 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.628 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.628 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.628 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:19.628 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366802+0000", 2024-09-18T04:40:19.628 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.628 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.628 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.628 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:19.628 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366800+0000", 2024-09-18T04:40:19.628 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:40:19.628 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.629 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.629 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:19.629 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368644+0000", 2024-09-18T04:40:19.629 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001844399 2024-09-18T04:40:19.629 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.629 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.629 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:19.629 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368681+0000", 2024-09-18T04:40:19.629 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6526999999999998e-05 2024-09-18T04:40:19.629 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.629 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.629 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:19.629 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368698+0000", 2024-09-18T04:40:19.629 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7272999999999999e-05 2024-09-18T04:40:19.630 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.630 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.630 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:19.630 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368773+0000", 2024-09-18T04:40:19.630 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4862000000000003e-05 2024-09-18T04:40:19.630 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.630 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.630 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:19.630 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368801+0000", 2024-09-18T04:40:19.630 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7925000000000001e-05 2024-09-18T04:40:19.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.631 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.631 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:19.631 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383261+0000", 2024-09-18T04:40:19.631 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014460377999999999 2024-09-18T04:40:19.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.631 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.631 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:19.631 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383316+0000", 2024-09-18T04:40:19.631 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4623999999999997e-05 2024-09-18T04:40:19.631 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.631 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:19.631 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.632 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.632 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:56 2.4 e51/17 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 51'4, mlcod=51'2)", 2024-09-18T04:40:19.632 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:40:19.632 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 97.237670828000006, 2024-09-18T04:40:19.632 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015105221, 2024-09-18T04:40:19.632 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:19.632 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:19.632 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:19.632 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.632 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:19.632 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:40:19.632 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.632 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.633 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.633 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:19.633 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368806+0000", 2024-09-18T04:40:19.633 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.633 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.633 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.633 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:19.633 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368805+0000", 2024-09-18T04:40:19.633 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:19.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.634 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.634 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:19.634 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371603+0000", 2024-09-18T04:40:19.634 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027982660000000002 2024-09-18T04:40:19.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.634 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.634 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:19.634 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371633+0000", 2024-09-18T04:40:19.634 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0073000000000002e-05 2024-09-18T04:40:19.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.634 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.634 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:19.634 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371652+0000", 2024-09-18T04:40:19.635 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8597000000000002e-05 2024-09-18T04:40:19.635 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.635 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.635 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:19.635 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371752+0000", 2024-09-18T04:40:19.635 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9772999999999995e-05 2024-09-18T04:40:19.635 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.635 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.635 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:19.635 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371793+0000", 2024-09-18T04:40:19.635 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1173e-05 2024-09-18T04:40:19.635 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.635 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.636 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:19.636 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383875+0000", 2024-09-18T04:40:19.636 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012081852000000001 2024-09-18T04:40:19.636 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.636 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.636 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:19.636 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383912+0000", 2024-09-18T04:40:19.636 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7045000000000002e-05 2024-09-18T04:40:19.636 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.636 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:19.636 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.636 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.636 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.636 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:36 2.5 e51/28 2:bd5b8dd8:ns::rbd_data.133e813ab5bc.0000000000000002:head v 51'4, mlcod=51'2)", 2024-09-18T04:40:19.637 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:40:19.637 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 97.068951837, 2024-09-18T04:40:19.637 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020758161000000001, 2024-09-18T04:40:19.637 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:19.637 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:19.637 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:19.637 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.637 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:19.637 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:40:19.637 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.637 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.638 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:19.638 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537525+0000", 2024-09-18T04:40:19.638 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.638 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.638 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:19.638 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.537523+0000", 2024-09-18T04:40:19.638 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:19.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.638 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.638 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:19.638 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541578+0000", 2024-09-18T04:40:19.638 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0040549710000000001 2024-09-18T04:40:19.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.639 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.639 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:19.639 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541608+0000", 2024-09-18T04:40:19.639 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9349e-05 2024-09-18T04:40:19.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.639 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.639 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:19.639 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541621+0000", 2024-09-18T04:40:19.639 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2736e-05 2024-09-18T04:40:19.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.639 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.639 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:19.639 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541715+0000", 2024-09-18T04:40:19.639 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4197999999999995e-05 2024-09-18T04:40:19.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.640 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.640 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:19.640 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.541755+0000", 2024-09-18T04:40:19.640 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9818999999999999e-05 2024-09-18T04:40:19.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.640 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.640 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:19.640 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558217+0000", 2024-09-18T04:40:19.640 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016462234999999999 2024-09-18T04:40:19.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.640 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.641 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:19.641 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.558284+0000", 2024-09-18T04:40:19.641 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6774999999999997e-05 2024-09-18T04:40:19.641 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.641 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:19.641 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.641 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.641 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.641 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:39 2.4 e51/17 2:304364d1:ns::rbd_data.133e813ab5bc.0000000000000004:head v 51'5, mlcod=51'3)", 2024-09-18T04:40:19.641 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:40:19.641 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 97.060812709999993, 2024-09-18T04:40:19.641 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016777558000000001, 2024-09-18T04:40:19.641 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:19.642 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:19.642 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:19.642 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.642 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:19.642 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:40:19.642 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.642 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.642 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.642 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:19.642 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545665+0000", 2024-09-18T04:40:19.642 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.642 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.642 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.642 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:19.643 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545663+0000", 2024-09-18T04:40:19.643 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:19.643 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.643 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.643 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:19.643 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548286+0000", 2024-09-18T04:40:19.643 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026233390000000001 2024-09-18T04:40:19.643 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.643 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.643 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:19.643 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548305+0000", 2024-09-18T04:40:19.643 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8425000000000001e-05 2024-09-18T04:40:19.643 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.644 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.644 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:19.644 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548316+0000", 2024-09-18T04:40:19.644 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1163e-05 2024-09-18T04:40:19.644 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.644 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.644 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:19.644 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548379+0000", 2024-09-18T04:40:19.644 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2780000000000005e-05 2024-09-18T04:40:19.644 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.644 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.644 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:19.644 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548399+0000", 2024-09-18T04:40:19.645 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0080000000000001e-05 2024-09-18T04:40:19.645 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.645 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.645 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:19.645 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562383+0000", 2024-09-18T04:40:19.645 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013984671000000001 2024-09-18T04:40:19.645 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.645 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.645 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:19.645 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562442+0000", 2024-09-18T04:40:19.645 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8700999999999999e-05 2024-09-18T04:40:19.645 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.646 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:19.646 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.646 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.646 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.646 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:37 2.5 e51/28 2:bf39b8e4:ns::rbd_data.133e813ab5bc.0000000000000003:head v 51'5, mlcod=51'2)", 2024-09-18T04:40:19.646 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:40:19.646 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 97.057944441000004, 2024-09-18T04:40:19.646 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014264492, 2024-09-18T04:40:19.646 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:19.646 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:19.646 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:19.646 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.646 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:19.647 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:40:19.647 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.647 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.647 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.647 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:19.647 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548533+0000", 2024-09-18T04:40:19.647 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.647 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.647 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.647 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:19.647 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.548531+0000", 2024-09-18T04:40:19.647 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:19.648 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.648 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.648 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:19.648 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551500+0000", 2024-09-18T04:40:19.648 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0029688129999999998 2024-09-18T04:40:19.648 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.648 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.648 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:19.648 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551516+0000", 2024-09-18T04:40:19.648 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5994e-05 2024-09-18T04:40:19.648 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.649 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.649 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:19.649 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551526+0000", 2024-09-18T04:40:19.649 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9299999999999998e-06 2024-09-18T04:40:19.649 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.649 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.649 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:19.649 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551577+0000", 2024-09-18T04:40:19.649 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0871000000000001e-05 2024-09-18T04:40:19.649 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.649 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.650 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:19.650 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551602+0000", 2024-09-18T04:40:19.650 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4813000000000001e-05 2024-09-18T04:40:19.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.650 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.650 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:19.650 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562741+0000", 2024-09-18T04:40:19.650 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011139428999999999 2024-09-18T04:40:19.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.650 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.650 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:19.650 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562797+0000", 2024-09-18T04:40:19.650 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6329000000000002e-05 2024-09-18T04:40:19.651 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.651 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:19.651 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.651 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.651 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.651 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:49 2.4 e51/17 2:20f0b0c9:ns::rbd_data.133e813ab5bc.0000000000000009:head v 51'6, mlcod=51'4)", 2024-09-18T04:40:19.651 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:40:19.651 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 97.026509883000003, 2024-09-18T04:40:19.651 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013849046, 2024-09-18T04:40:19.651 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:19.651 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:19.651 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:19.651 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.652 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:19.652 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:40:19.652 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.652 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.652 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.652 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:19.652 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579967+0000", 2024-09-18T04:40:19.652 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.653 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.653 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.653 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:19.653 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579966+0000", 2024-09-18T04:40:19.653 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:19.653 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.653 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.653 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:19.653 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582324+0000", 2024-09-18T04:40:19.653 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023578890000000002 2024-09-18T04:40:19.653 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.653 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.654 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:19.654 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582355+0000", 2024-09-18T04:40:19.654 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1485000000000001e-05 2024-09-18T04:40:19.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.654 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.654 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:19.654 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582371+0000", 2024-09-18T04:40:19.654 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6147999999999999e-05 2024-09-18T04:40:19.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.654 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.654 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:19.654 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582467+0000", 2024-09-18T04:40:19.654 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6059999999999998e-05 2024-09-18T04:40:19.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.655 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.655 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:19.655 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.582505+0000", 2024-09-18T04:40:19.655 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7679000000000003e-05 2024-09-18T04:40:19.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.655 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.655 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:19.655 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593743+0000", 2024-09-18T04:40:19.655 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011238356 2024-09-18T04:40:19.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.655 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.655 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:19.655 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593816+0000", 2024-09-18T04:40:19.656 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2935000000000006e-05 2024-09-18T04:40:19.656 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.656 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:19.656 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.656 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.656 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:53 2.4 e51/17 2:39f7f174:ns::rbd_data.133e813ab5bc.000000000000000b:head v 51'7, mlcod=51'4)", 2024-09-18T04:40:19.656 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:40:19.656 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 97.011808728000005, 2024-09-18T04:40:19.656 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0079914159999999995, 2024-09-18T04:40:19.656 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:19.656 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:19.656 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:19.657 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.657 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:19.657 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:40:19.657 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.657 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.657 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:19.657 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594668+0000", 2024-09-18T04:40:19.657 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.657 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.657 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:19.658 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594667+0000", 2024-09-18T04:40:19.658 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:19.658 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.658 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.658 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:19.658 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596446+0000", 2024-09-18T04:40:19.658 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001779033 2024-09-18T04:40:19.658 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.658 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.658 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:19.658 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596472+0000", 2024-09-18T04:40:19.658 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6120000000000001e-05 2024-09-18T04:40:19.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.659 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.659 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:19.659 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596500+0000", 2024-09-18T04:40:19.659 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7518e-05 2024-09-18T04:40:19.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.659 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.659 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:19.659 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596564+0000", 2024-09-18T04:40:19.659 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3880999999999994e-05 2024-09-18T04:40:19.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.659 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.659 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:19.660 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.596596+0000", 2024-09-18T04:40:19.660 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2638999999999997e-05 2024-09-18T04:40:19.660 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.660 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.660 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:19.660 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602605+0000", 2024-09-18T04:40:19.660 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0060080699999999999 2024-09-18T04:40:19.660 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.660 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.660 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:19.660 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602660+0000", 2024-09-18T04:40:19.661 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5396000000000003e-05 2024-09-18T04:40:19.661 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.661 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:19.661 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.661 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.661 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.661 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:60 2.5 e51/28 2:a06f51e4:ns::rbd_data.133e813ab5bc.000000000000000f:head v 51'6, mlcod=51'4)", 2024-09-18T04:40:19.661 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:40:19.661 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 96.996991742999995, 2024-09-18T04:40:19.662 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0093954989999999999, 2024-09-18T04:40:19.662 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:19.662 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:19.662 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:19.662 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.662 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:19.662 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:40:19.662 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.662 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.662 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:19.662 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609485+0000", 2024-09-18T04:40:19.662 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.663 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.663 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.663 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:19.663 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.609483+0000", 2024-09-18T04:40:19.663 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:19.663 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.663 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.663 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:19.663 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611767+0000", 2024-09-18T04:40:19.663 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002283724 2024-09-18T04:40:19.663 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.663 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.663 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:19.664 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611791+0000", 2024-09-18T04:40:19.664 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3547e-05 2024-09-18T04:40:19.664 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.664 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.664 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:19.664 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611800+0000", 2024-09-18T04:40:19.664 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4979999999999995e-06 2024-09-18T04:40:19.664 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.664 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.664 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:19.664 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611864+0000", 2024-09-18T04:40:19.664 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3974999999999996e-05 2024-09-18T04:40:19.664 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.665 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.665 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:19.665 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.611887+0000", 2024-09-18T04:40:19.665 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2633000000000002e-05 2024-09-18T04:40:19.665 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.665 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.665 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:19.665 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.618777+0000", 2024-09-18T04:40:19.665 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0068897979999999999 2024-09-18T04:40:19.665 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.665 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.665 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:19.665 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.618881+0000", 2024-09-18T04:40:19.666 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010437699999999999 2024-09-18T04:40:19.666 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.666 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:19.666 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.666 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.666 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.666 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:663 2.4 e57/17 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 57'13, mlcod=57'11)", 2024-09-18T04:40:19.666 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.355300+0000", 2024-09-18T04:40:19.666 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 88.251176807999997, 2024-09-18T04:40:19.666 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0049192760000000002, 2024-09-18T04:40:19.666 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:19.667 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:19.667 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:19.667 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.667 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:19.667 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355300+0000", 2024-09-18T04:40:19.667 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.667 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.667 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.667 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:19.667 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355300+0000", 2024-09-18T04:40:19.667 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.667 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.667 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.667 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:19.668 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355299+0000", 2024-09-18T04:40:19.668 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:19.668 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.668 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.668 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:19.668 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355876+0000", 2024-09-18T04:40:19.668 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00057730900000000003 2024-09-18T04:40:19.668 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.668 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.668 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:19.668 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355897+0000", 2024-09-18T04:40:19.668 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0477000000000001e-05 2024-09-18T04:40:19.669 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.669 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.669 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:19.669 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355905+0000", 2024-09-18T04:40:19.669 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5739999999999996e-06 2024-09-18T04:40:19.669 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.669 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.669 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:19.669 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.355989+0000", 2024-09-18T04:40:19.669 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4002e-05 2024-09-18T04:40:19.669 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.669 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.669 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:19.669 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.356041+0000", 2024-09-18T04:40:19.670 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1347000000000002e-05 2024-09-18T04:40:19.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.670 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.670 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:19.670 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.360172+0000", 2024-09-18T04:40:19.670 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0041315090000000002 2024-09-18T04:40:19.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.670 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.670 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:19.670 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.360220+0000", 2024-09-18T04:40:19.670 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7584999999999999e-05 2024-09-18T04:40:19.670 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.670 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:19.670 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.671 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.671 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.671 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:679 2.5 e57/28 2:b8f56ac7:::rbd_data.125fc09df6d1.0000000000000009:head v 57'20, mlcod=57'18)", 2024-09-18T04:40:19.671 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.445316+0000", 2024-09-18T04:40:19.671 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 88.161161331000002, 2024-09-18T04:40:19.671 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0047016760000000001, 2024-09-18T04:40:19.671 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:19.671 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:19.671 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:19.671 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.671 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:19.671 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445316+0000", 2024-09-18T04:40:19.671 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.671 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.672 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.672 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:19.672 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445316+0000", 2024-09-18T04:40:19.672 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.672 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.672 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:19.672 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445314+0000", 2024-09-18T04:40:19.672 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:19.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.672 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.672 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:19.672 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445947+0000", 2024-09-18T04:40:19.672 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00063298900000000003 2024-09-18T04:40:19.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.673 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.673 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:19.673 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445968+0000", 2024-09-18T04:40:19.673 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.054e-05 2024-09-18T04:40:19.673 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.673 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.673 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:19.673 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.445981+0000", 2024-09-18T04:40:19.673 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2697000000000001e-05 2024-09-18T04:40:19.673 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.673 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.673 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:19.674 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.446102+0000", 2024-09-18T04:40:19.674 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012189300000000001 2024-09-18T04:40:19.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.674 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.674 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:19.674 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.446151+0000", 2024-09-18T04:40:19.674 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8279000000000003e-05 2024-09-18T04:40:19.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.674 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.674 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:19.674 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.449953+0000", 2024-09-18T04:40:19.674 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003801979 2024-09-18T04:40:19.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.675 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.675 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:19.675 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.450018+0000", 2024-09-18T04:40:19.675 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4873e-05 2024-09-18T04:40:19.675 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.675 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:19.675 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.675 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.675 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.675 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4805.0:699 2.4 e57/17 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 57'17, mlcod=57'15)", 2024-09-18T04:40:19.675 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:51.566073+0000", 2024-09-18T04:40:19.675 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 88.040403818000001, 2024-09-18T04:40:19.675 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055398380000000001, 2024-09-18T04:40:19.676 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:19.676 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:19.676 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:19.676 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.676 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:19.676 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566073+0000", 2024-09-18T04:40:19.676 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.676 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.676 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.676 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:19.676 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566073+0000", 2024-09-18T04:40:19.676 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.676 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.676 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.677 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:19.677 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566071+0000", 2024-09-18T04:40:19.677 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:19.677 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.677 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.677 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:19.677 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566637+0000", 2024-09-18T04:40:19.677 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00056516699999999999 2024-09-18T04:40:19.677 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.677 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.677 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:19.677 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566668+0000", 2024-09-18T04:40:19.677 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1544999999999997e-05 2024-09-18T04:40:19.678 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.678 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.678 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:19.678 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566685+0000", 2024-09-18T04:40:19.678 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6857000000000001e-05 2024-09-18T04:40:19.678 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.678 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.678 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:19.678 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566816+0000", 2024-09-18T04:40:19.678 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013092799999999999 2024-09-18T04:40:19.678 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.678 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.678 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:19.678 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.566875+0000", 2024-09-18T04:40:19.678 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9583999999999997e-05 2024-09-18T04:40:19.678 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.679 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.679 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:19.679 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.571567+0000", 2024-09-18T04:40:19.679 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.004691682 2024-09-18T04:40:19.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.679 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.679 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:19.679 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:51.571613+0000", 2024-09-18T04:40:19.679 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6125000000000003e-05 2024-09-18T04:40:19.679 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.679 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:19.679 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.679 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.680 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:98 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'27, mlcod=60'25)", 2024-09-18T04:40:19.680 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.847978+0000", 2024-09-18T04:40:19.680 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 78.758499259000004, 2024-09-18T04:40:19.680 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0044617709999999998, 2024-09-18T04:40:19.680 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:19.680 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:19.680 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:19.680 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.680 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:19.680 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.847978+0000", 2024-09-18T04:40:19.680 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.680 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.680 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.681 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:19.681 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.847978+0000", 2024-09-18T04:40:19.681 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.681 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.681 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.681 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:19.681 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.847977+0000", 2024-09-18T04:40:19.681 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:19.681 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.681 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.681 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:19.681 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849038+0000", 2024-09-18T04:40:19.681 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0010614820000000001 2024-09-18T04:40:19.682 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.682 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.682 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:19.682 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849065+0000", 2024-09-18T04:40:19.682 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7127999999999998e-05 2024-09-18T04:40:19.682 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.682 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.682 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:19.682 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849082+0000", 2024-09-18T04:40:19.682 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7170999999999999e-05 2024-09-18T04:40:19.682 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.683 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.683 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:19.683 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849188+0000", 2024-09-18T04:40:19.683 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010605800000000001 2024-09-18T04:40:19.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.683 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.683 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:19.683 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.849233+0000", 2024-09-18T04:40:19.683 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4579000000000001e-05 2024-09-18T04:40:19.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.683 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.683 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:19.683 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.852370+0000", 2024-09-18T04:40:19.683 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00313751 2024-09-18T04:40:19.684 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.684 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.684 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:19.684 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.852440+0000", 2024-09-18T04:40:19.684 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9238000000000004e-05 2024-09-18T04:40:19.684 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.684 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:19.684 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.684 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.684 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.684 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:100 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'29, mlcod=64'27)", 2024-09-18T04:40:19.684 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.858766+0000", 2024-09-18T04:40:19.685 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 78.747710929999997, 2024-09-18T04:40:19.685 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0061964840000000004, 2024-09-18T04:40:19.685 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:19.685 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:19.685 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:19.685 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.685 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:19.685 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.858766+0000", 2024-09-18T04:40:19.686 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.686 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.686 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.686 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:19.686 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.858766+0000", 2024-09-18T04:40:19.686 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.686 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.686 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.686 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:19.686 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.858765+0000", 2024-09-18T04:40:19.686 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:19.686 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.687 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.687 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:19.687 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861612+0000", 2024-09-18T04:40:19.687 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028473169999999998 2024-09-18T04:40:19.687 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.687 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.687 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:19.687 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861643+0000", 2024-09-18T04:40:19.687 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1356000000000003e-05 2024-09-18T04:40:19.687 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.687 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.687 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:19.688 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861656+0000", 2024-09-18T04:40:19.688 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2899e-05 2024-09-18T04:40:19.688 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.688 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.688 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:19.688 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861773+0000", 2024-09-18T04:40:19.688 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011637999999999999 2024-09-18T04:40:19.688 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.688 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.688 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:19.688 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.861808+0000", 2024-09-18T04:40:19.689 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5002999999999997e-05 2024-09-18T04:40:19.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.689 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.689 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:19.689 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.864876+0000", 2024-09-18T04:40:19.689 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030680719999999998 2024-09-18T04:40:19.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.689 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.689 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:19.689 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.864963+0000", 2024-09-18T04:40:19.689 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7114000000000007e-05 2024-09-18T04:40:19.689 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.689 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:19.689 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.690 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.690 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.690 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:102 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'31, mlcod=64'29)", 2024-09-18T04:40:19.690 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.872098+0000", 2024-09-18T04:40:19.690 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 78.734379673000007, 2024-09-18T04:40:19.690 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0064676450000000002, 2024-09-18T04:40:19.690 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:19.690 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:19.690 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:19.690 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.690 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:19.690 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.872098+0000", 2024-09-18T04:40:19.690 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.690 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.691 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.691 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:19.691 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.872098+0000", 2024-09-18T04:40:19.691 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.691 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.691 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:19.691 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.872096+0000", 2024-09-18T04:40:19.691 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:19.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.692 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.692 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:19.692 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875176+0000", 2024-09-18T04:40:19.692 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030799809999999999 2024-09-18T04:40:19.692 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.692 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.692 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:19.692 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875204+0000", 2024-09-18T04:40:19.692 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8025999999999999e-05 2024-09-18T04:40:19.692 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.692 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.692 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:19.692 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875216+0000", 2024-09-18T04:40:19.693 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2072999999999999e-05 2024-09-18T04:40:19.693 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.693 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.693 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:19.693 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875307+0000", 2024-09-18T04:40:19.693 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0373999999999999e-05 2024-09-18T04:40:19.693 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.693 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.693 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:19.693 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.875349+0000", 2024-09-18T04:40:19.693 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2299000000000003e-05 2024-09-18T04:40:19.693 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.694 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.694 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:19.694 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.878496+0000", 2024-09-18T04:40:19.694 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0031468960000000002 2024-09-18T04:40:19.694 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.694 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.694 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:19.694 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.878565+0000", 2024-09-18T04:40:19.694 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9465e-05 2024-09-18T04:40:19.695 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.695 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:19.695 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.695 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.695 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:104 2.4 e64/62 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head v 64'33, mlcod=64'31)", 2024-09-18T04:40:19.695 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:00.885100+0000", 2024-09-18T04:40:19.695 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 78.721377137999994, 2024-09-18T04:40:19.695 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0051888159999999997, 2024-09-18T04:40:19.695 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:19.695 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:19.695 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:19.695 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.695 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:19.696 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.885100+0000", 2024-09-18T04:40:19.696 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.696 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.696 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:19.696 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.885100+0000", 2024-09-18T04:40:19.696 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.696 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.696 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:19.696 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.885098+0000", 2024-09-18T04:40:19.697 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:19.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.697 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.697 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:19.697 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886809+0000", 2024-09-18T04:40:19.697 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017110459999999999 2024-09-18T04:40:19.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.697 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.697 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:19.697 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886842+0000", 2024-09-18T04:40:19.697 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2638999999999997e-05 2024-09-18T04:40:19.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.697 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.697 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:19.698 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886855+0000", 2024-09-18T04:40:19.698 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3380000000000001e-05 2024-09-18T04:40:19.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.698 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.698 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:19.698 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886939+0000", 2024-09-18T04:40:19.698 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3733999999999995e-05 2024-09-18T04:40:19.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.698 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.698 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:19.698 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.886981+0000", 2024-09-18T04:40:19.698 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1956000000000003e-05 2024-09-18T04:40:19.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.699 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.699 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:19.699 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.890201+0000", 2024-09-18T04:40:19.699 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003219927 2024-09-18T04:40:19.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.699 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.699 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:19.699 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:00.890289+0000", 2024-09-18T04:40:19.699 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7997999999999993e-05 2024-09-18T04:40:19.699 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.699 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:19.699 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.700 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.700 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:145 2.4 e64/62 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 64'37, mlcod=64'35)", 2024-09-18T04:40:19.700 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:01.099398+0000", 2024-09-18T04:40:19.700 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 78.507079649999994, 2024-09-18T04:40:19.700 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0050078459999999998, 2024-09-18T04:40:19.700 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:19.700 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:19.700 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:19.700 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.700 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:19.700 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.099398+0000", 2024-09-18T04:40:19.700 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.700 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.701 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:19.701 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.099398+0000", 2024-09-18T04:40:19.701 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.701 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.701 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:19.701 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.099396+0000", 2024-09-18T04:40:19.701 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:19.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.701 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.701 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:19.701 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100199+0000", 2024-09-18T04:40:19.701 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00080338000000000002 2024-09-18T04:40:19.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.702 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.702 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:19.702 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100230+0000", 2024-09-18T04:40:19.702 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0716000000000002e-05 2024-09-18T04:40:19.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.702 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.702 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:19.702 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100243+0000", 2024-09-18T04:40:19.702 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3359e-05 2024-09-18T04:40:19.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.703 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.703 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:19.703 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100337+0000", 2024-09-18T04:40:19.703 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4387000000000003e-05 2024-09-18T04:40:19.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.703 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.703 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:19.703 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.100396+0000", 2024-09-18T04:40:19.703 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8164000000000001e-05 2024-09-18T04:40:19.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.703 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.703 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:19.703 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.104309+0000", 2024-09-18T04:40:19.703 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0039137800000000004 2024-09-18T04:40:19.704 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.704 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.704 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:19.704 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.104405+0000", 2024-09-18T04:40:19.704 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6039999999999995e-05 2024-09-18T04:40:19.704 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.704 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:19.704 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.704 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.704 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.704 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4970.0:147 2.4 e64/62 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head v 64'39, mlcod=64'37)", 2024-09-18T04:40:19.705 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:01.110658+0000", 2024-09-18T04:40:19.705 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 78.495819561000005, 2024-09-18T04:40:19.705 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0046316910000000003, 2024-09-18T04:40:19.705 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:19.705 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:19.705 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:19.705 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.705 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:19.705 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.110658+0000", 2024-09-18T04:40:19.705 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.705 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.706 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:19.706 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.110658+0000", 2024-09-18T04:40:19.706 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:19.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.706 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.706 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:19.706 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.110656+0000", 2024-09-18T04:40:19.706 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:19.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.706 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.707 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:19.707 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111444+0000", 2024-09-18T04:40:19.707 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00078762400000000001 2024-09-18T04:40:19.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.707 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.707 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:19.707 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111474+0000", 2024-09-18T04:40:19.707 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0085000000000001e-05 2024-09-18T04:40:19.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.707 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.707 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:19.707 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111486+0000", 2024-09-18T04:40:19.707 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2645e-05 2024-09-18T04:40:19.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.708 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.708 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:19.708 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111581+0000", 2024-09-18T04:40:19.708 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4432000000000007e-05 2024-09-18T04:40:19.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.708 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.708 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:19.708 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.111631+0000", 2024-09-18T04:40:19.708 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9901000000000002e-05 2024-09-18T04:40:19.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.708 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.708 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:19.709 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.115209+0000", 2024-09-18T04:40:19.709 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003577916 2024-09-18T04:40:19.709 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:19.709 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:19.709 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:19.709 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:01.115289+0000", 2024-09-18T04:40:19.710 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0704999999999994e-05 2024-09-18T04:40:19.710 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.710 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:19.710 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.710 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:19.710 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:19.710 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:40:19.711 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T04:40:19.713 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:40:19.714 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:40:19.707+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:40:19.813 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:40:19.807+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:40:19.824 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:40:19.824 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:40:19.824 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:40:19.824 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:40:19.835 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:40:19.852 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:40:19.874 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:40:19.894 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:40:19.914 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:40:19.907+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:40:19.921 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:40:19.945 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:40:19.962 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:19.962 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:19.967 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:40:19.969 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:40:19.969 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:40:19.969 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:40:19.969 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:40:19.969 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:40:19.969 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:40:19.972 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:40:20.023 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:40:20.019+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:40:20.116 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:40:20.111+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:40:20.133 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:20.133 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:20.143 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:40:20.218 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:40:20.211+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:40:20.259 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:20.259 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:20.269 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:40:20.319 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:40:20.315+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:40:20.381 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:20.381 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:20.390 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:40:20.419 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:40:20.415+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:40:20.499 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:20.499 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:20.508 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:40:20.554 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:40:20.551+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:40:20.615 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:20.616 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:20.621 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:40:20.615+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:40:20.625 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:40:20.722 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:40:20.719+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:40:20.733 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:20.734 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:20.743 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:40:20.823 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:40:20.819+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:40:20.850 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:20.850 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:20.860 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:40:20.924 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:40:20.919+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:40:20.940 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-18T04:40:20.969 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:20.969 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:20.979 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:40:21.024 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:40:21.019+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:40:21.081 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:21.081 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:21.091 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:40:21.126 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:40:21.119+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:40:21.200 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:21.200 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:21.209 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:40:21.227 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:40:21.223+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:40:21.319 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:21.319 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:21.327 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:40:21.323+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:40:21.330 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:40:21.428 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:40:21.423+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:40:21.435 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:21.435 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:21.445 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:40:21.530 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:40:21.523+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:40:21.553 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:21.554 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:21.564 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:40:21.631 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:40:21.627+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:40:21.672 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:21.672 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:21.682 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:40:21.731 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:40:21.727+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:40:21.752 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 2, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 2] 2024-09-18T04:40:21.752 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-18T04:40:21.793 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:21.793 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:21.803 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:40:21.846 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:40:21.843+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:40:21.933 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:40:21.927+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:40:21.942 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-09-18T04:40:21.944 INFO:tasks.workunit.client.0.smithi115.stderr: Removing all snapshots: 25% complete... Removing all snapshots: 50% complete... Removing all snapshots: 75% complete... Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2024-09-18T04:40:21.953 INFO:tasks.workunit.client.0.smithi115.stderr:2024-09-18T04:40:21.946+0000 7fb1ee14b700 0 -- 172.21.15.115:0/2915691943 >> [v2:172.21.15.28:6816/3849236161,v1:172.21.15.28:6817/3849236161] conn(0x7fb1d4008be0 msgr2=0x7fb1d400b070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-18T04:40:21.964 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd -p rbd --namespace '' remove testrbdnbd17342 2024-09-18T04:40:21.967 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:21.968 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:21.977 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:40:21.980 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T04:40:22.034 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:40:22.031+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:40:22.090 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:22.090 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:22.101 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:40:22.133 INFO:tasks.workunit.client.0.smithi115.stderr:2024-09-18T04:40:22.126+0000 7fd69e0c9700 0 -- 172.21.15.115:0/4024576069 >> [v2:172.21.15.28:6816/3849236161,v1:172.21.15.28:6817/3849236161] conn(0x7fd684008be0 msgr2=0x7fd68400b070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-18T04:40:22.135 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:40:22.131+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:40:22.165 INFO:tasks.workunit.client.0.smithi115.stderr: Removing image: 6% complete... Removing image: 12% complete... Removing image: 18% complete... Removing image: 25% complete... Removing image: 31% complete... Removing image: 37% complete... Removing image: 43% complete... Removing image: 50% complete... Removing image: 56% complete... Removing image: 62% complete... Removing image: 68% complete... Removing image: 75% complete... Removing image: 81% complete... Removing image: 87% complete... Removing image: 93% complete...2024-09-18T04:40:22.158+0000 7fd69f32b700 0 -- 172.21.15.115:0/4024576069 >> [v2:172.21.15.82:6816/1267644232,v1:172.21.15.82:6817/1267644232] conn(0x55c638b390c0 msgr2=0x55c638b3b4b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-18T04:40:22.166 INFO:tasks.workunit.client.0.smithi115.stderr: Removing image: 100% complete...done. 2024-09-18T04:40:22.173 INFO:tasks.workunit.client.0.smithi115.stderr:+ for ns in '' ${NS} 2024-09-18T04:40:22.174 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd -p rbd --namespace ns status testrbdnbd17342 2024-09-18T04:40:22.174 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:40:22.206 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:40:22.230 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:40:22.236 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:40:22.231+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:40:22.238 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:40:22.238 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:40:22.238 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:40:22.238 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:40:22.238 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:40:22.245 INFO:tasks.workunit.client.0.smithi115.stdout:Watchers: none 2024-09-18T04:40:22.247 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:40:22.251 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:40:22.252 INFO:tasks.workunit.client.0.smithi115.stderr:+ for s in 0.5 1 2 4 8 16 32 2024-09-18T04:40:22.252 INFO:tasks.workunit.client.0.smithi115.stderr:+ sleep 0.5 2024-09-18T04:40:22.277 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:40:22.306 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:40:22.308 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:40:22.308 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:40:22.308 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:40:22.308 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:40:22.308 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:40:22.308 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:40:22.363 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:40:22.363 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:40:22.363 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:40:22.363 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:40:22.363 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.363 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:33 2.1 e51/24 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head v 51'4, mlcod=51'2)", 2024-09-18T04:40:22.363 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:40:22.363 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 100.078981861, 2024-09-18T04:40:22.363 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018702106, 2024-09-18T04:40:22.364 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:22.364 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:22.364 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:22.364 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.364 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:22.364 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:40:22.364 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:22.364 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.364 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.364 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:22.364 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279198+0000", 2024-09-18T04:40:22.364 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:22.364 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.364 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.365 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:22.365 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.279197+0000", 2024-09-18T04:40:22.365 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:22.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.365 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.365 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:22.365 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282682+0000", 2024-09-18T04:40:22.365 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0034848689999999998 2024-09-18T04:40:22.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.365 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.365 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:22.365 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282709+0000", 2024-09-18T04:40:22.366 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7599000000000001e-05 2024-09-18T04:40:22.366 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.366 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.366 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:22.366 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282720+0000", 2024-09-18T04:40:22.366 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.092e-05 2024-09-18T04:40:22.366 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.366 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.366 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:22.366 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282791+0000", 2024-09-18T04:40:22.366 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1268999999999998e-05 2024-09-18T04:40:22.366 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.367 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.367 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:22.367 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.282843+0000", 2024-09-18T04:40:22.367 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1771999999999998e-05 2024-09-18T04:40:22.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.367 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.367 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:22.367 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.297830+0000", 2024-09-18T04:40:22.367 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014987228999999999 2024-09-18T04:40:22.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.367 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.367 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:22.368 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.297901+0000", 2024-09-18T04:40:22.368 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0209999999999994e-05 2024-09-18T04:40:22.368 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:22.368 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:22.368 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:22.368 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.368 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.368 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:38 2.0 e51/17 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head v 51'10, mlcod=51'8)", 2024-09-18T04:40:22.368 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:40:22.368 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 100.077563431, 2024-09-18T04:40:22.368 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.023984472, 2024-09-18T04:40:22.368 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:22.369 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:22.369 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:22.369 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.369 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:22.369 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:40:22.369 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:22.369 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.369 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.369 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:22.369 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280617+0000", 2024-09-18T04:40:22.369 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:22.369 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.369 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.370 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:22.370 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.280614+0000", 2024-09-18T04:40:22.370 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:40:22.370 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.370 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.370 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:22.370 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293094+0000", 2024-09-18T04:40:22.370 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012479571 2024-09-18T04:40:22.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.371 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.371 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:22.371 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293111+0000", 2024-09-18T04:40:22.371 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7354999999999999e-05 2024-09-18T04:40:22.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.371 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.371 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:22.371 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293121+0000", 2024-09-18T04:40:22.371 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6979999999999993e-06 2024-09-18T04:40:22.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.371 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.371 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:22.371 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293180+0000", 2024-09-18T04:40:22.372 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9333000000000003e-05 2024-09-18T04:40:22.372 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.372 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.372 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:22.372 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.293203+0000", 2024-09-18T04:40:22.372 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2633000000000002e-05 2024-09-18T04:40:22.372 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.372 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.372 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:22.372 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304534+0000", 2024-09-18T04:40:22.372 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011330761 2024-09-18T04:40:22.372 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.372 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.372 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:22.373 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.304601+0000", 2024-09-18T04:40:22.373 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7657000000000006e-05 2024-09-18T04:40:22.373 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:22.373 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:22.373 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:22.373 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.373 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.373 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:46 2.6 2:7ba25245:::rbd_data.125fc09df6d1.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:22.373 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:40:22.373 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 100.050020631, 2024-09-18T04:40:22.373 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.035604595000000003, 2024-09-18T04:40:22.373 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:22.373 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:22.373 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:22.374 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:40:22.374 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:40:22.374 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 46 2024-09-18T04:40:22.374 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.374 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:22.374 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.374 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:22.374 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:40:22.374 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:22.374 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.374 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.374 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:22.374 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308160+0000", 2024-09-18T04:40:22.374 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:22.375 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.375 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.375 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:22.375 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.308158+0000", 2024-09-18T04:40:22.375 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:22.375 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.375 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.375 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:22.375 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313191+0000", 2024-09-18T04:40:22.375 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0050323929999999996 2024-09-18T04:40:22.375 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.375 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.375 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:22.376 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313210+0000", 2024-09-18T04:40:22.376 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9673e-05 2024-09-18T04:40:22.376 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.376 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.376 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:22.376 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313221+0000", 2024-09-18T04:40:22.376 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0444e-05 2024-09-18T04:40:22.376 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.376 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.376 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:22.376 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313294+0000", 2024-09-18T04:40:22.377 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2650000000000004e-05 2024-09-18T04:40:22.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.377 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.377 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:22.377 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313493+0000", 2024-09-18T04:40:22.377 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019968300000000001 2024-09-18T04:40:22.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.377 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.377 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:40:22.377 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313666+0000", 2024-09-18T04:40:22.377 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00017248599999999999 2024-09-18T04:40:22.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.377 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.377 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:22.377 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.323175+0000", 2024-09-18T04:40:22.378 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0095088819999999998 2024-09-18T04:40:22.378 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.378 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.378 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:22.378 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343651+0000", 2024-09-18T04:40:22.378 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020475892999999998 2024-09-18T04:40:22.378 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.378 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.378 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:22.378 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343715+0000", 2024-09-18T04:40:22.378 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4443999999999995e-05 2024-09-18T04:40:22.378 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.378 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.378 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:22.379 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343764+0000", 2024-09-18T04:40:22.379 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9280000000000003e-05 2024-09-18T04:40:22.379 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:22.379 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:22.379 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:22.379 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.379 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.379 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:47 2.4 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:22.379 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:40:22.379 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 100.044938594, 2024-09-18T04:40:22.379 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.032570693999999997, 2024-09-18T04:40:22.379 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:22.379 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:22.379 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:22.380 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:40:22.380 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:40:22.380 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 47 2024-09-18T04:40:22.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.380 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:22.380 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.380 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:22.380 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:40:22.380 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:22.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.380 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.380 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:22.381 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313242+0000", 2024-09-18T04:40:22.381 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:22.381 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.381 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.381 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:22.381 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.313240+0000", 2024-09-18T04:40:22.381 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:22.381 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.381 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.381 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:22.381 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325825+0000", 2024-09-18T04:40:22.381 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012584425999999999 2024-09-18T04:40:22.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.382 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.382 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:22.382 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325848+0000", 2024-09-18T04:40:22.382 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3025999999999999e-05 2024-09-18T04:40:22.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.382 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.382 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:22.382 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325863+0000", 2024-09-18T04:40:22.382 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5152e-05 2024-09-18T04:40:22.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.382 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.382 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:22.382 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325938+0000", 2024-09-18T04:40:22.383 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4982999999999997e-05 2024-09-18T04:40:22.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.383 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.383 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:22.383 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326090+0000", 2024-09-18T04:40:22.383 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015207900000000001 2024-09-18T04:40:22.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.383 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.383 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:40:22.383 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.326242+0000", 2024-09-18T04:40:22.383 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015163500000000001 2024-09-18T04:40:22.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.383 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.383 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:22.384 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343015+0000", 2024-09-18T04:40:22.384 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016773414 2024-09-18T04:40:22.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.384 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.384 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:22.384 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345664+0000", 2024-09-18T04:40:22.384 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026487870000000001 2024-09-18T04:40:22.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.384 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.384 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:22.384 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345739+0000", 2024-09-18T04:40:22.384 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5018999999999994e-05 2024-09-18T04:40:22.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.385 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.385 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:22.385 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.345812+0000", 2024-09-18T04:40:22.385 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3547000000000006e-05 2024-09-18T04:40:22.385 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:22.385 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:22.385 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:22.385 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.385 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.385 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:48 2.6 2:61db7a3a:::rbd_data.125fc09df6d1.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:22.385 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:40:22.385 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 100.032281922, 2024-09-18T04:40:22.385 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.025064388, 2024-09-18T04:40:22.386 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:22.386 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:22.386 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:22.386 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:40:22.386 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:40:22.386 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 48 2024-09-18T04:40:22.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.386 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:22.386 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.386 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:22.386 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:40:22.386 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:22.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.387 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:22.387 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325898+0000", 2024-09-18T04:40:22.387 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:22.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.387 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:22.387 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.325897+0000", 2024-09-18T04:40:22.387 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:22.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.387 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:22.387 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333697+0000", 2024-09-18T04:40:22.388 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0078002649999999998 2024-09-18T04:40:22.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.388 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.388 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:22.388 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333716+0000", 2024-09-18T04:40:22.388 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8910999999999999e-05 2024-09-18T04:40:22.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.388 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.388 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:22.388 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333728+0000", 2024-09-18T04:40:22.388 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1936000000000001e-05 2024-09-18T04:40:22.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.389 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.389 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:22.389 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333792+0000", 2024-09-18T04:40:22.389 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4010999999999994e-05 2024-09-18T04:40:22.389 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.389 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.389 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:22.389 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.334082+0000", 2024-09-18T04:40:22.389 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00028955799999999998 2024-09-18T04:40:22.389 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.389 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.389 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:40:22.389 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.334326+0000", 2024-09-18T04:40:22.390 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00024399099999999999 2024-09-18T04:40:22.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.390 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.390 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:22.390 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350745+0000", 2024-09-18T04:40:22.390 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016419474999999999 2024-09-18T04:40:22.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.390 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.390 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:22.390 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350808+0000", 2024-09-18T04:40:22.390 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3244999999999995e-05 2024-09-18T04:40:22.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.390 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.390 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:22.391 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350892+0000", 2024-09-18T04:40:22.391 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4074999999999997e-05 2024-09-18T04:40:22.391 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.391 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.391 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:22.391 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.350963+0000", 2024-09-18T04:40:22.391 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0412000000000003e-05 2024-09-18T04:40:22.391 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:22.391 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:22.391 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:22.391 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.391 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.391 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:51 2.6 2:775d7b11:::rbd_data.125fc09df6d1.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:22.391 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:40:22.392 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 100.024429786, 2024-09-18T04:40:22.392 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.023824221, 2024-09-18T04:40:22.392 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:22.392 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:22.392 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:22.392 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:40:22.392 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:40:22.392 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 51 2024-09-18T04:40:22.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.392 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:22.392 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.392 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:22.392 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:40:22.393 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:22.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.393 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.393 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:22.393 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333750+0000", 2024-09-18T04:40:22.393 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:22.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.393 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.393 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:22.393 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.333749+0000", 2024-09-18T04:40:22.393 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:22.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.393 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.394 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:22.394 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340899+0000", 2024-09-18T04:40:22.394 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0071500340000000004 2024-09-18T04:40:22.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.394 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.394 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:22.394 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340929+0000", 2024-09-18T04:40:22.394 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0093000000000001e-05 2024-09-18T04:40:22.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.394 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.394 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:22.394 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.340943+0000", 2024-09-18T04:40:22.394 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4019000000000001e-05 2024-09-18T04:40:22.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.395 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.395 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:22.395 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.343834+0000", 2024-09-18T04:40:22.395 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028918120000000001 2024-09-18T04:40:22.395 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.395 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.395 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:22.395 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344020+0000", 2024-09-18T04:40:22.395 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000185892 2024-09-18T04:40:22.395 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.395 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.395 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:40:22.396 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.344190+0000", 2024-09-18T04:40:22.396 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016946800000000001 2024-09-18T04:40:22.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.396 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.396 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:22.396 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.351291+0000", 2024-09-18T04:40:22.396 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.007100805 2024-09-18T04:40:22.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.396 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.396 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:22.396 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357470+0000", 2024-09-18T04:40:22.396 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0061797570000000001 2024-09-18T04:40:22.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.396 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.397 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:22.397 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357533+0000", 2024-09-18T04:40:22.397 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2906999999999999e-05 2024-09-18T04:40:22.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.397 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.397 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:22.397 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357575+0000", 2024-09-18T04:40:22.397 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1403999999999999e-05 2024-09-18T04:40:22.397 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:22.397 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:22.397 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:22.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.397 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.398 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4703.0:56 2.4 2:2ecd49f5:::rbd_data.125fc09df6d1.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:22.398 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:40:22.398 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 100.008384693, 2024-09-18T04:40:22.398 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.034295870999999999, 2024-09-18T04:40:22.398 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:22.398 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:22.398 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:22.398 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4703", 2024-09-18T04:40:22.398 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:40:22.398 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 56 2024-09-18T04:40:22.398 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.398 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:22.398 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.399 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:22.399 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:40:22.399 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:22.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.399 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.399 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:22.399 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349796+0000", 2024-09-18T04:40:22.399 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:22.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.399 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.399 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:22.399 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.349794+0000", 2024-09-18T04:40:22.399 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:22.400 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.400 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.400 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:22.400 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366632+0000", 2024-09-18T04:40:22.400 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01683755 2024-09-18T04:40:22.400 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.400 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.400 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:22.400 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366669+0000", 2024-09-18T04:40:22.400 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7351000000000002e-05 2024-09-18T04:40:22.400 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.400 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.400 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:22.401 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366686+0000", 2024-09-18T04:40:22.401 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6507999999999998e-05 2024-09-18T04:40:22.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.401 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.401 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:22.401 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.366962+0000", 2024-09-18T04:40:22.401 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00027655900000000001 2024-09-18T04:40:22.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.401 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.401 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:22.401 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.367452+0000", 2024-09-18T04:40:22.401 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00049025299999999996 2024-09-18T04:40:22.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.401 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.402 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:40:22.402 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.368113+0000", 2024-09-18T04:40:22.402 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00066093399999999996 2024-09-18T04:40:22.402 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.402 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.402 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:22.402 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383719+0000", 2024-09-18T04:40:22.402 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015606106999999999 2024-09-18T04:40:22.402 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.402 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.402 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:22.402 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384035+0000", 2024-09-18T04:40:22.403 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00031590399999999999 2024-09-18T04:40:22.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.403 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.403 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:22.403 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384067+0000", 2024-09-18T04:40:22.403 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.129e-05 2024-09-18T04:40:22.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.403 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.403 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:22.403 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.384091+0000", 2024-09-18T04:40:22.403 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4788000000000001e-05 2024-09-18T04:40:22.403 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:22.403 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:22.404 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:22.404 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.404 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.404 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:57 2.0 e51/17 2:0cdd070e:::rbd_data.125fc09df6d1.000000000000000d:head v 51'21, mlcod=51'19)", 2024-09-18T04:40:22.404 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:40:22.404 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 100.000557206, 2024-09-18T04:40:22.404 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.025216311000000002, 2024-09-18T04:40:22.404 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:22.404 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:22.404 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:22.404 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.404 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:22.405 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:40:22.405 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:22.405 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.405 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.405 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:22.405 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:40:22.405 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:22.405 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.405 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.405 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:22.405 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.357623+0000", 2024-09-18T04:40:22.405 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:40:22.406 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.406 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.406 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:22.406 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363844+0000", 2024-09-18T04:40:22.406 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0062218450000000002 2024-09-18T04:40:22.406 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.406 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.406 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:22.406 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363879+0000", 2024-09-18T04:40:22.406 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4382999999999999e-05 2024-09-18T04:40:22.406 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.406 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.407 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:22.407 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363893+0000", 2024-09-18T04:40:22.407 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4348999999999999e-05 2024-09-18T04:40:22.407 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.407 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.407 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:22.407 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.363977+0000", 2024-09-18T04:40:22.407 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3499999999999997e-05 2024-09-18T04:40:22.407 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.407 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.407 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:22.407 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364009+0000", 2024-09-18T04:40:22.407 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2292000000000002e-05 2024-09-18T04:40:22.408 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.408 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.408 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:22.408 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.382781+0000", 2024-09-18T04:40:22.408 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018771626 2024-09-18T04:40:22.408 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.408 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.408 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:22.408 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.382839+0000", 2024-09-18T04:40:22.408 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8758000000000002e-05 2024-09-18T04:40:22.408 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:22.408 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:22.408 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:22.408 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.409 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.409 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4703.0:59 2.0 e51/17 2:19c52cde:::rbd_data.125fc09df6d1.000000000000000f:head v 51'22, mlcod=51'19)", 2024-09-18T04:40:22.409 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:40:22.409 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 99.993897279999999, 2024-09-18T04:40:22.409 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.019497112, 2024-09-18T04:40:22.409 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:22.409 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:22.409 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:22.409 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.409 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:22.409 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:40:22.410 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:22.410 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.410 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.410 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:22.410 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364283+0000", 2024-09-18T04:40:22.410 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:22.410 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.410 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.410 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:22.410 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.364282+0000", 2024-09-18T04:40:22.410 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:22.411 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.411 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.411 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:22.411 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371823+0000", 2024-09-18T04:40:22.411 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0075411829999999999 2024-09-18T04:40:22.411 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.411 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.411 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:22.411 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371837+0000", 2024-09-18T04:40:22.411 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4336e-05 2024-09-18T04:40:22.411 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.411 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.412 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:22.412 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.371846+0000", 2024-09-18T04:40:22.412 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9770000000000006e-06 2024-09-18T04:40:22.412 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.412 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.412 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:22.412 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.373566+0000", 2024-09-18T04:40:22.412 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017192539999999999 2024-09-18T04:40:22.412 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.412 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.412 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:22.412 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.373595+0000", 2024-09-18T04:40:22.413 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8975999999999998e-05 2024-09-18T04:40:22.413 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.413 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.413 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:22.413 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383740+0000", 2024-09-18T04:40:22.413 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01014525 2024-09-18T04:40:22.413 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.413 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.413 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:22.413 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.383780+0000", 2024-09-18T04:40:22.413 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0275999999999999e-05 2024-09-18T04:40:22.413 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:22.413 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:22.413 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:22.414 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.414 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.414 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:34 2.6 2:66289171:ns::rbd_data.133e813ab5bc.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:22.414 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.524015+0000", 2024-09-18T04:40:22.414 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 99.834165670999994, 2024-09-18T04:40:22.414 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027582111999999999, 2024-09-18T04:40:22.414 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:22.414 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:22.414 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:22.414 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:40:22.414 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:22.414 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 34 2024-09-18T04:40:22.415 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.415 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:22.415 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.415 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:22.415 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524015+0000", 2024-09-18T04:40:22.415 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:22.415 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.415 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.415 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:22.415 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524015+0000", 2024-09-18T04:40:22.415 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:22.415 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.415 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.416 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:22.416 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.524012+0000", 2024-09-18T04:40:22.416 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:40:22.416 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.416 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.416 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:22.416 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534956+0000", 2024-09-18T04:40:22.416 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010943403000000001 2024-09-18T04:40:22.416 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.416 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.416 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:22.416 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534978+0000", 2024-09-18T04:40:22.416 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1965999999999999e-05 2024-09-18T04:40:22.417 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.417 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.417 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:22.417 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534990+0000", 2024-09-18T04:40:22.417 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2065999999999999e-05 2024-09-18T04:40:22.417 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.417 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.417 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:22.417 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535065+0000", 2024-09-18T04:40:22.417 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5523000000000002e-05 2024-09-18T04:40:22.417 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.417 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.417 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:22.418 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535208+0000", 2024-09-18T04:40:22.418 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014309500000000001 2024-09-18T04:40:22.418 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.418 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.418 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:40:22.418 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535343+0000", 2024-09-18T04:40:22.418 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013437400000000001 2024-09-18T04:40:22.418 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.418 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.418 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:22.418 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.546576+0000", 2024-09-18T04:40:22.418 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011233425999999999 2024-09-18T04:40:22.418 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.419 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.419 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:22.419 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551498+0000", 2024-09-18T04:40:22.419 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0049219349999999997 2024-09-18T04:40:22.419 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.419 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.419 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:22.419 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551554+0000", 2024-09-18T04:40:22.419 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6400999999999998e-05 2024-09-18T04:40:22.419 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.419 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.419 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:22.419 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.551597+0000", 2024-09-18T04:40:22.419 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2283000000000002e-05 2024-09-18T04:40:22.420 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:22.420 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:22.420 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:22.420 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.420 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.420 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:33 2.1 e51/24 2:8b40f1c5:ns::rbd_data.133e813ab5bc.0000000000000000:head v 51'7, mlcod=51'5)", 2024-09-18T04:40:22.420 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.531767+0000", 2024-09-18T04:40:22.420 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 99.826413754000001, 2024-09-18T04:40:22.420 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015370295000000001, 2024-09-18T04:40:22.420 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:22.420 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:22.420 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:22.420 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.420 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:22.421 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531767+0000", 2024-09-18T04:40:22.421 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:22.421 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.421 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.421 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:22.421 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531767+0000", 2024-09-18T04:40:22.421 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:22.421 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.421 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.421 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:22.421 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.531765+0000", 2024-09-18T04:40:22.421 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:22.422 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.422 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.422 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:22.422 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534258+0000", 2024-09-18T04:40:22.422 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002493561 2024-09-18T04:40:22.422 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.422 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.422 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:22.422 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534288+0000", 2024-09-18T04:40:22.422 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.976e-05 2024-09-18T04:40:22.422 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.422 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.423 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:22.423 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534300+0000", 2024-09-18T04:40:22.423 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1928e-05 2024-09-18T04:40:22.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.423 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.423 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:22.423 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534367+0000", 2024-09-18T04:40:22.423 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7045999999999995e-05 2024-09-18T04:40:22.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.423 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.423 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:22.423 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.534394+0000", 2024-09-18T04:40:22.423 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6903999999999999e-05 2024-09-18T04:40:22.424 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.424 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.424 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:22.424 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.547073+0000", 2024-09-18T04:40:22.424 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012679365 2024-09-18T04:40:22.424 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.424 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.424 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:22.424 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.547137+0000", 2024-09-18T04:40:22.424 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3691999999999999e-05 2024-09-18T04:40:22.424 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:22.424 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:22.425 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:22.425 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.425 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.425 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:39 2.4 2:304364d1:ns::rbd_data.133e813ab5bc.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:22.425 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.535013+0000", 2024-09-18T04:40:22.425 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 99.823166818999994, 2024-09-18T04:40:22.425 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027764487000000001, 2024-09-18T04:40:22.425 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:22.425 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:22.425 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:22.425 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:40:22.425 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:22.425 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 39 2024-09-18T04:40:22.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.426 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:22.426 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.426 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:22.426 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535013+0000", 2024-09-18T04:40:22.426 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:22.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.426 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.426 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:22.426 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535013+0000", 2024-09-18T04:40:22.426 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:22.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.426 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.426 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:22.427 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.535012+0000", 2024-09-18T04:40:22.427 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:22.427 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.427 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.427 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:22.427 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545016+0000", 2024-09-18T04:40:22.427 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01000322 2024-09-18T04:40:22.427 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.427 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.427 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:22.427 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545035+0000", 2024-09-18T04:40:22.427 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9845999999999999e-05 2024-09-18T04:40:22.427 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.428 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.428 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:22.428 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545045+0000", 2024-09-18T04:40:22.428 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3640000000000005e-06 2024-09-18T04:40:22.428 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.428 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.428 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:22.428 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545152+0000", 2024-09-18T04:40:22.428 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000107025 2024-09-18T04:40:22.428 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.428 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.428 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:22.428 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545338+0000", 2024-09-18T04:40:22.429 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018643500000000001 2024-09-18T04:40:22.429 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.429 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.429 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:40:22.429 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545492+0000", 2024-09-18T04:40:22.429 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015393199999999999 2024-09-18T04:40:22.429 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.429 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.429 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:22.429 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.553549+0000", 2024-09-18T04:40:22.429 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0080569170000000002 2024-09-18T04:40:22.429 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.429 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.430 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:22.430 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562693+0000", 2024-09-18T04:40:22.430 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0091441609999999996 2024-09-18T04:40:22.430 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.430 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.430 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:22.430 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562748+0000", 2024-09-18T04:40:22.430 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5047000000000003e-05 2024-09-18T04:40:22.430 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.430 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.430 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:22.430 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.562778+0000", 2024-09-18T04:40:22.430 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9652e-05 2024-09-18T04:40:22.431 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:22.431 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:22.431 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:22.431 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.431 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.431 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:40 2.6 2:784d6cd1:ns::rbd_data.133e813ab5bc.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:22.431 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.545062+0000", 2024-09-18T04:40:22.431 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 99.813117767999998, 2024-09-18T04:40:22.431 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.032771623, 2024-09-18T04:40:22.431 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:22.431 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:22.431 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:22.431 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:40:22.431 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:22.432 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 40 2024-09-18T04:40:22.432 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.432 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:22.432 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.432 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:22.432 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545062+0000", 2024-09-18T04:40:22.432 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:22.432 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.432 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.432 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:22.432 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545062+0000", 2024-09-18T04:40:22.433 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:22.433 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.433 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.433 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:22.433 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.545061+0000", 2024-09-18T04:40:22.433 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:22.433 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.433 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.433 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:22.433 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552477+0000", 2024-09-18T04:40:22.433 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.007415914 2024-09-18T04:40:22.433 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.433 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.433 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:22.434 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552500+0000", 2024-09-18T04:40:22.434 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3546000000000001e-05 2024-09-18T04:40:22.434 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.434 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.434 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:22.434 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552515+0000", 2024-09-18T04:40:22.434 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4606999999999999e-05 2024-09-18T04:40:22.434 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.434 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.434 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:22.434 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552661+0000", 2024-09-18T04:40:22.434 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000146007 2024-09-18T04:40:22.434 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.435 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.435 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:22.435 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.552876+0000", 2024-09-18T04:40:22.435 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000214618 2024-09-18T04:40:22.435 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.435 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.435 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:40:22.435 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.553061+0000", 2024-09-18T04:40:22.435 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018564100000000001 2024-09-18T04:40:22.435 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.435 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.435 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:22.435 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.565612+0000", 2024-09-18T04:40:22.435 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012550301999999999 2024-09-18T04:40:22.436 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.436 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.436 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:22.436 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577695+0000", 2024-09-18T04:40:22.436 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012083493000000001 2024-09-18T04:40:22.436 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.436 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.436 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:22.436 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577772+0000", 2024-09-18T04:40:22.436 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7158999999999998e-05 2024-09-18T04:40:22.436 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.436 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.436 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:22.436 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577834+0000", 2024-09-18T04:40:22.437 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1851000000000001e-05 2024-09-18T04:40:22.437 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:22.437 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:22.437 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:22.437 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.437 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.437 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:43 2.6 2:759f27b6:ns::rbd_data.133e813ab5bc.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:22.437 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.555868+0000", 2024-09-18T04:40:22.437 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 99.802312274000002, 2024-09-18T04:40:22.437 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.022332497999999999, 2024-09-18T04:40:22.437 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:22.437 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:22.438 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:22.438 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:40:22.438 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:22.438 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 43 2024-09-18T04:40:22.438 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.438 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:22.438 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.438 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:22.438 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555868+0000", 2024-09-18T04:40:22.438 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:22.438 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.438 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.438 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:22.438 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555868+0000", 2024-09-18T04:40:22.439 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:22.439 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.439 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.439 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:22.439 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.555865+0000", 2024-09-18T04:40:22.439 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999971 2024-09-18T04:40:22.439 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.439 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.439 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:22.439 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561031+0000", 2024-09-18T04:40:22.439 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0051658770000000001 2024-09-18T04:40:22.439 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.439 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.439 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:22.440 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561048+0000", 2024-09-18T04:40:22.440 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6569e-05 2024-09-18T04:40:22.440 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.440 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.440 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:22.440 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561055+0000", 2024-09-18T04:40:22.440 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.328e-06 2024-09-18T04:40:22.440 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.440 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.440 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:22.440 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561114+0000", 2024-09-18T04:40:22.440 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9540000000000003e-05 2024-09-18T04:40:22.440 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.441 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.441 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:22.441 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561314+0000", 2024-09-18T04:40:22.441 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019965799999999999 2024-09-18T04:40:22.441 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.441 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.441 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:40:22.441 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.561473+0000", 2024-09-18T04:40:22.441 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000158904 2024-09-18T04:40:22.441 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.441 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.442 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:22.442 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.567015+0000", 2024-09-18T04:40:22.442 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055420859999999999 2024-09-18T04:40:22.442 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.442 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.442 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:22.442 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577982+0000", 2024-09-18T04:40:22.442 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010966872000000001 2024-09-18T04:40:22.442 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.442 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.442 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:22.442 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578162+0000", 2024-09-18T04:40:22.442 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018050999999999999 2024-09-18T04:40:22.443 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.443 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.443 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:22.443 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578200+0000", 2024-09-18T04:40:22.443 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8000999999999998e-05 2024-09-18T04:40:22.443 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:22.443 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:22.443 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:22.443 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.443 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.443 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:48 2.6 2:7c225365:ns::rbd_data.133e813ab5bc.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:22.443 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:40:22.443 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 99.794513050999996, 2024-09-18T04:40:22.444 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.024576377, 2024-09-18T04:40:22.444 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:22.444 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:22.444 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:22.444 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:40:22.444 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:22.444 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 48 2024-09-18T04:40:22.444 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.445 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:22.445 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.445 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:22.445 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:40:22.445 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:22.445 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.445 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.445 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:22.445 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:40:22.445 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:22.445 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.445 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.445 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:22.446 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.563666+0000", 2024-09-18T04:40:22.446 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:22.446 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.446 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.446 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:22.446 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570331+0000", 2024-09-18T04:40:22.446 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0066653329999999998 2024-09-18T04:40:22.446 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.446 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.446 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:22.446 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570348+0000", 2024-09-18T04:40:22.446 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6507e-05 2024-09-18T04:40:22.446 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.446 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.447 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:22.447 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570355+0000", 2024-09-18T04:40:22.447 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2749999999999998e-06 2024-09-18T04:40:22.447 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.447 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.447 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:22.447 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570396+0000", 2024-09-18T04:40:22.447 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0809000000000002e-05 2024-09-18T04:40:22.447 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.447 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.447 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:22.447 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570571+0000", 2024-09-18T04:40:22.447 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000174823 2024-09-18T04:40:22.447 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.448 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.448 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:40:22.448 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570722+0000", 2024-09-18T04:40:22.448 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015154599999999999 2024-09-18T04:40:22.448 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.448 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.448 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:22.448 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579756+0000", 2024-09-18T04:40:22.448 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0090337000000000004 2024-09-18T04:40:22.448 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.448 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.448 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:22.448 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588093+0000", 2024-09-18T04:40:22.448 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.008336553 2024-09-18T04:40:22.449 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.449 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.449 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:22.449 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588173+0000", 2024-09-18T04:40:22.449 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0333999999999994e-05 2024-09-18T04:40:22.449 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.449 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.449 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:22.449 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.588244+0000", 2024-09-18T04:40:22.449 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0573000000000003e-05 2024-09-18T04:40:22.449 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:22.449 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:22.449 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:22.450 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.450 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.450 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:49 2.4 2:20f0b0c9:ns::rbd_data.133e813ab5bc.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:22.450 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.570370+0000", 2024-09-18T04:40:22.450 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 99.787810723999996, 2024-09-18T04:40:22.450 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.029633981, 2024-09-18T04:40:22.450 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:22.450 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:22.450 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:22.450 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:40:22.450 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:22.450 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 49 2024-09-18T04:40:22.450 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.451 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:22.451 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.451 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:22.451 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570370+0000", 2024-09-18T04:40:22.451 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:22.451 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.451 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.451 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:22.451 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570370+0000", 2024-09-18T04:40:22.451 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:22.451 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.451 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.451 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:22.452 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.570369+0000", 2024-09-18T04:40:22.452 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:22.452 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.452 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.452 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:22.452 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579090+0000", 2024-09-18T04:40:22.452 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0087212339999999996 2024-09-18T04:40:22.452 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.452 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.452 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:22.452 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579109+0000", 2024-09-18T04:40:22.452 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9120999999999999e-05 2024-09-18T04:40:22.452 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.452 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.453 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:22.453 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579123+0000", 2024-09-18T04:40:22.453 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4382000000000001e-05 2024-09-18T04:40:22.453 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.453 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.453 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:22.453 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579221+0000", 2024-09-18T04:40:22.453 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.7503999999999993e-05 2024-09-18T04:40:22.453 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.453 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.453 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:22.453 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579553+0000", 2024-09-18T04:40:22.453 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00033254200000000002 2024-09-18T04:40:22.454 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.454 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.454 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:40:22.454 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579817+0000", 2024-09-18T04:40:22.454 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00026317099999999998 2024-09-18T04:40:22.454 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.454 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.454 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:22.454 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.599905+0000", 2024-09-18T04:40:22.454 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020088575000000001 2024-09-18T04:40:22.454 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.454 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.454 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:22.455 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.599923+0000", 2024-09-18T04:40:22.455 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7425999999999999e-05 2024-09-18T04:40:22.455 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.455 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.455 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:22.455 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.599973+0000", 2024-09-18T04:40:22.455 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0649999999999998e-05 2024-09-18T04:40:22.455 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.455 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.455 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:22.455 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.600004+0000", 2024-09-18T04:40:22.455 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0361999999999998e-05 2024-09-18T04:40:22.455 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:22.455 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:22.456 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:22.456 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.456 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.456 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:50 2.1 e51/24 2:8c706374:ns::rbd_data.133e813ab5bc.000000000000000a:head v 51'8, mlcod=51'6)", 2024-09-18T04:40:22.456 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.577029+0000", 2024-09-18T04:40:22.456 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 99.781151352999998, 2024-09-18T04:40:22.456 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.017362855, 2024-09-18T04:40:22.456 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:22.456 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:22.456 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:22.456 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.456 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:22.456 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577029+0000", 2024-09-18T04:40:22.457 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:22.457 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.457 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.457 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:22.457 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577029+0000", 2024-09-18T04:40:22.457 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:22.457 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.457 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.457 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:22.457 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.577027+0000", 2024-09-18T04:40:22.457 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:22.457 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.457 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.458 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:22.458 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578880+0000", 2024-09-18T04:40:22.458 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0018532080000000001 2024-09-18T04:40:22.458 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.458 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.458 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:22.458 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578906+0000", 2024-09-18T04:40:22.458 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5573999999999999e-05 2024-09-18T04:40:22.458 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.458 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.458 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:22.458 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578920+0000", 2024-09-18T04:40:22.458 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4483e-05 2024-09-18T04:40:22.459 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.459 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.459 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:22.459 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.578983+0000", 2024-09-18T04:40:22.459 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2248000000000004e-05 2024-09-18T04:40:22.459 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.459 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.459 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:22.459 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.579016+0000", 2024-09-18T04:40:22.459 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3364000000000001e-05 2024-09-18T04:40:22.459 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.459 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.459 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:22.459 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594342+0000", 2024-09-18T04:40:22.460 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01532564 2024-09-18T04:40:22.460 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.460 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.460 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:22.460 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.594392+0000", 2024-09-18T04:40:22.460 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0068999999999998e-05 2024-09-18T04:40:22.460 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:22.460 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:22.460 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:22.460 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.460 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.460 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.4926.0:45 2.0 e51/17 2:0fbfa3dc:ns::rbd_data.133e813ab5bc.0000000000000007:head v 51'24, mlcod=51'22)", 2024-09-18T04:40:22.460 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.581892+0000", 2024-09-18T04:40:22.460 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 99.776288680999997, 2024-09-18T04:40:22.461 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015646006, 2024-09-18T04:40:22.461 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:22.461 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:22.461 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:22.461 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.461 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:22.461 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581892+0000", 2024-09-18T04:40:22.461 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:22.461 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.461 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.461 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:22.461 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581892+0000", 2024-09-18T04:40:22.461 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:22.462 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.462 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.462 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:22.462 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.581890+0000", 2024-09-18T04:40:22.462 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:22.462 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.462 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.462 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:22.462 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586188+0000", 2024-09-18T04:40:22.462 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0042985769999999996 2024-09-18T04:40:22.462 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.462 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.462 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:22.462 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586210+0000", 2024-09-18T04:40:22.463 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1069000000000001e-05 2024-09-18T04:40:22.463 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.463 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.463 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:22.463 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586221+0000", 2024-09-18T04:40:22.463 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1287000000000001e-05 2024-09-18T04:40:22.463 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.463 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.463 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:22.463 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586313+0000", 2024-09-18T04:40:22.463 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1935999999999995e-05 2024-09-18T04:40:22.463 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.463 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.463 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:22.464 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.586368+0000", 2024-09-18T04:40:22.464 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5578000000000002e-05 2024-09-18T04:40:22.464 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.464 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.464 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:22.464 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.597473+0000", 2024-09-18T04:40:22.464 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011105139999999999 2024-09-18T04:40:22.464 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.464 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.464 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:22.464 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.597538+0000", 2024-09-18T04:40:22.464 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4133000000000004e-05 2024-09-18T04:40:22.464 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:22.465 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:22.465 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:22.465 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.465 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.465 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:53 2.4 2:39f7f174:ns::rbd_data.133e813ab5bc.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:22.465 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.584263+0000", 2024-09-18T04:40:22.465 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 99.773916807000006, 2024-09-18T04:40:22.465 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018701180000000001, 2024-09-18T04:40:22.465 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:22.465 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:22.465 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:22.465 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:40:22.465 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:22.466 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 53 2024-09-18T04:40:22.466 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.466 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:22.466 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.466 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:22.466 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584263+0000", 2024-09-18T04:40:22.466 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:22.466 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.466 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.466 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:22.466 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584263+0000", 2024-09-18T04:40:22.466 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:22.466 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.467 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.467 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:22.467 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.584262+0000", 2024-09-18T04:40:22.467 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:22.467 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.467 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.467 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:22.467 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.591617+0000", 2024-09-18T04:40:22.467 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0073550600000000001 2024-09-18T04:40:22.467 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.467 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.467 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:22.467 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.591637+0000", 2024-09-18T04:40:22.468 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0432999999999999e-05 2024-09-18T04:40:22.468 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.468 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.468 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:22.468 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.591646+0000", 2024-09-18T04:40:22.468 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7849999999999997e-06 2024-09-18T04:40:22.468 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.468 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.468 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:22.468 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593478+0000", 2024-09-18T04:40:22.468 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001831495 2024-09-18T04:40:22.468 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.468 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.469 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:22.469 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593661+0000", 2024-09-18T04:40:22.469 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018364999999999999 2024-09-18T04:40:22.469 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.469 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.469 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:40:22.469 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.593834+0000", 2024-09-18T04:40:22.469 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000173065 2024-09-18T04:40:22.469 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.469 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.469 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:22.469 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.600670+0000", 2024-09-18T04:40:22.469 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0068354970000000003 2024-09-18T04:40:22.470 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.470 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.470 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:22.470 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602851+0000", 2024-09-18T04:40:22.470 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002181653 2024-09-18T04:40:22.470 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.470 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.470 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:22.470 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602930+0000", 2024-09-18T04:40:22.470 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8045000000000002e-05 2024-09-18T04:40:22.470 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.470 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.470 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:22.471 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.602965+0000", 2024-09-18T04:40:22.471 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5102999999999999e-05 2024-09-18T04:40:22.471 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:22.471 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:22.471 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:22.471 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.471 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.471 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.4926.0:56 2.6 2:72453c3e:ns::rbd_data.133e813ab5bc.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:22.471 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:38:42.595988+0000", 2024-09-18T04:40:22.471 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 99.762192404999993, 2024-09-18T04:40:22.471 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.030296172, 2024-09-18T04:40:22.471 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:22.471 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:22.472 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:22.472 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4926", 2024-09-18T04:40:22.472 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:22.472 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 56 2024-09-18T04:40:22.472 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.472 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:22.472 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.472 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:22.472 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.595988+0000", 2024-09-18T04:40:22.472 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:22.472 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.472 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.472 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:22.473 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.595988+0000", 2024-09-18T04:40:22.473 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:22.473 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.473 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.473 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:22.473 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.595986+0000", 2024-09-18T04:40:22.473 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:40:22.473 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.473 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.473 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:22.473 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605262+0000", 2024-09-18T04:40:22.473 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0092767240000000001 2024-09-18T04:40:22.473 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.473 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.473 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:22.474 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605282+0000", 2024-09-18T04:40:22.474 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9307e-05 2024-09-18T04:40:22.474 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.474 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.474 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:22.474 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605292+0000", 2024-09-18T04:40:22.474 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9510000000000001e-06 2024-09-18T04:40:22.474 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.474 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.474 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:22.474 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605369+0000", 2024-09-18T04:40:22.474 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7846e-05 2024-09-18T04:40:22.474 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.474 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.475 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:22.475 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605668+0000", 2024-09-18T04:40:22.475 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00029896099999999999 2024-09-18T04:40:22.475 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.475 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.475 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-09-18T04:40:22.475 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.605906+0000", 2024-09-18T04:40:22.475 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000237758 2024-09-18T04:40:22.475 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.475 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.475 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:22.475 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.613532+0000", 2024-09-18T04:40:22.475 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0076260590000000001 2024-09-18T04:40:22.476 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.476 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.476 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:22.476 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.626116+0000", 2024-09-18T04:40:22.476 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012584092999999999 2024-09-18T04:40:22.476 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.476 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.476 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:22.476 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.626216+0000", 2024-09-18T04:40:22.476 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9630999999999995e-05 2024-09-18T04:40:22.476 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:22.476 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:22.476 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:22.476 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:38:42.626284+0000", 2024-09-18T04:40:22.477 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8034999999999995e-05 2024-09-18T04:40:22.477 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:22.477 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:22.477 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:22.477 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:22.477 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:22.477 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:40:22.477 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:40:22.359+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:40:22.478 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:40:22.480 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:40:22.475+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:40:22.580 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:40:22.575+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:40:22.595 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:40:22.595 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:40:22.595 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:40:22.596 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:40:22.605 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:40:22.681 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:40:22.675+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:40:22.719 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:40:22.719 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:40:22.719 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:40:22.719 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:40:22.719 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:40:22.728 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:40:22.754 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd -p rbd --namespace ns status testrbdnbd17342 2024-09-18T04:40:22.754 INFO:tasks.workunit.client.0.smithi115.stderr:+ grep 'Watchers: none' 2024-09-18T04:40:22.782 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:40:22.775+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:40:22.826 INFO:tasks.workunit.client.0.smithi115.stdout:Watchers: none 2024-09-18T04:40:22.826 INFO:tasks.workunit.client.0.smithi115.stderr:+ break 2024-09-18T04:40:22.826 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd -p rbd --namespace ns snap purge testrbdnbd17342 2024-09-18T04:40:22.838 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:40:22.838 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:40:22.838 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:40:22.838 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:40:22.838 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.839 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:40:22.839 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:40:22.839 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 175.045679882, 2024-09-18T04:40:22.839 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017399218000000001, 2024-09-18T04:40:22.839 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:22.839 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:22.839 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:22.839 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:40:22.839 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:40:22.839 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 114 2024-09-18T04:40:22.839 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.839 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:22.839 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.840 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:22.840 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:40:22.840 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:22.840 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.840 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.840 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:22.840 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786901+0000", 2024-09-18T04:40:22.840 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:22.840 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.840 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.840 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:22.840 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786900+0000", 2024-09-18T04:40:22.840 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:40:22.841 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.841 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.841 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:22.841 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786908+0000", 2024-09-18T04:40:22.841 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8199999999999997e-06 2024-09-18T04:40:22.841 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.841 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.841 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:22.841 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786925+0000", 2024-09-18T04:40:22.841 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6849000000000001e-05 2024-09-18T04:40:22.841 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.841 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.841 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:22.842 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.786934+0000", 2024-09-18T04:40:22.842 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6299999999999993e-06 2024-09-18T04:40:22.842 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.842 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.842 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:22.842 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.800926+0000", 2024-09-18T04:40:22.842 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013991632 2024-09-18T04:40:22.842 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.842 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.842 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:22.842 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801060+0000", 2024-09-18T04:40:22.842 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013378199999999999 2024-09-18T04:40:22.842 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.842 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.843 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:40:22.843 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.801166+0000", 2024-09-18T04:40:22.843 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010607099999999999 2024-09-18T04:40:22.843 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.843 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.843 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:22.843 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803026+0000", 2024-09-18T04:40:22.843 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001860469 2024-09-18T04:40:22.843 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.843 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.843 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:22.843 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804245+0000", 2024-09-18T04:40:22.843 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0012185900000000001 2024-09-18T04:40:22.844 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.844 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.844 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:22.844 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804278+0000", 2024-09-18T04:40:22.844 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3272000000000003e-05 2024-09-18T04:40:22.844 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.844 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.844 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:22.844 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804300+0000", 2024-09-18T04:40:22.844 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2266e-05 2024-09-18T04:40:22.844 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:22.844 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:22.844 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:22.845 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.845 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.845 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:40:22.845 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:40:22.845 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 175.04476698400001, 2024-09-18T04:40:22.845 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017521662, 2024-09-18T04:40:22.845 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:22.845 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:22.845 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:22.845 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:40:22.846 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:40:22.846 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 116 2024-09-18T04:40:22.846 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.846 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:22.846 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.846 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:22.846 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:40:22.846 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:22.846 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.846 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.846 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:22.846 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787814+0000", 2024-09-18T04:40:22.847 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:22.847 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.847 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.847 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:22.847 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787813+0000", 2024-09-18T04:40:22.847 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:22.847 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.847 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.847 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:22.847 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787824+0000", 2024-09-18T04:40:22.847 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1157e-05 2024-09-18T04:40:22.847 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.848 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.848 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:22.848 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787840+0000", 2024-09-18T04:40:22.848 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6269e-05 2024-09-18T04:40:22.848 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.848 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.848 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:22.848 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787853+0000", 2024-09-18T04:40:22.848 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3156000000000001e-05 2024-09-18T04:40:22.848 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.848 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.848 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:22.848 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803110+0000", 2024-09-18T04:40:22.849 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015256736 2024-09-18T04:40:22.849 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.849 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.849 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:22.849 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803197+0000", 2024-09-18T04:40:22.849 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7195999999999997e-05 2024-09-18T04:40:22.849 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.849 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.849 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:40:22.849 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803308+0000", 2024-09-18T04:40:22.849 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111005 2024-09-18T04:40:22.849 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.849 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.849 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:22.850 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804316+0000", 2024-09-18T04:40:22.850 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010073949999999999 2024-09-18T04:40:22.850 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.850 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.850 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:22.850 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805232+0000", 2024-09-18T04:40:22.850 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091636300000000001 2024-09-18T04:40:22.850 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.850 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.850 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:22.850 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805277+0000", 2024-09-18T04:40:22.850 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4864000000000003e-05 2024-09-18T04:40:22.850 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.850 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.851 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:22.851 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805336+0000", 2024-09-18T04:40:22.851 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8968000000000002e-05 2024-09-18T04:40:22.851 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:22.851 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:22.851 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:22.851 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.851 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.851 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4186.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-18T04:40:22.851 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:40:22.851 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 175.04471492900001, 2024-09-18T04:40:22.851 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.017983453, 2024-09-18T04:40:22.851 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:22.851 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:22.852 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:22.852 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4186", 2024-09-18T04:40:22.852 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.82:0/1176872315", 2024-09-18T04:40:22.852 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 117 2024-09-18T04:40:22.852 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.852 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:22.852 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.852 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:22.852 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:40:22.852 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:22.852 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.852 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.852 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:22.853 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787866+0000", 2024-09-18T04:40:22.853 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:22.853 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.853 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.853 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:22.853 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787865+0000", 2024-09-18T04:40:22.853 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:40:22.853 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.853 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.853 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:22.853 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787889+0000", 2024-09-18T04:40:22.853 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4244999999999999e-05 2024-09-18T04:40:22.853 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.854 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.854 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:22.854 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787905+0000", 2024-09-18T04:40:22.854 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5739999999999998e-05 2024-09-18T04:40:22.854 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.854 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.854 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:22.854 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.787918+0000", 2024-09-18T04:40:22.854 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.252e-05 2024-09-18T04:40:22.854 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.854 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.854 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:22.854 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803693+0000", 2024-09-18T04:40:22.854 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015774872999999998 2024-09-18T04:40:22.855 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.855 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.855 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:22.855 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803775+0000", 2024-09-18T04:40:22.855 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2761000000000003e-05 2024-09-18T04:40:22.855 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.855 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.855 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:40:22.855 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.803887+0000", 2024-09-18T04:40:22.855 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111708 2024-09-18T04:40:22.855 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.855 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.855 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:22.855 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.804823+0000", 2024-09-18T04:40:22.856 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00093625100000000003 2024-09-18T04:40:22.856 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.856 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.856 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:22.856 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805804+0000", 2024-09-18T04:40:22.856 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00098114699999999992 2024-09-18T04:40:22.856 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.856 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.856 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:22.856 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805831+0000", 2024-09-18T04:40:22.856 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6401999999999999e-05 2024-09-18T04:40:22.856 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.856 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.856 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:22.857 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:27.805850+0000", 2024-09-18T04:40:22.857 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8862e-05 2024-09-18T04:40:22.857 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:22.857 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:22.857 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:22.857 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.857 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.857 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:36 2.7 2:fe1b02ad:::rbd_data.125fc09df6d1.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:22.857 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:40:22.857 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 100.566858819, 2024-09-18T04:40:22.857 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.040387804999999999, 2024-09-18T04:40:22.857 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:22.857 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:22.857 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:22.858 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:40:22.858 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:40:22.858 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 36 2024-09-18T04:40:22.858 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.858 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:22.858 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.858 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:22.858 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:40:22.858 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:22.858 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.858 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.858 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:22.858 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265722+0000", 2024-09-18T04:40:22.859 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:22.859 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.859 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.859 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:22.859 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.265720+0000", 2024-09-18T04:40:22.859 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:40:22.859 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.859 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.859 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:22.859 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272590+0000", 2024-09-18T04:40:22.859 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.006870508 2024-09-18T04:40:22.859 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.859 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.859 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:22.860 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272623+0000", 2024-09-18T04:40:22.860 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2716000000000003e-05 2024-09-18T04:40:22.860 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.860 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.860 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:22.860 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272637+0000", 2024-09-18T04:40:22.860 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4175999999999999e-05 2024-09-18T04:40:22.860 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.860 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.860 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:22.860 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272779+0000", 2024-09-18T04:40:22.860 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000141392 2024-09-18T04:40:22.860 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.860 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.861 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:22.861 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273031+0000", 2024-09-18T04:40:22.861 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00025249099999999998 2024-09-18T04:40:22.861 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.861 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.861 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:40:22.861 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273219+0000", 2024-09-18T04:40:22.861 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000187531 2024-09-18T04:40:22.861 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.861 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.861 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:22.861 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.302806+0000", 2024-09-18T04:40:22.861 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.029587553999999999 2024-09-18T04:40:22.862 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.862 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.862 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:22.862 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306004+0000", 2024-09-18T04:40:22.862 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0031982629999999998 2024-09-18T04:40:22.862 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.862 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.862 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:22.862 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306071+0000", 2024-09-18T04:40:22.862 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6829999999999995e-05 2024-09-18T04:40:22.862 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.862 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.862 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:22.862 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306110+0000", 2024-09-18T04:40:22.863 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8874000000000001e-05 2024-09-18T04:40:22.863 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:22.863 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:22.863 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:22.863 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.863 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.863 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:38 2.0 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:22.863 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:40:22.863 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 100.559905282, 2024-09-18T04:40:22.863 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.033754348000000003, 2024-09-18T04:40:22.863 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:22.863 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:22.863 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:22.864 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:40:22.864 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:40:22.864 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 38 2024-09-18T04:40:22.864 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.864 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:22.864 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.864 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:22.864 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:40:22.864 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:22.864 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.864 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.864 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:22.864 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272676+0000", 2024-09-18T04:40:22.865 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:22.865 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.865 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.865 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:22.865 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272673+0000", 2024-09-18T04:40:22.865 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:40:22.865 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.865 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.865 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:22.865 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277166+0000", 2024-09-18T04:40:22.865 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044922249999999999 2024-09-18T04:40:22.865 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.866 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.866 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:22.866 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277189+0000", 2024-09-18T04:40:22.866 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3283000000000001e-05 2024-09-18T04:40:22.866 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.866 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.866 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:22.866 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277205+0000", 2024-09-18T04:40:22.866 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6305000000000001e-05 2024-09-18T04:40:22.866 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.866 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.866 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:22.866 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277290+0000", 2024-09-18T04:40:22.866 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5135000000000004e-05 2024-09-18T04:40:22.867 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.867 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.867 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:22.867 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277531+0000", 2024-09-18T04:40:22.867 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00024014699999999999 2024-09-18T04:40:22.867 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.867 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.867 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:22.867 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277793+0000", 2024-09-18T04:40:22.867 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00026224499999999998 2024-09-18T04:40:22.867 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.867 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.867 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:22.868 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303990+0000", 2024-09-18T04:40:22.868 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.026197295999999998 2024-09-18T04:40:22.868 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.868 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.868 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:22.868 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306340+0000", 2024-09-18T04:40:22.868 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.002349524 2024-09-18T04:40:22.868 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.868 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.868 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:22.869 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306394+0000", 2024-09-18T04:40:22.869 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3934999999999998e-05 2024-09-18T04:40:22.869 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.869 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.869 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:22.869 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306430+0000", 2024-09-18T04:40:22.869 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6690000000000003e-05 2024-09-18T04:40:22.869 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:22.869 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:22.869 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:22.869 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.869 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.870 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:39 2.7 2:e9c909c5:::rbd_data.125fc09df6d1.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:22.870 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:40:22.870 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 100.55534567700001, 2024-09-18T04:40:22.870 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.029110673, 2024-09-18T04:40:22.870 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:22.870 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:22.870 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:22.870 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:40:22.870 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:40:22.870 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 39 2024-09-18T04:40:22.870 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.870 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:22.870 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.870 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:22.871 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:40:22.871 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:22.871 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.871 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.871 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:22.871 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277235+0000", 2024-09-18T04:40:22.871 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:22.871 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.871 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.871 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:22.871 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.277234+0000", 2024-09-18T04:40:22.871 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:22.871 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.871 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.872 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:22.872 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281351+0000", 2024-09-18T04:40:22.872 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0041170920000000001 2024-09-18T04:40:22.872 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.872 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.872 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:22.872 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281375+0000", 2024-09-18T04:40:22.872 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.446e-05 2024-09-18T04:40:22.872 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.872 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.872 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:22.872 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.281390+0000", 2024-09-18T04:40:22.872 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4309e-05 2024-09-18T04:40:22.872 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.873 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.873 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:22.873 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.282306+0000", 2024-09-18T04:40:22.873 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091662000000000004 2024-09-18T04:40:22.873 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.873 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.873 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:22.873 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.282613+0000", 2024-09-18T04:40:22.873 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00030663799999999997 2024-09-18T04:40:22.873 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.873 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.873 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:40:22.873 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283001+0000", 2024-09-18T04:40:22.873 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00038801599999999998 2024-09-18T04:40:22.874 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.874 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.874 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:22.874 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306239+0000", 2024-09-18T04:40:22.874 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023238466999999999 2024-09-18T04:40:22.874 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.874 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.874 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:22.874 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306259+0000", 2024-09-18T04:40:22.874 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9837999999999999e-05 2024-09-18T04:40:22.874 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.874 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.874 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:22.875 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306299+0000", 2024-09-18T04:40:22.875 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0194000000000002e-05 2024-09-18T04:40:22.875 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.875 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.875 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:22.875 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306346+0000", 2024-09-18T04:40:22.875 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6845000000000003e-05 2024-09-18T04:40:22.875 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:22.875 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:22.875 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:22.875 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.875 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.875 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:46 2.6 e51/17 2:7ba25245:::rbd_data.125fc09df6d1.0000000000000006:head v 51'2, mlcod=0'0)", 2024-09-18T04:40:22.876 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:40:22.876 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 100.51942454500001, 2024-09-18T04:40:22.876 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019843098, 2024-09-18T04:40:22.876 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:22.876 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:22.876 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:22.876 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.876 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:22.876 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:40:22.876 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:22.876 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.876 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.876 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:22.877 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313157+0000", 2024-09-18T04:40:22.877 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:22.877 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.877 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.877 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:22.877 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.313156+0000", 2024-09-18T04:40:22.877 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:40:22.877 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.877 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.877 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:22.877 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320144+0000", 2024-09-18T04:40:22.877 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.006988345 2024-09-18T04:40:22.877 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.878 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.878 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:22.878 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320175+0000", 2024-09-18T04:40:22.878 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.0861999999999997e-05 2024-09-18T04:40:22.878 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.878 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.878 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:22.878 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320188+0000", 2024-09-18T04:40:22.878 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2714999999999999e-05 2024-09-18T04:40:22.878 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.878 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.878 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:22.878 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320267+0000", 2024-09-18T04:40:22.878 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9586999999999995e-05 2024-09-18T04:40:22.879 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.879 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.879 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:22.879 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.320311+0000", 2024-09-18T04:40:22.879 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3529000000000003e-05 2024-09-18T04:40:22.879 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.879 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.879 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:22.879 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.332949+0000", 2024-09-18T04:40:22.879 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012638267999999999 2024-09-18T04:40:22.879 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.879 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.879 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:22.879 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.333000+0000", 2024-09-18T04:40:22.879 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0736999999999999e-05 2024-09-18T04:40:22.880 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:22.880 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:22.880 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:22.880 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.880 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.880 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:57 2.0 2:0cdd070e:::rbd_data.125fc09df6d1.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:22.880 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:40:22.880 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 100.48356697, 2024-09-18T04:40:22.880 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.033329974999999998, 2024-09-18T04:40:22.880 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:22.880 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:22.880 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:22.880 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:40:22.881 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:40:22.881 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 57 2024-09-18T04:40:22.881 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.881 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:22.881 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.881 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:22.881 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:40:22.881 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:22.881 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.881 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.881 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:22.881 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349014+0000", 2024-09-18T04:40:22.881 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:22.882 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.882 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.882 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:22.882 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.349013+0000", 2024-09-18T04:40:22.882 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:22.882 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.882 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.882 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:22.882 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356138+0000", 2024-09-18T04:40:22.882 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071258010000000002 2024-09-18T04:40:22.882 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.882 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.882 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:22.882 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356168+0000", 2024-09-18T04:40:22.883 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9679000000000002e-05 2024-09-18T04:40:22.883 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.883 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.883 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:22.883 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356181+0000", 2024-09-18T04:40:22.883 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3164e-05 2024-09-18T04:40:22.883 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.883 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.883 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:22.883 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356254+0000", 2024-09-18T04:40:22.883 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3182999999999994e-05 2024-09-18T04:40:22.883 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.883 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.883 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:22.884 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356453+0000", 2024-09-18T04:40:22.884 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019876599999999999 2024-09-18T04:40:22.884 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.884 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.884 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:22.884 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356627+0000", 2024-09-18T04:40:22.884 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017340199999999999 2024-09-18T04:40:22.884 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.884 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.884 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:22.884 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.369200+0000", 2024-09-18T04:40:22.884 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012573302 2024-09-18T04:40:22.884 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.884 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.885 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:22.885 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382262+0000", 2024-09-18T04:40:22.885 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013062549 2024-09-18T04:40:22.885 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.885 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.885 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:22.885 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382310+0000", 2024-09-18T04:40:22.885 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7366e-05 2024-09-18T04:40:22.885 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.885 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.885 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:22.885 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.382344+0000", 2024-09-18T04:40:22.885 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4319999999999997e-05 2024-09-18T04:40:22.886 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:22.886 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:22.886 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:22.886 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.886 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.886 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:59 2.0 2:19c52cde:::rbd_data.125fc09df6d1.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:22.886 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:40:22.886 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 100.476375805, 2024-09-18T04:40:22.886 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.027137247999999999, 2024-09-18T04:40:22.886 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:22.886 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:22.886 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:22.886 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:40:22.887 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:40:22.887 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 59 2024-09-18T04:40:22.887 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.887 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:22.887 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.887 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:22.887 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:40:22.887 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:22.887 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.887 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.887 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:22.887 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356205+0000", 2024-09-18T04:40:22.887 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:22.887 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.888 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.888 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:22.888 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.356204+0000", 2024-09-18T04:40:22.888 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:22.888 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.888 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.888 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:22.888 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362661+0000", 2024-09-18T04:40:22.888 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0064565129999999997 2024-09-18T04:40:22.888 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.888 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.888 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:22.888 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362684+0000", 2024-09-18T04:40:22.889 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3688000000000001e-05 2024-09-18T04:40:22.889 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.889 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.889 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:22.889 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.362699+0000", 2024-09-18T04:40:22.889 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4798e-05 2024-09-18T04:40:22.889 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.889 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.889 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:22.889 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363016+0000", 2024-09-18T04:40:22.889 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031737300000000003 2024-09-18T04:40:22.889 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.890 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.890 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:22.890 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363201+0000", 2024-09-18T04:40:22.890 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000184131 2024-09-18T04:40:22.890 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.890 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.890 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:22.890 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.363343+0000", 2024-09-18T04:40:22.890 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014236699999999999 2024-09-18T04:40:22.890 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.890 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.890 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:22.890 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.372189+0000", 2024-09-18T04:40:22.891 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0088464819999999993 2024-09-18T04:40:22.891 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.891 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.891 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:22.891 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383269+0000", 2024-09-18T04:40:22.891 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011079594 2024-09-18T04:40:22.891 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.891 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.891 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:22.891 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383314+0000", 2024-09-18T04:40:22.891 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4932000000000003e-05 2024-09-18T04:40:22.891 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.892 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.892 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:22.892 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.383343+0000", 2024-09-18T04:40:22.892 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8600999999999999e-05 2024-09-18T04:40:22.892 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:22.892 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:22.892 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:22.892 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.892 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.892 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:40 2.6 e51/17 2:784d6cd1:ns::rbd_data.133e813ab5bc.0000000000000005:head v 51'7, mlcod=51'5)", 2024-09-18T04:40:22.892 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:40:22.893 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 100.280126071, 2024-09-18T04:40:22.893 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020497496, 2024-09-18T04:40:22.893 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:22.893 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:22.893 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:22.893 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.893 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:22.893 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:40:22.893 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:22.893 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.893 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.893 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:22.893 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552455+0000", 2024-09-18T04:40:22.894 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:22.894 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.894 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.894 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:22.894 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552454+0000", 2024-09-18T04:40:22.894 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:22.894 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.894 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.894 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:22.894 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556448+0000", 2024-09-18T04:40:22.894 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0039940990000000001 2024-09-18T04:40:22.894 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.894 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.895 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:22.895 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556481+0000", 2024-09-18T04:40:22.895 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3778000000000001e-05 2024-09-18T04:40:22.895 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.895 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.895 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:22.895 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556493+0000", 2024-09-18T04:40:22.895 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1622e-05 2024-09-18T04:40:22.895 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.895 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.895 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:22.895 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556556+0000", 2024-09-18T04:40:22.895 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2582999999999994e-05 2024-09-18T04:40:22.896 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.896 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.896 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:22.896 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.556592+0000", 2024-09-18T04:40:22.896 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5880999999999997e-05 2024-09-18T04:40:22.896 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.896 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.896 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:22.896 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.572786+0000", 2024-09-18T04:40:22.896 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016194571000000001 2024-09-18T04:40:22.896 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.896 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.896 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:22.897 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.572953+0000", 2024-09-18T04:40:22.897 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000166436 2024-09-18T04:40:22.897 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:22.897 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:22.897 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:22.897 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.897 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.897 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:45 2.0 2:0fbfa3dc:ns::rbd_data.133e813ab5bc.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:22.897 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:40:22.897 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 100.268911772, 2024-09-18T04:40:22.897 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.036558168000000002, 2024-09-18T04:40:22.897 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:22.897 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:22.898 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:22.898 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:40:22.898 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:22.898 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 45 2024-09-18T04:40:22.898 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.898 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:22.898 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.898 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:22.898 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:40:22.898 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:22.898 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.898 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.898 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:22.898 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563669+0000", 2024-09-18T04:40:22.899 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:22.899 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.899 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.899 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:22.899 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.563667+0000", 2024-09-18T04:40:22.899 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:40:22.899 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.899 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.899 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:22.899 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578594+0000", 2024-09-18T04:40:22.899 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014927147999999999 2024-09-18T04:40:22.899 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.899 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.899 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:22.900 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578612+0000", 2024-09-18T04:40:22.900 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7623999999999999e-05 2024-09-18T04:40:22.900 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.900 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.900 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:22.900 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578620+0000", 2024-09-18T04:40:22.900 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9070000000000007e-06 2024-09-18T04:40:22.900 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.900 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.900 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:22.900 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.578702+0000", 2024-09-18T04:40:22.900 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2349999999999996e-05 2024-09-18T04:40:22.900 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.901 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.901 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:22.901 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.579028+0000", 2024-09-18T04:40:22.901 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032622899999999998 2024-09-18T04:40:22.901 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.901 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.901 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:22.901 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.579343+0000", 2024-09-18T04:40:22.901 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031448899999999999 2024-09-18T04:40:22.901 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.901 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.901 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:22.901 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.590078+0000", 2024-09-18T04:40:22.901 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010734946 2024-09-18T04:40:22.902 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.902 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.902 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:22.902 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600089+0000", 2024-09-18T04:40:22.902 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010011195000000001 2024-09-18T04:40:22.902 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.902 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.902 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:22.902 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600177+0000", 2024-09-18T04:40:22.903 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8425999999999997e-05 2024-09-18T04:40:22.903 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.903 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.903 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:22.903 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.600228+0000", 2024-09-18T04:40:22.903 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0216999999999997e-05 2024-09-18T04:40:22.903 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:22.903 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:22.903 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:22.903 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.903 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.904 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:54 2.0 2:04f17ef6:ns::rbd_data.133e813ab5bc.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:22.904 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:40:22.904 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 100.247339935, 2024-09-18T04:40:22.904 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.024612999, 2024-09-18T04:40:22.904 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:22.904 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:22.904 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:22.904 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:40:22.904 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:22.904 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 54 2024-09-18T04:40:22.904 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.905 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:22.905 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.905 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:22.905 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:40:22.905 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:22.905 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.905 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.905 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:22.905 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585241+0000", 2024-09-18T04:40:22.905 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:22.905 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.905 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.905 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:22.905 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.585239+0000", 2024-09-18T04:40:22.906 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:22.906 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.906 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.906 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:22.906 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593702+0000", 2024-09-18T04:40:22.906 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0084632990000000005 2024-09-18T04:40:22.906 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.906 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.906 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:22.906 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593719+0000", 2024-09-18T04:40:22.906 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.664e-05 2024-09-18T04:40:22.906 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.906 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.906 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:22.908 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593727+0000", 2024-09-18T04:40:22.909 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6839999999999996e-06 2024-09-18T04:40:22.909 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.909 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.909 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:22.909 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593767+0000", 2024-09-18T04:40:22.909 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0330999999999997e-05 2024-09-18T04:40:22.909 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.909 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.909 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:22.909 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.593905+0000", 2024-09-18T04:40:22.909 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013754800000000001 2024-09-18T04:40:22.909 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.909 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.909 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:22.910 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.594013+0000", 2024-09-18T04:40:22.910 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000108805 2024-09-18T04:40:22.910 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.910 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.910 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:22.910 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604678+0000", 2024-09-18T04:40:22.910 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010664391 2024-09-18T04:40:22.910 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.910 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.910 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:22.910 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609662+0000", 2024-09-18T04:40:22.910 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0049846100000000004 2024-09-18T04:40:22.910 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.911 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.911 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:22.911 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609788+0000", 2024-09-18T04:40:22.911 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012518800000000001 2024-09-18T04:40:22.911 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.911 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.911 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:22.911 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609854+0000", 2024-09-18T04:40:22.911 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6556000000000004e-05 2024-09-18T04:40:22.911 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:22.911 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:22.911 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:22.911 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.911 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.912 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:58 2.7 2:fc7c6cd3:ns::rbd_data.133e813ab5bc.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:22.912 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:40:22.912 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 100.234497506, 2024-09-18T04:40:22.912 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.027690566999999999, 2024-09-18T04:40:22.912 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:22.912 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:22.912 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:22.912 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:40:22.912 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:22.912 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 58 2024-09-18T04:40:22.912 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.912 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:22.913 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.913 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:22.913 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:40:22.913 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:22.913 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.913 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.913 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:22.913 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598084+0000", 2024-09-18T04:40:22.913 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:22.913 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.913 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.913 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:22.913 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.598082+0000", 2024-09-18T04:40:22.914 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:22.914 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.914 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.914 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:22.914 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604832+0000", 2024-09-18T04:40:22.914 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0067504510000000002 2024-09-18T04:40:22.914 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.914 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.914 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:22.914 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604852+0000", 2024-09-18T04:40:22.914 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0228999999999999e-05 2024-09-18T04:40:22.914 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.915 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.915 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:22.915 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604870+0000", 2024-09-18T04:40:22.915 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7952e-05 2024-09-18T04:40:22.915 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.915 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.915 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:22.915 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.604936+0000", 2024-09-18T04:40:22.915 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5431000000000004e-05 2024-09-18T04:40:22.915 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.915 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.915 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:22.915 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605085+0000", 2024-09-18T04:40:22.916 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014910300000000001 2024-09-18T04:40:22.916 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.916 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.916 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5", 2024-09-18T04:40:22.916 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605205+0000", 2024-09-18T04:40:22.916 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000119887 2024-09-18T04:40:22.916 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.916 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.916 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:22.916 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625586+0000", 2024-09-18T04:40:22.916 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020381399000000001 2024-09-18T04:40:22.917 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.917 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.917 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:22.917 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625628+0000", 2024-09-18T04:40:22.917 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1873999999999999e-05 2024-09-18T04:40:22.917 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.917 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.917 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:22.917 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625722+0000", 2024-09-18T04:40:22.917 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3787000000000002e-05 2024-09-18T04:40:22.917 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.917 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.917 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:22.917 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625774+0000", 2024-09-18T04:40:22.918 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2503999999999997e-05 2024-09-18T04:40:22.918 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:22.918 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:22.918 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:22.918 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.918 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.918 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:56 2.6 e51/17 2:72453c3e:ns::rbd_data.133e813ab5bc.000000000000000d:head v 51'10, mlcod=51'8)", 2024-09-18T04:40:22.918 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:40:22.918 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 100.227300398, 2024-09-18T04:40:22.918 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019766552999999999, 2024-09-18T04:40:22.918 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:22.918 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:22.918 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:22.919 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.919 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:22.919 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:40:22.919 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:22.919 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.919 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.919 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:22.919 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605281+0000", 2024-09-18T04:40:22.919 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:22.919 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.919 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.919 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:22.919 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605279+0000", 2024-09-18T04:40:22.920 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:22.920 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.920 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.920 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:22.920 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609417+0000", 2024-09-18T04:40:22.920 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0041378059999999999 2024-09-18T04:40:22.920 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.920 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.920 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:22.920 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609443+0000", 2024-09-18T04:40:22.920 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6098e-05 2024-09-18T04:40:22.920 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.920 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.921 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:22.921 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609458+0000", 2024-09-18T04:40:22.921 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4671000000000001e-05 2024-09-18T04:40:22.921 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.921 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.921 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:22.921 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609553+0000", 2024-09-18T04:40:22.921 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5265000000000004e-05 2024-09-18T04:40:22.921 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.921 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.921 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:22.921 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.609583+0000", 2024-09-18T04:40:22.921 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9839e-05 2024-09-18T04:40:22.921 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.922 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.922 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:22.922 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625004+0000", 2024-09-18T04:40:22.922 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015420639999999999 2024-09-18T04:40:22.922 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.922 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.922 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:22.922 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625047+0000", 2024-09-18T04:40:22.922 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3665999999999998e-05 2024-09-18T04:40:22.922 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:22.922 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:22.922 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:22.922 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.922 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.923 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:99 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:40:22.923 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.582718+0000", 2024-09-18T04:40:22.923 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 86.249863489000006, 2024-09-18T04:40:22.923 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019362575, 2024-09-18T04:40:22.923 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:22.923 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:22.923 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:22.923 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:40:22.923 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:40:22.923 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 99 2024-09-18T04:40:22.923 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.923 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:22.923 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.923 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:22.924 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582718+0000", 2024-09-18T04:40:22.924 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:22.924 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.924 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.924 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:22.924 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582718+0000", 2024-09-18T04:40:22.924 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:22.924 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.924 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.924 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:22.924 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582717+0000", 2024-09-18T04:40:22.924 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:40:22.925 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.925 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.925 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:22.925 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582720+0000", 2024-09-18T04:40:22.925 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2239999999999998e-06 2024-09-18T04:40:22.925 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.925 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.925 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:22.925 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582723+0000", 2024-09-18T04:40:22.925 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.258e-06 2024-09-18T04:40:22.925 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.925 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.925 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:22.926 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.582726+0000", 2024-09-18T04:40:22.926 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8870000000000001e-06 2024-09-18T04:40:22.926 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.926 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.926 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:22.926 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.592018+0000", 2024-09-18T04:40:22.926 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0092926420000000003 2024-09-18T04:40:22.926 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.926 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.926 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:22.926 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.592214+0000", 2024-09-18T04:40:22.926 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000195267 2024-09-18T04:40:22.926 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.927 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.927 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:22.927 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.592913+0000", 2024-09-18T04:40:22.927 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00069924600000000005 2024-09-18T04:40:22.927 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.927 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.927 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:22.927 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597493+0000", 2024-09-18T04:40:22.927 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045797629999999997 2024-09-18T04:40:22.927 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.927 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.927 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:22.927 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.602009+0000", 2024-09-18T04:40:22.928 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045159559999999998 2024-09-18T04:40:22.928 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.928 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.928 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:22.928 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.602042+0000", 2024-09-18T04:40:22.928 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3084999999999999e-05 2024-09-18T04:40:22.928 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.928 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.928 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:22.928 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.602080+0000", 2024-09-18T04:40:22.928 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8600000000000003e-05 2024-09-18T04:40:22.928 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:22.928 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:22.929 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:22.929 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.929 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.929 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:104 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:40:22.929 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.583165+0000", 2024-09-18T04:40:22.929 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 86.249416190999995, 2024-09-18T04:40:22.929 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023679603, 2024-09-18T04:40:22.929 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:22.929 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:22.929 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:22.929 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:40:22.929 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:40:22.930 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 104 2024-09-18T04:40:22.930 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.930 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:22.930 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.930 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:22.930 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583165+0000", 2024-09-18T04:40:22.930 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:22.930 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.930 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.930 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:22.930 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583165+0000", 2024-09-18T04:40:22.930 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:22.930 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.930 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.931 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:22.931 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583164+0000", 2024-09-18T04:40:22.931 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:40:22.931 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.931 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.931 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:22.931 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583167+0000", 2024-09-18T04:40:22.931 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9210000000000001e-06 2024-09-18T04:40:22.931 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.931 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.931 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:22.931 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583171+0000", 2024-09-18T04:40:22.931 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9759999999999997e-06 2024-09-18T04:40:22.931 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.932 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.932 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:22.932 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583174+0000", 2024-09-18T04:40:22.932 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9960000000000002e-06 2024-09-18T04:40:22.932 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.932 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.932 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:22.932 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.594373+0000", 2024-09-18T04:40:22.932 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011198665999999999 2024-09-18T04:40:22.932 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.932 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.932 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:22.932 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.594434+0000", 2024-09-18T04:40:22.932 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.1446999999999996e-05 2024-09-18T04:40:22.933 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.933 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.933 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:22.933 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.594692+0000", 2024-09-18T04:40:22.933 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00025756599999999998 2024-09-18T04:40:22.933 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.933 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.933 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:22.933 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599302+0000", 2024-09-18T04:40:22.933 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0046099640000000002 2024-09-18T04:40:22.933 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.933 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.933 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:22.934 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606710+0000", 2024-09-18T04:40:22.934 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0074077739999999998 2024-09-18T04:40:22.934 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.934 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.934 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:22.934 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606798+0000", 2024-09-18T04:40:22.934 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8133999999999993e-05 2024-09-18T04:40:22.934 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.934 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.934 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:22.934 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606845+0000", 2024-09-18T04:40:22.934 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6653000000000002e-05 2024-09-18T04:40:22.934 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:22.935 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:22.935 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:22.935 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.935 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.935 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:105 2.0 2:1e6e806e:::rbd_data.125fc09df6d1.0000000000000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:40:22.935 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.583193+0000", 2024-09-18T04:40:22.935 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 86.249388572000001, 2024-09-18T04:40:22.935 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.023713649, 2024-09-18T04:40:22.935 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:22.935 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:22.935 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:22.935 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:40:22.936 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:40:22.936 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 105 2024-09-18T04:40:22.936 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.936 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:22.936 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.936 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:22.936 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583193+0000", 2024-09-18T04:40:22.936 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:22.936 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.936 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.936 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:22.936 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583193+0000", 2024-09-18T04:40:22.936 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:22.937 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.937 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.937 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:22.937 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583192+0000", 2024-09-18T04:40:22.937 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999995 2024-09-18T04:40:22.937 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.937 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.937 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:22.937 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583194+0000", 2024-09-18T04:40:22.937 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.305e-06 2024-09-18T04:40:22.937 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.937 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.937 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:22.937 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583198+0000", 2024-09-18T04:40:22.938 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9369999999999997e-06 2024-09-18T04:40:22.938 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.938 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.938 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:22.938 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.583202+0000", 2024-09-18T04:40:22.938 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6480000000000001e-06 2024-09-18T04:40:22.938 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.938 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.938 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:22.938 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.595304+0000", 2024-09-18T04:40:22.938 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012102063999999999 2024-09-18T04:40:22.938 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.938 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.939 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:22.939 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.595390+0000", 2024-09-18T04:40:22.939 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5493000000000003e-05 2024-09-18T04:40:22.939 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.939 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.939 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:22.939 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.595509+0000", 2024-09-18T04:40:22.939 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011966199999999999 2024-09-18T04:40:22.939 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.939 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.939 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:22.939 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599358+0000", 2024-09-18T04:40:22.939 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0038486689999999999 2024-09-18T04:40:22.940 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.940 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.940 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:22.940 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606859+0000", 2024-09-18T04:40:22.940 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0075010789999999999 2024-09-18T04:40:22.940 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.940 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.940 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:22.940 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606887+0000", 2024-09-18T04:40:22.940 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7773e-05 2024-09-18T04:40:22.940 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.940 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.940 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:22.941 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606906+0000", 2024-09-18T04:40:22.941 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9417000000000001e-05 2024-09-18T04:40:22.941 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:22.941 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:22.941 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:22.941 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.941 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.941 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:106 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:40:22.941 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.584425+0000", 2024-09-18T04:40:22.941 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 86.248156276000003, 2024-09-18T04:40:22.941 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.022529293999999998, 2024-09-18T04:40:22.941 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:22.941 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:22.941 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:22.942 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:40:22.942 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:40:22.942 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 106 2024-09-18T04:40:22.942 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.942 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:22.942 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.942 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:22.942 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584425+0000", 2024-09-18T04:40:22.942 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:22.942 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.942 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.942 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:22.942 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584425+0000", 2024-09-18T04:40:22.943 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:22.943 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.943 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.943 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:22.943 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584424+0000", 2024-09-18T04:40:22.943 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:40:22.943 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.943 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.943 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:22.943 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584430+0000", 2024-09-18T04:40:22.943 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6389999999999997e-06 2024-09-18T04:40:22.943 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.943 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.943 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:22.944 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584456+0000", 2024-09-18T04:40:22.944 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6188999999999999e-05 2024-09-18T04:40:22.944 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.944 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.944 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:22.944 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584463+0000", 2024-09-18T04:40:22.944 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7340000000000002e-06 2024-09-18T04:40:22.944 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.944 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.944 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:22.944 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597537+0000", 2024-09-18T04:40:22.944 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013074161000000001 2024-09-18T04:40:22.944 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.944 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.945 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:22.945 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597598+0000", 2024-09-18T04:40:22.945 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0930000000000001e-05 2024-09-18T04:40:22.945 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.945 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.945 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:22.945 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.597922+0000", 2024-09-18T04:40:22.945 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032473000000000002 2024-09-18T04:40:22.945 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.945 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.945 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:22.945 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.601103+0000", 2024-09-18T04:40:22.945 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0031809849999999999 2024-09-18T04:40:22.946 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.946 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.946 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:22.946 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606912+0000", 2024-09-18T04:40:22.946 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058083550000000003 2024-09-18T04:40:22.946 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.946 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.946 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:22.946 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606935+0000", 2024-09-18T04:40:22.946 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2759999999999999e-05 2024-09-18T04:40:22.946 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.946 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.946 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:22.947 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606954+0000", 2024-09-18T04:40:22.947 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9602999999999998e-05 2024-09-18T04:40:22.947 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:22.947 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:22.947 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:22.947 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.947 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.947 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:109 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:40:22.947 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.585231+0000", 2024-09-18T04:40:22.947 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 86.247349655999997, 2024-09-18T04:40:22.947 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.021767496000000001, 2024-09-18T04:40:22.947 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:22.947 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:22.947 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:22.948 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:40:22.948 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:40:22.948 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 109 2024-09-18T04:40:22.948 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.948 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:22.948 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.948 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:22.948 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585231+0000", 2024-09-18T04:40:22.948 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:22.948 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.948 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.948 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:22.949 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585231+0000", 2024-09-18T04:40:22.949 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:22.949 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.949 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.949 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:22.949 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585230+0000", 2024-09-18T04:40:22.949 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:40:22.949 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.949 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.949 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:22.949 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585237+0000", 2024-09-18T04:40:22.949 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.618e-06 2024-09-18T04:40:22.949 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.949 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.950 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:22.950 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585256+0000", 2024-09-18T04:40:22.950 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9048000000000001e-05 2024-09-18T04:40:22.950 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.950 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.950 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:22.950 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.585274+0000", 2024-09-18T04:40:22.950 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8153000000000001e-05 2024-09-18T04:40:22.950 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.950 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.950 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:22.950 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.598284+0000", 2024-09-18T04:40:22.950 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013009682999999999 2024-09-18T04:40:22.950 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.950 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.951 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:22.951 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.598383+0000", 2024-09-18T04:40:22.951 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.8722999999999997e-05 2024-09-18T04:40:22.951 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.951 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.951 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:22.951 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.598829+0000", 2024-09-18T04:40:22.951 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00044623999999999998 2024-09-18T04:40:22.951 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.951 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.951 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:22.951 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.601149+0000", 2024-09-18T04:40:22.951 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0023204150000000002 2024-09-18T04:40:22.951 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.951 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.952 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:22.952 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606960+0000", 2024-09-18T04:40:22.952 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058106390000000003 2024-09-18T04:40:22.952 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.952 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.952 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:22.952 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606982+0000", 2024-09-18T04:40:22.952 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2252e-05 2024-09-18T04:40:22.952 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.952 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.952 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:22.952 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.606999+0000", 2024-09-18T04:40:22.952 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6871000000000002e-05 2024-09-18T04:40:22.952 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:22.952 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:22.953 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:22.953 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.953 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.953 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:110 2.0 2:08ed9131:::rbd_object_map.125fc09df6d1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:40:22.953 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.586640+0000", 2024-09-18T04:40:22.953 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 86.245941337000005, 2024-09-18T04:40:22.953 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020400911000000001, 2024-09-18T04:40:22.953 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:22.953 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:22.953 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:22.953 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:40:22.953 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:40:22.953 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 110 2024-09-18T04:40:22.953 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.954 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:22.954 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.954 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:22.954 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586640+0000", 2024-09-18T04:40:22.954 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:22.954 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.954 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.954 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:22.954 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586640+0000", 2024-09-18T04:40:22.954 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:22.954 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.954 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.954 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:22.955 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586639+0000", 2024-09-18T04:40:22.955 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:40:22.955 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.955 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.955 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:22.955 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586652+0000", 2024-09-18T04:40:22.955 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2482e-05 2024-09-18T04:40:22.955 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.955 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.955 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:22.955 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586660+0000", 2024-09-18T04:40:22.955 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1659999999999994e-06 2024-09-18T04:40:22.955 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.956 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.956 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:22.956 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.586666+0000", 2024-09-18T04:40:22.956 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6900000000000003e-06 2024-09-18T04:40:22.956 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.956 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.956 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:22.956 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599381+0000", 2024-09-18T04:40:22.956 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012714328 2024-09-18T04:40:22.956 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.956 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.956 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:22.956 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599436+0000", 2024-09-18T04:40:22.957 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4802999999999997e-05 2024-09-18T04:40:22.957 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.957 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.957 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:22.957 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.599629+0000", 2024-09-18T04:40:22.957 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000193921 2024-09-18T04:40:22.957 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.957 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.957 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:22.957 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.601174+0000", 2024-09-18T04:40:22.957 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001544857 2024-09-18T04:40:22.957 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.957 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.957 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:22.957 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.607004+0000", 2024-09-18T04:40:22.958 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058300820000000003 2024-09-18T04:40:22.958 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.958 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.958 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:22.958 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.607024+0000", 2024-09-18T04:40:22.958 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9888e-05 2024-09-18T04:40:22.958 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:22.958 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:22.958 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:22.958 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.607041+0000", 2024-09-18T04:40:22.958 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6450999999999999e-05 2024-09-18T04:40:22.958 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:22.958 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:22.958 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:22.959 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:22.959 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:22.959 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:40:22.960 INFO:tasks.workunit.client.0.smithi115.stderr: Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2024-09-18T04:40:22.960 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.0 to 1 (10000) 2024-09-18T04:40:22.961 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:40:22.961 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:40:22.955+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:40:22.963 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-18T04:40:22.970 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd -p rbd --namespace ns remove testrbdnbd17342 2024-09-18T04:40:23.061 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:40:23.055+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:40:23.077 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:40:23.077 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:40:23.077 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:40:23.077 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:40:23.088 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:40:23.150 INFO:tasks.workunit.client.0.smithi115.stderr: Removing image: 6% complete... Removing image: 12% complete... Removing image: 18% complete... Removing image: 25% complete... Removing image: 31% complete... Removing image: 37% complete... Removing image: 43% complete... Removing image: 50% complete... Removing image: 56% complete...2024-09-18T04:40:23.142+0000 7fd30c7ae700 0 -- 172.21.15.115:0/741569642 >> [v2:172.21.15.28:6816/3849236161,v1:172.21.15.28:6817/3849236161] conn(0x7fd2e4007e30 msgr2=0x7fd2e400a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-18T04:40:23.161 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:40:23.155+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:40:23.199 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:40:23.199 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:40:23.199 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:40:23.199 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:40:23.200 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:40:23.209 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:40:23.262 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:40:23.259+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:40:23.310 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:40:23.311 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:40:23.311 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:40:23.311 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:40:23.311 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.311 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.5431.0:1 3.3 3:d80af3cb:::rbd_id.testrbdnbd17342:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-18T04:40:23.311 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:13.542818+0000", 2024-09-18T04:40:23.311 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 69.765190945000001, 2024-09-18T04:40:23.311 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048493400000000002, 2024-09-18T04:40:23.311 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:23.311 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:40:23.311 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:23.311 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.5431", 2024-09-18T04:40:23.311 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3147136083", 2024-09-18T04:40:23.312 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 1 2024-09-18T04:40:23.312 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.312 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:23.312 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.312 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:23.312 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542818+0000", 2024-09-18T04:40:23.312 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:23.312 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.312 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.312 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:23.312 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542818+0000", 2024-09-18T04:40:23.312 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:23.312 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.313 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.313 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:23.313 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542816+0000", 2024-09-18T04:40:23.313 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:23.313 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.313 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.313 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:23.313 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542828+0000", 2024-09-18T04:40:23.313 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2699e-05 2024-09-18T04:40:23.313 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.313 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.313 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:23.313 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542852+0000", 2024-09-18T04:40:23.314 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3176e-05 2024-09-18T04:40:23.314 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.314 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.314 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:23.314 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542874+0000", 2024-09-18T04:40:23.314 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1967000000000002e-05 2024-09-18T04:40:23.314 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.314 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.314 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:23.314 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.542984+0000", 2024-09-18T04:40:23.314 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011026099999999999 2024-09-18T04:40:23.314 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.314 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.314 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:23.315 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543161+0000", 2024-09-18T04:40:23.315 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017720299999999999 2024-09-18T04:40:23.315 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.315 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.315 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:23.315 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543303+0000", 2024-09-18T04:40:23.315 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014174700000000001 2024-09-18T04:40:23.315 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:23.315 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:23.315 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:23.315 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.315 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.316 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.5431.0:2 3.3 3:d80af3cb:::rbd_id.testrbdnbd17342:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-18T04:40:23.316 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:13.543564+0000", 2024-09-18T04:40:23.316 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 69.764444459000003, 2024-09-18T04:40:23.316 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010399929999999999, 2024-09-18T04:40:23.316 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:23.316 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:40:23.316 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:23.316 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.5431", 2024-09-18T04:40:23.316 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3147136083", 2024-09-18T04:40:23.316 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 2 2024-09-18T04:40:23.316 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.316 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:23.316 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.317 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:23.317 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543564+0000", 2024-09-18T04:40:23.317 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:23.317 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.317 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.317 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:23.322 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543564+0000", 2024-09-18T04:40:23.322 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:23.322 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.322 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.322 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:23.322 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543563+0000", 2024-09-18T04:40:23.322 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:23.322 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.323 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.323 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:23.323 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543571+0000", 2024-09-18T04:40:23.323 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4270000000000008e-06 2024-09-18T04:40:23.323 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.323 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.323 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:23.323 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543591+0000", 2024-09-18T04:40:23.323 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.929e-05 2024-09-18T04:40:23.323 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.323 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.323 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:23.323 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543600+0000", 2024-09-18T04:40:23.324 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.1390000000000001e-06 2024-09-18T04:40:23.324 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.324 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.324 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:23.324 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543724+0000", 2024-09-18T04:40:23.324 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012458400000000001 2024-09-18T04:40:23.324 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.324 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.324 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:23.324 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.543968+0000", 2024-09-18T04:40:23.324 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00024323199999999999 2024-09-18T04:40:23.324 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.324 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.325 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:23.325 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:13.544604+0000", 2024-09-18T04:40:23.325 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00063675799999999996 2024-09-18T04:40:23.325 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:23.325 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:23.325 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:23.325 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.325 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.326 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "MOSDPGPush(3.3 97/96 [PushOp(3:c4c92e5a:::rbd_trash:head, version: 53'1, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:c4c92e5a:::rbd_trash:head@53'1, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-18T04:40:23.326 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:40:05.833692+0000", 2024-09-18T04:40:23.326 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 17.474316725000001, 2024-09-18T04:40:23.326 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00061637300000000003, 2024-09-18T04:40:23.326 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:23.326 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:40:23.326 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:23.326 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.326 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:23.326 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:05.833692+0000", 2024-09-18T04:40:23.326 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:23.326 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.326 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.327 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:23.327 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:05.833692+0000", 2024-09-18T04:40:23.327 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:23.327 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.327 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.327 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:23.327 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:05.833691+0000", 2024-09-18T04:40:23.327 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:40:23.328 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.328 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.328 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:23.328 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:05.833709+0000", 2024-09-18T04:40:23.328 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7838000000000001e-05 2024-09-18T04:40:23.328 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.328 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.329 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:23.329 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:05.833745+0000", 2024-09-18T04:40:23.329 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5651000000000001e-05 2024-09-18T04:40:23.329 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.329 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.329 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:23.329 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:05.833763+0000", 2024-09-18T04:40:23.329 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8428e-05 2024-09-18T04:40:23.329 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.329 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.330 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:23.330 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:05.833832+0000", 2024-09-18T04:40:23.330 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8919999999999997e-05 2024-09-18T04:40:23.330 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.330 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.330 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:23.330 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:05.833845+0000", 2024-09-18T04:40:23.330 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2680000000000001e-05 2024-09-18T04:40:23.331 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.331 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.331 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:23.331 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:05.834308+0000", 2024-09-18T04:40:23.331 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00046391200000000003 2024-09-18T04:40:23.331 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:23.332 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:23.332 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:23.332 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.332 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.332 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "MOSDPGPush(3.3 97/96 [PushOp(3:d80af3cb:::rbd_id.testrbdnbd17342:head, version: 69'2, data_included: [0~16], data_size: 16, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d80af3cb:::rbd_id.testrbdnbd17342:head@69'2, size: 16, copy_subset: [0~16], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:16, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-18T04:40:23.332 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:40:05.840323+0000", 2024-09-18T04:40:23.332 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 17.467685598999999, 2024-09-18T04:40:23.332 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00084282300000000003, 2024-09-18T04:40:23.332 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:23.332 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:40:23.332 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:23.332 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.333 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:23.333 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:05.840323+0000", 2024-09-18T04:40:23.333 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:23.333 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.333 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.333 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:23.333 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:05.840323+0000", 2024-09-18T04:40:23.333 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:23.333 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.333 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.334 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:23.334 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:05.840322+0000", 2024-09-18T04:40:23.334 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:40:23.334 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.334 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.334 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:23.334 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:05.840339+0000", 2024-09-18T04:40:23.334 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6767999999999999e-05 2024-09-18T04:40:23.334 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.334 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.334 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:23.335 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:05.840358+0000", 2024-09-18T04:40:23.335 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8946000000000001e-05 2024-09-18T04:40:23.335 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.335 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.335 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:23.335 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:05.840382+0000", 2024-09-18T04:40:23.335 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4408000000000001e-05 2024-09-18T04:40:23.335 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.335 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.335 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:23.335 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:05.840491+0000", 2024-09-18T04:40:23.335 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000108126 2024-09-18T04:40:23.335 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.336 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.336 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:23.336 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:05.840504+0000", 2024-09-18T04:40:23.336 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3519e-05 2024-09-18T04:40:23.336 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.336 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.336 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:23.336 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:05.841166+0000", 2024-09-18T04:40:23.336 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00066203699999999998 2024-09-18T04:40:23.336 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:23.336 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:23.336 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:23.336 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:23.336 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:23.337 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:40:23.338 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:40:23.364 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:40:23.359+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:40:23.379 INFO:tasks.workunit.client.0.smithi115.stderr: Removing image: 62% complete... Removing image: 68% complete... Removing image: 75% complete... Removing image: 81% complete... Removing image: 87% complete... Removing image: 93% complete...2024-09-18T04:40:23.374+0000 7fd30ad4b700 0 -- 172.21.15.115:0/741569642 >> [v2:172.21.15.82:6808/2746051132,v1:172.21.15.82:6809/2746051132] conn(0x7fd2f0008be0 msgr2=0x7fd2f000b070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-18T04:40:23.396 INFO:tasks.workunit.client.0.smithi115.stderr: Removing image: 100% complete...done. 2024-09-18T04:40:23.403 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd namespace remove rbd/ns 2024-09-18T04:40:23.464 INFO:tasks.ceph.osd.3.smithi082.stderr:2024-09-18T04:40:23.459+0000 7f6b67da0700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21116) UID: 0 2024-09-18T04:40:23.485 INFO:tasks.workunit.client.0.smithi115.stderr:+ rbd config global rm global rbd_default_pool 2024-09-18T04:40:23.489 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:40:23.489 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:40:23.489 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-09-18T04:40:23.489 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:40:23.499 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:40:23.538 INFO:tasks.workunit.client.0.smithi115.stderr:+ ceph osd pool delete new_default_pool17342 new_default_pool17342 --yes-i-really-really-mean-it 2024-09-18T04:40:23.566 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:40:23.559+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:40:23.604 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:40:23.604 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-09-18T04:40:23.604 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-09-18T04:40:23.604 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-09-18T04:40:23.604 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:40:23.614 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:40:23.667 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-09-18T04:40:23.663+0000 7fa9390ea700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21117) UID: 0 2024-09-18T04:40:23.768 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-18T04:40:23.763+0000 7fb3e45d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33792) UID: 0 2024-09-18T04:40:23.776 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-09-18T04:40:23.776 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-09-18T04:40:23.776 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-09-18T04:40:23.776 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-09-18T04:40:23.776 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.776 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-18T04:40:23.776 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:40:23.777 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 169.30040624, 2024-09-18T04:40:23.777 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0095781189999999995, 2024-09-18T04:40:23.777 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:23.777 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:23.777 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:23.777 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:40:23.777 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:40:23.777 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 1 2024-09-18T04:40:23.777 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.777 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:23.777 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.778 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:23.778 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:40:23.778 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:23.778 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.778 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.778 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:23.778 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469865+0000", 2024-09-18T04:40:23.778 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:23.778 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.778 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.778 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:23.778 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469862+0000", 2024-09-18T04:40:23.778 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:40:23.778 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.779 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.779 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:23.779 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469879+0000", 2024-09-18T04:40:23.779 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7431999999999998e-05 2024-09-18T04:40:23.779 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.779 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.779 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:23.779 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469896+0000", 2024-09-18T04:40:23.779 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6742999999999999e-05 2024-09-18T04:40:23.779 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.779 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.779 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:23.780 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.469915+0000", 2024-09-18T04:40:23.780 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8991000000000002e-05 2024-09-18T04:40:23.780 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.780 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.780 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:23.780 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470397+0000", 2024-09-18T04:40:23.780 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048167799999999998 2024-09-18T04:40:23.780 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.780 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.780 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-09-18T04:40:23.780 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.470412+0000", 2024-09-18T04:40:23.780 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5526e-05 2024-09-18T04:40:23.781 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.781 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.781 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:23.781 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473279+0000", 2024-09-18T04:40:23.781 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0028667469999999998 2024-09-18T04:40:23.781 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.781 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.781 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:23.781 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473441+0000", 2024-09-18T04:40:23.781 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000162082 2024-09-18T04:40:23.781 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.781 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.781 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:40:23.782 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.473573+0000", 2024-09-18T04:40:23.782 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000132032 2024-09-18T04:40:23.782 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.782 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.782 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:23.782 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.474826+0000", 2024-09-18T04:40:23.782 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001252717 2024-09-18T04:40:23.782 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.782 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.782 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:23.782 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479307+0000", 2024-09-18T04:40:23.782 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044813220000000003 2024-09-18T04:40:23.782 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.783 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.783 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:23.783 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479366+0000", 2024-09-18T04:40:23.783 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9024999999999998e-05 2024-09-18T04:40:23.783 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.783 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.783 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:23.783 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:34.479443+0000", 2024-09-18T04:40:23.783 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6339000000000002e-05 2024-09-18T04:40:23.783 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:23.783 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:23.783 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:23.783 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.784 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.784 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4214.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-18T04:40:23.784 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:40:23.784 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 167.28739782, 2024-09-18T04:40:23.784 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082699560000000002, 2024-09-18T04:40:23.784 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:23.784 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:23.784 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:23.784 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4214", 2024-09-18T04:40:23.784 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.28:0/1372876187", 2024-09-18T04:40:23.784 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 7 2024-09-18T04:40:23.784 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.784 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:23.785 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.785 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:23.785 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:40:23.785 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:23.785 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.785 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.785 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:23.785 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482873+0000", 2024-09-18T04:40:23.785 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:23.785 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.785 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.785 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:23.785 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482870+0000", 2024-09-18T04:40:23.785 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999967 2024-09-18T04:40:23.786 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.786 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.786 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:23.786 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482889+0000", 2024-09-18T04:40:23.786 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.965e-05 2024-09-18T04:40:23.786 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.786 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.786 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:23.786 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482938+0000", 2024-09-18T04:40:23.786 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8548999999999999e-05 2024-09-18T04:40:23.786 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.786 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.786 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:23.786 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.482963+0000", 2024-09-18T04:40:23.787 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5075999999999998e-05 2024-09-18T04:40:23.787 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.787 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.787 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:23.787 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483121+0000", 2024-09-18T04:40:23.787 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015819400000000001 2024-09-18T04:40:23.787 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.787 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.787 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op must wait for map", 2024-09-18T04:40:23.787 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.483138+0000", 2024-09-18T04:40:23.788 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7068999999999999e-05 2024-09-18T04:40:23.788 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.788 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.788 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:23.788 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488666+0000", 2024-09-18T04:40:23.788 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0055282049999999996 2024-09-18T04:40:23.788 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.788 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.788 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:23.788 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488783+0000", 2024-09-18T04:40:23.788 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000116737 2024-09-18T04:40:23.788 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.788 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.789 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:23.789 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.488991+0000", 2024-09-18T04:40:23.789 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020781400000000001 2024-09-18T04:40:23.789 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.789 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.789 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:23.789 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.490554+0000", 2024-09-18T04:40:23.789 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0015628370000000001 2024-09-18T04:40:23.789 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.789 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.789 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:23.789 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491019+0000", 2024-09-18T04:40:23.789 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000465608 2024-09-18T04:40:23.790 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.790 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.790 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:23.790 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491087+0000", 2024-09-18T04:40:23.790 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7988e-05 2024-09-18T04:40:23.790 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.790 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.790 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:23.790 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:37:36.491143+0000", 2024-09-18T04:40:23.790 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5538999999999998e-05 2024-09-18T04:40:23.790 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:23.790 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:23.790 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:23.790 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.791 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.791 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:34 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:23.791 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.261758+0000", 2024-09-18T04:40:23.791 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 101.508512669, 2024-09-18T04:40:23.791 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.042024191000000002, 2024-09-18T04:40:23.791 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:23.791 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:23.791 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:23.791 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:40:23.791 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:40:23.791 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 34 2024-09-18T04:40:23.791 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.791 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:23.792 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.792 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:23.792 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.261758+0000", 2024-09-18T04:40:23.792 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:23.792 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.792 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.792 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:23.792 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.261758+0000", 2024-09-18T04:40:23.792 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:23.792 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.792 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.792 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:23.792 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.261756+0000", 2024-09-18T04:40:23.793 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:23.793 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.793 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.793 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:23.793 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.270785+0000", 2024-09-18T04:40:23.793 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0090292340000000006 2024-09-18T04:40:23.793 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.793 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.793 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:23.793 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.270890+0000", 2024-09-18T04:40:23.793 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000104541 2024-09-18T04:40:23.793 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.793 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.793 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:23.794 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.270991+0000", 2024-09-18T04:40:23.794 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000100778 2024-09-18T04:40:23.794 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.794 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.794 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:23.794 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.271157+0000", 2024-09-18T04:40:23.794 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016653999999999999 2024-09-18T04:40:23.794 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.794 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.794 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:23.794 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.271591+0000", 2024-09-18T04:40:23.794 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00043409500000000001 2024-09-18T04:40:23.794 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.794 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.795 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:40:23.795 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.272017+0000", 2024-09-18T04:40:23.795 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000425769 2024-09-18T04:40:23.795 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.795 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.795 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:23.795 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.294246+0000", 2024-09-18T04:40:23.795 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.022229237999999998 2024-09-18T04:40:23.795 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.795 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.795 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:23.795 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303571+0000", 2024-09-18T04:40:23.795 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0093250239999999995 2024-09-18T04:40:23.796 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.796 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.796 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:23.796 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303705+0000", 2024-09-18T04:40:23.796 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000133831 2024-09-18T04:40:23.796 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.796 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.796 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:23.796 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.303782+0000", 2024-09-18T04:40:23.796 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7156000000000005e-05 2024-09-18T04:40:23.796 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:23.796 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:23.796 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:23.797 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.797 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.797 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:36 2.7 e51/32 2:fe1b02ad:::rbd_data.125fc09df6d1.0000000000000002:head v 51'1, mlcod=0'0)", 2024-09-18T04:40:23.797 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.273441+0000", 2024-09-18T04:40:23.797 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 101.49682968800001, 2024-09-18T04:40:23.797 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.032400768000000003, 2024-09-18T04:40:23.797 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:23.797 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:23.797 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:23.797 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.797 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:23.797 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273441+0000", 2024-09-18T04:40:23.797 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:23.797 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.798 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.798 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:23.798 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273441+0000", 2024-09-18T04:40:23.798 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:23.798 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.798 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.798 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:23.798 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.273440+0000", 2024-09-18T04:40:23.798 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:40:23.798 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.798 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.799 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:23.799 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278723+0000", 2024-09-18T04:40:23.799 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0052829909999999999 2024-09-18T04:40:23.799 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.799 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.799 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:23.799 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278752+0000", 2024-09-18T04:40:23.799 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9178000000000001e-05 2024-09-18T04:40:23.799 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.799 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.799 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:23.799 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278761+0000", 2024-09-18T04:40:23.799 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7900000000000005e-06 2024-09-18T04:40:23.800 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.800 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.800 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:23.800 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278864+0000", 2024-09-18T04:40:23.800 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000102699 2024-09-18T04:40:23.800 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.800 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.800 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:23.800 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.278900+0000", 2024-09-18T04:40:23.800 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6001999999999999e-05 2024-09-18T04:40:23.800 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.800 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.800 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:23.801 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.305793+0000", 2024-09-18T04:40:23.801 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.026893344 2024-09-18T04:40:23.801 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.801 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.801 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:23.801 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.305842+0000", 2024-09-18T04:40:23.801 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8720000000000001e-05 2024-09-18T04:40:23.801 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:23.801 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:23.801 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:23.801 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.801 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.801 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4703.0:39 2.7 e51/32 2:e9c909c5:::rbd_data.125fc09df6d1.0000000000000005:head v 51'2, mlcod=0'0)", 2024-09-18T04:40:23.801 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.283175+0000", 2024-09-18T04:40:23.802 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 101.487095999, 2024-09-18T04:40:23.802 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.022864102000000001, 2024-09-18T04:40:23.802 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:23.802 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:23.802 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:23.802 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.802 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:23.802 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283175+0000", 2024-09-18T04:40:23.802 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:23.802 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.802 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.802 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:23.802 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283175+0000", 2024-09-18T04:40:23.803 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:23.803 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.803 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.803 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:23.803 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.283173+0000", 2024-09-18T04:40:23.803 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:23.803 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.803 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.803 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:23.803 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.285951+0000", 2024-09-18T04:40:23.803 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0027784540000000001 2024-09-18T04:40:23.803 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.803 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.803 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:23.804 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.285970+0000", 2024-09-18T04:40:23.804 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8855000000000002e-05 2024-09-18T04:40:23.804 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.804 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.804 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:23.804 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.285984+0000", 2024-09-18T04:40:23.804 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4252e-05 2024-09-18T04:40:23.804 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.804 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.804 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:23.804 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.286041+0000", 2024-09-18T04:40:23.804 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6431999999999998e-05 2024-09-18T04:40:23.804 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.804 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.805 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:23.805 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.286086+0000", 2024-09-18T04:40:23.805 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5423000000000002e-05 2024-09-18T04:40:23.805 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.805 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.805 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:23.805 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.305998+0000", 2024-09-18T04:40:23.805 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019911697999999999 2024-09-18T04:40:23.805 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.806 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.806 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:23.806 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.306039+0000", 2024-09-18T04:40:23.806 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0973999999999998e-05 2024-09-18T04:40:23.806 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:23.806 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:23.806 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:23.806 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.806 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.806 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4703.0:50 2.3 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:23.806 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.312563+0000", 2024-09-18T04:40:23.806 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 101.45770797199999, 2024-09-18T04:40:23.807 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.032636001999999997, 2024-09-18T04:40:23.807 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:23.807 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:23.807 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:23.807 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4703", 2024-09-18T04:40:23.807 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/395557385", 2024-09-18T04:40:23.807 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 50 2024-09-18T04:40:23.807 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.807 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:23.807 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.807 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:23.807 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.312563+0000", 2024-09-18T04:40:23.807 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:23.808 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.808 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.808 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:23.808 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.312563+0000", 2024-09-18T04:40:23.808 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:23.808 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.808 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.808 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:23.808 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.312561+0000", 2024-09-18T04:40:23.808 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:23.808 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.808 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.808 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:23.809 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321709+0000", 2024-09-18T04:40:23.809 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0091486650000000003 2024-09-18T04:40:23.809 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.809 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.809 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:23.809 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321739+0000", 2024-09-18T04:40:23.809 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9666e-05 2024-09-18T04:40:23.809 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.809 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.809 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:23.809 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321751+0000", 2024-09-18T04:40:23.809 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2086000000000001e-05 2024-09-18T04:40:23.809 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.810 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.810 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:23.810 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321833+0000", 2024-09-18T04:40:23.810 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1842999999999996e-05 2024-09-18T04:40:23.810 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.810 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.810 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:23.810 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.321981+0000", 2024-09-18T04:40:23.810 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000147441 2024-09-18T04:40:23.810 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.810 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.810 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:40:23.810 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.322109+0000", 2024-09-18T04:40:23.810 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012851200000000001 2024-09-18T04:40:23.811 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.811 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.811 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:23.811 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.332829+0000", 2024-09-18T04:40:23.811 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010720225999999999 2024-09-18T04:40:23.811 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.811 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.811 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:23.811 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.345039+0000", 2024-09-18T04:40:23.811 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012209831000000001 2024-09-18T04:40:23.811 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.811 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.811 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:23.811 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.345120+0000", 2024-09-18T04:40:23.812 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1145999999999999e-05 2024-09-18T04:40:23.812 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.812 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.812 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:23.812 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.345199+0000", 2024-09-18T04:40:23.812 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8546000000000003e-05 2024-09-18T04:40:23.812 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:23.812 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:23.812 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:23.812 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.812 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.813 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:30 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:23.813 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.516122+0000", 2024-09-18T04:40:23.813 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 101.25414881, 2024-09-18T04:40:23.813 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0091865030000000004, 2024-09-18T04:40:23.813 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:23.813 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:23.813 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:23.813 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:40:23.813 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:23.813 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 30 2024-09-18T04:40:23.813 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.813 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:23.813 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.813 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:23.814 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516122+0000", 2024-09-18T04:40:23.814 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:23.814 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.814 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.814 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:23.814 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516122+0000", 2024-09-18T04:40:23.814 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:23.814 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.814 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.814 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:23.814 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516120+0000", 2024-09-18T04:40:23.814 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:23.814 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.814 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.815 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:23.815 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516129+0000", 2024-09-18T04:40:23.815 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.85e-06 2024-09-18T04:40:23.815 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.815 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.815 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:23.815 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516155+0000", 2024-09-18T04:40:23.815 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5882000000000001e-05 2024-09-18T04:40:23.815 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.815 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.815 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:23.816 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.516175+0000", 2024-09-18T04:40:23.816 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.022e-05 2024-09-18T04:40:23.816 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.816 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.816 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:23.816 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519780+0000", 2024-09-18T04:40:23.816 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003604187 2024-09-18T04:40:23.816 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.816 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.816 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:23.816 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519964+0000", 2024-09-18T04:40:23.816 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00018391999999999999 2024-09-18T04:40:23.816 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.816 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.817 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:40:23.817 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.520794+0000", 2024-09-18T04:40:23.817 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00083078100000000005 2024-09-18T04:40:23.817 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.817 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.817 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:23.817 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.524129+0000", 2024-09-18T04:40:23.817 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033342419999999998 2024-09-18T04:40:23.817 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.817 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.817 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:23.817 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525239+0000", 2024-09-18T04:40:23.817 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001110778 2024-09-18T04:40:23.817 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.818 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.818 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:23.818 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525277+0000", 2024-09-18T04:40:23.818 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7596999999999999e-05 2024-09-18T04:40:23.818 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.818 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.818 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:23.818 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525308+0000", 2024-09-18T04:40:23.818 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.1535999999999998e-05 2024-09-18T04:40:23.818 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:23.818 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:23.818 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:23.818 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.818 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.819 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:32 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:23.819 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.519217+0000", 2024-09-18T04:40:23.819 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 101.25105349, 2024-09-18T04:40:23.819 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082266370000000002, 2024-09-18T04:40:23.819 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:23.819 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:23.819 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:23.819 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:40:23.819 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:23.819 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 32 2024-09-18T04:40:23.819 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.819 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:23.819 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.819 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:23.820 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519217+0000", 2024-09-18T04:40:23.820 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:23.820 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.820 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.820 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:23.820 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519217+0000", 2024-09-18T04:40:23.820 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:23.820 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.820 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.820 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:23.820 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519215+0000", 2024-09-18T04:40:23.820 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:23.820 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.820 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.821 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:23.821 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519229+0000", 2024-09-18T04:40:23.821 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3329999999999999e-05 2024-09-18T04:40:23.821 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.821 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.821 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:23.821 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519268+0000", 2024-09-18T04:40:23.821 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9533000000000002e-05 2024-09-18T04:40:23.821 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.821 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.821 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:23.821 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.519285+0000", 2024-09-18T04:40:23.821 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6446000000000001e-05 2024-09-18T04:40:23.821 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.822 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.822 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:23.822 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.522358+0000", 2024-09-18T04:40:23.822 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030734310000000002 2024-09-18T04:40:23.822 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.822 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.822 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:23.822 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.522551+0000", 2024-09-18T04:40:23.822 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019317600000000001 2024-09-18T04:40:23.822 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.822 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.822 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:40:23.822 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.523262+0000", 2024-09-18T04:40:23.822 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00071069999999999998 2024-09-18T04:40:23.823 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.823 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.823 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:23.823 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.525318+0000", 2024-09-18T04:40:23.823 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0020561469999999999 2024-09-18T04:40:23.823 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.823 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.823 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:23.823 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.527376+0000", 2024-09-18T04:40:23.823 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0020584969999999998 2024-09-18T04:40:23.823 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.823 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.824 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:23.824 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.527416+0000", 2024-09-18T04:40:23.824 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9054000000000003e-05 2024-09-18T04:40:23.824 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.824 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.824 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:23.824 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.527444+0000", 2024-09-18T04:40:23.824 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8347000000000002e-05 2024-09-18T04:40:23.824 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:23.824 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:23.824 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:23.824 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.825 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.825 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4926.0:42 2.3 2:dc582dfb:ns::rbd_object_map.133e813ab5bc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-18T04:40:23.825 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.551160+0000", 2024-09-18T04:40:23.825 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 101.219111098, 2024-09-18T04:40:23.825 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010692757000000001, 2024-09-18T04:40:23.825 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:23.825 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:23.825 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:23.825 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4926", 2024-09-18T04:40:23.825 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3040645444", 2024-09-18T04:40:23.825 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 42 2024-09-18T04:40:23.826 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.826 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:23.826 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.826 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:23.826 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551160+0000", 2024-09-18T04:40:23.826 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:23.826 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.826 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.826 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:23.826 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551160+0000", 2024-09-18T04:40:23.826 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:23.827 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.827 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.827 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:23.827 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551159+0000", 2024-09-18T04:40:23.827 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:40:23.828 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.828 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.828 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:23.828 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551166+0000", 2024-09-18T04:40:23.828 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1029999999999996e-06 2024-09-18T04:40:23.829 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.829 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.829 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:23.829 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551177+0000", 2024-09-18T04:40:23.829 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1384e-05 2024-09-18T04:40:23.829 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.830 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.830 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:23.830 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.551185+0000", 2024-09-18T04:40:23.830 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1990000000000006e-06 2024-09-18T04:40:23.830 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.830 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.831 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:23.831 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552021+0000", 2024-09-18T04:40:23.831 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00083574400000000003 2024-09-18T04:40:23.831 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.831 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.831 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:23.831 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.552339+0000", 2024-09-18T04:40:23.831 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031771399999999997 2024-09-18T04:40:23.832 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.832 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.832 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:40:23.832 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.553140+0000", 2024-09-18T04:40:23.832 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00080088100000000003 2024-09-18T04:40:23.832 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.832 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.832 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:23.832 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.554585+0000", 2024-09-18T04:40:23.832 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001445801 2024-09-18T04:40:23.832 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.832 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.832 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:23.833 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.561711+0000", 2024-09-18T04:40:23.833 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071260109999999998 2024-09-18T04:40:23.833 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.833 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.833 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:23.834 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.561790+0000", 2024-09-18T04:40:23.834 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8967000000000005e-05 2024-09-18T04:40:23.834 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.834 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.834 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:23.834 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.561852+0000", 2024-09-18T04:40:23.834 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2045999999999995e-05 2024-09-18T04:40:23.834 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:23.834 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:23.834 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:23.834 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.834 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.835 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.4926.0:58 2.7 e51/32 2:fc7c6cd3:ns::rbd_data.133e813ab5bc.000000000000000e:head v 51'7, mlcod=51'5)", 2024-09-18T04:40:23.835 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:42.605333+0000", 2024-09-18T04:40:23.835 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 101.164937893, 2024-09-18T04:40:23.835 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020085902999999999, 2024-09-18T04:40:23.835 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:23.835 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:23.835 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:23.835 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.835 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:23.835 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605333+0000", 2024-09-18T04:40:23.835 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:23.835 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.835 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.835 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:23.836 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605333+0000", 2024-09-18T04:40:23.836 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:23.836 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.836 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.836 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:23.836 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.605332+0000", 2024-09-18T04:40:23.836 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:23.836 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.836 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.836 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:23.836 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608667+0000", 2024-09-18T04:40:23.836 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033350300000000001 2024-09-18T04:40:23.837 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.837 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.837 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:23.837 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608706+0000", 2024-09-18T04:40:23.837 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8951e-05 2024-09-18T04:40:23.837 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.837 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.837 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:23.837 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608725+0000", 2024-09-18T04:40:23.837 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9675999999999999e-05 2024-09-18T04:40:23.837 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.838 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.838 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:23.838 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608798+0000", 2024-09-18T04:40:23.838 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3135999999999999e-05 2024-09-18T04:40:23.838 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.838 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.838 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:23.838 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.608824+0000", 2024-09-18T04:40:23.838 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5284000000000001e-05 2024-09-18T04:40:23.838 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.838 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.838 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:23.838 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625370+0000", 2024-09-18T04:40:23.839 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016545975000000001 2024-09-18T04:40:23.839 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.839 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.839 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:23.839 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:42.625419+0000", 2024-09-18T04:40:23.839 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9076999999999998e-05 2024-09-18T04:40:23.839 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:23.839 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:23.839 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:23.839 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.839 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.839 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:611 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:40:23.839 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.069561+0000", 2024-09-18T04:40:23.840 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 92.700709338999999, 2024-09-18T04:40:23.840 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0072330299999999997, 2024-09-18T04:40:23.840 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:23.840 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:23.840 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:23.840 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:40:23.840 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:40:23.840 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 611 2024-09-18T04:40:23.840 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.840 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:23.840 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.840 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:23.840 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.069561+0000", 2024-09-18T04:40:23.840 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:23.841 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.841 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.841 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:23.841 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.069561+0000", 2024-09-18T04:40:23.841 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:23.841 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.841 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.841 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:23.841 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.069559+0000", 2024-09-18T04:40:23.841 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999976 2024-09-18T04:40:23.841 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.841 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.841 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:23.842 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070801+0000", 2024-09-18T04:40:23.842 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0012423359999999999 2024-09-18T04:40:23.842 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.842 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.842 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:23.842 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070828+0000", 2024-09-18T04:40:23.842 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6387e-05 2024-09-18T04:40:23.842 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.842 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.842 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:23.842 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070840+0000", 2024-09-18T04:40:23.842 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2689e-05 2024-09-18T04:40:23.842 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.843 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.843 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:23.843 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.070990+0000", 2024-09-18T04:40:23.843 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014946800000000001 2024-09-18T04:40:23.843 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.843 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.843 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:23.843 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.071128+0000", 2024-09-18T04:40:23.843 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013796699999999999 2024-09-18T04:40:23.843 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.843 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.843 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:40:23.844 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.071318+0000", 2024-09-18T04:40:23.844 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000190106 2024-09-18T04:40:23.844 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.844 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.844 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:23.844 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.075711+0000", 2024-09-18T04:40:23.844 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.004393002 2024-09-18T04:40:23.844 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.844 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.844 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:23.844 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.076665+0000", 2024-09-18T04:40:23.844 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00095390399999999999 2024-09-18T04:40:23.845 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.845 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.845 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:23.845 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.076742+0000", 2024-09-18T04:40:23.845 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7112000000000003e-05 2024-09-18T04:40:23.845 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.845 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.845 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:23.845 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.076794+0000", 2024-09-18T04:40:23.845 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2475999999999997e-05 2024-09-18T04:40:23.845 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:23.845 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:23.845 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:23.846 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.846 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.846 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4805.0:687 2.3 2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:40:23.846 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.487994+0000", 2024-09-18T04:40:23.846 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 92.282276355999997, 2024-09-18T04:40:23.846 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0074167690000000001, 2024-09-18T04:40:23.846 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:23.846 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:23.846 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:23.846 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4805", 2024-09-18T04:40:23.846 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2880437649", 2024-09-18T04:40:23.847 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 687 2024-09-18T04:40:23.847 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.847 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:23.847 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.847 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:23.847 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.487994+0000", 2024-09-18T04:40:23.847 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:23.847 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.847 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.847 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:23.847 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.487994+0000", 2024-09-18T04:40:23.847 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:23.847 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.847 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.848 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:23.848 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.487993+0000", 2024-09-18T04:40:23.848 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:40:23.848 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.848 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.848 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:23.848 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.488979+0000", 2024-09-18T04:40:23.848 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00098620700000000001 2024-09-18T04:40:23.848 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.848 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.848 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:23.848 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489002+0000", 2024-09-18T04:40:23.848 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2500000000000001e-05 2024-09-18T04:40:23.849 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.849 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.849 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:23.849 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489020+0000", 2024-09-18T04:40:23.849 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7691000000000001e-05 2024-09-18T04:40:23.849 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.849 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.849 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:23.849 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489091+0000", 2024-09-18T04:40:23.849 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1307e-05 2024-09-18T04:40:23.849 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.849 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.849 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:23.850 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489219+0000", 2024-09-18T04:40:23.850 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012766000000000001 2024-09-18T04:40:23.850 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.850 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.850 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:40:23.850 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.489387+0000", 2024-09-18T04:40:23.850 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016855400000000001 2024-09-18T04:40:23.850 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.850 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.850 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:23.850 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495243+0000", 2024-09-18T04:40:23.850 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058563189999999996 2024-09-18T04:40:23.850 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.851 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.851 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:23.851 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495266+0000", 2024-09-18T04:40:23.851 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2810999999999999e-05 2024-09-18T04:40:23.851 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.851 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.851 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:23.851 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495332+0000", 2024-09-18T04:40:23.851 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5469999999999995e-05 2024-09-18T04:40:23.851 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.851 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.851 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:23.852 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.495411+0000", 2024-09-18T04:40:23.852 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9395999999999997e-05 2024-09-18T04:40:23.852 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:23.852 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:23.852 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:23.852 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.852 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.852 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4972.0:13 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-18T04:40:23.852 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:51.948263+0000", 2024-09-18T04:40:23.852 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 91.822007748999994, 2024-09-18T04:40:23.852 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0091358970000000005, 2024-09-18T04:40:23.852 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:23.852 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-09-18T04:40:23.853 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:23.853 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4972", 2024-09-18T04:40:23.853 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2280251472", 2024-09-18T04:40:23.853 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 13 2024-09-18T04:40:23.853 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.853 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:23.853 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.853 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:23.853 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948263+0000", 2024-09-18T04:40:23.853 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:23.853 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.853 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.854 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:23.854 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948263+0000", 2024-09-18T04:40:23.854 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:23.854 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.854 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.854 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:23.854 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948261+0000", 2024-09-18T04:40:23.854 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:23.854 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.854 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.854 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:23.854 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948273+0000", 2024-09-18T04:40:23.854 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1576999999999999e-05 2024-09-18T04:40:23.854 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.855 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.855 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:23.855 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948305+0000", 2024-09-18T04:40:23.855 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2209000000000003e-05 2024-09-18T04:40:23.855 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.855 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.855 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:23.855 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948325+0000", 2024-09-18T04:40:23.855 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.022e-05 2024-09-18T04:40:23.855 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.855 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.855 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:23.855 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948455+0000", 2024-09-18T04:40:23.856 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000129624 2024-09-18T04:40:23.856 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.856 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.856 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:23.856 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.948598+0000", 2024-09-18T04:40:23.856 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014376699999999999 2024-09-18T04:40:23.856 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.856 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.856 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:23.856 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:51.957399+0000", 2024-09-18T04:40:23.856 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0088004299999999997 2024-09-18T04:40:23.856 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:23.856 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:23.856 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:23.857 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.857 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.857 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4874.0:101 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-18T04:40:23.857 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:38:56.576367+0000", 2024-09-18T04:40:23.857 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 87.193904040999996, 2024-09-18T04:40:23.857 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0078671020000000008, 2024-09-18T04:40:23.857 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:23.857 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:23.857 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:23.857 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4874", 2024-09-18T04:40:23.857 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/3875109589", 2024-09-18T04:40:23.857 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 101 2024-09-18T04:40:23.858 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.858 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:23.858 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.858 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:23.858 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576367+0000", 2024-09-18T04:40:23.858 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:23.858 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.858 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.858 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:23.858 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576367+0000", 2024-09-18T04:40:23.858 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:23.858 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.858 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.858 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:23.859 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576364+0000", 2024-09-18T04:40:23.859 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999971 2024-09-18T04:40:23.859 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.859 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.859 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:23.859 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576378+0000", 2024-09-18T04:40:23.859 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3879999999999999e-05 2024-09-18T04:40:23.859 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.859 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.859 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:23.859 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576410+0000", 2024-09-18T04:40:23.859 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2286000000000002e-05 2024-09-18T04:40:23.859 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.860 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.860 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:23.860 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576425+0000", 2024-09-18T04:40:23.860 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5290000000000001e-05 2024-09-18T04:40:23.860 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.860 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.860 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:23.860 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576551+0000", 2024-09-18T04:40:23.860 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012555199999999999 2024-09-18T04:40:23.860 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.860 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.860 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:23.860 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576719+0000", 2024-09-18T04:40:23.861 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016787400000000001 2024-09-18T04:40:23.861 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.861 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.861 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:40:23.861 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.576951+0000", 2024-09-18T04:40:23.861 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00023252400000000001 2024-09-18T04:40:23.861 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.861 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.861 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:23.861 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.581459+0000", 2024-09-18T04:40:23.861 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045074410000000001 2024-09-18T04:40:23.862 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.862 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.862 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:23.862 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584076+0000", 2024-09-18T04:40:23.862 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0026173860000000002 2024-09-18T04:40:23.862 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.862 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.862 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:23.862 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584160+0000", 2024-09-18T04:40:23.862 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4191999999999996e-05 2024-09-18T04:40:23.862 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.862 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.862 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:23.863 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:38:56.584234+0000", 2024-09-18T04:40:23.863 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.339e-05 2024-09-18T04:40:23.863 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:23.863 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:23.863 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:23.863 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.863 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.863 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:44 2.3 2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:40:23.863 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.550448+0000", 2024-09-18T04:40:23.863 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 83.219822851999993, 2024-09-18T04:40:23.863 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082117590000000008, 2024-09-18T04:40:23.863 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:23.863 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:23.863 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:23.863 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:40:23.864 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:40:23.864 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 44 2024-09-18T04:40:23.864 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.864 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:23.864 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.864 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:23.864 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.550448+0000", 2024-09-18T04:40:23.864 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:23.864 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.864 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.864 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:23.864 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.550448+0000", 2024-09-18T04:40:23.864 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:23.864 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.865 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.865 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:23.865 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.550445+0000", 2024-09-18T04:40:23.865 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999971 2024-09-18T04:40:23.865 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.865 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.865 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:23.865 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.551904+0000", 2024-09-18T04:40:23.865 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001458322 2024-09-18T04:40:23.865 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.865 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.865 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:23.865 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.551938+0000", 2024-09-18T04:40:23.865 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4359000000000001e-05 2024-09-18T04:40:23.865 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.866 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.866 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:23.866 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.551959+0000", 2024-09-18T04:40:23.866 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1403000000000002e-05 2024-09-18T04:40:23.866 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.866 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.866 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:23.866 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.552049+0000", 2024-09-18T04:40:23.866 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9970999999999996e-05 2024-09-18T04:40:23.866 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.866 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.866 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:23.866 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.552190+0000", 2024-09-18T04:40:23.866 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000140271 2024-09-18T04:40:23.867 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.867 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.867 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:40:23.867 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.552385+0000", 2024-09-18T04:40:23.867 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019498700000000001 2024-09-18T04:40:23.867 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.867 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.867 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:23.867 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.556308+0000", 2024-09-18T04:40:23.867 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00392333 2024-09-18T04:40:23.867 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.867 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.867 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:23.867 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.558479+0000", 2024-09-18T04:40:23.868 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0021715889999999998 2024-09-18T04:40:23.868 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.868 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.868 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:23.868 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.558616+0000", 2024-09-18T04:40:23.868 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013613399999999999 2024-09-18T04:40:23.868 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.868 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.868 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:23.868 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.558660+0000", 2024-09-18T04:40:23.868 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4053e-05 2024-09-18T04:40:23.868 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:23.868 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:23.869 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:23.869 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.869 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.869 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:100 2.4 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:40:23.869 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.856389+0000", 2024-09-18T04:40:23.869 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 82.913881610999994, 2024-09-18T04:40:23.869 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0081480800000000003, 2024-09-18T04:40:23.869 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:23.869 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:23.869 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:23.869 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:40:23.870 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:40:23.870 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 100 2024-09-18T04:40:23.870 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.870 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:23.870 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.870 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:23.870 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.856389+0000", 2024-09-18T04:40:23.870 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:23.870 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.870 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.870 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:23.870 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.856389+0000", 2024-09-18T04:40:23.870 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:23.871 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.871 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.871 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:23.871 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.856388+0000", 2024-09-18T04:40:23.871 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.999999 2024-09-18T04:40:23.871 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.871 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.871 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:23.871 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857295+0000", 2024-09-18T04:40:23.871 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00090725599999999995 2024-09-18T04:40:23.871 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.871 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.871 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:23.871 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857311+0000", 2024-09-18T04:40:23.872 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6092000000000001e-05 2024-09-18T04:40:23.872 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.872 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.872 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:23.872 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857319+0000", 2024-09-18T04:40:23.872 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9200000000000004e-06 2024-09-18T04:40:23.872 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.872 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.872 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:23.872 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857388+0000", 2024-09-18T04:40:23.872 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8591999999999997e-05 2024-09-18T04:40:23.872 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.872 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.873 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:23.873 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857535+0000", 2024-09-18T04:40:23.873 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014684299999999999 2024-09-18T04:40:23.873 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.873 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.873 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:40:23.873 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.857747+0000", 2024-09-18T04:40:23.873 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000212361 2024-09-18T04:40:23.873 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.873 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.873 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:23.873 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.860831+0000", 2024-09-18T04:40:23.873 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030841789999999998 2024-09-18T04:40:23.873 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.874 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.874 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:23.874 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.864361+0000", 2024-09-18T04:40:23.874 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0035298780000000002 2024-09-18T04:40:23.874 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.874 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.874 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:23.874 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.864456+0000", 2024-09-18T04:40:23.874 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4557999999999998e-05 2024-09-18T04:40:23.874 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.874 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.874 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:23.874 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.864537+0000", 2024-09-18T04:40:23.875 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1577999999999997e-05 2024-09-18T04:40:23.875 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:23.875 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:23.875 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:23.875 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.875 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.875 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4970.0:102 2.4 2:33d2ea4e:::rbd_data.125fc09df6d1.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-18T04:40:23.875 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:39:00.869773+0000", 2024-09-18T04:40:23.875 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 82.900497954000002, 2024-09-18T04:40:23.875 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0084430359999999992, 2024-09-18T04:40:23.875 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:23.875 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:23.875 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:23.875 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4970", 2024-09-18T04:40:23.875 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/554259839", 2024-09-18T04:40:23.876 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 102 2024-09-18T04:40:23.876 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.876 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:23.876 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.876 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:23.876 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.869773+0000", 2024-09-18T04:40:23.876 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:23.876 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.876 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.876 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:23.876 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.869773+0000", 2024-09-18T04:40:23.876 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:23.876 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.876 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.877 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:23.877 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.869771+0000", 2024-09-18T04:40:23.877 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:23.877 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.877 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.877 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:23.877 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870629+0000", 2024-09-18T04:40:23.877 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00085760999999999997 2024-09-18T04:40:23.877 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.877 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.877 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:23.877 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870647+0000", 2024-09-18T04:40:23.878 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8363999999999999e-05 2024-09-18T04:40:23.878 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.878 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.878 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:23.878 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870655+0000", 2024-09-18T04:40:23.878 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0099999999999995e-06 2024-09-18T04:40:23.878 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.878 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.878 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:23.878 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870803+0000", 2024-09-18T04:40:23.879 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014798 2024-09-18T04:40:23.879 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.879 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.879 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:23.879 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.870951+0000", 2024-09-18T04:40:23.879 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014715999999999999 2024-09-18T04:40:23.880 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.880 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.880 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:40:23.880 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.871145+0000", 2024-09-18T04:40:23.880 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000194253 2024-09-18T04:40:23.880 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.881 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.881 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:23.881 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.877033+0000", 2024-09-18T04:40:23.881 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0058883099999999999 2024-09-18T04:40:23.881 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.881 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.881 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:23.881 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.878078+0000", 2024-09-18T04:40:23.881 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001044747 2024-09-18T04:40:23.881 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.881 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.881 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:23.881 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.878160+0000", 2024-09-18T04:40:23.882 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1904999999999996e-05 2024-09-18T04:40:23.882 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.882 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.882 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:23.882 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:39:00.878216+0000", 2024-09-18T04:40:23.882 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6082000000000003e-05 2024-09-18T04:40:23.882 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:23.882 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:23.882 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:23.882 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.882 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.882 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.6061.0:189 2.3 2:d670d898:::journal.125fc09df6d1:head [watch watch cookie 140218200024176] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-18T04:40:23.882 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:40:14.880927+0000", 2024-09-18T04:40:23.882 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 8.8893433890000004, 2024-09-18T04:40:23.883 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0085105679999999996, 2024-09-18T04:40:23.883 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:23.883 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:23.883 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:23.883 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.6061", 2024-09-18T04:40:23.883 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/2011291291", 2024-09-18T04:40:23.883 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 189 2024-09-18T04:40:23.883 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.883 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:23.883 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.883 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:23.883 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:14.880927+0000", 2024-09-18T04:40:23.883 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:23.883 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.883 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.884 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:23.884 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:14.880927+0000", 2024-09-18T04:40:23.884 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:23.884 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.884 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.884 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:23.884 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:14.880925+0000", 2024-09-18T04:40:23.884 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:23.884 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.884 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.884 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:23.884 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:14.880937+0000", 2024-09-18T04:40:23.884 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1659e-05 2024-09-18T04:40:23.884 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.885 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.885 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:23.885 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:14.880960+0000", 2024-09-18T04:40:23.885 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3113e-05 2024-09-18T04:40:23.885 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.885 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.885 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:23.885 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:14.880972+0000", 2024-09-18T04:40:23.885 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.222e-05 2024-09-18T04:40:23.885 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.885 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.885 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:23.885 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:14.881121+0000", 2024-09-18T04:40:23.885 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014843699999999999 2024-09-18T04:40:23.886 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.886 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.886 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op must wait for map", 2024-09-18T04:40:23.886 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:14.881136+0000", 2024-09-18T04:40:23.886 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5373e-05 2024-09-18T04:40:23.886 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.886 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.886 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:23.886 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:14.887284+0000", 2024-09-18T04:40:23.886 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0061473919999999998 2024-09-18T04:40:23.886 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.886 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.886 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:23.886 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:14.887438+0000", 2024-09-18T04:40:23.886 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000154117 2024-09-18T04:40:23.887 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.887 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.887 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2", 2024-09-18T04:40:23.887 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:14.887918+0000", 2024-09-18T04:40:23.887 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048078599999999997 2024-09-18T04:40:23.887 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.887 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.887 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:23.887 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:14.889178+0000", 2024-09-18T04:40:23.887 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001259453 2024-09-18T04:40:23.887 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.887 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.887 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:23.887 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:14.889246+0000", 2024-09-18T04:40:23.887 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8125000000000003e-05 2024-09-18T04:40:23.888 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.888 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.888 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:23.888 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:14.889330+0000", 2024-09-18T04:40:23.888 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4011999999999995e-05 2024-09-18T04:40:23.888 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.888 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.888 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:23.888 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:14.889438+0000", 2024-09-18T04:40:23.888 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010788100000000001 2024-09-18T04:40:23.888 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:23.888 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:23.888 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:23.889 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.889 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.889 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.6193.0:85 2.4 2:304364d1:ns::rbd_data.133e813ab5bc.0000000000000004:head [delete] snapc 5=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e110)", 2024-09-18T04:40:23.889 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:40:23.147661+0000", 2024-09-18T04:40:23.889 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.62260955799999995, 2024-09-18T04:40:23.889 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0082984430000000008, 2024-09-18T04:40:23.889 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:23.889 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:23.889 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:23.889 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.6193", 2024-09-18T04:40:23.889 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/741569642", 2024-09-18T04:40:23.889 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 85 2024-09-18T04:40:23.889 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.890 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:23.890 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.890 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:23.890 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:23.147661+0000", 2024-09-18T04:40:23.890 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:23.890 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.890 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.890 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:23.890 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:23.147661+0000", 2024-09-18T04:40:23.890 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:23.890 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.890 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.890 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:23.891 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:23.147660+0000", 2024-09-18T04:40:23.891 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:23.891 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.891 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.891 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:23.891 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:23.147668+0000", 2024-09-18T04:40:23.891 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5059999999999995e-06 2024-09-18T04:40:23.891 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.891 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.891 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:23.891 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:23.147690+0000", 2024-09-18T04:40:23.891 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1549999999999999e-05 2024-09-18T04:40:23.891 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.891 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.892 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:23.892 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:23.147701+0000", 2024-09-18T04:40:23.892 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1090000000000001e-05 2024-09-18T04:40:23.892 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.892 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.892 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:23.892 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:23.147899+0000", 2024-09-18T04:40:23.892 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019835400000000001 2024-09-18T04:40:23.892 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.892 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.892 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:23.892 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:23.148106+0000", 2024-09-18T04:40:23.892 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020703799999999999 2024-09-18T04:40:23.893 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.893 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.893 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:40:23.893 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:23.148402+0000", 2024-09-18T04:40:23.893 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00029601200000000001 2024-09-18T04:40:23.893 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.893 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.893 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:23.893 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:23.153828+0000", 2024-09-18T04:40:23.893 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0054259709999999999 2024-09-18T04:40:23.893 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.893 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.893 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:23.894 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:23.155878+0000", 2024-09-18T04:40:23.894 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0020495159999999999 2024-09-18T04:40:23.894 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.894 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.894 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:23.894 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:23.155920+0000", 2024-09-18T04:40:23.894 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.2292000000000001e-05 2024-09-18T04:40:23.894 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.894 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.894 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:23.894 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:23.155960+0000", 2024-09-18T04:40:23.894 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9671e-05 2024-09-18T04:40:23.894 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:23.895 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:23.895 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:23.895 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.895 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.895 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.6193.0:90 2.4 2:20f0b0c9:ns::rbd_data.133e813ab5bc.0000000000000009:head [delete] snapc 5=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e110)", 2024-09-18T04:40:23.895 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-09-18T04:40:23.147861+0000", 2024-09-18T04:40:23.895 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.62240934199999998, 2024-09-18T04:40:23.895 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011491652999999999, 2024-09-18T04:40:23.895 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-09-18T04:40:23.895 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:23.895 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-09-18T04:40:23.895 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.6193", 2024-09-18T04:40:23.895 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.115:0/741569642", 2024-09-18T04:40:23.895 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 90 2024-09-18T04:40:23.896 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.896 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-09-18T04:40:23.896 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.896 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-09-18T04:40:23.896 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:23.147861+0000", 2024-09-18T04:40:23.896 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:23.896 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.896 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.896 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-09-18T04:40:23.896 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:23.147861+0000", 2024-09-18T04:40:23.896 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-09-18T04:40:23.897 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.897 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.897 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-09-18T04:40:23.897 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:23.147860+0000", 2024-09-18T04:40:23.897 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4294967295.9999981 2024-09-18T04:40:23.897 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.897 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.897 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-09-18T04:40:23.897 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:23.147869+0000", 2024-09-18T04:40:23.897 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8529999999999998e-06 2024-09-18T04:40:23.897 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.897 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.898 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-09-18T04:40:23.898 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:23.147895+0000", 2024-09-18T04:40:23.898 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6083000000000001e-05 2024-09-18T04:40:23.898 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.898 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.898 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-09-18T04:40:23.898 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:23.147911+0000", 2024-09-18T04:40:23.898 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5973000000000001e-05 2024-09-18T04:40:23.898 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.898 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.898 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-09-18T04:40:23.898 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:23.150871+0000", 2024-09-18T04:40:23.898 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0029608519999999999 2024-09-18T04:40:23.898 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.899 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.899 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-09-18T04:40:23.899 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:23.150994+0000", 2024-09-18T04:40:23.899 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000122094 2024-09-18T04:40:23.899 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.899 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.899 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0", 2024-09-18T04:40:23.899 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:23.151133+0000", 2024-09-18T04:40:23.899 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013963199999999999 2024-09-18T04:40:23.899 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.899 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.900 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-09-18T04:40:23.900 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:23.154337+0000", 2024-09-18T04:40:23.900 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0032042749999999999 2024-09-18T04:40:23.900 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.900 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.900 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:23.900 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:23.159294+0000", 2024-09-18T04:40:23.900 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0049562089999999996 2024-09-18T04:40:23.900 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.900 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.900 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-09-18T04:40:23.900 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:23.159329+0000", 2024-09-18T04:40:23.900 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5244999999999999e-05 2024-09-18T04:40:23.901 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-09-18T04:40:23.901 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-09-18T04:40:23.901 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-09-18T04:40:23.901 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-09-18T04:40:23.159353+0000", 2024-09-18T04:40:23.901 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4181000000000001e-05 2024-09-18T04:40:23.901 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:23.901 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:23.901 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:23.901 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-09-18T04:40:23.901 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-09-18T04:40:23.901 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-09-18T04:40:23.903 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:40:23.903 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:40:23.899+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:40:23.954 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.1 to 1 (10000) 2024-09-18T04:40:23.955 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-18T04:40:23.972 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-18T04:40:24.004 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-09-18T04:40:23.999+0000 7fba1379f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21120) UID: 0 2024-09-18T04:40:24.037 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:40:24.037 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:40:24.037 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:40:24.037 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:40:24.047 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:40:24.057 INFO:tasks.workunit.client.0.smithi115.stderr:pool 'new_default_pool17342' does not exist 2024-09-18T04:40:24.067 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-18T04:40:24.067 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-18T04:40:24.081 INFO:tasks.workunit:Stopping ['rbd/rbd-nbd.sh'] on client.0... 2024-09-18T04:40:24.082 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-18T04:40:24.105 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:40:24.099+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:40:24.152 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:40:24.152 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:40:24.152 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-09-18T04:40:24.152 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-09-18T04:40:24.152 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:40:24.162 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:40:24.206 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-18T04:40:24.203+0000 7ff1c1c94700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20957) UID: 0 2024-09-18T04:40:24.280 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:40:24.281 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-09-18T04:40:24.281 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-09-18T04:40:24.281 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-09-18T04:40:24.281 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.281 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:7, version: 79'64, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:c07a6e46:::rbd_data.125fc09df6d1.000000000000000a:7@79'64, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-18T04:40:24.281 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.769830+0000", 2024-09-18T04:40:24.281 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 37.506587703999998, 2024-09-18T04:40:24.281 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0040223419999999999, 2024-09-18T04:40:24.281 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:24.281 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:24.281 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:24.281 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.282 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:24.282 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.769830+0000", 2024-09-18T04:40:24.282 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:24.282 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.282 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.282 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:24.282 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.769830+0000", 2024-09-18T04:40:24.282 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:24.282 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.282 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.283 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:24.283 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.769829+0000", 2024-09-18T04:40:24.283 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:24.283 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.283 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.283 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:24.283 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.770812+0000", 2024-09-18T04:40:24.283 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00098347800000000009 2024-09-18T04:40:24.283 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.283 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.283 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:24.283 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.770834+0000", 2024-09-18T04:40:24.283 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1648999999999999e-05 2024-09-18T04:40:24.283 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.284 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.284 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:24.284 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.770847+0000", 2024-09-18T04:40:24.284 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3169999999999999e-05 2024-09-18T04:40:24.284 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.284 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.284 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:24.284 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.770969+0000", 2024-09-18T04:40:24.284 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000121721 2024-09-18T04:40:24.284 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.284 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.284 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:24.285 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.770989+0000", 2024-09-18T04:40:24.285 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9944000000000001e-05 2024-09-18T04:40:24.285 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.285 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.285 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:24.285 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.773852+0000", 2024-09-18T04:40:24.286 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028632319999999998 2024-09-18T04:40:24.286 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:24.286 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:24.286 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:24.286 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.286 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.286 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head, version: 81'75, data_included: [0~49152,1097728~999424], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:c9c4e5e0:::rbd_data.125fc09df6d1.0000000000000001:head@81'75, size: 2146304, copy_subset: [0~2146304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-18T04:40:24.286 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:39:46.782052+0000", 2024-09-18T04:40:24.286 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 37.494365219000002, 2024-09-18T04:40:24.287 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0037389630000000001, 2024-09-18T04:40:24.287 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:24.287 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:24.287 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:24.287 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.287 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:24.287 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.782052+0000", 2024-09-18T04:40:24.287 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:24.287 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.288 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.288 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:24.288 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.782052+0000", 2024-09-18T04:40:24.288 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:24.288 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.288 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.288 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:24.288 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.782051+0000", 2024-09-18T04:40:24.288 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:24.288 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.288 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.288 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:24.289 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.783658+0000", 2024-09-18T04:40:24.289 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0016072149999999999 2024-09-18T04:40:24.289 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.289 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.289 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:24.289 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.783698+0000", 2024-09-18T04:40:24.289 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9394000000000003e-05 2024-09-18T04:40:24.289 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.289 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.289 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:24.289 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.783733+0000", 2024-09-18T04:40:24.289 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5428e-05 2024-09-18T04:40:24.289 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.289 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.289 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:24.290 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.783854+0000", 2024-09-18T04:40:24.290 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012067599999999999 2024-09-18T04:40:24.290 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.290 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.290 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:24.290 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.783869+0000", 2024-09-18T04:40:24.290 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5554999999999999e-05 2024-09-18T04:40:24.290 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.290 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.290 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:24.290 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:39:46.785791+0000", 2024-09-18T04:40:24.290 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001921629 2024-09-18T04:40:24.290 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:24.290 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:24.290 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:24.290 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.291 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.291 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.5732.0:3 2.1 2:9370a598:::rbd_header.125fc09df6d1:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e102)", 2024-09-18T04:40:24.291 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:40:13.100500+0000", 2024-09-18T04:40:24.291 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 11.175917224000001, 2024-09-18T04:40:24.291 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0035679359999999999, 2024-09-18T04:40:24.291 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:24.292 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:24.292 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:24.292 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.5732", 2024-09-18T04:40:24.292 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/1418224569", 2024-09-18T04:40:24.292 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 3 2024-09-18T04:40:24.292 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.292 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:24.292 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.292 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:24.292 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.100500+0000", 2024-09-18T04:40:24.293 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:24.293 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.293 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.293 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:24.293 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.100500+0000", 2024-09-18T04:40:24.293 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:24.293 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.293 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.293 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:24.293 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.100497+0000", 2024-09-18T04:40:24.293 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999971 2024-09-18T04:40:24.293 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.293 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.293 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:24.294 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.100511+0000", 2024-09-18T04:40:24.294 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3753e-05 2024-09-18T04:40:24.294 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.294 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.294 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:24.294 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.100534+0000", 2024-09-18T04:40:24.294 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3108999999999998e-05 2024-09-18T04:40:24.294 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.294 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.294 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:24.294 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.100555+0000", 2024-09-18T04:40:24.294 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.084e-05 2024-09-18T04:40:24.294 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.294 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.295 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:24.295 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.100645+0000", 2024-09-18T04:40:24.295 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9859999999999997e-05 2024-09-18T04:40:24.295 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.295 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.295 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-09-18T04:40:24.295 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.100652+0000", 2024-09-18T04:40:24.295 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4109999999999999e-06 2024-09-18T04:40:24.295 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.295 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.295 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:24.295 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.103598+0000", 2024-09-18T04:40:24.296 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0029458940000000001 2024-09-18T04:40:24.296 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.296 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.296 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:24.296 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.103731+0000", 2024-09-18T04:40:24.296 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013292999999999999 2024-09-18T04:40:24.296 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.296 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.296 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:24.296 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.104068+0000", 2024-09-18T04:40:24.296 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00033679199999999998 2024-09-18T04:40:24.296 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:24.297 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:24.297 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:24.297 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.297 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.297 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.6267.0:3 2.1 2:9370a598:::rbd_header.125fc09df6d1:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e103)", 2024-09-18T04:40:24.297 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:40:13.945412+0000", 2024-09-18T04:40:24.297 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 10.331005466000001, 2024-09-18T04:40:24.297 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0035634999999999998, 2024-09-18T04:40:24.297 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:24.297 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:24.297 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:24.297 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.6267", 2024-09-18T04:40:24.297 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/3449046013", 2024-09-18T04:40:24.298 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 3 2024-09-18T04:40:24.298 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.298 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:24.298 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.298 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:24.298 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.945412+0000", 2024-09-18T04:40:24.298 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:24.298 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.298 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.298 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:24.298 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.945412+0000", 2024-09-18T04:40:24.299 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:24.299 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.299 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.299 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:24.299 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.945409+0000", 2024-09-18T04:40:24.299 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:40:24.299 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.299 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.299 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:24.299 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.945423+0000", 2024-09-18T04:40:24.299 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4014e-05 2024-09-18T04:40:24.299 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.299 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.299 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:24.300 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.945448+0000", 2024-09-18T04:40:24.300 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5065000000000001e-05 2024-09-18T04:40:24.300 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.300 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.300 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:24.300 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.945470+0000", 2024-09-18T04:40:24.300 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2127e-05 2024-09-18T04:40:24.300 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.300 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.300 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:24.300 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.945636+0000", 2024-09-18T04:40:24.300 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016542299999999999 2024-09-18T04:40:24.300 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.301 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.301 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-09-18T04:40:24.301 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.945654+0000", 2024-09-18T04:40:24.301 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8165999999999999e-05 2024-09-18T04:40:24.301 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.301 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.301 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:24.301 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.948489+0000", 2024-09-18T04:40:24.301 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002835317 2024-09-18T04:40:24.301 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.301 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.302 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:24.302 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.948622+0000", 2024-09-18T04:40:24.302 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013259400000000001 2024-09-18T04:40:24.302 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.302 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.302 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:24.302 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:13.948975+0000", 2024-09-18T04:40:24.302 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00035329099999999999 2024-09-18T04:40:24.302 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:24.302 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:24.303 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:24.303 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.303 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.303 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.6061.0:21 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [read 2560~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e103)", 2024-09-18T04:40:24.303 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:40:14.538767+0000", 2024-09-18T04:40:24.303 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 9.7376498120000008, 2024-09-18T04:40:24.303 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003584575, 2024-09-18T04:40:24.303 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:24.303 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:24.303 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:24.303 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.6061", 2024-09-18T04:40:24.303 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2011291291", 2024-09-18T04:40:24.303 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 21 2024-09-18T04:40:24.304 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.304 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:24.304 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.304 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:24.304 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.538767+0000", 2024-09-18T04:40:24.304 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:24.304 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.304 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.304 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:24.304 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.538767+0000", 2024-09-18T04:40:24.304 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:24.304 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.305 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.305 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:24.305 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.538767+0000", 2024-09-18T04:40:24.305 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:40:24.305 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.305 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.305 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:24.305 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.538769+0000", 2024-09-18T04:40:24.305 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8870000000000001e-06 2024-09-18T04:40:24.305 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.305 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.305 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:24.305 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.538773+0000", 2024-09-18T04:40:24.305 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8e-06 2024-09-18T04:40:24.306 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.306 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.306 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:24.306 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.538776+0000", 2024-09-18T04:40:24.306 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3629999999999998e-06 2024-09-18T04:40:24.306 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.306 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.306 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:24.306 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.541842+0000", 2024-09-18T04:40:24.306 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030658730000000002 2024-09-18T04:40:24.306 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.306 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.306 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:24.306 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.542072+0000", 2024-09-18T04:40:24.307 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00023009500000000001 2024-09-18T04:40:24.307 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.307 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.307 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:24.307 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.542352+0000", 2024-09-18T04:40:24.307 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00027983700000000001 2024-09-18T04:40:24.307 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:24.307 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:24.307 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:24.308 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.308 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.308 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.6061.0:22 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [read 3072~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e103)", 2024-09-18T04:40:24.308 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:40:14.538779+0000", 2024-09-18T04:40:24.308 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 9.7376381779999992, 2024-09-18T04:40:24.308 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0042496519999999996, 2024-09-18T04:40:24.308 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:24.308 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:24.308 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:24.308 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.6061", 2024-09-18T04:40:24.308 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2011291291", 2024-09-18T04:40:24.308 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 22 2024-09-18T04:40:24.308 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.309 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:24.309 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.309 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:24.309 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.538779+0000", 2024-09-18T04:40:24.309 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:24.309 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.309 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.309 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:24.309 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.538779+0000", 2024-09-18T04:40:24.309 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:24.309 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.310 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.310 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:24.310 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.538779+0000", 2024-09-18T04:40:24.310 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:40:24.310 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.310 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.310 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:24.310 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.538781+0000", 2024-09-18T04:40:24.310 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.835e-06 2024-09-18T04:40:24.310 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.310 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.310 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:24.311 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.538784+0000", 2024-09-18T04:40:24.311 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7340000000000002e-06 2024-09-18T04:40:24.311 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.311 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.311 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:24.311 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.538788+0000", 2024-09-18T04:40:24.311 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5559999999999999e-06 2024-09-18T04:40:24.311 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.311 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.311 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:24.311 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.542460+0000", 2024-09-18T04:40:24.311 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0036717410000000001 2024-09-18T04:40:24.311 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.312 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.312 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:24.312 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.542705+0000", 2024-09-18T04:40:24.312 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00024554699999999999 2024-09-18T04:40:24.312 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.312 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.312 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:24.312 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.543029+0000", 2024-09-18T04:40:24.312 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00032351999999999999 2024-09-18T04:40:24.312 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:24.312 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:24.312 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:24.312 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.313 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.313 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.6061.0:23 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [read 3584~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e103)", 2024-09-18T04:40:24.313 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:40:14.538791+0000", 2024-09-18T04:40:24.313 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 9.7376261399999997, 2024-09-18T04:40:24.313 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0047975300000000004, 2024-09-18T04:40:24.313 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:24.313 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:24.313 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:24.313 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.6061", 2024-09-18T04:40:24.313 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2011291291", 2024-09-18T04:40:24.313 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 23 2024-09-18T04:40:24.313 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.314 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:24.314 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.314 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:24.314 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.538791+0000", 2024-09-18T04:40:24.314 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:24.314 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.314 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.314 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:24.315 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.538791+0000", 2024-09-18T04:40:24.315 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:24.315 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.315 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.315 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:24.315 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.538791+0000", 2024-09-18T04:40:24.315 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:40:24.315 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.315 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.315 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:24.315 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.538793+0000", 2024-09-18T04:40:24.315 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6950000000000001e-06 2024-09-18T04:40:24.315 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.316 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.316 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:24.316 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.538796+0000", 2024-09-18T04:40:24.316 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7079999999999999e-06 2024-09-18T04:40:24.316 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.316 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.316 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:24.316 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.538799+0000", 2024-09-18T04:40:24.316 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1659999999999998e-06 2024-09-18T04:40:24.317 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.317 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.317 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:24.317 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.543113+0000", 2024-09-18T04:40:24.317 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0043137690000000003 2024-09-18T04:40:24.317 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.317 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.318 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:24.318 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.543364+0000", 2024-09-18T04:40:24.318 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00025053299999999999 2024-09-18T04:40:24.318 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.318 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.318 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:24.318 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.543589+0000", 2024-09-18T04:40:24.318 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000224942 2024-09-18T04:40:24.318 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:24.318 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:24.318 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:24.319 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.319 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.319 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.6061.0:30 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [read 7168~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e103)", 2024-09-18T04:40:24.319 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:40:14.544315+0000", 2024-09-18T04:40:24.319 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 9.7321019579999994, 2024-09-18T04:40:24.319 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0038419959999999999, 2024-09-18T04:40:24.319 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:24.319 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:24.319 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:24.319 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.6061", 2024-09-18T04:40:24.319 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2011291291", 2024-09-18T04:40:24.319 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 30 2024-09-18T04:40:24.319 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.320 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:24.320 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.320 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:24.320 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.544315+0000", 2024-09-18T04:40:24.320 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:24.320 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.320 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.320 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:24.320 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.544315+0000", 2024-09-18T04:40:24.320 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:24.320 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.321 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.321 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:24.321 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.544315+0000", 2024-09-18T04:40:24.321 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:40:24.321 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.321 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.321 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:24.321 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.544317+0000", 2024-09-18T04:40:24.321 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.782e-06 2024-09-18T04:40:24.321 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.321 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.321 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:24.322 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.544320+0000", 2024-09-18T04:40:24.322 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6669999999999998e-06 2024-09-18T04:40:24.322 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.322 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.322 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:24.322 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.544324+0000", 2024-09-18T04:40:24.322 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3909999999999998e-06 2024-09-18T04:40:24.322 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.322 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.322 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:24.322 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.547774+0000", 2024-09-18T04:40:24.322 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0034506329999999998 2024-09-18T04:40:24.322 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.322 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.323 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:24.323 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.547934+0000", 2024-09-18T04:40:24.323 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000159576 2024-09-18T04:40:24.323 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.323 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.323 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:24.323 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.548157+0000", 2024-09-18T04:40:24.323 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000223255 2024-09-18T04:40:24.323 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:24.323 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:24.323 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:24.323 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.323 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.323 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.6061.0:31 2.1 2:8bbdc09f:::rbd_data.125fc09df6d1.0000000000000000:head [read 7680~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e103)", 2024-09-18T04:40:24.324 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:40:14.544327+0000", 2024-09-18T04:40:24.324 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 9.7320907220000006, 2024-09-18T04:40:24.324 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0044879840000000004, 2024-09-18T04:40:24.324 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:24.324 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:24.324 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:24.324 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.6061", 2024-09-18T04:40:24.324 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2011291291", 2024-09-18T04:40:24.324 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 31 2024-09-18T04:40:24.324 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.324 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:24.324 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.324 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:24.325 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.544327+0000", 2024-09-18T04:40:24.325 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:24.325 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.325 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.325 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:24.325 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.544327+0000", 2024-09-18T04:40:24.325 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:24.325 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.325 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.325 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:24.325 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.544326+0000", 2024-09-18T04:40:24.325 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:40:24.326 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.326 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.326 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:24.326 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.544338+0000", 2024-09-18T04:40:24.326 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2163e-05 2024-09-18T04:40:24.326 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.327 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.327 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:24.327 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.544343+0000", 2024-09-18T04:40:24.327 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.605e-06 2024-09-18T04:40:24.328 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.328 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.328 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:24.328 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.544346+0000", 2024-09-18T04:40:24.328 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4369999999999999e-06 2024-09-18T04:40:24.328 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.328 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.329 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:24.329 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.548234+0000", 2024-09-18T04:40:24.329 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0038871259999999999 2024-09-18T04:40:24.329 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.329 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.330 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:24.330 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.548480+0000", 2024-09-18T04:40:24.330 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00024613800000000002 2024-09-18T04:40:24.330 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.330 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.330 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:24.330 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.548815+0000", 2024-09-18T04:40:24.330 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000334796 2024-09-18T04:40:24.331 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:24.331 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:24.331 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:24.331 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.331 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.331 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.6061.0:86 2.1 2:9370a598:::rbd_header.125fc09df6d1:head [call rbd.snapshot_get in=15b,call rbd.parent_overlap_get in=21b,call rbd.get_flags in=12b,call rbd.get_protection_status in=24b,call rbd.snapshot_get in=15b,call rbd.parent_overlap_get in=21b,call rbd.get_flags in=12b,call rbd.get_protection_status in=24b,call rbd.snapshot_get in=15b,call rbd.parent_overlap_get in=21b,call rbd.get_flags in=12b,call rbd.get_protection_status in=24b,call rbd.snapshot_get in=15b,call rbd.parent_overlap_get in=21b,call rbd.get_flags in=12b,call rbd.get_protection_status in=24b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e103)", 2024-09-18T04:40:24.331 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:40:14.750931+0000", 2024-09-18T04:40:24.331 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 9.5254862609999993, 2024-09-18T04:40:24.331 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0035763380000000001, 2024-09-18T04:40:24.331 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:24.332 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:24.332 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:24.332 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.6061", 2024-09-18T04:40:24.332 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2011291291", 2024-09-18T04:40:24.332 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 86 2024-09-18T04:40:24.332 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.332 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:24.332 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.332 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:24.332 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.750931+0000", 2024-09-18T04:40:24.332 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:24.332 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.332 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.333 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:24.333 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.750931+0000", 2024-09-18T04:40:24.333 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:24.333 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.333 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.333 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:24.333 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.750930+0000", 2024-09-18T04:40:24.333 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:24.333 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.333 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.333 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:24.334 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.750936+0000", 2024-09-18T04:40:24.334 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2659999999999998e-06 2024-09-18T04:40:24.334 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.334 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.334 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:24.334 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.750947+0000", 2024-09-18T04:40:24.334 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0305999999999999e-05 2024-09-18T04:40:24.334 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.334 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.334 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:24.334 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.750954+0000", 2024-09-18T04:40:24.334 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1740000000000003e-06 2024-09-18T04:40:24.334 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.335 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.335 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:24.335 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.752128+0000", 2024-09-18T04:40:24.335 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0011746809999999999 2024-09-18T04:40:24.335 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.335 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.335 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for rw locks", 2024-09-18T04:40:24.335 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.752369+0000", 2024-09-18T04:40:24.335 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00024055499999999999 2024-09-18T04:40:24.335 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.335 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.335 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:24.335 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.753313+0000", 2024-09-18T04:40:24.335 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00094424699999999995 2024-09-18T04:40:24.336 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.336 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.336 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:24.336 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.753424+0000", 2024-09-18T04:40:24.336 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000110958 2024-09-18T04:40:24.336 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.336 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.336 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:24.336 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.754507+0000", 2024-09-18T04:40:24.336 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0010830989999999999 2024-09-18T04:40:24.336 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:24.336 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:24.336 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:24.336 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.337 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.337 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.6291.0:35 2.1 2:9370a598:::rbd_header.125fc09df6d1:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e103)", 2024-09-18T04:40:24.337 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:40:14.751317+0000", 2024-09-18T04:40:24.337 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 9.5251004570000006, 2024-09-18T04:40:24.337 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003608694, 2024-09-18T04:40:24.337 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:24.337 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:24.337 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:24.341 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.6291", 2024-09-18T04:40:24.341 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/1667696675", 2024-09-18T04:40:24.341 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 35 2024-09-18T04:40:24.341 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.341 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:24.341 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.341 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:24.341 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.751317+0000", 2024-09-18T04:40:24.341 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:24.341 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.341 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.341 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:24.342 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.751317+0000", 2024-09-18T04:40:24.342 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:24.342 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.342 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.342 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:24.342 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.751316+0000", 2024-09-18T04:40:24.342 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:40:24.342 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.342 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.342 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:24.342 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.751320+0000", 2024-09-18T04:40:24.343 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7529999999999999e-06 2024-09-18T04:40:24.343 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.343 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.343 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:24.343 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.751327+0000", 2024-09-18T04:40:24.343 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1790000000000002e-06 2024-09-18T04:40:24.343 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.343 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.343 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:24.343 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.751333+0000", 2024-09-18T04:40:24.343 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4870000000000002e-06 2024-09-18T04:40:24.343 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.343 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.344 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:24.344 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.752785+0000", 2024-09-18T04:40:24.344 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001452577 2024-09-18T04:40:24.344 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.344 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.344 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for rw locks", 2024-09-18T04:40:24.344 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.752865+0000", 2024-09-18T04:40:24.344 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9290999999999997e-05 2024-09-18T04:40:24.344 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.344 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.344 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:24.344 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.754760+0000", 2024-09-18T04:40:24.344 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001895189 2024-09-18T04:40:24.345 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.345 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.345 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:24.345 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.754845+0000", 2024-09-18T04:40:24.345 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5567000000000003e-05 2024-09-18T04:40:24.345 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.345 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.345 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:24.345 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:14.754925+0000", 2024-09-18T04:40:24.345 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0186999999999996e-05 2024-09-18T04:40:24.345 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:24.345 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:24.345 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:24.346 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.346 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.346 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.5837.0:26 2.1 2:9370a598:::rbd_header.125fc09df6d1:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e106)", 2024-09-18T04:40:24.346 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:40:18.915605+0000", 2024-09-18T04:40:24.346 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 5.3608119930000004, 2024-09-18T04:40:24.346 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0079842929999999999, 2024-09-18T04:40:24.346 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:24.346 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:24.346 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:24.346 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.5837", 2024-09-18T04:40:24.346 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/2915691943", 2024-09-18T04:40:24.346 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 26 2024-09-18T04:40:24.347 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.347 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:24.347 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.347 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:24.347 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:18.915605+0000", 2024-09-18T04:40:24.347 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:24.347 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.347 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.347 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:24.347 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:18.915605+0000", 2024-09-18T04:40:24.347 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:24.347 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.347 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.348 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:24.348 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:18.915603+0000", 2024-09-18T04:40:24.348 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999976 2024-09-18T04:40:24.348 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.348 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.348 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:24.348 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:18.915617+0000", 2024-09-18T04:40:24.348 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4290999999999999e-05 2024-09-18T04:40:24.348 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.349 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.349 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:24.349 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:18.915648+0000", 2024-09-18T04:40:24.349 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0467000000000002e-05 2024-09-18T04:40:24.349 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.349 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.349 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:24.349 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:18.915661+0000", 2024-09-18T04:40:24.349 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3505e-05 2024-09-18T04:40:24.349 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.349 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.349 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:24.349 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:18.915735+0000", 2024-09-18T04:40:24.350 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4357000000000007e-05 2024-09-18T04:40:24.350 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.350 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.350 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-09-18T04:40:24.350 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:18.915743+0000", 2024-09-18T04:40:24.350 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.401e-06 2024-09-18T04:40:24.350 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.350 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.350 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:24.350 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:18.917972+0000", 2024-09-18T04:40:24.350 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0022289319999999999 2024-09-18T04:40:24.350 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.350 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.350 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:24.351 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:18.918038+0000", 2024-09-18T04:40:24.351 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5782000000000001e-05 2024-09-18T04:40:24.351 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.351 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.351 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:24.351 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:18.918405+0000", 2024-09-18T04:40:24.351 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00036765700000000002 2024-09-18T04:40:24.351 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.351 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.351 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:24.351 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:18.923452+0000", 2024-09-18T04:40:24.351 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0050466369999999997 2024-09-18T04:40:24.351 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.351 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.351 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:24.352 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:18.923476+0000", 2024-09-18T04:40:24.352 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3992e-05 2024-09-18T04:40:24.352 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.352 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.352 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:24.352 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:18.923552+0000", 2024-09-18T04:40:24.352 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6531000000000003e-05 2024-09-18T04:40:24.352 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.352 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.352 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:24.352 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:18.923590+0000", 2024-09-18T04:40:24.352 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7143e-05 2024-09-18T04:40:24.352 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:24.353 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:24.353 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:24.353 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.353 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.353 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop_reply(client.5837.0:26 2.1 e106/86 ondisk, result = 0)", 2024-09-18T04:40:24.353 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:40:18.919683+0000", 2024-09-18T04:40:24.353 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 5.3567337879999997, 2024-09-18T04:40:24.353 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0037719369999999999, 2024-09-18T04:40:24.353 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:24.353 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:24.353 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:24.354 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.354 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:24.354 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:18.919683+0000", 2024-09-18T04:40:24.354 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:24.354 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.354 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.354 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:24.355 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:18.919683+0000", 2024-09-18T04:40:24.355 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:24.355 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.355 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.355 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:24.355 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:18.919682+0000", 2024-09-18T04:40:24.356 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:24.356 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.356 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.356 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:24.356 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:18.919689+0000", 2024-09-18T04:40:24.356 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5910000000000001e-06 2024-09-18T04:40:24.356 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.356 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.356 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:24.356 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:18.919699+0000", 2024-09-18T04:40:24.356 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0193e-05 2024-09-18T04:40:24.356 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.357 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.357 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:24.357 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:18.919706+0000", 2024-09-18T04:40:24.357 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9670000000000004e-06 2024-09-18T04:40:24.357 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.357 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.358 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:24.358 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:18.923433+0000", 2024-09-18T04:40:24.358 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0037266589999999998 2024-09-18T04:40:24.358 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.358 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.358 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:24.358 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:18.923444+0000", 2024-09-18T04:40:24.359 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0712e-05 2024-09-18T04:40:24.359 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.359 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.359 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:24.359 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:18.923455+0000", 2024-09-18T04:40:24.360 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1799e-05 2024-09-18T04:40:24.360 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:24.360 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:24.360 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:24.360 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.360 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.361 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.6193.0:81 2.1 2:8b40f1c5:ns::rbd_data.133e813ab5bc.0000000000000000:head [delete] snapc 5=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e110)", 2024-09-18T04:40:24.361 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:40:23.350069+0000", 2024-09-18T04:40:24.361 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.92634866699999996, 2024-09-18T04:40:24.361 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013974098000000001, 2024-09-18T04:40:24.361 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:24.362 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:24.362 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:24.362 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.6193", 2024-09-18T04:40:24.362 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/741569642", 2024-09-18T04:40:24.362 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 81 2024-09-18T04:40:24.362 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.362 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:24.362 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.363 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:24.363 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.350069+0000", 2024-09-18T04:40:24.363 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:24.363 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.363 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.363 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:24.363 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.350069+0000", 2024-09-18T04:40:24.363 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:24.363 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.363 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.363 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:24.363 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.350065+0000", 2024-09-18T04:40:24.363 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999962 2024-09-18T04:40:24.363 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.364 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.364 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:24.364 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.350080+0000", 2024-09-18T04:40:24.364 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5384999999999999e-05 2024-09-18T04:40:24.364 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.364 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.364 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:24.364 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.350102+0000", 2024-09-18T04:40:24.364 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1705999999999999e-05 2024-09-18T04:40:24.364 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.364 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.364 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:24.365 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.350117+0000", 2024-09-18T04:40:24.365 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5125e-05 2024-09-18T04:40:24.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.365 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.365 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:24.365 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.350288+0000", 2024-09-18T04:40:24.365 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00017105899999999999 2024-09-18T04:40:24.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.365 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.365 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:24.365 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.350992+0000", 2024-09-18T04:40:24.365 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00070439000000000005 2024-09-18T04:40:24.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.365 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.366 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:24.366 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.351343+0000", 2024-09-18T04:40:24.366 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000350479 2024-09-18T04:40:24.366 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.366 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.366 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:24.366 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.363938+0000", 2024-09-18T04:40:24.366 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012595465 2024-09-18T04:40:24.366 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.366 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.366 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:24.366 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.363950+0000", 2024-09-18T04:40:24.366 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1219e-05 2024-09-18T04:40:24.366 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.367 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.367 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:24.367 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.363998+0000", 2024-09-18T04:40:24.367 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8819000000000001e-05 2024-09-18T04:40:24.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.367 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.367 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:24.367 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.364043+0000", 2024-09-18T04:40:24.367 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4205000000000001e-05 2024-09-18T04:40:24.367 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:24.367 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:24.367 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:24.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.368 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.368 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.6193.0:83 2.5 2:bd5b8dd8:ns::rbd_data.133e813ab5bc.0000000000000002:head [delete] snapc 5=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e110)", 2024-09-18T04:40:24.368 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:40:23.350140+0000", 2024-09-18T04:40:24.368 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.92627685400000004, 2024-09-18T04:40:24.368 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015170520999999999, 2024-09-18T04:40:24.368 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:24.368 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:24.368 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:24.368 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.6193", 2024-09-18T04:40:24.368 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/741569642", 2024-09-18T04:40:24.368 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 83 2024-09-18T04:40:24.368 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.368 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:24.369 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.369 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:24.369 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.350140+0000", 2024-09-18T04:40:24.369 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:24.369 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.369 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.369 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:24.369 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.350140+0000", 2024-09-18T04:40:24.369 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:24.369 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.369 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.369 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:24.370 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.350140+0000", 2024-09-18T04:40:24.370 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:24.370 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.370 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.370 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:24.370 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.350144+0000", 2024-09-18T04:40:24.370 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.882e-06 2024-09-18T04:40:24.370 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.370 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.370 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:24.370 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.350157+0000", 2024-09-18T04:40:24.370 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2748999999999999e-05 2024-09-18T04:40:24.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.371 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.371 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:24.371 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.350165+0000", 2024-09-18T04:40:24.371 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4939999999999997e-06 2024-09-18T04:40:24.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.371 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.371 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:24.371 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.350288+0000", 2024-09-18T04:40:24.371 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000123361 2024-09-18T04:40:24.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.371 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.371 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:24.372 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.350966+0000", 2024-09-18T04:40:24.372 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00067803999999999998 2024-09-18T04:40:24.372 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.372 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.372 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:40:24.372 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.351343+0000", 2024-09-18T04:40:24.372 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000376831 2024-09-18T04:40:24.372 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.372 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.372 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:24.372 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.363753+0000", 2024-09-18T04:40:24.372 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012410205000000001 2024-09-18T04:40:24.372 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.373 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.373 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:24.373 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.365165+0000", 2024-09-18T04:40:24.373 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0014122869999999999 2024-09-18T04:40:24.373 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.373 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.373 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:24.373 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.365243+0000", 2024-09-18T04:40:24.373 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7653999999999999e-05 2024-09-18T04:40:24.373 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.374 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.374 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:24.374 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.365311+0000", 2024-09-18T04:40:24.374 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7792000000000004e-05 2024-09-18T04:40:24.374 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:24.374 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:24.374 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:24.374 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.374 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.374 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.6193.0:84 2.5 2:bf39b8e4:ns::rbd_data.133e813ab5bc.0000000000000003:head [delete] snapc 5=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e110)", 2024-09-18T04:40:24.374 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:40:23.350182+0000", 2024-09-18T04:40:24.374 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.92623485500000002, 2024-09-18T04:40:24.374 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015311785, 2024-09-18T04:40:24.375 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:24.375 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:24.375 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:24.375 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.6193", 2024-09-18T04:40:24.375 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/741569642", 2024-09-18T04:40:24.375 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 84 2024-09-18T04:40:24.375 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.375 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:24.375 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.375 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:24.375 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.350182+0000", 2024-09-18T04:40:24.375 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:24.375 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.375 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.376 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:24.376 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.350182+0000", 2024-09-18T04:40:24.376 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:24.376 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.376 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.376 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:24.376 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.350182+0000", 2024-09-18T04:40:24.376 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999995 2024-09-18T04:40:24.376 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.376 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.376 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:24.376 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.350186+0000", 2024-09-18T04:40:24.376 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2250000000000002e-06 2024-09-18T04:40:24.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.377 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.377 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:24.377 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.350214+0000", 2024-09-18T04:40:24.377 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7948000000000001e-05 2024-09-18T04:40:24.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.377 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.377 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:24.377 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.350223+0000", 2024-09-18T04:40:24.377 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5799999999999998e-06 2024-09-18T04:40:24.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.377 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.378 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:24.378 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.361289+0000", 2024-09-18T04:40:24.378 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011065480000000001 2024-09-18T04:40:24.378 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.378 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.378 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:24.378 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.361414+0000", 2024-09-18T04:40:24.378 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012549800000000001 2024-09-18T04:40:24.378 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.378 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.378 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:40:24.378 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.361545+0000", 2024-09-18T04:40:24.378 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013036300000000001 2024-09-18T04:40:24.378 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.379 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.379 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:24.379 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.365147+0000", 2024-09-18T04:40:24.379 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003602418 2024-09-18T04:40:24.379 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.379 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.379 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:24.379 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.365332+0000", 2024-09-18T04:40:24.379 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018470199999999999 2024-09-18T04:40:24.379 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.379 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.379 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:24.379 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.365426+0000", 2024-09-18T04:40:24.379 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3955000000000004e-05 2024-09-18T04:40:24.379 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.380 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.380 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:24.380 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.365494+0000", 2024-09-18T04:40:24.380 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8273000000000002e-05 2024-09-18T04:40:24.380 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:24.380 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:24.380 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:24.381 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.381 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.381 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.6193.0:91 2.1 2:8c706374:ns::rbd_data.133e813ab5bc.000000000000000a:head [delete] snapc 5=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e110)", 2024-09-18T04:40:24.381 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:40:23.350288+0000", 2024-09-18T04:40:24.381 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.926129549, 2024-09-18T04:40:24.382 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015264348, 2024-09-18T04:40:24.382 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:24.382 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:24.382 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:24.382 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.6193", 2024-09-18T04:40:24.382 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/741569642", 2024-09-18T04:40:24.383 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 91 2024-09-18T04:40:24.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.383 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:24.383 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.383 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:24.383 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.350288+0000", 2024-09-18T04:40:24.383 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:24.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.383 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.383 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:24.384 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.350288+0000", 2024-09-18T04:40:24.384 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:24.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.384 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.384 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:24.384 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.350286+0000", 2024-09-18T04:40:24.384 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:24.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.384 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.384 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:24.384 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.350296+0000", 2024-09-18T04:40:24.384 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.7799999999999995e-06 2024-09-18T04:40:24.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.385 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.385 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:24.385 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.350322+0000", 2024-09-18T04:40:24.385 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5962e-05 2024-09-18T04:40:24.385 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.385 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.385 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:24.385 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.350336+0000", 2024-09-18T04:40:24.385 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.365e-05 2024-09-18T04:40:24.385 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.385 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.385 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:24.385 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.361257+0000", 2024-09-18T04:40:24.386 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010921022000000001 2024-09-18T04:40:24.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.386 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.386 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:24.386 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.361566+0000", 2024-09-18T04:40:24.386 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00030932900000000001 2024-09-18T04:40:24.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.386 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.386 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1", 2024-09-18T04:40:24.386 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.361653+0000", 2024-09-18T04:40:24.386 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.6790000000000001e-05 2024-09-18T04:40:24.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.386 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.386 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:24.387 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.364263+0000", 2024-09-18T04:40:24.387 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002610785 2024-09-18T04:40:24.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.387 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:24.387 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.365064+0000", 2024-09-18T04:40:24.387 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00080037399999999996 2024-09-18T04:40:24.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.387 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:24.388 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.365143+0000", 2024-09-18T04:40:24.388 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8949000000000006e-05 2024-09-18T04:40:24.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.388 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.388 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:24.388 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.365552+0000", 2024-09-18T04:40:24.388 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00040923700000000001 2024-09-18T04:40:24.388 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:24.388 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:24.388 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:24.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.388 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.389 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.6193.0:96 2.5 2:a06f51e4:ns::rbd_data.133e813ab5bc.000000000000000f:head [delete] snapc 5=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e110)", 2024-09-18T04:40:24.389 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:40:23.350351+0000", 2024-09-18T04:40:24.389 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.926066151, 2024-09-18T04:40:24.389 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016030117999999999, 2024-09-18T04:40:24.389 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:24.389 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T04:40:24.389 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-09-18T04:40:24.389 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.6193", 2024-09-18T04:40:24.389 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "172.21.15.115:0/741569642", 2024-09-18T04:40:24.389 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 96 2024-09-18T04:40:24.389 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.389 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:24.390 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.390 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:24.390 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.350351+0000", 2024-09-18T04:40:24.390 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:24.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.390 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.390 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:24.390 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.350351+0000", 2024-09-18T04:40:24.390 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:24.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.390 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.390 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:24.390 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.350350+0000", 2024-09-18T04:40:24.390 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:24.391 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.391 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.391 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:24.391 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.350355+0000", 2024-09-18T04:40:24.391 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0259999999999998e-06 2024-09-18T04:40:24.391 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.391 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.391 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:24.391 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.350365+0000", 2024-09-18T04:40:24.391 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.73e-06 2024-09-18T04:40:24.391 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.391 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.391 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:24.391 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.350373+0000", 2024-09-18T04:40:24.392 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6040000000000004e-06 2024-09-18T04:40:24.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.392 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.392 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:24.392 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.363799+0000", 2024-09-18T04:40:24.392 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013426400999999999 2024-09-18T04:40:24.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.392 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.392 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:24.392 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.363931+0000", 2024-09-18T04:40:24.392 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013205899999999999 2024-09-18T04:40:24.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.392 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.393 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0", 2024-09-18T04:40:24.393 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.364004+0000", 2024-09-18T04:40:24.393 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3085999999999998e-05 2024-09-18T04:40:24.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.393 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.393 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-09-18T04:40:24.393 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.365553+0000", 2024-09-18T04:40:24.394 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0015486950000000001 2024-09-18T04:40:24.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.394 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.394 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-09-18T04:40:24.394 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.366333+0000", 2024-09-18T04:40:24.394 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00077982599999999996 2024-09-18T04:40:24.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.394 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.394 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-09-18T04:40:24.394 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.366362+0000", 2024-09-18T04:40:24.394 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9190999999999999e-05 2024-09-18T04:40:24.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.395 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.395 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:24.395 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.366381+0000", 2024-09-18T04:40:24.395 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9392999999999999e-05 2024-09-18T04:40:24.395 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:24.395 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:24.395 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:24.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.396 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.396 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop_reply(client.6193.0:83 2.5 e110/86 ondisk, result = 0)", 2024-09-18T04:40:24.396 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:40:23.354386+0000", 2024-09-18T04:40:24.396 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.92203112099999995, 2024-09-18T04:40:24.396 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0093819109999999997, 2024-09-18T04:40:24.396 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:24.396 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:24.396 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:24.396 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.396 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:24.397 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.354386+0000", 2024-09-18T04:40:24.397 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:24.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.397 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.397 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:24.397 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.354386+0000", 2024-09-18T04:40:24.397 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:24.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.397 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.397 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:24.397 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.354385+0000", 2024-09-18T04:40:24.397 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.9999986 2024-09-18T04:40:24.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.397 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.398 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:24.398 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.354393+0000", 2024-09-18T04:40:24.398 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1820000000000006e-06 2024-09-18T04:40:24.398 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.398 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.398 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:24.398 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.354411+0000", 2024-09-18T04:40:24.398 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8275e-05 2024-09-18T04:40:24.398 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.398 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.398 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:24.398 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.354423+0000", 2024-09-18T04:40:24.398 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2464e-05 2024-09-18T04:40:24.398 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.399 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.399 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:24.399 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.363726+0000", 2024-09-18T04:40:24.399 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0093027100000000005 2024-09-18T04:40:24.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.399 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.399 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:24.399 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.363741+0000", 2024-09-18T04:40:24.399 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4725e-05 2024-09-18T04:40:24.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.399 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.399 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:24.399 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.363768+0000", 2024-09-18T04:40:24.399 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7157000000000001e-05 2024-09-18T04:40:24.400 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:24.400 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:24.400 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:24.400 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.400 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.400 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop_reply(client.6193.0:81 2.1 e110/86 ondisk, result = 0)", 2024-09-18T04:40:24.400 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-09-18T04:40:23.355776+0000", 2024-09-18T04:40:24.400 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.92064146899999999, 2024-09-18T04:40:24.400 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0081663489999999998, 2024-09-18T04:40:24.400 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-09-18T04:40:24.400 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-09-18T04:40:24.400 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-09-18T04:40:24.400 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.401 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-09-18T04:40:24.401 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.355776+0000", 2024-09-18T04:40:24.401 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:24.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.401 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.401 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-09-18T04:40:24.401 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.355776+0000", 2024-09-18T04:40:24.401 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-09-18T04:40:24.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.401 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.401 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-09-18T04:40:24.401 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.355775+0000", 2024-09-18T04:40:24.401 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4294967295.999999 2024-09-18T04:40:24.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.401 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.402 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-09-18T04:40:24.402 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.355784+0000", 2024-09-18T04:40:24.402 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1770000000000004e-06 2024-09-18T04:40:24.402 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.402 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.402 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-09-18T04:40:24.402 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.355843+0000", 2024-09-18T04:40:24.402 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9373000000000002e-05 2024-09-18T04:40:24.402 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.402 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.403 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-09-18T04:40:24.403 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.355854+0000", 2024-09-18T04:40:24.403 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0472000000000001e-05 2024-09-18T04:40:24.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.403 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.403 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-09-18T04:40:24.403 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.363926+0000", 2024-09-18T04:40:24.403 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0080721160000000007 2024-09-18T04:40:24.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.403 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.403 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-09-18T04:40:24.403 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.363933+0000", 2024-09-18T04:40:24.403 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5240000000000003e-06 2024-09-18T04:40:24.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-18T04:40:24.404 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-18T04:40:24.404 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-09-18T04:40:24.404 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-09-18T04:40:23.363942+0000", 2024-09-18T04:40:24.404 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7909999999999993e-06 2024-09-18T04:40:24.404 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:24.404 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:24.404 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:24.404 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-18T04:40:24.404 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-18T04:40:24.404 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:40:24.405 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T04:40:24.407 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-18T04:40:24.403+0000 7f6aae926700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20958) UID: 0 2024-09-18T04:40:24.408 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T04:40:24.408 DEBUG:teuthology.parallel:result is None 2024-09-18T04:40:24.408 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-18T04:40:24.424 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-18T04:40:24.424 DEBUG:teuthology.orchestra.run.smithi115:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-18T04:40:24.437 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-18T04:40:24.437 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-09-18T04:40:24.446 INFO:tasks.thrashosds:joining thrashosds 2024-09-18T04:40:24.563 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:40:24.564 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-09-18T04:40:24.564 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-09-18T04:40:24.564 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:40:24.574 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-18T04:40:24.601 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-09-18T04:40:24.639 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-09-18T04:40:24.674 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-09-18T04:40:24.703 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:24.704 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:24.704 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-09-18T04:40:24.714 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-18T04:40:24.729 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-09-18T04:40:24.767 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-09-18T04:40:24.770 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T04:40:24.770 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T04:40:24.770 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T04:40:24.770 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T04:40:24.770 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T04:40:24.770 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T04:40:24.826 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:24.826 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:24.835 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T04:40:24.971 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-09-18T04:40:24.971 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.3 to 1 (10000) 2024-09-18T04:40:24.983 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-18T04:40:24.994 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:24.994 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:25.004 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-18T04:40:25.104 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:25.104 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:25.114 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-18T04:40:25.210 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:25.210 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:25.220 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T04:40:25.336 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:25.336 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:25.346 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-18T04:40:25.452 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.5 to 1 (10000) 2024-09-18T04:40:25.456 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:25.456 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:25.466 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-18T04:40:25.467 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-18T04:40:25.566 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:25.566 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:25.576 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-18T04:40:25.680 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:25.680 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:25.690 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-18T04:40:25.803 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:25.803 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:25.813 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-18T04:40:25.922 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:25.922 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:25.931 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-18T04:40:26.034 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:26.034 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:26.045 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-18T04:40:26.147 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:26.147 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:26.157 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-18T04:40:26.266 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:26.266 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:26.276 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T04:40:26.394 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:26.394 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:26.405 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-18T04:40:26.457 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.2 to 1 (10000) 2024-09-18T04:40:26.472 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-18T04:40:26.546 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:26.546 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:26.557 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-18T04:40:26.664 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-09-18T04:40:26.665 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T04:40:26.983 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-18T04:40:27.462 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.4 to 1 (10000) 2024-09-18T04:40:27.467 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-09-18T04:40:27.476 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-18T04:40:27.476 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T04:40:27.481 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-18T04:40:27.777 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:40:27.779 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-18T04:40:27.795 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":201,"stamp":"2024-09-18T04:40:27.639871+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8601,"num_read_kb":183296,"num_write":1123,"num_write_kb":273318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":39960576,"num_keys_recovered":25,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1214,"ondisk_log_size":1214,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":5,"kb":566231040,"kb_used":208488,"kb_used_data":7868,"kb_used_omap":17,"kb_used_meta":192302,"kb_avail":566022552,"statfs":{"total":579820584960,"available":579607093248,"internally_reserved":0,"allocated":8056832,"data_stored":4299246,"data_compressed":66040,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":18101,"internal_metadata":196917579},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-129048798,"num_objects":-54,"num_object_clones":-14,"num_object_copies":-108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-54,"num_whiteouts":-10,"num_read":471,"num_read_kb":407,"num_write":105,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.696449"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":255,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-18T04:40:24.923063+0000","last_change":"2024-09-18T04:37:27.423069+0000","last_active":"2024-09-18T04:40:24.923063+0000","last_peered":"2024-09-18T04:40:24.923063+0000","last_clean":"2024-09-18T04:40:24.923063+0000","last_became_active":"2024-09-18T04:37:27.422350+0000","last_became_peered":"2024-09-18T04:37:27.422350+0000","last_unstale":"2024-09-18T04:40:24.923063+0000","last_undegraded":"2024-09-18T04:40:24.923063+0000","last_fullsized":"2024-09-18T04:40:24.923063+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:26.406292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:26.406292+0000","last_clean_scrub_stamp":"2024-09-18T04:37:26.406292+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:31:45.852149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"110'170","reported_seq":791,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-18T04:40:25.443507+0000","last_change":"2024-09-18T04:40:22.947240+0000","last_active":"2024-09-18T04:40:25.443507+0000","last_peered":"2024-09-18T04:40:25.443507+0000","last_clean":"2024-09-18T04:40:25.443507+0000","last_became_active":"2024-09-18T04:39:46.718066+0000","last_became_peered":"2024-09-18T04:39:46.718066+0000","last_unstale":"2024-09-18T04:40:25.443507+0000","last_undegraded":"2024-09-18T04:40:25.443507+0000","last_fullsized":"2024-09-18T04:40:25.443507+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T07:09:22.599338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011466040000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":748,"num_read_kb":17282,"num_write":161,"num_write_kb":25730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":9486336,"num_keys_recovered":9,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"5","length":"4"}]},{"pgid":"2.0","version":"110'255","reported_seq":1455,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-18T04:40:24.923095+0000","last_change":"2024-09-18T04:40:22.949604+0000","last_active":"2024-09-18T04:40:24.923095+0000","last_peered":"2024-09-18T04:40:24.923095+0000","last_clean":"2024-09-18T04:40:24.923095+0000","last_became_active":"2024-09-18T04:37:34.470841+0000","last_became_peered":"2024-09-18T04:37:34.470841+0000","last_unstale":"2024-09-18T04:40:24.923095+0000","last_undegraded":"2024-09-18T04:40:24.923095+0000","last_fullsized":"2024-09-18T04:40:24.923095+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":255,"log_dups_size":0,"ondisk_log_size":255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:16:34.528783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052682599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1817,"num_read_kb":40557,"num_write":337,"num_write_kb":45342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"4","length":"5"}]},{"pgid":"2.1","version":"110'317","reported_seq":2365,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-18T04:40:23.456324+0000","last_change":"2024-09-18T04:40:22.942796+0000","last_active":"2024-09-18T04:40:23.456324+0000","last_peered":"2024-09-18T04:40:23.456324+0000","last_clean":"2024-09-18T04:40:23.456324+0000","last_became_active":"2024-09-18T04:39:46.716998+0000","last_became_peered":"2024-09-18T04:39:46.716998+0000","last_unstale":"2024-09-18T04:40:23.456324+0000","last_undegraded":"2024-09-18T04:40:23.456324+0000","last_fullsized":"2024-09-18T04:40:23.456324+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":317,"log_dups_size":0,"ondisk_log_size":317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:37:53.810382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001074104,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4741,"num_read_kb":31642,"num_write":206,"num_write_kb":27778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":5308416,"num_keys_recovered":13,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"4","length":"5"}]},{"pgid":"2.2","version":"20'2","reported_seq":270,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-18T04:40:25.443410+0000","last_change":"2024-09-18T04:40:22.947782+0000","last_active":"2024-09-18T04:40:25.443410+0000","last_peered":"2024-09-18T04:40:25.443410+0000","last_clean":"2024-09-18T04:40:25.443410+0000","last_became_active":"2024-09-18T04:37:34.473088+0000","last_became_peered":"2024-09-18T04:37:34.473088+0000","last_unstale":"2024-09-18T04:40:25.443410+0000","last_undegraded":"2024-09-18T04:40:25.443410+0000","last_fullsized":"2024-09-18T04:40:25.443410+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:17:57.352397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001752392,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":139,"num_read_kb":139,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"5","length":"4"}]},{"pgid":"2.4","version":"110'65","reported_seq":146,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-18T04:40:25.443520+0000","last_change":"2024-09-18T04:40:22.947195+0000","last_active":"2024-09-18T04:40:25.443520+0000","last_peered":"2024-09-18T04:40:25.443520+0000","last_clean":"2024-09-18T04:40:25.443520+0000","last_became_active":"2024-09-18T04:38:59.462279+0000","last_became_peered":"2024-09-18T04:38:59.462279+0000","last_unstale":"2024-09-18T04:40:25.443520+0000","last_undegraded":"2024-09-18T04:40:25.443520+0000","last_fullsized":"2024-09-18T04:40:25.443520+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:32:58.894411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011357990000000001,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":101,"num_read_kb":16665,"num_write":51,"num_write_kb":36867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":12582912,"num_keys_recovered":3,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"5","length":"4"}]},{"pgid":"2.5","version":"110'89","reported_seq":444,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-18T04:40:23.467977+0000","last_change":"2024-09-18T04:40:22.942807+0000","last_active":"2024-09-18T04:40:23.467977+0000","last_peered":"2024-09-18T04:40:23.467977+0000","last_clean":"2024-09-18T04:40:23.467977+0000","last_became_active":"2024-09-18T04:39:46.717423+0000","last_became_peered":"2024-09-18T04:39:46.717423+0000","last_unstale":"2024-09-18T04:40:23.467977+0000","last_undegraded":"2024-09-18T04:40:23.467977+0000","last_fullsized":"2024-09-18T04:40:23.467977+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T09:52:29.555572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086846899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":248,"num_read_kb":31406,"num_write":66,"num_write_kb":49154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"4","length":"5"}]},{"pgid":"2.6","version":"110'100","reported_seq":355,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-18T04:40:23.448948+0000","last_change":"2024-09-18T04:40:23.022503+0000","last_active":"2024-09-18T04:40:23.448948+0000","last_peered":"2024-09-18T04:40:23.448948+0000","last_clean":"2024-09-18T04:40:23.448948+0000","last_became_active":"2024-09-18T04:37:34.472940+0000","last_became_peered":"2024-09-18T04:37:34.472940+0000","last_unstale":"2024-09-18T04:40:23.448948+0000","last_undegraded":"2024-09-18T04:40:23.448948+0000","last_fullsized":"2024-09-18T04:40:23.448948+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:18:29.266815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012195979999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":146,"num_read_kb":28719,"num_write":77,"num_write_kb":57349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4","length":"5"}]},{"pgid":"2.7","version":"110'99","reported_seq":489,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-18T04:40:24.923139+0000","last_change":"2024-09-18T04:40:22.952155+0000","last_active":"2024-09-18T04:40:24.923139+0000","last_peered":"2024-09-18T04:40:24.923139+0000","last_clean":"2024-09-18T04:40:24.923139+0000","last_became_active":"2024-09-18T04:38:05.753962+0000","last_became_peered":"2024-09-18T04:38:05.753962+0000","last_unstale":"2024-09-18T04:40:24.923139+0000","last_undegraded":"2024-09-18T04:40:24.923139+0000","last_fullsized":"2024-09-18T04:40:24.923139+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:53:52.224098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030414399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":545,"num_read_kb":16786,"num_write":64,"num_write_kb":28680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"4","length":"5"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8485,"num_read_kb":183196,"num_write":964,"num_write_kb":270902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":39960576,"num_keys_recovered":25,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3627,"internal_metadata":0},"log_size":1097,"ondisk_log_size":1097,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":66040,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":13,"seq":55834574888,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32412,"kb_used_data":1296,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339428,"statfs":{"total":96636764160,"available":96603574272,"internally_reserved":0,"allocated":1327104,"data_stored":521386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:39:28 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.85099999999999998,"15min":0.86499999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.0469999999999999,"5min":1.5249999999999999,"15min":1.5249999999999999},"last":1.012},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.79700000000000004,"15min":0.80800000000000005},"min":{"1min":0.53500000000000003,"5min":0.502,"15min":0.502},"max":{"1min":0.996,"5min":1.369,"15min":1.369},"last":0.95899999999999996}]},{"osd":1,"last update":"Wed Sep 18 04:39:28 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.83999999999999997,"15min":0.85199999999999998},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.091,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.79500000000000004,"15min":0.79900000000000004},"min":{"1min":0.59999999999999998,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":1.1000000000000001}]},{"osd":2,"last update":"Wed Sep 18 04:38:26 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.4930000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":1.0680000000000001},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":1.278}]},{"osd":3,"last update":"Wed Sep 18 04:39:28 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.75700000000000001,"15min":0.77000000000000002},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.0169999999999999,"5min":1.22,"15min":1.22},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.78900000000000003,"15min":0.79700000000000004},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":1.139}]},{"osd":4,"last update":"Wed Sep 18 04:39:28 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.77800000000000002,"15min":0.78500000000000003},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.998,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":1.544},{"interface":"front","average":{"1min":0.751,"5min":0.82499999999999996,"15min":0.83699999999999997},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.075,"5min":1.121,"15min":1.121},"last":1.5109999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574888,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32424,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339416,"statfs":{"total":96636764160,"available":96603561984,"internally_reserved":0,"allocated":1335296,"data_stored":526145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:39:31 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.76600000000000001,"15min":0.77200000000000002},"min":{"1min":0.52200000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.044,"5min":1.175,"15min":1.175},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.78500000000000003,"15min":0.79500000000000004},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.1060000000000001,"5min":1.258,"15min":1.258},"last":0.60499999999999998}]},{"osd":1,"last update":"Wed Sep 18 04:39:31 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.72199999999999998,"15min":0.72799999999999998},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.95899999999999996,"5min":1.002,"15min":1.002},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.875,"15min":0.88400000000000001},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":0.73899999999999999}]},{"osd":2,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.619,"5min":1.619,"15min":1.619},"last":0.82499999999999996}]},{"osd":3,"last update":"Wed Sep 18 04:39:31 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.81299999999999994,"15min":0.82299999999999995},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0129999999999999,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":2.302,"5min":2.302,"15min":2.302},"last":0.79200000000000004}]},{"osd":5,"last update":"Wed Sep 18 04:39:31 2024","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.88200000000000001,"15min":0.88700000000000001},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.365,"5min":1.365,"15min":1.365},"last":0.95099999999999996},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.89000000000000001,"15min":0.88800000000000001},"min":{"1min":0.42599999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":2.0979999999999999,"5min":2.0979999999999999,"15min":2.0979999999999999},"last":1.0900000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574889,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36572,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335268,"statfs":{"total":96636764160,"available":96599314432,"internally_reserved":0,"allocated":1388544,"data_stored":1111735,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 04:39:28 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.59999999999999998,"15min":0.60899999999999999},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.78000000000000003,"5min":1.113,"15min":1.113},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.63,"15min":0.64200000000000002},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.89900000000000002,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.61499999999999999}]},{"osd":2,"last update":"Wed Sep 18 04:38:28 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.8180000000000001,"5min":1.8180000000000001,"15min":1.8180000000000001},"last":0.72399999999999998}]},{"osd":3,"last update":"Wed Sep 18 04:39:28 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.81499999999999995,"15min":0.81799999999999995},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":0.99399999999999999,"5min":1.169,"15min":1.169},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.81899999999999995,"15min":0.82599999999999996},"min":{"1min":0.49199999999999999,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.97699999999999998,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.97299999999999998}]},{"osd":4,"last update":"Wed Sep 18 04:39:28 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.84599999999999997,"15min":0.85399999999999998},"min":{"1min":0.61699999999999999,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.0600000000000001,"5min":1.073,"15min":1.073},"last":1.042},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.89600000000000002,"15min":0.90800000000000003},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.94099999999999995}]},{"osd":5,"last update":"Wed Sep 18 04:39:28 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.91600000000000004,"15min":0.93100000000000005},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.1479999999999999,"5min":1.8919999999999999,"15min":1.8919999999999999},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.95399999999999996,"15min":0.97199999999999998},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.3140000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":1.1140000000000001}]}]},{"osd":3,"up_from":12,"seq":51539607592,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32476,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339364,"statfs":{"total":96636764160,"available":96603508736,"internally_reserved":0,"allocated":1388544,"data_stored":1111735,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.93200000000000005,"15min":0.95299999999999996},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.264,"5min":2.665,"15min":2.665},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.86799999999999999,"15min":0.878},"min":{"1min":0.55700000000000005,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.0649999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.66900000000000004}]},{"osd":1,"last update":"Wed Sep 18 04:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.86299999999999999,"15min":0.88300000000000001},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.121,"5min":2.4510000000000001,"15min":2.4510000000000001},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.77200000000000002,"15min":0.78600000000000003},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.046,"5min":1.077,"15min":1.077},"last":0.63}]},{"osd":2,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":2.577,"5min":2.577,"15min":2.577},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":1.155}]},{"osd":4,"last update":"Wed Sep 18 04:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.72499999999999998,"15min":0.72699999999999998},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.97399999999999998,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.751,"15min":0.749},"min":{"1min":0.49199999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.0209999999999999,"5min":1.044,"15min":1.044},"last":1.0760000000000001}]},{"osd":5,"last update":"Wed Sep 18 04:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.73599999999999999,"15min":0.73899999999999999},"min":{"1min":0.56399999999999995,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.069,"5min":1.173,"15min":1.173},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.877,"5min":0.86899999999999999,"15min":0.86799999999999999},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.087,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":1.0369999999999999}]}]},{"osd":2,"up_from":86,"seq":369367187465,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29904,"kb_used_data":1280,"kb_used_omap":9,"kb_used_meta":28598,"kb_avail":94341936,"statfs":{"total":96636764160,"available":96606142464,"internally_reserved":0,"allocated":1310720,"data_stored":516645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10134,"internal_metadata":29284458},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58299999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72599999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.101}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.876}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.071}]}]},{"osd":1,"up_from":12,"seq":51539607592,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44700,"kb_used_data":1276,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94327140,"statfs":{"total":96636764160,"available":96590991360,"internally_reserved":0,"allocated":1306624,"data_stored":511600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:39:27 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.60299999999999998,"15min":0.59499999999999997},"min":{"1min":0.433,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.625,"15min":0.627},"min":{"1min":0.42599999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.871,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.69699999999999995}]},{"osd":2,"last update":"Wed Sep 18 04:39:27 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.66200000000000003,"15min":0.66300000000000003},"min":{"1min":0.46700000000000003,"5min":0.436,"15min":0.436},"max":{"1min":1.05,"5min":1.327,"15min":1.327},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.66400000000000003,"15min":0.66700000000000004},"min":{"1min":0.43099999999999999,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.872,"5min":1.002,"15min":1.002},"last":0.85599999999999998}]},{"osd":3,"last update":"Wed Sep 18 04:39:27 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.79900000000000004,"15min":0.80100000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.22,"5min":1.22,"15min":1.22},"last":0.93000000000000005},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.86799999999999999,"15min":0.871},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.78400000000000003}]},{"osd":4,"last update":"Wed Sep 18 04:39:27 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"min":{"1min":0.63600000000000001,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.018,"5min":1.236,"15min":1.236},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.93000000000000005,"15min":0.93100000000000005},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.268,"5min":1.268,"15min":1.268},"last":0.89300000000000002}]},{"osd":5,"last update":"Wed Sep 18 04:39:27 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.90700000000000003,"15min":0.91100000000000003},"min":{"1min":0.71899999999999997,"5min":0.628,"15min":0.628},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.159,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.98099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3627,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T04:40:27.797 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T04:40:28.097 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:40:28.100 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-18T04:40:28.113 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":201,"stamp":"2024-09-18T04:40:27.639871+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8601,"num_read_kb":183296,"num_write":1123,"num_write_kb":273318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":39960576,"num_keys_recovered":25,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1214,"ondisk_log_size":1214,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":5,"kb":566231040,"kb_used":208488,"kb_used_data":7868,"kb_used_omap":17,"kb_used_meta":192302,"kb_avail":566022552,"statfs":{"total":579820584960,"available":579607093248,"internally_reserved":0,"allocated":8056832,"data_stored":4299246,"data_compressed":66040,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":18101,"internal_metadata":196917579},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-129048798,"num_objects":-54,"num_object_clones":-14,"num_object_copies":-108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-54,"num_whiteouts":-10,"num_read":471,"num_read_kb":407,"num_write":105,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.696449"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":255,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-18T04:40:24.923063+0000","last_change":"2024-09-18T04:37:27.423069+0000","last_active":"2024-09-18T04:40:24.923063+0000","last_peered":"2024-09-18T04:40:24.923063+0000","last_clean":"2024-09-18T04:40:24.923063+0000","last_became_active":"2024-09-18T04:37:27.422350+0000","last_became_peered":"2024-09-18T04:37:27.422350+0000","last_unstale":"2024-09-18T04:40:24.923063+0000","last_undegraded":"2024-09-18T04:40:24.923063+0000","last_fullsized":"2024-09-18T04:40:24.923063+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:26.406292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:26.406292+0000","last_clean_scrub_stamp":"2024-09-18T04:37:26.406292+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:31:45.852149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"110'170","reported_seq":791,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-18T04:40:25.443507+0000","last_change":"2024-09-18T04:40:22.947240+0000","last_active":"2024-09-18T04:40:25.443507+0000","last_peered":"2024-09-18T04:40:25.443507+0000","last_clean":"2024-09-18T04:40:25.443507+0000","last_became_active":"2024-09-18T04:39:46.718066+0000","last_became_peered":"2024-09-18T04:39:46.718066+0000","last_unstale":"2024-09-18T04:40:25.443507+0000","last_undegraded":"2024-09-18T04:40:25.443507+0000","last_fullsized":"2024-09-18T04:40:25.443507+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T07:09:22.599338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011466040000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":748,"num_read_kb":17282,"num_write":161,"num_write_kb":25730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":9486336,"num_keys_recovered":9,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"5","length":"4"}]},{"pgid":"2.0","version":"110'255","reported_seq":1455,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-18T04:40:24.923095+0000","last_change":"2024-09-18T04:40:22.949604+0000","last_active":"2024-09-18T04:40:24.923095+0000","last_peered":"2024-09-18T04:40:24.923095+0000","last_clean":"2024-09-18T04:40:24.923095+0000","last_became_active":"2024-09-18T04:37:34.470841+0000","last_became_peered":"2024-09-18T04:37:34.470841+0000","last_unstale":"2024-09-18T04:40:24.923095+0000","last_undegraded":"2024-09-18T04:40:24.923095+0000","last_fullsized":"2024-09-18T04:40:24.923095+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":255,"log_dups_size":0,"ondisk_log_size":255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:16:34.528783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052682599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1817,"num_read_kb":40557,"num_write":337,"num_write_kb":45342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"4","length":"5"}]},{"pgid":"2.1","version":"110'317","reported_seq":2365,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-18T04:40:23.456324+0000","last_change":"2024-09-18T04:40:22.942796+0000","last_active":"2024-09-18T04:40:23.456324+0000","last_peered":"2024-09-18T04:40:23.456324+0000","last_clean":"2024-09-18T04:40:23.456324+0000","last_became_active":"2024-09-18T04:39:46.716998+0000","last_became_peered":"2024-09-18T04:39:46.716998+0000","last_unstale":"2024-09-18T04:40:23.456324+0000","last_undegraded":"2024-09-18T04:40:23.456324+0000","last_fullsized":"2024-09-18T04:40:23.456324+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":317,"log_dups_size":0,"ondisk_log_size":317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:37:53.810382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001074104,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4741,"num_read_kb":31642,"num_write":206,"num_write_kb":27778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":5308416,"num_keys_recovered":13,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"4","length":"5"}]},{"pgid":"2.2","version":"20'2","reported_seq":270,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-18T04:40:25.443410+0000","last_change":"2024-09-18T04:40:22.947782+0000","last_active":"2024-09-18T04:40:25.443410+0000","last_peered":"2024-09-18T04:40:25.443410+0000","last_clean":"2024-09-18T04:40:25.443410+0000","last_became_active":"2024-09-18T04:37:34.473088+0000","last_became_peered":"2024-09-18T04:37:34.473088+0000","last_unstale":"2024-09-18T04:40:25.443410+0000","last_undegraded":"2024-09-18T04:40:25.443410+0000","last_fullsized":"2024-09-18T04:40:25.443410+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:17:57.352397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001752392,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":139,"num_read_kb":139,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"5","length":"4"}]},{"pgid":"2.4","version":"110'65","reported_seq":146,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-18T04:40:25.443520+0000","last_change":"2024-09-18T04:40:22.947195+0000","last_active":"2024-09-18T04:40:25.443520+0000","last_peered":"2024-09-18T04:40:25.443520+0000","last_clean":"2024-09-18T04:40:25.443520+0000","last_became_active":"2024-09-18T04:38:59.462279+0000","last_became_peered":"2024-09-18T04:38:59.462279+0000","last_unstale":"2024-09-18T04:40:25.443520+0000","last_undegraded":"2024-09-18T04:40:25.443520+0000","last_fullsized":"2024-09-18T04:40:25.443520+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:32:58.894411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011357990000000001,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":101,"num_read_kb":16665,"num_write":51,"num_write_kb":36867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":12582912,"num_keys_recovered":3,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"5","length":"4"}]},{"pgid":"2.5","version":"110'89","reported_seq":444,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-18T04:40:23.467977+0000","last_change":"2024-09-18T04:40:22.942807+0000","last_active":"2024-09-18T04:40:23.467977+0000","last_peered":"2024-09-18T04:40:23.467977+0000","last_clean":"2024-09-18T04:40:23.467977+0000","last_became_active":"2024-09-18T04:39:46.717423+0000","last_became_peered":"2024-09-18T04:39:46.717423+0000","last_unstale":"2024-09-18T04:40:23.467977+0000","last_undegraded":"2024-09-18T04:40:23.467977+0000","last_fullsized":"2024-09-18T04:40:23.467977+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T09:52:29.555572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086846899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":248,"num_read_kb":31406,"num_write":66,"num_write_kb":49154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"4","length":"5"}]},{"pgid":"2.6","version":"110'100","reported_seq":355,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-18T04:40:23.448948+0000","last_change":"2024-09-18T04:40:23.022503+0000","last_active":"2024-09-18T04:40:23.448948+0000","last_peered":"2024-09-18T04:40:23.448948+0000","last_clean":"2024-09-18T04:40:23.448948+0000","last_became_active":"2024-09-18T04:37:34.472940+0000","last_became_peered":"2024-09-18T04:37:34.472940+0000","last_unstale":"2024-09-18T04:40:23.448948+0000","last_undegraded":"2024-09-18T04:40:23.448948+0000","last_fullsized":"2024-09-18T04:40:23.448948+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:18:29.266815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012195979999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":146,"num_read_kb":28719,"num_write":77,"num_write_kb":57349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4","length":"5"}]},{"pgid":"2.7","version":"110'99","reported_seq":489,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-18T04:40:24.923139+0000","last_change":"2024-09-18T04:40:22.952155+0000","last_active":"2024-09-18T04:40:24.923139+0000","last_peered":"2024-09-18T04:40:24.923139+0000","last_clean":"2024-09-18T04:40:24.923139+0000","last_became_active":"2024-09-18T04:38:05.753962+0000","last_became_peered":"2024-09-18T04:38:05.753962+0000","last_unstale":"2024-09-18T04:40:24.923139+0000","last_undegraded":"2024-09-18T04:40:24.923139+0000","last_fullsized":"2024-09-18T04:40:24.923139+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:53:52.224098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030414399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":545,"num_read_kb":16786,"num_write":64,"num_write_kb":28680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"4","length":"5"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8485,"num_read_kb":183196,"num_write":964,"num_write_kb":270902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":39960576,"num_keys_recovered":25,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3627,"internal_metadata":0},"log_size":1097,"ondisk_log_size":1097,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":66040,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":13,"seq":55834574888,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32412,"kb_used_data":1296,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339428,"statfs":{"total":96636764160,"available":96603574272,"internally_reserved":0,"allocated":1327104,"data_stored":521386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:39:28 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.85099999999999998,"15min":0.86499999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.0469999999999999,"5min":1.5249999999999999,"15min":1.5249999999999999},"last":1.012},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.79700000000000004,"15min":0.80800000000000005},"min":{"1min":0.53500000000000003,"5min":0.502,"15min":0.502},"max":{"1min":0.996,"5min":1.369,"15min":1.369},"last":0.95899999999999996}]},{"osd":1,"last update":"Wed Sep 18 04:39:28 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.83999999999999997,"15min":0.85199999999999998},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.091,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.79500000000000004,"15min":0.79900000000000004},"min":{"1min":0.59999999999999998,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":1.1000000000000001}]},{"osd":2,"last update":"Wed Sep 18 04:38:26 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.4930000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":1.0680000000000001},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":1.278}]},{"osd":3,"last update":"Wed Sep 18 04:39:28 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.75700000000000001,"15min":0.77000000000000002},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.0169999999999999,"5min":1.22,"15min":1.22},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.78900000000000003,"15min":0.79700000000000004},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":1.139}]},{"osd":4,"last update":"Wed Sep 18 04:39:28 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.77800000000000002,"15min":0.78500000000000003},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.998,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":1.544},{"interface":"front","average":{"1min":0.751,"5min":0.82499999999999996,"15min":0.83699999999999997},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.075,"5min":1.121,"15min":1.121},"last":1.5109999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574888,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32424,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339416,"statfs":{"total":96636764160,"available":96603561984,"internally_reserved":0,"allocated":1335296,"data_stored":526145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:39:31 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.76600000000000001,"15min":0.77200000000000002},"min":{"1min":0.52200000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.044,"5min":1.175,"15min":1.175},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.78500000000000003,"15min":0.79500000000000004},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.1060000000000001,"5min":1.258,"15min":1.258},"last":0.60499999999999998}]},{"osd":1,"last update":"Wed Sep 18 04:39:31 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.72199999999999998,"15min":0.72799999999999998},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.95899999999999996,"5min":1.002,"15min":1.002},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.875,"15min":0.88400000000000001},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":0.73899999999999999}]},{"osd":2,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.619,"5min":1.619,"15min":1.619},"last":0.82499999999999996}]},{"osd":3,"last update":"Wed Sep 18 04:39:31 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.81299999999999994,"15min":0.82299999999999995},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0129999999999999,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":2.302,"5min":2.302,"15min":2.302},"last":0.79200000000000004}]},{"osd":5,"last update":"Wed Sep 18 04:39:31 2024","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.88200000000000001,"15min":0.88700000000000001},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.365,"5min":1.365,"15min":1.365},"last":0.95099999999999996},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.89000000000000001,"15min":0.88800000000000001},"min":{"1min":0.42599999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":2.0979999999999999,"5min":2.0979999999999999,"15min":2.0979999999999999},"last":1.0900000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574889,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36572,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335268,"statfs":{"total":96636764160,"available":96599314432,"internally_reserved":0,"allocated":1388544,"data_stored":1111735,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 04:39:28 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.59999999999999998,"15min":0.60899999999999999},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.78000000000000003,"5min":1.113,"15min":1.113},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.63,"15min":0.64200000000000002},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.89900000000000002,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.61499999999999999}]},{"osd":2,"last update":"Wed Sep 18 04:38:28 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.8180000000000001,"5min":1.8180000000000001,"15min":1.8180000000000001},"last":0.72399999999999998}]},{"osd":3,"last update":"Wed Sep 18 04:39:28 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.81499999999999995,"15min":0.81799999999999995},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":0.99399999999999999,"5min":1.169,"15min":1.169},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.81899999999999995,"15min":0.82599999999999996},"min":{"1min":0.49199999999999999,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.97699999999999998,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.97299999999999998}]},{"osd":4,"last update":"Wed Sep 18 04:39:28 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.84599999999999997,"15min":0.85399999999999998},"min":{"1min":0.61699999999999999,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.0600000000000001,"5min":1.073,"15min":1.073},"last":1.042},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.89600000000000002,"15min":0.90800000000000003},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.94099999999999995}]},{"osd":5,"last update":"Wed Sep 18 04:39:28 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.91600000000000004,"15min":0.93100000000000005},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.1479999999999999,"5min":1.8919999999999999,"15min":1.8919999999999999},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.95399999999999996,"15min":0.97199999999999998},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.3140000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":1.1140000000000001}]}]},{"osd":3,"up_from":12,"seq":51539607592,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32476,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339364,"statfs":{"total":96636764160,"available":96603508736,"internally_reserved":0,"allocated":1388544,"data_stored":1111735,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.93200000000000005,"15min":0.95299999999999996},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.264,"5min":2.665,"15min":2.665},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.86799999999999999,"15min":0.878},"min":{"1min":0.55700000000000005,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.0649999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.66900000000000004}]},{"osd":1,"last update":"Wed Sep 18 04:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.86299999999999999,"15min":0.88300000000000001},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.121,"5min":2.4510000000000001,"15min":2.4510000000000001},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.77200000000000002,"15min":0.78600000000000003},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.046,"5min":1.077,"15min":1.077},"last":0.63}]},{"osd":2,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":2.577,"5min":2.577,"15min":2.577},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":1.155}]},{"osd":4,"last update":"Wed Sep 18 04:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.72499999999999998,"15min":0.72699999999999998},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.97399999999999998,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.751,"15min":0.749},"min":{"1min":0.49199999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.0209999999999999,"5min":1.044,"15min":1.044},"last":1.0760000000000001}]},{"osd":5,"last update":"Wed Sep 18 04:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.73599999999999999,"15min":0.73899999999999999},"min":{"1min":0.56399999999999995,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.069,"5min":1.173,"15min":1.173},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.877,"5min":0.86899999999999999,"15min":0.86799999999999999},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.087,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":1.0369999999999999}]}]},{"osd":2,"up_from":86,"seq":369367187465,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29904,"kb_used_data":1280,"kb_used_omap":9,"kb_used_meta":28598,"kb_avail":94341936,"statfs":{"total":96636764160,"available":96606142464,"internally_reserved":0,"allocated":1310720,"data_stored":516645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10134,"internal_metadata":29284458},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58299999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72599999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.101}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.876}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.071}]}]},{"osd":1,"up_from":12,"seq":51539607592,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44700,"kb_used_data":1276,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94327140,"statfs":{"total":96636764160,"available":96590991360,"internally_reserved":0,"allocated":1306624,"data_stored":511600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:39:27 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.60299999999999998,"15min":0.59499999999999997},"min":{"1min":0.433,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.625,"15min":0.627},"min":{"1min":0.42599999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.871,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.69699999999999995}]},{"osd":2,"last update":"Wed Sep 18 04:39:27 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.66200000000000003,"15min":0.66300000000000003},"min":{"1min":0.46700000000000003,"5min":0.436,"15min":0.436},"max":{"1min":1.05,"5min":1.327,"15min":1.327},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.66400000000000003,"15min":0.66700000000000004},"min":{"1min":0.43099999999999999,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.872,"5min":1.002,"15min":1.002},"last":0.85599999999999998}]},{"osd":3,"last update":"Wed Sep 18 04:39:27 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.79900000000000004,"15min":0.80100000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.22,"5min":1.22,"15min":1.22},"last":0.93000000000000005},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.86799999999999999,"15min":0.871},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.78400000000000003}]},{"osd":4,"last update":"Wed Sep 18 04:39:27 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"min":{"1min":0.63600000000000001,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.018,"5min":1.236,"15min":1.236},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.93000000000000005,"15min":0.93100000000000005},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.268,"5min":1.268,"15min":1.268},"last":0.89300000000000002}]},{"osd":5,"last update":"Wed Sep 18 04:39:27 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.90700000000000003,"15min":0.91100000000000003},"min":{"1min":0.71899999999999997,"5min":0.628,"15min":0.628},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.159,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.98099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3627,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T04:40:28.114 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-18T04:40:28.114 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T04:40:28.114 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-09-18T04:40:28.114 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-18T04:40:28.114 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-18T04:40:28.115 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok config set filestore_inject_stall 3 2024-09-18T04:40:28.227 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:40:28.228 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-18T04:40:28.228 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:40:28.474 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-09-18T04:40:33.238 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-18T04:40:33.385 INFO:teuthology.orchestra.run.smithi028.stdout:mon.a: {} 2024-09-18T04:40:33.400 INFO:teuthology.orchestra.run.smithi028.stdout:mon.b: {} 2024-09-18T04:40:33.415 INFO:teuthology.orchestra.run.smithi028.stdout:mon.c: {} 2024-09-18T04:40:33.417 INFO:teuthology.orchestra.run.smithi028.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-18T04:40:33.417 INFO:teuthology.orchestra.run.smithi028.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-18T04:40:33.417 INFO:teuthology.orchestra.run.smithi028.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-18T04:40:33.433 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-18T04:40:34.467 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.0 to 1 (10000) 2024-09-18T04:40:34.483 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-09-18T04:40:35.472 INFO:teuthology.orchestra.run.smithi028.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-18T04:40:35.486 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-18T04:40:36.485 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.1 to 1 (10000) 2024-09-18T04:40:36.499 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-09-18T04:40:37.491 INFO:teuthology.orchestra.run.smithi028.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-18T04:40:37.506 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-18T04:40:38.502 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.3 to 1 (10000) 2024-09-18T04:40:38.515 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-09-18T04:40:39.513 INFO:teuthology.orchestra.run.smithi028.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-18T04:40:39.529 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-18T04:40:40.517 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.4 to 1 (10000) 2024-09-18T04:40:40.532 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-09-18T04:40:41.528 INFO:teuthology.orchestra.run.smithi028.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-18T04:40:41.544 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-18T04:40:42.537 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.5 to 1 (10000) 2024-09-18T04:40:42.549 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-09-18T04:40:43.546 INFO:teuthology.orchestra.run.smithi028.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-18T04:40:43.562 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-18T04:40:44.561 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.2 to 1 (10000) 2024-09-18T04:40:44.575 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-09-18T04:40:45.565 INFO:teuthology.orchestra.run.smithi028.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-18T04:40:45.581 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-18T04:40:45.581 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-18T04:40:45.581 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-18T04:40:45.581 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-18T04:40:45.581 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-18T04:40:45.581 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T04:40:45.887 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:40:45.887 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":129,"fsid":"be6146ba-a991-4f6e-b359-99cbedc9f9b1","created":"2024-09-18T04:37:15.676634+0000","modified":"2024-09-18T04:40:45.550211+0000","last_up_change":"2024-09-18T04:39:45.699693+0000","last_in_change":"2024-09-18T04:40:04.812503+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-18T04:37:25.654950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-18T04:37:32.672462+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":13,"snap_epoch":110,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2.2699999809265137,"score_stable":2.2699999809265137,"optimal_score":0.82999998331069946,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6177002d-72e6-4888-bcde-29b6a92d9dc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6801","nonce":1554407851},{"type":"v1","addr":"172.21.15.28:6803","nonce":1554407851}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6805","nonce":1554407851},{"type":"v1","addr":"172.21.15.28:6807","nonce":1554407851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6813","nonce":1554407851},{"type":"v1","addr":"172.21.15.28:6815","nonce":1554407851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":1554407851},{"type":"v1","addr":"172.21.15.28:6811","nonce":1554407851}]},"public_addr":"172.21.15.28:6803/1554407851","cluster_addr":"172.21.15.28:6807/1554407851","heartbeat_back_addr":"172.21.15.28:6815/1554407851","heartbeat_front_addr":"172.21.15.28:6811/1554407851","state":["exists","up"]},{"osd":1,"uuid":"4f62cb09-3457-418c-bb69-491d9b70914c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":511141401},{"type":"v1","addr":"172.21.15.28:6802","nonce":511141401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":511141401},{"type":"v1","addr":"172.21.15.28:6806","nonce":511141401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":511141401},{"type":"v1","addr":"172.21.15.28:6814","nonce":511141401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":511141401},{"type":"v1","addr":"172.21.15.28:6810","nonce":511141401}]},"public_addr":"172.21.15.28:6802/511141401","cluster_addr":"172.21.15.28:6806/511141401","heartbeat_back_addr":"172.21.15.28:6814/511141401","heartbeat_front_addr":"172.21.15.28:6810/511141401","state":["exists","up"]},{"osd":2,"uuid":"854fe031-3b87-4290-9c2f-3fb196fb806c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":12,"last_clean_end":76,"up_from":86,"up_thru":92,"down_at":77,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3849236161},{"type":"v1","addr":"172.21.15.28:6817","nonce":3849236161}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":3849236161},{"type":"v1","addr":"172.21.15.28:6819","nonce":3849236161}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":3849236161},{"type":"v1","addr":"172.21.15.28:6823","nonce":3849236161}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3849236161},{"type":"v1","addr":"172.21.15.28:6821","nonce":3849236161}]},"public_addr":"172.21.15.28:6817/3849236161","cluster_addr":"172.21.15.28:6819/3849236161","heartbeat_back_addr":"172.21.15.28:6823/3849236161","heartbeat_front_addr":"172.21.15.28:6821/3849236161","state":["exists","up"]},{"osd":3,"uuid":"c2625f98-780b-4f86-bf36-a48d2a227756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":2746051132},{"type":"v1","addr":"172.21.15.82:6809","nonce":2746051132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":2746051132},{"type":"v1","addr":"172.21.15.82:6811","nonce":2746051132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":2746051132},{"type":"v1","addr":"172.21.15.82:6815","nonce":2746051132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":2746051132},{"type":"v1","addr":"172.21.15.82:6813","nonce":2746051132}]},"public_addr":"172.21.15.82:6809/2746051132","cluster_addr":"172.21.15.82:6811/2746051132","heartbeat_back_addr":"172.21.15.82:6815/2746051132","heartbeat_front_addr":"172.21.15.82:6813/2746051132","state":["exists","up"]},{"osd":4,"uuid":"deb22d86-4081-484d-96eb-9cb9d96b60bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":2005712460},{"type":"v1","addr":"172.21.15.82:6801","nonce":2005712460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":2005712460},{"type":"v1","addr":"172.21.15.82:6803","nonce":2005712460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":2005712460},{"type":"v1","addr":"172.21.15.82:6807","nonce":2005712460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":2005712460},{"type":"v1","addr":"172.21.15.82:6805","nonce":2005712460}]},"public_addr":"172.21.15.82:6801/2005712460","cluster_addr":"172.21.15.82:6803/2005712460","heartbeat_back_addr":"172.21.15.82:6807/2005712460","heartbeat_front_addr":"172.21.15.82:6805/2005712460","state":["exists","up"]},{"osd":5,"uuid":"061d332c-f51b-403d-bb18-eba415c97e7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":1267644232},{"type":"v1","addr":"172.21.15.82:6817","nonce":1267644232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":1267644232},{"type":"v1","addr":"172.21.15.82:6819","nonce":1267644232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":1267644232},{"type":"v1","addr":"172.21.15.82:6823","nonce":1267644232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":1267644232},{"type":"v1","addr":"172.21.15.82:6821","nonce":1267644232}]},"public_addr":"172.21.15.82:6817/1267644232","cluster_addr":"172.21.15.82:6819/1267644232","heartbeat_back_addr":"172.21.15.82:6823/1267644232","heartbeat_front_addr":"172.21.15.82:6821/1267644232","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T04:37:22.192087+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T04:37:22.017833+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T04:37:22.755774+0000","dead_epoch":77},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T04:37:22.663785+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T04:37:22.531758+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T04:37:22.400774+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.5","osds":[2,0]},{"pgid":"2.7","osds":[3,5]}],"pg_upmap_items":[{"pgid":"2.1","mappings":[{"from":2,"to":1}]},{"pgid":"2.4","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T04:40:45.903 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-18T04:40:45.903 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T04:40:46.203 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:40:46.203 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":129,"fsid":"be6146ba-a991-4f6e-b359-99cbedc9f9b1","created":"2024-09-18T04:37:15.676634+0000","modified":"2024-09-18T04:40:45.550211+0000","last_up_change":"2024-09-18T04:39:45.699693+0000","last_in_change":"2024-09-18T04:40:04.812503+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-18T04:37:25.654950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-18T04:37:32.672462+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":13,"snap_epoch":110,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2.2699999809265137,"score_stable":2.2699999809265137,"optimal_score":0.82999998331069946,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6177002d-72e6-4888-bcde-29b6a92d9dc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6801","nonce":1554407851},{"type":"v1","addr":"172.21.15.28:6803","nonce":1554407851}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6805","nonce":1554407851},{"type":"v1","addr":"172.21.15.28:6807","nonce":1554407851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6813","nonce":1554407851},{"type":"v1","addr":"172.21.15.28:6815","nonce":1554407851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":1554407851},{"type":"v1","addr":"172.21.15.28:6811","nonce":1554407851}]},"public_addr":"172.21.15.28:6803/1554407851","cluster_addr":"172.21.15.28:6807/1554407851","heartbeat_back_addr":"172.21.15.28:6815/1554407851","heartbeat_front_addr":"172.21.15.28:6811/1554407851","state":["exists","up"]},{"osd":1,"uuid":"4f62cb09-3457-418c-bb69-491d9b70914c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":511141401},{"type":"v1","addr":"172.21.15.28:6802","nonce":511141401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":511141401},{"type":"v1","addr":"172.21.15.28:6806","nonce":511141401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":511141401},{"type":"v1","addr":"172.21.15.28:6814","nonce":511141401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":511141401},{"type":"v1","addr":"172.21.15.28:6810","nonce":511141401}]},"public_addr":"172.21.15.28:6802/511141401","cluster_addr":"172.21.15.28:6806/511141401","heartbeat_back_addr":"172.21.15.28:6814/511141401","heartbeat_front_addr":"172.21.15.28:6810/511141401","state":["exists","up"]},{"osd":2,"uuid":"854fe031-3b87-4290-9c2f-3fb196fb806c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":12,"last_clean_end":76,"up_from":86,"up_thru":92,"down_at":77,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3849236161},{"type":"v1","addr":"172.21.15.28:6817","nonce":3849236161}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":3849236161},{"type":"v1","addr":"172.21.15.28:6819","nonce":3849236161}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":3849236161},{"type":"v1","addr":"172.21.15.28:6823","nonce":3849236161}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3849236161},{"type":"v1","addr":"172.21.15.28:6821","nonce":3849236161}]},"public_addr":"172.21.15.28:6817/3849236161","cluster_addr":"172.21.15.28:6819/3849236161","heartbeat_back_addr":"172.21.15.28:6823/3849236161","heartbeat_front_addr":"172.21.15.28:6821/3849236161","state":["exists","up"]},{"osd":3,"uuid":"c2625f98-780b-4f86-bf36-a48d2a227756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":2746051132},{"type":"v1","addr":"172.21.15.82:6809","nonce":2746051132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":2746051132},{"type":"v1","addr":"172.21.15.82:6811","nonce":2746051132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":2746051132},{"type":"v1","addr":"172.21.15.82:6815","nonce":2746051132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":2746051132},{"type":"v1","addr":"172.21.15.82:6813","nonce":2746051132}]},"public_addr":"172.21.15.82:6809/2746051132","cluster_addr":"172.21.15.82:6811/2746051132","heartbeat_back_addr":"172.21.15.82:6815/2746051132","heartbeat_front_addr":"172.21.15.82:6813/2746051132","state":["exists","up"]},{"osd":4,"uuid":"deb22d86-4081-484d-96eb-9cb9d96b60bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":2005712460},{"type":"v1","addr":"172.21.15.82:6801","nonce":2005712460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":2005712460},{"type":"v1","addr":"172.21.15.82:6803","nonce":2005712460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":2005712460},{"type":"v1","addr":"172.21.15.82:6807","nonce":2005712460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":2005712460},{"type":"v1","addr":"172.21.15.82:6805","nonce":2005712460}]},"public_addr":"172.21.15.82:6801/2005712460","cluster_addr":"172.21.15.82:6803/2005712460","heartbeat_back_addr":"172.21.15.82:6807/2005712460","heartbeat_front_addr":"172.21.15.82:6805/2005712460","state":["exists","up"]},{"osd":5,"uuid":"061d332c-f51b-403d-bb18-eba415c97e7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":1267644232},{"type":"v1","addr":"172.21.15.82:6817","nonce":1267644232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":1267644232},{"type":"v1","addr":"172.21.15.82:6819","nonce":1267644232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":1267644232},{"type":"v1","addr":"172.21.15.82:6823","nonce":1267644232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":1267644232},{"type":"v1","addr":"172.21.15.82:6821","nonce":1267644232}]},"public_addr":"172.21.15.82:6817/1267644232","cluster_addr":"172.21.15.82:6819/1267644232","heartbeat_back_addr":"172.21.15.82:6823/1267644232","heartbeat_front_addr":"172.21.15.82:6821/1267644232","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T04:37:22.192087+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T04:37:22.017833+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T04:37:22.755774+0000","dead_epoch":77},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T04:37:22.663785+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T04:37:22.531758+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T04:37:22.400774+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.5","osds":[2,0]},{"pgid":"2.7","osds":[3,5]}],"pg_upmap_items":[{"pgid":"2.1","mappings":[{"from":2,"to":1}]},{"pgid":"2.4","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T04:40:46.219 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-18T04:40:46.220 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-18T04:40:46.220 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-18T04:40:46.220 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-18T04:40:46.221 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-18T04:40:46.221 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-18T04:40:46.376 INFO:teuthology.orchestra.run.smithi028.stdout:55834574894 2024-09-18T04:40:46.376 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-18T04:40:46.386 INFO:teuthology.orchestra.run.smithi028.stdout:51539607597 2024-09-18T04:40:46.386 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-18T04:40:46.409 INFO:teuthology.orchestra.run.smithi028.stdout:55834574893 2024-09-18T04:40:46.410 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-18T04:40:46.437 INFO:teuthology.orchestra.run.smithi028.stdout:369367187470 2024-09-18T04:40:46.437 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-18T04:40:46.438 INFO:teuthology.orchestra.run.smithi028.stdout:51539607597 2024-09-18T04:40:46.438 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-18T04:40:46.439 INFO:teuthology.orchestra.run.smithi028.stdout:55834574893 2024-09-18T04:40:46.440 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-18T04:40:46.747 INFO:teuthology.orchestra.run.smithi028.stdout:55834574892 2024-09-18T04:40:46.753 INFO:teuthology.orchestra.run.smithi028.stdout:51539607596 2024-09-18T04:40:46.762 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574894 got 55834574892 for osd.0 2024-09-18T04:40:46.767 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607597 got 51539607596 for osd.3 2024-09-18T04:40:46.851 INFO:teuthology.orchestra.run.smithi028.stdout:55834574891 2024-09-18T04:40:46.865 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574893 got 55834574891 for osd.4 2024-09-18T04:40:46.896 INFO:teuthology.orchestra.run.smithi028.stdout:55834574891 2024-09-18T04:40:46.910 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574893 got 55834574891 for osd.5 2024-09-18T04:40:46.944 INFO:teuthology.orchestra.run.smithi028.stdout:369367187469 2024-09-18T04:40:46.950 INFO:teuthology.orchestra.run.smithi028.stdout:51539607596 2024-09-18T04:40:46.958 INFO:tasks.ceph.ceph_manager.ceph:need seq 369367187470 got 369367187469 for osd.2 2024-09-18T04:40:46.963 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607597 got 51539607596 for osd.1 2024-09-18T04:40:47.763 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-18T04:40:47.768 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-18T04:40:47.868 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-18T04:40:47.911 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-18T04:40:47.959 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-18T04:40:47.964 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-18T04:40:48.109 INFO:teuthology.orchestra.run.smithi028.stdout:51539607596 2024-09-18T04:40:48.135 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607597 got 51539607596 for osd.3 2024-09-18T04:40:48.219 INFO:teuthology.orchestra.run.smithi028.stdout:55834574892 2024-09-18T04:40:48.233 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574894 got 55834574892 for osd.0 2024-09-18T04:40:48.294 INFO:teuthology.orchestra.run.smithi028.stdout:51539607596 2024-09-18T04:40:48.308 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607597 got 51539607596 for osd.1 2024-09-18T04:40:48.316 INFO:teuthology.orchestra.run.smithi028.stdout:369367187469 2024-09-18T04:40:48.328 INFO:tasks.ceph.ceph_manager.ceph:need seq 369367187470 got 369367187469 for osd.2 2024-09-18T04:40:48.330 INFO:teuthology.orchestra.run.smithi028.stdout:55834574891 2024-09-18T04:40:48.344 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574893 got 55834574891 for osd.4 2024-09-18T04:40:48.347 INFO:teuthology.orchestra.run.smithi028.stdout:55834574891 2024-09-18T04:40:48.360 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574893 got 55834574891 for osd.5 2024-09-18T04:40:49.136 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-18T04:40:49.235 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-18T04:40:49.309 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-18T04:40:49.330 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-18T04:40:49.345 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-18T04:40:49.361 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-18T04:40:49.516 INFO:teuthology.orchestra.run.smithi028.stdout:51539607597 2024-09-18T04:40:49.534 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607597 got 51539607597 for osd.3 2024-09-18T04:40:49.535 DEBUG:teuthology.parallel:result is None 2024-09-18T04:40:49.619 INFO:teuthology.orchestra.run.smithi028.stdout:51539607597 2024-09-18T04:40:49.636 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607597 got 51539607597 for osd.1 2024-09-18T04:40:49.636 DEBUG:teuthology.parallel:result is None 2024-09-18T04:40:49.678 INFO:teuthology.orchestra.run.smithi028.stdout:369367187470 2024-09-18T04:40:49.680 INFO:teuthology.orchestra.run.smithi028.stdout:55834574894 2024-09-18T04:40:49.691 INFO:tasks.ceph.ceph_manager.ceph:need seq 369367187470 got 369367187470 for osd.2 2024-09-18T04:40:49.692 DEBUG:teuthology.parallel:result is None 2024-09-18T04:40:49.696 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574894 got 55834574894 for osd.0 2024-09-18T04:40:49.696 DEBUG:teuthology.parallel:result is None 2024-09-18T04:40:49.709 INFO:teuthology.orchestra.run.smithi028.stdout:55834574893 2024-09-18T04:40:49.722 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574893 got 55834574893 for osd.5 2024-09-18T04:40:49.722 DEBUG:teuthology.parallel:result is None 2024-09-18T04:40:49.748 INFO:teuthology.orchestra.run.smithi028.stdout:55834574893 2024-09-18T04:40:49.761 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574893 got 55834574893 for osd.4 2024-09-18T04:40:49.761 DEBUG:teuthology.parallel:result is None 2024-09-18T04:40:49.761 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-18T04:40:49.761 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T04:40:50.052 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:40:50.054 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-18T04:40:50.065 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":226,"stamp":"2024-09-18T04:40:49.645730+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8601,"num_read_kb":183296,"num_write":1123,"num_write_kb":273318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":39960576,"num_keys_recovered":25,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1214,"ondisk_log_size":1214,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":5,"kb":566231040,"kb_used":201484,"kb_used_data":9076,"kb_used_omap":17,"kb_used_meta":192302,"kb_avail":566029556,"statfs":{"total":579820584960,"available":579614265344,"internally_reserved":0,"allocated":9293824,"data_stored":4774930,"data_compressed":66040,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":18101,"internal_metadata":196917579},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.108058"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":272,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T04:40:44.557504+0000","last_change":"2024-09-18T04:37:27.423069+0000","last_active":"2024-09-18T04:40:44.557504+0000","last_peered":"2024-09-18T04:40:44.557504+0000","last_clean":"2024-09-18T04:40:44.557504+0000","last_became_active":"2024-09-18T04:37:27.422350+0000","last_became_peered":"2024-09-18T04:37:27.422350+0000","last_unstale":"2024-09-18T04:40:44.557504+0000","last_undegraded":"2024-09-18T04:40:44.557504+0000","last_fullsized":"2024-09-18T04:40:44.557504+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:26.406292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:26.406292+0000","last_clean_scrub_stamp":"2024-09-18T04:37:26.406292+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:31:45.852149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"110'170","reported_seq":807,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T04:40:44.720562+0000","last_change":"2024-09-18T04:40:22.947240+0000","last_active":"2024-09-18T04:40:44.720562+0000","last_peered":"2024-09-18T04:40:44.720562+0000","last_clean":"2024-09-18T04:40:44.720562+0000","last_became_active":"2024-09-18T04:39:46.718066+0000","last_became_peered":"2024-09-18T04:39:46.718066+0000","last_unstale":"2024-09-18T04:40:44.720562+0000","last_undegraded":"2024-09-18T04:40:44.720562+0000","last_fullsized":"2024-09-18T04:40:44.720562+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T07:09:22.599338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011466040000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":748,"num_read_kb":17282,"num_write":161,"num_write_kb":25730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":9486336,"num_keys_recovered":9,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"110'255","reported_seq":1474,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T04:40:44.557512+0000","last_change":"2024-09-18T04:40:22.949604+0000","last_active":"2024-09-18T04:40:44.557512+0000","last_peered":"2024-09-18T04:40:44.557512+0000","last_clean":"2024-09-18T04:40:44.557512+0000","last_became_active":"2024-09-18T04:37:34.470841+0000","last_became_peered":"2024-09-18T04:37:34.470841+0000","last_unstale":"2024-09-18T04:40:44.557512+0000","last_undegraded":"2024-09-18T04:40:44.557512+0000","last_fullsized":"2024-09-18T04:40:44.557512+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":255,"log_dups_size":0,"ondisk_log_size":255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:16:34.528783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052682599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1817,"num_read_kb":40557,"num_write":337,"num_write_kb":45342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"110'317","reported_seq":2382,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T04:40:43.538497+0000","last_change":"2024-09-18T04:40:22.942796+0000","last_active":"2024-09-18T04:40:43.538497+0000","last_peered":"2024-09-18T04:40:43.538497+0000","last_clean":"2024-09-18T04:40:43.538497+0000","last_became_active":"2024-09-18T04:39:46.716998+0000","last_became_peered":"2024-09-18T04:39:46.716998+0000","last_unstale":"2024-09-18T04:40:43.538497+0000","last_undegraded":"2024-09-18T04:40:43.538497+0000","last_fullsized":"2024-09-18T04:40:43.538497+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":317,"log_dups_size":0,"ondisk_log_size":317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:37:53.810382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001074104,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4741,"num_read_kb":31642,"num_write":206,"num_write_kb":27778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":5308416,"num_keys_recovered":13,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"20'2","reported_seq":286,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T04:40:44.720539+0000","last_change":"2024-09-18T04:40:22.947782+0000","last_active":"2024-09-18T04:40:44.720539+0000","last_peered":"2024-09-18T04:40:44.720539+0000","last_clean":"2024-09-18T04:40:44.720539+0000","last_became_active":"2024-09-18T04:37:34.473088+0000","last_became_peered":"2024-09-18T04:37:34.473088+0000","last_unstale":"2024-09-18T04:40:44.720539+0000","last_undegraded":"2024-09-18T04:40:44.720539+0000","last_fullsized":"2024-09-18T04:40:44.720539+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:17:57.352397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001752392,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":139,"num_read_kb":139,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"110'65","reported_seq":162,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T04:40:44.720562+0000","last_change":"2024-09-18T04:40:22.947195+0000","last_active":"2024-09-18T04:40:44.720562+0000","last_peered":"2024-09-18T04:40:44.720562+0000","last_clean":"2024-09-18T04:40:44.720562+0000","last_became_active":"2024-09-18T04:38:59.462279+0000","last_became_peered":"2024-09-18T04:38:59.462279+0000","last_unstale":"2024-09-18T04:40:44.720562+0000","last_undegraded":"2024-09-18T04:40:44.720562+0000","last_fullsized":"2024-09-18T04:40:44.720562+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:32:58.894411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011357990000000001,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":101,"num_read_kb":16665,"num_write":51,"num_write_kb":36867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":12582912,"num_keys_recovered":3,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"110'89","reported_seq":461,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T04:40:43.538566+0000","last_change":"2024-09-18T04:40:22.942807+0000","last_active":"2024-09-18T04:40:43.538566+0000","last_peered":"2024-09-18T04:40:43.538566+0000","last_clean":"2024-09-18T04:40:43.538566+0000","last_became_active":"2024-09-18T04:39:46.717423+0000","last_became_peered":"2024-09-18T04:39:46.717423+0000","last_unstale":"2024-09-18T04:40:43.538566+0000","last_undegraded":"2024-09-18T04:40:43.538566+0000","last_fullsized":"2024-09-18T04:40:43.538566+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T09:52:29.555572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086846899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":248,"num_read_kb":31406,"num_write":66,"num_write_kb":49154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"110'100","reported_seq":375,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T04:40:44.554158+0000","last_change":"2024-09-18T04:40:23.022503+0000","last_active":"2024-09-18T04:40:44.554158+0000","last_peered":"2024-09-18T04:40:44.554158+0000","last_clean":"2024-09-18T04:40:44.554158+0000","last_became_active":"2024-09-18T04:37:34.472940+0000","last_became_peered":"2024-09-18T04:37:34.472940+0000","last_unstale":"2024-09-18T04:40:44.554158+0000","last_undegraded":"2024-09-18T04:40:44.554158+0000","last_fullsized":"2024-09-18T04:40:44.554158+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:18:29.266815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012195979999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":146,"num_read_kb":28719,"num_write":77,"num_write_kb":57349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"110'99","reported_seq":508,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T04:40:44.557595+0000","last_change":"2024-09-18T04:40:22.952155+0000","last_active":"2024-09-18T04:40:44.557595+0000","last_peered":"2024-09-18T04:40:44.557595+0000","last_clean":"2024-09-18T04:40:44.557595+0000","last_became_active":"2024-09-18T04:38:05.753962+0000","last_became_peered":"2024-09-18T04:38:05.753962+0000","last_unstale":"2024-09-18T04:40:44.557595+0000","last_undegraded":"2024-09-18T04:40:44.557595+0000","last_fullsized":"2024-09-18T04:40:44.557595+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:53:52.224098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030414399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":545,"num_read_kb":16786,"num_write":64,"num_write_kb":28680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8485,"num_read_kb":183196,"num_write":964,"num_write_kb":270902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":39960576,"num_keys_recovered":25,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3627,"internal_metadata":0},"log_size":1097,"ondisk_log_size":1097,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":66040,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":13,"seq":55834574893,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32608,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339232,"statfs":{"total":96636764160,"available":96603373568,"internally_reserved":0,"allocated":1523712,"data_stored":596698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.82799999999999996,"15min":0.85799999999999998},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.1060000000000001,"5min":1.5249999999999999,"15min":1.5249999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.77700000000000002,"15min":0.80100000000000005},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":0.96099999999999997,"5min":1.369,"15min":1.369},"last":0.69699999999999995}]},{"osd":1,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.82099999999999995,"15min":0.84599999999999997},"min":{"1min":0.59099999999999997,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":0.997,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.98199999999999998},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.78600000000000003,"15min":0.79600000000000004},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":1.01}]},{"osd":2,"last update":"Wed Sep 18 04:38:26 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.4930000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.83799999999999997}]},{"osd":3,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.73899999999999999,"15min":0.76400000000000001},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.90500000000000003,"5min":1.22,"15min":1.22},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.78100000000000003,"15min":0.79400000000000004},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.1819999999999999,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":1.0880000000000001}]},{"osd":4,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.77000000000000002,"15min":0.78200000000000003},"min":{"1min":0.45400000000000001,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.544,"5min":1.544,"15min":1.544},"last":0.95799999999999996},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.80400000000000005,"15min":0.82999999999999996},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.5109999999999999,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":0.79700000000000004}]}]},{"osd":4,"up_from":13,"seq":55834574893,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32608,"kb_used_data":1484,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339232,"statfs":{"total":96636764160,"available":96603373568,"internally_reserved":0,"allocated":1519616,"data_stored":596679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.747,"15min":0.76600000000000001},"min":{"1min":0.47899999999999998,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.94799999999999995,"5min":1.175,"15min":1.175},"last":0.88},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.77100000000000002,"15min":0.79000000000000004},"min":{"1min":0.60499999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.98999999999999999,"5min":1.258,"15min":1.258},"last":0.64600000000000002}]},{"osd":1,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.71599999999999997,"15min":0.72599999999999998},"min":{"1min":0.51000000000000001,"5min":0.438,"15min":0.438},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.84899999999999998,"15min":0.875},"min":{"1min":0.49399999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.1259999999999999,"5min":1.153,"15min":1.153},"last":1.2050000000000001}]},{"osd":2,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":1},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.619,"5min":1.619,"15min":1.619},"last":1.113}]},{"osd":3,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.79000000000000004,"15min":0.81499999999999995},"min":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.123,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.78900000000000003,"15min":0.79700000000000004},"min":{"1min":0.52100000000000002,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.98399999999999999,"5min":2.302,"15min":2.302},"last":0.60999999999999999}]},{"osd":5,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.86899999999999999,"15min":0.88300000000000001},"min":{"1min":0.55100000000000005,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.0820000000000001,"5min":1.365,"15min":1.365},"last":0.747},{"interface":"front","average":{"1min":0.879,"5min":0.88900000000000001,"15min":0.88700000000000001},"min":{"1min":0.59199999999999997,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.2250000000000001,"5min":2.0979999999999999,"15min":2.0979999999999999},"last":0.78200000000000003}]}]},{"osd":0,"up_from":13,"seq":55834574894,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36772,"kb_used_data":1560,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335068,"statfs":{"total":96636764160,"available":96599109632,"internally_reserved":0,"allocated":1597440,"data_stored":1191741,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.59299999999999997,"15min":0.60699999999999998},"min":{"1min":0.41399999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.76400000000000001,"5min":1.113,"15min":1.113},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.61599999999999999,"15min":0.63800000000000001},"min":{"1min":0.41399999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.85199999999999998,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.58699999999999997}]},{"osd":2,"last update":"Wed Sep 18 04:38:28 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.8180000000000001,"5min":1.8180000000000001,"15min":1.8180000000000001},"last":0.89100000000000001}]},{"osd":3,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.82099999999999995,"15min":0.81999999999999995},"min":{"1min":0.64100000000000001,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.93400000000000005},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.80800000000000005,"15min":0.82199999999999995},"min":{"1min":0.66000000000000003,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.97499999999999998,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.96899999999999997}]},{"osd":4,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":0.879,"15min":0.86499999999999999},"min":{"1min":0.66400000000000003,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":3.3740000000000001,"5min":3.3740000000000001,"15min":3.3740000000000001},"last":1.0589999999999999},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.89400000000000002,"15min":0.90700000000000003},"min":{"1min":0.66400000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.117,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":1.2769999999999999}]},{"osd":5,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.89000000000000001,"15min":0.92200000000000004},"min":{"1min":0.66900000000000004,"5min":0.496,"15min":0.496},"max":{"1min":1.006,"5min":1.8919999999999999,"15min":1.8919999999999999},"last":1.1439999999999999},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.94899999999999995,"15min":0.96999999999999997},"min":{"1min":0.69599999999999995,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.2370000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":1.1919999999999999}]}]},{"osd":3,"up_from":12,"seq":51539607597,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32680,"kb_used_data":1560,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339160,"statfs":{"total":96636764160,"available":96603299840,"internally_reserved":0,"allocated":1597440,"data_stored":1191741,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.88700000000000001,"15min":0.93799999999999994},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.145,"5min":2.665,"15min":2.665},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.83999999999999997,"15min":0.86799999999999999},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.81699999999999995}]},{"osd":1,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.82799999999999996,"15min":0.872},"min":{"1min":0.55500000000000005,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.93799999999999994,"5min":2.4510000000000001,"15min":2.4510000000000001},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.76000000000000001,"15min":0.78200000000000003},"min":{"1min":0.57799999999999996,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.93600000000000005,"5min":1.077,"15min":1.077},"last":0.69599999999999995}]},{"osd":2,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":2.577,"5min":2.577,"15min":2.577},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.91200000000000003}]},{"osd":4,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.71399999999999997,"15min":0.72299999999999998},"min":{"1min":0.46999999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.91700000000000004,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.76000000000000001,"15min":0.752},"min":{"1min":0.44500000000000001,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.302,"5min":1.302,"15min":1.302},"last":0.59799999999999998}]},{"osd":5,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.74399999999999999,"15min":0.74199999999999999},"min":{"1min":0.60699999999999998,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":0.89300000000000002,"5min":1.173,"15min":1.173},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.88,"15min":0.872},"min":{"1min":0.72799999999999998,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.1779999999999999,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.76100000000000001}]}]},{"osd":2,"up_from":86,"seq":369367187470,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30104,"kb_used_data":1484,"kb_used_omap":9,"kb_used_meta":28598,"kb_avail":94341736,"statfs":{"total":96636764160,"available":96605937664,"internally_reserved":0,"allocated":1519616,"data_stored":596679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10134,"internal_metadata":29284458},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70999999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88900000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0409999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98199999999999998}]}]},{"osd":1,"up_from":12,"seq":51539607598,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36712,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335128,"statfs":{"total":96636764160,"available":96599171072,"internally_reserved":0,"allocated":1536000,"data_stored":601392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:40:27 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.59199999999999997,"15min":0.59099999999999997},"min":{"1min":0.42099999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.746,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.61299999999999999,"15min":0.623},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.40200000000000002}]},{"osd":2,"last update":"Wed Sep 18 04:40:48 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.439},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.499}]},{"osd":3,"last update":"Wed Sep 18 04:40:27 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.80100000000000005},"min":{"1min":0.52300000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.85399999999999998,"15min":0.86599999999999999},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.01,"5min":1.194,"15min":1.194},"last":0.70899999999999996}]},{"osd":4,"last update":"Wed Sep 18 04:40:27 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.83199999999999996,"15min":0.83799999999999997},"min":{"1min":0.63500000000000001,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.121,"5min":1.236,"15min":1.236},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.90700000000000003,"15min":0.92300000000000004},"min":{"1min":0.64300000000000002,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.0720000000000001,"5min":1.268,"15min":1.268},"last":0.75}]},{"osd":5,"last update":"Wed Sep 18 04:40:27 2024","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.88800000000000001,"15min":0.90400000000000003},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.1479999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.91000000000000003,"15min":0.92200000000000004},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.1879999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.58799999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3627,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T04:40:50.067 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T04:40:50.361 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:40:50.363 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-18T04:40:50.374 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":226,"stamp":"2024-09-18T04:40:49.645730+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8601,"num_read_kb":183296,"num_write":1123,"num_write_kb":273318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":39960576,"num_keys_recovered":25,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1214,"ondisk_log_size":1214,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":5,"kb":566231040,"kb_used":201484,"kb_used_data":9076,"kb_used_omap":17,"kb_used_meta":192302,"kb_avail":566029556,"statfs":{"total":579820584960,"available":579614265344,"internally_reserved":0,"allocated":9293824,"data_stored":4774930,"data_compressed":66040,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":18101,"internal_metadata":196917579},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.108058"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":272,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T04:40:44.557504+0000","last_change":"2024-09-18T04:37:27.423069+0000","last_active":"2024-09-18T04:40:44.557504+0000","last_peered":"2024-09-18T04:40:44.557504+0000","last_clean":"2024-09-18T04:40:44.557504+0000","last_became_active":"2024-09-18T04:37:27.422350+0000","last_became_peered":"2024-09-18T04:37:27.422350+0000","last_unstale":"2024-09-18T04:40:44.557504+0000","last_undegraded":"2024-09-18T04:40:44.557504+0000","last_fullsized":"2024-09-18T04:40:44.557504+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:26.406292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:26.406292+0000","last_clean_scrub_stamp":"2024-09-18T04:37:26.406292+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:31:45.852149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"110'170","reported_seq":807,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T04:40:44.720562+0000","last_change":"2024-09-18T04:40:22.947240+0000","last_active":"2024-09-18T04:40:44.720562+0000","last_peered":"2024-09-18T04:40:44.720562+0000","last_clean":"2024-09-18T04:40:44.720562+0000","last_became_active":"2024-09-18T04:39:46.718066+0000","last_became_peered":"2024-09-18T04:39:46.718066+0000","last_unstale":"2024-09-18T04:40:44.720562+0000","last_undegraded":"2024-09-18T04:40:44.720562+0000","last_fullsized":"2024-09-18T04:40:44.720562+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T07:09:22.599338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011466040000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":748,"num_read_kb":17282,"num_write":161,"num_write_kb":25730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":9486336,"num_keys_recovered":9,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"110'255","reported_seq":1474,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T04:40:44.557512+0000","last_change":"2024-09-18T04:40:22.949604+0000","last_active":"2024-09-18T04:40:44.557512+0000","last_peered":"2024-09-18T04:40:44.557512+0000","last_clean":"2024-09-18T04:40:44.557512+0000","last_became_active":"2024-09-18T04:37:34.470841+0000","last_became_peered":"2024-09-18T04:37:34.470841+0000","last_unstale":"2024-09-18T04:40:44.557512+0000","last_undegraded":"2024-09-18T04:40:44.557512+0000","last_fullsized":"2024-09-18T04:40:44.557512+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":255,"log_dups_size":0,"ondisk_log_size":255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:16:34.528783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052682599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1817,"num_read_kb":40557,"num_write":337,"num_write_kb":45342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"110'317","reported_seq":2382,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T04:40:43.538497+0000","last_change":"2024-09-18T04:40:22.942796+0000","last_active":"2024-09-18T04:40:43.538497+0000","last_peered":"2024-09-18T04:40:43.538497+0000","last_clean":"2024-09-18T04:40:43.538497+0000","last_became_active":"2024-09-18T04:39:46.716998+0000","last_became_peered":"2024-09-18T04:39:46.716998+0000","last_unstale":"2024-09-18T04:40:43.538497+0000","last_undegraded":"2024-09-18T04:40:43.538497+0000","last_fullsized":"2024-09-18T04:40:43.538497+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":317,"log_dups_size":0,"ondisk_log_size":317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:37:53.810382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001074104,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4741,"num_read_kb":31642,"num_write":206,"num_write_kb":27778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":5308416,"num_keys_recovered":13,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"20'2","reported_seq":286,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T04:40:44.720539+0000","last_change":"2024-09-18T04:40:22.947782+0000","last_active":"2024-09-18T04:40:44.720539+0000","last_peered":"2024-09-18T04:40:44.720539+0000","last_clean":"2024-09-18T04:40:44.720539+0000","last_became_active":"2024-09-18T04:37:34.473088+0000","last_became_peered":"2024-09-18T04:37:34.473088+0000","last_unstale":"2024-09-18T04:40:44.720539+0000","last_undegraded":"2024-09-18T04:40:44.720539+0000","last_fullsized":"2024-09-18T04:40:44.720539+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:17:57.352397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001752392,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":139,"num_read_kb":139,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"110'65","reported_seq":162,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T04:40:44.720562+0000","last_change":"2024-09-18T04:40:22.947195+0000","last_active":"2024-09-18T04:40:44.720562+0000","last_peered":"2024-09-18T04:40:44.720562+0000","last_clean":"2024-09-18T04:40:44.720562+0000","last_became_active":"2024-09-18T04:38:59.462279+0000","last_became_peered":"2024-09-18T04:38:59.462279+0000","last_unstale":"2024-09-18T04:40:44.720562+0000","last_undegraded":"2024-09-18T04:40:44.720562+0000","last_fullsized":"2024-09-18T04:40:44.720562+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:32:58.894411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011357990000000001,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":101,"num_read_kb":16665,"num_write":51,"num_write_kb":36867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":12582912,"num_keys_recovered":3,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"110'89","reported_seq":461,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T04:40:43.538566+0000","last_change":"2024-09-18T04:40:22.942807+0000","last_active":"2024-09-18T04:40:43.538566+0000","last_peered":"2024-09-18T04:40:43.538566+0000","last_clean":"2024-09-18T04:40:43.538566+0000","last_became_active":"2024-09-18T04:39:46.717423+0000","last_became_peered":"2024-09-18T04:39:46.717423+0000","last_unstale":"2024-09-18T04:40:43.538566+0000","last_undegraded":"2024-09-18T04:40:43.538566+0000","last_fullsized":"2024-09-18T04:40:43.538566+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T09:52:29.555572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086846899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":248,"num_read_kb":31406,"num_write":66,"num_write_kb":49154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"110'100","reported_seq":375,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T04:40:44.554158+0000","last_change":"2024-09-18T04:40:23.022503+0000","last_active":"2024-09-18T04:40:44.554158+0000","last_peered":"2024-09-18T04:40:44.554158+0000","last_clean":"2024-09-18T04:40:44.554158+0000","last_became_active":"2024-09-18T04:37:34.472940+0000","last_became_peered":"2024-09-18T04:37:34.472940+0000","last_unstale":"2024-09-18T04:40:44.554158+0000","last_undegraded":"2024-09-18T04:40:44.554158+0000","last_fullsized":"2024-09-18T04:40:44.554158+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:18:29.266815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012195979999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":146,"num_read_kb":28719,"num_write":77,"num_write_kb":57349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"110'99","reported_seq":508,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T04:40:44.557595+0000","last_change":"2024-09-18T04:40:22.952155+0000","last_active":"2024-09-18T04:40:44.557595+0000","last_peered":"2024-09-18T04:40:44.557595+0000","last_clean":"2024-09-18T04:40:44.557595+0000","last_became_active":"2024-09-18T04:38:05.753962+0000","last_became_peered":"2024-09-18T04:38:05.753962+0000","last_unstale":"2024-09-18T04:40:44.557595+0000","last_undegraded":"2024-09-18T04:40:44.557595+0000","last_fullsized":"2024-09-18T04:40:44.557595+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:53:52.224098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030414399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":545,"num_read_kb":16786,"num_write":64,"num_write_kb":28680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8485,"num_read_kb":183196,"num_write":964,"num_write_kb":270902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":39960576,"num_keys_recovered":25,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3627,"internal_metadata":0},"log_size":1097,"ondisk_log_size":1097,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":66040,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":13,"seq":55834574893,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32608,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339232,"statfs":{"total":96636764160,"available":96603373568,"internally_reserved":0,"allocated":1523712,"data_stored":596698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.82799999999999996,"15min":0.85799999999999998},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.1060000000000001,"5min":1.5249999999999999,"15min":1.5249999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.77700000000000002,"15min":0.80100000000000005},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":0.96099999999999997,"5min":1.369,"15min":1.369},"last":0.69699999999999995}]},{"osd":1,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.82099999999999995,"15min":0.84599999999999997},"min":{"1min":0.59099999999999997,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":0.997,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.98199999999999998},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.78600000000000003,"15min":0.79600000000000004},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":1.01}]},{"osd":2,"last update":"Wed Sep 18 04:38:26 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.4930000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.83799999999999997}]},{"osd":3,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.73899999999999999,"15min":0.76400000000000001},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.90500000000000003,"5min":1.22,"15min":1.22},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.78100000000000003,"15min":0.79400000000000004},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.1819999999999999,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":1.0880000000000001}]},{"osd":4,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.77000000000000002,"15min":0.78200000000000003},"min":{"1min":0.45400000000000001,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.544,"5min":1.544,"15min":1.544},"last":0.95799999999999996},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.80400000000000005,"15min":0.82999999999999996},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.5109999999999999,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":0.79700000000000004}]}]},{"osd":4,"up_from":13,"seq":55834574893,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32608,"kb_used_data":1484,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339232,"statfs":{"total":96636764160,"available":96603373568,"internally_reserved":0,"allocated":1519616,"data_stored":596679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.747,"15min":0.76600000000000001},"min":{"1min":0.47899999999999998,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.94799999999999995,"5min":1.175,"15min":1.175},"last":0.88},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.77100000000000002,"15min":0.79000000000000004},"min":{"1min":0.60499999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.98999999999999999,"5min":1.258,"15min":1.258},"last":0.64600000000000002}]},{"osd":1,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.71599999999999997,"15min":0.72599999999999998},"min":{"1min":0.51000000000000001,"5min":0.438,"15min":0.438},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.84899999999999998,"15min":0.875},"min":{"1min":0.49399999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.1259999999999999,"5min":1.153,"15min":1.153},"last":1.2050000000000001}]},{"osd":2,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":1},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.619,"5min":1.619,"15min":1.619},"last":1.113}]},{"osd":3,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.79000000000000004,"15min":0.81499999999999995},"min":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.123,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.78900000000000003,"15min":0.79700000000000004},"min":{"1min":0.52100000000000002,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.98399999999999999,"5min":2.302,"15min":2.302},"last":0.60999999999999999}]},{"osd":5,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.86899999999999999,"15min":0.88300000000000001},"min":{"1min":0.55100000000000005,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.0820000000000001,"5min":1.365,"15min":1.365},"last":0.747},{"interface":"front","average":{"1min":0.879,"5min":0.88900000000000001,"15min":0.88700000000000001},"min":{"1min":0.59199999999999997,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.2250000000000001,"5min":2.0979999999999999,"15min":2.0979999999999999},"last":0.78200000000000003}]}]},{"osd":0,"up_from":13,"seq":55834574894,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36772,"kb_used_data":1560,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335068,"statfs":{"total":96636764160,"available":96599109632,"internally_reserved":0,"allocated":1597440,"data_stored":1191741,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.59299999999999997,"15min":0.60699999999999998},"min":{"1min":0.41399999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.76400000000000001,"5min":1.113,"15min":1.113},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.61599999999999999,"15min":0.63800000000000001},"min":{"1min":0.41399999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.85199999999999998,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.58699999999999997}]},{"osd":2,"last update":"Wed Sep 18 04:38:28 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.8180000000000001,"5min":1.8180000000000001,"15min":1.8180000000000001},"last":0.89100000000000001}]},{"osd":3,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.82099999999999995,"15min":0.81999999999999995},"min":{"1min":0.64100000000000001,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.93400000000000005},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.80800000000000005,"15min":0.82199999999999995},"min":{"1min":0.66000000000000003,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.97499999999999998,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.96899999999999997}]},{"osd":4,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":0.879,"15min":0.86499999999999999},"min":{"1min":0.66400000000000003,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":3.3740000000000001,"5min":3.3740000000000001,"15min":3.3740000000000001},"last":1.0589999999999999},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.89400000000000002,"15min":0.90700000000000003},"min":{"1min":0.66400000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.117,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":1.2769999999999999}]},{"osd":5,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.89000000000000001,"15min":0.92200000000000004},"min":{"1min":0.66900000000000004,"5min":0.496,"15min":0.496},"max":{"1min":1.006,"5min":1.8919999999999999,"15min":1.8919999999999999},"last":1.1439999999999999},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.94899999999999995,"15min":0.96999999999999997},"min":{"1min":0.69599999999999995,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.2370000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":1.1919999999999999}]}]},{"osd":3,"up_from":12,"seq":51539607597,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32680,"kb_used_data":1560,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339160,"statfs":{"total":96636764160,"available":96603299840,"internally_reserved":0,"allocated":1597440,"data_stored":1191741,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.88700000000000001,"15min":0.93799999999999994},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.145,"5min":2.665,"15min":2.665},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.83999999999999997,"15min":0.86799999999999999},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.81699999999999995}]},{"osd":1,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.82799999999999996,"15min":0.872},"min":{"1min":0.55500000000000005,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.93799999999999994,"5min":2.4510000000000001,"15min":2.4510000000000001},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.76000000000000001,"15min":0.78200000000000003},"min":{"1min":0.57799999999999996,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.93600000000000005,"5min":1.077,"15min":1.077},"last":0.69599999999999995}]},{"osd":2,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":2.577,"5min":2.577,"15min":2.577},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.91200000000000003}]},{"osd":4,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.71399999999999997,"15min":0.72299999999999998},"min":{"1min":0.46999999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.91700000000000004,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.76000000000000001,"15min":0.752},"min":{"1min":0.44500000000000001,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.302,"5min":1.302,"15min":1.302},"last":0.59799999999999998}]},{"osd":5,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.74399999999999999,"15min":0.74199999999999999},"min":{"1min":0.60699999999999998,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":0.89300000000000002,"5min":1.173,"15min":1.173},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.88,"15min":0.872},"min":{"1min":0.72799999999999998,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.1779999999999999,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.76100000000000001}]}]},{"osd":2,"up_from":86,"seq":369367187470,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30104,"kb_used_data":1484,"kb_used_omap":9,"kb_used_meta":28598,"kb_avail":94341736,"statfs":{"total":96636764160,"available":96605937664,"internally_reserved":0,"allocated":1519616,"data_stored":596679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10134,"internal_metadata":29284458},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70999999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88900000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0409999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98199999999999998}]}]},{"osd":1,"up_from":12,"seq":51539607598,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36712,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335128,"statfs":{"total":96636764160,"available":96599171072,"internally_reserved":0,"allocated":1536000,"data_stored":601392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:40:27 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.59199999999999997,"15min":0.59099999999999997},"min":{"1min":0.42099999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.746,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.61299999999999999,"15min":0.623},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.40200000000000002}]},{"osd":2,"last update":"Wed Sep 18 04:40:48 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.439},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.499}]},{"osd":3,"last update":"Wed Sep 18 04:40:27 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.80100000000000005},"min":{"1min":0.52300000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.85399999999999998,"15min":0.86599999999999999},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.01,"5min":1.194,"15min":1.194},"last":0.70899999999999996}]},{"osd":4,"last update":"Wed Sep 18 04:40:27 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.83199999999999996,"15min":0.83799999999999997},"min":{"1min":0.63500000000000001,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.121,"5min":1.236,"15min":1.236},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.90700000000000003,"15min":0.92300000000000004},"min":{"1min":0.64300000000000002,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.0720000000000001,"5min":1.268,"15min":1.268},"last":0.75}]},{"osd":5,"last update":"Wed Sep 18 04:40:27 2024","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.88800000000000001,"15min":0.90400000000000003},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.1479999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.91000000000000003,"15min":0.92200000000000004},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.1879999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.58799999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3627,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T04:40:50.376 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-18T04:40:50.376 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-09-18T04:40:50.703 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-18T04:40:50.713 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-18T04:40:50.713 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T04:40:51.021 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:40:51.024 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-18T04:40:51.037 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":226,"stamp":"2024-09-18T04:40:49.645730+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8601,"num_read_kb":183296,"num_write":1123,"num_write_kb":273318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":39960576,"num_keys_recovered":25,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1214,"ondisk_log_size":1214,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":5,"kb":566231040,"kb_used":201484,"kb_used_data":9076,"kb_used_omap":17,"kb_used_meta":192302,"kb_avail":566029556,"statfs":{"total":579820584960,"available":579614265344,"internally_reserved":0,"allocated":9293824,"data_stored":4774930,"data_compressed":66040,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":18101,"internal_metadata":196917579},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.108058"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":272,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T04:40:44.557504+0000","last_change":"2024-09-18T04:37:27.423069+0000","last_active":"2024-09-18T04:40:44.557504+0000","last_peered":"2024-09-18T04:40:44.557504+0000","last_clean":"2024-09-18T04:40:44.557504+0000","last_became_active":"2024-09-18T04:37:27.422350+0000","last_became_peered":"2024-09-18T04:37:27.422350+0000","last_unstale":"2024-09-18T04:40:44.557504+0000","last_undegraded":"2024-09-18T04:40:44.557504+0000","last_fullsized":"2024-09-18T04:40:44.557504+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:26.406292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:26.406292+0000","last_clean_scrub_stamp":"2024-09-18T04:37:26.406292+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:31:45.852149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"110'170","reported_seq":807,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T04:40:44.720562+0000","last_change":"2024-09-18T04:40:22.947240+0000","last_active":"2024-09-18T04:40:44.720562+0000","last_peered":"2024-09-18T04:40:44.720562+0000","last_clean":"2024-09-18T04:40:44.720562+0000","last_became_active":"2024-09-18T04:39:46.718066+0000","last_became_peered":"2024-09-18T04:39:46.718066+0000","last_unstale":"2024-09-18T04:40:44.720562+0000","last_undegraded":"2024-09-18T04:40:44.720562+0000","last_fullsized":"2024-09-18T04:40:44.720562+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T07:09:22.599338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011466040000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":748,"num_read_kb":17282,"num_write":161,"num_write_kb":25730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":9486336,"num_keys_recovered":9,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"110'255","reported_seq":1474,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T04:40:44.557512+0000","last_change":"2024-09-18T04:40:22.949604+0000","last_active":"2024-09-18T04:40:44.557512+0000","last_peered":"2024-09-18T04:40:44.557512+0000","last_clean":"2024-09-18T04:40:44.557512+0000","last_became_active":"2024-09-18T04:37:34.470841+0000","last_became_peered":"2024-09-18T04:37:34.470841+0000","last_unstale":"2024-09-18T04:40:44.557512+0000","last_undegraded":"2024-09-18T04:40:44.557512+0000","last_fullsized":"2024-09-18T04:40:44.557512+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":255,"log_dups_size":0,"ondisk_log_size":255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:16:34.528783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052682599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1817,"num_read_kb":40557,"num_write":337,"num_write_kb":45342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"110'317","reported_seq":2382,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T04:40:43.538497+0000","last_change":"2024-09-18T04:40:22.942796+0000","last_active":"2024-09-18T04:40:43.538497+0000","last_peered":"2024-09-18T04:40:43.538497+0000","last_clean":"2024-09-18T04:40:43.538497+0000","last_became_active":"2024-09-18T04:39:46.716998+0000","last_became_peered":"2024-09-18T04:39:46.716998+0000","last_unstale":"2024-09-18T04:40:43.538497+0000","last_undegraded":"2024-09-18T04:40:43.538497+0000","last_fullsized":"2024-09-18T04:40:43.538497+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":317,"log_dups_size":0,"ondisk_log_size":317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:37:53.810382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001074104,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4741,"num_read_kb":31642,"num_write":206,"num_write_kb":27778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":5308416,"num_keys_recovered":13,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"20'2","reported_seq":286,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T04:40:44.720539+0000","last_change":"2024-09-18T04:40:22.947782+0000","last_active":"2024-09-18T04:40:44.720539+0000","last_peered":"2024-09-18T04:40:44.720539+0000","last_clean":"2024-09-18T04:40:44.720539+0000","last_became_active":"2024-09-18T04:37:34.473088+0000","last_became_peered":"2024-09-18T04:37:34.473088+0000","last_unstale":"2024-09-18T04:40:44.720539+0000","last_undegraded":"2024-09-18T04:40:44.720539+0000","last_fullsized":"2024-09-18T04:40:44.720539+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:17:57.352397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001752392,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":139,"num_read_kb":139,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"110'65","reported_seq":162,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T04:40:44.720562+0000","last_change":"2024-09-18T04:40:22.947195+0000","last_active":"2024-09-18T04:40:44.720562+0000","last_peered":"2024-09-18T04:40:44.720562+0000","last_clean":"2024-09-18T04:40:44.720562+0000","last_became_active":"2024-09-18T04:38:59.462279+0000","last_became_peered":"2024-09-18T04:38:59.462279+0000","last_unstale":"2024-09-18T04:40:44.720562+0000","last_undegraded":"2024-09-18T04:40:44.720562+0000","last_fullsized":"2024-09-18T04:40:44.720562+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:32:58.894411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011357990000000001,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":101,"num_read_kb":16665,"num_write":51,"num_write_kb":36867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":12582912,"num_keys_recovered":3,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"110'89","reported_seq":461,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T04:40:43.538566+0000","last_change":"2024-09-18T04:40:22.942807+0000","last_active":"2024-09-18T04:40:43.538566+0000","last_peered":"2024-09-18T04:40:43.538566+0000","last_clean":"2024-09-18T04:40:43.538566+0000","last_became_active":"2024-09-18T04:39:46.717423+0000","last_became_peered":"2024-09-18T04:39:46.717423+0000","last_unstale":"2024-09-18T04:40:43.538566+0000","last_undegraded":"2024-09-18T04:40:43.538566+0000","last_fullsized":"2024-09-18T04:40:43.538566+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T09:52:29.555572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086846899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":248,"num_read_kb":31406,"num_write":66,"num_write_kb":49154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"110'100","reported_seq":375,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T04:40:44.554158+0000","last_change":"2024-09-18T04:40:23.022503+0000","last_active":"2024-09-18T04:40:44.554158+0000","last_peered":"2024-09-18T04:40:44.554158+0000","last_clean":"2024-09-18T04:40:44.554158+0000","last_became_active":"2024-09-18T04:37:34.472940+0000","last_became_peered":"2024-09-18T04:37:34.472940+0000","last_unstale":"2024-09-18T04:40:44.554158+0000","last_undegraded":"2024-09-18T04:40:44.554158+0000","last_fullsized":"2024-09-18T04:40:44.554158+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:18:29.266815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012195979999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":146,"num_read_kb":28719,"num_write":77,"num_write_kb":57349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"110'99","reported_seq":508,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T04:40:44.557595+0000","last_change":"2024-09-18T04:40:22.952155+0000","last_active":"2024-09-18T04:40:44.557595+0000","last_peered":"2024-09-18T04:40:44.557595+0000","last_clean":"2024-09-18T04:40:44.557595+0000","last_became_active":"2024-09-18T04:38:05.753962+0000","last_became_peered":"2024-09-18T04:38:05.753962+0000","last_unstale":"2024-09-18T04:40:44.557595+0000","last_undegraded":"2024-09-18T04:40:44.557595+0000","last_fullsized":"2024-09-18T04:40:44.557595+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:53:52.224098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030414399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":545,"num_read_kb":16786,"num_write":64,"num_write_kb":28680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8485,"num_read_kb":183196,"num_write":964,"num_write_kb":270902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":39960576,"num_keys_recovered":25,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3627,"internal_metadata":0},"log_size":1097,"ondisk_log_size":1097,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":66040,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":13,"seq":55834574893,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32608,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339232,"statfs":{"total":96636764160,"available":96603373568,"internally_reserved":0,"allocated":1523712,"data_stored":596698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.82799999999999996,"15min":0.85799999999999998},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.1060000000000001,"5min":1.5249999999999999,"15min":1.5249999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.77700000000000002,"15min":0.80100000000000005},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":0.96099999999999997,"5min":1.369,"15min":1.369},"last":0.69699999999999995}]},{"osd":1,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.82099999999999995,"15min":0.84599999999999997},"min":{"1min":0.59099999999999997,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":0.997,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.98199999999999998},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.78600000000000003,"15min":0.79600000000000004},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":1.01}]},{"osd":2,"last update":"Wed Sep 18 04:38:26 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.4930000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.83799999999999997}]},{"osd":3,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.73899999999999999,"15min":0.76400000000000001},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.90500000000000003,"5min":1.22,"15min":1.22},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.78100000000000003,"15min":0.79400000000000004},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.1819999999999999,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":1.0880000000000001}]},{"osd":4,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.77000000000000002,"15min":0.78200000000000003},"min":{"1min":0.45400000000000001,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.544,"5min":1.544,"15min":1.544},"last":0.95799999999999996},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.80400000000000005,"15min":0.82999999999999996},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.5109999999999999,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":0.79700000000000004}]}]},{"osd":4,"up_from":13,"seq":55834574893,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32608,"kb_used_data":1484,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339232,"statfs":{"total":96636764160,"available":96603373568,"internally_reserved":0,"allocated":1519616,"data_stored":596679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.747,"15min":0.76600000000000001},"min":{"1min":0.47899999999999998,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.94799999999999995,"5min":1.175,"15min":1.175},"last":0.88},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.77100000000000002,"15min":0.79000000000000004},"min":{"1min":0.60499999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.98999999999999999,"5min":1.258,"15min":1.258},"last":0.64600000000000002}]},{"osd":1,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.71599999999999997,"15min":0.72599999999999998},"min":{"1min":0.51000000000000001,"5min":0.438,"15min":0.438},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.84899999999999998,"15min":0.875},"min":{"1min":0.49399999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.1259999999999999,"5min":1.153,"15min":1.153},"last":1.2050000000000001}]},{"osd":2,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":1},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.619,"5min":1.619,"15min":1.619},"last":1.113}]},{"osd":3,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.79000000000000004,"15min":0.81499999999999995},"min":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.123,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.78900000000000003,"15min":0.79700000000000004},"min":{"1min":0.52100000000000002,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.98399999999999999,"5min":2.302,"15min":2.302},"last":0.60999999999999999}]},{"osd":5,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.86899999999999999,"15min":0.88300000000000001},"min":{"1min":0.55100000000000005,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.0820000000000001,"5min":1.365,"15min":1.365},"last":0.747},{"interface":"front","average":{"1min":0.879,"5min":0.88900000000000001,"15min":0.88700000000000001},"min":{"1min":0.59199999999999997,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.2250000000000001,"5min":2.0979999999999999,"15min":2.0979999999999999},"last":0.78200000000000003}]}]},{"osd":0,"up_from":13,"seq":55834574894,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36772,"kb_used_data":1560,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335068,"statfs":{"total":96636764160,"available":96599109632,"internally_reserved":0,"allocated":1597440,"data_stored":1191741,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.59299999999999997,"15min":0.60699999999999998},"min":{"1min":0.41399999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.76400000000000001,"5min":1.113,"15min":1.113},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.61599999999999999,"15min":0.63800000000000001},"min":{"1min":0.41399999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.85199999999999998,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.58699999999999997}]},{"osd":2,"last update":"Wed Sep 18 04:38:28 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.8180000000000001,"5min":1.8180000000000001,"15min":1.8180000000000001},"last":0.89100000000000001}]},{"osd":3,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.82099999999999995,"15min":0.81999999999999995},"min":{"1min":0.64100000000000001,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.93400000000000005},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.80800000000000005,"15min":0.82199999999999995},"min":{"1min":0.66000000000000003,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.97499999999999998,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.96899999999999997}]},{"osd":4,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":0.879,"15min":0.86499999999999999},"min":{"1min":0.66400000000000003,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":3.3740000000000001,"5min":3.3740000000000001,"15min":3.3740000000000001},"last":1.0589999999999999},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.89400000000000002,"15min":0.90700000000000003},"min":{"1min":0.66400000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.117,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":1.2769999999999999}]},{"osd":5,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.89000000000000001,"15min":0.92200000000000004},"min":{"1min":0.66900000000000004,"5min":0.496,"15min":0.496},"max":{"1min":1.006,"5min":1.8919999999999999,"15min":1.8919999999999999},"last":1.1439999999999999},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.94899999999999995,"15min":0.96999999999999997},"min":{"1min":0.69599999999999995,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.2370000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":1.1919999999999999}]}]},{"osd":3,"up_from":12,"seq":51539607597,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32680,"kb_used_data":1560,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339160,"statfs":{"total":96636764160,"available":96603299840,"internally_reserved":0,"allocated":1597440,"data_stored":1191741,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.88700000000000001,"15min":0.93799999999999994},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.145,"5min":2.665,"15min":2.665},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.83999999999999997,"15min":0.86799999999999999},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.81699999999999995}]},{"osd":1,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.82799999999999996,"15min":0.872},"min":{"1min":0.55500000000000005,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.93799999999999994,"5min":2.4510000000000001,"15min":2.4510000000000001},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.76000000000000001,"15min":0.78200000000000003},"min":{"1min":0.57799999999999996,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.93600000000000005,"5min":1.077,"15min":1.077},"last":0.69599999999999995}]},{"osd":2,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":2.577,"5min":2.577,"15min":2.577},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.91200000000000003}]},{"osd":4,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.71399999999999997,"15min":0.72299999999999998},"min":{"1min":0.46999999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.91700000000000004,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.76000000000000001,"15min":0.752},"min":{"1min":0.44500000000000001,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.302,"5min":1.302,"15min":1.302},"last":0.59799999999999998}]},{"osd":5,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.74399999999999999,"15min":0.74199999999999999},"min":{"1min":0.60699999999999998,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":0.89300000000000002,"5min":1.173,"15min":1.173},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.88,"15min":0.872},"min":{"1min":0.72799999999999998,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.1779999999999999,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.76100000000000001}]}]},{"osd":2,"up_from":86,"seq":369367187470,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30104,"kb_used_data":1484,"kb_used_omap":9,"kb_used_meta":28598,"kb_avail":94341736,"statfs":{"total":96636764160,"available":96605937664,"internally_reserved":0,"allocated":1519616,"data_stored":596679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10134,"internal_metadata":29284458},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70999999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88900000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0409999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98199999999999998}]}]},{"osd":1,"up_from":12,"seq":51539607598,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36712,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335128,"statfs":{"total":96636764160,"available":96599171072,"internally_reserved":0,"allocated":1536000,"data_stored":601392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:40:27 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.59199999999999997,"15min":0.59099999999999997},"min":{"1min":0.42099999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.746,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.61299999999999999,"15min":0.623},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.40200000000000002}]},{"osd":2,"last update":"Wed Sep 18 04:40:48 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.439},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.499}]},{"osd":3,"last update":"Wed Sep 18 04:40:27 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.80100000000000005},"min":{"1min":0.52300000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.85399999999999998,"15min":0.86599999999999999},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.01,"5min":1.194,"15min":1.194},"last":0.70899999999999996}]},{"osd":4,"last update":"Wed Sep 18 04:40:27 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.83199999999999996,"15min":0.83799999999999997},"min":{"1min":0.63500000000000001,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.121,"5min":1.236,"15min":1.236},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.90700000000000003,"15min":0.92300000000000004},"min":{"1min":0.64300000000000002,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.0720000000000001,"5min":1.268,"15min":1.268},"last":0.75}]},{"osd":5,"last update":"Wed Sep 18 04:40:27 2024","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.88800000000000001,"15min":0.90400000000000003},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.1479999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.91000000000000003,"15min":0.92200000000000004},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.1879999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.58799999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3627,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T04:40:51.039 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T04:40:51.351 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:40:51.356 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-18T04:40:51.366 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":226,"stamp":"2024-09-18T04:40:49.645730+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8601,"num_read_kb":183296,"num_write":1123,"num_write_kb":273318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":39960576,"num_keys_recovered":25,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1214,"ondisk_log_size":1214,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":5,"kb":566231040,"kb_used":201484,"kb_used_data":9076,"kb_used_omap":17,"kb_used_meta":192302,"kb_avail":566029556,"statfs":{"total":579820584960,"available":579614265344,"internally_reserved":0,"allocated":9293824,"data_stored":4774930,"data_compressed":66040,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":18101,"internal_metadata":196917579},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.108058"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":272,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T04:40:44.557504+0000","last_change":"2024-09-18T04:37:27.423069+0000","last_active":"2024-09-18T04:40:44.557504+0000","last_peered":"2024-09-18T04:40:44.557504+0000","last_clean":"2024-09-18T04:40:44.557504+0000","last_became_active":"2024-09-18T04:37:27.422350+0000","last_became_peered":"2024-09-18T04:37:27.422350+0000","last_unstale":"2024-09-18T04:40:44.557504+0000","last_undegraded":"2024-09-18T04:40:44.557504+0000","last_fullsized":"2024-09-18T04:40:44.557504+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:26.406292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:26.406292+0000","last_clean_scrub_stamp":"2024-09-18T04:37:26.406292+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:31:45.852149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"110'170","reported_seq":807,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T04:40:44.720562+0000","last_change":"2024-09-18T04:40:22.947240+0000","last_active":"2024-09-18T04:40:44.720562+0000","last_peered":"2024-09-18T04:40:44.720562+0000","last_clean":"2024-09-18T04:40:44.720562+0000","last_became_active":"2024-09-18T04:39:46.718066+0000","last_became_peered":"2024-09-18T04:39:46.718066+0000","last_unstale":"2024-09-18T04:40:44.720562+0000","last_undegraded":"2024-09-18T04:40:44.720562+0000","last_fullsized":"2024-09-18T04:40:44.720562+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T07:09:22.599338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011466040000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":748,"num_read_kb":17282,"num_write":161,"num_write_kb":25730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":9486336,"num_keys_recovered":9,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"110'255","reported_seq":1474,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T04:40:44.557512+0000","last_change":"2024-09-18T04:40:22.949604+0000","last_active":"2024-09-18T04:40:44.557512+0000","last_peered":"2024-09-18T04:40:44.557512+0000","last_clean":"2024-09-18T04:40:44.557512+0000","last_became_active":"2024-09-18T04:37:34.470841+0000","last_became_peered":"2024-09-18T04:37:34.470841+0000","last_unstale":"2024-09-18T04:40:44.557512+0000","last_undegraded":"2024-09-18T04:40:44.557512+0000","last_fullsized":"2024-09-18T04:40:44.557512+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":255,"log_dups_size":0,"ondisk_log_size":255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:16:34.528783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052682599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1817,"num_read_kb":40557,"num_write":337,"num_write_kb":45342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"110'317","reported_seq":2382,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T04:40:43.538497+0000","last_change":"2024-09-18T04:40:22.942796+0000","last_active":"2024-09-18T04:40:43.538497+0000","last_peered":"2024-09-18T04:40:43.538497+0000","last_clean":"2024-09-18T04:40:43.538497+0000","last_became_active":"2024-09-18T04:39:46.716998+0000","last_became_peered":"2024-09-18T04:39:46.716998+0000","last_unstale":"2024-09-18T04:40:43.538497+0000","last_undegraded":"2024-09-18T04:40:43.538497+0000","last_fullsized":"2024-09-18T04:40:43.538497+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":317,"log_dups_size":0,"ondisk_log_size":317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:37:53.810382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001074104,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4741,"num_read_kb":31642,"num_write":206,"num_write_kb":27778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":5308416,"num_keys_recovered":13,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"20'2","reported_seq":286,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T04:40:44.720539+0000","last_change":"2024-09-18T04:40:22.947782+0000","last_active":"2024-09-18T04:40:44.720539+0000","last_peered":"2024-09-18T04:40:44.720539+0000","last_clean":"2024-09-18T04:40:44.720539+0000","last_became_active":"2024-09-18T04:37:34.473088+0000","last_became_peered":"2024-09-18T04:37:34.473088+0000","last_unstale":"2024-09-18T04:40:44.720539+0000","last_undegraded":"2024-09-18T04:40:44.720539+0000","last_fullsized":"2024-09-18T04:40:44.720539+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:17:57.352397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001752392,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":139,"num_read_kb":139,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"110'65","reported_seq":162,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T04:40:44.720562+0000","last_change":"2024-09-18T04:40:22.947195+0000","last_active":"2024-09-18T04:40:44.720562+0000","last_peered":"2024-09-18T04:40:44.720562+0000","last_clean":"2024-09-18T04:40:44.720562+0000","last_became_active":"2024-09-18T04:38:59.462279+0000","last_became_peered":"2024-09-18T04:38:59.462279+0000","last_unstale":"2024-09-18T04:40:44.720562+0000","last_undegraded":"2024-09-18T04:40:44.720562+0000","last_fullsized":"2024-09-18T04:40:44.720562+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:32:58.894411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011357990000000001,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":101,"num_read_kb":16665,"num_write":51,"num_write_kb":36867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":12582912,"num_keys_recovered":3,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"110'89","reported_seq":461,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T04:40:43.538566+0000","last_change":"2024-09-18T04:40:22.942807+0000","last_active":"2024-09-18T04:40:43.538566+0000","last_peered":"2024-09-18T04:40:43.538566+0000","last_clean":"2024-09-18T04:40:43.538566+0000","last_became_active":"2024-09-18T04:39:46.717423+0000","last_became_peered":"2024-09-18T04:39:46.717423+0000","last_unstale":"2024-09-18T04:40:43.538566+0000","last_undegraded":"2024-09-18T04:40:43.538566+0000","last_fullsized":"2024-09-18T04:40:43.538566+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T09:52:29.555572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086846899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":248,"num_read_kb":31406,"num_write":66,"num_write_kb":49154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"110'100","reported_seq":375,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T04:40:44.554158+0000","last_change":"2024-09-18T04:40:23.022503+0000","last_active":"2024-09-18T04:40:44.554158+0000","last_peered":"2024-09-18T04:40:44.554158+0000","last_clean":"2024-09-18T04:40:44.554158+0000","last_became_active":"2024-09-18T04:37:34.472940+0000","last_became_peered":"2024-09-18T04:37:34.472940+0000","last_unstale":"2024-09-18T04:40:44.554158+0000","last_undegraded":"2024-09-18T04:40:44.554158+0000","last_fullsized":"2024-09-18T04:40:44.554158+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:18:29.266815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012195979999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":146,"num_read_kb":28719,"num_write":77,"num_write_kb":57349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"110'99","reported_seq":508,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T04:40:44.557595+0000","last_change":"2024-09-18T04:40:22.952155+0000","last_active":"2024-09-18T04:40:44.557595+0000","last_peered":"2024-09-18T04:40:44.557595+0000","last_clean":"2024-09-18T04:40:44.557595+0000","last_became_active":"2024-09-18T04:38:05.753962+0000","last_became_peered":"2024-09-18T04:38:05.753962+0000","last_unstale":"2024-09-18T04:40:44.557595+0000","last_undegraded":"2024-09-18T04:40:44.557595+0000","last_fullsized":"2024-09-18T04:40:44.557595+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:53:52.224098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030414399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":545,"num_read_kb":16786,"num_write":64,"num_write_kb":28680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8485,"num_read_kb":183196,"num_write":964,"num_write_kb":270902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":39960576,"num_keys_recovered":25,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3627,"internal_metadata":0},"log_size":1097,"ondisk_log_size":1097,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":66040,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":13,"seq":55834574893,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32608,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339232,"statfs":{"total":96636764160,"available":96603373568,"internally_reserved":0,"allocated":1523712,"data_stored":596698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.82799999999999996,"15min":0.85799999999999998},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.1060000000000001,"5min":1.5249999999999999,"15min":1.5249999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.77700000000000002,"15min":0.80100000000000005},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":0.96099999999999997,"5min":1.369,"15min":1.369},"last":0.69699999999999995}]},{"osd":1,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.82099999999999995,"15min":0.84599999999999997},"min":{"1min":0.59099999999999997,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":0.997,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.98199999999999998},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.78600000000000003,"15min":0.79600000000000004},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":1.01}]},{"osd":2,"last update":"Wed Sep 18 04:38:26 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.4930000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.83799999999999997}]},{"osd":3,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.73899999999999999,"15min":0.76400000000000001},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.90500000000000003,"5min":1.22,"15min":1.22},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.78100000000000003,"15min":0.79400000000000004},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.1819999999999999,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":1.0880000000000001}]},{"osd":4,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.77000000000000002,"15min":0.78200000000000003},"min":{"1min":0.45400000000000001,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.544,"5min":1.544,"15min":1.544},"last":0.95799999999999996},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.80400000000000005,"15min":0.82999999999999996},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.5109999999999999,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":0.79700000000000004}]}]},{"osd":4,"up_from":13,"seq":55834574893,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32608,"kb_used_data":1484,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339232,"statfs":{"total":96636764160,"available":96603373568,"internally_reserved":0,"allocated":1519616,"data_stored":596679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.747,"15min":0.76600000000000001},"min":{"1min":0.47899999999999998,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.94799999999999995,"5min":1.175,"15min":1.175},"last":0.88},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.77100000000000002,"15min":0.79000000000000004},"min":{"1min":0.60499999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.98999999999999999,"5min":1.258,"15min":1.258},"last":0.64600000000000002}]},{"osd":1,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.71599999999999997,"15min":0.72599999999999998},"min":{"1min":0.51000000000000001,"5min":0.438,"15min":0.438},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.84899999999999998,"15min":0.875},"min":{"1min":0.49399999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.1259999999999999,"5min":1.153,"15min":1.153},"last":1.2050000000000001}]},{"osd":2,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":1},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.619,"5min":1.619,"15min":1.619},"last":1.113}]},{"osd":3,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.79000000000000004,"15min":0.81499999999999995},"min":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.123,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.78900000000000003,"15min":0.79700000000000004},"min":{"1min":0.52100000000000002,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.98399999999999999,"5min":2.302,"15min":2.302},"last":0.60999999999999999}]},{"osd":5,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.86899999999999999,"15min":0.88300000000000001},"min":{"1min":0.55100000000000005,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.0820000000000001,"5min":1.365,"15min":1.365},"last":0.747},{"interface":"front","average":{"1min":0.879,"5min":0.88900000000000001,"15min":0.88700000000000001},"min":{"1min":0.59199999999999997,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.2250000000000001,"5min":2.0979999999999999,"15min":2.0979999999999999},"last":0.78200000000000003}]}]},{"osd":0,"up_from":13,"seq":55834574894,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36772,"kb_used_data":1560,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335068,"statfs":{"total":96636764160,"available":96599109632,"internally_reserved":0,"allocated":1597440,"data_stored":1191741,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.59299999999999997,"15min":0.60699999999999998},"min":{"1min":0.41399999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.76400000000000001,"5min":1.113,"15min":1.113},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.61599999999999999,"15min":0.63800000000000001},"min":{"1min":0.41399999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.85199999999999998,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.58699999999999997}]},{"osd":2,"last update":"Wed Sep 18 04:38:28 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.8180000000000001,"5min":1.8180000000000001,"15min":1.8180000000000001},"last":0.89100000000000001}]},{"osd":3,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.82099999999999995,"15min":0.81999999999999995},"min":{"1min":0.64100000000000001,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.93400000000000005},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.80800000000000005,"15min":0.82199999999999995},"min":{"1min":0.66000000000000003,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.97499999999999998,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.96899999999999997}]},{"osd":4,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":0.879,"15min":0.86499999999999999},"min":{"1min":0.66400000000000003,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":3.3740000000000001,"5min":3.3740000000000001,"15min":3.3740000000000001},"last":1.0589999999999999},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.89400000000000002,"15min":0.90700000000000003},"min":{"1min":0.66400000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.117,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":1.2769999999999999}]},{"osd":5,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.89000000000000001,"15min":0.92200000000000004},"min":{"1min":0.66900000000000004,"5min":0.496,"15min":0.496},"max":{"1min":1.006,"5min":1.8919999999999999,"15min":1.8919999999999999},"last":1.1439999999999999},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.94899999999999995,"15min":0.96999999999999997},"min":{"1min":0.69599999999999995,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.2370000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":1.1919999999999999}]}]},{"osd":3,"up_from":12,"seq":51539607597,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32680,"kb_used_data":1560,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339160,"statfs":{"total":96636764160,"available":96603299840,"internally_reserved":0,"allocated":1597440,"data_stored":1191741,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.88700000000000001,"15min":0.93799999999999994},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.145,"5min":2.665,"15min":2.665},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.83999999999999997,"15min":0.86799999999999999},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.81699999999999995}]},{"osd":1,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.82799999999999996,"15min":0.872},"min":{"1min":0.55500000000000005,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.93799999999999994,"5min":2.4510000000000001,"15min":2.4510000000000001},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.76000000000000001,"15min":0.78200000000000003},"min":{"1min":0.57799999999999996,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.93600000000000005,"5min":1.077,"15min":1.077},"last":0.69599999999999995}]},{"osd":2,"last update":"Wed Sep 18 04:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":2.577,"5min":2.577,"15min":2.577},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.91200000000000003}]},{"osd":4,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.71399999999999997,"15min":0.72299999999999998},"min":{"1min":0.46999999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.91700000000000004,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.76000000000000001,"15min":0.752},"min":{"1min":0.44500000000000001,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.302,"5min":1.302,"15min":1.302},"last":0.59799999999999998}]},{"osd":5,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.74399999999999999,"15min":0.74199999999999999},"min":{"1min":0.60699999999999998,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":0.89300000000000002,"5min":1.173,"15min":1.173},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.88,"15min":0.872},"min":{"1min":0.72799999999999998,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.1779999999999999,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.76100000000000001}]}]},{"osd":2,"up_from":86,"seq":369367187470,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30104,"kb_used_data":1484,"kb_used_omap":9,"kb_used_meta":28598,"kb_avail":94341736,"statfs":{"total":96636764160,"available":96605937664,"internally_reserved":0,"allocated":1519616,"data_stored":596679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10134,"internal_metadata":29284458},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70999999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88900000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0409999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98199999999999998}]}]},{"osd":1,"up_from":12,"seq":51539607598,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36712,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335128,"statfs":{"total":96636764160,"available":96599171072,"internally_reserved":0,"allocated":1536000,"data_stored":601392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:40:27 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.59199999999999997,"15min":0.59099999999999997},"min":{"1min":0.42099999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.746,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.61299999999999999,"15min":0.623},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.40200000000000002}]},{"osd":2,"last update":"Wed Sep 18 04:40:48 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.439},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.499}]},{"osd":3,"last update":"Wed Sep 18 04:40:27 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.80100000000000005},"min":{"1min":0.52300000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.85399999999999998,"15min":0.86599999999999999},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.01,"5min":1.194,"15min":1.194},"last":0.70899999999999996}]},{"osd":4,"last update":"Wed Sep 18 04:40:27 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.83199999999999996,"15min":0.83799999999999997},"min":{"1min":0.63500000000000001,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.121,"5min":1.236,"15min":1.236},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.90700000000000003,"15min":0.92300000000000004},"min":{"1min":0.64300000000000002,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.0720000000000001,"5min":1.268,"15min":1.268},"last":0.75}]},{"osd":5,"last update":"Wed Sep 18 04:40:27 2024","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.88800000000000001,"15min":0.90400000000000003},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.1479999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.91000000000000003,"15min":0.92200000000000004},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.1879999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.58799999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3627,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T04:40:51.367 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-18T04:40:51.367 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T04:40:51.669 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:40:51.672 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-18T04:40:51.684 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":227,"stamp":"2024-09-18T04:40:51.646318+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8601,"num_read_kb":183296,"num_write":1123,"num_write_kb":273318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":39960576,"num_keys_recovered":25,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1214,"ondisk_log_size":1214,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":5,"kb":566231040,"kb_used":201512,"kb_used_data":9112,"kb_used_omap":17,"kb_used_meta":192302,"kb_avail":566029528,"statfs":{"total":579820584960,"available":579614236672,"internally_reserved":0,"allocated":9330688,"data_stored":4789012,"data_compressed":66040,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":18101,"internal_metadata":196917579},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002201"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":272,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T04:40:44.557504+0000","last_change":"2024-09-18T04:37:27.423069+0000","last_active":"2024-09-18T04:40:44.557504+0000","last_peered":"2024-09-18T04:40:44.557504+0000","last_clean":"2024-09-18T04:40:44.557504+0000","last_became_active":"2024-09-18T04:37:27.422350+0000","last_became_peered":"2024-09-18T04:37:27.422350+0000","last_unstale":"2024-09-18T04:40:44.557504+0000","last_undegraded":"2024-09-18T04:40:44.557504+0000","last_fullsized":"2024-09-18T04:40:44.557504+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:26.406292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:26.406292+0000","last_clean_scrub_stamp":"2024-09-18T04:37:26.406292+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:31:45.852149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"110'170","reported_seq":807,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T04:40:44.720562+0000","last_change":"2024-09-18T04:40:22.947240+0000","last_active":"2024-09-18T04:40:44.720562+0000","last_peered":"2024-09-18T04:40:44.720562+0000","last_clean":"2024-09-18T04:40:44.720562+0000","last_became_active":"2024-09-18T04:39:46.718066+0000","last_became_peered":"2024-09-18T04:39:46.718066+0000","last_unstale":"2024-09-18T04:40:44.720562+0000","last_undegraded":"2024-09-18T04:40:44.720562+0000","last_fullsized":"2024-09-18T04:40:44.720562+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T07:09:22.599338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011466040000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":748,"num_read_kb":17282,"num_write":161,"num_write_kb":25730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":9486336,"num_keys_recovered":9,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"110'255","reported_seq":1474,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T04:40:44.557512+0000","last_change":"2024-09-18T04:40:22.949604+0000","last_active":"2024-09-18T04:40:44.557512+0000","last_peered":"2024-09-18T04:40:44.557512+0000","last_clean":"2024-09-18T04:40:44.557512+0000","last_became_active":"2024-09-18T04:37:34.470841+0000","last_became_peered":"2024-09-18T04:37:34.470841+0000","last_unstale":"2024-09-18T04:40:44.557512+0000","last_undegraded":"2024-09-18T04:40:44.557512+0000","last_fullsized":"2024-09-18T04:40:44.557512+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":255,"log_dups_size":0,"ondisk_log_size":255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:16:34.528783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052682599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1817,"num_read_kb":40557,"num_write":337,"num_write_kb":45342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"110'317","reported_seq":2383,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T04:40:44.765761+0000","last_change":"2024-09-18T04:40:22.942796+0000","last_active":"2024-09-18T04:40:44.765761+0000","last_peered":"2024-09-18T04:40:44.765761+0000","last_clean":"2024-09-18T04:40:44.765761+0000","last_became_active":"2024-09-18T04:39:46.716998+0000","last_became_peered":"2024-09-18T04:39:46.716998+0000","last_unstale":"2024-09-18T04:40:44.765761+0000","last_undegraded":"2024-09-18T04:40:44.765761+0000","last_fullsized":"2024-09-18T04:40:44.765761+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":317,"log_dups_size":0,"ondisk_log_size":317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:37:53.810382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001074104,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4741,"num_read_kb":31642,"num_write":206,"num_write_kb":27778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":5308416,"num_keys_recovered":13,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"20'2","reported_seq":286,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T04:40:44.720539+0000","last_change":"2024-09-18T04:40:22.947782+0000","last_active":"2024-09-18T04:40:44.720539+0000","last_peered":"2024-09-18T04:40:44.720539+0000","last_clean":"2024-09-18T04:40:44.720539+0000","last_became_active":"2024-09-18T04:37:34.473088+0000","last_became_peered":"2024-09-18T04:37:34.473088+0000","last_unstale":"2024-09-18T04:40:44.720539+0000","last_undegraded":"2024-09-18T04:40:44.720539+0000","last_fullsized":"2024-09-18T04:40:44.720539+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:17:57.352397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001752392,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":139,"num_read_kb":139,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"110'65","reported_seq":162,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T04:40:44.720562+0000","last_change":"2024-09-18T04:40:22.947195+0000","last_active":"2024-09-18T04:40:44.720562+0000","last_peered":"2024-09-18T04:40:44.720562+0000","last_clean":"2024-09-18T04:40:44.720562+0000","last_became_active":"2024-09-18T04:38:59.462279+0000","last_became_peered":"2024-09-18T04:38:59.462279+0000","last_unstale":"2024-09-18T04:40:44.720562+0000","last_undegraded":"2024-09-18T04:40:44.720562+0000","last_fullsized":"2024-09-18T04:40:44.720562+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:32:58.894411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011357990000000001,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":101,"num_read_kb":16665,"num_write":51,"num_write_kb":36867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":12582912,"num_keys_recovered":3,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"110'89","reported_seq":462,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T04:40:44.765728+0000","last_change":"2024-09-18T04:40:22.942807+0000","last_active":"2024-09-18T04:40:44.765728+0000","last_peered":"2024-09-18T04:40:44.765728+0000","last_clean":"2024-09-18T04:40:44.765728+0000","last_became_active":"2024-09-18T04:39:46.717423+0000","last_became_peered":"2024-09-18T04:39:46.717423+0000","last_unstale":"2024-09-18T04:40:44.765728+0000","last_undegraded":"2024-09-18T04:40:44.765728+0000","last_fullsized":"2024-09-18T04:40:44.765728+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T09:52:29.555572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086846899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":248,"num_read_kb":31406,"num_write":66,"num_write_kb":49154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"110'100","reported_seq":375,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T04:40:44.554158+0000","last_change":"2024-09-18T04:40:23.022503+0000","last_active":"2024-09-18T04:40:44.554158+0000","last_peered":"2024-09-18T04:40:44.554158+0000","last_clean":"2024-09-18T04:40:44.554158+0000","last_became_active":"2024-09-18T04:37:34.472940+0000","last_became_peered":"2024-09-18T04:37:34.472940+0000","last_unstale":"2024-09-18T04:40:44.554158+0000","last_undegraded":"2024-09-18T04:40:44.554158+0000","last_fullsized":"2024-09-18T04:40:44.554158+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:18:29.266815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012195979999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":146,"num_read_kb":28719,"num_write":77,"num_write_kb":57349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"110'99","reported_seq":508,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T04:40:44.557595+0000","last_change":"2024-09-18T04:40:22.952155+0000","last_active":"2024-09-18T04:40:44.557595+0000","last_peered":"2024-09-18T04:40:44.557595+0000","last_clean":"2024-09-18T04:40:44.557595+0000","last_became_active":"2024-09-18T04:38:05.753962+0000","last_became_peered":"2024-09-18T04:38:05.753962+0000","last_unstale":"2024-09-18T04:40:44.557595+0000","last_undegraded":"2024-09-18T04:40:44.557595+0000","last_fullsized":"2024-09-18T04:40:44.557595+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:53:52.224098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030414399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":545,"num_read_kb":16786,"num_write":64,"num_write_kb":28680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8485,"num_read_kb":183196,"num_write":964,"num_write_kb":270902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":39960576,"num_keys_recovered":25,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3627,"internal_metadata":0},"log_size":1097,"ondisk_log_size":1097,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":66040,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":13,"seq":55834574894,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32616,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339224,"statfs":{"total":96636764160,"available":96603365376,"internally_reserved":0,"allocated":1536000,"data_stored":601392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.82799999999999996,"15min":0.85799999999999998},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.1060000000000001,"5min":1.5249999999999999,"15min":1.5249999999999999},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.77700000000000002,"15min":0.80100000000000005},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":0.96099999999999997,"5min":1.369,"15min":1.369},"last":0.83699999999999997}]},{"osd":1,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.82099999999999995,"15min":0.84599999999999997},"min":{"1min":0.59099999999999997,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":0.997,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.78600000000000003,"15min":0.79600000000000004},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.628}]},{"osd":2,"last update":"Wed Sep 18 04:40:49 2024","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.278,"5min":1.278,"15min":1.278},"last":0.91100000000000003}]},{"osd":3,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.73899999999999999,"15min":0.76400000000000001},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.90500000000000003,"5min":1.22,"15min":1.22},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.78100000000000003,"15min":0.79400000000000004},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.1819999999999999,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.86699999999999999}]},{"osd":4,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.77000000000000002,"15min":0.78200000000000003},"min":{"1min":0.45400000000000001,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.544,"5min":1.544,"15min":1.544},"last":0.753},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.80400000000000005,"15min":0.82999999999999996},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.5109999999999999,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":0.71999999999999997}]}]},{"osd":4,"up_from":13,"seq":55834574894,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32612,"kb_used_data":1496,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339228,"statfs":{"total":96636764160,"available":96603369472,"internally_reserved":0,"allocated":1531904,"data_stored":601373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.747,"15min":0.76600000000000001},"min":{"1min":0.47899999999999998,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.94799999999999995,"5min":1.175,"15min":1.175},"last":0.96199999999999997},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.77100000000000002,"15min":0.79000000000000004},"min":{"1min":0.60499999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.98999999999999999,"5min":1.258,"15min":1.258},"last":0.86899999999999999}]},{"osd":1,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.71599999999999997,"15min":0.72599999999999998},"min":{"1min":0.51000000000000001,"5min":0.438,"15min":0.438},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.84899999999999998,"15min":0.875},"min":{"1min":0.49399999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.1259999999999999,"5min":1.153,"15min":1.153},"last":1.0049999999999999}]},{"osd":2,"last update":"Wed Sep 18 04:40:46 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1,"5min":1,"15min":1},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.87,"5min":0.87,"15min":0.87},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.113,"5min":1.113,"15min":1.113},"last":0.91500000000000004}]},{"osd":3,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.79000000000000004,"15min":0.81499999999999995},"min":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.123,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.78900000000000003,"15min":0.79700000000000004},"min":{"1min":0.52100000000000002,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.98399999999999999,"5min":2.302,"15min":2.302},"last":0.68000000000000005}]},{"osd":5,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.86899999999999999,"15min":0.88300000000000001},"min":{"1min":0.55100000000000005,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.0820000000000001,"5min":1.365,"15min":1.365},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.879,"5min":0.88900000000000001,"15min":0.88700000000000001},"min":{"1min":0.59199999999999997,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.2250000000000001,"5min":2.0979999999999999,"15min":2.0979999999999999},"last":0.80800000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574895,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36772,"kb_used_data":1560,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335068,"statfs":{"total":96636764160,"available":96599109632,"internally_reserved":0,"allocated":1597440,"data_stored":1191741,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.59299999999999997,"15min":0.60699999999999998},"min":{"1min":0.41399999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.76400000000000001,"5min":1.113,"15min":1.113},"last":0.503},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.61599999999999999,"15min":0.63800000000000001},"min":{"1min":0.41399999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.85199999999999998,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.67200000000000004}]},{"osd":2,"last update":"Wed Sep 18 04:40:47 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.53300000000000003}]},{"osd":3,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.82099999999999995,"15min":0.81999999999999995},"min":{"1min":0.64100000000000001,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.80800000000000005,"15min":0.82199999999999995},"min":{"1min":0.66000000000000003,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.97499999999999998,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":1.0960000000000001}]},{"osd":4,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":0.879,"15min":0.86499999999999999},"min":{"1min":0.66400000000000003,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":3.3740000000000001,"5min":3.3740000000000001,"15min":3.3740000000000001},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.89400000000000002,"15min":0.90700000000000003},"min":{"1min":0.66400000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.117,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.93999999999999995}]},{"osd":5,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.89000000000000001,"15min":0.92200000000000004},"min":{"1min":0.66900000000000004,"5min":0.496,"15min":0.496},"max":{"1min":1.006,"5min":1.8919999999999999,"15min":1.8919999999999999},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.94899999999999995,"15min":0.96999999999999997},"min":{"1min":0.69599999999999995,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.2370000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":0.96999999999999997}]}]},{"osd":3,"up_from":12,"seq":51539607598,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32680,"kb_used_data":1560,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339160,"statfs":{"total":96636764160,"available":96603299840,"internally_reserved":0,"allocated":1597440,"data_stored":1191741,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.88700000000000001,"15min":0.93799999999999994},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.145,"5min":2.665,"15min":2.665},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.83999999999999997,"15min":0.86799999999999999},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.70099999999999996}]},{"osd":1,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.82799999999999996,"15min":0.872},"min":{"1min":0.55500000000000005,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.93799999999999994,"5min":2.4510000000000001,"15min":2.4510000000000001},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.76000000000000001,"15min":0.78200000000000003},"min":{"1min":0.57799999999999996,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.93600000000000005,"5min":1.077,"15min":1.077},"last":0.66100000000000003}]},{"osd":2,"last update":"Wed Sep 18 04:40:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.1970000000000001,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.621}]},{"osd":4,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.71399999999999997,"15min":0.72299999999999998},"min":{"1min":0.46999999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.91700000000000004,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.76000000000000001,"15min":0.752},"min":{"1min":0.44500000000000001,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.302,"5min":1.302,"15min":1.302},"last":0.88300000000000001}]},{"osd":5,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.74399999999999999,"15min":0.74199999999999999},"min":{"1min":0.60699999999999998,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":0.89300000000000002,"5min":1.173,"15min":1.173},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.88,"15min":0.872},"min":{"1min":0.72799999999999998,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.1779999999999999,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.83899999999999997}]}]},{"osd":2,"up_from":86,"seq":369367187471,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30120,"kb_used_data":1496,"kb_used_omap":9,"kb_used_meta":28598,"kb_avail":94341720,"statfs":{"total":96636764160,"available":96605921280,"internally_reserved":0,"allocated":1531904,"data_stored":601373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10134,"internal_metadata":29284458},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:40:47 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.5069999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":2.2010000000000001,"5min":2.2010000000000001,"15min":2.2010000000000001},"last":0.70099999999999996}]},{"osd":1,"last update":"Wed Sep 18 04:40:47 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":2.7730000000000001,"5min":2.7730000000000001,"15min":2.7730000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.6539999999999999,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":0.53200000000000003}]},{"osd":3,"last update":"Wed Sep 18 04:40:47 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":1.8959999999999999,"5min":1.8959999999999999,"15min":1.8959999999999999},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"min":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":2.1669999999999998,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":0.751}]},{"osd":4,"last update":"Wed Sep 18 04:40:47 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":2.2160000000000002,"5min":2.2160000000000002,"15min":2.2160000000000002},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":2.3420000000000001,"5min":2.3420000000000001,"15min":2.3420000000000001},"last":0.89100000000000001}]},{"osd":5,"last update":"Wed Sep 18 04:40:47 2024","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":2.4199999999999999,"5min":2.4199999999999999,"15min":2.4199999999999999},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":2.0270000000000001,"5min":2.0270000000000001,"15min":2.0270000000000001},"last":0.78000000000000003}]}]},{"osd":1,"up_from":12,"seq":51539607598,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36712,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335128,"statfs":{"total":96636764160,"available":96599171072,"internally_reserved":0,"allocated":1536000,"data_stored":601392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:40:27 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.59199999999999997,"15min":0.59099999999999997},"min":{"1min":0.42099999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.746,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.61299999999999999,"15min":0.623},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.40200000000000002}]},{"osd":2,"last update":"Wed Sep 18 04:40:48 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.439},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.499}]},{"osd":3,"last update":"Wed Sep 18 04:40:27 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.80100000000000005},"min":{"1min":0.52300000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.85399999999999998,"15min":0.86599999999999999},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.01,"5min":1.194,"15min":1.194},"last":0.70899999999999996}]},{"osd":4,"last update":"Wed Sep 18 04:40:27 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.83199999999999996,"15min":0.83799999999999997},"min":{"1min":0.63500000000000001,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.121,"5min":1.236,"15min":1.236},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.90700000000000003,"15min":0.92300000000000004},"min":{"1min":0.64300000000000002,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.0720000000000001,"5min":1.268,"15min":1.268},"last":0.75}]},{"osd":5,"last update":"Wed Sep 18 04:40:27 2024","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.88800000000000001,"15min":0.90400000000000003},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.1479999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.91000000000000003,"15min":0.92200000000000004},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.1879999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.58799999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3627,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T04:40:51.686 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T04:40:51.989 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:40:51.990 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":129,"fsid":"be6146ba-a991-4f6e-b359-99cbedc9f9b1","created":"2024-09-18T04:37:15.676634+0000","modified":"2024-09-18T04:40:45.550211+0000","last_up_change":"2024-09-18T04:39:45.699693+0000","last_in_change":"2024-09-18T04:40:04.812503+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-18T04:37:25.654950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-18T04:37:32.672462+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":13,"snap_epoch":110,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2.2699999809265137,"score_stable":2.2699999809265137,"optimal_score":0.82999998331069946,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6177002d-72e6-4888-bcde-29b6a92d9dc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6801","nonce":1554407851},{"type":"v1","addr":"172.21.15.28:6803","nonce":1554407851}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6805","nonce":1554407851},{"type":"v1","addr":"172.21.15.28:6807","nonce":1554407851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6813","nonce":1554407851},{"type":"v1","addr":"172.21.15.28:6815","nonce":1554407851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":1554407851},{"type":"v1","addr":"172.21.15.28:6811","nonce":1554407851}]},"public_addr":"172.21.15.28:6803/1554407851","cluster_addr":"172.21.15.28:6807/1554407851","heartbeat_back_addr":"172.21.15.28:6815/1554407851","heartbeat_front_addr":"172.21.15.28:6811/1554407851","state":["exists","up"]},{"osd":1,"uuid":"4f62cb09-3457-418c-bb69-491d9b70914c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":511141401},{"type":"v1","addr":"172.21.15.28:6802","nonce":511141401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":511141401},{"type":"v1","addr":"172.21.15.28:6806","nonce":511141401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":511141401},{"type":"v1","addr":"172.21.15.28:6814","nonce":511141401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":511141401},{"type":"v1","addr":"172.21.15.28:6810","nonce":511141401}]},"public_addr":"172.21.15.28:6802/511141401","cluster_addr":"172.21.15.28:6806/511141401","heartbeat_back_addr":"172.21.15.28:6814/511141401","heartbeat_front_addr":"172.21.15.28:6810/511141401","state":["exists","up"]},{"osd":2,"uuid":"854fe031-3b87-4290-9c2f-3fb196fb806c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":12,"last_clean_end":76,"up_from":86,"up_thru":92,"down_at":77,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3849236161},{"type":"v1","addr":"172.21.15.28:6817","nonce":3849236161}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":3849236161},{"type":"v1","addr":"172.21.15.28:6819","nonce":3849236161}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":3849236161},{"type":"v1","addr":"172.21.15.28:6823","nonce":3849236161}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3849236161},{"type":"v1","addr":"172.21.15.28:6821","nonce":3849236161}]},"public_addr":"172.21.15.28:6817/3849236161","cluster_addr":"172.21.15.28:6819/3849236161","heartbeat_back_addr":"172.21.15.28:6823/3849236161","heartbeat_front_addr":"172.21.15.28:6821/3849236161","state":["exists","up"]},{"osd":3,"uuid":"c2625f98-780b-4f86-bf36-a48d2a227756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":2746051132},{"type":"v1","addr":"172.21.15.82:6809","nonce":2746051132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":2746051132},{"type":"v1","addr":"172.21.15.82:6811","nonce":2746051132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":2746051132},{"type":"v1","addr":"172.21.15.82:6815","nonce":2746051132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":2746051132},{"type":"v1","addr":"172.21.15.82:6813","nonce":2746051132}]},"public_addr":"172.21.15.82:6809/2746051132","cluster_addr":"172.21.15.82:6811/2746051132","heartbeat_back_addr":"172.21.15.82:6815/2746051132","heartbeat_front_addr":"172.21.15.82:6813/2746051132","state":["exists","up"]},{"osd":4,"uuid":"deb22d86-4081-484d-96eb-9cb9d96b60bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":2005712460},{"type":"v1","addr":"172.21.15.82:6801","nonce":2005712460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":2005712460},{"type":"v1","addr":"172.21.15.82:6803","nonce":2005712460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":2005712460},{"type":"v1","addr":"172.21.15.82:6807","nonce":2005712460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":2005712460},{"type":"v1","addr":"172.21.15.82:6805","nonce":2005712460}]},"public_addr":"172.21.15.82:6801/2005712460","cluster_addr":"172.21.15.82:6803/2005712460","heartbeat_back_addr":"172.21.15.82:6807/2005712460","heartbeat_front_addr":"172.21.15.82:6805/2005712460","state":["exists","up"]},{"osd":5,"uuid":"061d332c-f51b-403d-bb18-eba415c97e7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":1267644232},{"type":"v1","addr":"172.21.15.82:6817","nonce":1267644232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":1267644232},{"type":"v1","addr":"172.21.15.82:6819","nonce":1267644232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":1267644232},{"type":"v1","addr":"172.21.15.82:6823","nonce":1267644232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":1267644232},{"type":"v1","addr":"172.21.15.82:6821","nonce":1267644232}]},"public_addr":"172.21.15.82:6817/1267644232","cluster_addr":"172.21.15.82:6819/1267644232","heartbeat_back_addr":"172.21.15.82:6823/1267644232","heartbeat_front_addr":"172.21.15.82:6821/1267644232","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T04:37:22.192087+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T04:37:22.017833+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T04:37:22.755774+0000","dead_epoch":77},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T04:37:22.663785+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T04:37:22.531758+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T04:37:22.400774+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.5","osds":[2,0]},{"pgid":"2.7","osds":[3,5]}],"pg_upmap_items":[{"pgid":"2.1","mappings":[{"from":2,"to":1}]},{"pgid":"2.4","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T04:40:53.011 INFO:tasks.ceph:Scrubbing osd.0 2024-09-18T04:40:53.011 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-18T04:40:53.178 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:40:53.178 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-18T04:40:53.178 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:40:53.193 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-18T04:40:53.518 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 0 to deep-scrub 2024-09-18T04:40:53.529 INFO:tasks.ceph:Scrubbing osd.1 2024-09-18T04:40:53.529 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-18T04:40:53.681 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:40:53.682 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-18T04:40:53.682 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:40:53.694 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-18T04:40:53.999 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 1 to deep-scrub 2024-09-18T04:40:54.014 INFO:tasks.ceph:Scrubbing osd.2 2024-09-18T04:40:54.014 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-18T04:40:54.165 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:40:54.166 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-18T04:40:54.166 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:40:54.184 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-18T04:40:54.496 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 2 to deep-scrub 2024-09-18T04:40:54.507 INFO:tasks.ceph:Scrubbing osd.3 2024-09-18T04:40:54.508 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-18T04:40:54.660 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:40:54.660 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-18T04:40:54.661 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:40:54.673 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-18T04:40:54.987 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 3 to deep-scrub 2024-09-18T04:40:54.998 INFO:tasks.ceph:Scrubbing osd.4 2024-09-18T04:40:54.998 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-18T04:40:55.152 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:40:55.152 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-18T04:40:55.152 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:40:55.166 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-18T04:40:55.474 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 4 to deep-scrub 2024-09-18T04:40:55.485 INFO:tasks.ceph:Scrubbing osd.5 2024-09-18T04:40:55.485 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-18T04:40:55.632 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-18T04:40:55.632 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-18T04:40:55.632 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-18T04:40:55.645 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-18T04:40:55.958 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 5 to deep-scrub 2024-09-18T04:40:55.969 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T04:40:56.273 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:40:56.275 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-18T04:40:56.288 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":229,"stamp":"2024-09-18T04:40:55.647395+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8601,"num_read_kb":183296,"num_write":1123,"num_write_kb":273318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":39960576,"num_keys_recovered":25,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1214,"ondisk_log_size":1214,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":5,"kb":566231040,"kb_used":201512,"kb_used_data":9112,"kb_used_omap":17,"kb_used_meta":192302,"kb_avail":566029528,"statfs":{"total":579820584960,"available":579614236672,"internally_reserved":0,"allocated":9330688,"data_stored":4789012,"data_compressed":66040,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":18101,"internal_metadata":196917579},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.091380"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":273,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T04:40:45.558841+0000","last_change":"2024-09-18T04:37:27.423069+0000","last_active":"2024-09-18T04:40:45.558841+0000","last_peered":"2024-09-18T04:40:45.558841+0000","last_clean":"2024-09-18T04:40:45.558841+0000","last_became_active":"2024-09-18T04:37:27.422350+0000","last_became_peered":"2024-09-18T04:37:27.422350+0000","last_unstale":"2024-09-18T04:40:45.558841+0000","last_undegraded":"2024-09-18T04:40:45.558841+0000","last_fullsized":"2024-09-18T04:40:45.558841+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:26.406292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:26.406292+0000","last_clean_scrub_stamp":"2024-09-18T04:37:26.406292+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"110'170","reported_seq":807,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T04:40:44.720562+0000","last_change":"2024-09-18T04:40:22.947240+0000","last_active":"2024-09-18T04:40:44.720562+0000","last_peered":"2024-09-18T04:40:44.720562+0000","last_clean":"2024-09-18T04:40:44.720562+0000","last_became_active":"2024-09-18T04:39:46.718066+0000","last_became_peered":"2024-09-18T04:39:46.718066+0000","last_unstale":"2024-09-18T04:40:44.720562+0000","last_undegraded":"2024-09-18T04:40:44.720562+0000","last_fullsized":"2024-09-18T04:40:44.720562+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T07:09:22.599338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011466040000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":748,"num_read_kb":17282,"num_write":161,"num_write_kb":25730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":9486336,"num_keys_recovered":9,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"110'255","reported_seq":1475,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T04:40:45.558871+0000","last_change":"2024-09-18T04:40:22.949604+0000","last_active":"2024-09-18T04:40:45.558871+0000","last_peered":"2024-09-18T04:40:45.558871+0000","last_clean":"2024-09-18T04:40:45.558871+0000","last_became_active":"2024-09-18T04:37:34.470841+0000","last_became_peered":"2024-09-18T04:37:34.470841+0000","last_unstale":"2024-09-18T04:40:45.558871+0000","last_undegraded":"2024-09-18T04:40:45.558871+0000","last_fullsized":"2024-09-18T04:40:45.558871+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":255,"log_dups_size":0,"ondisk_log_size":255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052682599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1817,"num_read_kb":40557,"num_write":337,"num_write_kb":45342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"110'317","reported_seq":2384,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T04:40:46.827223+0000","last_change":"2024-09-18T04:40:22.942796+0000","last_active":"2024-09-18T04:40:46.827223+0000","last_peered":"2024-09-18T04:40:46.827223+0000","last_clean":"2024-09-18T04:40:46.827223+0000","last_became_active":"2024-09-18T04:39:46.716998+0000","last_became_peered":"2024-09-18T04:39:46.716998+0000","last_unstale":"2024-09-18T04:40:46.827223+0000","last_undegraded":"2024-09-18T04:40:46.827223+0000","last_fullsized":"2024-09-18T04:40:46.827223+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":317,"log_dups_size":0,"ondisk_log_size":317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001074104,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4741,"num_read_kb":31642,"num_write":206,"num_write_kb":27778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":5308416,"num_keys_recovered":13,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"20'2","reported_seq":286,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T04:40:44.720539+0000","last_change":"2024-09-18T04:40:22.947782+0000","last_active":"2024-09-18T04:40:44.720539+0000","last_peered":"2024-09-18T04:40:44.720539+0000","last_clean":"2024-09-18T04:40:44.720539+0000","last_became_active":"2024-09-18T04:37:34.473088+0000","last_became_peered":"2024-09-18T04:37:34.473088+0000","last_unstale":"2024-09-18T04:40:44.720539+0000","last_undegraded":"2024-09-18T04:40:44.720539+0000","last_fullsized":"2024-09-18T04:40:44.720539+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:17:57.352397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001752392,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":139,"num_read_kb":139,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"110'65","reported_seq":162,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T04:40:44.720562+0000","last_change":"2024-09-18T04:40:22.947195+0000","last_active":"2024-09-18T04:40:44.720562+0000","last_peered":"2024-09-18T04:40:44.720562+0000","last_clean":"2024-09-18T04:40:44.720562+0000","last_became_active":"2024-09-18T04:38:59.462279+0000","last_became_peered":"2024-09-18T04:38:59.462279+0000","last_unstale":"2024-09-18T04:40:44.720562+0000","last_undegraded":"2024-09-18T04:40:44.720562+0000","last_fullsized":"2024-09-18T04:40:44.720562+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:32:58.894411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011357990000000001,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":101,"num_read_kb":16665,"num_write":51,"num_write_kb":36867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":12582912,"num_keys_recovered":3,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"110'89","reported_seq":463,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T04:40:46.827225+0000","last_change":"2024-09-18T04:40:22.942807+0000","last_active":"2024-09-18T04:40:46.827225+0000","last_peered":"2024-09-18T04:40:46.827225+0000","last_clean":"2024-09-18T04:40:46.827225+0000","last_became_active":"2024-09-18T04:39:46.717423+0000","last_became_peered":"2024-09-18T04:39:46.717423+0000","last_unstale":"2024-09-18T04:40:46.827225+0000","last_undegraded":"2024-09-18T04:40:46.827225+0000","last_fullsized":"2024-09-18T04:40:46.827225+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086846899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":248,"num_read_kb":31406,"num_write":66,"num_write_kb":49154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"110'100","reported_seq":387,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T04:40:54.447303+0000","last_change":"2024-09-18T04:40:54.447675+0000","last_active":"2024-09-18T04:40:54.447303+0000","last_peered":"2024-09-18T04:40:54.447303+0000","last_clean":"2024-09-18T04:40:54.447303+0000","last_became_active":"2024-09-18T04:37:34.472940+0000","last_became_peered":"2024-09-18T04:37:34.472940+0000","last_unstale":"2024-09-18T04:40:54.447303+0000","last_undegraded":"2024-09-18T04:40:54.447303+0000","last_fullsized":"2024-09-18T04:40:54.447303+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'100","last_scrub_stamp":"2024-09-18T04:40:54.447172+0000","last_deep_scrub":"110'100","last_deep_scrub_stamp":"2024-09-18T04:40:54.447172+0000","last_clean_scrub_stamp":"2024-09-18T04:40:54.447172+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:18:13.300712+0000","scrub_duration":0.0039970709999999996,"objects_trimmed":0,"snaptrim_duration":0.0012195979999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":146,"num_read_kb":28719,"num_write":77,"num_write_kb":57349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"110'99","reported_seq":509,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T04:40:45.558917+0000","last_change":"2024-09-18T04:40:22.952155+0000","last_active":"2024-09-18T04:40:45.558917+0000","last_peered":"2024-09-18T04:40:45.558917+0000","last_clean":"2024-09-18T04:40:45.558917+0000","last_became_active":"2024-09-18T04:38:05.753962+0000","last_became_peered":"2024-09-18T04:38:05.753962+0000","last_unstale":"2024-09-18T04:40:45.558917+0000","last_undegraded":"2024-09-18T04:40:45.558917+0000","last_fullsized":"2024-09-18T04:40:45.558917+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T04:37:33.445423+0000","last_clean_scrub_stamp":"2024-09-18T04:37:33.445423+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030414399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":545,"num_read_kb":16786,"num_write":64,"num_write_kb":28680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8485,"num_read_kb":183196,"num_write":964,"num_write_kb":270902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":39960576,"num_keys_recovered":25,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3627,"internal_metadata":0},"log_size":1097,"ondisk_log_size":1097,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":66040,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":13,"seq":55834574894,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32616,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339224,"statfs":{"total":96636764160,"available":96603365376,"internally_reserved":0,"allocated":1536000,"data_stored":601392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.82799999999999996,"15min":0.85799999999999998},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.1060000000000001,"5min":1.5249999999999999,"15min":1.5249999999999999},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.77700000000000002,"15min":0.80100000000000005},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":0.96099999999999997,"5min":1.369,"15min":1.369},"last":0.83699999999999997}]},{"osd":1,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.82099999999999995,"15min":0.84599999999999997},"min":{"1min":0.59099999999999997,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":0.997,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.78600000000000003,"15min":0.79600000000000004},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.628}]},{"osd":2,"last update":"Wed Sep 18 04:40:49 2024","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.278,"5min":1.278,"15min":1.278},"last":0.91100000000000003}]},{"osd":3,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.73899999999999999,"15min":0.76400000000000001},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.90500000000000003,"5min":1.22,"15min":1.22},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.78100000000000003,"15min":0.79400000000000004},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.1819999999999999,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.86699999999999999}]},{"osd":4,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.77000000000000002,"15min":0.78200000000000003},"min":{"1min":0.45400000000000001,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.544,"5min":1.544,"15min":1.544},"last":0.753},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.80400000000000005,"15min":0.82999999999999996},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.5109999999999999,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":0.71999999999999997}]}]},{"osd":4,"up_from":13,"seq":55834574894,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32612,"kb_used_data":1496,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339228,"statfs":{"total":96636764160,"available":96603369472,"internally_reserved":0,"allocated":1531904,"data_stored":601373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.747,"15min":0.76600000000000001},"min":{"1min":0.47899999999999998,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.94799999999999995,"5min":1.175,"15min":1.175},"last":0.96199999999999997},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.77100000000000002,"15min":0.79000000000000004},"min":{"1min":0.60499999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.98999999999999999,"5min":1.258,"15min":1.258},"last":0.86899999999999999}]},{"osd":1,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.71599999999999997,"15min":0.72599999999999998},"min":{"1min":0.51000000000000001,"5min":0.438,"15min":0.438},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.84899999999999998,"15min":0.875},"min":{"1min":0.49399999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.1259999999999999,"5min":1.153,"15min":1.153},"last":1.0049999999999999}]},{"osd":2,"last update":"Wed Sep 18 04:40:46 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1,"5min":1,"15min":1},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.87,"5min":0.87,"15min":0.87},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.113,"5min":1.113,"15min":1.113},"last":0.91500000000000004}]},{"osd":3,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.79000000000000004,"15min":0.81499999999999995},"min":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.123,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.78900000000000003,"15min":0.79700000000000004},"min":{"1min":0.52100000000000002,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.98399999999999999,"5min":2.302,"15min":2.302},"last":0.68000000000000005}]},{"osd":5,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.86899999999999999,"15min":0.88300000000000001},"min":{"1min":0.55100000000000005,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.0820000000000001,"5min":1.365,"15min":1.365},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.879,"5min":0.88900000000000001,"15min":0.88700000000000001},"min":{"1min":0.59199999999999997,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.2250000000000001,"5min":2.0979999999999999,"15min":2.0979999999999999},"last":0.80800000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574895,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36772,"kb_used_data":1560,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335068,"statfs":{"total":96636764160,"available":96599109632,"internally_reserved":0,"allocated":1597440,"data_stored":1191741,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.59299999999999997,"15min":0.60699999999999998},"min":{"1min":0.41399999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.76400000000000001,"5min":1.113,"15min":1.113},"last":0.503},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.61599999999999999,"15min":0.63800000000000001},"min":{"1min":0.41399999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.85199999999999998,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.67200000000000004}]},{"osd":2,"last update":"Wed Sep 18 04:40:47 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.53300000000000003}]},{"osd":3,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.82099999999999995,"15min":0.81999999999999995},"min":{"1min":0.64100000000000001,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.80800000000000005,"15min":0.82199999999999995},"min":{"1min":0.66000000000000003,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.97499999999999998,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":1.0960000000000001}]},{"osd":4,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":0.879,"15min":0.86499999999999999},"min":{"1min":0.66400000000000003,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":3.3740000000000001,"5min":3.3740000000000001,"15min":3.3740000000000001},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.89400000000000002,"15min":0.90700000000000003},"min":{"1min":0.66400000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.117,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.93999999999999995}]},{"osd":5,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.89000000000000001,"15min":0.92200000000000004},"min":{"1min":0.66900000000000004,"5min":0.496,"15min":0.496},"max":{"1min":1.006,"5min":1.8919999999999999,"15min":1.8919999999999999},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.94899999999999995,"15min":0.96999999999999997},"min":{"1min":0.69599999999999995,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.2370000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":0.96999999999999997}]}]},{"osd":3,"up_from":12,"seq":51539607599,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32680,"kb_used_data":1560,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339160,"statfs":{"total":96636764160,"available":96603299840,"internally_reserved":0,"allocated":1597440,"data_stored":1191741,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.88700000000000001,"15min":0.93799999999999994},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.145,"5min":2.665,"15min":2.665},"last":0.754},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.83999999999999997,"15min":0.86799999999999999},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.68999999999999995}]},{"osd":1,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.82799999999999996,"15min":0.872},"min":{"1min":0.55500000000000005,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.93799999999999994,"5min":2.4510000000000001,"15min":2.4510000000000001},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.76000000000000001,"15min":0.78200000000000003},"min":{"1min":0.57799999999999996,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.93600000000000005,"5min":1.077,"15min":1.077},"last":0.49199999999999999}]},{"osd":2,"last update":"Wed Sep 18 04:40:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.1970000000000001,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.63500000000000001}]},{"osd":4,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.71399999999999997,"15min":0.72299999999999998},"min":{"1min":0.46999999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.91700000000000004,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.76000000000000001,"15min":0.752},"min":{"1min":0.44500000000000001,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.302,"5min":1.302,"15min":1.302},"last":0.93500000000000005}]},{"osd":5,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.74399999999999999,"15min":0.74199999999999999},"min":{"1min":0.60699999999999998,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":0.89300000000000002,"5min":1.173,"15min":1.173},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.88,"15min":0.872},"min":{"1min":0.72799999999999998,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.1779999999999999,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.78900000000000003}]}]},{"osd":2,"up_from":86,"seq":369367187472,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30120,"kb_used_data":1496,"kb_used_omap":9,"kb_used_meta":28598,"kb_avail":94341720,"statfs":{"total":96636764160,"available":96605921280,"internally_reserved":0,"allocated":1531904,"data_stored":601373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10134,"internal_metadata":29284458},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:40:47 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.5069999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":2.2010000000000001,"5min":2.2010000000000001,"15min":2.2010000000000001},"last":0.54300000000000004}]},{"osd":1,"last update":"Wed Sep 18 04:40:47 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":2.7730000000000001,"5min":2.7730000000000001,"15min":2.7730000000000001},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.6539999999999999,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":0.56399999999999995}]},{"osd":3,"last update":"Wed Sep 18 04:40:47 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":1.8959999999999999,"5min":1.8959999999999999,"15min":1.8959999999999999},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"min":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":2.1669999999999998,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":1.0349999999999999}]},{"osd":4,"last update":"Wed Sep 18 04:40:47 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":2.2160000000000002,"5min":2.2160000000000002,"15min":2.2160000000000002},"last":1.1160000000000001},{"interface":"front","average":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":2.3420000000000001,"5min":2.3420000000000001,"15min":2.3420000000000001},"last":0.77000000000000002}]},{"osd":5,"last update":"Wed Sep 18 04:40:47 2024","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":2.4199999999999999,"5min":2.4199999999999999,"15min":2.4199999999999999},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":2.0270000000000001,"5min":2.0270000000000001,"15min":2.0270000000000001},"last":0.84899999999999998}]}]},{"osd":1,"up_from":12,"seq":51539607599,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36712,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335128,"statfs":{"total":96636764160,"available":96599171072,"internally_reserved":0,"allocated":1536000,"data_stored":601392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:40:27 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.59199999999999997,"15min":0.59099999999999997},"min":{"1min":0.42099999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.746,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.61299999999999999,"15min":0.623},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.64500000000000002}]},{"osd":2,"last update":"Wed Sep 18 04:40:48 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.61199999999999999}]},{"osd":3,"last update":"Wed Sep 18 04:40:27 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.80100000000000005},"min":{"1min":0.52300000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.85399999999999998,"15min":0.86599999999999999},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.01,"5min":1.194,"15min":1.194},"last":0.69799999999999995}]},{"osd":4,"last update":"Wed Sep 18 04:40:27 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.83199999999999996,"15min":0.83799999999999997},"min":{"1min":0.63500000000000001,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.121,"5min":1.236,"15min":1.236},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.90700000000000003,"15min":0.92300000000000004},"min":{"1min":0.64300000000000002,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.0720000000000001,"5min":1.268,"15min":1.268},"last":0.81299999999999994}]},{"osd":5,"last update":"Wed Sep 18 04:40:27 2024","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.88800000000000001,"15min":0.90400000000000003},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.1479999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.92800000000000005},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.91000000000000003,"15min":0.92200000000000004},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.1879999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.88300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3627,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T04:40:56.292 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-18T04:37:26.406292+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=4, tm_min=37, tm_sec=26, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=4, tm_min=40, tm_sec=52, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T04:40:56.292 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-18T04:37:33.445423+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=4, tm_min=37, tm_sec=33, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=4, tm_min=40, tm_sec=52, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T04:40:56.292 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-18T04:37:33.445423+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=4, tm_min=37, tm_sec=33, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=4, tm_min=40, tm_sec=52, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T04:40:56.293 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-18T04:37:33.445423+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=4, tm_min=37, tm_sec=33, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=4, tm_min=40, tm_sec=52, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T04:40:56.293 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-18T04:37:33.445423+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=4, tm_min=37, tm_sec=33, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=4, tm_min=40, tm_sec=52, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T04:40:56.293 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-18T04:37:33.445423+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=4, tm_min=37, tm_sec=33, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=4, tm_min=40, tm_sec=52, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T04:40:56.293 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-18T04:37:33.445423+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=4, tm_min=37, tm_sec=33, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=4, tm_min=40, tm_sec=52, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T04:40:56.293 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-18T04:37:33.445423+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=4, tm_min=37, tm_sec=33, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=4, tm_min=40, tm_sec=52, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T04:40:56.293 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-18T04:41:16.294 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T04:41:16.615 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-18T04:41:16.617 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-18T04:41:16.629 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":239,"stamp":"2024-09-18T04:41:15.652807+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8601,"num_read_kb":183296,"num_write":1123,"num_write_kb":273318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":39960576,"num_keys_recovered":25,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1220,"ondisk_log_size":1220,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":5,"kb":566231040,"kb_used":201484,"kb_used_data":9112,"kb_used_omap":17,"kb_used_meta":192302,"kb_avail":566029556,"statfs":{"total":579820584960,"available":579614265344,"internally_reserved":0,"allocated":9330688,"data_stored":4789012,"data_compressed":66040,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":18101,"internal_metadata":196917579},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003214"},"pg_stats":[{"pgid":"1.0","version":"129'119","reported_seq":286,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T04:40:56.169968+0000","last_change":"2024-09-18T04:40:56.170195+0000","last_active":"2024-09-18T04:40:56.169968+0000","last_peered":"2024-09-18T04:40:56.169968+0000","last_clean":"2024-09-18T04:40:56.169968+0000","last_became_active":"2024-09-18T04:37:27.422350+0000","last_became_peered":"2024-09-18T04:37:27.422350+0000","last_unstale":"2024-09-18T04:40:56.169968+0000","last_undegraded":"2024-09-18T04:40:56.169968+0000","last_fullsized":"2024-09-18T04:40:56.169968+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'119","last_scrub_stamp":"2024-09-18T04:40:56.169919+0000","last_deep_scrub":"129'119","last_deep_scrub_stamp":"2024-09-18T04:40:56.169919+0000","last_clean_scrub_stamp":"2024-09-18T04:40:56.169919+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T09:37:03.858668+0000","scrub_duration":0.0084846880000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"129'171","reported_seq":820,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T04:40:56.139293+0000","last_change":"2024-09-18T04:40:56.139568+0000","last_active":"2024-09-18T04:40:56.139293+0000","last_peered":"2024-09-18T04:40:56.139293+0000","last_clean":"2024-09-18T04:40:56.139293+0000","last_became_active":"2024-09-18T04:39:46.718066+0000","last_became_peered":"2024-09-18T04:39:46.718066+0000","last_unstale":"2024-09-18T04:40:56.139293+0000","last_undegraded":"2024-09-18T04:40:56.139293+0000","last_fullsized":"2024-09-18T04:40:56.139293+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'171","last_scrub_stamp":"2024-09-18T04:40:56.139241+0000","last_deep_scrub":"129'171","last_deep_scrub_stamp":"2024-09-18T04:40:56.139241+0000","last_clean_scrub_stamp":"2024-09-18T04:40:56.139241+0000","objects_scrubbed":1,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:03:44.144478+0000","scrub_duration":0.0066915569999999999,"objects_trimmed":0,"snaptrim_duration":0.0011466040000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":748,"num_read_kb":17282,"num_write":161,"num_write_kb":25730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":9486336,"num_keys_recovered":9,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"110'255","reported_seq":1486,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T04:41:00.147996+0000","last_change":"2024-09-18T04:41:00.148243+0000","last_active":"2024-09-18T04:41:00.147996+0000","last_peered":"2024-09-18T04:41:00.147996+0000","last_clean":"2024-09-18T04:41:00.147996+0000","last_became_active":"2024-09-18T04:37:34.470841+0000","last_became_peered":"2024-09-18T04:37:34.470841+0000","last_unstale":"2024-09-18T04:41:00.147996+0000","last_undegraded":"2024-09-18T04:41:00.147996+0000","last_fullsized":"2024-09-18T04:41:00.147996+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'255","last_scrub_stamp":"2024-09-18T04:41:00.147945+0000","last_deep_scrub":"110'255","last_deep_scrub_stamp":"2024-09-18T04:41:00.147945+0000","last_clean_scrub_stamp":"2024-09-18T04:41:00.147945+0000","objects_scrubbed":0,"log_size":255,"log_dups_size":0,"ondisk_log_size":255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:02:56.743951+0000","scrub_duration":0.0039430940000000003,"objects_trimmed":0,"snaptrim_duration":0.00052682599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1817,"num_read_kb":40557,"num_write":337,"num_write_kb":45342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"110'317","reported_seq":2395,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T04:40:56.484794+0000","last_change":"2024-09-18T04:40:56.485080+0000","last_active":"2024-09-18T04:40:56.484794+0000","last_peered":"2024-09-18T04:40:56.484794+0000","last_clean":"2024-09-18T04:40:56.484794+0000","last_became_active":"2024-09-18T04:39:46.716998+0000","last_became_peered":"2024-09-18T04:39:46.716998+0000","last_unstale":"2024-09-18T04:40:56.484794+0000","last_undegraded":"2024-09-18T04:40:56.484794+0000","last_fullsized":"2024-09-18T04:40:56.484794+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'317","last_scrub_stamp":"2024-09-18T04:40:56.484743+0000","last_deep_scrub":"110'317","last_deep_scrub_stamp":"2024-09-18T04:40:56.484743+0000","last_clean_scrub_stamp":"2024-09-18T04:40:56.484743+0000","objects_scrubbed":0,"log_size":317,"log_dups_size":0,"ondisk_log_size":317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:24:39.674020+0000","scrub_duration":0.0037338800000000002,"objects_trimmed":0,"snaptrim_duration":0.001074104,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4741,"num_read_kb":31642,"num_write":206,"num_write_kb":27778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":5308416,"num_keys_recovered":13,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"129'3","reported_seq":299,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T04:40:59.107370+0000","last_change":"2024-09-18T04:40:59.107608+0000","last_active":"2024-09-18T04:40:59.107370+0000","last_peered":"2024-09-18T04:40:59.107370+0000","last_clean":"2024-09-18T04:40:59.107370+0000","last_became_active":"2024-09-18T04:37:34.473088+0000","last_became_peered":"2024-09-18T04:37:34.473088+0000","last_unstale":"2024-09-18T04:40:59.107370+0000","last_undegraded":"2024-09-18T04:40:59.107370+0000","last_fullsized":"2024-09-18T04:40:59.107370+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'3","last_scrub_stamp":"2024-09-18T04:40:59.107321+0000","last_deep_scrub":"129'3","last_deep_scrub_stamp":"2024-09-18T04:40:59.107321+0000","last_clean_scrub_stamp":"2024-09-18T04:40:59.107321+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:21:39.843582+0000","scrub_duration":0.0072788769999999996,"objects_trimmed":0,"snaptrim_duration":0.001752392,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":139,"num_read_kb":139,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"129'67","reported_seq":177,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T04:41:05.166067+0000","last_change":"2024-09-18T04:41:05.166067+0000","last_active":"2024-09-18T04:41:05.166067+0000","last_peered":"2024-09-18T04:41:05.166067+0000","last_clean":"2024-09-18T04:41:05.166067+0000","last_became_active":"2024-09-18T04:38:59.462279+0000","last_became_peered":"2024-09-18T04:38:59.462279+0000","last_unstale":"2024-09-18T04:41:05.166067+0000","last_undegraded":"2024-09-18T04:41:05.166067+0000","last_fullsized":"2024-09-18T04:41:05.166067+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'67","last_scrub_stamp":"2024-09-18T04:41:05.165747+0000","last_deep_scrub":"129'67","last_deep_scrub_stamp":"2024-09-18T04:41:05.165747+0000","last_clean_scrub_stamp":"2024-09-18T04:41:05.165747+0000","objects_scrubbed":2,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:28:50.947607+0000","scrub_duration":0.0076497609999999997,"objects_trimmed":0,"snaptrim_duration":0.0011357990000000001,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":101,"num_read_kb":16665,"num_write":51,"num_write_kb":36867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":12582912,"num_keys_recovered":3,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"110'89","reported_seq":474,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T04:40:57.469313+0000","last_change":"2024-09-18T04:40:57.469561+0000","last_active":"2024-09-18T04:40:57.469313+0000","last_peered":"2024-09-18T04:40:57.469313+0000","last_clean":"2024-09-18T04:40:57.469313+0000","last_became_active":"2024-09-18T04:39:46.717423+0000","last_became_peered":"2024-09-18T04:39:46.717423+0000","last_unstale":"2024-09-18T04:40:57.469313+0000","last_undegraded":"2024-09-18T04:40:57.469313+0000","last_fullsized":"2024-09-18T04:40:57.469313+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'89","last_scrub_stamp":"2024-09-18T04:40:57.469262+0000","last_deep_scrub":"110'89","last_deep_scrub_stamp":"2024-09-18T04:40:57.469262+0000","last_clean_scrub_stamp":"2024-09-18T04:40:57.469262+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T09:44:27.111539+0000","scrub_duration":0.0044293170000000003,"objects_trimmed":0,"snaptrim_duration":0.00086846899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":248,"num_read_kb":31406,"num_write":66,"num_write_kb":49154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"110'100","reported_seq":387,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T04:40:54.447303+0000","last_change":"2024-09-18T04:40:54.447675+0000","last_active":"2024-09-18T04:40:54.447303+0000","last_peered":"2024-09-18T04:40:54.447303+0000","last_clean":"2024-09-18T04:40:54.447303+0000","last_became_active":"2024-09-18T04:37:34.472940+0000","last_became_peered":"2024-09-18T04:37:34.472940+0000","last_unstale":"2024-09-18T04:40:54.447303+0000","last_undegraded":"2024-09-18T04:40:54.447303+0000","last_fullsized":"2024-09-18T04:40:54.447303+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'100","last_scrub_stamp":"2024-09-18T04:40:54.447172+0000","last_deep_scrub":"110'100","last_deep_scrub_stamp":"2024-09-18T04:40:54.447172+0000","last_clean_scrub_stamp":"2024-09-18T04:40:54.447172+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:18:13.300712+0000","scrub_duration":0.0039970709999999996,"objects_trimmed":0,"snaptrim_duration":0.0012195979999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":146,"num_read_kb":28719,"num_write":77,"num_write_kb":57349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"110'99","reported_seq":520,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T04:40:57.157689+0000","last_change":"2024-09-18T04:40:57.158060+0000","last_active":"2024-09-18T04:40:57.157689+0000","last_peered":"2024-09-18T04:40:57.157689+0000","last_clean":"2024-09-18T04:40:57.157689+0000","last_became_active":"2024-09-18T04:38:05.753962+0000","last_became_peered":"2024-09-18T04:38:05.753962+0000","last_unstale":"2024-09-18T04:40:57.157689+0000","last_undegraded":"2024-09-18T04:40:57.157689+0000","last_fullsized":"2024-09-18T04:40:57.157689+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'99","last_scrub_stamp":"2024-09-18T04:40:57.157640+0000","last_deep_scrub":"110'99","last_deep_scrub_stamp":"2024-09-18T04:40:57.157640+0000","last_clean_scrub_stamp":"2024-09-18T04:40:57.157640+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:42:28.727075+0000","scrub_duration":0.0033977600000000001,"objects_trimmed":0,"snaptrim_duration":0.0030414399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":545,"num_read_kb":16786,"num_write":64,"num_write_kb":28680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8485,"num_read_kb":183196,"num_write":964,"num_write_kb":270902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":39960576,"num_keys_recovered":25,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3627,"internal_metadata":0},"log_size":1101,"ondisk_log_size":1101,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":66040,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":13,"seq":55834574898,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32612,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339228,"statfs":{"total":96636764160,"available":96603369472,"internally_reserved":0,"allocated":1536000,"data_stored":601392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.82799999999999996,"15min":0.85799999999999998},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.1060000000000001,"5min":1.5249999999999999,"15min":1.5249999999999999},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.77700000000000002,"15min":0.80100000000000005},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":0.96099999999999997,"5min":1.369,"15min":1.369},"last":0.91600000000000004}]},{"osd":1,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.82099999999999995,"15min":0.84599999999999997},"min":{"1min":0.59099999999999997,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":0.997,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.99099999999999999},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.78600000000000003,"15min":0.79600000000000004},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.70199999999999996}]},{"osd":2,"last update":"Wed Sep 18 04:40:49 2024","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.94299999999999995},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.278,"5min":1.278,"15min":1.278},"last":0.96999999999999997}]},{"osd":3,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.73899999999999999,"15min":0.76400000000000001},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.90500000000000003,"5min":1.22,"15min":1.22},"last":0.623},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.78100000000000003,"15min":0.79400000000000004},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.1819999999999999,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.86399999999999999}]},{"osd":4,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.77000000000000002,"15min":0.78200000000000003},"min":{"1min":0.45400000000000001,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.544,"5min":1.544,"15min":1.544},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.80400000000000005,"15min":0.82999999999999996},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.5109999999999999,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":0.73699999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574898,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32612,"kb_used_data":1496,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339228,"statfs":{"total":96636764160,"available":96603369472,"internally_reserved":0,"allocated":1531904,"data_stored":601373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.747,"15min":0.76600000000000001},"min":{"1min":0.47899999999999998,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.94799999999999995,"5min":1.175,"15min":1.175},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.77100000000000002,"15min":0.79000000000000004},"min":{"1min":0.60499999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.98999999999999999,"5min":1.258,"15min":1.258},"last":0.58499999999999996}]},{"osd":1,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.71599999999999997,"15min":0.72599999999999998},"min":{"1min":0.51000000000000001,"5min":0.438,"15min":0.438},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.84899999999999998,"15min":0.875},"min":{"1min":0.49399999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.1259999999999999,"5min":1.153,"15min":1.153},"last":0.97099999999999997}]},{"osd":2,"last update":"Wed Sep 18 04:40:46 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1,"5min":1,"15min":1},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.87,"5min":0.87,"15min":0.87},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.113,"5min":1.113,"15min":1.113},"last":0.77600000000000002}]},{"osd":3,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.79000000000000004,"15min":0.81499999999999995},"min":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.123,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.78900000000000003,"15min":0.79700000000000004},"min":{"1min":0.52100000000000002,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.98399999999999999,"5min":2.302,"15min":2.302},"last":0.92700000000000005}]},{"osd":5,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.86899999999999999,"15min":0.88300000000000001},"min":{"1min":0.55100000000000005,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.0820000000000001,"5min":1.365,"15min":1.365},"last":0.871},{"interface":"front","average":{"1min":0.879,"5min":0.88900000000000001,"15min":0.88700000000000001},"min":{"1min":0.59199999999999997,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.2250000000000001,"5min":2.0979999999999999,"15min":2.0979999999999999},"last":0.83799999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574899,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36768,"kb_used_data":1560,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335072,"statfs":{"total":96636764160,"available":96599113728,"internally_reserved":0,"allocated":1597440,"data_stored":1191741,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.59299999999999997,"15min":0.60699999999999998},"min":{"1min":0.41399999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.76400000000000001,"5min":1.113,"15min":1.113},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.61599999999999999,"15min":0.63800000000000001},"min":{"1min":0.41399999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.85199999999999998,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.54500000000000004}]},{"osd":2,"last update":"Wed Sep 18 04:40:47 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.50800000000000001}]},{"osd":3,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.82099999999999995,"15min":0.81999999999999995},"min":{"1min":0.64100000000000001,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.93000000000000005},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.80800000000000005,"15min":0.82199999999999995},"min":{"1min":0.66000000000000003,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.97499999999999998,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.74399999999999999}]},{"osd":4,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":0.879,"15min":0.86499999999999999},"min":{"1min":0.66400000000000003,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":3.3740000000000001,"5min":3.3740000000000001,"15min":3.3740000000000001},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.89400000000000002,"15min":0.90700000000000003},"min":{"1min":0.66400000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.117,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.83299999999999996}]},{"osd":5,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.89000000000000001,"15min":0.92200000000000004},"min":{"1min":0.66900000000000004,"5min":0.496,"15min":0.496},"max":{"1min":1.006,"5min":1.8919999999999999,"15min":1.8919999999999999},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.94899999999999995,"15min":0.96999999999999997},"min":{"1min":0.69599999999999995,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.2370000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":0.879}]}]},{"osd":3,"up_from":12,"seq":51539607604,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32672,"kb_used_data":1560,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339168,"statfs":{"total":96636764160,"available":96603308032,"internally_reserved":0,"allocated":1597440,"data_stored":1191741,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.88700000000000001,"15min":0.93799999999999994},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.145,"5min":2.665,"15min":2.665},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.83999999999999997,"15min":0.86799999999999999},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.75}]},{"osd":1,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.82799999999999996,"15min":0.872},"min":{"1min":0.55500000000000005,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.93799999999999994,"5min":2.4510000000000001,"15min":2.4510000000000001},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.76000000000000001,"15min":0.78200000000000003},"min":{"1min":0.57799999999999996,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.93600000000000005,"5min":1.077,"15min":1.077},"last":0.72899999999999998}]},{"osd":2,"last update":"Wed Sep 18 04:40:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.1970000000000001,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.86499999999999999}]},{"osd":4,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.71399999999999997,"15min":0.72299999999999998},"min":{"1min":0.46999999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.91700000000000004,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.501},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.76000000000000001,"15min":0.752},"min":{"1min":0.44500000000000001,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.302,"5min":1.302,"15min":1.302},"last":0.68300000000000005}]},{"osd":5,"last update":"Wed Sep 18 04:40:31 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.74399999999999999,"15min":0.74199999999999999},"min":{"1min":0.60699999999999998,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":0.89300000000000002,"5min":1.173,"15min":1.173},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.88,"15min":0.872},"min":{"1min":0.72799999999999998,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.1779999999999999,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.82799999999999996}]}]},{"osd":2,"up_from":86,"seq":369367187476,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30112,"kb_used_data":1496,"kb_used_omap":9,"kb_used_meta":28598,"kb_avail":94341728,"statfs":{"total":96636764160,"available":96605929472,"internally_reserved":0,"allocated":1531904,"data_stored":601373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10134,"internal_metadata":29284458},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:40:47 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.5069999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":2.2010000000000001,"5min":2.2010000000000001,"15min":2.2010000000000001},"last":0.625}]},{"osd":1,"last update":"Wed Sep 18 04:40:47 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":2.7730000000000001,"5min":2.7730000000000001,"15min":2.7730000000000001},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.6539999999999999,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":0.53600000000000003}]},{"osd":3,"last update":"Wed Sep 18 04:40:47 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":1.8959999999999999,"5min":1.8959999999999999,"15min":1.8959999999999999},"last":0.94499999999999995},{"interface":"front","average":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"min":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":2.1669999999999998,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":0.89600000000000002}]},{"osd":4,"last update":"Wed Sep 18 04:40:47 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":2.2160000000000002,"5min":2.2160000000000002,"15min":2.2160000000000002},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":2.3420000000000001,"5min":2.3420000000000001,"15min":2.3420000000000001},"last":0.91400000000000003}]},{"osd":5,"last update":"Wed Sep 18 04:40:47 2024","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":2.4199999999999999,"5min":2.4199999999999999,"15min":2.4199999999999999},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":2.0270000000000001,"5min":2.0270000000000001,"15min":2.0270000000000001},"last":0.84999999999999998}]}]},{"osd":1,"up_from":12,"seq":51539607603,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36708,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335132,"statfs":{"total":96636764160,"available":96599175168,"internally_reserved":0,"allocated":1536000,"data_stored":601392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 04:40:27 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.59199999999999997,"15min":0.59099999999999997},"min":{"1min":0.42099999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.746,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.61299999999999999,"15min":0.623},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.502}]},{"osd":2,"last update":"Wed Sep 18 04:40:48 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.56599999999999995}]},{"osd":3,"last update":"Wed Sep 18 04:40:27 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.80100000000000005},"min":{"1min":0.52300000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.75},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.85399999999999998,"15min":0.86599999999999999},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.01,"5min":1.194,"15min":1.194},"last":0.72599999999999998}]},{"osd":4,"last update":"Wed Sep 18 04:40:27 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.83199999999999996,"15min":0.83799999999999997},"min":{"1min":0.63500000000000001,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.121,"5min":1.236,"15min":1.236},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.90700000000000003,"15min":0.92300000000000004},"min":{"1min":0.64300000000000002,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.0720000000000001,"5min":1.268,"15min":1.268},"last":0.84199999999999997}]},{"osd":5,"last update":"Wed Sep 18 04:40:27 2024","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.88800000000000001,"15min":0.90400000000000003},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.1479999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.88},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.91000000000000003,"15min":0.92200000000000004},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.1879999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.95399999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33020,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3627,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T04:41:16.631 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-18T04:41:16.959 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-18T04:41:16.960 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-18T04:41:16.960 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-18T04:41:16.960 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T04:41:17.047 INFO:tasks.ceph.osd.0:Stopped 2024-09-18T04:41:17.048 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-18T04:41:17.048 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T04:41:21.515 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-09-18T04:41:23.150 INFO:tasks.ceph.osd.1:Stopped 2024-09-18T04:41:23.150 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-18T04:41:23.150 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T04:41:27.222 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-09-18T04:41:27.222 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-09-18T04:41:29.252 INFO:tasks.ceph.osd.2:Stopped 2024-09-18T04:41:29.253 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-18T04:41:29.253 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T04:41:32.829 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-09-18T04:41:32.829 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-09-18T04:41:35.355 INFO:tasks.ceph.osd.3:Stopped 2024-09-18T04:41:35.355 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-18T04:41:35.355 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T04:41:35.442 INFO:tasks.ceph.osd.4:Stopped 2024-09-18T04:41:35.443 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-18T04:41:35.443 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T04:41:38.234 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2024-09-18T04:41:38.235 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-09-18T04:41:41.545 INFO:tasks.ceph.osd.5:Stopped 2024-09-18T04:41:41.545 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-18T04:41:41.546 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-18T04:41:41.546 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T04:41:41.577 INFO:tasks.ceph.mgr.x:Stopped 2024-09-18T04:41:41.577 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-18T04:41:41.577 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-18T04:41:41.578 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T04:41:41.616 INFO:tasks.ceph.mon.a:Stopped 2024-09-18T04:41:41.616 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-18T04:41:41.616 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T04:41:41.663 INFO:tasks.ceph.mon.c:Stopped 2024-09-18T04:41:41.663 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-18T04:41:41.663 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T04:41:41.706 INFO:tasks.ceph.mon.b:Stopped 2024-09-18T04:41:41.706 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-18T04:41:41.707 DEBUG:teuthology.orchestra.run.smithi028:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | head -n 1 2024-09-18T04:41:41.729 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi028.front.sepia.ceph.com 2024-09-18T04:41:41.729 DEBUG:teuthology.orchestra.run.smithi028:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-18T04:41:41.947 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi028.front.sepia.ceph.com 2024-09-18T04:41:41.948 DEBUG:teuthology.orchestra.run.smithi028:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-18T04:41:42.246 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi028.front.sepia.ceph.com 2024-09-18T04:41:42.247 DEBUG:teuthology.orchestra.run.smithi028:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-18T04:41:42.568 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi082.front.sepia.ceph.com 2024-09-18T04:41:42.568 DEBUG:teuthology.orchestra.run.smithi082:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-18T04:41:42.739 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi082.front.sepia.ceph.com 2024-09-18T04:41:42.739 DEBUG:teuthology.orchestra.run.smithi082:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-18T04:41:42.997 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi082.front.sepia.ceph.com 2024-09-18T04:41:42.997 DEBUG:teuthology.orchestra.run.smithi082:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-18T04:41:43.184 INFO:tasks.ceph:Archiving mon data... 2024-09-18T04:41:43.185 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-09-17_22:16:03-rbd-reef-distro-default-smithi/7909789/data/mon.a.tgz 2024-09-18T04:41:43.185 DEBUG:teuthology.orchestra.run.smithi028:> mktemp 2024-09-18T04:41:43.192 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/tmp.N2cSLRvmV9 2024-09-18T04:41:43.192 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.N2cSLRvmV9 2024-09-18T04:41:43.236 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2024-09-18T04:41:43.278 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0666 /tmp/tmp.N2cSLRvmV9 2024-09-18T04:41:43.302 DEBUG:teuthology.orchestra.remote:smithi028:/tmp/tmp.N2cSLRvmV9 is 305KB 2024-09-18T04:41:43.364 DEBUG:teuthology.orchestra.run.smithi028:> rm -fr /tmp/tmp.N2cSLRvmV9 2024-09-18T04:41:43.371 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-09-17_22:16:03-rbd-reef-distro-default-smithi/7909789/data/mon.c.tgz 2024-09-18T04:41:43.371 DEBUG:teuthology.orchestra.run.smithi028:> mktemp 2024-09-18T04:41:43.378 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/tmp.0fCYVplyQ7 2024-09-18T04:41:43.378 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.0fCYVplyQ7 2024-09-18T04:41:43.538 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0666 /tmp/tmp.0fCYVplyQ7 2024-09-18T04:41:43.560 DEBUG:teuthology.orchestra.remote:smithi028:/tmp/tmp.0fCYVplyQ7 is 426KB 2024-09-18T04:41:43.620 DEBUG:teuthology.orchestra.run.smithi028:> rm -fr /tmp/tmp.0fCYVplyQ7 2024-09-18T04:41:43.626 DEBUG:teuthology.misc:Transferring archived files from smithi082:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-09-17_22:16:03-rbd-reef-distro-default-smithi/7909789/data/mon.b.tgz 2024-09-18T04:41:43.627 DEBUG:teuthology.orchestra.run.smithi082:> mktemp 2024-09-18T04:41:43.633 INFO:teuthology.orchestra.run.smithi082.stdout:/tmp/tmp.zq0Idy3vvQ 2024-09-18T04:41:43.633 DEBUG:teuthology.orchestra.run.smithi082:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.zq0Idy3vvQ 2024-09-18T04:41:43.743 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 0666 /tmp/tmp.zq0Idy3vvQ 2024-09-18T04:41:43.764 DEBUG:teuthology.orchestra.remote:smithi082:/tmp/tmp.zq0Idy3vvQ is 426KB 2024-09-18T04:41:43.829 DEBUG:teuthology.orchestra.run.smithi082:> rm -fr /tmp/tmp.zq0Idy3vvQ 2024-09-18T04:41:43.836 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-18T04:41:43.836 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-18T04:41:43.838 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-18T04:41:43.840 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-18T04:41:43.857 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-09-18T04:41:43.857 DEBUG:teuthology.orchestra.run.smithi028:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-18T04:41:43.859 DEBUG:teuthology.orchestra.run.smithi082:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-18T04:41:43.862 DEBUG:teuthology.orchestra.run.smithi115:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-18T04:41:43.883 INFO:teuthology.orchestra.run.smithi028.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-18T04:41:43.891 INFO:teuthology.orchestra.run.smithi082.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-18T04:41:43.927 INFO:teuthology.orchestra.run.smithi115.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-18T04:41:43.934 INFO:tasks.ceph:Archiving crash dumps... 2024-09-18T04:41:43.935 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-17_22:16:03-rbd-reef-distro-default-smithi/7909789/remote/smithi028/crash 2024-09-18T04:41:43.936 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-18T04:41:43.954 DEBUG:teuthology.misc:Transferring archived files from smithi082:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-17_22:16:03-rbd-reef-distro-default-smithi/7909789/remote/smithi082/crash 2024-09-18T04:41:43.954 DEBUG:teuthology.orchestra.run.smithi082:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-18T04:41:43.970 DEBUG:teuthology.misc:Transferring archived files from smithi115:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-17_22:16:03-rbd-reef-distro-default-smithi/7909789/remote/smithi115/crash 2024-09-18T04:41:43.971 DEBUG:teuthology.orchestra.run.smithi115:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-18T04:41:43.988 INFO:tasks.ceph:Compressing logs... 2024-09-18T04:41:43.988 DEBUG:teuthology.orchestra.run.smithi028:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-18T04:41:43.990 DEBUG:teuthology.orchestra.run.smithi082:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-18T04:41:43.992 DEBUG:teuthology.orchestra.run.smithi115:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-18T04:41:45.309 INFO:tasks.ceph:Archiving logs... 2024-09-18T04:41:45.309 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-17_22:16:03-rbd-reef-distro-default-smithi/7909789/remote/smithi028/log 2024-09-18T04:41:45.310 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-18T04:41:45.700 DEBUG:teuthology.misc:Transferring archived files from smithi082:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-17_22:16:03-rbd-reef-distro-default-smithi/7909789/remote/smithi082/log 2024-09-18T04:41:45.701 DEBUG:teuthology.orchestra.run.smithi082:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-18T04:41:45.832 DEBUG:teuthology.misc:Transferring archived files from smithi115:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-17_22:16:03-rbd-reef-distro-default-smithi/7909789/remote/smithi115/log 2024-09-18T04:41:45.833 DEBUG:teuthology.orchestra.run.smithi115:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-18T04:41:46.028 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-18T04:41:46.039 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-18T04:41:46.039 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-18T04:41:46.042 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-18T04:41:46.044 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-18T04:41:46.134 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-18T04:41:46.143 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-18T04:41:46.143 DEBUG:teuthology.orchestra.run.smithi028:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-18T04:41:46.146 DEBUG:teuthology.orchestra.run.smithi082:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-18T04:41:46.148 DEBUG:teuthology.orchestra.run.smithi115:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-18T04:41:46.165 INFO:teuthology.orchestra.run.smithi028.stdout: remote refid st t when poll reach delay offset jitter 2024-09-18T04:41:46.165 INFO:teuthology.orchestra.run.smithi028.stdout:============================================================================== 2024-09-18T04:41:46.165 INFO:teuthology.orchestra.run.smithi028.stdout:+hv01.front.sepi 67.205.162.81 3 u 28 64 177 0.071 -0.228 0.895 2024-09-18T04:41:46.165 INFO:teuthology.orchestra.run.smithi028.stdout:+hv02.front.sepi 96.245.170.99 3 u 33 64 177 0.062 -0.574 0.427 2024-09-18T04:41:46.165 INFO:teuthology.orchestra.run.smithi028.stdout:*hv03.front.sepi 74.6.168.72 3 u 35 64 177 0.090 -1.148 0.695 2024-09-18T04:41:46.165 INFO:teuthology.orchestra.run.smithi028.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T04:41:46.166 INFO:teuthology.orchestra.run.smithi115.stdout: remote refid st t when poll reach delay offset jitter 2024-09-18T04:41:46.166 INFO:teuthology.orchestra.run.smithi115.stdout:============================================================================== 2024-09-18T04:41:46.167 INFO:teuthology.orchestra.run.smithi115.stdout:+hv01.front.sepi 67.205.162.81 3 u 26 64 177 0.088 1.944 0.638 2024-09-18T04:41:46.167 INFO:teuthology.orchestra.run.smithi115.stdout:*hv02.front.sepi 96.245.170.99 3 u 29 64 177 0.069 -0.278 1.190 2024-09-18T04:41:46.167 INFO:teuthology.orchestra.run.smithi115.stdout:+hv03.front.sepi 74.6.168.72 3 u 26 64 177 0.061 0.716 1.066 2024-09-18T04:41:46.167 INFO:teuthology.orchestra.run.smithi115.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T04:41:46.169 INFO:teuthology.orchestra.run.smithi082.stdout: remote refid st t when poll reach delay offset jitter 2024-09-18T04:41:46.169 INFO:teuthology.orchestra.run.smithi082.stdout:============================================================================== 2024-09-18T04:41:46.169 INFO:teuthology.orchestra.run.smithi082.stdout:+hv01.front.sepi 67.205.162.81 3 u 26 64 177 0.064 2.099 0.678 2024-09-18T04:41:46.169 INFO:teuthology.orchestra.run.smithi082.stdout:*hv02.front.sepi 96.245.170.99 3 u 24 64 177 0.080 0.303 0.154 2024-09-18T04:41:46.169 INFO:teuthology.orchestra.run.smithi082.stdout:+hv03.front.sepi 74.6.168.72 3 u 29 64 177 0.095 -0.547 0.108 2024-09-18T04:41:46.169 INFO:teuthology.orchestra.run.smithi082.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T04:41:46.170 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-18T04:41:46.179 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-18T04:41:46.179 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-18T04:41:46.187 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-18T04:41:46.198 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-18T04:41:46.207 INFO:teuthology.task.internal:Duration was 769.746836 seconds 2024-09-18T04:41:46.207 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-18T04:41:46.216 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-18T04:41:46.216 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-18T04:41:46.218 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-18T04:41:46.220 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-18T04:41:46.273 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-18T04:41:46.273 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi028.front.sepia.ceph.com 2024-09-18T04:41:46.273 DEBUG:teuthology.orchestra.run.smithi028:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-18T04:41:46.286 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi082.front.sepia.ceph.com 2024-09-18T04:41:46.287 DEBUG:teuthology.orchestra.run.smithi082:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-18T04:41:46.302 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi115.front.sepia.ceph.com 2024-09-18T04:41:46.303 DEBUG:teuthology.orchestra.run.smithi115:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-18T04:41:46.318 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-18T04:41:46.318 DEBUG:teuthology.orchestra.run.smithi028:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-18T04:41:46.321 DEBUG:teuthology.orchestra.run.smithi082:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-18T04:41:46.322 DEBUG:teuthology.orchestra.run.smithi115:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-18T04:41:46.359 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-18T04:41:46.359 DEBUG:teuthology.orchestra.run.smithi028:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-18T04:41:46.399 DEBUG:teuthology.orchestra.run.smithi082:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-18T04:41:46.403 DEBUG:teuthology.orchestra.run.smithi115:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-18T04:41:46.485 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-18T04:41:46.495 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-18T04:41:46.495 DEBUG:teuthology.orchestra.run.smithi028:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-18T04:41:46.509 DEBUG:teuthology.orchestra.run.smithi082:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-18T04:41:46.525 DEBUG:teuthology.orchestra.run.smithi115:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-18T04:41:46.542 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-18T04:41:46.552 DEBUG:teuthology.orchestra.run.smithi028:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-18T04:41:46.554 DEBUG:teuthology.orchestra.run.smithi082:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-18T04:41:46.567 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern = core 2024-09-18T04:41:46.567 DEBUG:teuthology.orchestra.run.smithi115:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-18T04:41:46.580 INFO:teuthology.orchestra.run.smithi082.stdout:kernel.core_pattern = core 2024-09-18T04:41:46.581 INFO:teuthology.orchestra.run.smithi115.stdout:kernel.core_pattern = core 2024-09-18T04:41:46.608 DEBUG:teuthology.orchestra.run.smithi028:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-18T04:41:46.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T04:41:46.644 DEBUG:teuthology.orchestra.run.smithi082:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-18T04:41:46.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T04:41:46.649 DEBUG:teuthology.orchestra.run.smithi115:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-18T04:41:46.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T04:41:46.655 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-18T04:41:46.665 INFO:teuthology.task.internal:Transferring archived files... 2024-09-18T04:41:46.666 DEBUG:teuthology.misc:Transferring archived files from smithi028:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-17_22:16:03-rbd-reef-distro-default-smithi/7909789/remote/smithi028 2024-09-18T04:41:46.666 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-18T04:41:46.694 DEBUG:teuthology.misc:Transferring archived files from smithi082:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-17_22:16:03-rbd-reef-distro-default-smithi/7909789/remote/smithi082 2024-09-18T04:41:46.694 DEBUG:teuthology.orchestra.run.smithi082:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-18T04:41:46.720 DEBUG:teuthology.misc:Transferring archived files from smithi115:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-17_22:16:03-rbd-reef-distro-default-smithi/7909789/remote/smithi115 2024-09-18T04:41:46.721 DEBUG:teuthology.orchestra.run.smithi115:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-18T04:41:46.747 INFO:teuthology.task.internal:Removing archive directory... 2024-09-18T04:41:46.747 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-18T04:41:46.749 DEBUG:teuthology.orchestra.run.smithi082:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-18T04:41:46.759 DEBUG:teuthology.orchestra.run.smithi115:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-18T04:41:46.796 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-18T04:41:46.809 INFO:teuthology.task.internal:Not uploading archives. 2024-09-18T04:41:46.809 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-18T04:41:46.820 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-18T04:41:46.820 DEBUG:teuthology.orchestra.run.smithi028:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-18T04:41:46.822 DEBUG:teuthology.orchestra.run.smithi082:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-18T04:41:46.825 DEBUG:teuthology.orchestra.run.smithi115:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-18T04:41:46.828 INFO:teuthology.orchestra.run.smithi028.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 18 04:41 /home/ubuntu/cephtest 2024-09-18T04:41:46.830 INFO:teuthology.orchestra.run.smithi082.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 18 04:41 /home/ubuntu/cephtest 2024-09-18T04:41:46.832 INFO:teuthology.orchestra.run.smithi115.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 18 04:41 /home/ubuntu/cephtest 2024-09-18T04:41:46.834 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-18T04:41:46.845 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-18T04:41:46.866 INFO:teuthology.run:Summary data: description: rbd/device/{base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{ubuntu_20.04} thrashers/default thrashosds-health workloads/rbd_nbd} duration: 769.7468361854553 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-18T04:41:46.866 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-18T04:41:46.954 INFO:teuthology.run:pass